Communication Efficiency for Federated Learning in IoT Systems

Abstract

The use of Artificial Intelligence (AI) increases every day in today’s society. Traditionally, AI models are trained on a central server using raw data collected from participants. However, centralizing all data raises privacy concerns, prompting the development of distributed machine learning (ML) approaches. One such approach is Federated Learning (FL), where raw training data remains on local devices, and only model updates are sent to the server for aggregation. While this shift towards distributed ML improves privacy, it introduces a new significant challenge: communication efficiency, which is the focus of this project. This project investigates strategies to improve communication efficiency in FL. It studies the trade-off between model accuracy and communication cost when using different algorithms: Quantization for sketched updates, Low-Rank Adaptation (LoRA) for structured updates, and Communication-Mitigated FL for selective updates (CMFL). The experiments are conducted on local hardware representative of Internet of Things (IoT) edge devices, specifically 30 Raspberry Pi 4 units. Through graphs and tables, we demonstrate the differences of the algorithms using different distributions, and using different training datasets. The results show that using quantization leads to a big reduction in the communication cost, with almost no change to the accuracy. LoRA shares a similar result in communication cost; however, with a larger percentage loss in the accuracy. LoRA is also more efficient in terms of Central Processing Unit (CPU) usage for the participants; however, it also increases the memory usage. CMFL showed potential in filtering out bad updates; however, it comes with a need of fine tuning. In conclusion, there are algorithms that reduce the communication cost while maintaining a high accuracy. Depending on what the model’s purpose is and the model’s training data, different algorithms yield different results in terms of accuracy, CPU usage, and average memory usage. Therefore, it is important to select the algorithm based on the intended use of the model.