Privacy in Gossip Learning - A study of privacy risks and defenses in Gossip Learning

Abstract

Gossip learning is a fully decentralized machine learning method where gradient information are exchanged peer-to-peer during training, improving scalability and avoiding raw data sharing. However, especially in networks where gradients are directly shared, adversarial nodes can still reconstruct original data with high fidelity using gradient inversion. This thesis investigates the privacy vulnerabilities of gossip learning, evaluates existing defenses, and proposes a novel approach. Replacing gradient sharing with model state sharing was found to mitigate the risk of attackers performing gradient inversion. Model sharing requires adversaries to first compute gradients from successive model states, which is difficult unless a majority of nodes collaborate in the attack. However, data reconstruction is still possible duringinitialization and near convergence. Thus, gossip learning is not inherently private even with model sharing. To counter these risks, both existing and new defenses have been tested. Differential privacy provides sufficient protection but leads to a loss in model accuracy, making it a suitable baseline defense. This work introduces the use of homogeneous batches as a novel defense, providing partial protection for all data points without affecting accuracy at all. In combination with differential privacy, complete protection against inversion attacks was achieved with a reduced performance degradation compared to the baseline defense alone. While the results are specific to the experimental setup, they suggest this defense has an improved privacy-accuracy tradeoff.