The Impact of Duplicate Data on Federated Learning: An Overlooked Issue in Data Quality and Privacy-Preserving Governance
Wei Shao, Associate Research Professor, Shandong Computer Science Center (National Supercomputing Jinan Center), Qilu University of Technology (Shandong Academy of Sciences), China
Abstract:
Federated learning enables collaborative model training across diverse clients without directly sharing raw data,
and has become an important paradigm for privacy-preserving data collaboration in real-world applications. However,
most existing research focuses on model optimization and secure aggregation, while the quality of data before training has
received much less attention.
In real-world scenarios, cross-client data redundancy and overlaps are prevalent due to
cross-platform synchronization or repeated user registration. Such data redundancy is far
from a negligible preprocessing issue and can induce systematic drawbacks to federated
learning pipelines. This talk will first discuss how duplicate data affects federated learning
in terms of training efficiency and model effectiveness, and then further present our
series of privacy-preserving decentralized deduplication solutions for different privacy
and efficiency requirements. These methods enable secure cross-client redundancy
detection while preserving data locality, and provide a practical basis for privacy-aware
data quality governance before training, thereby enhancing the performance of federated
learning.
CV:
Dr. Wei Shao currently works as an associate research professor at Shandong Computer Science Center (National Supercomputer Center in Jinan),
Qilu University of Technology (Shandong Academy of Sciences) in China. She leads and participates in multiple national and provincial research projects
including the National Key R&D Programs, and the National Natural Science Foundation projects. She has published 20+ high-impact papers in top
international journals and conferences, and possesses 10+ national invention patents. Her research focuses on applied cryptography, blockchain security,
privacy-preserving digital identity and trustworthy federated learning. She has proposed multiple innovative schemes including user-controlled anonymous digital identities,
auditable revocable permissioned blockchain mechanisms, and blockchain-based privacy-preserving federated learning frameworks.