NJU International Fellowship Initiative Lecture

Reproducible Research Is Both Hard and Helpful: A Case Study on Federated Learning

Time: June 3, 2025 (Tuesday) 10.00 a.m.

Location: 229, Computer Science Building, Xianlin Campus

Speaker:

Baochun Li received his Ph.D. degree from the University of Illinois at Urbana-Champaign, Urbana, in 2000. Since 2000, he has been with the University of Toronto, where he is currently a Professor. He holds the Bell Canada Endowed Chair in Computer Engineering since August 2005. He has co-authored more than 480 research papers, with a total of over 27,000 citations, an H-index of 89 and an i10-index of 356, according to Google Scholar Citations. He was the recipient of the IEEE Communications Society Leonard G. Abraham Award in the Field of Communications Systems in 2000, the Multimedia Communications Best Paper Award from the IEEE Communications Society in 2009, the University of Toronto McLean Award in 2009, the Best Paper Award from IEEE INFOCOM in 2023, and the IEEE INFOCOM Achievement Award in 2024. He is a Fellow of the Canadian Academy of Engineering, a Fellow of the Engineering Institute of Canada, and a Fellow of IEEE.

Abstract:

Modern networking and distributed systems are highly complex, and research on these systems involves a large number of moving parts, making research results much less likely to be reproducible. However, reproducible research is not just a gem waiting to be discovered, but a necessary characteristic to move a research field forward. In this talk, I will discuss some of my experiences over the past three years on why reproducible research is hard, with a focus on federated learning, a popular decentralized paradigm that allows a large number of clients to collaboratively train shared machine learning models. There have been thousands of papers on federated learning published each year recently, including multiple sessions in recent INFOCOM technical programs. Yet, many of the proposed algorithms have not been benchmarked, compared, and fairly evaluated in the same open-source benchmarking framework at a reasonable scale, and their performance claims have not been evaluated in a fair and reproducible comparison study. Using actual code from Plato—a new open-source framework that I developed from scratch—as examples, I will show why this is the case from a technical point of view, and how much existing claims in the literature over most major research topics have deviated substantially from our experiences in Plato. In conclusion, I will present several guiding principles and recent technological advances that can make research more reproducible, and advocate for more benchmarking platforms that allow research ideas to be compared fairly under the same roof.