​​​​​​​​​Title​

Trends and advancements in deep neural network communication​​​​

Abstract

Due to their great performance and scalability properties neural networks have become ubiquitous building
blocks of many applications. With the rise of mobile and IoT, these models now are also being increasingly applied
in distributed settings, where the owners of the data are separated by limited communication channels and privacy
constraints. To address the challenges of these distributed environments, a wide range of training and evaluation
schemes have been developed, which require the communication of neural network parametrizations. These novel
approaches, which bring the “intelligence to the data” have many advantages over traditional cloud solutions such as
privacy preservation, increased security and device autonomy, communication efficiency and a greater training speed. This paper gives an overview on the recent advancements and challenges in this new field of research at the intersection of machine learning and communications.

Keywords

Distributed training, federated learning, model compression, neural networks, on-device inference

Author​s

​​​Felix Sattler  
Fraunhofer HHI, Germany 
​ ​
Felix Sattler received an M.Sc. degree in computer science, a M.Sc. degree in applied mathematics and a B.Sc. degree in Mathematics all from Technische Universität Berlin. He is currently with the Machine Learning Group, Fraunhofer Heinrich Hertz Institute, Berlin, Germany. His research interests include distributed machine learning, neural networks and multi-task learning.​

​
​

​​​Thomas Wiegand 
Fraunhofer HHI and Technische Universität ​Berlin​,Ger​many 

Thomas Wiegand is a professor in the department of Electrical Engineering and Computer Science at the Technical University of Berlin and is jointly heading the Fraunhofer Heinrich Hertz Institute, Berlin, Germany. He received the Dipl.-Ing. degree in Electrical Engineering from the Technical University of Hamburg-Harburg, Germany, in 1995 and the Dr.-Ing. degree from the University of Erlangen-Nuremberg, Germany, in 2000. Since 1995, he has been an active participant in standardization for multimedia with many successful submissions to ITU-T and ISO/IEC. In 2000, he was appointed as the Associated Rapporteur of ITU-T VCEG and from 2005-2009, he was Co-Chair of ISO/IEC MPEG Video. The projects that he co-chaired for the development of the H.264/MPEG-AVC standard have been recognized by an ATAS Primetime Emmy Engineering Award. He was also a recipient of a ATAS Primetime Emmy Engineering Award for the development of H.265/MPEG-HEVC and a pair of NATAS Technology & Engineering Emmy Awards. For his research in video coding and transmission, he received numerous awards including the Vodafone Innovations Award, the EURASIP Group Technical Achievement Award, the Eduard Rhein Technology Award, the Karl Heinz Beckurts Award, the IEEE Masaru Ibuka Technical Field Award, and the IMTC Leadership Award. He received multiple best paper awards for his publications. Si​nce 2014, Thomson Reuters named him in their list of “The World’s Most Influential Scientific Minds” as one of the most cited researchers in his field. He is a recipient of the ITU150 Award. He has been elected to the German National Acdemy of Engineering (Acatech) and the National Academy of Science (Leopoldina). Since 2018, he has been appointed the chair of the ITU/WHO Focus Group on Artificial Intelligence for Health.

Fraunhofer HHI, Germany 
​ 
Wojciech Samek founded and is heading the Machine Learning Group at Fraunhofer Heinrich Hertz Institute since 2014. He studied computer science at Humboldt University of Berlin, Heriot-Watt University and University of Edinburgh from 2004 to 2010 and received the Dr. rer. nat. degree with distinction (summa cum laude) from the Technical University of Berlin in 2014. In 2009 he was visiting researcher at NASA Ames Research Center, Mountain View, CA, and in 2012 and 2013 he had several short-term research stays at ATR International, Kyoto, Japan. He was awarded scholarships from the European Union's Erasmus Mundus programme, the Studienstiftung des deutschen Volkes and the DFG Research Training Group GRK 1589/1. He is PI at the Berlin Institute for the Foundation of Learning and Data (BIFOLD), member of the European Lab for Learning and Intelligent Systems (ELLIS) and associated faculty at the DFG graduate school BIOQIC. Furthermore, he is an editorial board member of Digital Signal Processing, PLOS ONE and IEEE Transactions on Neural Networks and Learning Systems and an elected member of the IEEE MLSP Technical Committee. He is part of various international standardization initiatives, including the MPEG AHG on Compression of Neural Networks for Multimedia Content Description and Analysis, and was organizer of special sessions, workshops and tutorials at top-tier machine learning and signal processing conferences (NIPS, ICML, CVPR, ICASSP, MICCAI), has received multiple best paper awards, and has authored more than 100 journal and conference papers, predominantly in the areas deep learning, interpretable machine learning, neural network compression and federated learning.

 

​