Page 78 - ITU Journal, ICT Discoveries, Volume 3, No. 1, June 2020 Special issue: The future of video and immersive media

ITU Journal: ICT Discoveries, Vol. 3(1), June 2020

Table 1 – Communication characteristics of different Embedded ML pipelines and popular respective compression techniques used in the
literature to reduce communication.
On-Device Distributed Federated Peer-to-Peer
Inference Training Learning Learning
Communication:
trained models/ models/ models/
• Objects model gradients
model updates model updates model updates
server all clients some clients all clients
• Flow
→ clients → all clients ↔ server → some clients
• Frequency low high medium high
• Redundancy low high medium low

Compression Techniques:
• Trained Compression:
→ Pruning [28][74][84] - [45] -
→ Trained Quantization [77][28][80] - [45] -
→ Distillation [31] - - -
• Lossy Compression:
→ Quantization [18][17] [3][78][75][8] [45][11][59] [55] [44]
→ Sparsification - [49][2] [59][45][11] [44]
→ Sketching - [35] [47] [35]
→ Low-Rank Approx. - [72] [45] -
• Error Accumulation - [49][63][39] [58] [66]
• Communication Delay - [86][62][58] [52] [76]
• Loss-Less Compression [79][81] [58] [59] -

2.2 Federated Learning the distribution of data among the clients will usually
be “non-iid” meaning that any particular user’s local
Federated learning [52][48][37] allows multiple parties to dataset will not be representative of the whole distri-
jointly train a neural network on their combined data, bution. The amount of local data is also typically un-
without having to compromise the privacy of any of the balanced among clients, since different users may make
participants. This is achieved by iterating over multi- use of their device or a specific application to a differ-
ple communication rounds of the following three step ent extent. Many scenarios are imaginable in which the
protocol:
total number of clients participating in the optimiza-
(1) The server selects a subset of the entire client pop- tion is much larger than the average number of training
ulation to participate in this communication round data examples per client. The intrinsic heterogeneity
and communicates a common model initialization of client data in federated learning introduces new chal-
to these clients. lenges when it comes to designing (communication effi-
cient) training algorithms.
(2) Next, the selected clients compute an update to the A major issue in federated learning is the massive com-
model initialization using their private local data. munication overhead that arises from sending around
the model updates. When naively following the feder-
(3) Finally, the participating clients communicate their ated learning protocol, every participating client has to
model updates back to the server where they are ag- download and upload a full model during every train-
gregated (by e.g. an averaging operation) to create ing iteration. Every such update is of the same size as
a new master model which is used as the initializa- the full model, which can be in the range of gigabytes
tion point of the next communication round.
for modern architectures with millions of parameters.
Since private data never leaves the local devices, feder- At the same time, mobile connections are often slow,
ated learning can provide strong privacy guarantees to expensive and unreliable, aggravating the problem fur-
the participants. These guarantees can be made rigor- ther.
ous by applying homomorphic encryption to the com- Compression for Federated Learning: The most
municated parameter updates [9] or by concealing them widely used method for reducing communication over-
with differentially private mechanisms [24]. head in federated learning (see Table 1) is to delay syn-
Since in most federated learning applications the train- chronization by letting the clients perform multiple local
ing data on a given client is generated based on the updates instead of just one [38]. Experiments show that
specific environment or usage pattern of the sensor, this way communication can be delayed for up to multi-

56 © International Telecommunication Union, 2020