• Docs >
  • Benchmarking Results for MPI-based federated learning
Shortcuts

Benchmarking Results for MPI-based federated learning

Please visit the following link to check the latest benchmark experimental results: https://app.wandb.ai/automl/fedml/reports/FedML-Benchmark-Experimental-Results–VmlldzoxODE2NTU FedML white paper (https://arxiv.org/pdf/2007.13518.pdf) also summarizes the dataset list and related benchmarks. We refer the hyper-parameters and reproduce results from many top-tier ML conferences. Please check details of our reference hyperparameters as follows.

Linear Models

Data

Model

Alg

Partition

#C

#C_p

bs

c_opt

lr

e

#R

acc

MNIST

LR

FedAvg

Power Law

1000

10

10

SGD

0.03

1

>100

>75

Federated EMNIST

LR

FedAvg

Power Law

200

10

10

SGD

0.003

1

>200

10~40

Synthetic(α,β)

LR

FedAvg

Power Law

30

10

10

SGD

0.01

1

>200

>60

Note: #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch_size; c_opt = client optimizer; e = epoch; #R = number of rounds; acc = accuracy. For Synthetic(α,β), (α,β) is chosen from (0,0), (0.5,0.5), (1,1)

  • MNIST – Logistic Regression – FedAvg

    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’

    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’

    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation

    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description

    • comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10

    • accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10

  • Federated EMNIST – Logistic Regression-FedAvg

    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’

    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’

    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation

    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description

    • comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10

    • accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10

  • Synthetic(α,β) – Logistic Regression -FedAvg

    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’

    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’

    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation

    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’

    • epochs: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Hyperparameters & evaluation metrics’

    • comm_round: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6

    • accuracy: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6

Lightweight and shallow neural network models

Task

Data Set

Model

Alogrithm

Partition Method

Partition Alpha

client_num_in_total

client_num_per_round

batch_size

client_optimizer

lr

wd

epochs

comm_round

accuracy

CV

Federated EMNIST

CNN (2 Conv + 2 FC)

FedAvg

Power Law

3400

10

20

SGD

0.1

-

1

>1500

84.9

CV

CIFAR-100

ResNet-18+group normalization

FedAvg

Pachinko Allocation

100/500(ex/cli)

500

10

20

SGD

0.1

-

1

>4000

44.7

NLP

Shakespeare

RNN (2 LSTM + 1 FC)

FedAvg

realistic patition

715

10

4

SGD

1

-

1

>1200

56.9

NLP

StackOverflow

RNN (1 LSTM + 2 FC)

FedAvg

Pachinko Allocation

342477

50

16

SGD

pow(10,-0.5)

-

1

>1500

19.5

  • Federated EMNIST-CNN-FedAvg (https://openreview.net/pdf?id=LkFG3lB13U5)

    • Patition Method: ‘Adaptive federated optimization’ (https://openreview.net/pdf?id=LkFG3lB13U5), page 23, Appendix C.2

    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2

    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7

    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1

    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8

    • wd (learning rate decay): ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 2

    • epochs: ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 1

    • comm_round:‘Adaptive federated optimization’, page28, Appendix E.1, figure 3

    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1

  • CIFAR-100 – ResNet18 -FedAvg

    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 3

    • Patition_alpha: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 2

    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2

    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7

    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1

    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8

    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1

    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1

  • Shakespeare – RNN – FedAvg

    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.3

    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2

    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7

    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1

    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8

    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1

    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1

  • StackOverflow – RNN – FedAvg

    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.4, Paragraph 2

    • client_num_in_total: ‘Adaptive federated optimization’, page 25, Appendix C.4, Paragraph 1

    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7

    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1

    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8

    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’

    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1

    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1

Benchmarking using modern DNNs

Data

Model

Alg

# C

# C_p

bs

c_opt

lr

wd

e

round

IID acc

non-IID acc

CIFAR10

ResNet-56

FedAvg

10

10

64

SGD

0.001

0.001

20

100

93.19

87.12

CIFAR100

ResNet-56

FedAvg

10

10

64

SGD

0.001

0.001

20

100

68.91

64.70

CINIC10

ResNet-56

FedAvg

10

10

64

SGD

0.001

0.001

20

100

82.57

73.49

CIFAR10

MobileNet

FedAvg

10

10

64

SGD

0.001

0.001

20

100

91.12

86.32

CIFAR100

MobileNet

FedAvg

10

10

64

SGD

0.001

0.001

20

100

55.12

53.54

CINIC10

MobileNet

FedAvg

10

10

64

SGD

0.001

0.001

20

100

79.95

71.23

Note: Non-IID distribution is set using LDA ( LDA = Latent Dirichlet Allocation) with alpha = 0.5; #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch size; c_opt = client optimizer.