Hyperparameter Tuning for Federated Learning Speed and Accuracy

Overview

Welcome to the "Hyperparameter Tuning for Federated Learning Speed and Accuracy" project. This project focuses on optimizing hyperparameters in federated learning models to enhance their performance and accuracy. We developed a tool named SEEL (Simulated Easy Efficient Learning) to automatically evaluate hyperparameter configurations within the IBM Federated Learning framework. This project aims to address the challenges associated with federated learning, including ensuring model accuracy across heterogeneous datasets and managing performance overheads.

Project Description

Federated Learning (FL) decentralizes the data used to train machine learning models, moving the training process to edge devices instead of centralizing data on a server. This approach addresses issues related to data privacy, scalability, and practicality. Hyperparameters control the training process, influencing both local and global aspects of the federated learning models.

Key Features

• SEEL Tool: Automatically sweeps and evaluates hyperparameter configurations, providing insights into their impact on model performance.
• IBM Federated Learning Framework: Utilizes IBMFL for training and evaluating models with various hyperparameter settings.
• Focus on Key Hyperparameters: Analyzes the effects of hyperparameters such as max timeout, number of rounds, and termination accuracy.

Methodology

IBM's Python-based Federated Learning Framework

IBM Federated Learning (IBMFL) is an open-source framework supporting multiple deep learning models and fusion algorithms. It allows easy definition of models and hyperparameters, facilitating the tuning process.

Local Hyperparameters

These control the training process at each edge device (party). Examples include:

• Learning Rate: Affects the step size of each iteration.
• Number of Epochs: Determines the number of training passes on local data.
• Termination Accuracy: Specifies the accuracy threshold for terminating training.

Global Hyperparameters

These are managed by the central server (aggregator) and influence the overall training process. Examples include:

• Max Timeout: The maximum time the aggregator waits for parties to complete their training rounds.
• Percent Quorum: The minimum percentage of parties required to complete training before proceeding.
• Number of Rounds: The number of training rounds between parties and the aggregator.

SEEL: Simulated Easy Efficient Learning

SEEL automates hyperparameter tuning by iteratively adjusting configurations and aggregating results. It uses the sed text stream editor, tmux, and other tools to modify configuration files and retrain models efficiently.

Evaluating the Impact of Hyperparameters

We focused on evaluating the following criteria:

• Loss: The value of the loss function indicating model performance.
• Accuracy: The proportion of correctly classified examples.
• Precision Weighted: The weighted average of precision scores for each class, considering class imbalances.

Results & Discussion

Experiments

We used SEEL to evaluate the impact of hyperparameters over a range of values. The search space for hyperparameters included max timeout, number of rounds, and termination accuracy.

Hyperparameter Search Spaces

Hyperparameter	Min Value	Max Value	Increment
Max Timeout	0	200	50
Rounds	2	4	1
Termination Accuracy	0.95	0.99	0.01

Observations

• Accuracy: Higher accuracy is generally achieved with a higher number of rounds and a larger max timeout.
• Precision: Precision varies significantly between different parties, highlighting the heterogeneous nature of federated learning.
• Loss: Lower loss is inversely correlated with higher accuracy, emphasizing the importance of tuning the number of rounds and termination accuracy.

Future Work

Expanding the Hyperparameter Search Space

Future work will involve testing a broader range of hyperparameters and values to gain deeper insights into their impacts on model performance.

Hyperparameter Evaluations

Detailed evaluations of individual hyperparameters will help in understanding their specific effects on federated learning models.

Conclusion

This project presents SEEL, a tool for automatically evaluating the efficacy of hyperparameters in federated learning models using IBM's FL framework. Our experiments demonstrate the impact of hyperparameters on model precision, accuracy, and loss, providing valuable insights for optimizing federated learning processes.

Paper