FedPromo: Federated Lightweight Proxy Models At The Edge for Fine-Grained Image Classification with Foundation Models

Federated learning (FL) is an established paradigm for training deep learning models on decentralized data, particularly relevant in Internet of Things (IoT) scenarios. However, as model sizes grow, conventional FL approaches require significant computational resources, which may not be feasible for resource-constrained IoT devices. We introduce FedPromo, a novel framework that enables efficient adaptation of classification heads trained in a federated way to large-scale foundation models (FMs) stored on a central server without requiring explicit data sharing. Instead of directly training the large model on client devices, FedPromo optimizes lightweight proxy models via FL, reducing computational overhead, energy consumption, and bandwidth usage while maintaining privacy. We evaluate our method in cross-domain fine-grained image classification via a two-stage process: server-side knowledge distillation (KD) aligns representations of a large-scale FM with those of a compact counterpart, then the compact model encoder is deployed to devices for local classifier learning. These classifiers are aggregated and transferred back to the FM, enabling learning of fine-grained representations without direct access to user data. Through novel regularization strategies, our framework enables decentralized multidomain learning, balancing performance, privacy, and resource efficiency for wide-scale IoT deployment. Extensive experiments on five image classification benchmarks and feature representation analysis demonstrate that FedPromo outperforms existing methods when employing mobile-targeted efficient architectures, making it suitable for IoT devices such as smart home devices, industrial sensors, and phones. Across domains, FedPromo enjoys average gains of 4.8% (9.1%) top-1 (top-5) accuracy on the clients and 7.5% (9.7%) on the server with respect to the best competitors.