Foundation Models, especially Large Language Models (LLMs) and Large Multimodal Models (LMMs), have recently attracted a lot of interest in AI/ML research. Being trained using self-supervised learning methods on vast amounts of data, the Foundation Models have shown unprecedented abilities to handle, with minimal adaptation, a wide range of downstream tasks where, traditionally, one used bespoke task-specific machine learning pipelines. Most of the work on Foundation Models has taken place in the context of natural language and image/video, but they hold promise for CPS/IoT applications as well. Conventional machine learning pipelines trained using supervised learning remain the norm for processing sensor data and performing control actions in CPS/IoT. However, they are hungry for difficult-to-obtain annotated data and brittle to domain shifts caused by deployment time variations and dynamics in the physical environment, hardware platforms, and human subjects. Moreover, CPS-IoT applications often require higher-level spatiotemporal reasoning over sensory data, planning of actions, and meeting real-world regulations, all challenging tasks for task-specific models trained on limited data.
Foundation Models offer an opportunity to address these challenges in CPS-IoT via large-scale pooling of abundant but unlabeled sensory data from IoT devices, development of standardized multimodal embeddings for various sensor types, and training on diverse platform-task combinations to create reusable models with good generalization capabilities across tasks and deployments. However, the Foundation Models must also cope with challenges unique to the CPS/IoT domain, such as dynamic environments, resource-constrained platforms, the impact of inference latency, appropriate handling of signal-level concepts such as sampling rates, etc. FMSys Workshop will explore challenges and opportunities relating to Foundation Models in CPS-IoT systems. FMSys welcomes submissions from researchers from academia and industry that explore the latest developments in this area and encourages encompassing diverse domains, including sensing, systems, security, and applications.
Topics of interest include (but are not limited to)
- Foundation models on sensor data analytics, e.g., acoustic, light, motion, RF sensor.
- Benchmark datasets/evaluations of foundation models on sensor data.
- Applications of foundation models in healthcare, smart city, virtual reality, social media, etc..
- Foundation models for reasoning, planning, and control.
- Efficient fine-tuning and inference of foundation models for CPS/IoT applications.
- Novel machine learning (e.g., federated and multi-modal learning) systems based on foundation models.
- Foundation models for human-computer interaction or human-AI collaboration.
- Trending study of Penetrative AI that uses Large Language Models (LLMs) to support CPS/IoT applications.
- Trustworthiness of foundation models.