Foundation Models, especially Large Language Models (LLMs) and Large Multimodal Models (LMMs), have recently attracted a lot of interest in AI/ML research. Being trained using self-supervised learning methods on vast amounts of data, the Foundation Models have shown unprecedented abilities to handle, with minimal adaptation, a wide range of downstream tasks where, traditionally, one used bespoke task-specific machine learning pipelines. Most of the work on Foundation Models has taken place in the context of natural language and image/video, but they hold promise for CPS/IoT applications as well. Conventional machine learning pipelines trained using supervised learning remain the norm for processing sensor data and performing control actions in CPS/IoT. However, they are hungry for difficult-to-obtain annotated data and brittle to domain shifts caused by deployment time variations and dynamics in the physical environment, hardware platforms, and human subjects. Moreover, CPS-IoT applications often require higher-level spatiotemporal reasoning over sensory data, planning of actions, and meeting real-world regulations, all challenging tasks for task-specific models trained on limited data.
Foundation Models offer an opportunity to address these challenges in CPS-IoT via large-scale pooling of abundant but unlabeled sensory data from IoT devices, development of standardized multimodal embeddings for various sensor types, and training on diverse platform-task combinations to create reusable models with good generalization capabilities across tasks and deployments. However, the Foundation Models must also cope with challenges unique to the CPS/IoT domain, such as dynamic environments, resource-constrained platforms, the impact of inference latency, appropriate handling of signal-level concepts such as sampling rates, etc. FMSys Workshop will explore challenges and opportunities relating to Foundation Models in CPS-IoT systems. FMSys welcomes submissions from researchers from academia and industry that explore the latest developments in this area and encourages encompassing diverse domains, including sensing, systems, security, and applications.
- Foundation models on sensor data analytics, e.g., acoustic, light, motion, RF sensor.
- Benchmark datasets/evaluations of foundation models on sensor data.
- Applications of foundation models in healthcare, smart city, virtual reality, social media, etc..
- Foundation models for reasoning, planning, and control.
- Efficient fine-tuning and inference of foundation models for CPS/IoT applications.
- Novel machine learning (e.g., federated and multi-modal learning) systems based on foundation models.
- Foundation models for human-computer interaction or human-AI collaboration.
- Trending study of Penetrative AI that uses Large Language Models (LLMs) to support CPS/IoT applications.
- Trustworthiness of foundation models.