3rd International Workshop on
Testing Distributed Internet of Things Systems

Held in conjunction with ACM EuroSys 2025 and ACM ASPLOS 2025

March 31, 2025, in Rotterdam, the Netherlands

The 3rd International Workshop on Testing Distributed Internet of Things Systems (TDIS) again brought together computer systems researchers and practitioners who focus on benchmarking, measuring, and evaluating distributed IoT systems across the edge-cloud continuum to provide a forum for ongoing work presentations and discussions.

Many thanks!

The third TDIS workshop has been held on March 31, 2025, with over 20 participants. We were very pleased with the excellent invited talks, the interesting paper presentations, and the engaged discussions.

We are most grateful to all authors and programme committee members for their invaluable contributions to the workshop. Again, a big Thank You to all participants!

Call for Papers

The Internet of Things (IoT), cloud computing, and machine learning will allow more adaptive and responsive services for our cities, houses, and infrastructures. Yet, this vision of intelligent cyber-physical systems will not be implemented with centralized cloud resources alone. Such resources are simply too far away from sensor-equipped IoT devices, leading to high latencies, bandwidth bottlenecks, and unnecessary energy consumption. Additionally, there are often privacy and safety requirements mandating distributed architectures. Therefore, new distributed computing system architectures have emerged with edge and fog computing, bringing computing and storage closer to devices to realize in situ IoT services and in some cases extend AI deployment right to the edge of the network.

The resulting heterogeneous, distributed, and dynamic environments make it difficult to implement dependable, performant, and efficient IoT systems. Furthermore, it is far less clear how IoT systems can be sufficiently tested before they are deployed to control our cities, homes, and infrastructures. Yet, continuous testing in realistic test environments is essential for such systems. For instance, IoT systems might be deployed to continuously optimize the operation of critical urban infrastructures, including public transport systems, energy grids, water networks, and medical services. New versions of such IoT systems must be evaluated thoroughly before they can be deployed and relied on. Furthermore, the behavior of such IoT systems must be tested under the expected computing environment conditions, including variations of such conditions, given the inherently unsteady nature of IoT environments.

TDIS 2025 will provide a forum for ongoing work presentations and discussions on topics such as test frameworks, hybrid testbeds, (co-)simulation, testing automation and methodologies, monitoring, benchmarking, metrics, resilience testing, energy and emissions testing, data integrity and bias testing, test tool usability, as well as IoT applications and empirical evaluations. We welcome submissions that describe initial ideas and visions just as much as reports on preliminary results, practical tools, and completed projects.

Topics of Interest

Topics of interest include but are not limited to:

  • Physical and hybrid IoT testbeds
    • Heterogeneous and distributed IoT device testbeds
    • Containerization and virtualization in IoT testing
    • Scalability and flexibility in testbed design for diverse IoT scenarios
  • Emulation and (co-)simulation test frameworks
    • Emulation frameworks that mimic real-world conditions
    • Simulating realistic network conditions
    • Co-simulation of IoT environment conditions and application domains
  • Testing practices and methodologies
    • Integrating IoT testing in continuous deployment pipelines
    • Automated and AI-based testing across IoT-edge-cloud environments
    • Representativeness, reproducibility, and repeatability of experiments, benchmarks, and tests in edge/fog/cloud computing environments
  • Monitoring, benchmarking, and metrics
    • (Low overhead) monitoring, tracing, and profiling for distributed IoT systems
    • Benchmark suites for evaluating IoT performance
    • Metrics for assessing performance, dependability, or usability of IoT applications
  • Testing energy efficiency, carbon awareness, and resource usage
    • Techniques for measuring or modeling energy consumption in IoT systems
    • Test frameworks, testbeds, and datasets for evaluating carbon-aware computer systems
    • Testing of resource usage and resource management strategies in edge-to-cloud computing environments
  • Load and resilience testing
    • Load/stress testing to ensure system robustness
    • Security testing and resilience assessment in connected devices
    • Techniques for automated failure injection in distributed IoT systems (e.g. implementing chaos engineering practices)
  • Testing data quality assurance and privacy preservation techniques
    • Assessing methods to ensure data integrity and authenticity in IoT communications
    • Testing the detection and mitigation of biases in data-driven IoT applications
    • Evaluating privacy-preserving techniques in IoT data collection and analysis
  • Usability of testbeds, testing frameworks, benchmarks, and experimentation tools
    • Evaluating interfaces of IoT systems
    • User-centric evaluation methodologies for IoT applications
    • Incorporating user feedback into IoT system development and testing
  • Empirical evaluations of IoT applications
    • Empirical analysis of IoT system performance and user experiences
    • Lessons learned from implementing and/or deploying large-scale IoT systems in real-world settings
    • Cross-disciplinary studies that integrate IoT technologies into specific domains

Important Dates

  • workshop paper submission deadline: January 24 February 4, 2025 (AoE) plus a 72-hour grace period (final)
  • notification of acceptance: February 21, 2025
  • camera-ready submission: March 7, 2025
  • workshop: March 31, 2025

Program

The workshop was held on March 31, starting at 14:00h CEST (after the clocks will spring forward in the night before) in Rotterdam, the Netherlands, as part of the EuroSys and ASPLOS 2025 workshops. The program is planned as follows:

  • 14:00 – 14:10 Welcome
  • 14:10 – 14:50 Keynote #1

    • Beyond Accuracy: Revealing the True Capabilities of Prediction Models Used for Cloud Resource Management Through Systematic Experimentation. Thaleia Dimitra Doudali, Assistant Professor at the IMDEA Software Institute in Madrid, Spain

      Abstract: Cloud resource management plays a crucial role in ensuring that computing resources such as CPU, memory, storage, and networking are allocated efficiently to meet user demand while minimizing costs. Predictive models are widely used in cloud environments to forecast workloads, optimize resource allocation, and improve resource savings. Traditionally, metrics like prediction accuracy have been used to assess the effectiveness of these models in resource management. However, our research demonstrates that accuracy alone can be misleading. Through visualization, we found cases where machine learning models weren’t truly “learning,” and simple history-based methods were just as effective. Additionally, in a comparative analysis of production-level predictors, we observed that static system thresholds limited model usage to just 6% of the time. In conclusion, this talk emphasizes the importance of systematically testing and evaluating how prediction models are integrated into cloud resource management. We advocate for moving beyond traditional metrics to ensure that the models are genuinely adding value, rather than merely offering seemingly accurate but ultimately ineffective predictions.

      Speaker: Dr. Thaleia Dimitra Doudali is an Assistant Professor at the IMDEA Software Institute in Spain. She holds a Ph.D. in Computer Science from Georgia Tech, USA, and an undergraduate degree in Electrical and Computer Engineering from NTUA, Greece. Her research focuses on leveraging methodologies such as machine learning to enhance computer system-level resource management for emerging computing platforms. She has received the 2024 César Nombela and 2021 Juan de la Cierva award grants and was recognized as a Rising Star in EECS in 2020. Beyond her research, Thaleia is a strong advocate for diversity in STEM through her active participation in the NGO ‘Greek Women in STEM’ and the ‘MAR’ chapter of Women in HPC.

  • 14:50 – 15:30 Paper Session #1 (full papers: 15min presentation followed by 5min Q/A)
    • Reckon-ing Kubernetes at the Edge using Emulated Clusters (full paper). Alessandro Sassi (University of Cambridge / Politecnico di Milano), Christopher Jensen (University of Cambridge / Microsoft Research), and Richard Mortier (University of Cambridge).
    • AtlasFL: A Federated Learning Workload Generator with Energy and Carbon Emission Support (full paper). Moysis Symeonides (University of Cyprus, Cyprus) and Demetris Trihinas (University of Nicosia, Cyprus).
  • 15:30 – 16:00 Coffee break
  • 16:00 – 16:40 Keynote #2

    • From Devices to DDoS: Methods for Measuring IoT Exploitation. Harm Griffioen, Assistant Professor at the Delft University of Technology, the Netherlands

      Abstract: The increasing integration of Internet of Things (IoT) devices into everyday life has introduced numerous opportunities, but also significant security risks. IoT devices, often with limited resources and weak security measures, can easily become targets for exploitation, including being co-opted into botnets or used in Distributed Denial of Service (DDoS) attacks.
      In this keynote, Harm Griffioen, Assistant Professor at Delft University of Technology, will explore a range of methodologies to identify and measure the vulnerabilities and failures of IoT devices when they are hijacked for malicious purposes. Drawing from his expertise in network security and cyber threat intelligence, he will discuss techniques for detecting compromised IoT devices, measuring their behavior in botnets, and analyzing their role in large-scale DDoS attacks.

      Speaker: Dr. Harm Griffioen is an Assistant Professor specializing in Cyber Security at Delft University of Technology in the Netherlands. His research interests encompass network security, cyber threat intelligence, and the security of distributed Internet of Things (IoT) systems. He has contributed to understanding and mitigating security threats in IoT environments, including analyzing large-scale botnet attacks like Mirai. His work emphasizes the development of robust testing methodologies to ensure the reliability and security of distributed IoT systems across the edge-cloud continuum. Next to his academic career, he has worked as a Cyber Threat Analyst.

  • 16:40 – 17:30 Paper Session #2 (full papers: 15min presentation followed by 5min Q/A; short papers: 10min presentation followed by 5min Q/A)
    • LoRaLive: Efficient LoRaWAN Traffic Generation (full paper). Vadim Safronov and Richard Mortier (University of Cambridge).
    • Towards A Modular End-To-End Machine Learning Benchmarking Framework (short paper). Robert Bayer, Ties Robroek, and Pinar Tozun (IT University of Copenhagen).
    • On-Demand Network QoS Adaptation for Automated Tele-Operated Driving (short paper). Belma Turkovic, Rintse van de Vlasakker, Srinath Potnuru, Ramon S. Schwartz, Lucia D’Acunto (TNO), Oliver Held, Stepan Karasek (Roboauto), Rakshith Kusumakar, Andrea Cremona, and Raana Fardanian (V-Tron).

Workshop Organizers

Workshop Chairs

Program Committee

  • David Bermbach, Technische Universität Berlin, Germany
  • Nicolae Cleju, Technical University of Iași, Romania
  • Abdessalam Elhabbash, University of Lancaster, UK
  • Andreas Dionysiou, Frederick University, Cyprus
  • Saeid Ghafouri, Queen’s University Belfast, UK
  • Paul Harvey, University of Glasgow, UK
  • Shashikant Ilager, University of Amsterdam, the Netherlands
  • Panagiotis Ilia, Cyprus University of Technology, Cyprus
  • Aleksandar Ilic, University of Lisbon, Portugal
  • Vasileios Karakostas, University of Athens, Greece
  • Iacovos G. Kolokasis, ICS-FORTH, Greece
  • Angelos Marnerides, University of Cyprus, Cyprus
  • Fotis Nikolaidis, SuperDuper GmbH, Greece
  • Lukas Pirl, Hasso Plattner Institute, University of Potsdam, Germany
  • Sri Pramodh Rachuri, Stony Brook University, New York, USA
  • Philipp Raith, Vienna University of Technology, Austria
  • Thomas Rausch, LocalStack, Austria
  • Qun Song, Singapore University of Technology and Design, Singapore
  • Tom Spink, University of St Andrews, UK
  • Moysis Symeonides, University of Cyprus, Cyprus
  • Theodoros Toliopoulos, Aristotle University of Thessaloniki, Greece
  • Amjad Ullah, Edinburgh Napier, UK
  • Eleni Tzirita Zacharatou, IT University of Copenhagen, Denmark
  • Steffen Zeuch, DFKI, Germany

Previous Workshop Editions

  • Our 1st workshop edition, TDIS 2021, was held with IEEE IC2E 2021, a program of eight talks, and up to 33 participants joining us online.
  • Our 2nd workshop edition, TDIS 2022, was held with IEEE IC2E 2022, a program of seven talks, and around 25 participants on location.