Download to read offline
Uploaded byAndy Zaidman
an inconvenient truth in software testing: what is our environmental impact?
Presentation given at the Dutch Testing Day ("Testdag) 2025. Presentation is about the environmental impact of software testing and continuous integration
Food◦
More Related Content
Similar to an inconvenient truth in software testing: what is our environmental impact?
an inconvenient truth in software testing: what is our environmental impact?
- 1. an inconvenient truth insoftware testing: what is our environmental impact? andy zaidman / tu delft october 31st, 2025
- 3. Who is Andy? JoinedTU Delft in 2006 Full Professor in Software Engineering (since 2019) Head of Department (since 2024) 🔎 Software Testing & Software Quality - Software design, static analysis tooling, code reviewing - Automatic generation of software tests using GenAI, search-based, … - Testing with environment
- 4. Nature inspires (my 2023sabbatical at University of Victoria in British Columbia, Canada)
- 5. Closer to home:TU Delft climate initiative Echo: our newest building on campus
- 6. State of theUnion on Green ICT
- 7. The ICT sectorand the environment Estimations from 2020… 15—20%of the world’s electricity consumption 1.8—3.9%of the world’s CO2 emissions
- 8. Energy consumption andICT Images generated by DALL-E
- 9.
- 13. AI Everyone talks invests uses and we knowit is energy draining…
- 14.
- 15.
- 16. What do wereally know about how developers test?
- 18. Main insights 1. Wetracked ~2400 developers for ≥5 months If developers write test code, they typically don’t execute tests in the IDE… 2. We analyzed 2,640,825 Java and Ruby TRAVIS CI builds testing is the single most important reason why CI builds fail
- 19. How many CIbuilds happen in a year on GitHub?
- 20. Project # buildsper year Apache Flink 3218 Apache Maven 200 Apache Seatunnel 1457 Cruise-control 68 Elasticsearch 5025 Google Guave 231 Junit5 334 OpenEMS 228 Spring Boot 5535 Spring Framework 2564 Project # builds per year Apache Iceberg 1472 Apache Groovy 671 Apache IoTDB 1742 Openrewrite 1499 Apache Dubbo 1282 Hibernate ORM 2483 Spring Cloud GCP 493 Teku 754 Apache ShenYu 524 Egeria 1080
- 21. Project # buildsper year Apache Flink 3218 Apache Maven 200 Apache Seatunnel 1457 Cruise-control 68 Elasticsearch 5025 Google Guave 231 Junit5 334 OpenEMS 228 Spring Boot 5535 Spring Framework 2564 More than 5000 builds per year… that’s more than 23 per “working day”…
- 22. What is theenergy cost of a single build?
- 23. Hardware based measurement •Minisforum EM680 mini-PC • Gets its power through USB-C, no battery • Great little machine • Relatively fast with AMD Ryzen 7 6800U chip; 8 cores/16 threads (CPU 15W TDP) (same chip as in some Lenovo laptops) • 16GB RAM, fast 512GB SSD
- 24. Measurement setup CT-3 USB-Cpower meter
- 25. Disclaimer (!!) Disclaimer aboutthe Thermal Design Power (TDP) TDP = the power consumption under the maximum theoretical load. • AMD Ryzen 7 6800U chip has 15W TDP • Intel Xeon chips used by many cloud providers have an 85W TDP
- 26. Disclaimer (!!) Disclaimer aboutthe Thermal Design Power (TDP) TDP = the power consumption under the maximum theoretical load. • AMD Ryzen 7 6800U chip has 15W TDP • Intel Xeon chips used by many cloud providers have an 85W TDP
- 27.
- 28. Project Number ofbuilds Energy cost of 1 build Apache Flink 3218 7379 mWh Apache Maven 200 1573 mWh Apache Seatunnel 1457 16506 mWh Cruise-control 68 8996 mWh Elasticsearch 5025 32131mWh Google Guave 231 4969 mWh Junit5 334 3115 mWh OpenEMS 228 2753 mWh Spring Boot 5535 11727 mWh Spring Framework 2564 6133 mWh mWh = milli Watt per hour
- 29. Perspective… • 32131 milliWatt hour = 32.131 Watt hour ≈ 3 hours of using a 10 Watt LED light bulb Elasticsearch 5025 32131mWh
- 30. What is theenergy cost of one year of building (2022)?
- 31. Project Number ofbuilds Energy cost of yearly builds Apache Flink 3218 23.746 kWh Apache Maven 200 0.315 kWh Apache Seatunnel 1457 24.049 kWh Cruise-control 68 0.612 kWh Elasticsearch 5025 161.458kWh Google Guave 231 1.148 kWh Junit5 334 1.040 kWh OpenEMS 228 0.628 kWh Spring Boot 5535 64.909 kWh Spring Framework 2564 15.725 kWh
- 32. Perspective • 161 kWh≈ 10% of the yearly household energy consumption of a citizen of the European Union Elasticsearch 5025 161.458kWh
- 33. Perspective • 161 kWh≈ 10% of the yearly household energy consumption of a citizen of the European Union • Or… a one-way drive to wintersport using an electric car (a bit “optimistic” in winter conditions) Elasticsearch 5025 161.458kWh
- 34. From simulation to“real world”…
- 35. Server hardware Extrapolating… yearlyenergy becomes 915 kWh ≈ 55% of the yearly household energy of an EU citizen ≈ or 5 single way car rides to Basel in Switzerland 15W efficient (laptop) chip 85W server chip (Intel Xeon)
- 36. Docker Dockerize your build +10%energy consumption 915 kWh+ 10% ~1006 kWh
- 37. Multi-platform builds • Oursimulations are for 1 build • Some builds are done for different platforms • Most extreme: 1 project with 14 builds per commit (Different platforms, different Java versions, etc.) 1006 kWh *14 ~14 mWh
- 38. Multi-platform builds • Oursimulations are for 1 build • Some builds are done for different platforms • Most extreme: 1 project with 14 builds per commit (Different platforms, different Java versions, etc.) 1006 kWh *14 ~14 mWh 41 round-trips to Switzerland
- 39. Where is theenergy drain? In building? Or… in testing?
- 40. Building versus testing •First observed patterns: • Smaller projects spend more energy on compiling/building • Larger projects (# classes) spend more energy on testing • Projects that use a lot of IO-operations in their tests… more energy spent on testing
- 41. How much energydo we consume? What is the environmental impact?
- 44. Environmental impact? • Verymuch dependent on the carbon intensity • Theoretically, you could be building/testing your software entirely with green energy • Take ElasticSearch: • Assuming annual energy consumption of CI builds 915 kWh of electricity • Assuming your data center is in the EU: 305.6 kg of CO2 The Netherlands Egypt ~3500km flight
- 45.
- 46. git commit –m“Feature done’ git push • Datacenter energy consumption oblivous to the software engineer • We haven’t talked about the workstation energy consumption • Running tools: IDE, linters, compiler, … • Running tests locally • …
- 47. AI (again) When talkingto big IT companies about energy… SW build
- 48.
- 49.
- 50. What’s up withresearch? • Do we need to execute all tests every build? Test selection • Do we need to build every commit? Build batching • Can we make our tests more energy efficient? Code optimization Goal: Same quality, less energy
- 51. What’s up withresearch? • Do we need to execute all tests every build? Test selection • Do we need to build every commit? Build batching • Can we make our tests more energy efficient? Code optimization Goal: Same quality, less energy
- 52.
- 53. Additional resources • A.Zaidman. An Inconvenient Truth in Software Engineering? The Environmental Impact of Testing Open Source Java Projects Proceedings of the International Conference on Automation of Software Test (AST), pages 215-219. ACM, 2024. • R. Arntzenius, X. Liu, A. Zaidman. On The Energy Consumption of Continuous Integration in Open-Source Java Projects. In Submission • Q. Perez, R. Lefeuvre, T. Degueule, O. Barais, B. Combemale. Software Frugality in an Accelerating World: the Case of Continuous Integration. Arxiv, October 2024. • M. Beller, G. Gousios, A. Panichella, S. Proksch, S. Amann, A. Zaidman. Developer Testing in The IDE: Patterns, Beliefs, And Behavior. IEEE Transactions on Software Engineering (TSE), 45(3), 261-284, 2019. • M. Beller, G. Gousios, A. Zaidman. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. Proceedings of the International Conference on Mining Software Repositories (MSR), pages 356-367. IEEE, 2017. • C. Pang; A. Hindle; B. Adams; A.E. Hassan. What Do Programmers Know about Software Energy Consumption? IEEE Software 33(3), 83-89, 2016. • E.A. Santos, C McLean, C. Solinas, A. Hindle. How does docker affect energy consumption? Evaluating workloads in and out of Docker containers Journal of Systems and Software 146 (12): 14-25, 2018.