How to Make Your UI Tests Stable, Scalable, and Maintainable

How to Make Your UI Tests Stable, Scalable, and Maintainable

• 1.
  How to MakeYour UI Tests Stable, Scalable, and Maintainable
• 2.
  2 SmartBear’s technology stack
• 3.
  3 Quick Intro Nikhil KaulJeffrey Martin
• 4.
  4 On our platetoday 1 Why you shouldn't tolerate flakiness? 2 Common reasons for flaky UI tests 4 What to do with flaky UI tests? 5 How to separate signal from noise? 3 How to overcome these challenges?
• 5.
  5 Flaky tests shouldn'tbe tolerated • 300 tests • Each test has .5% failure rate • Pass rate for each tests: 99.5% • Pass rate for the suite: (99.5%)^300 = 22.23%
• 6.
  6 Understand the natureof flakiness Tech ComplexitiesTest EnviromentTest DataTest Design
• 7.
  7 Understand the natureof flakiness Tech ComplexitiesTest EnviromentTest DataTest Design
• 8.
  8 Don’t interact withUI elements directly Test services seperately from the UI Use page objects Automate at three levels ‒ Unit ‒ Service ‒ UI Use UI encapsulation Build modular test design ‒ Identify user journeys ‒ Unit ‒ Service ‒ UI Automate at three levels Use UI encapsulation ‒ Page Objects UIServiceUnit
• 9.
  9 How to implementin real world Add Item Click on an item with name XYZ Clickitem() Set element with name ABC to quantity Z Set element() Click on button labeled place in cart clickbutton() What? How?
• 10.
  10 Understand the natureof flakiness Tech ComplexitiesTest EnviromentTest DataTest Design
• 11.
  11 Separate the testdata from test design Test services seperately from the UI Use page objects Automate at three levels ‒ Unit ‒ Service ‒ UI Use page objects Build modular test design – Test specific data – Reference dataLimit data dependency between tests Tests should run independently Using databases ‒ Conduct tests inside a transaction
• 12.
  12 Understand the natureof flakiness Tech ComplexitiesTest EnviromentTest DataTest Design
• 13.
  13 Limit your dependencyon environments Test services seperately from the UI Use page objects Automate at three levels ‒ Unit ‒ Service ‒ UI Use page objects Build modular test design ‒ Application API ‒ HTML API ‒ Identify user journeys Limit calls to external systems Parameterize connections Hermetic Servers
• 14.
  14 Understand the natureof flakiness Tech ComplexitiesTest EnviromentTest DataTest Design
• 15.
  15 Automate at threelevels ‒ Unit ‒ Service ‒ UI Use page objects Build modular test design‒ Application API ‒ HTML API Limit calls to external systems Parameterize connections Hermetic Servers Technical complexities can be overcome AJAX Calls ‒ Callback ‒ Polling Interval ‒ WaitUntil No Response Calls ‒ Fake objects ‒ Mock ‒ Virtual Service
• 16.
  16 Acceptance test buildsshould never be red Separate flaky from healthy tests Limit the number of tests as well as the time
• 17.
  17 Use statistics formeasuring test stability Sample Size (N) • 50 Desired confidence (α) • 95% confidence P • 29/50 = .58 1-p • 21/50 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝐶𝐼 = 𝑝 ± z1−α 2 𝑝(1 − p) n 𝐶𝐼 = .58 ± 1.96 .58(1 − .58) 50 𝐶𝐼 = .58 ± .14 𝐶𝐼 = between 44% and 72%. ±z1−α 2 = . ±z.95 = 1.96
• 18.
  18 Takeaways 1 Flaky tests shouldn'tbe tolerated 2 Understand the nature of flakiness 4 Use statistics for measuring test stability 3 Acceptance test builds should never be red
• 19.
  Questions?