Download as PDF, PPTX
![Our Quality Journey
• A year ago, many defects in production
• “[one] week where every single deploy had to
either be rolled back or have a fast follow”
• Today - very few escaped production defects,
many small uneventful deploys](https://image.slidesharecdn.com/getting-by-without-qa-150513125825-lva1-app6892/75/Getting-By-Without-QA-8-2048.jpg)
![Our Quality Journey
• A year ago, many defects in production
• “[one] week where every single deploy had to
either be rolled back or have a fast follow”
• Today - very few escaped production defects,
many small uneventful deploys](https://crownmelresort.com/image.slidesharecdn.com/getting-by-without-qa-150513125825-lva1-app6892/75/Getting-By-Without-QA-8-2048.jpg)
Uploaded byDave King
Getting By Without "QA"
The document discusses the journey of Tilt, a company without a dedicated QA team, emphasizing the importance of team responsibility for quality and rapid deployment of code. Initially facing many production defects, they successfully improved their testing processes over a year, resulting in fewer bugs and enhanced product stability. Key lessons include investing in test improvement, choosing appropriate technologies, and integrating testing as a team priority for product enhancement.
![[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...](https://cdn.slidesharecdn.com/ss_thumbnails/dpesummitseptember20-212023-howimprovingthetestingexperiencegoesbeyondqualityadeveloperproductivityp-230924000523-453e8af8-thumbnail.jpg?width=640&height=640&fit=bounds)
Getting By Without "QA"
- 1. Getting By Without “QA” SanFrancisco Baking in the Quality Meetup 05/12/2015 Dave King
- 2. About Me • Programming20+ years • Penn State PhD 2009 • Team Lead at Tilt • Here’s a picture of my dog
- 3. What is Tilt? Tiltis the the easiest way to collect money from a group.
- 4.
- 5. Tilt Engineering • ~80people at company, ~30 engineers • CI/CD - ship code to production daily
- 6. Is QA Necessary? •No QA specialists on engineering team • “Quality is everyone’s job” • We’ve seen both sides of the “No QA” coin
- 7. Is QA Necessary? GoodBad Team responsible for quality Quality is defined by the team QA doesn’t bother you about bugs CEO bothers you about bugs Ship code to production fast Ship bugs to production fast Nobody tells you to write tests Nobody tells you to write tests
- 8. Our Quality Journey •A year ago, many defects in production • “[one] week where every single deploy had to either be rolled back or have a fast follow” • Today - very few escaped production defects, many small uneventful deploys
- 9. One Year Ago •Tests run on every branch prior to merge - takes ~30 minutes • Unstable tests - “rerun until green” mentality
- 10. One Year Ago •Single team member fixed test suite when it broke
- 11. One Year Ago •Very small smoke suite run in staging prior to production deploy • Tested only 3 core flows • Tests written in Perl
- 12. Today • Tests rewrittenusing Nightwatch.js • Entire team responsible for expansion/ maintenance of test suite
- 13. Today • Selenium Suitesfor 15 core flows • ~60-70 scenarios • “Create Tilt”, “Contribute to Tilt”, “Auth”, etc • Run every 10 minutes vs staging • All passing when I left for this meetup 😎
- 14.
- 15. Lessons Learned • Youneed to invest in test improvement
- 16. Investment is Important •Can’t say “fix the tests” and expect results • Team may not know the technologies involved • Team may not feel like they have time to investigate and learn
- 17. How Can YouInvest? • Hire people who have done it before • Give the team exploratory time to improve tests • Represent functional tests in JIRA: “Write Test” stories
- 18. Lessons Learned • Continuallyrun tests against staging
- 19. Test Against Staging •Dev environments have low fidelity • A failing test is everyone’s problem • Tests rerun themselves to green, smoothing out “flappers”
- 20. Lessons Learned • Choiceof technology matters
- 21. Choice of Technology Matters •Past job - write tests in Ruby, write code in JavaScript/Python • Nobody knows the tools involved, language idioms, etc • “Copy-paste” mentality, “lol what’s a Gemfile”
- 22. Choice of Technology Matters •Tilt uses Nightwatch.js, written in JavaScript • Add libraries to tests using npm • We use babel.js to make our tests even better
- 23. Choice of Technology Matters •register babel in nightwatch.conf.js • Now, enjoy ES6 in your tests! var createTestUser = (appId) => { return (token) => { return request({ method: 'POST', url: `https://graph.facebook.com/$ {appId}/accounts/test-users?access_token=${token}`, json: true }); }; };
- 24. Choice of Technology Matters •Nightwatch.js provides a good foundation for a functional test suite • Page objects and commands are “built in” • Hand-rolled suite needed to reinvent these
- 25. Lessons Learned • Splittests into a separate repository
- 26. Separate Test Repository •Easier to see what’s important for things to work • Slightly looser code review standards 😉
- 27. Lessons Learned • Improvingyour product improves test stability
- 28. Improve Your Product •User experience is stochastic • Tests set an upper bound, “wait 5 seconds…” Test fails if user experience is worse than predefined wait time!
- 29. Improve Your Product •You can improve unstable tests by changing the waits to more accurately reflect your application’s behavior • You can also improve the behavior of the application!
- 30. Improve Your Product •After rewriting tests, some jobs were still only 65-70% stable • Hard to know if a build on staging was good or not!
- 31. Improve Your Product •Over the last three months, we’ve made big steps to improve the product: • Rewrote contribution lightboxes in React.js • Massively sped up our campaign page load • Removed page refresh after facebook login
- 32. Improve Your Product •Now all tests 95%+ reliable, very few “flappers” • But we didn’t really spend time tunneling to “improve the tests” • Benefit to no-QA - team members can invest time into product improvements too!
- 33. Our Next Steps •Continue to expand test suite to cover all core flows • Add functional coverage for tracking calls • Add cross-browser coverage • Investigate visual testing (screenshot diffs)