Download as PDF, PPTX
Uploaded byDinis Cruz
Start with passing tests (tdd for bugs) v0.5 (22 sep 2016)
This document discusses improving the Test-Driven Development (TDD) process for bug fixing by starting with passing tests rather than failed tests. It highlights the importance of real-time unit test execution and effective communication among team members through tests to enhance productivity and identify root causes of issues. The presentation emphasizes that successful application security practices require collaboration between developers and QA teams while continuously improving test APIs and execution environments.
More Related Content
![[DevSecOps Live] DevSecOps: Challenges and Opportunities](https://cdn.slidesharecdn.com/ss_thumbnails/20200316dsochallengesopportunities-200416111818-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Start with passing tests (tdd for bugs) v0.5 (22 sep 2016)
Start with passing tests (tdd for bugs) v0.5 (22 sep 2016)
- 1. Start with passingTests (TDD for bugs) London, 22nd Sep 2016, v0.5
- 2. Me • Developer for25 years • AppSec for 13 years • Day jobs: • Leader OWASP O2 Platform project • Application Security Training for JBI Training • Part of AppSec team of: • The Hut Group • BBC • AppSec Consultant and Mentor
- 3. • @Leanpub (buyfor 0$ ) • http://leanpub.com/u/DinisCruz – Books Published
- 4. • published atLeanpub (buy for 0$ ) – http://leanpub.com/u/DinisCruz Books under development Major revision with lots of new content (based on Maturity Models app) Ideas shown in this presentation and a lot more
- 6.
- 8.
- 10.
- 11. • For bugs,always start with a passing test – Common workflow on TDD is to write failed tests. – The problem with this approach is that it only works for a very specific scenario (when fixing bugs). – This presentation will present a different workflow which will make the coding and testing of those much much easier, faster, simpler, secure and thorough' Key concept: Turning TDD upside down
- 12. • For TDDto be productive you need – Real time unit test execution (when hands lift) – Real time code coverage • TDD focus needs to be on – making developers more productive – preventing developers from switching context • You need something like Wallaby or NCrunch • If 99% code coverage doesn’t happen ‘by default’ TDD workflow is not working TDD
- 13. 9 9 %C O D E C O V E R A G E …is not the destination …it is ‘base camp’
- 15. TDD in WebStormwith WallabyJS
- 16. What happens whenyou increase attack surface
- 17. You want atest to fail
- 18. TDD in WebStormwith WallabyJS • … but is a topic for another talk :)
- 19. TDD WORKFLOW FORA BUG
- 20. • Bug isfound • Issue created and documented • Issue prioritised and scheduled • Fix • QA • Deploy – hope is has been fully fixed? Normal flow for bugs and security issues
- 21. • Bug found,Issue created, Scheduled • TDD loop – Write test that fails – Fix Issue – Test passes – All done? • QA – why is this needed? With TDD
- 22. • Very fewdo this loop in pure TDD – very hard to do without realtime test execution and code coverage • Needs test API & Infrastructure to be there already • are you able to spend an extra 4h, 8h,16h fixing a blind spot of your test APIs? • End result tends to be very shallow tests • Root cause of issue is usually not identified (or fixed)
- 23. • Given thesebugs: • Login + open page X + enter data + submit data = BUG happens • Create object + populate with data + do action = BUG happens • call service A + inner call to Service B + bad data returned from Service B = BUG happens • If you write a failed test, what happens when a problem occurs before we reach the Bug happens How do you know what failed?
- 24. • Tests areonly written while in ‘fix mode’ • Very little time to improve test APIs for test execution environment – There are really hard problems in Testing, that will need time to be solved • Creating test proxies • Creating ‘hot’ run-time environments • Improving test execution and code coverage visualisation (when Wallaby or NCrunch is not available) • Will break the build • Will create a high threshold of what can be tested The problem with failed tests
- 25. • Missing Headerson Production (HSTS for example) • Debug messages shown on production • SQL Injection on user controller field • Wrong product shown on page • Wrong link generated on page and email • Performance issues (for example: slow search) • Inconsistent error messages returned from Web Service • Message Queues accepting message with bad data • Expose user info via parameter manipulation • Over-posting on Controller’s Model bindings • Use of Strings on models to hold data (‘strings should to be banned’) Example of bugs
- 26. • Aren’t theythe current behaviour of the application? • Isn’t that how the application was designed • If you are writing a test to replicate the issue, should that test: Pass or Fail? • My definition of feature is: Once the code hits the main ‘develop’ branch, it’s behaviour is a feature (regardless if the code is in production) Are these bugs or features?
- 27. Until you writethe (passing) test, you don’t really know the root cause of the issue
- 28. 1.Find issue 2.Create PoC(Proof-of-Concept) 3.Report issue (on PDF or Bug tracking) 4.Dev ‘fixes’ issue 5.Fix pushed to production 6.Is it really fixed? If yes, end of story 7.Find variation of issue 8.go back to 2. Bug/Security fix merry-go-round
- 29. • Why isQA finding issues that the devs are not aware of, or have been fixed before? • Most QA teams are clicking on things and running scripts • QA teams: – should be reviewing the Tests created/changed since the last release – need to have developers since they need to write Tests – need to have senior developers since they need to improve the Test’s APIs and execution/development Environments QA - Most are not being efficient
- 30.
- 31. • Write Testfirst • Understand the issue • Improve test APIs and Frameworks • Write one Test per affected layer (or call stack) – start with e2e (end-to-end) and end with pure Unit Tests (on the root cause of the issue) – For affected each method, write tests for it’s callers (creating a reverse tree) • By definition, the fix of original issue(s) will break these tests • Transform these ‘passing bug tests’ into ‘passing regression tests’ • QA team (and manager/architect/security) reviews the test’s diff Suggested workflow
- 32. • Team membersneed to communicate using Tests • Tests should be easier to write than explanations (text, screenshot, videos) • Teams collaboration should be done via tests (from dev to QA, from QA to dev, from Team A to team B) • Managers need to write tests (and demand powerful/easy-to-use APIs) Tests need to be the lingua franca
- 33. • It iscrazy that we tend to promote the best developers into management position (since that is the ‘only’ career path) and then make them do everything except coding • These ‘senior devs, now managers/architects’ tend to be the best programmers in the building and usually love coding • The solution is to make them read and write tests • Teams need to communicate using tests • Printing tests (on paper) is a very effective way to code review Managers need to code
- 34. When creating testson the ‘Fix’ stage, the focus (& time allocated) is on fixing the bug (not on testing it) When creating tests on the ‘Issue Creation’ stage, the focus (& time allocated) is on how to test it and what is its root cause
- 35. CASE STUDY: WHENI CREATED A VULNERABILITY
- 36. • Here isthe code I wrote (at the Data Layer) • This method is designed to be called by the controller (i.e. rest api endpoint): Feature request: Allow data editing on UI
- 37. Feature request: Allowdata editing on UI
- 38. Regression test thatpasses on issue
- 43. Fix for Pathtransversal
- 44.
- 45. LET’S SEE HOWIT LOOKED IN THE CODE
- 46. …before the vulnis created
- 47. …when the vulnis created
- 48.
- 49. …after issues arecreated
- 50.
- 51. …updating issues after1st fix
- 52.
- 53.