Download as PDF, PPTX
![pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking gear, etc.,
upon which we’re going to try to carefully apply a
complicated set of changes and hope that nothing
goes wrong. Cross your fingers… here goes.](https://image.slidesharecdn.com/cd-principles-and-practice-mike-brittain-140402165511-phpapp02/75/Principles-and-Practices-in-Continuous-Deployment-at-Etsy-7-2048.jpg)
![US","region":"US","detected_currency_code":"USD","detected_language":"en-
US","detected_region":"US","accept-languages":"en-US","cdn-
provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0"
,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst
antRequest":"0","isMozPrefetchRequest":"0","listing_ids":
[104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11
4767467,117266897,157579748],"scheduled_modules_content_ids":
[10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte
nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl
ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in
ternational.languages.ja;international.languages.no;international.languages.pl;internationa
l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_
toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r
ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo
ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c
srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_enc
Analytics connected to config names](https://image.slidesharecdn.com/cd-principles-and-practice-mike-brittain-140402165511-phpapp02/75/Principles-and-Practices-in-Continuous-Deployment-at-Etsy-73-2048.jpg)
![pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking gear, etc.,
upon which we’re going to try to carefully apply a
complicated set of changes and hope that nothing
goes wrong. Cross your fingers… here goes.](https://crownmelresort.com/image.slidesharecdn.com/cd-principles-and-practice-mike-brittain-140402165511-phpapp02/75/Principles-and-Practices-in-Continuous-Deployment-at-Etsy-7-2048.jpg)
![US","region":"US","detected_currency_code":"USD","detected_language":"en-
US","detected_region":"US","accept-languages":"en-US","cdn-
provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0"
,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst
antRequest":"0","isMozPrefetchRequest":"0","listing_ids":
[104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11
4767467,117266897,157579748],"scheduled_modules_content_ids":
[10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte
nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl
ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in
ternational.languages.ja;international.languages.no;international.languages.pl;internationa
l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_
toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r
ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo
ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c
srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_enc
Analytics connected to config names](https://crownmelresort.com/image.slidesharecdn.com/cd-principles-and-practice-mike-brittain-140402165511-phpapp02/75/Principles-and-Practices-in-Continuous-Deployment-at-Etsy-73-2048.jpg)
Uploaded byMike Brittain
Principles and Practices in Continuous Deployment at Etsy
This document discusses principles and practices of continuous deployment at Etsy. It describes how Etsy moved from deploying code changes every 2-3 weeks with stressful release processes, to deploying over 30 times per day. The key principles that enabled this are innovating continuously, resolving scaling issues quickly, minimizing recovery time from failures, and prioritizing employee well-being over stressful releases. Automated testing, deployment to staging environments, dark launches, and extensive monitoring allow for frequent, low-risk deployments to production.
More Related Content
Similar to Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
- 1. Principles and Practicesin Continuous Deployment Mike Brittain Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
- 3. “Continuous Deployment” Process bywhich our team deploys software changes to production services over 30 times per day.
- 4. Where we started Principlesfor our Engineering team Continuous Deployment Business case
- 5. Five years ago… 2-3weeks of code changes Release and rollback plans Traffic and infrastructure management (Ops) 6-14 hours
- 6. Five years ago… “DeploymentArmy” Stressful, especially when things go wrong Long days and late nights Scheduled downtime
- 7. pro·duc·tion [pruh-duhk-shuhn] (n) 1.This complex system of application code, distributed services, servers, networking gear, etc., upon which we’re going to try to carefully apply a complicated set of changes and hope that nothing goes wrong. Cross your fingers… here goes.
- 8. Software for large-scaleweb sites has been traditionally written by one group of people, then released and operated by a different group. These two groups have very different levels of visibility into how the software works.
- 9.
- 10. “…frequent and prolongedoutages.” 2010 CAPACITY PLAN
- 11.
- 12.
- 13. Innovate or die Resolvescaling hurdles
- 14. Innovate or die Resolvescaling hurdles Mean-time-to-recovery
- 15. “Quality is notjust testing pre-release. It also includes our adaptability and response time.” - Jeff Sussna at ALM Forum, 2014
- 16. Innovate or die Resolvescaling hurdles Mean-time-to-recovery
- 17. Innovate or die Resolvescaling hurdles Mean-time-to-recovery Healthy and talented engineering team
- 18. Autonomy, Mastery, Purpose. “Drive:The surprising truth about what motivates us.” ~Dan Pink, at RSA http://youtu.be/u6XAPnuFjJc
- 19. Innovate or die Resolvescaling hurdles Mean-time-to-recovery Healthy and talented engineering team Stop stressing about releases
- 20.
- 21. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ In a softwarerelease process Fail Fast means releasing undeployed code as fast as possible, instead of waiting for a weekly release to break.
- 22.
- 23. SECRET WEAPON: Hired asVP, Tech-Ops at Etsy
- 24.
- 25. Frequent check-ins directlyto mainline. Continuous Deployment Continuous Delivery ✓ ✓
- 26. Continuous Integration and Automatedtests. Continuous Deployment Continuous Delivery ✓ ✓
- 27. Keep the buildgreen. We’re always ready to release. Continuous Deployment Continuous Delivery ✓ ✓
- 28. “One button” deploys. ContinuousDeployment Continuous Delivery ✓ ✓
- 30. Business dictates whena build is deployed. Continuous Deployment Continuous Delivery ✓
- 31. Every passing buildis deployed to production. Continuous Deployment Continuous Delivery ✓
- 32. All enhancements aregated by Config Flags. (“Branch in code”) Continuous Deployment Continuous Delivery ✓ ?
- 33. Most of thebuilds we deploy are “dark” changes.
- 34. CSS rules andproperties Copy in templates (e.g. typos) New, un-referenced code (e.g. classes, funcs, templates) Code paths behind disabled config flags etc…
- 35. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
- 36. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
- 37. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
- 38. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Continuous Delivery release pipeline Dev / Integration Staging Production
- 39. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Continuous Delivery release pipeline Dev / Integration Staging Production Assumptions: Staging is a perfect reflection of Production, with respect to hardware, configurations, data, overall load, capacity, etc. Deploy process is infallible.
- 40. “What do youmean, ‘it’s not working in production?’ I TESTED IT BEFORE WE RELEASED!”
- 41. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Dev / Integration Staging Production Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval
- 42. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Dev / Integration Staging Production Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback
- 43. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback "Because you’re integrating so frequently, there is significantly less back- tracking to discover where things went wrong , so you can spend more time building features.” ! —ThoughtWorks ! ! http://www.thoughtworks.com/continuous-integration Dev / Integration Staging Production
- 44. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback Dev / Integration Staging Production Where’s the bug? ! In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests?
- 45. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback Dev / Integration Staging Production Where’s the bug? ! In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests? ! Data out of sync? Server configurations out of sync? Capacity vs. current load? Deployment script?
- 46. Dev Team VersionControl Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback Dev / Integration Staging Production How will we know when something is wrong in production? ! How long will it take to resolve the issue? Check in Trigger
- 47. We aim toreduce fundamental surprise in every release.
- 48. Furthermore, we optimizefor detecting and recovering from failures quickly.
- 49. Pre-production validation Code deployedto de-pooled application (web) servers touching prod services and databases. Smoke tests Integration tests Functional tests User-Acceptance (ad hoc)
- 50. Production validation Exactly thesame server configs, services and data as pre-prod, but this is where we introduce application code to live traffic.
- 51. Production validation Exactly thesame server configs, services and data as pre-prod, but this is where we introduce application code to live traffic. Smoke tests (esp. over public hostnames) User-Acceptance testing behind config flags Gratuitous monitoring Customer support and forums
- 52. Single release Many releases 50K LOC/month Few opportunitiesfor failure Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve written per release Imagine that we’ll write
- 53.
- 54. Monitoring PHP Warnings BugReports and Help Requests
- 55.
- 56.
- 57. Check in Trigger Feedback Trigger FeedbackApproval Approval Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline Feedback
- 58. Dev Pre-Production (“Princess”) Production Check in Trigger FeedbackTrigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI
- 59. Dev Pre-Production (“Princess”) Production Check in Trigger FeedbackTrigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI Approval Approval Feedback Feedback Feedback
- 60. Dev Pre-Production (“Princess”) Production Check in Trigger FeedbackTrigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI Approval Approval Feedback Feedback Feedback Approval Feedback Approval Feedback
- 61. “Allow buttons properlyto inherit color from their parent node.”
- 64. Five years ago… 2-3weeks of code changes Release and rollback plans Traffic and infrastructure management (Ops) 6-14 hours
- 65. Five years ago… “DeploymentArmy” Stressful, especially when things go wrong Long days and late nights Scheduled downtime
- 66. Why do wedo this?
- 67. Innovate or die. Resolvescaling hurdles. Mean-time-to-recovery. Healthy and talented engineering team. Stop stressing about releases.
- 68. Innovate or die. Resolvescaling hurdles. Mean-time-to-recovery. Healthy and talented engineering team. Stop stressing about releases.
- 69. Admin-launch and whitelist Ramp-uppublic traffic
- 70.
- 71.
- 72.
- 73.
- 74.
- 75. Observed impact Time seriesdata for duration of the experiment
- 76. Observed impact Time seriesdata for duration of the experiment
- 77.
- 78. “I want tofind out whether buyers will favor a single price for the product that includes shipping.” https://www.etsy.com/shop/lucra
- 79. Eligibility requirements: - Mustbe first page of visit - Buyer & seller in same region - etc…
- 80. Time: < 8hours Staff: One ! Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.
- 81. We do notbundle the item price and shipping cost together today. ! https://www.etsy.com/shop/lucra
- 83.
- 84. Ambitious Product Goal Monolithic Buildingand measuring many things at once.
- 85. Ambitious Product Goal Monolithic Buildingand measuring many things at once. Iterative One thing at a time, our design goal is always in sight.
- 86. Time: < 8hours Staff: One ! Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled. Deployed Deployed
- 87. Is this forme?
- 88. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ “Maybe this isjust viable for a single developer … your site will be down. A lot.”
- 89.
- 90. @mikebrittain Very end of2009 Today DEPLOYMENTSPERDAY APPCODE CONFIGFILES
- 91. $1.35 Billion Goodssold in 2013 60+ Million Unique visitors per month ! 175+ Committers, everyone deploys http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
- 93. Thank you. Mike Brittain EngineeringDirector, Etsy @mikebrittain mikebrittain.com/talks