Downloaded 49 times
Uploaded byJosh Evans
Engineering Netflix Global Operations in the Cloud
The document discusses Netflix's operational challenges in managing a rapidly scaling cloud infrastructure, specifically focusing on accelerating innovation and maintaining high quality amidst complexity. It outlines the application of operations engineering practices to enhance service delivery and automation, including the use of chaos engineering for resilience. Key elements include continuous improvement, real-time analytics, and systematic monitoring to achieve operational excellence and competitive advantage.
More Related Content
Similar to Engineering Netflix Global Operations in the Cloud
Engineering Netflix Global Operations in the Cloud
- 1. © 2015, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Josh Evans - Director of Operations Engineering Engineering Netflix Global Operations in the Cloud
- 2.
- 3. • Two OperationalChallenges • Operational Excellence • Operations Engineering Our Journey
- 4. Our Journey • TwoOperational Challenges • Operational Excellence • Operations Engineering
- 5.
- 10. ● Every facetof the product ● 1400 AB tests in the last year & accelerating Continuous Innovation
- 11. Challenge #1: Accelerate Innovationand Rate of Change
- 12.
- 13. 100,000s of requestsper second 1000s of Global Starts per Second
- 14. Approaching Global Reach October- Spain, Portugal, Italy Early 2016 - Korea, Taiwan, Singapore, Hong Kong 65m members 100m ~60 counties 200
- 15.
- 16. Netflix CDN (Open Connect) Cloud ControlPlane Internet The Bigger Picture Service Partners Service Partners
- 17. Challenge #2: Sustain &Improve Quality in the face of ever growing scale & complexity
- 18. Our Journey • TwoOperational Challenges • Operational Excellence • Operations Engineering
- 19.
- 22. Availability vs. Rateof Change Rate of Change Availability(nines) 6 5 4 3 2 1 0 1 10 100 1000 99.9999% 99.999% 99.99% 99.9% 99% 90% 31.5 seconds 5.26 minutes 52.56 minutes 8.76 hours 3.26 days 36.5 days Quality vs. Velocity
- 23. Availability vs. Rateof Change Rate of Change Availability(nines) 6 5 4 3 2 1 0 1 10 100 1000 99.9999% 99.999% 99.99% 99.9% 99% 90% 31.5 seconds 5.26 minutes 52.56 minutes 8.76 hours 3.26 days 36.5 days The Zero Sum Game
- 24. Availability vs. Rateof Change Rate of Change Availability(nines) 6 5 4 3 2 1 0 1 10 100 1000 99.9999% 99.999% 99.99% 99.9% 99% 90% 31.5 seconds 5.26 minutes 52.56 minutes 8.76 hours 3.26 days 36.5 days The Zero Sum Game
- 25. Availability vs. Rateof Change Rate of Change Availability(nines) 6 5 4 3 2 1 0 1 10 100 1000 99.9999% 99.999% 99.99% 99.9% 99% 90% Shifting the Curve
- 26. Operational Excellence isthe continuous improvement of the management, design, and function of operational environments to achieve greater quality, velocity, and competitive advantage.
- 27. Our Journey • TwoOperational Challenges • Operational Excellence • Operations Engineering
- 28. Build It design code build bake test deploy Run It operate configure monitor respond Youbuild it, you run it… …globally
- 29.
- 30. Operations Engineering isthe application of software engineering practices and principles to achieve and sustain operational excellence. • automation • modular components • tools & services • best practices
- 31. Our Journey –Operations Engineering • Engineering Tools • Insight & Real-time Analytics • Performance & Reliability • Leverage
- 32. Our Journey • EngineeringTools • Insight & Real-time Analytics • Performance & Reliability • Leverage
- 33. Data Center ● Delayedprovisioning ● Hand-crafted servers ● Variations and complexity Our Artisanal Past Delivery ● Late night, manual deployments ● Repeated mistakes ● Painful delays to production fixes
- 34. • productivity • velocity •quality Engineering Tools
- 35. • cloud management •delivery engine • automation platform
- 36.
- 37.
- 38.
- 39. The Paved Road •Stash • Gradle • Ubuntu • Jenkins • Spinnaker
- 40. Our Journey • EngineeringTools • Insight & Real-time Analytics • Performance & Reliability • Leverage
- 41. Insight & Real-TimeAnalytics
- 42.
- 43. An outage maynot be life or death but…
- 44. • DES ontime series data • Predict the future based on history • Favor recent history • Threshold-based alerts • 6-8 minute delay Anomaly Detection Alert!
- 45. Finer Granularity, ShorterTime Windows
- 46.
- 47. Median Absolute Deviation IQR LeastSquares HDI Voting
- 48. observe, orient, decide,act Alert! From 6-8 minutes to < 1 minute
- 49.
- 50.
- 51. How do wetake humans out of the equation?
- 52. Outlier Detection &Remediation
- 53. • Unsupervised machinelearning • Density-based clustering algorithm • Actions • Email, page • OOS, detach, terminate Kepler
- 54. An ounce ofprevention…
- 55. Old Version (v1.0) NewVersion (v1.1) Load BalancerCustomers 100 Servers 5 Servers 95% 5% Metrics Canary Release Process
- 56. Old Version (v1.0) NewVersion (v1.1) Load BalancerCustomers 0 Servers 100 Servers 100% Metrics Canary Release Process
- 57. Define • Metrics • Athreshold Every n minutes ● Classify metrics ● Compute score ● Make a decision Automatic Canary Analysis
- 58. • Systematic observationof facets & permutations • Unsupervised monitoring & decision- making • Automated tuning & recovery • Alerts with analysis Thinking Globally
- 59. Our Journey • EngineeringTools • Insight & Real-time Analytics • Performance & Reliability • Leverage
- 60.
- 61.
- 62. Chaos Engineering isthe discipline of experimenting on a distributed system in order to build confidence in the systems capability to withstand turbulent conditions in production.
- 63. Cluster A ClusterD Edge Cluster Cluster B Cluster C Imagine a monkey loose in your data center…
- 64. Xen Hypervisor vulnerability– 9/25/14 218 out of 2700+ Cassandra nodes rebooted 22 did not reboot successfully Automation handled the rest A State of Xen – Chaos Monkey & Cassandra
- 65. Device Service B ServiceC Internet EdgeZuul Service A ELB FIT Fault-Injection Testing (FIT) • Simulate service failures • Override by device or account • % of member traffic
- 66. Device Service B ServiceC Internet EdgeZuul Service A ELB FIT Fault-Injection Testing (FIT) • Simulate service failures • Override by device or account • % of member traffic
- 67.
- 68. The Internet DNS-based Routing Zuul Proxy BackChannel ###, ###, ###
- 69. • Alerting andMonitoring • Apache & Tomcat Hardening • Automated Canary Analysis • Autoscaling • Chaos Participation • Consistent Naming • ELB Configuration • Healthcheck Configured • Red-Black Pipeline • Squeeze Testing • Timeout & Fallback Tuning • Workload Reliability Production Ready?
- 70. Our Journey • EngineeringTools • Insight & Real-time Analytics • Performance & Reliability • Leverage
- 71. ● A federationof tools ● Common UI elements ● Deep linking Operational Tools as a Product
- 75. Canary Analysis Conformity Integration Tests Citrus Chaos Static UnitTests Deep Integration Modular Components Functional Testing
- 76. RTA auto-tuning • Alerts •Apache/Tomcat • Auto-scaling • Hystrix fallbacks RTA decision support • ACA • Citrus • Flow Conformity checks • Consistent names • ELBs • Health check • Red/black deployment Delivery integration • ACA • Citrus • FIT Production Ready – Automation & Integration
- 79.
- 80.
- 82. Speaker When? Where? EngineeringNetflix Global Operations in the Cloud Josh Evans Wed @11am Palazzo N Efficient Innovation: High-Velocity Cost Management at Netflix Andrew Park Wed @ 2:45pm Palazzo C Netflix Keystone: How Netflix Handles Data Streams Up to 8 Million Events Per Second Peter Bakas Wed @ 2:45pm San Polo 3501B A Day in the Life of a Netflix Engineer Using 37% of the Internet Dave Hahn Wed @ 4:15pm Venetian H Availability: The New Kind of Innovator’s Dilemma Coburn Watson Wed @ 4:15pm Marcello 4501B Real-Time Analytics In Service of Self-Healing Ecosystems Roy Rapoport Chris Sanden Wed @ 4:15pm Lido 3001B Running Spark and Presto on the Netflix Big Data Platform Daniel Weeks Thu @ 11am Palazzo F Splitting the Check on Compliance and Security: Keeping Developers and Auditors Happy in the Cloud Jason Chan Thu @ 11am Marcello 4501B @
- 83.