Uploaded byC4Media
Fault Tolerance at Speed
The document discusses fault tolerance in software development, emphasizing the importance of efficient designs and clustering technologies like Aeron and Raft for handling stateful services. It highlights the need for fault tolerance strategies that ensure service reliability and speed while addressing efficiency and resource management. The content also reflects on the evolution of distributed systems through the sharing of knowledge and innovation at QCon conferences.
More Related Content
Fault Tolerance at Speed
- 1. Fault Tolerance atSpeed Todd L. Montgomery @toddlmontgomery StoneTor
- 2. InfoQ.com: News &Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ aeron-cluster-raft/
- 3. Purpose of QCon -to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4.
- 5. What type ofFault Tolerance? What is Clustering? Why Aeron? Design for Speeding Up?
- 6. What type ofFault Tolerance? What is Clustering? Why Aeron? Design for Speeding Up? Efficiency
- 7.
- 8. We seem toassume efficiency/security/quality/etc. is a “special” characteristic added … later… if at all
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 16.
- 17.
- 18. Fault Tolerance ofState
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. Service ServiceService Log ArchiveLogArchive Log Archive
- 28.
- 29. Each Replicated Service Sameevent log Same input ordering Log replicated locally Replicated State Machines
- 30. Checkpoints / Snapshots Eventin the log “Rolling” up previous log events Replicated State Machines
- 31. When should aservice “consume” (or process) a log event?
- 32. Service ServiceService ArchiveArchive Archive 12 3 4 5 6 1 2 3 4 5 6 71 2
- 33. Once processed, Event cannot be altered Only process once event is stable
- 34. Raft Consensus Event mustbe recorded at majority of Replicas before being consumed by any Replica Replicated State Machines https://raft.github.io/
- 35. Service ServiceService ArchiveArchive Archive 12 3 4 5 6 1 2 3 4 5 6 71 2
- 36. Service ServiceService ArchiveArchive Archive 12 3 4 5 6 1 2 3 4 5 6 71 2
- 37. Strong Leader Elected memberof the Cluster Orders Input Disseminates Consensus Raft
- 38. Service ServiceService Archive ArchiveArchive ConsensusConsensusConsensus
- 39. Raft is An algorithmwith formal verification Replicated State Machines
- 40. Raft is not Aspecification Nor A complete system Replicated State Machines
- 41. More than Raft Leadertimestamps events Async, not RPC-based Timers The Real World
- 42. Service ServiceService Archive ArchiveArchive ConsensusConsensusConsensus *Leader Client
- 43.
- 44. Determinism Log is immutable Logcan be played, stopped, & replayed Each event is timestamped Services restarted from snapshot & log Benefits
- 45. What Can YouDo?
- 46. Distributed Key/Value Store DistributedTimers Distributed Locks
- 47. Matching Engines Order Management MarketSurveillance P&L, Risk, … Finance
- 48. Venue Ticketing /Reservations Auctions Beyond Hint - a contended database is a good indicator
- 49.
- 50. Efficient reliable UDPunicast, UDP multicast, and IPC message transport Java, C/C++, C#, Go Aeron https://github.com/real-logic/Aeron
- 51. And a littlebit more… Very fast Archival & Replay Aeron https://github.com/real-logic/Aeron
- 52.
- 53. All communications Aeron publications& subscriptions Aeron archival & replay Aeron shared counters
- 54. Consensus based on Aeronstream position
- 55. Batching Critical to efficientoperation Optimizing pipelined throughput
- 56. Flow Control Critical tocorrect operation
- 57.
- 58. Cache Hit/Miss Ratios BranchPrediction Allocation Rates Garbage Collection Inlining Optimizations
- 59.
- 60. Ownership, Dependency, &Coupling Complexity Layers of Abstraction (ain’t free) Resource Management
- 61.
- 62. "AmdahlsLaw" by Daniels220at English Wikipedia - Own work based on: File:AmdahlsLaw.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons
- 63. Universal Scalability Law 0 2 4 6 8 10 12 14 16 18 20 12 4 8 16 32 64 128 256 512 1024 Speedup Processors Amdahl USL
- 64.
- 65. Ingress Message, Sequence,Disseminate Client Follower X Leader Ingress Follower Y Log (multicast or serial unicast) Member Status Log Event Log Event
- 66. Followers Append Client Follower X Leader Ingress FollowerY Log (multicast or serial unicast) Member Status Append Position Append Position
- 67. Commit Message Client Follower X Leader Ingress FollowerY Log (multicast or serial unicast) Member Status Commit Position Commit Position
- 68.
- 69. FollowerLeader Log (multicast orserial unicast) Member Status Commit Position @4096 Append Position @6912 Log Event @8192 Stream Positions Archive Position @8096 Archive Position @7168 Store locally asynchronous to Position processing by Consensus, & Log processing by Service Batching: Log, Appends, Commits
- 70. Doesn’t this ComplicateRecovery?
- 71. Follower Recovery Positions Archive Position@8096 Archive Position @7168 A synchronous system doesn’t make this complexity go away! Election still needs to assert state of the cluster & locally catch-up Follower Follower Archive Position @7584 Commit Position @4096 Commit Position @4064 Commit Position @4032 Service Position @4096 Service Position @4064 Service Position @3776
- 72. Limitations of Efficiency Throughput& Latency
- 73. Client FollowersLeader Ingress Log (multicastor serial unicast) Member Status Commit Position Append Position Log Event Client to Service A: 0.5 RTT Client to Service Ox: 1 RTT Client to Service A (on Commit): 1.5 RTT Client to Service Ox (on Commit): 2 RTT Constant Delay Network Service A Service Ox Round-Trip Time (RTT)
- 74. Client to ServiceA: 50ns Client to Service Ox: 100ns Client to Service A (on Commit): 150ns Client to Service Ox (on Commit): 200ns Limits from Constant Delay Shared Memory RTT <100ns Client to Service A: 50us Client to Service Ox: 100us Client to Service A (on Commit): 150us Client to Service Ox (on Commit): 200us DC RTT <100us Client to Service A: 5us Client to Service Ox: 10us Client to Service A (on Commit): 15us Client to Service Ox (on Commit): 20us Rack (Kernel Bypass) RTT <10us
- 75. Measured Latency atThroughput RTT(us) 0 75 150 225 300 Percentile Min 0.50 0.90 0.99 0.9999 0.999999 Max 100K msgs/sec 200K msgs/sec Intel Xeon Gold 5118 (2.30GHz, 12 cores) 32GB DDR4 2400 MHz ECC RAM Intel Optane SSD 900P Series 480GB SolarFlare X2522-PLUS 10GbE NIC All servers are connected to an Arista 7150S CentOS Linux 7.7, kernel 4.4.195-1.el7.elrepo.x86_64 tuned for low-latency workload. Courtesy Mark Price Single client session, bursts of 20x 200B messages, 3-node cluster, Service(s) echo(es) the payload back.
- 76. Takeways Efficiency is partof design Power of a timestamped, replicated log Replicated State Machines
- 77. Current Status Aeron Archiving- fully supported Aeron Clustering - pre-release Sponsored by https://weareadaptive.com/
- 79. Aeron: https://github.com/real-logic/Aeron Twitter: @toddlmontgomery ThankYou! Questions? StoneTor
- 80. Watch the videowith slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ aeron-cluster-raft/