Downloaded 468 times
Uploaded byMongoDB
Sharding Methods for MongoDB
Sharding in MongoDB allows for horizontal scaling of data and operations across multiple servers. When determining if sharding is needed, factors like available storage, query throughput, and response latency on a single server are considered. The number of shards can be calculated based on total required storage, working memory size, and input/output operations per second across servers. Different types of sharding include range, tag-aware, and hashed sharding. Choosing a high cardinality shard key that matches query patterns is important for performance. Reasons to shard include scaling to large data volumes and query loads, enabling local writes in a globally distributed deployment, and improving backup and restore times.
In this document
Powered by AI
Introduction to sharding in MongoDB, agenda includes customer stories and various sharding methods.
Foursquare and CarFax examples illustrate usage of sharding: Foursquare with 50M users, 6B check-ins; CarFax with 13B+ documents, 12 shards.
Defines sharding and illustrates scaling through examples of read/write scalability in MongoDB clusters.
Criteria for when to shard: assessing disk space, query throughput, and IOPS; examples provided for estimating shard requirements.
Introduction to different sharding types: range, tag-aware, and hashed sharding; pros and cons of each method.
Characteristics of a good shard key for performance; importance of cardinality and distribution to avoid hotspots.Reasons to implement sharding: scale, local writes, tiered storage, and faster restore processes described.
Summary on determining number of shards based on storage, latency, throughput; scalability, and geo-aware considerations.
Resources for MongoDB sharding, webinars, and expert advice; invitation to a Q&A session.
Closing thanks and contact information for inquiries regarding MongoDB.
More Related Content
![[2018] MySQL 이중화 진화기](https://cdn.slidesharecdn.com/ss_thumbnails/cloudinfra03-190131073325-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Sharding Methods for MongoDB
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=640&height=640&fit=bounds)
Sharding Methods for MongoDB
- 1. Sharding Methods For MongoDB JayRunkel jay.runkel@mongodb.com @jayrunkel #MongoDB
- 2. 2 • Customer Stories •Sharding for Performance/Scale – When to shard? – How many shards do I need? • Types of Sharding • How to Pick a Shard Key • Sharding for Other Reasons Agenda
- 3.
- 4.
- 5. 5 • 50M users. •6B check-ins to date (6M per day growth). • 55M points of interest / venues. • 1.7M merchants using the platform for marketing • Operations Per Second: 300,000 • Documents: 5.5B Foursquare
- 6. 6 • 11 MongoDBclusters – 8 are sharded • Largest cluster has 15 shards (check ins) – Sharded on user id Foursquare clusters
- 7. 7 • Large dataset CarFax
- 8. 8 • 13 billion+documents – 1.5 billion documents added every year • 1 vehicle history report is > 200 documents • 12 Shards • 9-node replica sets • Replicas distributed across 3 data centers CarFax Shards
- 9.
- 10.
- 11. 12 Sharding Overview Primary Secondary Secondary Shard 1 Primary Secondary Secondary Shard2 Primary Secondary Secondary Shard 3 Primary Secondary Secondary Shard N … Query Router Query Router Query Router …… Driver Application
- 12.
- 13. 15 Scaling: Sharding Read/Write Scalability mongodmongod Key Range 0..50 Key Range 51..100
- 14. 16 Scaling: Sharding mongod mongodmongod mongod Key Range 0..25 Key Range 26..50 Key Range 51..75 Key Range 76.. 100 Read/Write Scalability
- 15. How do Iknow I need to shard?
- 16. 18 Does one server/replica… •Have enough disk space to store all my data? • Handle my query throughput (operations per second)? • Respond to queries fast enough (latency)?
- 17. 19 • Have enoughdisk space to store all my data? • Handle my query throughput (operations per second)? • Respond to queries fast enough (latency)? Does one server/replica set… Server Specs Disk Capacity Disk IOPS RAM Network Disk IOPS RAM Network
- 18. How many shardsdo I need?
- 19. 21 • Sum ofdisk space across shards > greater than required storage size Disk Space: How Many Shards Do I Need?
- 20. 22 • Sum ofdisk space across shards > greater than required storage size Disk Space: How Many Shards Do I Need? Example Storage size = 3 TB Server disk capacity = 2 TB 2 Shards Required
- 21. 23 • Working setshould fit in RAM – Sum of RAM across shards > Working Set • WorkSet = Indexes plus the set of documents accessed frequently • WorkSet in RAM – Shorter latency – Higher Throughput RAM: How Many Shards Do I Need?
- 22. 24 • Measuring IndexSize and Working Set db.stats() – index size of each collection db.serverStatus({ workingSet: 1}) – working set size estimate RAM: How Many Shards Do I Need?
- 23. 25 • Measuring IndexSize and Working Set db.stats() – index size of each collection db.serverStatus({ workingSet: 1}) – working set size estimate RAM: How Many Shards Do I Need? Example Working Set = 428 GB Server RAM = 128 GB 428/128 = 3.34 4 Shards Required
- 24. 26 • Sum ofIOPS across shards > greater than required IOPS • IOPS are difficult to estimate – Update doc – Update indexes – Append to journal – Log entry? • Best approach – build a prototype and measure Disk Throughput: How Many Shards Do I Need
- 25. 27 • Sum ofIOPS across shards > greater than required IOPS • IOPS are difficult to estimate – Update doc – Update indexes – Append to journal – Log entry? • Best approach – build a prototype and measure Disk Throughput: How Many Shards Do I Need Example Required IOPS = 11000 Server disk IOPS = 5000 3 Shards Required
- 26. 28 • S =ops/sec of a single server • G = required ops/sec • N = # of shards • G = N * S * .7 N = G/.7S OPS: How Many Shards Do I Need?
- 27. 29 • S =ops/sec of a single server • G = required ops/sec • N = # of shards • G = N * S * .7 N = G/.7S OPS: How Many Shards Do I Need? Sharding Overhead
- 28. 30 • S =ops/sec of a single server • G = required ops/sec • N = # of shards • G = N * S * .7 N = G/.7S OPS: How Many Shards Do I Need? Example S = 4000 G = 10000 N = 3.57 4 Shards
- 29.
- 30. 32 • Range • Tag-Aware •Hashed Sharding Types
- 31. 33 Range Sharding mongod mongodmongod mongod Key Range 0..25 Key Range 26..50 Key Range 51..75 Key Range 76.. 100 Read/Write Scalability
- 32. 34 Tag-Aware Sharding mongod mongodmongod mongod Shard Tags Shard Tag Start End Winter 23 Dec 21 Mar Spring 22 Mar 21 Jun Summer 21 Jun 23 Sep Fall 24 Sep 22 Dec Tag Ranges Winter Spring Summer Fall
- 33. 35 Hash-Sharding mongod mongod mongodmongod Hash Range 0000..4444 Hash Range 4445..8000 Hash Range i8001..aaaa Hash Range aaab..ffff
- 34. 36 Hashed shard key •Pros: – Evenly distributed writes • Cons: – Random data (and index) updates can be IO intensive – Range-based queries turn into scatter gather Shard 1 mongos Shard 2 Shard 3 Shard N
- 35.
- 36.
- 37. How do IPick A Shard Key
- 38. 40 Shard Key characteristics •A good shard key has: – sufficient cardinality – distributed writes – targeted reads ("query isolation") • Shard key should be in every query if possible – scatter gather otherwise • Choosing a good shard key is important! – affects performance and scalability – changing it later is expensive
- 39. 41 Low cardinality shardkey • Induces "jumbo chunks" • Examples: boolean field Shard 1 mongos Shard 2 Shard 3 Shard N [ a, b )
- 40. 42 Ascending shard key •Monotonically increasing shard key values cause "hot spots" on inserts • Examples: timestamps, _id Shard 1 mongos Shard 2 Shard 3 Shard N [ ISODate(…), $maxKe
- 41.
- 42. 44 • Scale – Datavolume – Query volume • Global deployment with local writes – Geography aware sharding • Tiered Storage • Fast backup restore Reasons to shard
- 43.
- 44. 46 • Save hardwarecosts • Put frequently accessed documents on fast servers – Infrequently accessed documents on less capable servers • Use Tag aware sharding Tiered Storage mongod mongod mongod mongod Current Current Archive Archive SSD SSD HDD HDD
- 45. 47 • 40 TBDatabase • 2 shards of 20 TB each • Challenge – Cannot meet restore SLA after data loss Fast Restore mongod mongod 20 TB 20 TB
- 46. 48 • 40 TBDatabase • 4 shards of 10 TB each • Solution – Reduce the restore time by 50% Fast Restore mongod mongod 10 TB 10 TB mongod mongod 10 TB 10 TB
- 47.
- 48. 50 • To determinerequired # of shards determine – Storage requirements – Latency requirements – Throughput requirements • Derive total – Disk capacity – Disk throughput – RAM • Calculate # of shards based upon individual server specs Determining the # of shards
- 49. 51 • Scalability • Geo-awareclusters • Tiered Storage • Reduce backup restore times Leverage Sharding For
- 50. 52 • MongoDB Manual: http://docs.mongodb.org/manual/sharding/ •Other Webinars: – How to Achieve Scale With MongoDB • White Papers – MongoDB Performance Best Practices – MongoDB Architecture Guide Sharding: Where to go from here…
- 51. Get Expert Adviceon Scaling. For Free. For a limited time, if you’re considering a commercial relationship with MongoDB, you can sign up for a free one hour consult about scaling with one of our MongoDB Engineers. Sign Up: http://bit.ly/1rkXcfN
- 52. 54 Webinar Q&A jay.runkel@mongodb.com @jayrunkel Stay tuned afterthe webinar and take our survey for your chance to win MongoDB swag.
- 53.
Editor's Notes
- #14 MongoDB provides horizontal scale-out for databases using a technique called sharding, which is trans- parent to applications. Sharding distributes data across multiple physical partitions called shards. Sharding allows MongoDB deployments to address the hardware limitations of a single server, such as bottlenecks in RAM or disk I/O, without adding complexity to the application. MongoDB supports three types of sharding: • Range-based Sharding. Documents are partitioned across shards according to the shard key value. Documents with shard key values “close” to one another are likely to be co-located on the same shard. This approach is well suited for applications that need to optimize range- based queries. • Hash-based Sharding. Documents are uniformly distributed according to an MD5 hash of the shard key value. Documents with shard key values “close” to one another are unlikely to be co-located on the same shard. This approach guarantees a uniform distribution of writes across shards, but is less optimal for range-based queries. • Tag-aware Sharding. Documents are partitioned according to a user-specified configuration that associates shard key ranges with shards. Users can optimize the physical location of documents for application requirements such as locating data in specific data centers. MongoDB automatically balances the data in the cluster as the data grows or the size of the cluster increases or decreases.
- #42 may consider hashing _id instead
- #43 may consider hashing _id instead
- #54 www.mongodb.com/lp/contact/scaling-101 http://www.mongodb.com/lp/contact/planning-for-scale