Odinot Stanislas, profile picture
Uploaded byOdinot Stanislas
PDF, PPTX24,447 views

Ceph: Open Source Storage Software Optimizations on Intel® Architecture for Cloud Workloads

The document discusses Ceph, an open-source, scalable storage solution that addresses the growing data demands and storage costs by optimizing its performance on Intel architecture. It highlights Ceph's features such as cache tiering, erasure coding, and its implementation in cloud environments, alongside best practices and Intel's role in enhancing its capabilities. Ceph is positioned as a preferred option for distributed storage with strong customer interest and several successful production deployments.

Embed presentation

Download as PDF, PPTX
1 Make the Future with China! Ceph: Open Source Storage Software Optimizations on Intel® Architecture for Cloud Workloads Jian Zhang – Software Engineer, Intel Corporation DATS005
2 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
3 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
4 The Problem: Data Big Bang From 2013 to 2020, the digital universe will grow by a factor of 10, from 4.4 ZB to 44 ZB It more than doubles every two years. Data needs are growing at a rate unsustainable with today’s infrastructure and labor costs Source: IDC – The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things - April 2014 1 2 3 4 7 12 19 30 48 77 125 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 1.2 COST CHALLENGES COTINUE TO GROW Storage Cost structure needs a fundamental shift Storage Capacity in TB 62% CAGR IT Budget 2% CAGR IT PROS WILL SHOULDER A GREATER STORAGE BURDEN 230 GB Per IT Pro 1,231 GB Per IT Pro
5 Diverse Workloads & Cost Drive Need for Distributed Storage Traditional Workloads Today’s Workloads Diverse Workloads Management Scale on Demand Increasing Complexity Cost Traditional Workloads Today’s Trends Storage Fabric Network Fabric Mobile HPC Big Data & Analytics CloudOLTP Email ERP Apps Apps Challenges CRM
6 Distributed Storage Traditional Scale-up Model SAN (Storage Area Network) Server High Availability (Failover) High perf workloads (e.g., database) Enterprise Mission Critical Hybrid Cloud Limited Scale Costly (Cap-ex and Op-ex) Pay as you Grow, massive on-demand scale Cost, Performance optimized Open and commercial solutions on x86 servers Applicable to cloud workloads Not a good fit for traditional high perf workloads • Ceph is the most popular† open source virtual block storage option. Also provides object, file (experimental). • Strong customer interest - several production implementations already. Distributed Storage Model Application Servers Storage Nodes Storage Client Metadata Servers Converged Network † OpenStack User Survey Insights: http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014
7 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
8 Ceph Introduction • Ceph is an open-source, massively scalable, software-defined storage system which provides object, block and file system storage in a single platform. It runs on commodity hardware—saving you costs, giving you flexibility—and because it’s in the Linux* kernel, it’s easy to consume. • Object Store (RADOSGW) - A bucket based REST gateway - Compatible with S3 and swift • File System (CEPH FS) - A POSIX-compliant distributed file system - Kernel client and FUSE • Block device service (RBD) - OpenStack* native support - Kernel client and QEMU/KVM driver RADOS A software-based, reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes and lightweight monitors LIBRADOS A library allowing apps to directly access RADOS RGW A web services gateway for object storage Application RBD A reliable, fully distributed block device CephFS A distributed file system with POSIX semantics Host/VM Client
9 Ceph Cluster Overview Client Servers Storage Servers • Ceph Clients - Block/Object/File system storage - User space or kernel driver • Peer to Peer via Ethernet - Direct access to storage - No centralized metadata = no bottlenecks • Ceph Storage Nodes - Data distributed and replicated across nodes - No single point of failure - Scale capacity and performance with additional nodes Ceph scales to 1000s of nodes Ethernet KVM Application Guest OS Application Guest OS KVM Application Guest OS Application Guest OS KVM Application Guest OS Application Guest OS RADOS RBD Object File RADOS RBD Object File RADOS RBD Object File SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD MON MON
10 Object Store Daemon (OSD) Read and Write Flow OSD Disk OSD OSD OSD Disk Disk Disk Compute Node RBD RADOS App OSD Disk OSD OSD OSD Disk Disk Disk OSD Disk OSD OSD OSD Disk Disk Disk 1 2 2 3 3 4 Server Server Server Compute Node RBD RADOS App 1 2 WriteRead 1 2 3 4 Client app writes data, RADOS sends data to primary OSD Primary OSD identifies replica OSDs and sends data, writes data to local disk Replica OSDs write data to local disk, signal completion to primary Primary OSD signals completion to client app Client app issues read request, RADOS sends request to primary OSD Primary OSD reads data from local disk and completes read request 1 2
11 Ceph WorkloadsStoragePerformance (IOPS,Throughput) Storage Capacity (PB) Lower Higher Virtual Machines LowerHigher Virtual Block Media Streaming Enterprise Dropbox Backup, Archive Remote Disks VDI App Storage HPC Batch analytics Mobile Content Depot Databases Block Object Current Future Intel Focus Test & Dev FS
12 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
13 Ceph Block Performance – Configuration 1x10Gb NIC Test Environment Compute Node • 2 nodes with Intel® Xeon™ processor x5570 @ 2.93GHz, 128GB mem • 1 node with IntelXeon processor E5 2680 @2.8GHz, 56GB mem Storage Node • IntelXeon processor E3-1275 v2 @ 3.5 GHz • 32GB Memory • 1xSSD for OS • 10x 3 TB 7200rpm • 2x 400GB Intel® SSD DC S3700 CEPH1 MON OSD1 OSD10… CEPH2 OSD1 OSD10… CEPH3 OSD1 OSD10… CEPH4 OSD1 OSD10… KVM VM1 FIO FIO VM40 CLIENT 1 KVM VM1 FIO FIO VM40 CLIENT 2 KVM VM1 FIO FIO VM40 CLIENT 3 2x10Gb NIC Note: See page #37, #38, #39 for system configuration and benchmark data
14 Ceph Block Performance – Measure Raw Performance 3 TB 7200 rpm 1 Run FIO on one HDD, collect disk IO performance Note: Sequential 64K (Client) = Random 512K (Ceph OSD) 2 Estimate cluster performance (include replication overhead for writes – 2x in this test) 3 TB 7200 rpm3 TB 7200 rpm3 TB 7200 rpm20 HDDs (Writes) 40 HDDs (Reads) 70 MB/s 80 MB/s 270 IOPS 160 IOPS Random 512K Write Read Random 4K Write Read 1400 MB/s 3200 MB/s 5400 IOPS 6400 IOPS Write Read Random 4K Write Read Random 512K Note: See page #37, #38, #39 for system configuration and benchmark data
15 0 1000 2000 3000 4000 5000 6000 7000 4K Rand Read - IOPS 4K Rand Write - IOPS 64K Seq Read - MB/s 64K Seq Write - MB/s Maximum Possible Peak Observed Ceph Block Performance – Test Results Drop OSD Cache Prepare Data (dd) Run FIO 1. 60GB Span 2. 4 IOs: Sequential (W,R), Random (W, R) 3. 100s warm-up, 600s test 4. RBD images – 1 to 120 Note: Random tests use Queue Depth=8, Sequential tests use Queue Depth=64 See page #39, #40, #41 for system configuration and benchmark data CEPH Cluster Performance 88% 64% 102% 94% Ceph performance is close to max cluster IO limit for all but random writes – room for further optimizations
16 Ceph Block Performance – Tuning effects 0 1000 2000 3000 4000 5000 6000 7000 64K Seq Write - MB/s 64K Seq Read - MB/s 4K Rand Write - IOPS 4K Rand Read - IOPS Ceph Block Performance Tuning Impact --compared with default ceph.conf Throughput W/ Tunings Throughput 2.36x 1.88x 3.59x 2.00x Best Tuning Knobs Read ahead = 2048 I/O merge & write cache Omap data on a separate Disk Large pg number: 81920 Note: See page #37, #38, #39 for system configuration and benchmark data
17 Ceph Object Performance – Configuration 1x10Gb NIC Test Environment Client Node • 2 nodes with Intel® Xeon™ processor x5570 @ 2.93GHz, 24GB mem Storage Node • IntelXeon processor E3-1280 v2 @ 3.6 GHz • 16 GB Memory • 1xSSD for OS • 10x1 TB 7200rpm • 3x 480GB Intel® SSD S530 CEPH1 MON OSD1 OSD10… CEPH2 OSD1 OSD10… CEPH3 OSD1 OSD10… CEPH4 OSD1 OSD10… KVM VM1 FIO FIO VM20 CLIENT 1 KVM VM1 FIO FIO VM20 CLIENT 2 2x10Gb NIC Note: See page #40, #41 for system configuration and benchmark data Rados Gateway (RGW) Rados Gateway • 1 nodes with Intel Xeon processor E5-2670 @ 2.6GHz, 64GB mem
18 Ceph Object Performance – Test Results Prepare the Data Run COSBench 1. 100 containers x 100 objects each 2. 4 IOs: 128K Read/Write; 10M Read/write 3. 100s warm-up, 600s test 4. COSBench workers – 1 to 2048 Note: COSBench is an Intel development open source cloud object storage benchmark https://github.com/intel-cloud/cosbench Note: See page #42, #43 for system configuration and benchmark data CEPH Cluster Performance Ceph Object performance is close to max cluster IO limit #con x # obj Object- Size RW-Mode Worker- Count Avg- ResTime 95%- ResTime Throughput Bandwidth Bottleneck -- -- -- ms ms op/s MB/s -- 100x100 128KB Read 80 10 20 7,951 971 RGW CPU Write 320 143 340 2,243 274 OSD CPU 10MB Read 160 1,365 3,870 117 1,118 RGW NIC Write 160 3,819 6,530 42 397 OSD NIC
19 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
20 Ceph Cache Tiering • An important feature towards Ceph enterprise readiness - Cost-effective - High performance tier W/ more SSDs - Use a pool of fast storage devices (Typically SSDs) and use it as a cache for an existing larger pool  E.g., Reads would first check the cache pool for a copy of the object, and then fall through to the existing pool if there is a miss • Cache tiering mode - Read only - Write back Application Ceph Storage Cluster Cache Pool (Replicated) Backing Pool (Erasure Coded)
21 Ceph Cache Tiering Optimization – Proxy Read/write Proxy the read/write operations while the object is missing in cache tier, promotion in background • 3.8x and 3.6x performance improvement respectively with proxy-write and proxy-read optimization Promotion logic Cache tier Base tier Client ① ② ③ ④ Promotion logic Cache tier Base tier Client ① ② ③ ④ Proxy logic Current design Proxy design 0 2000 4000 6000 8000 0 500 1000 1500 2000 RR RW Latency(ms) IOPS W/ CT IOPS W/ CT optimized IOPS W/ CT Latency W/ CT optimized Latency Cache Tiering optimization 4K random write Proxy-read and write significantly improved cache tiering performance Note: See page #37, #38, #39 for system configuration and benchmark data
22 Ceph Erasure Coding Ceph Storage Cluster Replicated Pool COPY COPY COPY Object Ceph Storage Cluster Erasure Coded Pool 1 Object 2 3 4 X Y Full Copies of stored objects • Very high durability • 3x (200% overhead) • Quicker recovery One Copy plus parity • Cost-effective durability • 1.5X (50% store overhead) • Expensive recovery
23 Ceph EC optimization – I/O hint • ISA-L EC library merged to Firefly • EC performance - Acceptable performance impact  <10% degradation for 10M objects large scale read/write tests - But: Compared with 3x replica, we now tolerate 40% object loss with 1.6x space • Rados I/O hint - Provide a hint to the storage system to classify the operate type brings differentiated storage services - Balance throughput and cost, and boost Storage performance - With rados I/O hint optimization, we can get even higher throughput compared W/O EC! Client Node RADOS Node Hypervisor Guest VM Qemu/Virtio Application RBD RADOS RADOS Protocol OSD FileStore NewStore… SSD Network Hint MemStore KVStore Hint engine 35% performance improvement for EC write with Rados I/O Hint SSD SSD SSD SSD Hint engine Hint engine Hint engine I/O hint policy engine Hint Note: See page #40, #41 for system configuration and benchmark data
24 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
25 How is Intel helping? • Deliver Storage workload optimized products and technologies - Optimize for Silicon, Flash* and Networking technologies • Working with the community to optimize Ceph on IA - Make Ceph run best on Intel® Architecture (IA) based platforms (performance, reliability and management) • Publish IA Reference Architectures - Share best practices with the ecosystem and community
26 CPU Platforms NVM Networking Server Boards & Systems Software Libraries Software Products Opensource Ecosystem Intel’s Product Portfolio for Ceph Intel® Cache Acceleration Software (Intel® CAS) • OEMs / ODMs • ISVs Intel® Storage Acceleration Library (Intel® ISA-L) • Virtual Storage Manager (VSM) • Ceph contributions: No.2 • Cephperf† • Reference Architectures Solution focus with Intel® platform and software ingredients. Deep collaboration with Red Hat* and Inktank* (by Red Hat) to deliver enterprise ready Ceph solutions. †Cephperf to be open sourced in Q2’15
27 VSM– Ceph Simplified Home page: https://01.org/virtual-storage-manager Code Repository: https://github.com/01org/virtual-storage- manager Issue Tracking: https://01.org/jira/browse/VSM Mailing list: http://vsm-discuss.33411.n7.nabble.com/ VSM (Virtual Storage Manager) - An open source Ceph management tool developed by Intel, and announced on 2014 Nov’s OpenStack* Paris summit, designed to help make day to day management of Ceph easier for storage administrators.
28 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
29 Mon Caching Compute Node RADOS Node RADOS Protocol RADOS Protocol OSD Journal Filestore SSD File System Network SSD MON Ceph – Best Deployment Practices Note: Refer to backup for detailed tuning recommendations. • XFS, deadline scheduler • Tune read_ahead_kb for sequential I/O read • Queue Depth – 64 for sequential, 8 for random Jumbo Frames for 10Gbps Use 10x of default queue params • In small cluster, can co- locate with OSDs with 4GB-6GB • Beyond 100 OSDs (100 HDDs or 10 nodes), deploy monitors on separate nodes • 3 monitors for < 200 nodes • One OSD process per disk • Approximately 1GHz Intel® Xeon™-class CPU per OSD • 1GB memory per OSD
30 Agenda • The Problem • Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
31 Summary • Cloud workloads and cost is driving the need for distributed storage solutions • Strong customer interest and lots of production implementations in Ceph • Intel is optimizing CEPH for Intel® Architecture
32 Next Steps • Take advantage of Intel software optimizations and reference architectures for your production deployments - Pilot “cephperf” in Q2’15 and give us feedback http://01.org/cephperf • Engage with open source communities for delivering enterprise features • Innovate storage offerings using value added features with Ceph
33 Additional Sources of Information • A PDF of this presentation is available from our Technical Session Catalog: www.intel.com/idfsessionsSZ. This URL is also printed on the top of Session Agenda Pages in the Pocket Guide. • More web based info: http://ceph.com • Intel® Solutions Reference Architectures www.intel.com/storage • Intel® Storage Acceleration Library (Open Source Version) - https://01.org/intel%C2%AE-storage-acceleration-library-open-source- version
34 Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. © 2015 Intel Corporation.
35 Risk Factors The above statements and any others in this document that refer to plans and expectations for the first quarter, the year and the future are forward- looking statements that involve a number of risks and uncertainties. Words such as "anticipates," "expects," "intends," "plans," "believes," "seeks," "estimates," "may," "will," "should" and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel's actual results, and variances from Intel's current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations. Demand for Intel’s products is highly variable and could differ from expectations due to factors including changes in the business and economic conditions; consumer confidence or income levels; customer acceptance of Intel’s and competitors’ products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Intel’s gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing quality/yields. Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing expenses, and Intel’s ability to respond quickly to technological developments and to introduce new features into existing products, which may result in restructuring and asset impairment charges. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Results may also be affected by the formal or informal imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice. Intel operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term. The amount, timing and execution of Intel’s stock repurchase program and dividend program could be affected by changes in Intel’s priorities for the use of cash, such as operational spending, capital spending, acquisitions, and as a result of changes to Intel’s cash flows and changes in tax laws. Product defects or errata (deviations from published specifications) may adversely impact our expenses, revenues and reputation. Intel’s results could be affected by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. Intel’s results may be affected by the timing of closing of acquisitions, divestitures and other significant transactions. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the company’s most recent reports on Form 10-Q, Form 10-K and earnings release. Rev. 1/15/15
36 Backup
37 Block Test Environment - Configuration Details Client Nodes CPU 2 x Intel ®Xeon™ x5570 @ 2.93GHz (4-core, 8 threads) (Qty: 2) 2x Intel Xeon E5 2680 @2.8GHz (16-core, 32 threads) (Qty: 1) Memory 128 GB (8GB * 16 DDR3 1333 MHZ) or 56GB (8GB * 7) for E5 server NIC 10Gb 82599EB Disks 1 HDD for OS Client VM CPU 1 X VCPU VCPUPIN Memory 512 MB Ceph Nodes CPU 1 x Intel Xeon E3-1275 V2 @ 3.5 GHz (4-core, 8 threads) Memory 32 GB (4 x 8GB DDR3 @ 1600 MHz) NIC 2 X 82599 10GbE HBA/C204 {SAS2008 PCI Express* Fusion-MPT SAS-2} / {6 Series/C200 Series Chipset Family SATA AHCI Controller} Disks 1 x SSDSA2SH064G1GC 2.5’’ 64GB for OS 2 x Intel SSDSC2BA40 400 GB SSD (Journal) 10 x Seagate* ST3000NM0033-9ZM 3.5’’ 3TB 7200rpm SATA HDD (Data) Ceph cluster OS CentOS 6.5 Kernel 2.6.32-431 Ceph 0.61.2 built from source Client host OS Ubuntu* 12.10 Kernel 3.6.3 Client VM OS Ubuntu 12.10 Kernel 3.5.0-17 • XFS as file system for Data Disk • Each Data Disk (SATA HDD) was parted into 1 partition for OSD daemon • Default replication setting (2 replicas), 7872 pgs. • Tunings - Set read_ahead_kb=2048 - MTU= 8000 • Change i/o scheduler to [deadline]: # Echo deadline >/sys/block/[dev]/queue/scheduler
38 Ceph RBD Tuned Configuration [global] debug default = 0 log file = /var/log/ceph/$name.log max open files = 131072 auth cluster required = none auth service required = none auth client required = none [osd] osd mkfs type = xfs osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog osd mkfs options xfs = -f -i size=2048 filestore max inline xattr size = 254 filestore max inline xattrs = 6 osd_op_threads=20 filestore_queue_max_ops=500 filestore_queue_committing_max_ops=5000 journal_max_write_entries=1000 journal_queue_max_ops=3000 objecter_inflight_ops=10240 filestore_queue_max_bytes=1048576000 filestore_queue_committing_max_bytes =1048576000 journal_max_write_bytes=1048576000 journal_queue_max_bytes=1048576000 ms_dispatch_throttle_bytes=1048576000 objecter_infilght_op_bytes=1048576000 filestore_max_sync_interval=10 filestore_flusher=false filestore_flush_min=0 filestore_sync_flush=true
39 Testing Methodology Storage interface Use QemuRBD as storage interface Tool • Use “dd” to prepare data for R/W tests • Use fio (ioengine=libaio, direct=1) to generate 4 IO patterns: sequential write/read, random write/read • Access Span: 60GB • For capping tests, Seq Read/Write (60MB/s), and Rand Read/Write (100 ops/s) • QoS Compliance: - For random 4k read/write cases: latency <= 20ms - For sequential 64K read/write cases: BW >= 54 MB/s Run rules • Drop osds page caches ( “1” > /proc/sys/vm/drop_caches) • 100 secs for warm up, 600 secs for data collection • Run 4KB/64KB tests under different # of rbds (1 to 120) Space allocation (per node) • Data Drive: - Sits on 10x 3TB HDD drives - So 4800GB/40 * 2 = 240GB data space will be used on each Data disk at 80 VMs. • Journal: - Sits on 2x 400GB SSD drives - One journal partition per data drive, 10GB
40 Object Test Environment - Configuration Details Client & RGW CPU Client: 2 x Intel ®Xeon™ x5570 @ 2.93GHz (4-core, 8 threads) (Qty: 2) GRW: 2x Intel Xeon E5 2670@2.6GHz (16-core, 32 threads) (Qty: 1) Memory 128 GB (8GB * 16 DDR3 1333 MHZ) or 56GB (8GB * 7) for E5 server NIC 10Gb 82599EB Disks 1 HDD for OS Ceph OSD Nodes CPU 1 x Intel Xeon E3-1280 V2 @ 3.6 GHz (4-core, 8 threads) Memory 32 GB (4 x 8GB DDR3 @ 1600 MHz) NIC 2 X 82599 10GbE HBA/C204 {SAS2308 PCI Express* Fusion-MPT SAS-2} / {6 Series/C200 Series Chipset Family SATA AHCI Controller} Disks 1 x SSDSA2SH064G1GC 2.5’’ 64GB for OS 3 x Intel SSDSC2CW480A3 480 GB SSD (Journal) 10 x Seagate* ST1000NM0011 3.5’’ 1TB 7200rpm SATA HDD (Data) Ceph cluster OS Ubuntu 14.04 Kernel 3.13.0 Ceph 0.61.8 built from source Client host OS Ubuntu* 12.10 Kernel 3.6.3 • XFS as file system for Data Disk • Each Data Disk (SATA HDD) was parted into 1 partition for OSD daemon • Default replication setting (3 replicas), 12416 pgs. • Tunings - Set read_ahead_kb=2048 - MTU= 8000
41 Ceph Rados Gateway (RGW) Tuning ceph-gateway –rgw enable ops log = false –rgw enable usage log = false –rgw thread pool size = 256 –Log disabled apache2-http-server –<IfModule mpm_worker_module> – ServerLimit 50 – StartServers 5 – MinSpareThreads 25 – MaxSpareThreads 75 – ThreadLimit 100 – ThreadsPerChild 100 – MaxClients 5000 – MaxRequestsPerChild 0 –</IfModule> GW instances –5 instances in one GW server –Access log turned off

More Related Content

PDF
Storage tiering and erasure coding in Ceph (SCaLE13x)
bySage Weil
 
PPTX
Ceph Introduction 2017
byKaran Singh
 
PDF
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
byKaran Singh
 
PPTX
Ceph Performance and Sizing Guide
byJose De La Rosa
 
PDF
BlueStore, A New Storage Backend for Ceph, One Year In
bySage Weil
 
PDF
Ceph Object Storage Reference Architecture Performance and Sizing Guide
byKaran Singh
 
PDF
Ceph - A distributed storage system
byItalo Santos
 
PDF
A crash course in CRUSH
bySage Weil
 
Storage tiering and erasure coding in Ceph (SCaLE13x)
bySage Weil
 
Ceph Introduction 2017
byKaran Singh
 
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
byKaran Singh
 
Ceph Performance and Sizing Guide
byJose De La Rosa
 
BlueStore, A New Storage Backend for Ceph, One Year In
bySage Weil
 
Ceph Object Storage Reference Architecture Performance and Sizing Guide
byKaran Singh
 
Ceph - A distributed storage system
byItalo Santos
 
A crash course in CRUSH
bySage Weil
 

What's hot

PDF
Ceph and RocksDB
bySage Weil
 
PDF
Ceph Block Devices: A Deep Dive
byRed_Hat_Storage
 
PDF
[OpenInfra Days Korea 2018] Day 2 - CEPH 운영자를 위한 Object Storage Performance T...
byOpenStack Korea Community
 
PPTX
ceph optimization on ssd ilsoo byun-short
byNAVER D2
 
PDF
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
byDanielle Womboldt
 
PDF
Ceph c01
byLâm Đào
 
PDF
Nick Fisk - low latency Ceph
byShapeBlue
 
PDF
Ceph scale testing with 10 Billion Objects
byKaran Singh
 
PDF
Red Hat OpenStack 17 저자직강+스터디그룹_1주차
byNalee Jang
 
PDF
[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기
byIan Choi
 
PDF
2021.02 new in Ceph Pacific Dashboard
byCeph Community
 
PPTX
OVN - Basics and deep dive
byTrinath Somanchi
 
PDF
Ceph RBD Update - June 2021
byCeph Community
 
PDF
OpenStack 개요 및 활용 사례 @ Community Open Camp with Microsoft
byIan Choi
 
PDF
LinuxCon 2015 Linux Kernel Networking Walkthrough
byThomas Graf
 
PDF
Ceph Month 2021: RADOS Update
byCeph Community
 
PDF
Using eBPF for High-Performance Networking in Cilium
byScyllaDB
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
PDF
Ceph issue 해결 사례
byOpen Source Consulting
 
PDF
2019.06.27 Intro to Ceph
byCeph Community
 
Ceph and RocksDB
bySage Weil
 
Ceph Block Devices: A Deep Dive
byRed_Hat_Storage
 
[OpenInfra Days Korea 2018] Day 2 - CEPH 운영자를 위한 Object Storage Performance T...
byOpenStack Korea Community
 
ceph optimization on ssd ilsoo byun-short
byNAVER D2
 
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
byDanielle Womboldt
 
Ceph c01
byLâm Đào
 
Nick Fisk - low latency Ceph
byShapeBlue
 
Ceph scale testing with 10 Billion Objects
byKaran Singh
 
Red Hat OpenStack 17 저자직강+스터디그룹_1주차
byNalee Jang
 
[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기
byIan Choi
 
2021.02 new in Ceph Pacific Dashboard
byCeph Community
 
OVN - Basics and deep dive
byTrinath Somanchi
 
Ceph RBD Update - June 2021
byCeph Community
 
OpenStack 개요 및 활용 사례 @ Community Open Camp with Microsoft
byIan Choi
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
byThomas Graf
 
Ceph Month 2021: RADOS Update
byCeph Community
 
Using eBPF for High-Performance Networking in Cilium
byScyllaDB
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Ceph issue 해결 사례
byOpen Source Consulting
 
2019.06.27 Intro to Ceph
byCeph Community
 

Viewers also liked

PDF
Kakao Openstack CI/CD
by어형 이
 
PDF
Cloud data center and openstack
byAndrew Yongjoon Kong
 
PPTX
OpenStack Cinder
byDeepti Ramakrishna
 
PDF
[OpenStack Days Korea 2016] Track1 - Red Hat enterprise Linux OpenStack Platform
byOpenStack Korea Community
 
PDF
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
byOpenStack Korea Community
 
PDF
[OpenStack Days Korea 2016] Track1 - 카카오는 오픈스택 기반으로 어떻게 5000VM을 운영하고 있을까?
byOpenStack Korea Community
 
Kakao Openstack CI/CD
by어형 이
 
Cloud data center and openstack
byAndrew Yongjoon Kong
 
OpenStack Cinder
byDeepti Ramakrishna
 
[OpenStack Days Korea 2016] Track1 - Red Hat enterprise Linux OpenStack Platform
byOpenStack Korea Community
 
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
byOpenStack Korea Community
 
[OpenStack Days Korea 2016] Track1 - 카카오는 오픈스택 기반으로 어떻게 5000VM을 운영하고 있을까?
byOpenStack Korea Community
 

Similar to Ceph: Open Source Storage Software Optimizations on Intel® Architecture for Cloud Workloads

PDF
Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS
byCeph Community
 
PPTX
Ceph Community Talk on High-Performance Solid Sate Ceph
byCeph Community
 
PPTX
Using Recently Published Ceph Reference Architectures to Select Your Ceph Con...
byPatrick McGarry
 
PPTX
Using Recently Published Ceph Reference Architectures to Select Your Ceph Con...
byCeph Community
 
PDF
NAVER Ceph Storage on ssd for Container
byJangseon Ryu
 
PDF
Red Hat Storage Day Atlanta - Designing Ceph Clusters Using Intel-Based Hardw...
byRed_Hat_Storage
 
PPTX
New Ceph capabilities and Reference Architectures
byKamesh Pemmaraju
 
PPTX
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
byRed_Hat_Storage
 
PDF
Ceph in 2023 and Beyond.pdf
byClyso GmbH
 
PDF
Ceph
byShravya Reddy
 
PDF
Red hat Storage Day LA - Designing Ceph Clusters Using Intel-Based Hardware
byRed_Hat_Storage
 
ODP
Ceph Day SF 2015 - Keynote
byCeph Community
 
PDF
Linux Stammtisch Munich: Ceph - Overview, Experiences and Outlook
byDanny Al-Gaaf
 
PPTX
openSUSE storage workshop 2016
byAlex Lau
 
PDF
Ceph as software define storage
byMahmoud Shiri Varamini
 
PDF
Backup management with Ceph Storage - Camilo Echevarne, Félix Barbeira
byCeph Community
 
PPTX
Dfs in iaa_s
byChih-Chieh Huang
 
PPTX
ceph-barcelona-v-1.2
byRanga Swami Reddy Muthumula
 
PPTX
Ceph barcelona-v-1.2
byRanga Swami Reddy Muthumula
 
PDF
Introduction into Ceph storage for OpenStack
byOpenStack_Online
 
Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS
byCeph Community
 
Ceph Community Talk on High-Performance Solid Sate Ceph
byCeph Community
 
Using Recently Published Ceph Reference Architectures to Select Your Ceph Con...
byPatrick McGarry
 
Using Recently Published Ceph Reference Architectures to Select Your Ceph Con...
byCeph Community
 
NAVER Ceph Storage on ssd for Container
byJangseon Ryu
 
Red Hat Storage Day Atlanta - Designing Ceph Clusters Using Intel-Based Hardw...
byRed_Hat_Storage
 
New Ceph capabilities and Reference Architectures
byKamesh Pemmaraju
 
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
byRed_Hat_Storage
 
Ceph in 2023 and Beyond.pdf
byClyso GmbH
 
Ceph
byShravya Reddy
 
Red hat Storage Day LA - Designing Ceph Clusters Using Intel-Based Hardware
byRed_Hat_Storage
 
Ceph Day SF 2015 - Keynote
byCeph Community
 
Linux Stammtisch Munich: Ceph - Overview, Experiences and Outlook
byDanny Al-Gaaf
 
openSUSE storage workshop 2016
byAlex Lau
 
Ceph as software define storage
byMahmoud Shiri Varamini
 
Backup management with Ceph Storage - Camilo Echevarne, Félix Barbeira
byCeph Community
 
Dfs in iaa_s
byChih-Chieh Huang
 
ceph-barcelona-v-1.2
byRanga Swami Reddy Muthumula
 
Ceph barcelona-v-1.2
byRanga Swami Reddy Muthumula
 
Introduction into Ceph storage for OpenStack
byOpenStack_Online
 

More from Odinot Stanislas

PDF
Silicon Photonics and datacenter
byOdinot Stanislas
 
PDF
Using a Field Programmable Gate Array to Accelerate Application Performance
byOdinot Stanislas
 
PDF
Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...
byOdinot Stanislas
 
PDF
SDN/NFV: Service Chaining
byOdinot Stanislas
 
PDF
SNIA : Swift Object Storage adding EC (Erasure Code)
byOdinot Stanislas
 
PDF
PCI Express* based Storage: Data Center NVM Express* Platform Topologies
byOdinot Stanislas
 
PDF
Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...
byOdinot Stanislas
 
PDF
Software Defined Storage - Open Framework and Intel® Architecture Technologies
byOdinot Stanislas
 
PDF
Virtualizing the Network to enable a Software Defined Infrastructure (SDI)
byOdinot Stanislas
 
PDF
Accelerate the SDN with Intel ONP
byOdinot Stanislas
 
PDF
Moving to PCI Express based SSD with NVM Express
byOdinot Stanislas
 
PDF
Intel Cloud Builder : Siveo
byOdinot Stanislas
 
PDF
Configuration and deployment guide for SWIFT on Intel Architecture
byOdinot Stanislas
 
PDF
Intel IT Open Cloud - What's under the Hood and How do we Drive it?
byOdinot Stanislas
 
PDF
Configuration and Deployment Guide For Memcached on Intel® Architecture
byOdinot Stanislas
 
PDF
Améliorer OpenStack avec les technologies Intel
byOdinot Stanislas
 
PDF
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
byOdinot Stanislas
 
PDF
Big Data and Intel® Intelligent Systems Solution for Intelligent transportation
byOdinot Stanislas
 
PDF
Big Data Solutions for Healthcare
byOdinot Stanislas
 
PDF
Protect Your Big Data with Intel<sup>®</sup> Xeon<sup>®</sup> Processors a..
byOdinot Stanislas
 
Silicon Photonics and datacenter
byOdinot Stanislas
 
Using a Field Programmable Gate Array to Accelerate Application Performance
byOdinot Stanislas
 
Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...
byOdinot Stanislas
 
SDN/NFV: Service Chaining
byOdinot Stanislas
 
SNIA : Swift Object Storage adding EC (Erasure Code)
byOdinot Stanislas
 
PCI Express* based Storage: Data Center NVM Express* Platform Topologies
byOdinot Stanislas
 
Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...
byOdinot Stanislas
 
Software Defined Storage - Open Framework and Intel® Architecture Technologies
byOdinot Stanislas
 
Virtualizing the Network to enable a Software Defined Infrastructure (SDI)
byOdinot Stanislas
 
Accelerate the SDN with Intel ONP
byOdinot Stanislas
 
Moving to PCI Express based SSD with NVM Express
byOdinot Stanislas
 
Intel Cloud Builder : Siveo
byOdinot Stanislas
 
Configuration and deployment guide for SWIFT on Intel Architecture
byOdinot Stanislas
 
Intel IT Open Cloud - What's under the Hood and How do we Drive it?
byOdinot Stanislas
 
Configuration and Deployment Guide For Memcached on Intel® Architecture
byOdinot Stanislas
 
Améliorer OpenStack avec les technologies Intel
byOdinot Stanislas
 
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
byOdinot Stanislas
 
Big Data and Intel® Intelligent Systems Solution for Intelligent transportation
byOdinot Stanislas
 
Big Data Solutions for Healthcare
byOdinot Stanislas
 
Protect Your Big Data with Intel<sup>®</sup> Xeon<sup>®</sup> Processors a..
byOdinot Stanislas
 

Recently uploaded

PDF
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 
PDF
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
PDF
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
PDF
Oracle MySQL HeatWave - Complete - Version 3
byLuca Bonesini
 
PDF
UnityNet AI Induced Harm & Youth Protection Position Statement 2025-11-15
byLHelferty
 
PDF
IDSECCONF2025 - Faisal Ilham - Semi Automating Vulnerability Scanner and Expl...
byidsecconf
 
PDF
Mutation Testing for Industrial Robotic Systems (FMAS 2025)
bySylvain Hallé
 
PPTX
Agentic AI presentation with Python Exercises
byParth Mane
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PDF
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
bywesley chun
 
PDF
IDSECCONF2025 - Nosa Shandy - Discovering and Disclosing Privacy Vulnerabilit...
byidsecconf
 
PDF
IDSECCONF2025 - Ali - DursGo–Web Security Scanner with AI Analysis.pdf
byidsecconf
 
PPTX
Building powerful web apps to improve productivity and engagement - Esri UK W...
byEsri UK
 
PPTX
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
PDF
Running Non-Cloud-Native Databases in Cloud-Native Environments_ Challenges a...
byAlkin Tezuysal
 
PPTX
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PDF
SELinux Policy Management in RHEL - RHCSA+.pdf
byLinuxCert Guru
 
PPTX
Redcentric Data Centres: vision, mission and values
byRedcentric
 
PPTX
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
PDF
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
Oracle MySQL HeatWave - Complete - Version 3
byLuca Bonesini
 
UnityNet AI Induced Harm & Youth Protection Position Statement 2025-11-15
byLHelferty
 
IDSECCONF2025 - Faisal Ilham - Semi Automating Vulnerability Scanner and Expl...
byidsecconf
 
Mutation Testing for Industrial Robotic Systems (FMAS 2025)
bySylvain Hallé
 
Agentic AI presentation with Python Exercises
byParth Mane
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
bywesley chun
 
IDSECCONF2025 - Nosa Shandy - Discovering and Disclosing Privacy Vulnerabilit...
byidsecconf
 
IDSECCONF2025 - Ali - DursGo–Web Security Scanner with AI Analysis.pdf
byidsecconf
 
Building powerful web apps to improve productivity and engagement - Esri UK W...
byEsri UK
 
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
Running Non-Cloud-Native Databases in Cloud-Native Environments_ Challenges a...
byAlkin Tezuysal
 
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
SELinux Policy Management in RHEL - RHCSA+.pdf
byLinuxCert Guru
 
Redcentric Data Centres: vision, mission and values
byRedcentric
 
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
In this document
Powered by AI

Overview of Ceph, open-source storage optimizations for cloud workloads presented by Jian Zhang from Intel.

Outline of the key topics to be discussed, including the problem of data growth and the introduction to Ceph.

Highlighting the rapid growth of the digital universe from 4.4 ZB in 2013 to 44 ZB by 2020 and the increasing cost challenges.

Emphasizing the shift from traditional workloads to today's diverse workloads and the necessity for distributed storage solutions.

Ceph as a scalable, software-defined storage system providing object, block, and file storage on commodity hardware.

Description of the architecture of a Ceph cluster, illustrating the distributed storage model and scalability.

Detailed explanation of how the Object Store Daemon (OSD) handles read and write operations within Ceph.

Comparison of storage performance for various workloads while showcasing IOPS and storage capacity.

Evaluation of Ceph block performance under specific configuration settings and analysis of performance results.

Test results indicating performance metrics for random and sequential read/write operations and tuning effects.

Configuration details for measuring Ceph's object performance, including test environments and results.

Explanation of Ceph cache tiering, its benefits, and performance optimization techniques.

Overview of Ceph's erasure coding for enhanced durability, cost efficiency, and performance optimizations.

Intel's efforts in optimizing Ceph on Intel Architecture and available resources for developers.

Recommended best practices for deploying a Ceph cluster to ensure optimal performance and reliability.

Summary of distributed storage needs and how Intel is supporting Ceph; call to action for engaging with the community.

Additional resources for further information about Ceph, Intel's contributions, and legal disclaimers.

Detailed configurations and methodologies for testing Ceph's performance across various parameters.

Ceph: Open Source Storage Software Optimizations on Intel® Architecture for Cloud Workloads

  • 1.
    1 Make the Futurewith China! Ceph: Open Source Storage Software Optimizations on Intel® Architecture for Cloud Workloads Jian Zhang – Software Engineer, Intel Corporation DATS005
  • 2.
    2 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 3.
    3 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 4.
    4 The Problem: DataBig Bang From 2013 to 2020, the digital universe will grow by a factor of 10, from 4.4 ZB to 44 ZB It more than doubles every two years. Data needs are growing at a rate unsustainable with today’s infrastructure and labor costs Source: IDC – The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things - April 2014 1 2 3 4 7 12 19 30 48 77 125 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 1.2 COST CHALLENGES COTINUE TO GROW Storage Cost structure needs a fundamental shift Storage Capacity in TB 62% CAGR IT Budget 2% CAGR IT PROS WILL SHOULDER A GREATER STORAGE BURDEN 230 GB Per IT Pro 1,231 GB Per IT Pro
  • 5.
    5 Diverse Workloads &Cost Drive Need for Distributed Storage Traditional Workloads Today’s Workloads Diverse Workloads Management Scale on Demand Increasing Complexity Cost Traditional Workloads Today’s Trends Storage Fabric Network Fabric Mobile HPC Big Data & Analytics CloudOLTP Email ERP Apps Apps Challenges CRM
  • 6.
    6 Distributed Storage Traditional Scale-upModel SAN (Storage Area Network) Server High Availability (Failover) High perf workloads (e.g., database) Enterprise Mission Critical Hybrid Cloud Limited Scale Costly (Cap-ex and Op-ex) Pay as you Grow, massive on-demand scale Cost, Performance optimized Open and commercial solutions on x86 servers Applicable to cloud workloads Not a good fit for traditional high perf workloads • Ceph is the most popular† open source virtual block storage option. Also provides object, file (experimental). • Strong customer interest - several production implementations already. Distributed Storage Model Application Servers Storage Nodes Storage Client Metadata Servers Converged Network † OpenStack User Survey Insights: http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014
  • 7.
    7 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 8.
    8 Ceph Introduction • Cephis an open-source, massively scalable, software-defined storage system which provides object, block and file system storage in a single platform. It runs on commodity hardware—saving you costs, giving you flexibility—and because it’s in the Linux* kernel, it’s easy to consume. • Object Store (RADOSGW) - A bucket based REST gateway - Compatible with S3 and swift • File System (CEPH FS) - A POSIX-compliant distributed file system - Kernel client and FUSE • Block device service (RBD) - OpenStack* native support - Kernel client and QEMU/KVM driver RADOS A software-based, reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes and lightweight monitors LIBRADOS A library allowing apps to directly access RADOS RGW A web services gateway for object storage Application RBD A reliable, fully distributed block device CephFS A distributed file system with POSIX semantics Host/VM Client
  • 9.
    9 Ceph Cluster Overview ClientServers Storage Servers • Ceph Clients - Block/Object/File system storage - User space or kernel driver • Peer to Peer via Ethernet - Direct access to storage - No centralized metadata = no bottlenecks • Ceph Storage Nodes - Data distributed and replicated across nodes - No single point of failure - Scale capacity and performance with additional nodes Ceph scales to 1000s of nodes Ethernet KVM Application Guest OS Application Guest OS KVM Application Guest OS Application Guest OS KVM Application Guest OS Application Guest OS RADOS RBD Object File RADOS RBD Object File RADOS RBD Object File SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD SSD SSD OSDOSD OSD MON MON
  • 10.
    10 Object Store Daemon(OSD) Read and Write Flow OSD Disk OSD OSD OSD Disk Disk Disk Compute Node RBD RADOS App OSD Disk OSD OSD OSD Disk Disk Disk OSD Disk OSD OSD OSD Disk Disk Disk 1 2 2 3 3 4 Server Server Server Compute Node RBD RADOS App 1 2 WriteRead 1 2 3 4 Client app writes data, RADOS sends data to primary OSD Primary OSD identifies replica OSDs and sends data, writes data to local disk Replica OSDs write data to local disk, signal completion to primary Primary OSD signals completion to client app Client app issues read request, RADOS sends request to primary OSD Primary OSD reads data from local disk and completes read request 1 2
  • 11.
    11 Ceph WorkloadsStoragePerformance (IOPS,Throughput) Storage Capacity (PB) LowerHigher Virtual Machines LowerHigher Virtual Block Media Streaming Enterprise Dropbox Backup, Archive Remote Disks VDI App Storage HPC Batch analytics Mobile Content Depot Databases Block Object Current Future Intel Focus Test & Dev FS
  • 12.
    12 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 13.
    13 Ceph Block Performance– Configuration 1x10Gb NIC Test Environment Compute Node • 2 nodes with Intel® Xeon™ processor x5570 @ 2.93GHz, 128GB mem • 1 node with IntelXeon processor E5 2680 @2.8GHz, 56GB mem Storage Node • IntelXeon processor E3-1275 v2 @ 3.5 GHz • 32GB Memory • 1xSSD for OS • 10x 3 TB 7200rpm • 2x 400GB Intel® SSD DC S3700 CEPH1 MON OSD1 OSD10… CEPH2 OSD1 OSD10… CEPH3 OSD1 OSD10… CEPH4 OSD1 OSD10… KVM VM1 FIO FIO VM40 CLIENT 1 KVM VM1 FIO FIO VM40 CLIENT 2 KVM VM1 FIO FIO VM40 CLIENT 3 2x10Gb NIC Note: See page #37, #38, #39 for system configuration and benchmark data
  • 14.
    14 Ceph Block Performance– Measure Raw Performance 3 TB 7200 rpm 1 Run FIO on one HDD, collect disk IO performance Note: Sequential 64K (Client) = Random 512K (Ceph OSD) 2 Estimate cluster performance (include replication overhead for writes – 2x in this test) 3 TB 7200 rpm3 TB 7200 rpm3 TB 7200 rpm20 HDDs (Writes) 40 HDDs (Reads) 70 MB/s 80 MB/s 270 IOPS 160 IOPS Random 512K Write Read Random 4K Write Read 1400 MB/s 3200 MB/s 5400 IOPS 6400 IOPS Write Read Random 4K Write Read Random 512K Note: See page #37, #38, #39 for system configuration and benchmark data
  • 15.
    15 0 1000 20003000 4000 5000 6000 7000 4K Rand Read - IOPS 4K Rand Write - IOPS 64K Seq Read - MB/s 64K Seq Write - MB/s Maximum Possible Peak Observed Ceph Block Performance – Test Results Drop OSD Cache Prepare Data (dd) Run FIO 1. 60GB Span 2. 4 IOs: Sequential (W,R), Random (W, R) 3. 100s warm-up, 600s test 4. RBD images – 1 to 120 Note: Random tests use Queue Depth=8, Sequential tests use Queue Depth=64 See page #39, #40, #41 for system configuration and benchmark data CEPH Cluster Performance 88% 64% 102% 94% Ceph performance is close to max cluster IO limit for all but random writes – room for further optimizations
  • 16.
    16 Ceph Block Performance– Tuning effects 0 1000 2000 3000 4000 5000 6000 7000 64K Seq Write - MB/s 64K Seq Read - MB/s 4K Rand Write - IOPS 4K Rand Read - IOPS Ceph Block Performance Tuning Impact --compared with default ceph.conf Throughput W/ Tunings Throughput 2.36x 1.88x 3.59x 2.00x Best Tuning Knobs Read ahead = 2048 I/O merge & write cache Omap data on a separate Disk Large pg number: 81920 Note: See page #37, #38, #39 for system configuration and benchmark data
  • 17.
    17 Ceph Object Performance– Configuration 1x10Gb NIC Test Environment Client Node • 2 nodes with Intel® Xeon™ processor x5570 @ 2.93GHz, 24GB mem Storage Node • IntelXeon processor E3-1280 v2 @ 3.6 GHz • 16 GB Memory • 1xSSD for OS • 10x1 TB 7200rpm • 3x 480GB Intel® SSD S530 CEPH1 MON OSD1 OSD10… CEPH2 OSD1 OSD10… CEPH3 OSD1 OSD10… CEPH4 OSD1 OSD10… KVM VM1 FIO FIO VM20 CLIENT 1 KVM VM1 FIO FIO VM20 CLIENT 2 2x10Gb NIC Note: See page #40, #41 for system configuration and benchmark data Rados Gateway (RGW) Rados Gateway • 1 nodes with Intel Xeon processor E5-2670 @ 2.6GHz, 64GB mem
  • 18.
    18 Ceph Object Performance– Test Results Prepare the Data Run COSBench 1. 100 containers x 100 objects each 2. 4 IOs: 128K Read/Write; 10M Read/write 3. 100s warm-up, 600s test 4. COSBench workers – 1 to 2048 Note: COSBench is an Intel development open source cloud object storage benchmark https://github.com/intel-cloud/cosbench Note: See page #42, #43 for system configuration and benchmark data CEPH Cluster Performance Ceph Object performance is close to max cluster IO limit #con x # obj Object- Size RW-Mode Worker- Count Avg- ResTime 95%- ResTime Throughput Bandwidth Bottleneck -- -- -- ms ms op/s MB/s -- 100x100 128KB Read 80 10 20 7,951 971 RGW CPU Write 320 143 340 2,243 274 OSD CPU 10MB Read 160 1,365 3,870 117 1,118 RGW NIC Write 160 3,819 6,530 42 397 OSD NIC
  • 19.
    19 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 20.
    20 Ceph Cache Tiering •An important feature towards Ceph enterprise readiness - Cost-effective - High performance tier W/ more SSDs - Use a pool of fast storage devices (Typically SSDs) and use it as a cache for an existing larger pool  E.g., Reads would first check the cache pool for a copy of the object, and then fall through to the existing pool if there is a miss • Cache tiering mode - Read only - Write back Application Ceph Storage Cluster Cache Pool (Replicated) Backing Pool (Erasure Coded)
  • 21.
    21 Ceph Cache TieringOptimization – Proxy Read/write Proxy the read/write operations while the object is missing in cache tier, promotion in background • 3.8x and 3.6x performance improvement respectively with proxy-write and proxy-read optimization Promotion logic Cache tier Base tier Client ① ② ③ ④ Promotion logic Cache tier Base tier Client ① ② ③ ④ Proxy logic Current design Proxy design 0 2000 4000 6000 8000 0 500 1000 1500 2000 RR RW Latency(ms) IOPS W/ CT IOPS W/ CT optimized IOPS W/ CT Latency W/ CT optimized Latency Cache Tiering optimization 4K random write Proxy-read and write significantly improved cache tiering performance Note: See page #37, #38, #39 for system configuration and benchmark data
  • 22.
    22 Ceph Erasure Coding CephStorage Cluster Replicated Pool COPY COPY COPY Object Ceph Storage Cluster Erasure Coded Pool 1 Object 2 3 4 X Y Full Copies of stored objects • Very high durability • 3x (200% overhead) • Quicker recovery One Copy plus parity • Cost-effective durability • 1.5X (50% store overhead) • Expensive recovery
  • 23.
    23 Ceph EC optimization– I/O hint • ISA-L EC library merged to Firefly • EC performance - Acceptable performance impact  <10% degradation for 10M objects large scale read/write tests - But: Compared with 3x replica, we now tolerate 40% object loss with 1.6x space • Rados I/O hint - Provide a hint to the storage system to classify the operate type brings differentiated storage services - Balance throughput and cost, and boost Storage performance - With rados I/O hint optimization, we can get even higher throughput compared W/O EC! Client Node RADOS Node Hypervisor Guest VM Qemu/Virtio Application RBD RADOS RADOS Protocol OSD FileStore NewStore… SSD Network Hint MemStore KVStore Hint engine 35% performance improvement for EC write with Rados I/O Hint SSD SSD SSD SSD Hint engine Hint engine Hint engine I/O hint policy engine Hint Note: See page #40, #41 for system configuration and benchmark data
  • 24.
    24 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 25.
    25 How is Intelhelping? • Deliver Storage workload optimized products and technologies - Optimize for Silicon, Flash* and Networking technologies • Working with the community to optimize Ceph on IA - Make Ceph run best on Intel® Architecture (IA) based platforms (performance, reliability and management) • Publish IA Reference Architectures - Share best practices with the ecosystem and community
  • 26.
    26 CPU Platforms NVM Networking Server Boards & Systems Software Libraries Software Products Opensource Ecosystem Intel’s ProductPortfolio for Ceph Intel® Cache Acceleration Software (Intel® CAS) • OEMs / ODMs • ISVs Intel® Storage Acceleration Library (Intel® ISA-L) • Virtual Storage Manager (VSM) • Ceph contributions: No.2 • Cephperf† • Reference Architectures Solution focus with Intel® platform and software ingredients. Deep collaboration with Red Hat* and Inktank* (by Red Hat) to deliver enterprise ready Ceph solutions. †Cephperf to be open sourced in Q2’15
  • 27.
    27 VSM– Ceph Simplified Homepage: https://01.org/virtual-storage-manager Code Repository: https://github.com/01org/virtual-storage- manager Issue Tracking: https://01.org/jira/browse/VSM Mailing list: http://vsm-discuss.33411.n7.nabble.com/ VSM (Virtual Storage Manager) - An open source Ceph management tool developed by Intel, and announced on 2014 Nov’s OpenStack* Paris summit, designed to help make day to day management of Ceph easier for storage administrators.
  • 28.
    28 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 29.
    29 Mon Caching Compute Node RADOS Node RADOS Protocol RADOS Protocol OSD JournalFilestore SSD File System Network SSD MON Ceph – Best Deployment Practices Note: Refer to backup for detailed tuning recommendations. • XFS, deadline scheduler • Tune read_ahead_kb for sequential I/O read • Queue Depth – 64 for sequential, 8 for random Jumbo Frames for 10Gbps Use 10x of default queue params • In small cluster, can co- locate with OSDs with 4GB-6GB • Beyond 100 OSDs (100 HDDs or 10 nodes), deploy monitors on separate nodes • 3 monitors for < 200 nodes • One OSD process per disk • Approximately 1GHz Intel® Xeon™-class CPU per OSD • 1GB memory per OSD
  • 30.
    30 Agenda • The Problem •Ceph Introduction • Ceph Performance • Ceph Cache Tiering and Erasure Code • Intel Product Portfolio for Ceph • Ceph Best Practices • Summary
  • 31.
    31 Summary • Cloud workloadsand cost is driving the need for distributed storage solutions • Strong customer interest and lots of production implementations in Ceph • Intel is optimizing CEPH for Intel® Architecture
  • 32.
    32 Next Steps • Takeadvantage of Intel software optimizations and reference architectures for your production deployments - Pilot “cephperf” in Q2’15 and give us feedback http://01.org/cephperf • Engage with open source communities for delivering enterprise features • Innovate storage offerings using value added features with Ceph
  • 33.
    33 Additional Sources ofInformation • A PDF of this presentation is available from our Technical Session Catalog: www.intel.com/idfsessionsSZ. This URL is also printed on the top of Session Agenda Pages in the Pocket Guide. • More web based info: http://ceph.com • Intel® Solutions Reference Architectures www.intel.com/storage • Intel® Storage Acceleration Library (Open Source Version) - https://01.org/intel%C2%AE-storage-acceleration-library-open-source- version
  • 34.
    34 Legal Notices andDisclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. © 2015 Intel Corporation.
  • 35.
    35 Risk Factors The abovestatements and any others in this document that refer to plans and expectations for the first quarter, the year and the future are forward- looking statements that involve a number of risks and uncertainties. Words such as "anticipates," "expects," "intends," "plans," "believes," "seeks," "estimates," "may," "will," "should" and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel's actual results, and variances from Intel's current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations. Demand for Intel’s products is highly variable and could differ from expectations due to factors including changes in the business and economic conditions; consumer confidence or income levels; customer acceptance of Intel’s and competitors’ products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Intel’s gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing quality/yields. Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing expenses, and Intel’s ability to respond quickly to technological developments and to introduce new features into existing products, which may result in restructuring and asset impairment charges. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Results may also be affected by the formal or informal imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice. Intel operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term. The amount, timing and execution of Intel’s stock repurchase program and dividend program could be affected by changes in Intel’s priorities for the use of cash, such as operational spending, capital spending, acquisitions, and as a result of changes to Intel’s cash flows and changes in tax laws. Product defects or errata (deviations from published specifications) may adversely impact our expenses, revenues and reputation. Intel’s results could be affected by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. Intel’s results may be affected by the timing of closing of acquisitions, divestitures and other significant transactions. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the company’s most recent reports on Form 10-Q, Form 10-K and earnings release. Rev. 1/15/15
  • 36.
  • 37.
    37 Block Test Environment- Configuration Details Client Nodes CPU 2 x Intel ®Xeon™ x5570 @ 2.93GHz (4-core, 8 threads) (Qty: 2) 2x Intel Xeon E5 2680 @2.8GHz (16-core, 32 threads) (Qty: 1) Memory 128 GB (8GB * 16 DDR3 1333 MHZ) or 56GB (8GB * 7) for E5 server NIC 10Gb 82599EB Disks 1 HDD for OS Client VM CPU 1 X VCPU VCPUPIN Memory 512 MB Ceph Nodes CPU 1 x Intel Xeon E3-1275 V2 @ 3.5 GHz (4-core, 8 threads) Memory 32 GB (4 x 8GB DDR3 @ 1600 MHz) NIC 2 X 82599 10GbE HBA/C204 {SAS2008 PCI Express* Fusion-MPT SAS-2} / {6 Series/C200 Series Chipset Family SATA AHCI Controller} Disks 1 x SSDSA2SH064G1GC 2.5’’ 64GB for OS 2 x Intel SSDSC2BA40 400 GB SSD (Journal) 10 x Seagate* ST3000NM0033-9ZM 3.5’’ 3TB 7200rpm SATA HDD (Data) Ceph cluster OS CentOS 6.5 Kernel 2.6.32-431 Ceph 0.61.2 built from source Client host OS Ubuntu* 12.10 Kernel 3.6.3 Client VM OS Ubuntu 12.10 Kernel 3.5.0-17 • XFS as file system for Data Disk • Each Data Disk (SATA HDD) was parted into 1 partition for OSD daemon • Default replication setting (2 replicas), 7872 pgs. • Tunings - Set read_ahead_kb=2048 - MTU= 8000 • Change i/o scheduler to [deadline]: # Echo deadline >/sys/block/[dev]/queue/scheduler
  • 38.
    38 Ceph RBD TunedConfiguration [global] debug default = 0 log file = /var/log/ceph/$name.log max open files = 131072 auth cluster required = none auth service required = none auth client required = none [osd] osd mkfs type = xfs osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog osd mkfs options xfs = -f -i size=2048 filestore max inline xattr size = 254 filestore max inline xattrs = 6 osd_op_threads=20 filestore_queue_max_ops=500 filestore_queue_committing_max_ops=5000 journal_max_write_entries=1000 journal_queue_max_ops=3000 objecter_inflight_ops=10240 filestore_queue_max_bytes=1048576000 filestore_queue_committing_max_bytes =1048576000 journal_max_write_bytes=1048576000 journal_queue_max_bytes=1048576000 ms_dispatch_throttle_bytes=1048576000 objecter_infilght_op_bytes=1048576000 filestore_max_sync_interval=10 filestore_flusher=false filestore_flush_min=0 filestore_sync_flush=true
  • 39.
    39 Testing Methodology Storage interface UseQemuRBD as storage interface Tool • Use “dd” to prepare data for R/W tests • Use fio (ioengine=libaio, direct=1) to generate 4 IO patterns: sequential write/read, random write/read • Access Span: 60GB • For capping tests, Seq Read/Write (60MB/s), and Rand Read/Write (100 ops/s) • QoS Compliance: - For random 4k read/write cases: latency <= 20ms - For sequential 64K read/write cases: BW >= 54 MB/s Run rules • Drop osds page caches ( “1” > /proc/sys/vm/drop_caches) • 100 secs for warm up, 600 secs for data collection • Run 4KB/64KB tests under different # of rbds (1 to 120) Space allocation (per node) • Data Drive: - Sits on 10x 3TB HDD drives - So 4800GB/40 * 2 = 240GB data space will be used on each Data disk at 80 VMs. • Journal: - Sits on 2x 400GB SSD drives - One journal partition per data drive, 10GB
  • 40.
    40 Object Test Environment- Configuration Details Client & RGW CPU Client: 2 x Intel ®Xeon™ x5570 @ 2.93GHz (4-core, 8 threads) (Qty: 2) GRW: 2x Intel Xeon E5 2670@2.6GHz (16-core, 32 threads) (Qty: 1) Memory 128 GB (8GB * 16 DDR3 1333 MHZ) or 56GB (8GB * 7) for E5 server NIC 10Gb 82599EB Disks 1 HDD for OS Ceph OSD Nodes CPU 1 x Intel Xeon E3-1280 V2 @ 3.6 GHz (4-core, 8 threads) Memory 32 GB (4 x 8GB DDR3 @ 1600 MHz) NIC 2 X 82599 10GbE HBA/C204 {SAS2308 PCI Express* Fusion-MPT SAS-2} / {6 Series/C200 Series Chipset Family SATA AHCI Controller} Disks 1 x SSDSA2SH064G1GC 2.5’’ 64GB for OS 3 x Intel SSDSC2CW480A3 480 GB SSD (Journal) 10 x Seagate* ST1000NM0011 3.5’’ 1TB 7200rpm SATA HDD (Data) Ceph cluster OS Ubuntu 14.04 Kernel 3.13.0 Ceph 0.61.8 built from source Client host OS Ubuntu* 12.10 Kernel 3.6.3 • XFS as file system for Data Disk • Each Data Disk (SATA HDD) was parted into 1 partition for OSD daemon • Default replication setting (3 replicas), 12416 pgs. • Tunings - Set read_ahead_kb=2048 - MTU= 8000
  • 41.
    41 Ceph Rados Gateway(RGW) Tuning ceph-gateway –rgw enable ops log = false –rgw enable usage log = false –rgw thread pool size = 256 –Log disabled apache2-http-server –<IfModule mpm_worker_module> – ServerLimit 50 – StartServers 5 – MinSpareThreads 25 – MaxSpareThreads 75 – ThreadLimit 100 – ThreadsPerChild 100 – MaxClients 5000 – MaxRequestsPerChild 0 –</IfModule> GW instances –5 instances in one GW server –Access log turned off