Downloaded 13 times
More Related Content
Similar to Best Practices for Building Open Source Data Layers
Best Practices for Building Open Source Data Layers
- 1. © 2016 IBMCorporation Christopher Bienko Worldwide Technical Sales – Cloud Data Services cdbienko@us.ibm.com August 15, 2016 Best Practices for Building Open Source Data Layers
- 2. Compose is for… Builders IBM Compose Platform
- 3. The Compose OpenSource Stack § A managed platform for open source databases-as-a-service －Provision services individually via Multi-tenant: deploy in minutes, scale enormously, develop effortlessly －Reserved infrastructure for production via Enterprise: guaranteed SLA, à la carte licensing to the entire Compose catalogue IBM Compose Platform
- 4. q Open SourceEnables Open Architectures –Avoid vendor lock-in –Community-driven projects leading the industry –Cut licensing fees & operationalize hardware costs q The Database Dilemma: Scoping is Hard q The Infrastructure Quandary: Scaling is Harder q Repeatable Deployments for Standardized Workflow Data Layer Requirements
- 5. The Compose OpenSource Stack IBM Compose Platform Key-Value Database for Distributed DBs etcd NoSQL (BSON) Document DB MongoDB Data Caching Key-Value DB Redis Scalable JSON Database for Real-Time Apps RethinkDB Asynchronous Messaging Layer RabbitMQ Extensible and Secure Object Relational DB PostgreSQL Full-Text Search Indexing Engine Elasticsearch
- 6. The Compose OpenSource Stack IBM SoftLayer Amazon AWS Compose services are deployable to both SoftLayer and AWS Available on SoftLayer as: § IBM-Managed service § Public Multi-Tenant Available on AWS as: § IBM-Managed service § Self-Hosted service § Public Multi-Tenant IBM Compose Platform MongoDB Redis Elasticsearch PostgreSQL RethinkDB RabbitMQ etcd * Self-Hosted services coming soon
- 7. The Database Dilemma– What to Choose? § Developers that discover the Compose platform are often already using 2–4 of these databases & services in their stack －On-premises or private/public cloud § “Have I deployed and configured my services the right way?” IBM Compose Platform MongoDB Redis Elasticsearch PostgreSQL RethinkDB RabbitMQ etcd If you are using one of these, very likely you are building or experimenting with others. New technologies add to larger architectures, but few will start a project with these alone.
- 8. q Open SourceEnables Open Architectures q The Database Dilemma: Scoping is Hard –Selecting the appropriate database on the first attempt is rare –Use cases evolve over time…as do database requirements –Platform services need to be as flexible as your workloads q The Infrastructure Quandary: Scaling is Harder q Repeatable Deployments for Standardized Workflow Data Layer Requirements
- 9. Containers for BestPractice, Repeatable Deployments IBM Compose Platform MongoDB Redis Elasticsearch RethinkDB RabbitMQ etcd Customer Private Infrastructure Compose Shared Platform Infrastructure Customer Compose Dashboard çè SSH Data Host A Data Host B Utility Host C PostgreSQL Data Capsule PostgreSQL Data Capsule PostgreSQL Haproxy Capsule PostgreSQL Arbiter Capsule VLAN çè çè SSH SSH RabbitMQ Message Broker Compose GRU Recipes Compose Dashboard www.compose.io Bluemix Console www.bluemix.net 2 1 çè HTTPs çè HTTPs 3 4 PostgreSQL
- 10. Compose Platform –3 Consumption Models § Compose Enterprise －For those needing à la carte access to the complete Compose catalogue －Dynamically mix & match, scale & deploy new combinations of Compose －Self-Hosted for those already managing their own virtual private cloud －IBM-Managed takes care of both infrastructure and Compose licensing IBM Compose Platform Self-Hosted Compose Enterprise Multi-Tenant Compose Public IBM-Managed Compose Enterprise Reserved, SLA-governed Enterprise infrastructure for unlimited licensing of the full Compose catalogue. Individual services for PAY-GO consumption.
- 11. q Open SourceEnables Open Architectures q The Database Dilemma: Scoping is Hard q The Infrastructure Quandary: Scaling is Harder –Compose services scale elastically and without downtime –Onboard new databases as your platform architecture matures –Reserve enterprise-grade infrastructure that’s managed for you q Repeatable Deployments for Standardized Workflow Data Layer Requirements
- 12. Simplify Infrastructure Across3 Configurations & 1 Bill IBM Compose Platform Self-Hosted Compose Enterprise Multi-Tenant Compose Public IBM-Managed Compose Enterprise Starter 16 GB RAM Transactional 64 GB RAM Large Transactional 256 GB RAM IBM-Managed Compose Enterprise supports three (3) infrastructure configurations. AWS Only SL & AWS SL & AWS
- 13. Evolve Composition ofEnterprise Services on the Fly IBM Compose Platform Self-Hosted Compose Enterprise Multi-Tenant Compose Public IBM-Managed Compose Enterprise Starter 16 GB RAM Transactional 64 GB RAM Large Transactional 256 GB RAM AWS Only SL & AWS SL & AWS MongoDB 640GB SSD, 64GB RAM Elasticsearch 320GB SSD, 32GB RAM Redis 16GB RAM Elasticsearch 160GB SSD, 16GB RAM MongoDB 320GB SSD, 32GB RAM MongoDB 320GB SSD, 32GB RAM OR OR IBM-Managed Compose Enterprise supports three (3) infrastructure configurations.
- 14. q Open SourceEnables Open Architectures q The Database Dilemma: Scoping is Hard q The Infrastructure Quandary: Scaling is Harder q Repeatable Deployments for Standardized Workflow –Services consistently deployed to best-practice configuration –SLA, 3-node HA, automated backups, at-rest encryption –Fully-managed infrastructure & elastically scalable databases Data Layer Requirements
- 15. Building a VRData Layer with Compose and IBM
- 16. Building a VRData Layer § Selecting the appropriate databases and services in support of a Virtual Reality (VR) headset data layer is complex § Multiple requirements with no one-size-fits-all solution: 1. Games played with the headset need a responsive, flexible database 2. In-store and in-game purchases rely upon dependable record keeping 3. Insight into customers and players requires reporting and analytics 4. VR headsets generate a wealth of sensor data, demanding streaming capabilities to ingest the data and big data tools to transform it Games Micro-transactions Analytics Sensors
- 17. Building a VRData Layer MongoDB PostgreSQL MySQL (Beta) Cloudant Redis Streams Elasticsearch dashDB for Transactions dashDB for Analytics Analytics, reporting, and data visualization In memory caching and near/real-time streaming analytics Transactional data, strongly consistent, systems of record Operational data, eventually consistent, mobile applications Apache Spark DB2 on Cloud ScyllaDB (Beta) Compose Services IBM CDS Services Games Micro-transactions Analytics Sensors
- 18. Building a VRData Layer MongoDB PostgreSQL MySQL (Beta) Cloudant Redis Streams Elasticsearch dashDB for Transactions dashDB for Analytics Apache Spark DB2 on Cloud ScyllaDB (Beta) Compose Services IBM CDS Services Games Micro-transactions Analytics Sensors Selecting a service from each category– across both the Compose and IBM CDS catalogues –yields a fully-fledged data layer for your VR platform! PostgreSQL or DB2? MongoDB or Cloudant? Not sure? Let us help.
- 19. Building a VRData Layer PostgreSQLCloudant dashDB for Analytics Compose Services IBM CDS Services Web & Mobile Applications Flexible JSON docs support operational data from mobile apps Relational, strongly consistent tables support transactional data from mobile apps Apache Spark RabbitMQ Redis
- 20. Building a VRData Layer PostgreSQLCloudant dashDB for Analytics Compose Services IBM CDS Services dashDB Schema Discovery Schema Discovery applies structure to unstructured JSON data for reporting & analytics Web & Mobile Applications Flexible JSON docs support operational data from mobile apps Relational, strongly consistent tables support transactional data from mobile apps Apache Spark RabbitMQ Redis
- 21. RabbitMQ Building a VRData Layer PostgreSQLCloudant dashDB for Analytics Compose Services IBM CDS Services dashDB Schema Discovery Schema Discovery applies structure to unstructured JSON data for reporting & analytics Web & Mobile Applications Slack DevOps & Support RabbitMQ notifies DevOps team’s Slack #channel when a mobile app’s micro-transactions hit a new milestone AMQP Flexible JSON docs support operational data from mobile apps Relational, strongly consistent tables support transactional data from mobile apps RedisRabbitMQ Apache Spark
- 22. RabbitMQ Redis Building a VRData Layer PostgreSQLCloudant dashDB for Analytics Compose Services IBM CDS Services dashDB Schema Discovery Schema Discovery applies structure to unstructured JSON data for reporting & analytics Web & Mobile Applications VR Headset IOT Sensor Data Spark will transform & filter IOT data before landing in Cloudant Spark-Cloudant Adapter Slack DevOps & Support RabbitMQ notifies DevOps team’s Slack #channel when a mobile app’s micro-transactions hit a new milestone AMQP Redis provides elastic, high-velocity caching for IOT data before ingestion into Spark Flexible JSON docs support operational data from mobile apps Relational, strongly consistent tables support transactional data from mobile apps RabbitMQ Apache Spark Redis
- 23. © 2016 IBMCorporation Christopher Bienko Worldwide Technical Sales – Cloud Data Services cdbienko@us.ibm.com August 15, 2016 Want to find out more? Watch the webinar: http://ibm.biz/BdrNVR