Download as PDF, PPTX
Uploaded bymark madsen
Building the Enterprise Data Lake: A look at architecture
The document discusses considerations for building an enterprise data lake. It notes that traditional data warehousing approaches do not scale well for new data sources like sensors and streaming data. It advocates adopting a data lake approach with separate systems for data acquisition, management, and access instead of a monolithic architecture. A data lake requires a distributed architecture and platform services to support various data flows, formats, and processing needs. The data architecture should not enforce models or limitations upfront but rather allow for evolution and change over time.
More Related Content
Similar to Building the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architecture
- 1. Building the Enterprise DataLake Considerations before you jump in December, 2015 Mark Madsen www.ThirdNature.net @markmadsen1
- 2. What This SessionIsn’t SQL.. . SQL! SQL? SQL
- 3. The craft modelof information delivery does not scale
- 4. © Third Nature,Inc. So we shifted to data publishing Industrialized data delivery for self-service access.
- 5. Events and sensorsare a relatively new data source Sensor data doesn’t fit well with current methods of modeling, collection and storage, or with the technology to process and analyze it.
- 6. There’s lots ofother new data involved
- 7. © Third Nature,Inc. You can store this data in an RDBMS, but…
- 8. These sorts ofthings slow user requests down Conclusion: any methodology built on the premise that you must know and model all the data first is untenable
- 9. © Third Nature,Inc. Analytics embiggens data volume problems Many of the processing problems are O(n2) or worse, so moderate data can be a problem for scale-up platforms
- 10. © Third Nature,Inc. Old market says: There’s nothing wrong with what you have, just keep buying new products from us
- 11. The emerging bigdata market has an answer…
- 12. © Third Nature,Inc. The data lake
- 13. © Third Nature,Inc. Views of the lake Is the business vs supports the business? Application vs infrastructure?
- 14. © Third Nature,Inc. The naïve idea of a data lake leads to predictable results
- 15. © Third Nature,Inc. You can’t install Hadoop and hope it solves all the problems Big data no 2
- 16. Slide 16 The answerisn’t just technology, it’s architecture
- 17. Schema In the DWworld both data and processing are bounded No consideration for feedback loops and change Processing only happens here Carefully controlled access here Nobodyherecreates newinformation Sources few and well understood Complex DI is controlled by IT Schemas are few and designed Tools are authorized, few in number and kind One way flow This is a monolithic, layered architecture
- 18. © Third Nature,Inc. In the big data world flow is unbounded and continuous Feedback loops allowed End-of-analysis dataset may be start of a BI dataset Continuous data integration and delivery Files are back as both input and storage Minimal barrier of / control on collection Areas of provisioned data Any shape in, rectangles out This needs a distributed service architecture
- 19. © Third Nature,Inc. Deconstructing data environments There are three things happening in a data warehouse: ▪ Data acquisition ▪ Data management ▪ Data delivery Isolate them from one another, allow read- write use, and you are on the path. Data Warehouse
- 20. Data lake subsystems/ components The acquisition component allows any data to be collected at any latency. The management component allows some data to be standardized and integrated. The access component provides access at any latency and via any means an application chooses. Processing can be done to any data at any time from any area. Data Acquisition Collect & Store Incremental Batch One-time copy Real time Data Lake Platform Services Data Management Process & Integrate Data Access Deliver & Use Data storage In reality, you are building three systems, not one. Avoid the monolith.
- 21. © Third Nature,Inc. Data lake functions depend on platform services Base Platform Services Data Movement MetadataData Persistence Workflow Management Processing Engines Dataflow Services Data Curation Data Access Services Data Acquisition Collect & Store Data Management Process & Integrate Data Access Deliver & Use Platformservicesneeded
- 22. DATA ARCHITECTURE We’re sofocused on the light switch that we’re not talking about the light
- 23. © Third Nature,Inc. Decouple the Data Architecture The core of the data lake isn’t a database or HDFS, it’s the data architecture that the tools implement. We need a data architecture that is not limiting: ▪ Deals with change easily and at scale ▪ Does not enforce requirements and models up front ▪ Does not limit the format or structure of data ▪ Assumes the range of data latencies in and out, from streaming to one-time bulk
- 24. © Third Nature,Inc. Food supply chain: an analogy for data Multiple contexts of use, differing quality levels You need to keep the original because just like baking, you can’t unmake dough once it’s mixed.
- 25. © Third Nature,Inc. Data architecture is required by the services, and vice versa Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data DataAcquisition Collect&Store Platform Services DataAccess Deliver&Use Data Management Process & Integrate
- 26. © Third Nature,Inc. The data areas map (mostly) to functional areas of the lake Collection can’t be limited by database scale and latency. Immutability, persistence and concurrency are required. Incremental Collect Batch One-time copy Real time Manage & Integrate Process, Deliver, Use
- 27. © Third Nature,Inc. Stages, not layers Some tools require specific repositories or models. Others can reach in to get what they need. Do not enforce a single access point or model.
- 28. © Third Nature,Inc. The geography has been redefined The box IT created: • not any data, rigidly typed data • not any form, tabular rows and columns of typed data • not any latency, persist what the DB can keep up with • not any process, only queries The digital world was diminished to only what’s inside the box until we forgot the box was there.
- 29. © Third Nature,Inc. Layered data architecture The DW assumed a single flat model of data, DB in the center. The data lake enables new ways to organize data: ▪ Raw – straight from the source ▪ Enhanced –cleaned, standardized ▪ Integrated – modeled, augmented, ~semi-persistent ▪ Derived – analytic output, pattern based sets, ephemeral Implies a new technology architecture and data modeling approaches.
- 30. © Third Nature,Inc. The data lake enables evolutionary design for data Evolutionary design is required because data needs change. You need a system not for stability – we have that in the DW - but for evolution and change, the data lake. Data Acquisition Collect & Store Incremental Batch One-time copy Real time Data Lake Platform Services Data Management Process & Integrate Data Access Deliver & Use Data storage You can’t build this all at once. You need to grow it over time.
- 31. © Third Nature,Inc. Away from “one throat to choke”, back to best of breed Tight coupling leads to efficient reuse and standardization, and to slow changes. In a rapidly evolving market componentized architectures, modularity and loose coupling are favorable over monolithic stacks, single-vendor architectures and tight coupling. Architecture, not blueprints: there is no single answer. It depends on your goals and starting position.
- 32. Questions?“When a newtechnology rolls over you, you're either part of the steamroller or part of the road.” – Stewart Brand
- 33. © Third Nature,Inc. CC Image Attributions Thanks to the people who supplied the creative commons licensed images used in this presentation: donuts_4_views.jpg - http://www.flickr.com/photos/le_hibou/76718773/ glass_buildings.jpg - http://www.flickr.com/photos/erikvanhannen/547701721
- 34. © Third Nature,Inc. About the Presenter Mark Madsen is president of Third Nature, a consulting and advisory firm focused on analytics, business intelligence and data management. Mark is an award-winning author, architect and CTO. Over the past ten years Mark received awards for his work from the American Productivity & Quality Center, TDWI, and the Smithsonian Institute. He is an international speaker, a contributor to Forbes, member of the O’Reilly Strata program committee. For more information or to contact Mark, follow @markmadsen on Twitter or visit http://ThirdNature.net
- 35. About Third Nature ThirdNature is a consulting and advisory firm focused on new and emerging technology and practices in information strategy, analytics, business intelligence and data management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place. Our goal is to help organizations solve problems using data. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in strategy and architecture, so we look at emerging technologies and markets, evaluating how technologies are applied to solve problems rather than evaluating product features.