Download as PDF, PPTX
Uploaded byThoughtworks
Next Generation Data Platforms - Deon Thomas
The document discusses next-generation data platforms that enhance the traditional understanding of big data through technologies capable of efficiently managing vast volumes of varied data at high speeds. It covers key concepts such as velocity, variety, volume, and value, as well as the evolution of databases and architectures like Hadoop and the Lambda architecture. The piece emphasizes the importance of adapting to new data challenges and opportunities to maintain competitive advantage.
More Related Content
Similar to Next Generation Data Platforms - Deon Thomas
Next Generation Data Platforms - Deon Thomas
- 1. NEXT GENERATION DATA PLATFORMS! Beyondthe traditional view
- 3. What is BigData ? A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis.
- 4. • Terabytes • Records • Tables, • Files • Structured • Unstructured • Semi-Structured • All the above • Batch • Realtime • Streams • Near Realtime Velocity Variety Volume
- 5. Volume! + Velocity = Value +Variety
- 6.
- 7.
- 8.
- 9. NoSQL ● no ACIDtransactions ● sharded indexes ● restricted Joins ● support columnar storage! In memory DB ● real time transactions ● not fully geared for enterprise level data ● variety of indexes ● complex joins HDFS GDF HBASE Database evolution
- 10. Hadoop - Hasbecome synonymous with Big Data.!
- 11.
- 12.
- 13.
- 14. Real Time vsBatch(MapReduce)!
- 15. 15 ACHIEVING VELOCITY (Parallelcomputing) Shared Memory Processor 1 Shared Data Lock (Shared Data) Worker (Shared Data) Unlock (Shared Data)
- 16.  ACHIEVING VELOCITY (Parallelcomputing) Shared Data Chunk 1 Chunk 2 Chunk 3 Message Passing (E.g. MapReduce) Processor 1 Processor 2 Processor 3
- 17.
- 19.
- 20.
- 21. Pull-based Batch Loads Enterprise Data Models ComplexETL Logic Poorly Suited to Non-Relational Data Emergent design is difficult Conventional Architectures!
- 22. OLAP (Online Analyticalprocessing) SELECT SUM(s.dollar_cost), s.product_key, p.description FROM SALES_FACT s … … … GROUP BY s.product_key, p.description
- 23. Why Databases! ● Transactionprocessing (ACID properties) ● SQL - Indexes and queries OLAP ● Transaction processing not needed for analytics o Moving of data via ETL ● Large volumes of data, indexes become irrelevant ● Schema or Write vs Schema Read!
- 24. Analytics is notjust SQL queries
- 25. There is moreto analytics than you think!
- 26. Do we losehope, how do we move forward!
- 27. 1. Variety -How to we deal with different kinds of data ? 2. Volume - How to we cope with large volume of data? 3. Velocity - How do we solve realtime problems? 4. Value - What is our value ?! Summary!
- 28.
- 29.
- 30. Real time processing+ Batch since 1983!
- 31. Should i adoptor should i not!
- 35. “Change before youhave to” Jack Welch
- 36.