Download to read offline
More Related Content
Similar to Bigdata (1) converted
Bigdata (1) converted
- 1. Introduction to BigData Submitted by R.Thilakavathi II - M.sc(cs)
- 2. What’s Big Data? Nosingle definition; here is from Wikipedia: • Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. • The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. • The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real- time roadway traffic conditions.” 2
- 3.
- 4. 12+ TBs of tweetdata every day 25+ TBs of log data every day ?TBsof dataeveryday 2+ billion people on the Web by end 2011 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide 100s of millions of GPS enabled devices sold annually 76 million smart meters in 2009… 200M by 2014
- 5. The Earthscope • TheEarthscope is the world's largest science project. Designed to track North America's geological evolution, this observatory records data over 3.8 million square miles, amassing 67 terabytes of data. It analyzes seismic slips in the San Andreas fault, sure, but also the plume of magma underneath Yellowstone and much, much more. (http://www.msnbc.msn.com/id/4436 3598/ns/technology_and_science- future_of_technology/#.TmetOdQ--uI)
- 6. Variety (Complexity) • RelationalData (Tables/Transaction/Legacy Data) • Text Data (Web) • Semi-structured Data (XML) • Graph Data – Social Network, Semantic Web (RDF), … • Streaming Data – You can only scan the data once • A single application can be generating/collecting many types of data • Big Public Data (online, weather, finance, etc) 6 To extract knowledge➔ all these types of data need to linked together
- 7. A Single Viewto the Customer Customer Social Media Gaming Entertain Banking Finance Our Known History Purchas e
- 8. Velocity (Speed) • Datais begin generated fast and need to be processed fast • Online Data Analytics • Late decisions ➔ missing opportunities • Examples – E-Promotions: Based on your current location, your purchase history, what you like ➔ send promotions right now for store next to you – Healthcare monitoring: sensors monitoring your activities and body ➔ any abnormal measurements require immediate reaction 8
- 9. Real-time/Fast Data Social mediaand networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Mobile devices (tracking all objects all the time) Sensor technology and networks (measuring all kinds of data) • The progress and innovation is no longer hindered by the ability to collect data • But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion 9
- 10. Real-Time Analytics/Decision Requirement Customer Influence Behavior Product Recommendations thatare Relevant & Compelling Friend Invitations to join a Game or Activity that expands business Preventing Fraud as it is Occurring & preventing more proactively Learning why Customers Switch to competitors and their offers; in time to Counter Improving the Marketing Effectiveness of a Promotion while it is still in Play
- 11. Some Make it4V’s 11
- 12. Harnessing Big Data •OLTP: Online Transaction Processing (DBMSs) • OLAP: Online Analytical Processing (Data Warehousing) • RTAP: Real-Time Analytics Processing (Big Data Architecture & technology) 12
- 13. The Model HasChanged… • The Model of Generating/Consuming Data has Changed Old Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data 13
- 14. What’s driving BigData - Ad-hoc querying and reporting - Data mining techniques - Structured data, typical sources - Small to mid-size datasets - Optimizations and predictive analytics - Complex statistical analysis - All types of data, and many sources - Very large datasets - More of a real-time 14
- 15. Big Data: Batch Processing& Distributed Data Store Hadoop/Spark; HBase/Cassandra BI Reporting OLAP & Dataware house Business Objects, SAS, Informatica, Cognos other SQL Reporting Tools Interactive Business Intelligence & In-memory RDBMS QliqView, Tableau, HANA Big Data: Real Time & Single View Graph Databases THE EVOLUTION OF BUSINESS INTELLIGENCE 1990’s 2000’s 2010’s Speed Scale Scale Speed
- 16. Big Data Analytics •Big data is more real-time in nature than traditional DW applications • Traditional DW architectures (e.g. Exadata, Teradata) are not well- suited for big data apps • Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps 16
- 17.
- 18. Benefits • Cost &management – Economies of scale, “out-sourced” resource management • Reduced Time to deployment – Ease of assembly, works “out of the box” • Scaling – On demand provisioning, co-locate data and compute • Reliability – Massive, redundant, shared resources • Sustainability – Hardware not owned
- 19. Infrastructure as aService (IaaS)
- 20. More Refined Categorization •Storage-as-a-service • Database-as-a-service • Information-as-a-service • Process-as-a-service • Application-as-a-service • Platform-as-a-service • Integration-as-a-service • Security-as-a-service • Management/ Governance-as-a-service • Testing-as-a-service • Infrastructure-as-a-service InfoWorld Cloud Computing Deep Dive
- 21. Enabling Technology: Virtualization Hardware OperatingSystem App App App Traditional Stack Hardware OS App App App Hypervisor OS OS Virtualized Stack
- 22. What does Azureplatform offer to developers?
- 23. June 3, 2008Slide 23 Google AppEngine vs. Amazon EC2/S3 Google’s AppEngine vs Amazon’s EC2 AppEngine: • Higher-level functionality (e.g., automatic scaling) • More restrictive (e.g., respond to URL only) • Proprietary lock-in EC2/S3: • Lower-level functionality • More flexible • Coarser billing model VMs Flat File Storage Python BigTable Other API’s
- 24. Cloud Resources • Hadoopon your local machine • Hadoop in a virtual machine on your local machine (Pseudo-Distributed on Ubuntu) • Hadoop in the clouds with Amazon EC2
- 25. Course Prerequisite • Prerequisite: –Java Programming / C++ – Data Structures and Algorithm – Computer Architecture – Basic Statistics and Probability – Database and Data Mining (preferred) 25
- 26.