Download as PDF, PPTX
More Related Content
Similar to Demystifying Data Engineering
Demystifying Data Engineering
- 1.
- 2. Data engineering • Softwareengineering with an emphasis on dealing with large amounts of data • A “specialty” of software engineering
- 3. Why now? • Alwaysvalue in scale, but it was previously too difficult / expensive • Economics and technology advances make these scales accessible
- 4. Enable others toanswer questions on dataset within latency constraints
- 5. Data engineering • Distributedsystems – consensus, consistency, availability, etc. • Parallel processing • Databases • Queuing
- 6. Data engineering • Human-faulttolerance • Metrics and monitoring • Multi-tenancy
- 7. BackType • When Ijoined: • Comment search by keyword • Comment search by user • Basic stats on commenters • Link search on Twitter
- 8.
- 9. BackType • Inflexible • Proneto corruption • Heavy operational burden • Not scalable • Not fault-tolerant
- 10. BackType • Enable askingany question (with high latency) • Allows exploration and experimentation • Establishes human-fault tolerance
- 11.
- 13. ElephantDB • Export resultsof MapReduce pipelines for querying • Low latency querying but out of date by many hours • Incredibly simple
- 15. • Infrastructure • Datapipelines • Abstractions Data engineering
- 16. Data pipeline example Tweets (S3) Normalize URLs Compute hourbucket Sum by hour/url Emit ElephantDB indexes
- 17. Data pipeline example Tweets (Kafka) Normalize URLs Compute hourbucket Update hour/ url bucket Cassandra
- 18.
- 20. Infrastructure • HDFS • MapReduce •Kafka • Storm • Spark • Cassandra • HBase • ElephantDB • Zookeeper
- 21. Streaming compute team atTwitter • Started streaming compute team at Twitter • One shared Storm cluster for entire company
- 22. Multi-tenancy • Independent applicationson same cluster • Topologies should not affect one another
- 23. Resource allocation • Topologiesshould be given an appropriate amount of resources
- 24. Initial approach • UseMesos to provide resource guarantees • Users include resources needed as part of topology submission
- 26. Solution • Implement newscheduler which gives production topologies dedicated hardware • Only Storm team can configure production topologies • Left-over machines are used as failover or for in-development topologies
- 28. Data Engineering vsData Science • Well-defined problems • No special statistics skills required • Larger scope • Not just analytics
- 29. Open source • Almostall major Big Data tools are open source (e.g. Hadoop, Storm, Spark, Kafka, Cassandra, HBase, etc.) • Many have commercial support
- 30. Open source • Veryimportant for recruiting data engineers • Strong developers want to work at places where they can be involved with open source
- 31. Open source • Developa technology brand for company (in conjunction with a tech blog) • Creating a popular open source project can give you access to lots of strong engineers
- 32. Open source • Identifystrong engineers in the community you may want to recruit • Learn best practices and get help from the people who know the tools the best • *Do not* expect to get “free work” on your projects
- 33. Ideal data engineer •Strong software engineering skills • Abstraction • Testing • Version control • Refactoring
- 34. Ideal data engineer •Strong software engineering skills • Strong algorithm skills
- 35. Ideal data engineer •Strong software engineering skills • Strong algorithm skills • Good at digging into open source code
- 36. Ideal data engineer •Strong software engineering skills • Strong algorithm skills • Good at digging into open source code • Good at stress testing
- 38. Finding strong dataengineers • Standard “coding on the whiteboard” interviews are near useless • Use take home projects to gauge general programming ability • The best is to see projects that require data engineering
- 39.