AWS SSA Webinar 21 - Getting Started with Data lakes on AWS

Editor's Notes

• #4 That’s why many customers are moving to a data lake architecture. A data lake is an architectural approach that helps you manage multiple data types from a wide variety of sources, both structured and unstructured, through a unified set of tools, so it's readily available to be categorized, processed, analyzed and consumed by diverse groups within an organization. Since data can be stored as-is, there is no need to convert it to a predefined schema and you no longer need to know what questions you want to ask of your data beforehand. batch, interactive, online, search, in-memory and other processing engines. It helps you manage multiple data types from a wide variety of sources, both structured and unstructured, through a unified set of tools
• #5 Data lakes allow you to break down data silos and bring data into a single central repository. You can store a wide variety of data formats, at any scale and at low cost. Data lakes provide you a single source of truth and allow you access to the same data using a variety of analytics and machine-learning tools.
• #9 Turns out there are a lot of steps involved in building data lakes 1/ Set up storage – Data lakes hold a massive amount of data. Before doing anything else, customers need to set up storage to hold all of that data. If they are using AWS they configure S3 buckets and partitions. If they are doing this on-prem they acquire hardware and set up large disk arrays to hold all of the data for their data lake. 2/ Move data -- Customers need to connect to different data sources on-premises, in the cloud, and on IoT devices. Then they need to collect and organize the relevant data sets from those sources, crawl the data to extract the schemas, and add metadata tags to the catalog. Customers do this today with a collection of file transfer and ETL tools, like AWS Glue. 3/ Clean and prepare data -- Next, that data must be carefully partitioned, indexed, and transformed to columnar formats to optimize for performance and cost. Customers need to clean, de-duplicate, and match related records. Today this is done using rigid and complex SQL statements that only work so well and are difficult to maintain. This process of collecting, cleaning, and transforming the incoming data is complex and must be manually monitored in order to avoid errors. 4/ Configure and enforce policies – Sensitive data must be secured according to compliance requirements. This means creating and applying data access, protection, and compliance policies to make sure you are meeting required standards. For example, restricting access to personally identifiable information (PII) at the table, column, or row level, encrypting all data, and keeping audit logs of who is accessing the data. Today customers use access control lists on S3 buckets or they use 3rd party encryption and access control software to secure the data. And for every analytics service that needs to access the data, customers need to create and maintain data access, protection and compliance policies for each one. For example, if you are running analysis against your data lake, using Redshift and Athena, you need to set up access control rules for each of these services. 5/ Make it easy to find data - Different people in your organizations, like analysts and data scientists, may have trouble finding and trusting data sets in the data lake. You need to make it easy for those end-users to easily find relevant and trusted data. To do this you must clearly label the data in a catalog of the data lake and provide users with the ability to access and analyze this data without making requests to IT. Each of these steps involve a lot of work because today a lot of it is done manually. Customers can spend months building data access and transformation workflows, mapping security and policy settings, and configuring tools and services for data movement, storage, cataloging, security, analytics, and machine learning. With all these steps, a fully productive data lake can take months to implement. TRANSITION: We’ve learned from the tens of thousands of customers running analytics on AWS that most customers that want to do analytics want to build a data lake, and many of them want this to be easier and faster than it is today.
• #10 A recent study by CrowdFlower who surveyed ~80 data scientists about their jobs. They found Data scientists spend 60% of their time on cleaning and organizing data. Collecting data sets comes second at 19% of their time, meaning data scientists spend around ~80% of their time on preparing and managing data for analysis. They also found data preparation was the least enjoyable parts of their work! https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#6493d6c76f63 The question we have to ask ourselves is if we can make data preparation easier? Can we minimize the time that people are collecting data sets and cleaning/organizing their data?
• #12 Lake Formation automates many of the steps we discussed, allowing customers to get started with just a few clicks from a single, unified dashboard. 1/ Identify, ingest, clean, and transform data: With Lake Formation, you can move, store, catalog, and clean your data faster. 2/ Enforce security policies across multiple services: Once your data sources are setup, you then define security, governance, and auditing policies in one place, and enforce those policies for all users and all applications. 3/ Gain and manage new insights: With Michigan you build a data catalog that describes available data sets and their appropriate business uses. This makes your users more productive by helping them find the right data set to analyze. By providing a catalog of your data and consistent security enforcement, Michigan makes it easier for your analysts and data scientists to combine multiple analytic tools, like Athena, Redshift, and EMR, across diverse data sets.
• #13 With just a few clicks, you can setup your data lake on Amazon S3 and start ingesting data that is readily queryable. Lake For To get started, you go to the Michigan dashboard in the AWS console, add your data sources and then Michigan will crawl those sources and move the data into your new Amazon S3 data lake. Michigan uses machine learning to automatically lay out the data in Amazon S3 partitions, change it into formats for faster analytics, like Apache Parquet and ORC, and also de-duplicates and finds matching records to increase data quality. From a single screen set up all of the permissions for your data lake and they will be implemented across all services accessing this data - analytics and machine learning services (Amazon Redshift, Amazon Athena, and Amazon EMR.) This reduces the hassle in re-defining policies across multiple services and provides consistent enforcement and compliance of those policies. mation …
• #17 Blueprints heavily leverage the functionality in AWS Glue: We use Glue crawlers and connections to connect and discover the raw data that needs to be ingested. We use Glue code-gen and jobs to generate the ingest code to bring that data into the data lake. We leverage the data catalog for organizing the metadata. We have added a workflow construct to stitch together crawlers, jobs, and allow for monitoring for individual workflows. They’re an natural extension of the AWS Glue capabilities.