Downloaded 11 times
![5
“…autonomous vehicles would have
to be driven hundreds of millions of
miles and sometimes hundreds of
billions of miles to demonstrate their
reliability in terms of fatalities and
injuries.” [1]
[1] Driving to Safety, Rand Corporation
https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-5-2048.jpg)
![48
Scaling considerations
Storage
§ Cloud based storage allows
ingestion of larger datasets
Compute
§ Quicker training iterations both
locally and on the cloud
§ Faster inference pipeline
Cost
§ CapEx è OpEx
Workflows
§ Self-serve analytics optimized for
agility along a maturity framework
"Glue code and pipeline jungles are symptomatic of
integration issues that may have a root cause in
overly separated “research” and “engineering”
roles…
… engineers and researchers are embedded together
on the same teams (and indeed, are often the same
people) can help reduce this source of friction
significantly [1].
[1] The Hidden Technical Debt of Machine Learning Systems
https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-48-2048.jpg)
![54
Conclusions
§ Well-architected systems [1] accelerate development of automated
semantic segmentation against large datasets
§ Local development provides smooth workflows for the development and
refinement of deep learning models
§ Cloud-based scaling of compute and storage can be leveraged on
Databricks to enable self-service analytics
§ Techniques that are relevant for ADAS development are equally applicable
across other domains such as medical, geo-exploration, etc.
[1] AWS Well-Architected
https://aws.amazon.com/architecture/well-architected/](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-54-2048.jpg)
![5
“…autonomous vehicles would have
to be driven hundreds of millions of
miles and sometimes hundreds of
billions of miles to demonstrate their
reliability in terms of fatalities and
injuries.” [1]
[1] Driving to Safety, Rand Corporation
https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf](https://crownmelresort.com/image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-5-2048.jpg)
![48
Scaling considerations
Storage
§ Cloud based storage allows
ingestion of larger datasets
Compute
§ Quicker training iterations both
locally and on the cloud
§ Faster inference pipeline
Cost
§ CapEx è OpEx
Workflows
§ Self-serve analytics optimized for
agility along a maturity framework
"Glue code and pipeline jungles are symptomatic of
integration issues that may have a root cause in
overly separated “research” and “engineering”
roles…
… engineers and researchers are embedded together
on the same teams (and indeed, are often the same
people) can help reduce this source of friction
significantly [1].
[1] The Hidden Technical Debt of Machine Learning Systems
https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf](https://crownmelresort.com/image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-48-2048.jpg)
![54
Conclusions
§ Well-architected systems [1] accelerate development of automated
semantic segmentation against large datasets
§ Local development provides smooth workflows for the development and
refinement of deep learning models
§ Cloud-based scaling of compute and storage can be leveraged on
Databricks to enable self-service analytics
§ Techniques that are relevant for ADAS development are equally applicable
across other domains such as medical, geo-exploration, etc.
[1] AWS Well-Architected
https://aws.amazon.com/architecture/well-architected/](https://crownmelresort.com/image.slidesharecdn.com/292arvindhosagrahara-210616155254/75/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-54-2048.jpg)
Uploaded byDatabricks
Processing Large Datasets for ADAS Applications using Apache Spark
The document outlines the use of Spark for processing large datasets in automated driving applications, focusing on semantic segmentation and the challenges of moving from prototype to production. It presents the architecture of the system, covering ETL processes, model training, and inference, while addressing design considerations like scaling, security, and governance. Key takeaways emphasize the importance of leveraging cloud-based solutions and effective workflow management to enhance the development of perception software for autonomous vehicles.
More Related Content
![[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...](https://cdn.slidesharecdn.com/ss_thumbnails/draganberic-lakehousearchitecturewithdeltalakeanddatabricks-221130080712-6e817e95-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic](https://cdn.slidesharecdn.com/ss_thumbnails/petarzecevic-databricksoverview-221130080703-c60d93de-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Processing Large Datasets for ADAS Applications using Apache Spark
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Processing Large Datasets for ADAS Applications using Apache Spark
- 1. 1 © 2020 TheMathWorks, Inc. Processing Large Datasets for ADAS applications using Spark Arvind Hosagrahara (ahosagra@mathworks.com) Lead Solutions Architect May 2021
- 2. 2 Outline § The BigPicture – Problem Statement / Challenges § Architecture of the system (ETL, Training, Inference) § Deep-Dive into Spark Workflows – Cluster Management – Interactive exploration – Pushdown of analytics to Spark Clusters – Experiment and metadata management § Design considerations – Scaling, Accuracy and Interoperability – Security, Governance, Model Lifecycle management § Conclusions / Key Takeaways § Q&A
- 3. 3 The Big Picture– Automated Driving How can I analyze & simulate scenarios? How can I design & deploy algorithms? How can I integrate & test systems? Control Planning Perception
- 4. 4 Semantic Segmentation What? § Semanticsegmentation is the classification of every pixel in an image/video Why? § Semantic segmentation: Higher quality perception outcomes, easier to analyze and validation of localization modules for AD. § Automation: Faster design iterations
- 5. 5 “…autonomous vehicles wouldhave to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their reliability in terms of fatalities and injuries.” [1] [1] Driving to Safety, Rand Corporation https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf
- 6. 6 Problem Statement How canwe: 1. Accelerate development of semantic segmentation models from prototype to production in order to build better perception software 2. Scale the workflows, capabilities and throughput of engineering processes by leveraging automation, cloud storage, compute and other infrastructure
- 7. 7 Challenges Business Challenges § Prototypeto production: establish system & software engineering processes § Verification and validation: connect data, embedded software and simulation § Workforce development: hire & integrate engineers with disparate background Technical Challenges § Handling large datasets and specialized formats § Leveraging the cloud to scale computational requirements § Enable self-serve analytics for the data scientist / engineer
- 8. 8 Demonstration / Results (Image-levelfeatures with batch normalization)
- 9.
- 10. 10 Demo / Results 3-Dsimulation data to train a semantic segmentation network and fine-tune it to real-world data using generative adversarial networks (GANs)
- 11.
- 12.
- 13. 13 Architecture of thedemo system Burst Data access Experiment Management / Governance / Security
- 14. 14 Burst Data access Experiment Management / Governance / Security Data access Experiment Management / Governance / Security Development Environment (Desktop/ Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest
- 15. 15 Data access Experiment Management / Governance / Security Development Environment (Desktop /Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest
- 16. 16 A simplified viewof the system architecture Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access
- 17. 17 The ETL (ExtractTransform Load) Pipeline § The ETL (Extract Transform Load) Pipeline Experiment Management / Governance / Security Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Recorded and Simulated Data (Batch and Streaming Ingest) Ingest § §
- 18. 18 The Training Pipeline § Experiment Management / Governance / Security DevelopmentEnvironment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Cloud-based training system (Spark Clusters / GPU support / Fast networks) Burst Ingest Data access § The Training Pipeline §
- 19. 19 The Inference Pipeline § Experiment Management / Governance / Security Storage (LargeDatasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Burst Ingest Data access § § The Inference Pipeline
- 20. 20 The Software Stack Experiment Management / Governance / Security DevelopmentEnvironment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access
- 21. 21 The Software Stack Experiment Management / Governance / Security DevelopmentEnvironment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access MATLAB® and Simulink® MATLAB Production Server™ MATLAB Webapp Server™ 3P Dashboards Amazon S3™ AWS Identity and Access Management (IAM) AWS™ MLflow™
- 22.
- 23. 23 Workflows supporting DevOps DevelopmentEnvironment Operational Environment TEST O P E R A T E DEPLOY MONITOR DESIGN B U I L D P L A N F E E D B A C K Cluster Management JDBC based connectivity to data sources Databricks- connect and interactive exploration CI/CD integration Jobs and Spark-Submit Model Lifecycle Management Experiment Management Coder based acceleration Embedded Applications Inference engine Central result, config, data management
- 24.
- 25. 25 Workflow: Cluster managementand Data Engineering via REST Engineers Developers Scientists Data Analyst Compute Cluster (Cloud/On-Prem) jobsubmission.m MATLAB HTTP Client MATLAB Integration (REST 2.0 API) • Create Clusters • CRUD with DBFS • Create Jobs • Authentication /Tokens • Secrets • Notifications • Libraries • Workspace Databricks specific runtime image MATLAB Runtime init_script Dockerfile dockerized runtime analytics.jar
- 26. 26 Video 1 Outline §Planning capacity and performance § Cluster creation § Data and application transfer via the DBFS API
- 27. 27 Workflow: JDBC andDatabase toolbox (a) Engineers Developers Scientists Data Analyst Existing Compute Cluster (Cloud/On-Prem) DatabaseToolbox ODBC 2.6.x JDBC 2.6.x Driver SQL
- 28. 28 Workflow: JDBC andDatabase toolbox (b) Engineers Developers Scientists Data Analyst Existing Compute Cluster (Cloud/On-Prem) ODBC 2.6.x JDBC 2.6.x Driver SparkSession SQL Databases / Delta Tables
- 29. 29 Workflow: Databricks-connect Engineers Developers Scientists Data Analyst ExistingCompute Cluster (Cloud/On-Prem) Databricks-connect Spark DAG Spark Master Spark API
- 30. 30 Video 2 Outline §Databricks-Connect workflows § Read/Write delta tables
- 31.
- 32. 32 The Training Pipeline(Local Experimentation) § Optimized for GPU § High-level API (“write less and do more”) § Language abstractions to improve accuracy (eg: imageDataAugmenter) § Workflows for ease-of-use and debug – apps for model selection – monitor accuracy, loss and training performance – many quick-start examples
- 33. 33 Workflow: Push-down viaMATLAB Compiler Engineers Developers Scientists Data Analyst Compute Cluster (Cloud/On-Prem) spark-submit job for New Clusters analytics.jar Spark DAG Spark Master analytics.m MATLAB Compiler/SDK MATLAB API for Spark Tall Arrays Libraries for Existing Clusters
- 34. 34 Video 3 Outline §SparkSession and RDD abstractions § Tall-based workflows § Deploytool and app based workflows – Compiler SDK for Java § mcc – Spark Submit § DBFS API to push application to Databricks
- 35. 35 Build and Test §Improve quality of code and models by testing on CI Servers (self-hosted or Cloud-hosted) § Package libraries for deployment § Adhere to agile development practices https://www.mathworks.com/solutions/continuous-integration.html
- 36. 36 The Training Pipeline(Cloud-based training) § Infrastructure defined as code § Leverage the latest hardware, optimized drivers and toolkits § APIs for command and control of Jobs, artifacts, notifications, execution, etc. § Capital Expenses (CapEx) è Operating Expenses (OpEx)
- 37. 37 Management of Experimentsand tracking of performance
- 38. 38 Video 4 Outline §MLFlow Demo – Creation of Experiments / Runs – Logging of Parameters and Metrics – Integration with Experiment Management features
- 39. 39 Training Benchmarks andPerformance VGG-16 based network on NVIDIA Quadro GV100
- 40.
- 41. 41 The Inference Pipeline §Directed Acyclic Graphs (DAG) used to resolve changes to: – Training models, ETL changes – The training data – New data / changes to the cache § Trained Models can be deployed as cloud services § Cloud Reference Architectures available on github.com to integrate with existing systems
- 42. 42 Scaling Inference onthe MATLAB Production Server § Scales gracefully § Optimal usage of available CPU and GPU resources § Can increase throughput as desired § PAYG models available on the cloud
- 43.
- 44. 44 The ETL Pipeline §Wide variety of Automotive and Engineering Formats (eg: ROSBag, MP4, MDF, BLF, PCAP) § RESTful endpoints or Client based access § Connectivity to Cloud Storage (S3) and distributed streaming systems (Kafka) § Horizontally scalable and designed to run 24x7x365 on headless cloud systems § Provides data proximity when executing on the cloud § Accelerated CUDA execution on NVIDIA® via GPU Coder™
- 45. 45 Putting it alltogether
- 46. 46 Build Graph (DAG) §Engine code § Trained models § New/Cached Data § Operates on S3 version metadata § Can be triggered manually or automatically (eg: via CI/CD pipelines)
- 47.
- 48. 48 Scaling considerations Storage § Cloudbased storage allows ingestion of larger datasets Compute § Quicker training iterations both locally and on the cloud § Faster inference pipeline Cost § CapEx è OpEx Workflows § Self-serve analytics optimized for agility along a maturity framework "Glue code and pipeline jungles are symptomatic of integration issues that may have a root cause in overly separated “research” and “engineering” roles… … engineers and researchers are embedded together on the same teams (and indeed, are often the same people) can help reduce this source of friction significantly [1]. [1] The Hidden Technical Debt of Machine Learning Systems https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
- 49. 49 Accuracy considerations § MoreTraining Data § Simulation / Scene generation – Roadrunner Simulation
- 50. 50 Interop considerations § ONNXinterop § Transfer learning techniques § Import/Export to – OpenDRIVE®, – FBX®, – glTF™, – OpenFlight, – OpenSceneGraph, – OBJ, and USD formats. § Automated driving simulators and game engines (eg: CARLA, VIRES VTD, NVIDIA DRIVE Sim®, Metamoto®, LGSVL, Baidu Apollo®, Unity®, and Unreal Engine®)
- 51. 51 Embedded Targeting § GPUCoder™ Support Package for NVIDIA GPUs § NVIDIA® DRIVE™ and Jetson hardware platforms
- 52. 52 Governance and Lifecyclemanagement § Local Experimentation § Centralized Tracking
- 53. 53 Security Considerations § CloudStorage Best Practices § ACL based permissions § IAM based identity and access
- 54. 54 Conclusions § Well-architected systems[1] accelerate development of automated semantic segmentation against large datasets § Local development provides smooth workflows for the development and refinement of deep learning models § Cloud-based scaling of compute and storage can be leveraged on Databricks to enable self-service analytics § Techniques that are relevant for ADAS development are equally applicable across other domains such as medical, geo-exploration, etc. [1] AWS Well-Architected https://aws.amazon.com/architecture/well-architected/
- 55. 55 Key Takeaways § Buildupon proven, safety compliant, and DevOps-ready development tools § Leveraging best-in-class simulation integration platform and modeling tools leads to faster time-to-market outcomes for ADAS development § Domain specific tooling allows practitioners to write less and do more § If this is interesting, please contact us at databricks@mathworks.com for more details or to get started.
- 56. 56 Q&A Contact us at:databricks@mathworks.com
- 57.