Download as PDF, PPTX
Uploaded byFuture Insights
Structuring Data from Unstructured Things. Sean Lorenz
The document discusses the importance of structuring data in the Internet of Things (IoT) era, emphasizing the need for adaptive applications and effective data manipulation using tools like the Python library Pandas. It highlights organizing various data types and managing time-series data efficiently for building data-driven IoT applications. Additionally, it touches on predictive modeling and multimodal sensor fusion for health monitoring and management.
More Related Content
Similar to Structuring Data from Unstructured Things. Sean Lorenz
Structuring Data from Unstructured Things. Sean Lorenz
- 1. Structuring Data fromUnstructured Things Sean Lorenz, Founder & CEO, Senter @seanlorenz | @SenterIoT
- 2. WE CURRENTLY LIVEIN THE INTERNET OF THING ERA
- 3. IS THIS ASGOOD AS THE IOT GETS?
- 4. WE WANT ADAPTIVE APPS FROMCROSS-MANUFACTURER DATA
- 5. APIs ARE THE LIFEBLOODOF THE IOT
- 6. WHAT DO YOUDO WITH ALL THAT DATA?
- 7.
- 8. NO MORE BAD IoTDASHBOARDS
- 9. MORE DATA STRUCTURING &SENSEMAKING
- 10. BUILDING DATA-DRIVEN IOTAPPS 1 2 3 4 INGEST ORGANIZE PREDICT ACCESS/STORE 2 ORGANIZE
- 11. timeseries images textsparse binary sparse analog INGEST - PROCESS DIFFERENT DATA TYPES
- 12. ORGANIZE - THESOLUTION??
- 13. ORGANIZE - PythonData Analysis Library (pandas) • A fast and efficient DataFrame object for data manipulation; • Tools for reading and writing data between in-memory data structures and different formats; • Intelligent data alignment and integrated handling of missing data • Columns can be inserted and deleted from data structures for size mutability; • High performance merging and joining of data sets; • Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data http://pandas.pydata.org/
- 14. ORGANIZE - PythonData Analysis Library (pandas) http://pandas.pydata.org/
- 15. ORGANIZE - HandlingTime Series Data A few things to remember: • Schema design that minimizes memory, disk I/O • How often do you aggregate the data? • Read/write to a database needs to be fast, reliable, scalable, adaptable • Dealing with uneven time period data inputs • How much of the raw data do you keep? • Appending existing vs. creating new DataFrames
- 16. ORGANIZE - 3Examples Twitter, Fitbit, Temperature
- 17. timeseries images textsparse binary sparse analog Deep RNN & LSTM coding of electrical activity to categorize activity peaks Deep RBM coding of facial anomaly detection from security cameras Deep RBM coding of CRM keywords & phrases for concept clustering Sparse PCA & LASSO of ERP system data for delivery probability Sparse Bayesian coding of IoT sensor data for smart trigger event notifications PREDICT - NOT ALL ALGORITHMS ARE CREATED EQUALLY
- 18. PREDICT - GoogleTensorFlow + LSTM RRN + time series data
- 19. ACCESS/STORE - So,so, so many options….
- 20. multimodal sensor fusion w/cognitive deep learning IoT home and health, phone app, & EHR data The Hub for Adaptive Connected Home Health +
- 21. LocationStates Activity Patterns HzBands O2Prediction MedClusters MedUsage Prediction Patient Health States PREDICTIVE HOMEHEALTH IoT EXAMPLE CARE PLAN ACTION
- 22.