Download to read offline
Uploaded bySafe Software
Taming the Chaos: How to Turn Unstructured Data into Decisions
Unstructured data is everywhere: buried in reports, scattered across image folders, or hidden in scanned documents. But with the right tools, it’s possible to extract meaning and turn chaos into clarity. In this webinar, you’ll discover how to use FME to process unstructured data types and unlock new insights. From document summarization and metadata extraction to AI-powered semantic search, we’ll walk through four practical demos that make your data work harder. You’ll learn how to: -Categorize a series of documents using transformers and workflows -Extract and structure key fields from messy PDFs or forms -Use OpenAI embeddings with FME to enable powerful semantic search -Map photo metadata and enrich it with GPS validation Whether you’re wrangling PDFs, images, or forms: this session will show how to streamline your unstructured data strategy.
More Related Content
Similar to Taming the Chaos: How to Turn Unstructured Data into Decisions
Taming the Chaos: How to Turn Unstructured Data into Decisions
- 1. This is WhereWe Will Write the Title for our Webinar
- 2.
- 3. Welcome to Livestorm. Afew ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. How to downloadslides 1. Hover over the slide deck in the webinar room 2. Click this button
- 5. Agenda 1 Introduction 2 Basicsof Prompt Engineering for FME 3 Summarizing Documents 4 Auto-Extract & Organize Info from Unstructured Docs 5 Using OpenAI to Create Embeddings 6 Image Metadata Extraction & Geo-tagging 7 Conclusion 8 Resources & Next Steps 9 Q&A Agenda
- 6.
- 7. “By 2025, IDCestimates there will be 175 zettabytes of data globally”... “with 80% of that data being unstructured.” - Forbes
- 8. ● Data withouta predefined format: text, images, PDFs, and more. ● Hard to query, organize, or visualize without transformation. ● Found in emails, reports, field photos, customer forms, and more. ● Often holds critical information hidden in plain sight. Unstructured Data & Why It Matters
- 9. Poll: What’s your maingoal for extracting information from unstructured data? (Select top 1-2)
- 10. Poll: How often doyou work with unstructured data in your day-to-day tasks?
- 11. Why Unstructured DataIs a Headache ● Manual processing is slow and error-prone. ● Data is inconsistent, non-standard, or incomplete. ● Difficult to search, filter, or analyze. ● High effort for low immediate value, unless automated.
- 12. What FME Enables: ●Extract structured meaning from messy sources. ● Automate classification, tagging, and conversion. ● Combine AI tools like OpenAI with geospatial intelligence. ● Power semantic search, dashboards, and maps from raw inputs.
- 13. The only All-Data,Any-AI Platform. FME Form FME Flow Data Movement and transformations (“ETL”) workflows are built here. Brings life to FME Form workflows FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME FME Realize Experience data in real world context, in real time.
- 14. With 500+ supporteddata types in FME. Unrivalled Data Support GIS CAD Database XML Raster 3D BIM Web Point Cloud Cloud Big Data IOT Graph BI Indoor Mapping AR/VR Generative AI Cloud Native Tabular
- 15.
- 16. Prompt Engineering 1. Beclear and specific in your prompt. The more context you provide, the better the model performs. 2. Role-based prompts (e.g., “You are a document classification assistant…”) help set expectations for the model. 3. Use structured output whenever possible. This makes downstream processing much easier, especially in FME, where structured JSON can be parsed into attributes using the JSONFlattener. 4. Enumerate options. If you want consistent classification (e.g., fixed categories), provide a defined list the model can choose from. 5. Ask for justification. Including fields like explanation or confidence_score can help with auditing and quality assurance.
- 17. Bad Prompt "Classify thistext." ● No context or structure ● No role or output format ● No categories or justification
- 18. Better Prompt Role-based: “Youare a document classification assistant.” Specific task: Classify text into one of these categories: "Financial Report", "Policy Document", "Technical Manual", "Legal Agreement", "Other" Structured output (JSON): { "category": "<selected_category>", "explanation": "<brief reasoning>", "confidence_score": "<0.0 to 1.0>" } Enumerated options: Fixed list ensures consistency. Auditable: Includes explanation and confidence score.
- 19.
- 20. OpenAIChatGPTConnector ● OpenAI: GPT-4o ●Azure OpenAI Model Catalog: Easy access to 1,672 models FME & Cloud Based AI Services AmazonBedrockConnector ● AI21 labs ● Claude ● Cohere ● Stable Diffusion ● Amazon Titan ● Llama GoogleGeminiConnector ● Gemini 1.5 Pro ● Model Garden: Easy access to 165 models Discover more on the FME Hub!
- 21. google-research/ albert ● Build aframework to exploit LLM models locally. ● Compliment the cloud based services. ● Data stays local reducing any security concerns. FME & AI Services Integrate Embedded AI Services
- 22. ● Many organizationsrely on lengthy PDF documents for reporting, compliance, and operations. ● Manually summarizing this content is time-consuming and prone to inconsistency. ● With FME, you can automate summarization using AI-powered large language models. ● Integrate the results directly into Excel, databases, dashboards, or alerting tools. Use Case Introduction Article: Getting Started with AI in FME: Extracting Insights from Unstructured Documents
- 23. OpenAIConnector The File SearchAPI empowers the OpenAIConnector to retrieve relevant information from uploaded files using combined semantic and keyword search logic. It “grounds” responses in your own document base—enhancing accuracy and domain knowledge at runtime. Documentation
- 24. Model: default gpt-5but can use gpt-4o Instructions: Persistent guidance that sets the assistant’s behavior, style, or goals for a conversation or request. User Prompt: The specific question or task the user wants answered in that interaction. Structured Output: A response format that returns results in a defined schema (e.g., JSON), making outputs consistent and machine-readable.
- 25. Slide Title Automate the extractionof key insights and structured data Goal Block Key Summarizing Documents Result Valuable insights often stay hidden inside large, messy files AI integrated into automated FME workflows Minutes instead of hours to process reports
- 26.
- 27. ● Use AdvancedSettings for Large Files ○ The OpenAIConnector has Advanced Settings where you can increase timeouts. ○ Helpful for processing larger files without hitting request limits. ● Adjust the Temperature Setting ○ Higher values (e.g., 0.8) → more creative, varied, and random output. ○ Lower values (e.g., 0.2) → more consistent, predictable output. ○ Keep it low for automation tasks to ensure repeatable results. Open AI Docs Lessons Learned
- 28. 4 Auto-Extract and Organize KeyInfo from Unstructured Docs
- 29. Directory and File PathnamesReader Reads file paths to pass to the OpenAIConnector OpenAIConnector Processes the files and categorizes them FileCopy Writer Moves the file to a new location and give them a proper filename Article: Getting Started with AI in FME: Classifying Unstructured PDF Files
- 30. Slide Title Automatically classify afolder containing 1,000 unknown PDF files into meaningful categories Goal Block Key Classifying PDFs Result Without automation, this task would require significant manual time and effort. Leverage FME and AI to automatically classify documents. Now you have organized files!
- 31.
- 32. ● Refine theprompt: ○ Language classification- sometimes it would produce en, ENGLISH or english ○ Filenames would often include extensions Lessons Learned
- 33.
- 34. The Problem withFree-Form Text ● Notes, reports, emails, and logs are full of unstructured text. ● Keyword search is limited — if you don’t guess the exact word, results are missed (e.g. “corrosion” ≠ “rusting”) ● People want to ask questions in their own words and still find the right information.
- 35. A Smarter SemanticWay to Search ● Find similar meaning: e.g. “pipe is corroded” ≈ “pipe is rusted out” ● Spot anomalies: detect unusual notes or outliers automatically ● Ask in plain language: explore data through simple API queries ● Discover patterns: group related behaviors and track changes over time FME + AI: Turn messy text into meaningful, searchable insights 🔍
- 36. Dictionary Corner ● Embeddings:Numbers that capture the meaning of text ● Embedding Models: AI services that turn text into numbers ○ OpenAI → text-embedding-3-small, text-embedding-3-large ○ Cohere → embed-english-v3.0, embed-multilingual-v3.0 ● Vectors: The format those numbers are stored in ● Vector Databases: Store and search vectors (e.g. Pinecone, Weaviate, Milvus, PostgreSQL) ● Semantic Search: Finds results by meaning, not exact words
- 37. Slide Title Make messy maintenance notes searchableby meaning. Goal Block Key From Text to Vectors: Building a Semantic Search Engine Result Notes are free text in a Word file — hard to filter or analyze. Use FME + AI embeddings + Postgres pgvector to store and search. Plain language queries return meaningful matches, not just keywords.
- 38.
- 39. Case Study: Semantic Searchwith OpenAI and Postgres Embeddings: Turning Unstructured Notes into Data
- 40.
- 41. Case Study: Semantic Searchwith OpenAI and Postgres “Severe conditions requiring immediate shutdown” Semantic Search: Finding Insights Beyond Keywords 🔍
- 42. Lessons Learned ● Prepyour text → Run QA; prep raw data for ingestion ● Prototype small → Use Sampler to save tokens during dev. ● Models matter → Balance size, accuracy, cost, and speed; keep all embeddings from the same model. ● Normalize, or don’t → Some models do that for you. ● Performance tip → Run FME Engine close to DB to cut latency. Let the DB do the work. ● Pick the right DB types → In lieu of vector? Use TEXT or JSONB and cast it. ● Be creative → FME can bridge gaps when features aren’t out-of-the-box.
- 43. Sub-Section Name Here ●Maintenance logs in any industry. ● Customer service chat records. ● Research papers or legal documents. ● Product descriptions for e-commerce search. ● Multimedia captions and metadata. ● What else? Better Decisions Through Smarter Search My favorite vector database? RepTileDB… optimized for fly retrieval 🪰
- 44.
- 45. The Problem: MetadataYou Can’t See ● Photos, PDFs, CAD, images = unstructured files ● They carry embedded metadata (time, location, device, authorship) ● Attributes are hidden inside the file ● Without extraction → no way to trust, validate, or analyze
- 46. Why Metadata Extraction Matters ●Metadata = structured clues inside unstructured files ● FME Readers + Transformers → expose hidden attributes ● Attributes become visible + comparable ● Enables automated QA/QC and confidence scoring
- 47. 🛑 Bad DataHurts ● Wasted effort ● Lost credibility ⚡ Fix with Metadata ● Automatic cross-checks ● Scalable QA/QC ● Trustworthy reports Data Quality Detective Work (49.2775,-123.1206) Dec 20, 2024 (-33.8586, 151.2139) Jul 3, 2020
- 48. Slide Title Verify that reportedsurvey information matches the actual photo metadata. Goal Block Key JPEG Metadata QA: Survey Verification Result Survey photos are unstructured JPEGs with hidden GPS/time data, not visible in raw form. Expose and process hidden EXIF attributes, then compare them to reported survey details. Confident, automated QA scoring ensures trustworthy survey records at scale.
- 49.
- 50. Lessons Learned ● Tamethe chaos – metadata is messy; rename attributes clearly and keep only what you need . ● Make it usable – expose and manage key values so downstream tools and AI can find patterns, context, and even explain obscure fields. ● Handle with caution – metadata isn’t always accurate (e.g. GPS can lie), and sharing photos can reveal more than you intended.
- 51. Sub-Section Name Here ●Asset inspections → confirm repair photos with time & place ● Environmental monitoring → validate sample collection metadata ● Compliance & audits → extract authorship, edit history ● AI workflows → enrich datasets with contextual attributes Metadata Mysteries: Solved
- 52.
- 53. Summary ● FME makesunstructured data usable: PDFs, images, forms, and photos ● Clean, structure, and enrich metadata for real insights ● Combine FME with AI for summarization, explanation, and semantic search ● Practical workflows: categorization, extraction, enrichment, validation
- 54. All Data. AnyAI. All Data Velocities Batch (ETL, Reverse ETL, ...) Event ( BPA, RPA, ...) Stream All Data Locations Any Cloud On-premises Hybrid Edge Containers Embedded Mixed All Data Types Unstructured Structured Spatial APIs Web Apps … Any AI Technology OpenAI Amazon Bedrock Google Gemini Ollama Deepseek Composite
- 55. 30+ 30K+ 128 140+ 25K+ years of solvingdata challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide 200K+ users worldwide
- 56.
- 57. Get our Ebook SpatialData for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base support.safe.com Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 58.
- 59. We’d love tohelp you get started. Get in touch with us at info@safe.com Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 60. ClaimYour Community Badge& Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: V91J70 ● Join bingo! community.safe.com/p/bingo25 Join the Community today! Next Steps
- 61.
- 62. ThankYou Recap of NextSteps 1 Follow us on LinkedIn! 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey