How AI and data engineering are changing data access with dbt MCP and Snowflake

Memory vs Tokens in AI Agents (aka: stop sending everything) Last 2 months I kept running into the same question: we’re giving LLMs way more context than they actually need — sometimes close to or over 1M tokens — every single turn for big dataset . In fact a big excel can over 1M token. In data science we don’t read every row for any big data; we select, aggregate, filter. But in AI agents we often just dump the whole history/RAG result/knowledge and hope for the best. Result: higher latency, higher cost, no better answer. Last week I switched to a simple rule: right info, right form, right time — not “everything, always.” Rebuilt my agents’ data flows and logic. Summaries or structured, querying, SQL select, refs for most turns, full text only when the task truly needs it. That alone cut tokens and made responses cleaner, and my agent can handle significant large content and data (No over 1M error return, no shocking bill again). Are you seeing this too — agents that “remember” by re-sending everything? Are you team send it all or team selective context? #AI #GenAI #AIAgents #LLM #RAG #PromptEngineering #MLOps #DataScience #CostOptimization

Building a Smarter Text-to-SQL Chatbot with Multi-Agent AI (Part 1) From Natural Language to Database Insights — Safely, Intelligently, and at Scale. Introducing the Multi-Agent Approach Instead of one massive model trying to do everything, imagine a team of specialized AI agents, each with a defined role — like a data analysis team. That’s what a multi-agent architecture brings. Query Rewriter Agent: Refine and clarifies user queries before processing. Schema Agent: Retrieves relevant tables, columns and relationship using RAG Query Generation Agent: Writes SQL from the enhanced question + schema context Validation Agent: Ensures SQL is safe, syntactically correct and read only. Execution Agent: Executes validated SQL on a replica database Visualization Agent: Displays results as tables and charts Explainability Agent: Converts raw SQL and output into human-friendly explanations. https://lnkd.in/g9mUM6Qv

🚀 Exploring the Future of Data: From Star Schemas to Vector Databases! 🚀 Just finished an insightful lecture on data warehousing and vector databases, and I’m excited to share some highlights: 🔹 Data Warehousing & Star Schema: Central fact tables connected to denormalized dimension tables enable faster and more flexible analytical queries. Surrogate keys improve query performance by replacing complex business keys with system-generated numeric keys. Finding the right granularity balances detail with performance for better decision-making. 🔹 Vector Databases & Embeddings: Move beyond traditional SQL limitations to semantic search powered by high-dimensional embeddings. Enable fuzzy, multilingual, and context-aware search—perfect for e-commerce, AI chatbots, and more. Techniques like cosine similarity and Euclidean distance help find meaning, not just exact matches. This blend of structured data design and cutting-edge semantic search is shaping the future of AI and business intelligence. Excited to learn more about how Retrieval-Augmented Generation (RAG) builds on this foundation! #DataWarehousing #VectorDatabases #AI #MachineLearning #SemanticSearch #DataScience #BusinessIntelligence #BigData #RAG

🧠 RAG 2.0: The Next Leap in How AI Thinks Most people think of Retrieval-Augmented Generation (RAG) as a way to feed documents into a language model. That’s RAG 1.0. RAG 2.0 is something else entirely — it’s where AI reasoning meets data architecture. Here’s the shift 👇 ➡️ RAG 1.0: Pulls unstructured data (PDFs, text, notes) and gives the model context. ➡️ RAG 2.0: Merges structured and unstructured sources — think databases, APIs, vector stores, and knowledge graphs — into a hybrid context system. Instead of just “retrieving,” the model now: • Understands relationships between structured data (e.g., SQL, CRM fields) • Combines it with narrative insights (emails, reports, transcripts) • Reasons across both to deliver contextual intelligence, not just answers. 💡 Why this matters: RAG 2.0 doesn’t just improve recall — it improves judgment. We’re entering an era where AI doesn’t just fetch information; it interprets, correlates, and decides. By 2026, companies won’t ask “How do we fine-tune a model?” They’ll ask “How do we unify our context layer?” 🚀 The winners won’t just have the best model — they’ll have the best memory system. 👉 Do you think RAG 2.0 will replace fine-tuning as the standard for enterprise AI reasoning? #ArtificialIntelligence #RAG #MachineLearning #LLMs #AI #DataEngineering #KnowledgeGraphs #Automation #EnterpriseAI #AIAgents #TechTrends #FutureOfWork #AIInnovation #DigitalTransformation

🤖🧠 PandasAI: Transforming Data Analysis with Conversational Artificial Intelligence 🗓️ 28 Oct 2025 📚 AI News & Trends In a world dominated by data, the ability to analyze and interpret information efficiently has become a core competitive advantage. From business intelligence dashboards to large-scale machine learning models, data-driven decision-making fuels innovation across industries. Yet, for most people, data analysis remains a technical challenge requiring coding expertise, statistical knowledge and familiarity with libraries like ... #PandasAI #ConversationalAI #DataAnalysis #ArtificialIntelligence #DataScience #MachineLearning

The MCP Server is now smarter! New SYSTEM_EXECUTE_SQL tool allows your agent to go beyond reasoning to directly query & analyze structured data for verified insights. This is key for production-scale, secure AI. Read all about it in Dash Desai's post. ⬇️

🤯 Just read a fascinating piece comparing Graph RAG and SQL RAG, and it truly sparked a reflection on how we approach intelligent data retrieval. It's a powerful reminder that while we have vast amounts of data, the real magic happens when we can access and synthesize it efficiently. Whether you're leveraging the interconnectedness of graph databases or the structured precision of SQL, the choice of RAG architecture isn't just a technical detail – it's a strategic decision that profoundly impacts the insights we can extract and the value we create. It really highlights the need to constantly evaluate our tools against our data's unique landscape. What are your thoughts on optimizing knowledge retrieval in your projects? I'd love to hear your experiences and insights! If you found this valuable, please like this post and follow for more discussions on data, AI, and marketing strategy! #RAG #GraphDatabases #SQLDatabases #DataStrategy #AI #KnowledgeRetrieval Read more: https://lnkd.in/ghDg2x-i

🍒 The Cherry on Top: Going the Extra Mile in Data Science This year, I learned that real value comes from building beyond the initial ask. 🤖 Built Agentic RAG applications with LLM integration—turning simple data collection into intelligent AI assistants ⚡ Achieved 96% performance improvements using Polars—reducing processing from 24 hours to 15 minutes 📊 Enabled 100GB+ dataset processing on limited hardware through out-of-core techniques and Deep Learning models 🔧 Created reusable frameworks: workshops, templates, and boilerplates that elevate the entire team's capabilities The lesson? Every project is an opportunity to build tomorrow's infrastructure, not just solve today's problem. Of course I can’t forget about Al Sayyida Shaima Al Busaidi guiding me through it all. What capabilities have you built this year that will compound in the next? #DataScience #MachineLearning #AI #ProfessionalGrowth #DataEngineeringq

🧹“Data Cleaning: Still the Unsung Hero of Analytics (Even in the Age of AI)” 🤖 AI can summarize data. It can even generate dashboards. But it still can’t fix bad logic, missing context, or broken source systems. That’s why data cleaning remains the quiet backbone of analytics. 💡In one of my recent projects, most of the real value wasn’t in the dashboard it was in tracing inconsistent records back to their root causes and rebuilding trust in the data itself. Because clean data doesn’t just make visuals look good it makes business decisions reliable. 🧠As analytics professionals, we don’t just build dashboards we safeguard data integrity, the foundation AI and executives both rely on. ⚙️ So yes, even in 2025, the most advanced tools still depend on one thing: Well-prepared data and human judgment. 💬 What’s the biggest data-cleaning nightmare you’ve encountered, and did AI help or hurt? 👇 #AnalyticsEngineering #DataQuality #BusinessIntelligence #AI #PowerQuery #SQL #DataAnalytics #DataGovernance #DataIntegrity

Today, let’s see about the 𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐩𝐡𝐚𝐬𝐞. I've seen the biggest impact come from understanding a few core truths in the Data Modeling phase. This is where your data truly starts to speak! Here are 5 critical points to master: 1️⃣ 𝐅𝐞𝐚𝐭𝐮𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠: Model is only as good as its inputs. Transform raw data into meaningful features (e.g., date -> day_of_week, is_weekend). It's where the magic often happens! 2️⃣ 𝐌𝐨𝐝𝐞𝐥 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧 = 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐲: Don't just pick the trendiest algorithm. Your choice must align with your business problem, the type of data, and the need for interpretability. Is it regression, classification, or something else? 3️⃣ 𝐇𝐲𝐩𝐞𝐫𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 𝐓𝐮𝐧𝐢𝐧𝐠 𝐢𝐬 𝐄𝐬𝐬𝐞𝐧𝐭𝐢𝐚𝐥: Models don't optimize themselves perfectly out-of-the-box. Use techniques like Grid Search or Random Search to find optimal settings that unlock peak performance (with proper cross-validation). 3️⃣ 𝐂𝐫𝐨𝐬𝐬-𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐘𝐨𝐮𝐫 𝐒𝐡𝐢𝐞𝐥𝐝: Always Evaluate on Unseen Data! K-Fold Cross-Validation ensures your model generalizes well and isn't just memorizing. Avoid overfitting at all costs. 4️⃣ 𝐈𝐭𝐞𝐫𝐚𝐭𝐞 & 𝐁𝐚𝐬𝐞𝐥𝐢𝐧𝐞: Model building is rarely one-and-done. Train, evaluate, refine features, try another model, tune, repeat! And always compare against a simple baseline, if you can't beat a simple average, you've got work to do! Mastering these transforms you from a code-runner into a strategic model builder. It’s where data turns into actionable intelligence. #DataScience #MachineLearning #DataModeling #AI #FeatureEngineering #ModelBuilding #MLOps