Factors That Determine Data Trustworthiness

One of the most powerful uses of AI is transforming unstructured data into structured formats. Structured data is often used for analytics and machine learning—but here’s the critical question: Can we trust the output? 👉 Structured ≠ Clean. Take this example: We can use AI to transform retail product reviews into structured fields like Product Quality, Delivery Experience, and Customer Sentiment, etc. This structured data is then fed into a machine learning model that helps merchants decide whether to continue working with a vendor based on return rates, sentiment trends, and product accuracy. Sounds powerful—but only if we apply Data Quality (DQ) checks before using that data in the model. Here’s what DQ management should include at least the following: 📌 Missing Value Checks – Are all critical fields populated? 📌 Valid Value Range: Ratings should be within 1–5, or sentiment should be one of {Positive, Negative, Mixed}. 📌 Consistent Categories – Are labels like “On Time” vs “on_time” standardized? 📌 Cross-field Logic – Does a “Negative” sentiment align with a “Excellent product quality” value? 📌 Outlier Detection – Are there reviews that contradict the overall trend? For example, a review with all negative fields but field "Recommend Vendor” has “Yes". 📌 Duplicate Records – Same review text or ID appearing more than once. AI can accelerate many processes—but DQ management processes is what make that data trustworthy.

Data quality is one of the most essential investments you can make when developing your data infrastructure. If you're data is "real-time" but it's wrong, guess what, you're gonna have a bad time. So how do you implement data quality into your pipelines? On a basic level you'll likely want to integrate some form of checks that could be anything from: - Anomaly and Range checks - These checks ensure that the data received fits an expected range or distribution. So let's say you only ever expect transactions of $5-$100 and you get a $999 transaction. That should set off alarms. In fact I have several cases where the business added new products or someone made a large business purchase that exceeded expectations that were flagged because of these checks - Data type checks - As the name suggests, this ensures that a date field is a date. This is important because if you're pulling files from a 3rd party they might send you headerless files that you have to trust they will keep sending you the same data in the same order. - Row count checks - A lot of businesses have a pretty steady rate of rows when it comes to fact tables. The number of transactions follow some sort of pattern, many are lower on the weekends and perhaps steadily growing over time. Row checks help ensure you don't see 2x the amount of rows because of a bad process or join. - Freshness checks - If you've worked in data long enough you'e likely had an executive bring up that your data was wrong. And it's less that the data was wrong, and more that the data was late(which is kind of wrong). Thus freshness checks make sure you know the data is late first so you can fix it or at least update those that need to know. - Category checks - The first category check I implemented was to ensure that every state abbreviation was valid. I assumed this would be true because they must use a drop down right? Well there were bad state abbreviations entered nonetheless As well as a few others. The next question would become how would you implement these checks and the solutions there range from setting up automated tasks that run during or after a table lands to dashboards to finally using far more developed tools that provide observability into far more than just a few data checks. If you're looking to dig deeper into the topic of data quality and how to implement it I have both a video and an article on the topic. 1. Video - How And Why Data Engineers Need To Care About Data Quality Now - And How To Implement It https://lnkd.in/gjMThSxY 2. Article - How And Why We Need To Implement Data Quality Now! https://lnkd.in/grWmDmkJ #dataengineering #datanalytics

Can you trust your dealership’s reports? Many dealer groups use tools like Power BI and Tableau to visualize data from their DMS, CRM, or other systems. These tools are incredibly powerful for reporting but here’s the uncomfortable truth I have discovered through countless calls. They only work if your data is accurate, consistent, and clean. Here’s the challenge I keep seeing in calls. Normalization gaps: Without a layer to standardize data (e.g., inconsistent op codes or naming conventions), insights across stores or brands can be misleading. Data hygiene issues: Duplicate records, stale customer info, and incomplete data lead to inaccurate calculations and blind spots. Fragmentation: Data flowing in from multiple systems (DMS, CRM, marketing tools) often doesn’t align, leaving leadership teams struggling to connect the dots. Take this example: If store A calls a “brake pad replacement” one thing and store B calls it something else, and this data is fed into your reporting without standardization, your service KPIs will never tell a true story. Or worse, imagine running a marketing campaign based on customer records that are 20% duplicates. These gaps aren’t just technical they’re business-critical. Inaccurate data leads to misinformed decisions, missed opportunities, and wasted resources. To truly trust your reports, you need: 1️⃣ Data normalization: Align fields and formats across systems to ensure consistency. 2️⃣ Hygiene processes: Remove duplicates, fix stale records, and validate data in real-time. 3️⃣ Centralized data: Integrate all your systems into a single source of truth to avoid fragmented insights. When these elements are in place, tools like Power BI and Tableau become exponentially more valuable. Instead of visualizing bad data, you’re unlocking reliable, actionable insights for every department—from sales to service to inventory. The question for dealer groups is this: Are you investing as much in your data quality as you are in your reporting tools? For the groups we’re working with at QoreAI, it’s transformative: ✅ Reports they can trust. ✅ Smarter decisions powered by accurate insights. ✅ Confidence in their data—and their strategies. If you’re not 100% confident in the accuracy of your reports, maybe the problem isn’t the tools but the data itself. What’s your biggest challenge when it comes to reporting? Let’s discuss below. #QoreAI #AutomotiveRetail #DataQuality #AIinAutomotive #DealerGroups #DataInsights

Perhaps no issue with regard to data-driven decision making is more important than the concept of validity. But what does validity mean? I wanted to share a publicly available editorial I helped coauthor on this topic ( https://lnkd.in/gwZfwbaP) that can provide guidance to both academics and those in industry. A few thoughts: •Validity claims concern the interpretation and use that are attached to data. For example, we interpret the producer price index for the general freight trucking, long-distance, less-than-truckload sector measured by the BLS as representing a broader construct of "LTL Prices in the United States". We can use these data for different purposes (e.g., budgeting for LTL costs in 2024, indexing our own contract prices, etc.). •The process that generates data is a key facet of validity. The trucking ton-mile index I produce performs well on this dimension because all inputs are generated by representative government data sources that themselves are benchmarked. •Data may be valid for one use but not another. Trucking firms' CSA scores can be used by shippers for carrier selection, but it would be very questionable to terminate operating authority solely because CSA scores are high. Implication: anytime you plan on using data to inform decisions, you need to keep the various facets of validity in mind. You should continually push data providers to provide evidence to support the validity claims they make from their data. #supplychain #supplychainmanagement #data #freight #trucking