Why Production and Data Intelligence Environments Differ

Post-deployment data science begins with understanding how production data differs from training data. This is where theory meets reality and can make or break your model’s performance. [1] In training, ground truth labels are available, making performance evaluation straightforward. But in production, especially for scenarios like predictive maintenance, true labels might not be accessible. How can you evaluate a model without them? (You can with performance estimation algorithms) [2] Then there’s data drift. Training data distributions are stable and predictable. But once in production, data can shift over time, and your model has to keep up to stay relevant. [3] Data quality also changes, i.e., training data is cleaned and standardized, but production data often arrives with quality issues that can drag down model performance if not addressed. [4] Lastly, concept drift. In training, your model is built on patterns that are familiar and consistent. But in production, new patterns and behaviors can emerge as the world evolves. This means your model may need to be retrained every time there is concept drift. Is your data science team ready for production data and all the challenges that come with it?

LLMs are shepherding in a new era of AI, no doubt about it. And while the volume and velocity of innovation is astounding, I feel that we are forgetting the importance of the quality of the data that powers this. There is definitely a lot of talk on what data is used to train the massive LLMs such as OpenAI, and there is a lot of talk on leveraging your own data through finetuning and RAG. I also see an increased attention on ops, whether it is LLMOps, MLOps or DataOps, all of which is great to keeping your system and data running. What I seeing far less attention to is managing your data, ensuring it is of high quality and that it is available when and where you need it. We all know about garbage in garbage out -- if you do not give your system good data, you will not get good results. I believe that this new era of AI means that data engineering and data infrastructure will become key. There are numerous challenges to get your system into production from a data perspective. Here are some key areas that I have seen causing challenges: 1. Data: The data used in development is often not representative of what is seen in production. This means the data cleaning and transforms may miss important aspects of production data. This in turn degrades the model performance as they were not trained and tested appropriately. Often new data sources are introduced in development that may not be available in production and they need to be identified early. 2. Pipelines: Moving our data/ETL pipelines from development to staging to production environments. Either the environment (libraries, versions, tools) have incompatibilities or the functions written in development were not tested in the other environments. This means broken pipelines or functions that need rewriting. 3. Scaling: Although your pipelines and systems worked fine in development, even with some stress testing, once you get to the production environment and do integration testing, you realize that the system is not scaling the way you expected and are not meeting the SLAs. This is true even for offline pipelines. Having the right infrastructure, platforms and teams in place to facilitate rapid innovation with seamless lifting to production is key to stay competitive. This is the one thing I see again and again being a large risk factor for many companies. What do you all think? Are there other key areas you believe are crucial to pay attention to in order to achieve efficient ways to get LLM and ML innovations into production?

On paper, everything works flawlessly. Test data runs through the pipelines, models are trained, and the results look promising… But the moment you switch to real-world data, reality hits hard. Test data is like practicing in a controlled environment consistent, predictable, and clean. Real data, on the other hand, is messy, inconsistent, and full of surprises. It comes from multiple integrations, different data models, and systems that rarely speak the same language. One of the biggest lessons I’ve learned is that testing in isolation is not enough. The real test happens when your solution has to handle the chaos of live environments, unexpected edge cases, and integrations that don’t always play nice. If you’re only validating with test data, you’re setting yourself up for surprises in production. The key is to embrace the complexity of real data as early as possible in the process. It’s not as smooth, but it’s the only way to ensure what you build actually works when it matters most. #DataEngineering #DataQuality #datagovernance #AI #data