How Synthetic Data Improves Decision-Making

When I have a conversation about AI with a layperson, reactions range from apocalyptic fears to unrestrained enthusiasm. Similarly, with the topic of whether to use synthetic data in corporate settings, perspectives among leaders vary widely. We're all cognizant that AI systems rely fundamentally on data. While most organizations possess vast data repositories, the challenge often lies in the quality rather than the quantity. A foundational data estate is a 21st century competitive advantage, and synthetic data has emerged as an increasingly compelling solution to address data quality in that estate. However, it raises another question. Can I trust synthetic more or less than experiential data? Inconveniently, it depends on context. High-quality data is accurate, complete, and relevant to the purpose for which its being used. Synthetic data can be generated to meet these criteria, but it must be done carefully to avoid introducing biases or inaccuracies, both of which are likely to occur to some measure in experiential data. Bottom line, there is no inherent hierarchical advantage between experiential data (what we might call natural data) and synthetic data—there are simply different characteristics and applications. What proves most trustworthy depends entirely on the specific context and intended purpose. I believe both forms of data deliver optimal value when employed with clarity about desired outcomes. Models trained on high-quality data deliver more reliable judgments on high impact topics like credit worthiness, healthcare treatments, and employment opportunities, thereby strengthening an organization's regulatory, reputational, and financial standing. For instance, in a recent visit a customer was grappling with a relatively modest dataset. They wanted to discern meaningful patterns within their limited data, concerned that an underrepresented data attribute or pattern might be critical to their analysis. A reasonable way of revealing potential patterns is to augment their dataset synthetically. The data set would maintain statistical integrity (the synthetic mimics the statistical properties and relationships of the original data) allowing any obscure patterns to emerge with clarity. We’re finding this method particularly useful for preserving privacy, identifying rare diseases or detecting sophisticated fraud. As we continue to proliferate AI across sectors, senior leaders must know it's not all "upside." Proper oversight mechanisms to verify that synthetic data accurately represents real-world conditions without introducing new distortions is a must. However, when approached with "responsible innovation" in mind, synthetic data offers a powerful tool for enabling organizations to augment limited datasets, test for bias, and enhance privacy protections, making synthetic data a competitive differentiator. #TrustworthyAI #ResponsibleInnovation #SyntheticData

Prostate cancer (PCa) is a leading cause of cancer-related mortality in men, with Gleason grading critical for prognosis and treatment decisions. Machine learning (ML) models offer potential for automated grading but are limited by dataset biases, staining variability, and data scarcity, reducing their generalizability. This study employs generative adversarial networks (GANs) to generate high-quality synthetic histopathological images to address these challenges. A conditional GAN (dcGAN) was developed and validated using expert pathologist review and Spatial Heterogeneous Recurrence Quantification Analysis (SHRQA), achieving 80% diagnostic quality approval. A convolutional neural network (EfficientNet) was trained on original and synthetic images and validated across TCGA, PANDA Challenge, and MAST trial datasets. Integrating synthetic images improved classification accuracy for Gleason 3 (26%, p = 0.0010), Gleason 4 (15%, p = 0.0274), and Gleason 5 (32%, p < 0.0001), with sensitivity and specificity reaching 81% and 92%, respectively. This study demonstrates that synthetic data significantly enhances ML-based Gleason grading accuracy and improves reproducibility, providing a scalable AI-driven solution for precision oncology. Mitigating bias in prostate cancer diagnosis using synthetic data for improved AI driven Gleason grading: https://lnkd.in/eUir5vtT Interesting study demonstrating that carefully validated synthetic data can significantly enhance AI-based Gleason grading in Prostate Cancer Diagnosis. The approach offers a scalable solution for improving diagnostic accuracy, potentially reducing over-diagnosis while maintaining high clinical performance. Paper and research by Derek Van Booven and larger team at the University of Miami

🚀 Generating High-Quality Synthetic Data — While Preserving Feature Relationships In today’s data-driven world, organizations urgently need realistic data for testing, development, and AI training—but privacy concerns and regulations like HIPAA and FERPA often make using real data impossible. That's where structured synthetic data comes in. Harpreet Singh and I developed a synthetic data generation pipeline that not only mimics the distribution of real data—but also preserves the relationships between features, something many approaches overlook. 🧠 Here's a look at what sets this approach apart: ✅ Preprocessing - Imputes missing values (median/mode/“Unknown”) - Encodes categoricals smartly: binary, one-hot, or frequency-based - Fixes skewed features using Box-Cox - Standardizes numerical data - Stores all parameters for full reversibility 🔍 Clustering with HDBSCAN Real data often comes from diverse subgroups (e.g., customer segments or patient cohorts). Using HDBSCAN, we automatically detect natural clusters without predefining their number. This ensures minority patterns aren’t averaged out. 📊 Per-Cluster Modeling Using Copulas Each cluster is modeled independently to capture local behavior. - First, we fit the best marginal distribution for each feature (normal, log-normal, gamma, etc.) - Then, using copulas (Gaussian, Student-T, Clayton), we preserve the inter-feature dependencies—ensuring we don’t just get realistic individual values, but also realistic combinations This step is crucial. It avoids scenarios like low-income customers buying large numbers of luxury items—something that happens when relationships aren't preserved. 🎯 Generation and Postprocessing - Samples are drawn from the fitted copula - Inverse CDF restores each feature’s shape - Reverse standardization and decoding returns everything to the original format - Categorical encodings are fully recovered (binary, one-hot, frequency) 🧪 Validation The pipeline doesn't stop at generation—it rigorously validates: - Kolmogorov-Smirnov and chi-square tests for distributions - Correlation matrix comparison (Pearson, Spearman) - Frobenius norms for dependency structure accuracy - Cluster proportion alignment ⚠️ Limitations: All variables are treated as continuous during dependency modeling—so while relationships are preserved broadly, some nuanced categorical interactions may be less precise. ✅ Use Cases: - Safe test data for dev teams - Realistic ML training data - Simulating rare edge cases - Privacy-preserving analysis in finance, health, and retail 📚 Full breakdown with code is here: 👉 https://lnkd.in/gS5a3Sk7 Let us know what you think—or if you'd like help implementing something similar for your team. If you find it useful, don't shy away from liking or reposting it. #SyntheticData #Privacy #AI #MachineLearning #DataScience #Copulas #HDBSCAN #DataEngineering

In the realm of building machine learning models, there are typically two primary data sources: organic data, stemming directly from customer activities, and synthetic data, generated artificially through a deliberate process. Each holds its unique value and serves a distinct purpose. This blog post, written by the Data Scientists at Expedia Group, shares how their team leveraged synthetic search data to enable flight price forecasting.   -- [Business need] The primary objective is to develop a price forecasting model that can offer future flight pricing predictions to customers. For instance, it aims to inform customers whether flight prices are likely to rise or fall in the next 7 days, aiding them in making informed purchasing decisions.    -- [Challenges] However, organic customer search data falls short due to its sparsity, even for the most popular routes. For instance, it's rare to see daily searches for two-way flights from SFO to LAX for every conceivable combination of departure and arrival dates in the upcoming three months. The limitations of this organic data are evident, making it challenging to construct a robust forecasting model.   -- [Solution] This is where synthetic search data comes into play. By systematically simulating search activities on the same route and under identical configurations, such as travel dates, on a regular basis, it provides a more comprehensive and reliable source of information. Leveraging synthetic data is a potent tool for systematic exploration, but it requires a well-balanced approach to ensure that the benefits outweigh the associated costs. Striking this balance is essential for unlocking the full potential of synthetic data in data science models. – – –  To better illustrate concepts in this and future tech blogs, I created one podcast "Snacks Weekly on Data Science" (https://lnkd.in/gKgaMvbh) to make them more accessible. It's now available on Spotify and Apple podcasts. Please check it out, and I appreciate your support! #machinelearning #datascience #search #synthetic #data #forecasting https://lnkd.in/gRjR5tTQ