How Data Restrictions Affect AI Training

🧠 The world is getting more and more worried about the future of AI... From sensationalist articles like ‘Attack of the psycho chatbot’ to tech visionaries like Elon Musk sounding the alarm on AI’s risks - the conversation is everywhere. My take? Bad data is quietly sabotaging our trust in technology. Remember the sexist chatbot Tay? That was influenced by data from Twitter trolls. Remember Google’s racist Photo app? Poor ethnic diversity in the training data meant the model failed to correctly identify people of color. The truth is, that biased data is the biggest threat to AI models - garbage in, garbage out. For those venturing into AI - be aware of these three biases in your training datasets: 🔍 Systemic bias Often - the bias isn’t intentional but is ingrained in the system where the data is collected. This means your dataset might be skewed from the start, unintentionally favoring or disadvantaging certain groups. 🛠 Selection bias It’s tempting to use a smaller sample from a large dataset for convenience. However, this can lead to non-representative data, inadvertently omitting crucial info that affects your model’s accuracy. 🍒 Confirmation bias We all love being right and will try and find information that confirms our beliefs. This can happen subconsciously based on how we set up and collect data, or consciously if we cherry-pick data that supports what we believe. Vigilance against these biases isn’t just good practice; it’s going to be essential for building trust in models used by people around the world. What steps are you taking to ensure unbiased AI models?

Prediction: In the next phase of AI, some gains will come from *scaling up* dataset & LLM size- and many will now come from *scaling down*. Bigger dataset/model != better anymore. Scaling up: In some frontier areas like multimodal where we're likely far from data scale saturation (e.g. image, video, time series, motion/control, etc) we'll see continued zero-to-one step changes from scaling up data/LLM sizes. This will be especially powerful where there are existing synthetic data generators to leverage (e.g. video game, gameplay, logic engines). Scaling down: In areas like text/chat/etc, scaling to better, bigger jack-of-all-trades generalists will have diminishing returns, due to exhaustion of both resources (hitting limit of data on the internet) and patience (business/developers want reliable results on actual use cases, not better brainteaser memorizers...). Here, base LLMs will continue to commoditize, and the game will be about training/tuning on small, carefully curated use case/domain-specific datasets for high performance (accuracy & cost) on specific tasks. We've already seen amazing results from the latter Snorkel AI! Intuition: - If you are training a toddler to read/talk, volume of raw data matters. - If you are training a new grad employee- you want a carefully curated curriculum for the task they are actually supposed to learn to do. You don't care about how many internet brainteasers they've memorized... you want them to perform on a specific set of tasks with high accuracy and speed. This is a big shift in how we think about LLM "scaling"- and it's all about how you curate & develop the data!

Here’s Why Data is the Lock and Key to AI's Future 🗝 The AI landscape is humming with innovation, yet one thing is abundantly clear: your AI is only as good as the data that feeds it. A Lamborghini without fuel is, after all, just an expensive piece of sculpture. 📊 Why Data Matters in AI Data and processing power are the twin engines driving AI. But as we face a shortage of specialized AI chips, companies are doubling down on sourcing quality data to win in AI. Epoch ai, a research firm, estimates that high-quality text for AI training could be exhausted by 2026. That's not far off. To put this in perspective, the latest AI models are trained on over 1 trillion words, dwarfing the 4 billion English words on Wikipedia! 🎯 Quality Over Quantity But it's not just about having the most data; it's about having the right data. Models perform significantly better when trained on high-quality, specialized datasets. So while AI models are gobbling up data like Pac-Man, there's a clear hierarchy on the menu. Long-form, factually accurate, and well-written content is the gourmet meal for these systems. Specialized data allows for fine-tuning, making AI models more effective for niche applications. 🚧 Challenges Ahead With demand for data scaling up, copyright battles are flaring up and companies that own vast data troves are becoming gatekeepers, dictating terms and raising the costs for access. For example, Adobe, which owns a treasure-trove of stock images, has an advantage in image creation AI. The lay of the land is changing, and fast. 🔄 The Data Flywheel Effect Companies are improving data quality through user interactions. Feedback mechanisms are increasingly built into AI tools, creating a “data flywheel” effect. As users give thumbs-up or thumbs-down, that information becomes a new layer of data, enriching the AI model's understanding and performance. 🔒 Unlocking Corporate Data Beyond public datasets, a goldmine lies within corporate walls. Think customer spending records, call-center transcripts, and more. However, this data is often unstructured and fragmented across systems. Businesses now have the opportunity, and frankly the imperative, to organize these data silos. Not only would this amplify their own AI capabilities but also add a crucial source to the broader data ecosystem. 🛠 The Road Ahead The narrative is clear: for AI to reach its fullest potential, data sourcing, quality, and management can't be afterthoughts; they are central to the plot. As AI continues to stretch its capabilities, the race for data isn't slowing down. It's not just about finding the data; it's about cultivating it, refining it, and recognizing its true value in the grand scheme of AI development. #AI #DataQuality #Innovation #DataManagement #AIandData

At the beginning of the year, one of my predictions for 2024 was this: A shift of attention towards restrictions on international data transfers, this time around because of their impact on AI development. "🔷 There is an undercurrent that might be moving below surface for the time being but that might become very relevant in our space in upcoming years: international data transfers rules seen as a tool to leverage data for the development of local AI providers and systems, and a barrier to fuel the development of AI technology elsewhere. 🔷Think about it. To everyone’s amazement, the US government decided through its Trade Representative last year to drop its demands for the free flow of data at the WTO, reversing a longstanding position. France has stubbornly pushed in the past couple of years for various strict data localization initiatives under the label of “data sovereignty”. France also threw the loudest opposition to regulating Generative AI in the EU AI Act, openly defending its local start-up champions in the space (like Mistral). 🔷 The EU included restrictions of international transfers of non-personal data in the Data Act, which just became applicable, and in the Data Governance Act. These clearly cannot be justified as stemming from concerns with the level of protection of personal data outside of the EU. 🔷 Might it be that in the race to grow AI champions, some jurisdictions are realizing that creating barriers to data produced within their digital realm will give them an advantage, especially if, like in the US, the sheer amount and variety of available data is outstanding? Ultimately, as was highlighted in Lazard’s latest Geopolitics of Artificial Intelligence Report, data is one of the four key bottlenecks for AI development, alongside computing power, talent and physical infrastructure, and all four of them will be increasingly weaponized in the AI race. So will the dust settling on international data transfers requirements in all of the data protection laws of the world after the last Schrems episode start being cleaned?" I wrote in January in my newsletter for the FPF Global Privacy community, and later published on LinkedIn (https://lnkd.in/gJCTNHww) I thought this was a wild card prediction, because we were just starting to see less stress to global data flows after a decade of localization and further restrictions (think the DPDPA moving pass the localization requirements, China relaxing its transfers regime, DFFT initiatives). BUT it was one of the fastest predictions that started to manifest 😅 . While not entirely motivated by such undercurrents, see this in yesterday's EO of the White House creating restrictions on the transfer of some personal data of Americans outside the US. More to come?

🚨 The AI Revolution Is Over—Data Shortages Are Killing LLM Innovation! 🚨 As someone deeply invested in the evolution of artificial intelligence, I’ve been closely tracking a trend that’s about to reshape everything: the world is running out of high-quality data to fuel Large Language Models (LLMs). In my latest video, I break down why this shortage could mark the end of the explosive growth era for AI, and what it means for our industry moving forward. 🎥 https://lnkd.in/eMswPxb3 In this episode, I discuss: Why training data is the real lifeblood of AI innovation How regulatory, copyright, and ethical challenges are impacting access to crucial datasets Why simply scaling up models doesn’t work without new data The new frontiers: smarter algorithms, synthetic data, and the legal battles reshaping AI’s future We’re entering a new phase—one where creativity, efficiency, and data stewardship matter more than ever. I invite you to watch the video and join the conversation. How do you see the industry adapting to these challenges? What innovative solutions are you seeing in your organization or network? Let’s tackle this next phase of AI together. #AI #MachineLearning #LLM #DataScience #ArtificialIntelligence #TechLeadership #Innovation

The AI industry is approaching a potential bottleneck due to a finite amount of high-quality public data online, which thus far, has been crucial in training increasingly powerful models. Companies like OpenAI and Google are exhausting the internet's data reserves, necessitating the search for new data sources. For a deeper dive, check out: https://lnkd.in/eA5w77Et   To continue advancing the performance of these models while addressing data scarcity, chip shortages, and power limitations, tech companies may consider: 🤖 Creating synthetic data from models 📺 Collecting transcripts from videos (oh, hi! YouTube) 👩💻 Improving data selection and curation methods 💡 Developing novel and less data-hungry training methods   Wherever there is a challenge, there is an opportunity - even for companies in other industries. Many enterprises own an abundance of rich data that they can either monetize (if not sensitive) or use to fine-tune models in ways that large tech companies cannot.   Eyes peeled, everyone. The next act of AI ingenuity is just unfolding… Would you like me to unpack each of the 4 approaches listed above? 👀