The Role of Domain Experts in AI Projects

Building collaboration between domain science and machine learning can be an interesting challenge - not unlike (meaningfully) using ML in a new business context. In microscopy and materials community, this process have started 5-7 years ago. At that time, many of my colleagues tried collaborating with ML experts - typically picking the top expert in a specific field in their university and trying to build a partnership. Very natural pathway - but very often it didn't work out. Experts tend to focus on specific areas of ML and use the same tools for all problems. Sometimes it is a good fit, but many times it isn't. Furtnermore, very often specific problem requires most simple tools - just scaled to realistic data set sizes. At that time, the approach that worked best was teaming up with enterprising and curious ML/CS grad students with broad knowledge of several methods and willing to explore domain science areas. Another (Artem Maksov, Oleg Ovchinnikov) was to train a student to use ML over several years. Both were a practical solution. Nowadays, the advancements in ML and Python ecosystem make it much simpler - a domain scientists can learn basic ML principles and lingo, and then moving from simple tools (scikit-learn, simple DCNNs) to complex ones can be done in partnership with ML experts or using open codes on GitHub. Even basic principles and experience for adapting simple workflows for domain specific problems allow sto build solid base fro translational AI. It's fun to build such course for ML in materials! https://lnkd.in/gaYs2ji4 #MachineLearning #Collaboration #DomainScience

Having domain expertise is much more valuable than people think. Let’s say you’re building a predictive model for drug discovery. Before you begin building the model, you need to consider a few things: 1. Proteins are the building blocks of life. 2. Drug discovery, simply put, involves identifying which drugs will interact with which proteins, and how. 3. Protein activity (i.e probability to be optimized into a novel drug target) is derived from protein functionality. 4. Protein functionality is directly attributed to a protein’s 3D structure. So long before you’ve opened up your IDE, you need to whiteboard how you intend to represent a protein’s 3D structure, in a way that is recognizable to the model. In ML terms, how are you going to ‘embed’ the 3D structure into your algorithm? Meaning, how do you map the 3D spatial coordinates of atoms (high-dimensional data) into a lower-dimensional space, where it can accurately be used as input? * This is the same as text embedding for NLP models — transforming raw text into continuous vectors — to capture the semantic relationships between words * Now, I’m not saying someone without a Bioinformatics, Cheminformatics or even Molecular Biology background couldn’t execute. But, it may take them longer to formulate these considerations; worst case they may never acknowledge these considerations at all, and rush full speed into prototyping naively. So before you think of what shiny new tool you’re gonna learn next, maybe try leaning on your domain expertise a bit more. It might get you further, faster. Note: Credit to Deep Learning for the Life Sciences for inspiring this post. _______ #biotech #ai #academia #industry #careers

The deeper I delve into AI, the more clearly I see that the relative values of different skillsets are being rebalanced. This shift has particularly large implications for career transitioners and students entering the data field... Recently, I posted about how the ability to build emotional ties and trust with stakeholders will be the most critical skill of the AI-era: https://lnkd.in/efMV6tdi Similarly, I believe the value of #domainexpertise (DE) will continue to grow, as the value of "technical stack" skills declines (as #AI increasingly assumes those duties). THE COMPONENTS OF DOMAIN EXPERTISE 🔸Factual Knowledge - the terminology, definitions, and data relevant to a domain 🔸Conceptual Knowledge - the theories, models, and structures that explain how things work within the domain 🔸Procedural Knowledge - how to perform domain-specific tasks, techniques and processes 🔸Strategic (aka Metacognitive) Knowledge - how to apply these components to solve problems and make decisions 🔸Tacit Knowledge - the implicit understanding, skills, insights, intuition, etc necessary for expert performance 🔸Contextual Knowledge - the industry-specific factors, regulatory environment, market dynamics, and cultural factors that define the full context in which the domain operates 🔸 Domain-Specific Data - the data sources and metrics essential for analysis and decision-making 🔸Problem Framing - the questions and factors to consider when tackling domain-specific challenges 🔸Interpretation - the ability to translate domain analyses into actionable insights 🔸 Continuous Learning - the discipline and adaptability to keep pace w/ new domain developments, trends, and best practices WHY IS DOMAIN EXPERTISE SO CRITICAL TO AI? There are two primary ways to improve AI models - improve the underlying models themselves or train them on better data. It is DE that generates this higher quality training data. I've been working since GPT4O was released on a custom Power BI GPT that is vastly outperforming both the base 4O model, and every GPT in the GPT store that I've tested it against. This is because mine captures IMO the top 15 books related to Power BI (6,000+ total pages), as well as datasets and data models, courses, articles/blogs, video and audio transcripts, images, thousands of code solutions, etc. - fully leveraging years of experience as a CCO and trainer in this domain. In the ultra-competive business world, where every org will have access to the same base models, the advantages afforded by a superior model trained on better data will be enormous, and those who have the DE to provide that edge - in health care, finance, law, construction, logistics, IT security, public policy, etc. will be in extraordinary demand. This is why IMO #career transitioners with DE from a different sector are entering at a perfect time, and why students should orient their studies to obtaning data skills in the context of building DE in a second area.

AI cannot do what experts are doing today But army of AI + domain experts is a lethal combo We saw this first hand from an AI startup at Upekkha Story of “Grunt Work as a Service” This is the story of a startup we worked with What do they do? They manage a marketplace of experts In the beginning, they built a product Experts login & deliver their services But, then they realized that for this to become valuable They needed a marketplace to showcase these experts to the right customers Good learning But, here’s the fun part After a few conversations with experts They started to notice a pattern None of the experts wanted to do the grunt work that comes along with their service They want to stick to the the cool parts - Sharing their expertise - Real world scenarios - Practical things And wished that grunt work just magically disappeared You may have guessed it by now The startup decided to offer “Grunt Work as a Service” While this was a huge struggle in their initial days ChatGPT & LLMs made their way into the world They figured out a way to get this grunt work done with AI This was much faster & much more systematic The expert is happy because if they were on another platform They would have to do all the grunt work But this way The whole offering changed From a managed marketplace to an end to end platform Army of AI plus the experts came together Because the AI cannot do what these experts are doing Bringing in very specific, domain specific smarts A whole spectrum of things will change with AI More than any of us can even imagine I believe AI + Humans = Future of SaaS What do you think?

If (1) you are a subject matter expert in an area and (2) suspect that a problem in your area can be solved using AI but (3) don't know much about AI and (4) aren't sure where to begin, check out this nice talk by Andrew Ng, *especially 22:00 - 31:15*, where he shares concrete examples of how his AI Fund partners with domain experts to create startups. My 2 cents: Your domain knowledge and your deep understanding of the problem are *incredibly valuable* (and arguably more important than AI knowledge). Don't let your lack of knowledge of AI deter you. Try to find an AI expert to complement you. https://lnkd.in/eURbgNAu

Summer Scribbles #3.1 - Leveraging Internal Business Knowledge for Robust AI Feature Engineering With the rapid pace of AI advancement, it can be tempting for enterprises to rely on off-the-shelf models and "out of the box" feature sets. However, successful adoption of AI requires taking advantage of a key asset: internal business knowledge. Thoughtful feature engineering based on deep domain expertise and institutional experience is a critical architectural principle for scalable, accurate AI. According to a 2021 State of Enterprise Machine Learning survey, 78% of organizations say lack of high-quality training data and features is a top barrier to AI success. Dedicated feature engineering based on institutional knowledge can help overcome this challenge. The same survey found companies with extensive feature engineering maturity achieved up to 5x higher model performance. A few years back there was a debate on value of tribal knowledge and whether democratization of data sources had taken over. A recent article by Michael Katz – articulated some of the key points of why? https://lnkd.in/gDsKzPdZ Many past AI failures can be traced back to the use of generic, publicly available features without sufficient customization. While external benchmarks provide a starting point, internal data understanding is indispensable for extracting maximum signal from enterprise data. I still thought it would be useful to capture how to balance the External/Internal Dataset capabilities in your architecture. (see #3.2 for more on the recommendations..)

Our latest research published in Harvard Business Review reveals the importance of specific human capabilities for successful use of AI. We observe two categories of human skills as critical. First are effective interpersonal skills - basic conflict resolution, communication, skills of disconnecting from emotions, and even mindfulness practices. Second is domain expertise, deep knowledge of one’s environment. Rushing to replace talent with AI is a huge mistake. Competitive advantage cannot be achieved without humans in the loop. Companies should focus on #reskilling #upskilling, and preserving domain knowledge among experienced talent while developing it among young inexperienced workers. #futureofwork #artificialintelligenceforbusiness #artificialintelligence #hr #humanresourcedevelopment #talentmanagement #ai D'Amore-McKim School of Business at Northeastern University Heather Hill Polly Mitchell-Guthrie Anne Robinson John Sicard Maria Villablanca Ted English John Wood https://lnkd.in/embyjKWW https://lnkd.in/eEh2ubRn

𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗜 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗿𝗲𝗽𝗹𝗮𝗰𝗲 𝗲𝘅𝗽𝗲𝗿𝘁𝘀; 𝗶𝘁 𝗮𝗺𝗽𝗹𝗶𝗳𝗶𝗲𝘀 𝘁𝗵𝗲𝗶𝗿 𝗲𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲! 👉 It’s about harnessing AI to enhance our human capabilities, not replace them. 🙇♂️ Let me walk you through my realization. As a healthcare practitioner deeply involved in integrating AI into our systems, I've learned it's not about tech for tech's sake. It's about the synergy between human intelligence and artificial intelligence. Here’s how my perspective evolved after deploying Generative AI in various sectors: 𝐇𝐞𝐚𝐥𝐭𝐡𝐜𝐚𝐫𝐞: "I 𝐧𝐞𝐞𝐝 AI to analyze complex patient data for personalized care." - But first, we must understand the unique healthcare challenges and data intricacies. 𝐄𝐝𝐮𝐜𝐚𝐭𝐢𝐨𝐧: "I 𝐧𝐞𝐞𝐝 AI to tailor learning to each student's needs." - Yet, identifying those needs requires human insight and empathy that AI alone can't provide. 𝐀𝐫𝐭 & 𝐃𝐞𝐬𝐢𝐠𝐧: "I 𝐧𝐞𝐞𝐝 AI to push creative boundaries." - And yet, the creative spark starts with a human idea. 𝐁𝐮𝐬𝐢𝐧𝐞𝐬𝐬: "I 𝐧𝐞𝐞𝐝 AI for precise market predictions." - But truly understanding market nuances comes from human experience and intuition. The Jobs-to-be-Done are complex, and time is precious. We must focus on: ✅ Integrating AI into human-led processes. ☑ Using AI to complement, not replace, human expertise. ✅ Combining AI-generated data with human understanding for decision-making. ☑ Ensuring AI tools are user-friendly for non-tech experts. Finding the right balance is key: A. AI tools must be intuitive and supportive. B. They require human expertise to interpret and apply their output effectively. C. They must fit into the existing culture and workflows. For instance, using AI to enhance patient care requires clinicians to interpret data with a human touch. Or in education, where AI informs, but teachers inspire. 𝐌𝐚𝐭𝐜𝐡𝐢𝐧𝐠 𝐀𝐈 𝐰𝐢𝐭𝐡 𝐭𝐡𝐞 𝐫𝐢𝐠𝐡𝐭 𝐫𝐨𝐥𝐞𝐬 is critical. And that’s where I come in. 👋 I'm Umer kHan, here to help you navigate the integration of Generative AI into your world, ensuring it's done with human insight at the forefront. Let's collaborate to create solutions where technology meets humanity. 👇 Feel free to reach out for a human-AI strategy session. #GenerativeAI #HealthcareInnovation #PersonalizedEducation #CreativeSynergy #BusinessIntelligence

With the latest breakthroughs from OpenAI this week, the future of work is once again thrust into the spotlight. The #TalentIntelligence machine inside my head is thinking about the number of functions that will radically shift in the next year. A tactical example: I believe a unique category that will embrace the shift is the fractional executive ecosystem. Experts who have decades of experience and know subject areas forward and backward (like me) will likely consider being augmented by AI. Imagine capturing the essence of your expertise and distilling it into a process that works tirelessly for you. How would you begin to document the intricate steps of an expert workflow? It's not just about the tasks you perform; it's about the decision trees you navigate daily without a second thought. The actions that you take subconsciously are the ones that others need your (or your AI helper’s) assistance. Consider the notable forks in your process - those critical junctures where your experience guides you to one path over another. What if you could map these out clearly for an AI to follow? Optimizing for different use cases is another layer of sophistication. Your workflow isn't one-size-fits-all; it's a tailored suit that adjusts to the occasion. How would you teach an AI to understand these nuances? As an expert you know the ten wrong answers for every right solution. Then there's the data - the lifeblood of any AI system. What data would you feed into this intelligent machine? More importantly, what do you expect out of it? The input/output relationship is a delicate dance that requires a deep understanding of your goals. Let's talk about the art of transforming your professional acumen into an automated powerhouse. How would you approach it? What tools would you use? What challenges do you foresee? I'm curious to hear your thoughts on this exciting frontier of AI in the workplace. Drop your insights below or DM me to exchange ideas! #ArtificialIntelligence #FutureOfWork #Innovation