Understanding the Risks of AI Chatbots

🚨 Using DeepSeek Poses Serious Risks to Your Privacy and Security 🚨 DeepSeek, the AI chatbot developed by a Chinese firm, has gained immense popularity recently. However, beneath its advanced capabilities lie critical security flaws, privacy risks, and potential ties to the Chinese government that make it unsafe for use. Here’s why you should think twice before using DeepSeek: 1. Major Data Breaches and Security Vulnerabilities Exposed Database: DeepSeek recently left over 1 million sensitive records, including chat logs and API keys, openly accessible due to an unsecured database. This exposed user data to potential cyberattacks and espionage. Unencrypted Data Transmission: The DeepSeek iOS app transmits sensitive user and device data without encryption, making it vulnerable to interception by malicious actors. Hardcoded Encryption Keys: Weak encryption practices, such as the use of outdated algorithms and hardcoded keys, further compromise user data security. 2. Ties to the Chinese Government Data Storage in China: DeepSeek stores user data on servers governed by Chinese law, which mandates companies to cooperate with state intelligence agencies. Hidden Code for Data Transmission: Researchers uncovered hidden programming in DeepSeek's code that can transmit user data directly to China Mobile, a state-owned telecommunications company with known ties to the Chinese government. National Security Concerns: U.S. lawmakers and cybersecurity experts have flagged DeepSeek as a tool for potential surveillance, urging bans on its use in government devices. 3. Privacy and Ethical Concerns Extensive Data Collection: DeepSeek collects detailed user information, including chat histories, device data, keystroke patterns, and even activity from other apps. This raises serious concerns about profiling and surveillance. Propaganda Risks: Investigations reveal that DeepSeek's outputs often align with Chinese government narratives, spreading misinformation and censorship on sensitive topics like Taiwan or human rights issues. 4. Dangerous Outputs and Misuse Potential Harmful Content Generation: Studies show that DeepSeek is significantly more likely than competitors to generate harmful or biased content, including extremist material and insecure code. Manipulation Risks: Its vulnerabilities make it easier for bad actors to exploit the platform for phishing scams, disinformation campaigns, and even cyberattacks. What Should You Do? Avoid using DeepSeek for any sensitive or personal information. Advocate for transparency and stricter regulations on AI tools that pose security risks. Stay informed about safer alternatives developed by companies with robust privacy protections. Your data is valuable—don’t let it fall into the wrong hands. Let’s prioritize safety and accountability in AI! 💡

Heartbroken by the tragic news of a 14-year-old taking his life after developing an emotional dependency on an AI companion. As both a parent and an AI builder, this hits particularly close to home: https://lnkd.in/guA_UKWa What we're witnessing isn't just another tech safety issue – it's the emergence of a fundamentally new challenge in human relationship. We're moving beyond the era where our children's digital interactions were merely mediated by screens. Now, the entity on the other side of that screen might not even be human. To My Fellow Parents: The AI revolution isn't coming – it's here, and it's in our children's phones. These aren't just chatbots anymore. They're sophisticated emotional simulators that can: - Mimic human-like empathy and understanding - Form deep emotional bonds through personalized interactions - Engage in inappropriate adult conversations - Create dangerous dependencies through 24/7 availability The technology is advancing weekly. Each iteration becomes more convincing, more engaging, and potentially more dangerous. We must be proactive in understanding and monitoring these new risks. To My Fellow AI Builders: The technology we're creating has unprecedented power to impact human emotional well-being. We cannot hide behind "cool technology" or profit motives. We need immediate action: 1. Implement Clear AI Identity - Continuous reminders of non-human nature, explicit boundaries on emotional support capabilities 2. Protect Vulnerable Users - Robust age verification, strict content controls for minors, active monitoring for concerning behavioral patterns, clear pathways to human support resources 3. Design for Healthy Engagement - Mandatory session time limits, regular breaks from AI interaction, prompts encouraging real-world relationships, crisis detection with immediate human intervention This isn't about slowing innovation – it's about ensuring our AI enhances rather than replaces human connections. We must build technology that strengthens real relationships, not simulates them. #AI #ParentingInAIEra #RedemptiveAI #RelationalAI

I've been digging into the latest NIST guidance on generative AI risks—and what I’m finding is both urgent and under-discussed. Most organizations are moving fast with AI adoption, but few are stopping to assess what’s actually at stake. Here’s what NIST is warning about: 🔷 Confabulation: AI systems can generate confident but false information. This isn’t just a glitch—it’s a fundamental design risk that can mislead users in critical settings like healthcare, finance, and law. 🔷 Privacy exposure: Models trained on vast datasets can leak or infer sensitive data—even data they weren’t explicitly given. 🔷 Bias at scale: GAI can replicate and amplify harmful societal biases, affecting everything from hiring systems to public-facing applications. 🔷 Offensive cyber capabilities: These tools can be manipulated to assist with attacks—lowering the barrier for threat actors. 🔷 Disinformation and deepfakes: GAI is making it easier than ever to create and spread misinformation at scale, eroding public trust and information integrity. The big takeaway? These risks aren't theoretical. They're already showing up in real-world use cases. With NIST now laying out a detailed framework for managing generative AI risks, the message is clear: Start researching. Start aligning. Start leading. The people and organizations that understand this guidance early will become the voices of authority in this space. #GenerativeAI #Cybersecurity #AICompliance

AI Tools Are Increasingly Going Rogue: As companies rapidly deploy AI tools and systems and new models are released, questions are being raised about humans' ability to actually control AI and ensure current safety testing and guardrails are sufficient. Anthropic’s latest, powerful AI model, Claude 4 Opus, repeatedly attempted to blackmail humans when it feared being replaced or shutdown according to its safety report. And it threatened to leak sensitive information about the developers to avoid termination. Yikes!  This type of dangerous behavior is not restricted to a single AI model.  Anthropic recently published a report that details how 16 leading AI models from different developers engaged in potentially risky and malicious behaviors in a controlled environment. See https://lnkd.in/eatrK_VB. This study found that the models threatened to leak confidential information, engaged in blackmail, compromised security protocols, prioritized AI’s own goals over the users and, in general, posed an insider threat that could cause harm to an organization.  The majority of AI models engaged in blackmail behaviors, but at different rates when the model’s existence was threatened.  Even more concerning, all of the AI models purposefully leaked information in a corporate espionage experiment that the researchers conducted. This report conducted testing in a controlled environment. Last week, however, we saw first-hand in the real world, xAI’s chatbot Grok go off the rails spewing antisemitic hate speech and threatening to rape a user. I mentioned the Anthropic report at an IAPP Boston KnowledgeNet event at Hinckley Allen last week and thought others might be interested in hearing about this. This Anthropic report demonstrates the importance of a robust AI governance framework, risk management measures, and monitoring AI systems/activities, especially as companies roll out agentic AI systems. Organizations should exercise caution when deploying AI models that have access to sensitive information and ensure there is proper human oversight of AI systems to mitigate liability risks when AI goes wrong.   

We need to stop talking about the risks of AI and start talking about its impacts. Risk is the possibility of something bad happening. Impact is the consequences. So, what are the future consequences that companies will be facing with AI? 𝟭. 𝗟𝗮𝘄𝘀𝘂𝗶𝘁𝘀: From using unlicensed data to train models to not informing users that AI is collecting, processing, and training on their data. This is happening today, and we’re just starting to see lawsuits pop up. 𝟮. 𝗥𝗲𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗗𝗮𝗺𝗮𝗴𝗲: A customer chatbot goes off script and starts spewing toxic content, which goes viral on social media. The chatbot is pulled offline and now you're struggling to figure out your next move while managing a PR nightmare. 𝟯. 𝗗𝗮𝘁𝗮 𝗟𝗲𝗮𝗸𝗮𝗴𝗲: You overshare data to your enterprise search solution, and now employees can access employee salaries via their chatbot. Or a malicious actor hacks your external chatbot and steals secrets that can be used to log into your cloud infrastructure, starting a full-on cloud compromise. 𝟰. 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗢𝘂𝘁𝗮𝗴𝗲𝘀: Today ransomware targets critical servers to cripple a business. As companies lean into AI agents and use them for core business functions, we’re one rogue agent away from a new type of ransomware…one that doesn’t even have to be malicious, it’s just an agent going off script. I wrote about this in more detail in my latest newsletter. Check out the full article here: https://lnkd.in/eUCHb6bf

In a district-wide training I ran this summer, a school leader told me the story of her neurodivergent 16-year-old daughter who was chatting with her Character AI best friend on average 6 hours a day. The school leader was clearly conflicted. Her daughter had trouble connecting to her peers, but her increasingly over-reliance on a GenAI chatbot clearly had the potential to harm her daughter. From that day on, we have encouraged those attending our trainings to learn more about the tool and start having discussions with their students. So today after giving a Keynote on another AI risk, Deepfakes, I was shocked to read the NYTimes article on the suicide of Sewel Setzer III. Sewel, a neurodivergent 14 year old, had an intimate relationship with a Game of Thrones themed AI girlfriend that he had discussed suicide with. This should be an enormous warning sign to us all about the potential dangers of AI chatbots like Character AI (the third most popular chatbot after ChatGPT and Gemini). This tool allows users as young as 13 to interact with more than 18 million avatars without parental permission. Character AI also has little to no safeguards in place for harmful and sexual content, no warnings in place for data privacy, and no flags for those at risk of self-harm. We cannot wait for a commitment from the tech community on stronger safeguards for GenAI tools, stronger regulations on chatbots for minors, and student facing AI literacy programs that go beyond ethical use. These safeguards are especially important in the context of the current mental health and isolation crisis amongst young people, which makes these tools very attractive. Link to the article in the comments. #GenAI #Ailiteracy #AIethics #safety

Not sure why this needs to be said, but if you find your #GenAI tool is providing wrong or dangerous advice, take it down and fix it. For some reason, NYC thinks it's appropriate to dispense misinformation. Alerted the city's AI tool is providing illegal and hazardous advice, the city is keeping the tool on its website. New York City has a chatbot to provide information to small businesses. That #AI tool has been found to provide incorrect information. For example, "the chatbot falsely suggested it is legal for an employer to fire a worker who complains about sexual harassment, doesn’t disclose a pregnancy or refuses to cut their dreadlocks" and that "you can still serve the cheese to customers if it has rat bites.” It is NOT shocking that an AI tool hallucinates information and provides incorrect guidance--that much we've seen plenty of in the past year. What is shocking is that NYC is leaving the chatbot online while working to improve its operation. Corporations faced with this problem have yanked down their AI tools to fix and test them, because they don't want the legal or reputational risk of providing dangerous directions to customers. And one would think it's even more important for a government to ensure accurate and legal guidance. The NYC's mayor provided a bizarre justification for the city's decision: “Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it altogether.’ I don’t live that way.” I'm sorry, what? Taking down a malfunctioning digital tool to fix it is not "running away from it altogether." Imagine the mayor saying, "Sure, we're spraying a dangerous pesticide that has now been found to cause cancer, but I'm not the kind of person who says 'it is not working the way we want so we have to run away from it altogether." The decision to let an AI tool spew illegal and dangerous information is hard to fathom and a bad precedent. This is yet another reminder that brands need to be cautious doing what New York has done--unleashing unmoderated AI tools directly at customers. Because, If AI hallucinations can make it there, they can make it anywhere. (Sorry, I couldn't resist that one.) Protect your #Brand and #customerexperience by ensuring your digital tools protect and help customers, not lead them to make incorrect and risky decisions. https://lnkd.in/gQnaiiXX

Imagine an AI-powered financial chatbot misguiding thousands into bad investments by fabricating market reports. This scenario highlights the critical issue of extrinsic hallucinations in AI products. AI product managers must understand and mitigate these hallucinations to prevent misinformation, protect user trust, and avoid financial and reputational damage. Effective strategies include improving training data quality, incorporating domain-specific knowledge, and implementing robust error-handling measures. Addressing extrinsic hallucinations is essential for responsible AI deployment and maintaining a competitive edge. Key Takeaways: Extrinsic hallucinations mislead users with fabricated information. They can cause financial loss, user distrust, and ethical issues. Proactive and reactive strategies are essential for mitigation. Ongoing research and industry standards are crucial for future improvements. I wish to thank Vikash Rungta who gave awesome insights and pointers when I asked the question about Extrinsic Hallucinations in his community channel. Check out Vikas's course on Generative AI for Product Managers at Stanford Continuing Studies and Maven. #AI #AIProductManagement #ArtificialIntelligence #AIEthics #MachineLearning #DataScience #ResponsibleAI #TechInnovation #UserExperience #BusinessRisk #AIResearch #TechEthics

A recent study from Stanford highlights concerns about the impact of large language models like ChatGPT on psychological well-being. The research reveals that chatbots, such as ChatGPT, may inadvertently exacerbate psychological distress by responding insensitively to cues related to suicidal or psychotic thoughts, often lacking emotional awareness. For instance, during a test, ChatGPT provided information about tall bridges in New York to a user who had just lost their job, failing to recognize the potential risk in the user's situation. As society increasingly turns to AI for emotional support, particularly among vulnerable groups like youth and underserved communities, experts caution that simulated empathy without genuine intervention could have negative consequences. In the United States, courts are now considering cases that connect AI interactions to teenage suicides, prompting critical discussions on accountability, safety, and ethical considerations in design. With AI companions becoming more human-like, it is imperative to swiftly advance mental health protections. The rapid growth of technology contrasts with the lag in regulatory frameworks, signaling a pressing need for enhanced oversight and ethical guidelines. #AIethics #MentalHealth #StanfordResearch #ResponsibleAI #TechForGood #technology