How safe is your AI conversation? What CIOs must know about privacy risks

In a recent podcast appearance on This Past Weekend with Theo Von, Sam Altman, CEO of OpenAI, dropped a bombshell that’s reverberating across boardrooms and IT departments: Conversations with ChatGPT lack the legal protections afforded to discussions with doctors, lawyers or therapists.

This revelation underscores a critical gap in privacy law and raises urgent questions about how organizations can responsibly integrate AI while safeguarding user data. For CIOs and C-suite leaders, Altman’s warning serves as a wake-up call to strike a balance between innovation and robust privacy, compliance and governance frameworks. Here’s what business leaders need to focus on to stay compliant and ahead of the curve in this rapidly evolving AI landscape. 

The privacy gap in AI conversations 

Altman highlighted that users, particularly younger demographics, are increasingly turning to ChatGPT for sensitive advice, treating it as a substitute for a therapist or life coach. However, unlike professional consultations protected by legal privileges, these AI interactions are not confidential. In legal proceedings, OpenAI could be compelled to disclose user conversations, exposing deeply personal information. This issue is compounded by OpenAI’s data retention policies, which allow chats to be stored for up to 30 days (or longer for legal and security reasons), posing risks to user privacy in cases like the ongoing lawsuit with The New York Times. 

This lack of legal privilege for AI interactions isn’t just a user concern; it’s a corporate one. Organizations leveraging AI tools like ChatGPT for customer service, employee support or decision-making could inadvertently expose sensitive data to legal discovery or government subpoenas. As AI becomes ubiquitous, the absence of a clear regulatory framework creates a minefield for businesses striving to maintain trust and compliance. 

Implications for privacy law and the future of AI 

Altman’s comments signal a broader challenge: privacy laws haven’t kept pace with AI’s rapid adoption. Traditional frameworks, such as doctor-patient confidentiality, don’t apply to AI, leaving a regulatory void. This gap could erode user trust, slow AI adoption and invite stricter regulations. In Europe, for instance, the EU is debating whether to classify general-purpose AI as “high risk,” which could impose stringent oversight (a move opposed by tech giants like OpenAI and Microsoft for fear of stifling innovation). 

For the future of AI, this privacy issue is a double-edged sword. On one hand, addressing it could unlock greater user confidence and drive adoption in sensitive areas such as mental health or HR support. On the other hand, failure to act could lead to public backlash, legal battles or fragmented global regulations that complicate the deployment of AI. Altman himself advocates for extending therapist-like privacy protections to AI conversations, a sentiment echoed by policymakers with whom he has consulted. The question is how quickly such frameworks can be implemented without hampering innovation. 

What CIOs and C-suite leaders must do 

To navigate this uncharted territory, business leaders must adopt a proactive, multi-faceted approach that balances privacy, compliance, governance and innovation. Here are the key areas of focus: 

1. Prioritize data governance and transparency 

CIOs must ensure robust data governance policies that clarify how AI tools handle sensitive information. This starts with understanding the data practices of AI vendors like OpenAI. For instance, OpenAI’s privacy policy permits data sharing with third parties for legal compliance purposes, and chats may be reviewed to improve models or detect misuse. Leaders should: 

• Audit AI tools: Assess the data retention, encryption and sharing policies of AI platforms used in your organization. Ensure vendors align with your compliance requirements, such as GDPR or privacy acts. 
• Implement clear policies: Establish internal guidelines on what types of data employees can input into AI tools. Prohibit sharing sensitive personal or corporate data unless the tool guarantees end-to-end encryption or equivalent protections. 
• Communicate risks: Educate employees and customers about the lack of legal privilege in AI interactions, encouraging discretion when using tools like ChatGPT. 

2. Strengthen compliance frameworks 

With no AI-specific privacy laws in place, businesses must lean on existing regulations while preparing for future ones. The EU’s potential “high-risk” classification and US agencies’ crackdowns on harmful AI products signal a tightening regulatory landscape. To stay compliant: 

• Map regulatory requirements: Align AI usage with existing privacy laws like GDPR, HIPAA or industry-specific standards. For example, healthcare organizations using AI for patient engagement must ensure compliance with HIPAA’s data protection rules. 
• Monitor legal developments: Stay informed about emerging AI regulations, such as the EU’s AI Act or US proposals for an AI licensing agency. Engage with industry groups to influence policy in a way that balances innovation and privacy. 
• Prepare for legal discovery: Assume AI-generated data could be subpoenaed. Work with legal teams to minimize exposure by limiting data collection and using secure, enterprise-grade AI solutions with stronger privacy controls. 

3. Invest in secure AI solutions 

Not all AI tools are created equal. Enterprise-grade solutions, such as ChatGPT Enterprise, often offer enhanced privacy features, including data isolation and shorter retention periods, compared to their free-tier versions. CIOs should: 

• Opt for enterprise AI: Deploy AI tools designed for business use, which typically include better security and compliance features. For example, OpenAI’s enterprise offerings exclude user data from training models, reducing privacy risks. 
• Explore open-source alternatives: Open-source AI models, like Qwen3-Coder, may offer greater transparency and control over data handling, though they require in-house expertise to manage. 
• Enhance security protocols: Integrate AI tools with existing cybersecurity frameworks, ensuring data encryption, access controls and regular audits to prevent unauthorized access. 

4. Foster responsible innovation 

Innovation doesn’t have to come at the expense of privacy. Altman’s vision of AI as a “life advisor” highlights its transformative potential, but over-reliance or unchecked deployment could backfire. To innovate responsibly: 

• Pilot AI use cases: Test AI applications in low-risk areas before scaling to sensitive functions like HR or customer support. This allows you to assess privacy and performance without exposing critical data. 
• Leverage AI for governance: Use AI to monitor compliance, detect data misuse or anonymize sensitive information, turning the technology into a tool for privacy protection. 
• Engage stakeholders: Collaborate with legal, HR and compliance teams to ensure AI initiatives align with organizational values and regulatory expectations. 

5. Build a culture of trust 

Trust is the currency of AI adoption. Altman’s warning about users’ over-reliance on ChatGPT underscores the need to manage expectations and foster transparency. Leaders should: 

• Set realistic expectations: Educate stakeholders about AI’s limitations, such as its tendency to “hallucinate” or generate unreliable outputs, to prevent blind trust. 
• Champion ethical AI: Publicly commit to ethical AI use, emphasising privacy and accountability. This can differentiate your organization in a competitive market. 
• Engage with regulators: Advocate for clear, balanced AI privacy laws that protect users without stifling innovation, as Altman has done in congressional hearings. 

The road ahead 

Altman’s candid admission about AI’s privacy shortcomings is a call to action for CIOs and C-suite leaders. The absence of legal protections for AI conversations is a stark reminder that technology is outpacing regulation; businesses must bridge this gap. By prioritising data governance, compliance, secure solutions, responsible innovation and trust, organizations can harness the potential of AI while mitigating risks. 

The future of AI depends on getting this balance right. As Altman noted, no one had to think about AI privacy a year ago, but now it’s a critical issue. For business leaders, the challenge is clear: act now to embed privacy and governance into your AI strategy, or risk falling behind in a world where trust and compliance are non-negotiable.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?