What you should never share with an AI chatbot

*ESET Press Release

AI chatbots, such as ChatGPT, Claude, and Copilot, are becoming increasingly popular. But how much should they really know about us?

Many users treat AI chatbots as friends or trusted confidants from whom they can seek advice. In reality, however, they aren’t human, nor is it always clear how they handle the information we share with them.

So, before seeking advice on mental health, medical, legal, or other personal matters, think carefully about the potential consequences.

AI chatbots remember more than you think

They usually store the information you provide so that it can be:

• be retrieved by you at a later time,
• be retained as part of a broader context of information about you or your work (memory),
• be reviewed by the platform provider to identify violations of the Terms of Service (ToS) and to improve security,
• be used to improve services and develop future versions of the models, among other things.

Not every prompt is automatically used to train the models; automatically used to train the models. The relevant practices vary depending on the platform, privacy policy, and account settings.

In addition, the information you share may be exposed due to security incidents on the provider’s side, through shared chat links or through integrations that forward prompts to third-party services.

Some offer zero data retention (ZDR) and commit to not using user data to train their models. Before choosing a specific platform or model, it’s important to review the provider’s data management policies and check the available privacy settings so you know how your information is collected, stored, and used.

Five Things You Should Never Share

Although there are platforms that emphasize privacy protection and digital data sovereignty, users should remain cautious. Company policies, terms of use, and priorities may change, meaning that the data you share today could be used in the future to train AI systems.

Here are some categories of information that are best kept private when using AI tools.

1. Passwords and credentials

This category includes usernames, passwords, recovery codes for multi-factor authentication (MFA), as well as answers to security questions used to recover or access accounts.

The ability of an AI chatbot to retain information in its memory may seem convenient. However, storing or sharing credentials through such tools poses serious security risks. If this information is exposed—for example, through improper data handling or unauthorized access—it could be used to compromise accounts

2. Sensitive Personally Identifiable Information (SPII)

This includes identification and passport information, home addresses, and financial data.

Some of this information may be provided to an AI service provider as part of a transaction, such as when paying for a subscription. In such cases, the handling of this information may be subject to specific security and regulatory compliance requirements. However, this does not mean it is equally safe to share it with a chatbot.

If they fall into the wrong hands, they could facilitate identity theft or other forms of fraud.

3. Confidential work-related information

This category includes internal reports, customer data, product designs, source code, and other non-public business information.

Using AI to summarize a confidential document may seem like an effective way to save time. However, if the tool is not authorized to handle such information or is not covered by the relevant agreement between your employer and the provider, uploading the document may violate the company’s internal policies, contractual obligations, or data protection requirements. In some cases, it may even need to be treated as a security incident.

A notable example is an incident reported in 2023, when Samsung employees allegedly fed ChatGPT proprietary source code, information about semiconductor testing, and notes from an internal meeting, in order to streamline daily tasks. The incident highlighted how easily a seemingly innocent workaround intended to boost productivity can lead to the transfer of confidential information outside the organization.

4. Medical and Other Highly Sensitive Personal Data

This includes medical records, test results, information about treatments, and other health-related data.

It is understandable that people may share sensitive information when seeking guidance on difficult personal situations. However, AI tools are no substitute for healthcare professionals and cannot provide the same personalized assessment and care as a doctor. Furthermore, the processing of health data is subject to specific protection requirements, while healthcare professionals are bound by specific legal and ethical obligations regarding medical confidentiality. For this reason, it is best to avoid sharing identifiable medical information with AI tools unless you have confirmed that their use is appropriate and safe.

5. Information Concerning Other Individuals

This category includes colleagues’ information, client records, private conversations, contact information, and other personal information about third parties.

Privacy protection applies not only to our own data but also to that of others. Sharing third-party information without proper authorization or, where required, their consent may violate ethical principles, contractual obligations, or even the law. At the same time, it may expose those individuals to risks if their information is disclosed or used in an unforeseen manner.

A relevant example occurred in 2025, when thousands of ChatGPT conversations that had been shared via public links appeared in search engine results. Some of these contained personal or professional information. The conversations had not been leaked from private accounts; however, the incident highlighted how a seemingly simple sharing choice can make a conversation much more public than users had realized.

Furthermore, keep in mind that a prompt isn’t limited to just the text you type. The documents you upload, screenshots, images, voice recordings, as well as information gathered through connected apps, may also contain sensitive data. Some of this information may be hidden in metadata, comments, or other content that isn’t immediately visible. All of this information may be accessible to and utilized by AI.

Think before you submit a prompt

Before sharing any information, ask yourself:

• Is this public information?
• Would I be willing to share it in a public forum?
• Do I have the necessary authorization to share it?
• Would I feel comfortable if this information were disclosed in the event of a data breach?

If the answer to any of these questions is no, do not enter the information into a chatbot.

Private conversations aren’t necessarily private

Private, incognito, or ephemeral chat features can limit the exposure of your information, for example, by keeping a conversation out of your visible history or excluding it from model training. However, the term “private” can create a misleading sense of security. The provider may retain conversations for a limited period of time in order to detect abuse, investigate security issues, or comply with legal obligations.

Furthermore, these features do not necessarily ensure the protection of information transmitted through plugins, connected apps, shared links, or other third-party services.

Use AI Safely

If, despite this, you still want to share information with the AI chatbot, you can craft your prompt in a way that minimizes the risks:

• Remove names and other identifying information.
• Use fictional or anonymized examples instead of real cases or incidents.
• Summarize sensitive documents yourself before asking the AI for help.
• Use approved corporate AI tools with appropriate data protection safeguards, if available in your workplace.

Just because a tool is approved doesn’t mean you can use it without restrictions. For added security, check which categories of classified data are permitted to be processed through the tool, whether features such as chat history and cache are enabled, as well as whether external plugins, links, or web search functions might transfer information outside the environment controlled by your organization.

Also keep in mind that an intelligent chatbot can extract information that is personally identifiable to you by analyzing your prompt history, even when you have anonymized the data you enter. Furthermore, deleting a conversation does not necessarily guarantee that all related data or copies of it have been deleted. Although deletion may limit the exposure of the information, it should not be considered proof that every trace of it has been eliminated.

For this reason, it is important to be aware of data retention policies and how each provider manages the information it collects, including its potential use for training AI models.

If you discover that you have shared sensitive information, take immediate action, depending on the nature and severity of the exposure. If passwords, API keys, or other credentials have been compromised, change or revoke them without delay. Delete the relevant conversation and clear any cached data, where possible. If your work data, customer data, or third-party data has been exposed, report the incident through your organization’s appropriate information security or data protection channel.

Users can reinforce these practices with the help of specialized security solutions for AI usage. AI security capabilities from trusted providers can analyze prompts and responses in real time, helping to prevent the sharing of sensitive information on public chatbots. At the same time, they help identify malicious links, scripts, and other potentially dangerous content.

Conclusion: Treat AI as a public forum

AI can boost productivity and creativity. However, users should exercise the same caution they would when sharing information with someone they know. In the age of AI chatbots, knowing what information to keep to yourself has become a fundamental digital skill. AI chatbots are suitable for general, everyday conversations, but if you want to protect your privacy, it’s best to keep sensitive information to yourself.

Frequently Asked Questions: Using AI Chatbots

Are conversations with AI chatbots private?

Not necessarily. How conversations are stored, monitored, or used varies depending on the provider, the product, the account type, and the privacy settings. Treat AI chatbots aimed at the general public as third-party services rather than confidential communication channels. Before sharing sensitive information, check the provider’s current policies regarding data handling.

Does deleting a conversation permanently delete the data?

Not necessarily. Deleting a conversation usually removes it from your visible history, but it may not immediately delete all retained copies or operational logs. Data retention periods and related exceptions vary by service. For more information, consult the provider’s privacy policy and guidelines regarding data retention.

Is anonymizing the information enough to make a prompt secure?

Anonymization reduces the risk, but it does not eliminate it. The combination of individual details, the broader context, or information from previous prompts may reveal the identity of a person or organization. Remove unnecessary details, use fictional examples whenever possible, and never submit information you are not authorized to share.

Are enterprise AI tools safer for work-related information?

Approved enterprise AI tools may provide stronger contractual guarantees, administrative controls, and restrictions on the use of customer data for model training. However, employees must still adhere to their organization’s policies regarding confidentiality and data use. They should only share information that the specific tool is authorized to process.

What should I do if I’ve shared sensitive information?

Delete the conversation, if possible; clear any cached data; and revoke access for connected apps, if applicable to your situation. If you have exposed passwords, recovery codes, or other credentials, change or revoke them immediately. If company, customer, or third-party data is involved, report the incident to your employer or the organization’s security team, rather than attempting to handle it on your own.