As soon as customer data goes into an AI system, the question comes up: is this actually allowed. Often it is. But the GDPR existed long before anyone talked about AI, and those rules apply in full. In the eyes of the law, an AI system is simply processing of personal data, with a few extra points to watch.
The same four questions as always
For every processing operation, with or without AI, you must be able to answer four things. If you cannot, the problem is not the AI.
- For what? A concrete purpose. “Producing quotes faster” is a purpose. “Keeping everything just in case” is not.
- On what legal basis? Usually that is the performance of a contract or a legitimate interest. In a business context, consent is often the weakest choice, especially with your own staff.
- How much? Only the data you need for that purpose. A system that forecasts delivery times does not need dates of birth.
- For how long? A retention period that fits the purpose, and that you actually enforce.
Where AI is slightly different
There are three points that come up specifically with AI. The first is the processor. If you use an AI service, your data passes through their systems. That requires a data processing agreement, and you want to know where the processing takes place and whether your input is used to train their models. Business subscriptions generally exclude that training, free versions often do not. That is a difference you can check on your own invoice.
The second is automated decision-making. If a system makes a decision with legal or similarly significant effects without human intervention, think of a credit refusal or the rejection of a job applicant, stricter requirements apply and in principle a person must be able to step in. For a system that drafts a quote which a colleague then checks, this does not come into play.
The third is explainability. Data subjects have a right of access and a right to understandable information about what happens to their data. With an AI system that pulls data from ten places, that is harder to answer if you have not recorded it in advance.
The most common mistake at SMEs is not legal but practical: employees pasting customer data or entire files into a free AI tool because it is more convenient. That is a data breach in the making, and it happens out of your sight. A short working agreement plus a well-functioning business alternative solves this faster than a ban.
Practical measures that really help
- Set up one business account with a data processing agreement, instead of five employees each with a free account.
- Work with as little data as possible in your integrations. An agent that matches invoices needs a customer number and an amount, not a complete customer file.
- Anonymise where you can. For recognising patterns in orders you rarely need names.
- Log what the system does, with which data and at whose request. You need that for an access request and in the event of an incident.
- Include your AI applications in your record of processing activities. You have to keep that record anyway.
- Agree who is in the loop for decisions that affect a customer or employee, and build that into the process itself, not only on paper.
What disappoints here
Two things. The first is that “the data stays in Europe” is less clear-cut than it sounds. Even with European storage, support or administration can take place from elsewhere. Ask your supplier to put it down in concrete terms instead of settling for a reassuring sentence on a product page.
The second is that privacy often costs functionality. Passing on less data sometimes means poorer answers. That is a real trade-off and not a matter of clever building. We prefer to show that choice explicitly rather than quietly make it for you.
If you cannot explain which data your system uses and why, you cannot justify it in the event of a complaint either.
When you need us and when you do not
For the basics you do not need us. A business account with a data processing agreement, a clear working agreement and updating your record of processing activities: you can do that yourself, or with the lawyer you already have. We are not a law firm and we do not pretend to be.
We are useful in the building itself: determining which data does and does not pass through an integration, setting up logging, and making sure human oversight sits in the process rather than in a manual. If you are in a sector with special categories of personal data, such as healthcare, a lawyer should be involved alongside us in any case.