Every AI workflow needs a verification layer

Testing out how good AI is at data protection

I have written previously about how AI will impact knowledge work. Simply put, AI makes the generation of content and knowledge much quicker and cheaper such that there may be less reliance on humans to do this sort of work.

But what AI does not change is the importance of taste and judgment - even if you can produce a document quickly, you still need to know whether that document contains everything it needs to contain and what you need to do with it.

In other words, you still need humans with subject-matter expertise to direct, verify and implement the outputs generated by AI systems.

I wanted to demonstrate this with a small test on some of the popular AI systems. Given that I work in tech governance, the test consisted of providing fictional scenarios with data protection issues and asking the AI system to address those issues.

ChatGPT was the first model I tested. I used the free version (though I am not sure what specific model this uses, this is not really apparent anymore, so it was probably some variant of GPT-5), with memory switched off and I also created a new chat for each scenario.

The main takeaway from this exercise is that AI is quite good at some things, but not everything. And if you use the general-purpose, out-of-the-box systems like ChatGPT for tasks like legal research, do so at your own risk.