With the AI boom and hype comes many AI systems. Some are built for general-purpose use - household names like ChatGPT and Claude may come to mind. Others are designed for more specific use cases, like RobinAI (for contract review) or Cursor AI (for writing code).
The plethora of systems available provide plenty of opportunities for organisations to innovate. AI can help produce new products or services or improve existing processes for building products or services.
Such opportunity still exists despite organisations needing to resort to the use of systems built by others. It might sometimes be preferable to develop your own system specifically built for your own use case with your own data, giving you a wide range of customisability. However, these systems built by the likes of OpenAI and Anthropic still come with plenty of flexibility. Their models are general-purpose which can be built on top of, fine-tuned or engineered with context and other tools to construct systems for a range of domains.
But with this opportunity comes risk, and these risks are just not limited to those which are legal in nature. AI development is itself an empirical science, whereby assessing behaviour and performance can only really be done by using models and monitoring them post-deployment. Models are often characterised as black-boxes possessing a level of complexity that renders them opaque and difficult to control. This can mean that AI systems risk failing to meet important business and legal requirements, with potentially significant consequences; poor ROI, user complaints and even regulatory intervention and legal action.
And so with these risks come responsibility. As Ethan Mollick notes in his book Co-Intelligence: Living and Working with AI, with AI becoming increasingly capable, “we’ll need to grapple with the awe and excitement of living with increasingly powerful alien co-intelligence.”1
For organisations procuring AI systems, this responsibility comes in the form of being aware of what you buy. In the world of AI, the principle caveat emptor (’buyer beware’) is crucial.
The procurement process
When procuring AI systems, organisations need to think about three main things:
What are the requirements that the AI system needs to meet?
How well does the AI system meet these requirements?
What measures can be put in place to ensure that the AI system meets the requirements?
These questions map to certain steps one must undertake before procuring an AI system:
Step 1 - Determine the requirements for the system
Step 2 - Assess the level of compliance of the system
Step 3 - Identify the right mitigation measures
Determining system requirements
To figure out the requirements that the system needs to meet, organisations need to think about the task(s) the system will be used for.
This is the essential starting point, even before thinking about what the applicable legal requirements are. Organisations need to know exactly what they want their system to do.
Requirements engineering involves determining both functional and non-functional requirements.
Functional requirements are about the specific functions that the system should execute. For example, if organisations are context engineering an AI system for a particular task, the steps required to complete the task will be contained in detailed prompts. Those prompts will instruct the information or documents required, the tools that need to be used and the outputs that need to be produced.
Non-functional requirements are about properties that the system should have. These type of requirements might be common across different AI systems, as they are general requirements that should be met in every case. This is where the legal and policy requirements come into play.
For AI systems, non-functional requirements could be derived from various AI governance frameworks or standards. For example, NIST’s AI Risk Management Framework provides a handy set of characteristics of trustworthy AI systems, which would require the system to be:
Valid and reliable
Safe
Secure and resilient
Accountable and transparent
Explainable and interpretable
Privacy-enhanced
Fair with harmful bias managed
When it comes to legal requirements, one area that organisations will often need to consider is data protection and privacy law. The GDPR is considered the ‘gold standard’ in this regard, and organisations may resort to using this as a benchmark when the deployment of the system will be global in scope and therefore involve the processing of personal data of people in the EU.
For AI procurement, the relevant GDPR requirements will be those in relation to ‘processors’, namely those entities that will be processing personal data on behalf of the organisation (the ‘controller’). There are two main questions here:
Will personal data be used? Under the GDPR, personal data means any information that can be used to identify an individual. If the task(s) that the system will complete involves the use of personal data, then the GDPR will be relevant to its use.
Will the AI system provider be processing personal data? This will typically be the case if the system is being accessed through an API or a chat interface. Such modalities will involve data being transmitted to the provider’s servers and therefore processing any personal data shared.
If the answer to both of the above questions is yes, then organisations will want the AI system provider to show that they are compliant with the GDPR. Forming part of the system requirements will therefore be data protection; requirements for ensuring that personal data are used in a responsible and ethical manner.
Additionally, if deploying the system on a global scale, the EU AI Act may also be of relevance. This legislation applies to a range of AI systems, and also what it calls ‘general purpose AI models’. These include LLMs like GPT-5 or Sonnet 4.5. If building on top of or using these models, then organisations will want to make sure that they meet the standards of the AI Act. This will also therefore form part of the requirements for the system.
Assessing compliance
The biggest hurdle with assessing compliance is visibility - organisations procuring AI will not have visibility of the whole system lifecycle.
What do I mean by this? Systems go through a lifecycle of development. That lifecycle will generally involve the designing, building, testing, deployment and maintenance of the system.
Organisations using a deployed system built by others are accessing the system as it exists at a certain point in time in that lifecycle. Consequently, there are certain aspects that user organisations will be able to see, which includes:
How the model has been deployed (e.g., API or chat interface)
The inputs it accepts (e.g., text or images)
How the system processes the input (e.g., LLM-based system processing inputs using tools or other resources)
The output produced by the system (e.g., a document)
Information on these visible aspects can mostly be found via publicly-available information. This includes model or system cards, API docs, terms of use, privacy policies and customer agreements. This is in addition to simply using system itself and observing how it works.
Less visible are the aspects of the system that do not directly form part of the user experience but nevertheless contribute to it. These include:
Training data
Model training and testing
Post-deployment monitoring (related to model performance and content moderation)
Model retraining
So how do organisations deal with these visibility constraints when assessing a third-party AI system?
Organisations can start by building templates out of their system requirements. These templates should cover each system requirement (functional and non-functional), what the requirement is about, how well the AI system meets the requirement and the evidence for this. Organisations can then use the publicly-available information for the AI system to fill out these templates as much as possible.
Anything that cannot be addressed, or needs further information or clarification, will require direct interaction with the AI system provider. This will likely be the case regarding aspects of the system that are less visible to end-users. The level of transparency given by providers here can certainly vary, since the information sought may touch on intellectual property or trade secrets. But some engagement might be better than no engagement.
Mitigation measures
Mitigation measures are about things organisations can do to help the AI system better meet the system requirements. They address the gaps identified in the assessment of the system.
These measures can be technical or organisational in nature. Technical measures are those that can be implemented into the AI system directly, such as toggling controls to ensure that input data are not used by the provider for retraining the model. Organisational measures are those that govern the use of the system, such as contractual clauses that require the data to only be processed for certain purposes (e.g., to deliver the service or for content safety).
The measures implemented for procured AI system will clearly depend on how well it meets the system requirements and where the gaps are. But the general aim of the measures will be to help ensure that the AI system operates as desired for the particular use case.
Other considerations
Beyond the implementation of mitigation measures, organisations may also want to consider further steps for effective AI procurement. Examples of this could include:
An AI inventory. Document and track the AI systems in use, what they are being used for and who is responsible for their use within the organisation.
Gathering Feedback. Constantly collect information on how well the AI system is working, and use this to evidence continued compliance with the relevant requirements.
Some may ask why all of these measures and processes are necessary? Why carry out AI procurement in this way?
Yes, some of these steps may seem cumbersome, maybe even bureaucratic. But that does not take away from their necessity, from both a business as well as a legal perspective.
The truth is, the earlier that governance is addressed, the better it is in the long-run. Organisations bother with governance because:
...by taking governance head on, and early on, it presents another means for longevity and outlasting the competition. When governance is built in, and not just an add-on, it puts you ahead of others that start thinking about it too late.
Do governance because it is hard. The greatest things are often the hardest to achieve.
Ethan Mollick, Co-Intelligence: Living and Working with AI (WH Allen 2024), p.62.