TL;DR
This newsletter is about the SCHUFA case on automated individual decision-making under the GDPR. It looks at the facts of the case, the court verdict and its implications for AI governance.
Here are the key takeaways:
The SCHUFA case is about a data subject who had her loan application rejected by a bank based on a credit score produced by a credit agency using her personal data. The case brought before the Court of Justice of the European Union (CJEU) concerned whether the credit scoring constituted automated individual decision-making under the GDPR.
The GDPR provides a definition of 'automated individual decision-making' and specifies when such processing may be carried out. Data subjects whose personal data are used for automated individual decision-making have the right to be informed of such processing and to receive further information about how its carried out.
The CJEU held in the SCHUFA case that a credit score generated by a credit agency is an automated individual decision if a third party (like a bank) "draws strongly" on it to make a decision. Accordingly, if one entity makes a decision that "draws strongly" on an algorithmic output produced by another, then the entity that produced the algorithmic output is effectively carrying out automated individual decision-making.
The case highlights the risk of automation bias in the context of AI. This is where the users of AI systems over-rely on the outputs of such systems to make decisions or act in certain situations.
Certain parts of the AI Act complement the GDPR in terms of mitigating the risk of automation bias. These include provisions on AI literacy and human oversight for high-risk AI systems.