Reuben Binns, our Research Fellow in Artificial Intelligence (AI), and Valeria Gallo, Technology Policy Adviser, discuss some of the key safeguards organisations should implement when using solely automated AI systems to make decisions with significant impacts on data subjects.
This post is part of our ongoing Call for Input on developing the ICO framework for auditing AI. We encourage you
to share your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
The
General Data Protection Regulation (GDPR) requires organisations to implement
suitable safeguards when processing personal data to make solely automated
decisions that have a legal or similarly significant impact on individuals. These
safeguards include the right for data subjects:
- to
obtain human intervention; - to
express their point of view; and - to
contest the decision made about them.
These safeguards cannot be token gestures. Guidance published by the European
Data Protection Board (EDPB) states that human intervention involve a review of
the decision, which
Data Protection Board (EDPB) states that human intervention involve a review of
the decision, which
“must be
carried out by someone who has the appropriate authority and capability to
change the decision”.
carried out by someone who has the appropriate authority and capability to
change the decision”.
The review should include a
“thorough
assessment of all the relevant data, including any additional information
provided by the data subject.”
assessment of all the relevant data, including any additional information
provided by the data subject.”
In this respect, the conditions
under which human intervention will qualify as meaningful are similar to those that
apply to human oversight in ‘non-solely automated’ systems.
However, a key difference is that in solely automated
contexts, human
intervention is only required on a case-by-case basis to safeguard the data
subject’s rights.
under which human intervention will qualify as meaningful are similar to those that
apply to human oversight in ‘non-solely automated’ systems.
However, a key difference is that in solely automated
contexts, human
intervention is only required on a case-by-case basis to safeguard the data
subject’s rights.
Why is this a particular issue for AI
systems?
systems?
The
type and complexity of the systems involved in making solely automated
decisions will affect the nature and severity of the risk to people’s data
protection rights and will raise different considerations, as well as compliance
and risk management challenges.
type and complexity of the systems involved in making solely automated
decisions will affect the nature and severity of the risk to people’s data
protection rights and will raise different considerations, as well as compliance
and risk management challenges.
Basic
systems, which automate a relatively small number of explicitly written rules
(eg a set of clearly expressed ‘if-then’
rules to determine a customer’s eligibility for a product) are unlikely to be
considered AI. It should also be relatively easy for a human reviewer to
identify and rectify any mistake, if a decision is challenged by a data subject
because of system’s high
interpretability.
systems, which automate a relatively small number of explicitly written rules
(eg a set of clearly expressed ‘if-then’
rules to determine a customer’s eligibility for a product) are unlikely to be
considered AI. It should also be relatively easy for a human reviewer to
identify and rectify any mistake, if a decision is challenged by a data subject
because of system’s high
interpretability.
However
other systems, such as those based on machine learning (ML), may be more
complex and present more challenges for meaningful human review. ML systems make
predictions or classifications about people based on data patterns. Even when
they are highly accurate,
they will occasionally reach the wrong decision in an individual case. Errors
may not be easy for a human reviewer to identify, understand or fix.
other systems, such as those based on machine learning (ML), may be more
complex and present more challenges for meaningful human review. ML systems make
predictions or classifications about people based on data patterns. Even when
they are highly accurate,
they will occasionally reach the wrong decision in an individual case. Errors
may not be easy for a human reviewer to identify, understand or fix.
While
not every challenge on the part of data subject will be valid, organisations
should expect that many could be. There are two particular reasons why this may
be the case in ML systems:
- The individual is an ‘outlier’, ie their
circumstances are substantially different from those considered in the training
data used to build the AI system. Because the ML model has not been trained on enough
data about similar individuals, it can make incorrect predictions or
classifications. - Assumptions in the AI design can be challenged, for example a continuous
variable such as age, might have been broken up (‘binned’) into discrete age
ranges, eg 20-39, as part of the modelling process. Finer-grained ‘bins’ may result
in a different model with substantially different predictions for people of
different ages. The validity of this data pre-processing and other design choices
may only come into question as a result of an individual’s challenge.
What should organisations do?
Organisations
should:
- consider the system
requirements necessary to support a meaningful human review from the design
phase. Particularly, the interpretability requirements and effective
user-interface design to support human reviews and interventions; - design and deliver
appropriate training and support for human reviewers; and
- give staff the
appropriate authority, incentives and support to address or escalate data
subjects’ concerns and, if necessary, override the AI system’s decision.
However,
there are some additional requirements and considerations organisations should
be aware of:
- The use of solely automated systems to make
decisions with legal or significant effects on data subjects will always
trigger the need for a Data Protection Impact Assessment(DPIA). DPIAs are a
compliance requirement, but also a helpful tool for organisations to reflect
carefully on the appropriateness of deploying a solely automated process. In
the case of AI systems, DPIAs should give particular consideration to the level
of complexity and interpretability of the system, and the organisation’s
ability to adequately protect individuals and their rights. - Our
ExplAIn
project is currently looking at how, and to what extent, complex AI
systems might affect an organisation’s ability to provide meaningful
explanations to data subjects. However, complex AI systems can also
impact the effectiveness of other mandatory safeguards. If a system is too
complex to explain, it may also be too complex to meaningfully contest, to
intervene on, to review, or to put an alternative point of view against. For
instance, if an AI system uses hundreds of features and a complex, non-linear
model to make a prediction, then it may be difficult for a data subject to
determine which variables or correlations to object to. Therefore safeguards
around solely automated AI systems are mutually supportive, and should be designed
holistically and with the data subject in mind. - The information about
the logic of a system and explanations of decisions should give data subjects the
necessary context to decide whether, and on what grounds, they would like to request
human intervention. In some cases, insufficient explanations may prompt data
subjects to resort to other rights unnecessarily. Requests for intervention,
expression of views, or contests are more likely to happen if data subjects don’t
feel they have a sufficient understanding of how the decision was reached. - The process for data
subjects to exercise their rights should be simple and user friendly. For instance,
if the result of the solely automated decision is communicated through a
website, the page should contain a link or clear information allowing the
individual to contact a member of staff who can intervene, without any undue
delays or complications. Organisations are expected to keep a record of
all decisions made by an AI system, as well as whether a data subject requested
human intervention, expressed any views, contested the decision, and whether
the decision was changed as a result. - Organisations should monitor and analyse
this data. If decisions are regularly changed in response to data subjects
exercising their rights, organisations will be expected to amend their systems
accordingly. Where the system is based on ML, this might involve including the
corrected decisions into fresh training data, so that similar mistakes are less
likely to happen in future. More substantially, they may identify a need to
collect more or better training data to fill in the gaps that led to the
erroneous decision, or modify the model-building process, ie by changing the
feature selection.
In addition to being a compliance
requirement, this is also an opportunity for organisations to improve the
performance of their AI systems and, in turn, build data subjects’ trust in them. However, if grave or frequent mistakes are identified,
organisations will need to take immediate steps to understand and rectify the
underlying issues and, if necessary, suspend the use of the automated system.
We would like to hear your views on this
topic and genuinely welcome any feedback on our current thinking. Please share
your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
topic and genuinely welcome any feedback on our current thinking. Please share
your views by leaving a comment below or by emailing us at AIAuditingFramework@ico.org.uk.
 |
 Dr Reuben Binns, a researcher working on AI and data protection, joined the ICO on a fixed term fellowship in December 2018. During his two-year term, Reuben will research and investigate a framework for auditing algorithms and conduct further in-depth research activities in AI and machine learning. |
|
|
Valeria Gallo is currently seconded to the ICO as a Technology Policy Adviser. She works with Reuben Binns, our Artificial Intelligence (AI) Research Fellow, on the development of the ICO Auditing Framework for AI. Prior to her secondment, Valeria was responsible for analysing and developing thought leadership on the impact of technological innovation on regulation and supervision of financial services firms.
|
|
