Can the EU make AI “trustworthy”? No – but they can make it just

Today, 4 June 2020, European Digital Rights (EDRi) submitted its answer to the European Commission’s consultation on the AI White Paper. On top of our response, in our additional paper we outline recommendations to the European Commission for a fundamental rights- based AI regulation. You can find our consultation response, recommendations paper, and answering guide for the public here.

How to ensure a “trustworthy AI” has been highly debated since the European Commission launched its White Paper on AI in February this year. Policymakers and industry have hosted numerous conversations about “innovation”, “Europe becoming a leader in AI”, and promoting a “Fair AI”.

Yet, a “fair” or “trustworthy” artificial intelligence seems a far way off. As governments, institutions and industry swiftly move to incorporate AI into their systems and decision-making processes – grave concerns remain as to how these changes will impact people, democracy and society as a whole.

EDRi’s response outlines the main risks AI poses for people, communities and society, and outlines recommendations for an improved, truly ‘human-centric’ legislative proposal on AI. We argue that the EU must reinforce the protections already embedded in the General Data Protection Regulation (GDPR), outline clear legal limits for AI by focusing on impermissible use, and foreground principles of collective impact, democratic oversight, accountability, and fundamental rights. Here’s a summary of our main points.

Put people before industrial policy

A ‘human centric’ approach to AI requires that considerations of safety, equality, privacy, and fundamental rights are the primary factors underpinning decisions as to whether to promote or invest in AI.

However, the European Commission’s White Paper proposal takes as a point of a departure the inherent economic benefits of promoting AI, particularly in the public sector. Promoting AI in the public sector as a whole, without requiring scientific evidence to justify the need or the purpose of such applications in some potentially harmful situations, is likely to have the most direct consequences on everyday peoples’ lives, particularly on marginalised groups.

Despite wide ranging applications that could advance our societies (such as some uses in the field of health), we have also seen the vast negative impacts of automated systems at play at the border, in predictive policing systems which exacerbate overpolicing of racialised communities, in ‘fraud detection’ systems which target poor, working class and migrant areas, and countless more examples [link to explainer]. All such examples highlight the potentially devastating consequences AI systems can have in the public sector, contesting the case for ‘promoting the uptake of AI.’ These examples highlight the need for AI regulation to be rooted in a human-centric approach.

The development of artificial intelligence technology offers huge potential opportunities for improving our economies and societies, but also extreme risks. Poorly-designed and governed AI will exacerbate power imbalances and inequality, increase discrimination, invade privacy and undermine a whole host of other rights. EU legislation must ensure that cannot happen. Nobody’s rights should be sacrificed on the altar of innovation.

said Chris Jones, Statewatch

Address collective harms of AI

The vast potential scale and impact AI systems challenges existing conceptions of harm. Whilst in many ways we can view the challenges posed by AI as fundamental rights issues, often the harms perpetrated are much broader, disadvantaging communities, economy, democracy and entire societies. From the impending threat of mass surveillance as a a result of biometric processing in publicly-accessible spaces, to the use of automated systems or ‘upload filters’ to moderate content on social media, to severe disruptions to the democratic process, we see the impact goes far beyond the level of the individual. One specificity of regulating AI is the need to address societal-level harms.

Prevent harms by focusing on impermissible use

Just as the problems with AI are collective and structural, so must be the solutions. The European Commission’s White Paper outlines some safeguards to address ‘high-risk’ AI, such as training data to correct for bias and ensuring human oversight. Whilst these safeguards are crucial, they will not address the irreparable harms which will result from a number of uses of AI.

“The EU must move beyond technical fixes for the complex problems posed by AI. Instead, the upcoming AI regulation must determine the legal limits, impermissible uses or ‘red-lines’ for AI applications. This is a necessary step for a people-centered, fundamental rights-based AI”

says Sarah Chander, Senior Policy Adviser, EDRi.

The EDRi network lists some of the impermissible uses of AI:

• indiscriminate biometric surveillance and biometric capture and processing in public spaces¹
• use of AI to solely determine access to or delivery of essential public services (such as social security, policing, migration control)
• uses of AI which purport to identify, analyse and assess emotion, mood, behaviour, and sensitive identity traits (such as race, disability) in the delivery of essential services
• predictive policing
• autonomous lethal weapons and other uses which identify targets for lethal force (such as law and immigration enforcement)

“The EU must ensure that states and companies meet their obligations and responsibilities to respect and promote human rights in the context of automated decision-making systems. EU institutions and national policymakers must explicitly recognise that there are legal limits to the use and impact of automation. No safeguard or remedy would make indiscriminate biometric surveillance or predictive policing acceptable, justified or compatible with human rights”

said Fanny Hidvegi, Europe Policy Manager at Access Now

Require democratic oversight for AI in the public sphere

The rapidly increasing deployment of AI systems presents a major governance issue. Due to the (designed) opacity of the systems, the complete lack of transparency from governments when such systems are deployed for use in public, essential functions, and the systematic lack of democratic oversight and engagement – AI is furthering the ‘power asymmetry between those who develop and employ AI technologies, and those who interact with and are subject to them.’²

As a result, decisions impacting public services will be more opaque, increasingly privately owned, and even less subject to democratic oversight. It is vital that the EU’s regulatory proposal on AI addresses this – implementing mandatory measures of democratic oversight for the procurement and deployment of AI in the public sector and essential services. More, the EU must explore methods of direct public engagement on AI systems. In this regard, authorities should be required to specifically consult marginalised groups likely to be disproportionately impacted by automated systems.

Implement the strongest possible fundamental rights protections

Regulation on AI must reinforce, rather than replace, the protections already embedded in the General Data Protection Regulation (GDPR). The European Commission has the opportunity to complement these protections with safeguards for AI. To put people first and provide the strongest possible protections, all systems should complete mandatory human rights impact assessments. This assessment should evaluate the collective, societal, institutional and governance implications the system poses, and outline adequate steps to mitigate this.

“The deployment of such systems for predictive purposes comes with high risks on human rights violations. Introducing ethical guidelines & standards for the design and deployment of these tools is welcome, but not enough. Instead, we need the European Union and Member States to ensure compliance with the applicable regulatory frameworks, and draw clear legal limits to ensure AI is always compatible with fundamental rights.”

says Eleftherios Chelioudakis – Homo Digitalis

EDRi’s position calls for fundamental rights to be prioritised in the regulatory proposal for all AI systems, not only those categorised as ‘high-risk’. We argue AI regulation should avoid creating loop-holes or exemptions based on sector, size of enterprise, or whether or not the system is deployed in the public sector.

“It is crucial for the EU to recognize that the adoption of AI applications is not inevitable. The design, development and deployment of systems must be tested against human rights standards in order to establish their appropriate and acceptable use. Red lines are thus an important piece of the AI governance puzzle. Recognizing impermissible use at the outset is particularly important because of the disproportionate, unequal and sometimes irreversible ways in which automated decision making systems impact societies.”

said Vidushi Marda, Senior Programme Officer, at ARTICLE 19

The rapid uptake of AI will fundamentally change our society. From a human rights’ perspective, AI systems have the ability to exacerbate surveillance and intrusion into our personal lives, fundamentally alter the delivery of public and essential services, vastly undermine vital data protection legislation, and disrupt the democratic process.

For some, AI will mean reinforced, deeper harms as such systems feed and embed existing processes of marginalisation. For all, the route to remedies, accountability, and justice will be ever-more unclear, as this power asymmetry further shifts to private actors, and public goods and services will be not only automated, but privately owned.

There is no “trustworthy AI” without clear red-lines for impermissable use, democratic oversight, and a truly fundamental rights-based approach to AI regulation. The European Union’s upcoming legislative proposal on artificial intelligence (AI) is a major opportunity change this; to protect people and democracy from the escalating economic, political and social issues posed by AI.

Footnotes:

1EDRi (2020). ‘Ban Biometric Mass Surveillance: A set of fundamental
rights demands for the European Commission and Member States’
https://edri.org/wp-content/uploads/2020/05/Paper-Ban-Biometric-Mass-Surveillance.pdf

2Council of Europe (2019). ‘Responsibility and AI DGI(2019)05 Rapporteur: Karen Yeung https://rm.coe.int/responsability-and-ai-en/168097d9c5

Read more:

EDRi Consultation Response: European Commission Consultation on the White Paper on Artificial Intelligence
https://edri.org/wp-content/uploads/2020/06/AI_EDRiConsultationResponse.pdf

EDRI Recommendations for a Fundamental-rights based Artificial Inelligence Regulation: Addressing collective harms, democratic oversight and impermissable use
https://edri.org/wp-content/uploads/2020/06/AI_EDRiRecommendations.pdf

Access Now Consultation Response: European Commission Consultation on the White Paper on Artificial Intelligence
https://www.accessnow.org/EU-white-paper-consultation

Bits of Freedom (2020). ‘Facial recognition: A convenient and efficient solution, looking for a problem?’
https://www.bitsoffreedom.nl/2020/01/29/facial-recognition-a-convenient-and-efficient-solution-looking-for-a-problem/

EDRi (2020). ‘Ban Biometric Mass Surveillance: A set of fundamental rights demands for the European Commission and Member States’
https://edri.org/wp-content/uploads/2020/05/Paper-Ban-Biometric-Mass-Surveillance.pdf

Privacy International and Article 19 (2018). ‘Privacy and Freedom of Expression in the Age of Artificial Intelligence’
https://www.article19.org/wp-content/uploads/2018/04/Privacy-and-Freedom-of-Expression-In-the-Age-of-Artificial-Intelligence-1.pdf