EDRi welcomes the European Commission’s globally significant step towards regulating the development and deployment of artificial intelligence systems. However, we would like to make suggestions to ensure that the AI Act is in line with the Charter of Fundamental Rights of the European Union (EU), future proof, and a role model for rights-protective future AI legislation around the world.

We raise concerns about the extent to which the Act protects fundamental rights and addresses broader structural, political and economic issues. In the attached paper, we highlight how the use of AI systems can systematically target, harm and exclude marginalised communities. We question how far the act can address these structural harms, due to its tendency towards de-regulation of all but the most narrowly-defined ‘unacceptable’ uses; the lack of obligations on users; and the lack of individual and collective redress for those subject to AI systems. EDRi thus recommends the following improvements:

1. Ensure effective protection against prohibited practices and address the full scope of unacceptable risks:
a) Strengthen art. 5 prohibitions to provide meaningful protection against fundamental rights violations and individual and collective harms;
b) Comprehensively prohibit biometric mass surveillance practices (RBI and automated recognition of human features in publicly-accessible spaces for any purpose);
c) Prohibit additional practices which are incompatible with fundamental rights and democracy: uses of AI in law enforcement or criminal justice that purport to predict future behaviour; in migration control in ways that undermine the right to claim asylum; to implement invasive surveillance, monitoring and algorithmic management in an employment and educational context; to categorise people on the basis of their human features; for emotion recognition; and in uses that constitute mass surveillance.

2. Adapt the AIA to ensure a holistic, democratic and future-proof framework:
a) Introduce a democratic, inclusive and accessible process for the insertion of new prohibitions, including criteria for ‘unacceptable risk’ and the addition of future prohibitions;
b) Ensure high risk ‘areas’ can be updated under art. 7;
c) Fix gaps relating to economic and environmental impact, structural inequality, migration control, law enforcement (LE), worker surveillance, mass surveillance and exports;
d) Remove loopholes in arts. 2.4 and 83 (international LE agreements);
e) Remove art. 47 broad public security exemption for conformity assessment;
f) Remove art. 54.1.a purpose limitation exemption in criminal justice.

3. Enhance obligations on users of all AI systems:
a) Mandate ex ante human rights impact assessments for high risk systems before putting into use;
b) Implement duty to cooperate with national authorities under arts. 65 and 67 for all systems;
c) Implement meaningful consultation duty;
d) Implement public authority notification requirement.

4. Implement meaningful public transparency for high risk systems:
a) Registration in EU database (art. 60) for AI systems that are put into use;
b) Add instructions for use for LE, migration, asylum and borders to public database;
c) Require providers to include access to conformity assessment as per article 13.2-3 under art. 60;
d) Require more thorough detail from providers to users under art. 13.3;
e) Remove art. 52 exemptions for transparency in detection, prevention or prosecution; for art. 52 uses, suspects should be notified post factum.

5. Facilitate accountability: Include oversight and enforcement infrastructures that work for people:
a) Ensure cohesive national enforcement structure;
b) Include individual and collective flagging and redress mechanisms;
c) Implement a more democratic governance infrastructure and greater independence for EAIB.

See EDRi’s paper for further analysis and recommendations here.