A Typology of Automated Law Enforcement Systems Under EU Law: What Standards for Defence Rights?

Table of Contents: 1. Introduction. – 2. AI-powered law enforcement. – 2.1. Typology. – 2.2. Critical reflections with a focus on the transition from high-risk to minimal-risk AI in law enforcement settings. – 3. What standards for defence rights in the era of AI? – 3.1 Implications for defence rights. – 3.2 How to safeguard defence rights in times of automated law enforcement. –3.2.1. Defence-rights-by-design. – 3.2.2. Defence rights in (EU secondary) law. – 4. Conclusions.

Abstract: New and emerging technologies are re-shaping police-citizen encounters – ranging from person-based predictive policing software deployed to spot likely wrongdoers to AI-driven software solutions employed to collect, organise, visualise and assess data for criminal investigation purposes. Hence, automated law enforcement requires a careful balancing act between effectiveness and innovation on the one hand, and protection of fundamental rights on the other. It is no coincidence that the EU AI Act classifies AI systems of this or similar kind in principle as high-risk systems and establishes functional requirements and additional safeguards in the form of legal obligations on the providers and deployers thereof. Additionally, the 2021 European Parliament's Resolution on AI in criminal law further specifies the fundamental rights scrutiny required in the case of automated law enforcement. Against this background, this article first presents a typology of automated law enforcement systems and discusses the overall approach in the EU AI Act towards the use of AI in the area of criminal law. Next, it focuses on defence rights standards, exploring how these standards may be reshaped at the EU level, incorporated into technical design, and enforced nationally. Such a combined solution should ensure that defence rights standards are adapted to the potential of the machine and to the power imbalance and fallibility inherent in its use.

Keywords: artificial intelligence – law enforcement – automated law enforcement – AI Act (AIA) – defence rights – human-rights-by-design.

1.   Introduction

Thinking of search engines, online shops, dating and self-tracking apps or similar, it becomes evident that decision-making supported by algorithms is anything but a far-fetched reality. New and emerging technologies, such as Artificial Intelligence (AI), including Machine Learning (ML), have ignited a passionate debate about whether we may entrust algorithms with more creative and, certainly, more complicated tasks that have previously been taken over primarily (though not exclusively) by humans. Examples include AI systems that can help a farmer decide upon pesticide treatments[1] and may support a public agriculture agency in deciding whether to fund that farmer’s activities.[2] Further, AI systems may help a supervision authority predict whether the same farmer is susceptible to committing fraud against the European Union (EU) budget,[3] and may even support the police in investigating such a fraud case.[4] Finally, an AI system may equally help the accused farmer to dissent the charges,[5] and support the court in ruling on this case.[6]

Automation in these (and many other) areas is no novelty. Taking law enforcement as an example, it strikes that mathematical-statistical calculations have a long history in crime control, having even penetrated criminal justice in the case of probation decisions.[7] Complexity, however, increases significantly when delving into how an AI-supported algorithm delivers a certain result that serves subsequently as a decision basis. The ability to ‘learn iteratively from data and think in concepts and eventually turn themselves into a source of new knowledge, generated by AI’[8] is a specific feature of such algorithms. This raises numerous questions: To whom do we attribute this ‘knowledge’? How can it be assessed? How can it be contested? The more this ‘knowledge’ is capitalised across a decision-making network, the more pressing an answer to these questions becomes. For instance, an AI-based software employed by the police can be used to make sense of voluminous datasets for the purposes of crime investigation to the extent that its output (e.g., a dynamic knowledge graph) may, subsequently, inform the prosecution files and be admitted as a piece of evidence in court. It is true that AI tools may not be unique as a source of investigative leads. In the past, DNA analysis has literally given rise to a new generation of forensic evidence.[9] Nonetheless, the decisive difference lies in the capacity of AI-supported decision-making systems to operate both automatically and at scale. For this reason, they have the potential to affect thousands, if not millions, of people.[10]

This reality has been fuelling a lively debate at national, regional, and international level. Its aim is to decode the impact the widespread use of AI applications in general and particularly in fields tightly linked to public governance (such as law enforcement) may have on fundamental rights.[11] Although not all rights attract the same amount of attention, the spectrum of affected rights is broad, ranging from privacy and data protection to freedom of expression and information, freedom of assembly and association, the right to non-discrimination, and the right to an effective remedy and to a fair trial.[12] Initially, although no substantial revisions of human rights laws had taken place, there has been a ‘mass production’ of AI ethical standards that seek to counterbalance the mismatch between technological development and regulation.[13] In that sense, the adoption of a Regulation laying down harmonised rules on Artificial Intelligence (EU Artificial Intelligence Act; hereinafter: AIA) at the EU level[14] – following numerous policy developments in the area of AI ethics[15] and a series of European Parliament (EP) Resolutions on topics related to AI[16] – is a turning point.[17] The AIA adopts a risk-based approach towards regulating AI,[18] essentially based on the systems’ impact on health, safety and fundamental rights (recitals 7, 46 among others). Overall, the AIA distinguishes between four different categories of risk: 

i) unacceptable risk, leading to prohibitions of certain AI practices that may be waived under concrete circumstances (Article 5);[19]

ii) high risk (Article 6), prompting a series of legal obligations to be fulfilled by AI providers, users, and third parties (Articles 8-49); 

iii) limited risk, giving rise to heightened transparency obligations (Article 50);[20] and 

iv) minimal risk, not leading to specific obligations but still giving rise to the voluntary application of codes of conduct to foster common standards (Article 95).

Against this backdrop, this article discusses automated law enforcement, examining how different AI systems employed in this area of public governance, which is traditionally shaped by power-imbalance, have been classified as prohibited, high-, limited- and minimal-risk AI by the EU legislator. This analysis results in a typology (Section II), which reveals that, even if numerous (heterogenous) AI systems employed for the purposes of law enforcement have been put under the ‘high risk’ umbrella, exceptions made to this central regulatory choice have the potential to jeopardise significantly fundamental rights and defence rights in particular. Next, this article provides a comprehensive overview of how automated law enforcement impacts defence rights (Section 3.1), a research question driven by the potential of the respective AI systems to generate ‘knowledge’ intended to penetrate criminal proceedings (recital 59 AIA). This is followed by a brief examination of the governance model established by the AIA and of the (co-) regulatory steps further required to enhance the protection of fundamental rights (Section 3.2). In particular, this article discusses how technical design can be informed by legal requirements for the protection of defence rights (defence rights by design) and what kind of defence rights scrutiny is required when public authorities employ AI for law enforcement purposes. Regarding the latter, the article uses EU law as a system of reference (defence rights in law).

For the purposes of the present analysis, this article builds on previous scholarship on automated law enforcement, AI ethics, the AIA (with a focus on studies that concentrate on fundamental rights implications), AI regulation and governance, including ‘ethics by design’ frameworks. Socio-legal and doctrinal legal research methods are combined to provide a typology of automated law enforcement systems and to suggest defence rights standards tackling the impact AI systems have on areas of public governance. In terms of legal doctrine, the analysis is predominantly EU-law based. However, this is not meant to downplay the existence and importance of other (and numerous) initiatives towards embedding the protection of human rights in national[21] and international AI laws.[22]

2.   AI-powered law enforcement

As AI is a new means of knowing about and understanding the world,[23] the legislature had significant difficulties when attempting to draft a comprehensive (and neutral) definition of AI. According to the work and findings of the High-Level Expert Group on AI (AI HLEG), a multi-disciplinary expert group that guided the first legislative steps in regulating AI at the EU level,[24] ‘AI is best understood as a sociotechnical concept’.[25] Using the AI HLEG findings as a starting point and considering the technical capabilities of AI systems, particularly compared to traditional software tools, Article 3(1) AIA defines them as ‘machine-based system[s] designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infer, from the input [they receive], how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments’.

This definition should assist scholars and practitioners in classifying and understanding multiple computational models that seek to ‘measure the strength of associations between a set of data points and an outcome of interest [in the form of a probabilistic distribution]’.[26] In a penal system, the data used to build and train such models are administrative information collected by police and criminal justice authorities (including historical crime data).[27] The interpretation of such data shall inform predictions or forecasts as to where and when the next crime will occur, but also as to who has the potential to commit it. In addition, AI-fuelled predictions or forecasts shall inform decisions of law enforcement and criminal justice authorities as to whom to surveil, search, arrest, prosecute, punish, incarcerate, release on parole etc. Ideally, the use of data-driven models shall turn the respective decisions into ‘evidence-based’ practices that are effective and free of bias.[28]

Besides crime prevention tools, AI tools equally facilitate crime investigation. Notably, they are associated with a general increase of investigative power, resulting from the possibility of matching seemingly disparate data and recognising trends and patterns in large data sets. This means that AI systems may be employed to investigate past crimes by means of organising voluminous data that gives insight into criminal practices and identities.[29] Given AI’s capacity to analyse massive amounts of data in a split second, the technology figures as important asset for the police striving to analyse data adequately (usually under time pressure) and take the necessary measures to enforce law and avert danger.[30] In short, AI is able to learn from past data and never grows tired.[31] Hence, the technology seems optimally suited to assist human agents. As will be shown below, these phenomenologies are reflected in the typology of automated law enforcement the AIA introduces as part of its risk-based approach towards regulating AI.[32]

2.1.  Typology

Following the trilogue negotiations, which were marked by significantly divergent approaches to the risks posed by the progressive automation in law enforcement settings, reflected in the Council’s General Approach to the AIA[33] and the EP’s Amendments thereto,[34] the AIA introduces two prohibitions in the use of AI for law enforcement purposes. Considering the substantial increase in surveillance capabilities of law enforcement authorities (LEAs) using facial recognition technologies (FRTs) and the significant impact on, inter alia, citizens’ privacy, Article 5(1) lit. h AIA prohibits the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purposes of law enforcement. This prohibition is subject to multiple exceptions concerning: the targeted search for specific victims of abduction, trafficking in or sexual exploitation of sexual beings and search for missing persons; the prevention of specific, substantial, and imminent threats to life or physical safety or genuine and foreseeable threat of a terrorist attack; and the localisation or identification of suspected offenders for the purpose of investigating, prosecuting or executing a criminal penalty for the crimes that are exhaustively listed in Annex II to the AIA. The latter includes both harmonised (e.g., terrorism, trafficking in human beings) and non-harmonised offences (e.g., murder, grievous bodily injury and rape). In both cases, the offence in question shall be punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least four years.[35] To enable these exceptions, the use of FRTS shall adhere to strict proportionality requirements, including limitations on time, location and personal scope, and be previously authorised by national law. This authorisation necessitates a fundamental rights impact assessment (Articles 5(2) and 27 of AIA), along with the respective AI system’s registration in the EU database (Article 49 AIA) and is to be granted by a judicial authority or an independent administrative body, ensuring that decisions resulting in adverse effects on individuals will not solely rely on the output of real-time remote biometric identification systems (Article 5(3) AIA). Additionally, it is required to notify the relevant market surveillance authority and the national data protection authority, as well as to update the applicable national law to include the mentioned authorisation procedure (Article 5(4–5) AIA). 

The second prohibition, which was adopted following the EP’s lead and the civil society’s calls,[36] refers to AI systems employed for ‘making risk assessments of natural persons in order to assess or predict the likelihood of a natural person committing a criminal offence, based solely on the profiling of a natural person or on assessing their personality traits and characteristics’ (Article 5(1) lit. d AIA). This prohibition is not absolute inasmuch the use of such systems remains possible, if these are employed to support the human assessment of an individual’s involvement in a criminal activity that is already grounded in objective and verifiable facts directly associated with said criminal activity (ibid). This scenario can be addressed, though, in the realm of high-risk AI.

The AIA provides for two categories of high-risk AI systems understood as systems that have ‘a significant harmful impact on the health, safety and fundamental rights of persons in the Union’ (recital 46). The first category encompasses AI systems that are intended to be used as safety components of a product or are themselves a product, covered by the Union harmonisation legislation listed in Annex I to AIA and – as such – are required to undergo a third-party conformity assessment, in order to be placed on the market or put into service (Article 6(1) AIA).^[37] In this case, the risk arises from the specificities of the sector where the AI system is to be deployed, and from its particular use.^[38] The second category of high-risk AI systems includes stand-alone AI systems (Article 6(2) AIA) employed in any of the following areas: biometrics; critical infrastructure; education and vocational training, employment; workers management and access to self-employment; access to and enjoyment of essential private services and essential public services and benefits; law enforcement; migration, asylum and border control management; administration of justice and democratic processes. For each of these areas, Annex III lists specific categories of AI systems that are to be considered as high-risk, depending on their intended purpose (Article 3(12) AIA).

As particularly regards the area of law enforcement, point 6 of Annex III to the AIA designates as high-risk AI systems, whose use is permitted under relevant Union or national law and which are intended to be used by or on behalf of LEAs, Union institutions, bodies, offices or agencies, in order to: a) assess a natural person’s risk of becoming the victim of criminal offences; b) serve as polygraphs or similar tools; c) evaluate the reliability of evidence in the course of the investigation or prosecution of criminal offences; d) assess the likelihood of a natural person of offending or re-offending not solely based on profiling of natural persons as referred to in Article 3(4) Directive (EU) 2016/680 (known as Law Enforcement Directive (LED)), or personality traits and characteristics or past criminal behaviour of natural persons or groups; e) profile natural persons as referred to in Article 3(4) LED in the course of the detection, investigation or prosecution of criminal offences.

As mentioned above, the providers of high-risk AI systems, including the aforementioned systems employed for law enforcement purposes, shall abide by minimum requirements related to risk management, data governance, technical documentation, record keeping, transparency and user information, human oversight, accuracy, robustness and cybersecurity (Articles 9–15 AIA). This should minimise the adverse impact of the use of such systems on affected individuals and groups of individuals. To attain this goal, these minimum requirements are coupled with obligations for the deployers, who shall perform, inter alia, an assessment of the impact on fundamental rights the use of the system in question may produce (Article 27(1) AIA).

The prohibited and high-risk AI systems do not encompass the entire range of automated law enforcement. For instance, AI-based software solutions, developed to assist the personnel of LEAs in investigating crime by means of, among other capabilities, analysing voluminous datasets do not fall – anymore (compared to the initial Commission’s Proposal and the EP’s stand on the matter) – under any of the above-mentioned categories. Instead, the drafters of the forthcoming AIA advocated an exception for crime analytics, an exception which was previously promoted by the Council as part of the General Approach to AIA (the same trend was detected in all previous Compromise Texts released by the Council Presidency).[39] Such systems may include a certain level of direct interaction with natural persons, fact that at first sight suggests that a limited risk can be inherent in their development and use in the law enforcement environment. However, pursuant to Article 50(1) AIA, the enhanced transparency requirements set out therein will not be applicable to ‘AI systems authorised by law to detect, prevent, investigate or prosecute criminal offences, subject to appropriate safeguards for the rights and freedoms of third parties, unless those systems are available for the public to report a criminal offence’. This is yet another exception meant to serve the investigative needs of LEAs (recital 123 AIA).

In sum, AI systems developed and deployed for law enforcement purposes that neither fall into the scope of the prohibitions provided for in Article 5 AIA nor are listed in point 6 of Annex III to AIA or serve crime reporting purposes (Article 50(1) AIA) are classified as minimal-risk AI. Hence, they are to be governed by primarily voluntary development and use standards. This approach is confirmed by the wording of Article 6(3) AIA that explicitly introduces exceptions to the automatic classification of AI systems employed in designated high-risk areas, including law enforcement, as listed in Annex III to the AIA. The primary justification for this exception is that such systems may be deployed in areas that are ostensibly high-risk without, however, materially influencing decision-making, whether human or automated, or significantly harming the health, safety, or fundamental rights of individuals (recital 53 AIA). This exception, which will not apply to AI systems that perform profiling of natural persons, encompasses cases where the AI system is intended to: a) perform a narrow procedural task; b) improve the result of a previously completed human activity; c) detect decision-making patterns or deviations from prior decision-making patterns, without replacing or influencing the previously completed human assessment, without proper human review; or d) perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III.

The Compromise Texts by the Council Presidency have laid the groundwork for such amendments to the perception of high-risk AI. Notably, seeking to address feedback of EU Member States (EU MSs) concerning the breadth of notions adopted to define high-risk AI systems, the Second Presidency Compromise Text introduced ‘another horizontal layer on the top of the high-risk classification made in Annex III’ including the following criteria: a) immediacy of the effect of the system’s output without human validation; or b) significance of the system’s output in the sense of being the sole basis or not being purely accessory in respect of actions or decisions to be taken by a human.[40] The Third Compromise Text crossed out the first criterion on the basis that such a criterion may be prone to circumvention by involving a human intermediary, and that automated systems should not be automatically considered high-risk. Additionally, it rephrased the second criterion by removing the reference to the ‘sole basis’, which was considered implicit.[41]

The criterion of ‘pure accessoriness’, which implies that the AI system assists in the decision-making procedure without replacing human judgment, was further scrutinised by means of introducing some examples in the then recital 32 (as revised by the drafters of the Third Compromise Text). More specifically, when the algorithmic output consists of the profiling of natural persons (as defined in Articles 4(4) of Regulation (EU) 2016/679, 3(4) of Directive (EU) 2016/680, 3(5) of Regulation (EU) 2018/1725), it should not be considered to be of accessory nature for the purposes of Art 6 AIA. This is not the case with AI systems that only have negligible or minor relevance for human action or decision, such as those employed for translation or document management purposes. This approach was preserved in the Fourth Compromise Text and the Council’s General Approach to the AIA, which renders ‘pure accessoriness’ into the prevailing additional horizontal layer on top of the implications for fundamental rights for the high-risk classification purposes.[42] In any event, there should be a causal link between the lack of pure accessoriness and a significant risk to health, safety or fundamental rights. 

Similar examples are now included in the revised recital 53 AIA (AI systems that transform unstructured data into structured data, classify incoming documents into categories, detect duplicates among a large number of applications, improve the language used in previously drafted documents, index, search, process text and speech or link data to other data sources, translate initial documents etc.). In doing so, the drafters of the AIA turn the spotlight (even just indirectly) onto the nature of the task entrusted to the AI system (e.g., profiling vs. translation). In that sense, the AIA does not take into consideration the communication between AI systems of purely and non-purely accessory character (e.g., an AI system translates social media posts and its output serves as input data for another AI system employed for profiling purposes). Further, it neglects the risks inherent in the use of purely accessory systems (e.g., an AI system intended to be employed for transcription purposes discriminates against members of national groups based on the dialect used in the recordings). Additionally, the ‘pure accessoriness’ criterion, which is reflected in the wording of and the use cases included in Article 6(3) AIA, fails to encompass the risks arising from the objectivity and the scientific language that surround AI and the ‘security’ feelings the human user may develop.[43] The purely accessory character of an AI system does not necessarily prevent a human operator from over-relying on the system. Equally, purely accessory systems may likewise createsignificant risks for people’s health, safety or fundamental rights. Hence, it is not only the circumstances of ‘pure accessoriness’ that require further specification, but also the significance of the risks arising from the lack of those.

Against this backdrop, (at least) a case-by-case analysis of the AI system in question and the risks arising from its use appears to be necessary. In a similar way, the EU legislator promotes the following solutions for the providers that consider that an AI system referred to in Annex III is not high-risk (Article 6(4) AIA): first, they shall document this assessment, before the system is placed on the market or put into service, in order to ensure traceability and transparency. Second, the respective AI systems must be registered pursuant to Article 49(2) AIA. Additionally, the aforementioned documentation should become available to the national competent authorities upon request. Lastly, further guidance shall be provided by the Commission, after consulting the European AI Board, concerning the implementation of Article 6; this guidance should include a comprehensive list of practical examples of both high-risk and non-high-risk AI system use cases. 

2.2.  Critical reflections with a focus on the transition from high-risk to minimal-risk AI in law enforcement settings 

To date, a significant number of scholars has stressed the implications of automated law enforcement for fundamental rights, taking into account the particularities of the criminal law ecosystem and the coercive measures citizens may be subject to when they find themselves under the police microscope (e.g., surveillance, arrest, detention).[44] Such concerns are reflected in recital 59 AIA, which underlines the ‘significant degree of power imbalance’ that characterises automated law enforcement. Additionally, the same recital expressly refers to the risk of discrimination and unfair treatment that may arise from the lack of high-quality data, accuracy and robustness. Further, it cautions that the use of AI may hamper the ‘exercise of important procedural fundamental rights, such as the right to an effective remedy and to a fair trial as well as the right of defence and the presumption of innocence, […] where such AI systems are not sufficiently transparent, explainable and documented’. Against this backdrop, point 6 Annex III to AIA includes, as shown above, a list of AI systems, in the case of which it was deemed appropriate to be classified as high-risk, insofar their use is permitted under relevant Union and national law.[45] This choice is driven by the acknowledgement that accuracy, reliability and transparency is particularly important in the law enforcement context, in order to ‘avoid adverse impacts, retain public trust and ensure accountability and effective redress’ (recital 59 AIA).

The selection of the respective AI systems has been purpose-oriented and reflects a distinction between prediction/prevention- and repression-driven uses of AI (Table 1). In cases where the focus lies on prediction/prevention, AI systems may be employed to detect future risks, often without there being a concrete suspicion. Further, such future-oriented systems may also be used in the context or even in the aftermath of a criminal trial, namely to predict recidivism. In the case of repression-driven AI systems, the algorithm mainly serves suspicion-based detection, investigation and prosecution of past crimes. 

Table 1. Categories of automated law enforcement (point 6 Annex III to EU AIA).

This distinction is not absolute to the extent that repurposing is possible or cannot be excluded a priori (cf. Article 3(13) AIA, which defines the notion of reasonably foreseeable misuse; this means that an AI system is not used in accordance with its intended purpose). For instance, an AI system initially intended to support LEAs in investigating crimes already committed may be repurposed to forecast the future commission of criminal offences and the circumstances thereof. In that sense, the distinction between prediction/prevention- and repression-driven uses may become blurry.

A major concern lies in the following: By placing all above-listed AI systems under the ‘high-risk’ umbrella, the drafters of the AIA do not seem to have scrutinised the different and specific fundamental rights implications (and concerns) that arise from each single AI system. What is meant hereby may be clarified by looking at the example of the presumption of innocence, under Article 6(2) of the European Convention on Human Rights (ECHR) and Article 48(1) of the EU Charter of Fundamental Rights (CFR), in terms of a procedural fundamental right. Its protective scope only extends to ‘natural persons who are suspects or accused persons in criminal proceedings’ (Article 2 of Directive (EU) 2016/343). This means: unless one adopts an extensive reading of the presumption of innocence, encompassing the moral and political claim of citizens to be treated by the State as law-abiding until proven otherwise,[46] those who have not yet acquired or have lost the procedural identity of suspect or accused person would fall outside its protective scope. Hence, they may ‘merely’ be singled-out by the algorithm as a high-risk to commit a/any crime in the future,[47]without being entitled to any procedural protection arising from the principle of the presumption of innocence (including, but not limited to, burden of proof rules and the right not to incriminate oneself).

By contrast, a more refined, purpose-centred approach was adopted in the draft report on the AIA released by the EP Committee on the Internal Market and Consumer Protection (IMCO) and the EP Committee on Civil Liberties, Justice and Home Affairs (LIBE) in April 2022.[48] The rapporteurs stated: ‘AI systems used by law enforcement authorities or on their behalf to predict the probability of a natural person to offend or to reoffend, based on profiling and individual risk-assessment hold a particular risk of discrimination against certain persons or groups of persons, as they violate human dignity as well as the key legal principle of presumption of innocence. Such AI systems should therefore be prohibited’ (proposal for a regulation recital 17(a) (new), emphasis added). 

Next, they suggested the deletion of point 6 lit. a and lit. e of Annex III to the AIA (based on the Commission’s Proposal) and listed such AI systems under the prohibited uses of AI. As mentioned above (Section 2.1), this proposal was partially adopted in the final text of the AIA, according to which (Article 5(1) lit. d) AI systems, employed for making risk assessments of natural persons in order to assess or predict the risk of a natural person committing a criminal offence, based solely on profiling or on assessing their personality traits and characteristics, are prohibited. This prohibition can be waived if such systems have an auxiliary function, namely when they support a human judgment already based on objective and verifiable facts directly linked to a criminal activity. In reality, however, it can always be claimed that such an AI system did not replace human judgment without any substantial proof beyond the user’s claim itself.

Besides this, the revised list of AI systems included in point 6 of Annex III to the AIA does not encompass anymore AI systems deployed as deepfake detectors and for purposes of crime analytics, following the respective suggestion included in all Council Presidency Compromise Texts as well as in the Council’s General Approach. Apparently, this is an effectiveness-oriented move that aligns well with the stand the European Police Chiefs took on the AIA. The latter had criticised (what they called) the blanket classification of entire areas of automated law enforcement as high-risk and suggested a case-by-case risk assessment instead.[49]Their view was substantiated by several examples, including AI systems like speech and text recognition, information extraction, object detection and image classification. It was submitted that such tools are deployed with the primary goal of ‘automating data pre-processing and processing tasks’ in order to ‘relieve human analysts from repetitive tasks and from exposure to gruesome materials, and allow them to focus on more cognitive tasks’.[50] Additionally, they are applied to data already seized in the context of a criminal investigation. In that sense, such systems do not involve automated decision-making or enable indiscriminate data analysis. Thus, these AI systems should be considered ‘purely accessory’ inasmuch as their output is evaluated and checked by specially trained and skilled police officers.

The advocates of this exception emphasize the added value inherent in, among others, AI-powered crime analytics, but fail to recognize the risks associated with the development and training of the respective AI systems. Notably, it is the data-intensive character that distinguishes such systems from other traditional digital solutions.^[51] Additionally, the decision to adopt an exception for crime analytics, which automatically leads to the classification of such systems as minimal-risk AI, demonstrates that those responsible for it have not properly considered the potential bias inherent in large data sets that are used to train the algorithm.

Next, this exception does not address the risks inherent in the unwitting or indirect shift from pattern-based to(wards) individual-based data mining that may encompass profiling.^[52] For example, defence rights might be interfered with, should the algorithmic output ‘intrude’ the criminal proceedings as a piece of evidence (depending on the national laws governing evidence admissibility in criminal proceedings). This is particularly the case when the targeted individual does not have access to appropriate means to rebut the conclusions reached against them based on the algorithmic output – with the use of the latter in criminal proceedings resulting in an impermissibly high ‘innocence threshold’, if not an indirect reversal of the burden of proof.[53] Similar risks may arise in the case of intentional repurposing[54] of AI systems (e.g., shift from repression-driven crime analytics into individual-based predictive policing). 

Interestingly, the First Senate of the German Federal Constitutional Court addressed this matter in the judgment of 16 February 2023 on the constitutionality of Article 25(a)(1) first alternative of the Security and Public Order Act for the Land Hesse and Article 49(1) first alternative of the Act on Data Processing by the Police for the Land Hamburg.[55] These provisions, which authorise the police to process personal data through automated data analysis and automated data interpretation, respectively, were found to violate the right to informational self-determination as expression of the general right to personality (Articles 2(1), 1(1) of the German Constitution) to the extent that the respective tools may be employed as a precautionary measure to prevent specific criminal acts.[56] In its proportionality assessment, the Court argued that ‘[g]iven the particularly broad wording of the powers, in terms of both the data and the methods concerned, the grounds for interference fall short of the constitutionally required threshold of an identifiable danger’ (emphasis added).[57] In this context, the Court examined the severity of interferences arising from automated data analysis or interpretation, both on their own and those resulting from prior data collection. This examination considered the principles of purpose limitation and change in purpose (e.g., using the same data to prevent future crime).[58] As regards the latter case, the Court stressed that a change in purpose would be permissible: ‘if the data of the police authorities concerns information that results, in an individual case, in a specific basis for further investigations aimed at detecting comparably serious criminal acts or averting impending dangers that, at least in the medium term, threaten weighty legal interests that are comparable to the legal interests whose protection justified the collection of the data in question’ (emphasis added).[59]

Regarding the interferences arising from automated data analysis or interpretation as such, it was stressed that further intrusions to individual rights result from processing large amounts of complex information. Such processing may enable the creation of a full profile based on algorithmic assumptions about relationships and connections surrounding the person concerned. Importantly, the information load easily renders the protection granted by the principle of purpose limitation inadequate.[60] Considering the breadth and depth of the intelligence information acquired by means of automated data analysis or interpretation, the Court also cautions to take into account the margin of error, the likelihood of discrimination, and the difficulties inherent in retracing the links generated by the software.[61] Interferences with the right to informational self-determination of this kind call out for higher protection requirements that apply to intrusive and covert surveillance. Hence, the threshold of an identifiable danger to particularly weighty legal interests (e.g., life, limb or personal freedom) is appropriate, given the number of adverse consequences affected individuals have to face.

These findings are particularly important concerning the distinctions introduced above concerning prediction/prevention- and repression-driven uses of AI, considering that one of the data analysis tools in question (i.e., the one employed by the police in Hesse) was by definition ‘always linked to a criminal act that has already been committed or, at a minimum to a suspicion that a criminal act has been committed’ but also allowed for a prognosis to be made on this basis.[62]

Against this backdrop, decisions like the one to delete AI systems employed for crime analytics purpose from point 6 of Annex III to the AIA require caution and clear checks and balances. This is particularly the case to the extent that the typology of high-risk automated law enforcement systems in AIA may be purpose-oriented, but, in reality, such systems are susceptible to repurposing. The risks associated with repurposing become more apparent in an environment shaped by power imbalance. This suggests the importance of the power to amend Annex III in accordance with Article 7 AIA. The latter allows for the amendment of Annex III by adding or modifying use cases of high-risk AI systems. This is applicable when these systems are intended for use in any areas listed in Annex III and have an adverse impact on fundamental rights that is equivalent to or greater than the risks posed by the high-risk AI systems already mentioned in Annex III. The AIA drafters have identified criteria such as data volume, disproportionate impact, and power imbalance (Article 7(2) lit. c, f, h AIA) to evaluate the extent of adverse impacts on fundamental rights. Notably, AI systems employed for crime analytics purposes already meet these criteria.

Instead, the approach adopted in the final text of the AIA fails to acknowledge the AIA’s potential to reshape the way LEAs exercise important tasks in the area of criminal law.[63] By contrast, the drafters of the EP Resolution of 6 October 2021 on ‘AI in criminal law and its use by the police and judicial authorities in criminal matters’ insightfully stressed that using AI in criminal law is not a mere technical feasibility, but ‘rather a political decision concerning the design and the objectives of law enforcement and of criminal justice systems’.[64] In that sense, the transition from high-risk to minimal risk in law enforcement settings should be viewed as a unilateral decision. Compared to the ‘other extreme’, namely an absolute prohibition or the automatic classification of all AI systems used for law enforcement purposes as high-risk AI, there has been a middle way: a ‘case-by-case analysis’ approach for AI systems’ uses like crime analytics. This approach would have allowed for more scrutiny and, importantly, prevented reliance solely on voluntary self-regulation measures in such a critical area of public governance.

3.   What standards for defence rights in the era of AI?

While the EU was progressing with its efforts to regulate AI, the European Agency for Fundamental Rights (FRA) stressed on several occasions the need for a rights-based approach that would ‘[guarantee] a high level of protection against possible wrongdoing related to new technologies’.[65] This approach to AI regulation should be holistic in the sense of ensuring respect for the full spectrum of fundamental rights.[66] As part of this holistic approach, the following analysis focuses on defence rights with the ultimate goal of suggesting new ways to protect them in the era of automated law enforcement. The scope of this analysis is limited; it does not address AI-based consumer products that – even though there is no intention to use them for evidentiary purposes – may convey a data-driven message that could ‘intrude’ criminal proceedings as a piece of evidence (e.g., driving assistants’ output).[67] Instead, the analysis here focuses exclusively on stand-alone AI systems employed in law enforcement settings based on the above-explained classifications in the context of the AIA. 

3.1.  Implications for defence rights

The EU MSs have to respect the right to a fair trial within the framework of the ECHR (Article 6) and the CFR (Articles 47–48). Further, they have to abide by EU secondary law that, as will be shown below (Section 3.2.2), has been adopted to establish minimum criminal procedural safeguards. At the level of the European Court of Human Rights (ECtHR), the jurisprudence of which the Court of Justice of the EU (CJEU) primarily follows in the area of fundamental rights, the primary concern is ‘to evaluate overall fairness of the criminal proceedings’.[68] The concept of charge is understood autonomously and encompasses the ‘official notification given to an individual by the competent authority of an allegation that he has committed a criminal offence’ (emphasis added). It further includes the test of whether the suspect’s situation has been affected substantially.[69] The criminal nature of the charge is assessed on the basis of the criteria set out in the ECtHR judgment of 8 June 1976 in Engel and Others v. the Netherlands, namely: a) classification in domestic law; b) nature of the offence; and c) severity of the penalty that the person concerned risks incurring.[70] Therefore, the fair trial guarantees[71] are applicable throughout the entire criminal procedure, encompassing the pre-trial stage, the sentencing process and, finally, the appeal proceedings against conviction or sentence that are in fact provided and the execution of the judgment. Accordingly, AI-supported judicial decision-making must equally comply with the fair trial guarantees enshrined in Article 6 ECHR (and arts. 47-48 CFR) throughout these various stages.[72]

This understanding of criminal proceedings leaves out of the protective scope of the fair trial guarantees the use of AI systems before a criminal charge is pressed, namely cases where the focus lies on a crime not yet committed and coercive measures taken on the basis of the crime forecast. Examples include police stops and searches (not based on concrete suspicion against a person) or ‘mere’ labelling of a person as presenting a heightened risk of criminality.[73] To date, this concern predominantly refers to predictive policing. Yet, it remains to be seen whether, in the future, AI-based crime forecasts can support the charges in criminal proceedings before courts, namely in cases where the prophecy (crime commission) appears to have been fulfilled. In the absence of concrete evidentiary rules, similar doubts arise in the case of recidivism algorithms. For instance, in the case of the COMPAS risk assessment, the Supreme Court of Wisconsin stressed that risk assessments may not be used as the determinant factor in sentencing. In particular, the software may not determine whether someone will be incarcerated or how severe his/her sentence will be. Additionally, the choice of a specific sentence has to be justified.[74] On the contrary, AI systems intended to be used for crime analytics purposes are designed to inform prosecution files and, thus, criminal proceedings; that is, their input may be classified as evidence, depending on the applicable national evidence admissibility rules (or the lack thereof).

Should the algorithmic output be admitted as evidence, equality of arms and the right to an adversarial hearing – two meta-rules arising from the right to a fair hearing (Article 6(1) ECHR; Articles 47-48 and 52(3) CFR) and partially overlapping to the special safeguards set out in Article 6(3) ECHR – are of central importance. These are the following: first, equality of arms presupposes that each party to a criminal trial is given a reasonable opportunity to present their case under such conditions that do not place them at a disadvantage vis-à-vis their opponent.[75] Second, the right to an adversarial hearing means that the parties must not only have knowledge of all evidence adduced and all observations filed but also have the possibility to comment on the evidence and observations to be able to influence the court’s decision.[76] Over time, the ECtHR has elaborated on these two principles. The following guidelines serve as the starting point for formulating criminal procedural safeguards in automated law enforcement settings:

-      Evidence admissibility rules must grant the defence the possibility to challenge expert findings effectively, in particular by introducing or obtaining alternative opinions and reports. This possibility may be impeded in cases where the prosecution fails to disclose the technical details on which an expert report is based. Consequently, the principle of equality of arms under Article 6(1) ECHR is compromised.[77]

-      Limited access to the case file on grounds of public interest may also amount to a breach of the principle of equality of arms. Respect for defence rights requires that access limitations must not prevent the evidence from being made available before the trial and the accused from being given the opportunity to comment on it through his/her lawyer in oral submissions, even if, in certain cases, specific justifications may be required to access a particular document in the file.[78] Disclosure of evidence related issues are also dealt with under the right to an adversarial hearing and to adequate time and facilities for the preparation of defence enshrined in Article 6(3) lit. b ECHR.[79]

-      The evidence that is relied upon for determining a person’s guilt or the evidence that could have enabled either their exoneration from guilt or the reduction of the sentence, is deemed relevant for exercising the right to an adversarial hearing. The evidence related to the admissibility, reliability and completeness of evidence directly relevant to the facts at issue (cf. the case of AI systems intended to be used for evaluating the reliability of evidence in criminal proceedings) is equally considered decisive for exercising the right to an adversarial hearing.[80] Therefore, national criminal procedural systems in which ‘the prosecuting authorities themselves attempt to assess what may or may not be relevant to the case, without any further procedural safeguards for the rights of the defence, cannot comply with the requirements of [article 6(1) ECHR]’.[81]

-      When competing interests justify withholding certain pieces of evidence (for instance, to preserve the fundamental rights of another person (e.g., victim), any difficulties the defence encounters due to these limitations must be counterbalanced sufficiently by the procedures followed by the judicial authorities.[82]

-      In the case of electronic data, the defence shall be able to get involved in the process of laying down the criteria for determining what may be relevant for disclosure, given that the prosecution may have access to a multitude of such information. Additionally, refusing the defence to have further searches of tagged or identified data carried out ‘in principle raises an issue with regard to the provision of adequate facilities for the preparation of the defence’.[83]

The inability (including the lack of sufficient means and knowledge) to confront evidence against oneself may also be assessed through the lens of the right ‘to examine or have examined witnesses against him’ (Article 6(3) lit. d ECHR), provided one adopts the autonomous and broad interpretation of the term ‘witness’ in ECtHR jurisprudence. The latter term encompasses any evidence (whether AI-generated/evaluated or not) that is used at trial and appeal proceedings and can serve as the basis for a conviction.[84]The ability to rebut evidence may likewise be examined in the light of the presumption of innocence (Articles 6(2) ECHR; 48 CFR), and the burden of proof in terms of a meta-rule arising therefrom. Even if the burden of proof lies on the prosecution or the court, the defence shall be able to address a(n) (indirect) shift of the burden, particularly where EU secondary law, following the ECtHR jurisprudence,[85] ‘leaves space’ to legal and factual presumptions of criminal liability (recital 22 of Directive (EU) 2016/343) (see below).[86] Should this evidence/presumption originate from an AI system (e.g. an AI-powered dynamic knowledge graph that suggests that A is the beneficial owner of company B that was involved in suspicious monetary transactions) employed for law enforcement purposes, the main obstacle to achieving the level of protection required by Article 6 ECHR and Articles 47-48 CFR is algorithmic opacity.[87] The latter may have multiple origins: a) intentional corporate or institutional self-protection and concealment, including secrecy of police actions; b) lack of technical skills required to scrutinise the algorithmic output amongst the judiciary and the defence (particularly relevant in cases where resort to experts may be limited by the procedural framework, for example for reasons of trial economy); c) mismatch between algorithmic sophistication and capabilities of human brains.[88]

Lack of insight into AI systems’ inner workings leading to (the production of) a certain output may adversely impact defence rights that go beyond the contestability of (the reliability of) evidence stricto sensu. The fairness principle enshrined in Article 6(1) ECHR encompasses the right to reasoning of judicial decisions with the aim of demonstrating that the parties have been heard. This obliges the judges to reason on the basis of objective arguments, and to preserve the defence rights.[89] Reasoned decisions are equally important concerning the defence right to appeal.[90] This must remain valid when AI systems generate evidence admitted in criminal proceedings as well as when they are intended to ‘merely’ support judicial decision-making.[91]

Lastly, the use of AI systems for law enforcement purposes is equally questionable in light of the so-called in dubio pro reo principle. According to the latter, any doubt as to the question of guilt is to benefit the suspect or accused person. Hence, this principle is considered a specific expression of the presumption of innocence (Articles 6(2) ECHR; 48 CFR), and it is violated when someone is found guilty but the decision is not sufficiently reasoned. The same holds true when the defence faces an extreme and unattainable innocence threshold that eliminates the slightest prospect of success.[92] This becomes relevant not only for AI systems that generate evidence but also for systems that support decision-making. Their use may adversely influence legal reasoning and/or present the defence with unsurmountable challenges as to proving otherwise, particularly concerning the resources required. In the case of algorithms supporting human decision-making, the decision at issue may not be about guilt but about dangerousness and (related) coercive measures taken in criminal proceedings (e.g., recidivism algorithms). In such cases, the question raised is whether the individual is guilty of a potential crime.[93] Such an automated judgment may, however, disregard the fact that crime results from human dispositions that cannot be pre-determined and, instead, are susceptible to change over time.[94] Further, the operation of the in dubio pro reo principle is challenged in a context where there seem to be no space for doubt. This can be the case to the extent that AI output appears to convey the air of objectivity.[95] In this context, the algorithmic output seems to ‘beat’ those apt to lie or make a mistake, namely human beings.[96] And even in cases where there is still doubt (e.g., in cases where there is information concerning the fallibility rate of a certain AI system), it remains to be seen how much doubt suffices for exonerating the defendant in the times of AI.[97]

3.2.  How to safeguard defence rights in times of automated law enforcement

Prior to and during the trilogue negotiations on the AIA, there have been several AI policy initiatives with a particular focus on AI ethics. Although the respective guidelines often correlate with human rights standards, the way they contribute to their protection remains rather unclear.[98] Such governance gaps may be filled with self-regulatory instruments adopted by non-State actors – yet again with questionable (and insufficient) impact from a human rights perspective.[99] It is the same private actors that do not always welcome legally mandated regulatory standards and enforcement in the AI field and, instead, consider that conventional regulation hampers innovation.[100]

The long-awaited AIA promised to fill this gap. However, it does so only partially. As already mentioned, the AIA contains a list of fundamental rights, including defence rights (recital 48), that are to be protected and seeks to primarily regulate those AI systems that may jeopardise these rights or have an adverse impact on human health and safety.[101] The requirements set out in Chapter III, Section 2 of the AIA are formulated in broad terms (‘Requirements for high-risk AI systems’), whereas the crafting of more concrete standards is entrusted to European Standardisation Organisations (ESOs) pursuant to Article 40 AIA.[102] Standardisation comes with the promise of promoting rapid technology transfer, ensuring interoperability of AI systems, enabling a uniform approach to IT security, and establishing uniform requirements that will support the implementation of legal requirements and ethical values, including the protection of fundamental rights (Article 40(3) AIA).[103] However, it is rather unclear how legal requirements and ethical values can be ‘translated’ into technical standards.[104]

Next, several practical burdens stand in the way of AI standardisation. According to Ebers, these burdens include: the rapid changes AI development is subject to (these changes make technical standards potentially obsolete in a short period of time); the lack of consensus concerning major ethical and legal questions pertaining to AI regulation (such as the definition of bias);[105] the learning capabilities of advanced AI systems based on machine learning and the probabilistic nature of many AI systems, making the establishment of quality criteria hard to achieve; the inability to test the quality of an AI system in abstracto, namely without applying it to a concrete set of input data; the variety of industries and sectors that may use AI; and the socio-technical nature of AI systems, making it necessary to test the systems’ quality in relation to the context they are embedded in.[106] Additionally, standardisation as a regulatory ‘resort’ reveals a strong tendency towards delegating significant regulatory functions to private stakeholders. This very trend has the potential to undermine State power significantly.[107] Of particular concern are the lack of democratic oversight and accountability, the missing or limited possibilities of other relevant stakeholders (e.g., EP, EU MSs, civil society organisations) to participate in and influence the standardisation process.[108] The latter may be due to procedural restraints, lack of expertise or insufficient resources, and the lack of judicial control over harmonised standards.[109]

The alternative to outsourcing standardisation to ESOs[110] would be the adoption of legally binding provisions for the essential requirements high-risk AI systems as defined in the AIA have to comply with. These provisions would include clear answers to questions of, for instance, how much transparency is required and which algorithmic discriminations are to be considered disproportionate and how others are to be mitigated (depending on, inter alia, contextual parameters). These answers should then be the subject of standardisation.[111] Considering the far-reaching legal and societal consequences of some AI systems, as Ebers suggests, this co-regulation model, namely the combination of comprehensive legally binding provisions and standardisation input, should be complemented by an improvement of the standardisation procedure with the aim of its democratisation.[112] This article argues that these two correction measures should be coupled with legislative action in the area of individual rights, considering that the AIA – despite the noble intention of its drafters to protect fundamental rights – does not contain any directly enforceable rights for individuals. The only exception is the newly introduced right to explanation of individual decision-making (Article 86 AIA),[113] which is granted to affected individuals subject to a decision taken by the deployer on the basis of a high-risk AI system’s output, which produces legal effects or similarly significantly affects that person in a way that they consider having an adverse impact on their health, safety or fundamental rights (Article 86(1) AIA). In particular, this right refers to the use of high-risk AI systems listed in Annex III, with the exception of systems listed under point 2 thereof. Hence, the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken as enshrined in the AIA does not cover the whole spectrum of automated law enforcement (e.g., AI-powered crime analytics). This means that the affected individuals will have to rely upon general fundamental rights laws, namely legislation that does not take into account the particularities of the AI reality, to exercise their right to effective remedy (Articles 13 ECHR, 47 CFR).

Against this backdrop, the following analysis pursues a twofold goal: First, it discusses how technical design can be informed by legal requirements on the protection of fundamental rights, thereby using defence rights as an example (Section 3.2.1). Second, it explores how other EU secondary and sector-specific laws on defence rights can and should be amended to address the novel AI realities (Section 3.2.2). 

3.2.1.   Defence-rights-by-design

In its Guidelines for Trustworthy AI, the AI HLEG describes several technical methods that can be incorporated in the phases of design and deployment of an AI system, including that of ‘Ethics and rule of law by design’. In particular, the AI HLEG refers to ‘value-by-design’ methods that can ‘provide precise and explicit links between the abstract principles which the system is required to respect and the specific implementation decisions’.[114] Similarly, the OECD classifies ‘human rights by design’ as a key AI principle dictating that ‘AI systems should be designed in a way that respects the rule of law, human rights, democratic values and diversity’.[115]

Although not all ethical values are translated into legal principles,^[116] key ethical values are embodied by human rights. Hence, ethically aware AI must respect human rights in a tangible way and, at the same time, consider the particularities of the societal context the system in question is embedded in.[117] To achieve this goal, it is necessary to ‘translate [otherwise abstract ethical values and] fundamental human rights into context-dependent design requirements through a structured, inclusive, and transparent process’.[118]

The term ‘design for values’ serves as an umbrella term that encompasses different ethics and technological design methods, such as the ‘value sensitive design’[119] and the ‘participatory design’,[120] the common denominator of which is that design requirements are socio-technical.[121] Value specification as a fundamental component of these methodologies stands for the gradual translation of abstract values into specific design requirements.[122] As part of this process, values first expand into (legal) norms that are in turn specified into socio-technical design requirements.[123] In this sense, value specification should be context-dependent.[124] Thus, it is important to spot the legal order within which this design model would operate to the extent that the legal system impacts on the applicable legal norms that embody the values in question. The same applies to the specific societal context a technology is embedded in. Crucially, there is always a certain margin of appreciation as to the optimal design requirements. Yet, all relevant stakeholders (from technology developers and end-users, policy- and law-makers, to civil society and affected individuals) should get the opportunity to initiate a debate as to the (dis-)advantages associated with each design option.[125]

Such a design process can be further informed by the tripartite methodology of ‘value sensitive design’, which encompasses: a)conceptual investigations leading to the identification of the relevant stakeholders and values for the technology in question; b)empirical investigations pinpointing the needs, views and experiences of the stakeholders in respect of the technology and the values the latter implicates; and c) technical investigations intended to implement and evaluate technical design solutions that are aligned with the values and norms the conceptual and empirical investigations have surfaced. These investigations should be intertwined.[126]

Human rights can be incorporated into this methodological approach at the level of value selection and norm identification [point 1) of the ‘value sensitive design’]. Concerning the development of AI systems to be employed for law enforcement purposes in the EU MSs, the values that should guide the technological design process are to be extracted from the ECHR and the CFR.^[127] This is because both the Convention and the Charter provide for ‘a well-established analytical framework through which tension and conflict between rights, and between rights and collective interests of considerable importance in democratic societies, are resolved in specific cases through the application of a structured form of reasoned evaluation’.[128] This framework is further informed by and developed through rulings issued by national or supranational judicial authorities, such as the ECtHR and the CJEU. (The corpus of) these rulings can help those participating in the design, development and implementation of AI systems navigate human rights compliance.[129]

Using the value of fairness as an example, Articles 6 ECHR and 47–48 CFR can serve as guidance (1^st level of value specification). The value of fairness in criminal proceedings encompasses, inter alia, the right to be presumed innocent (Articles 6(2) ECHR; 48 CFR) in terms of a right that is granted to everyone and can be invoked by ‘natural persons who are suspects or accused persons in criminal proceedings [… and] at all stages of criminal proceedings’ (Article 2 of Directive (EU) 2016/343) (2^nd level of value specification). If bias intrudes criminal proceedings (e.g., by means of taking into consideration the output of an algorithmic system trained on the basis of non-representative input data), the presumption of innocence stops applying equally to everyone (a case where the presumption of innocence and the right to non-discrimination may overlap).[130] This may be the case when an AI system employed to assess personality traits and past criminal behaviour to inform a police investigation or a criminal conviction takes into consideration, among other parameters, the ethnicity of the person into question. Applying a human-rights-by-design approach to such an algorithm would require taking into account the negative implications for the right to be presumed innocent from the earliest software design stage and to disregard those design options that obviously have a negative impact on that right. These demands could, for instance, be translated into the following coding rules (3^rd level of value specification): Ethnicity data may neither be collected nor evaluated. If, however, ethnicity data is collected or evaluated, a) justify the necessity of this action for performing a certain data processing operation, b) use a log to track the processing of ethnicity data (logging and black-box recording), and c) explain and report the calculation method and the indicators used to incorporate ethnicity data into the individual risk assessment report. If these justifications, logs and reports inform criminal proceedings, they should be made available and accessible to the data subject concerned in a language that they understand, so that they can prove whether their right to be presumed innocent has been violated. 

Next, as part of the design-for-human-rights process, it is important to ask whether and to what extent an AI system will replace the human agent who previously performed a certain activity alone. Hereby, the normative and institutional purposes (of using the respective AI system) should be considered.[131] In this regard, the so-called ‘nature-of-activities’ approach suggests to divide the activity in question into goal-directed and practice-oriented aspects and to identify the parts that present a special social value.[132]This approach should surface significant human-to-human interactions and, thus, bridge socio-technical gaps.[133] Also, it shouldenable an informed decision as to which parts of a certain activity may be fully automated and which ones should be entrusted to human agents. From a socio-technical point of view, it may also indicate where an innovative human-AI interaction may be the optimal solution.[134]

For instance, AI-driven crime analytics can be employed for automating the task of gathering, organising and visualising information for a multi-layered criminal scheme. This activity is usually performed by trained, experienced police agents (where needed, in collaboration with staff of the Prosecutor’s office), often (but not necessarily) with the help of technical systems.[135] AI-driven crime analytics aims to gather information of potentially probative value and to sort it out for crime prosecution purposes in a speedy and effective way (goal-oriented aspects). In democracies, however, police investigations should not concentrate solely on effectiveness criteria. Instead, they should primarily strive to create a relationship of trust between the State and its citizens. This trust relationship is reflected in the value choice that citizens do not live under mass surveillance[136] and manifests itself, inter alia, in the presumption of innocence.[137] Accordingly, law enforcement should serve to safeguard both security and freedom, and criminal investigations need to be based on concrete suspicion, not on vague assumptions or statistical correlations found in historical datasets.[138]

The requirements outlined above can be incorporated in a governance framework for AI that goes beyond the design and development of AI systems. In other words, an AI governance regime should also address the future use of AI systems. This use needs to be regulated by law and overviewed by independent regulatory authorities equipped with proper resources and appropriate investigative and enforcement powers.[139] As Yeung et al. stress, such a regime of AI governance must foresee input from technical and human rights experts, affected stakeholders and the general public.[140] Hence, the design, development and use of AI systems intended for use in law enforcement settings require deliberations from LEAs, the judiciary, defence lawyers, and civil society organisations.

3.2.2.   Defence rights in (EU secondary) law

Even when shaped on the basis of human rights considerations, AI design requirements cannot replace normative standards on the protection of individuals who are affected by AI systems. As regards defence rights, the EU legislator has further specified the obligations of national lawmakers in the so-called Roadmap Directives adopted on the basis of Article 82(2) lit. b of the Treaty on the Functioning of the EU (TFEU). The latter article confers on the EU the competence to adopt minimum rules on individual rights in criminal proceedings. As Mitsilegas explains, this competence is a rather functional one, meaning that it is justified ‘only to the extent necessary to facilitate mutual recognition’ (cf. Article 82(1) TFEU) in order to enhance mutual trust.[141] Nonetheless, Article 82(2) TFEU has been used to provide for human rights standards ‘applicable across the board, embracing not only cross-border cases involving mutual recognition, but also purely domestic cases’ (emphasis added).[142]

Considering the obligation of EU MSs to respect the right to a fair trial as enshrined in Article 6 ECHR and Articles 47–48 CFR and given the implications of automated law enforcement for defence rights (Section 3.1), the focus here lies on a matrix of provisions included in Directive 2012/13/EU on the right to information in criminal proceedings and in Directive 2016/343/EU on the strengthening of certain aspects of the presumption of innocence and of the right to be present at the trial in criminal proceedings. In particular with regard to the AIA typology of automated law enforcement systems (Section 2), the material scope of the following provisions should be re-visited, preferably at central (i.e., EU) level, given that AI expertise is often limited or at least diverse at national level. 

In Directive 2012/13/EU, the right to information about procedural rights (Article 3) is complemented by the right to information about the accusation (Article 6) and the right of access to the materials of the case (Article 7).[143] According to Article 6 of the Directive, the EU MSs shall ensure that: ‘suspects or accused persons are provided with information about the criminal act they are suspected or accused of having committed [, and that] information shall be provided promptly and in such detail as is necessary to safeguard the fairness of the proceedings and the effective exercise of the rights of the defence’ (paragraph 1, emphasis added). 

The same applies to those arrested or detained concerning the reasons for their arrest or detention, including the criminal act they are suspected or accused of having committed (paragraph 2). Next, at the latest on submission of the merits of the accusation to a court, detailed information must be provided on the accusation. This includes information on the nature and legal classification of the criminal offence, and information on the nature of participation by the accused person (paragraph 3).

Pursuant to Article 7 of Directive 2012/13/EU, these information rights are to be coupled with free-of-charge access for those arrested or their lawyers to the ‘documents related to the specific case in the possession of the competent authorities which are essential to challenging effectively, in accordance with national law, the lawfulness of the arrest or detention’ (paragraph 1). Further, ‘all material evidence in the possession of the competent authorities, whether for or against suspects or accused persons’[144] must be made accessible to those persons or their lawyers in order to safeguard procedural fairness and enable the defence preparation (paragraph 2). Provided this does not prejudice the right to a fair trial, derogations may be justified in cases where access to certain materials ‘may lead to a serious threat to the life or the fundamental rights of another person or if such refusal is strictly necessary to safeguard an important public interest, such as in cases where access could prejudice an ongoing investigation or seriously harm the national security of the Member State in which the criminal proceedings are instituted’ (paragraph 4). In such cases, the decision to refuse access must be taken by a judicial authority or at least be subject to judicial review.

These procedural rights are further ‘backed up’ by Article 8 of Directive 2012/13/EU. This article sets out express provisions on verification of actual respect of these rights (by means of the recording procedure specified in national law) and remedies.[145] In particular, remedies shall be provided in cases where the competent authorities fail to provide the information needed by suspects, accused persons or their lawyers to exercise the right to defence.

In the face of automated law enforcement, the material scope of the information and access rights mentioned above should be expanded in the sense that they require information on whether and which AI systems have been employed. In addition, the defence should have a legally guaranteed access to a clear evidence trail that gives information on the respective design features of the AI system used, the assessment procedure and the outcome thereof. Further, meaningful information should be provided on the system’s operation and implementation. Such requirements for traceability and accessibility may enable the suspect or accused person to exercise the right to defence effectively (e.g., by inviting an external expert to audit the respective material). Access must not be made dependent on the will of the corporate actors that are involved in the development of the AI system in question. Further, refusal of access may not be justified ‘automatically’ in the name of police secrecy. Rather, it must be decided by a court or be subject to judicial review, just as Article 7(4) of Directive 2012/13/EU demands.

Access to meaningful information is equally critical for respecting the right of suspects and accused persons to be presumed innocent according to Article 3 of Directive (EU) 2016/343 and other rights tightly linked to it. Those include the duty of EU MSs to ensure that ‘the burden of proof for establishing the guilt of suspects and accused persons is on the prosecution’ (Article 6(1) of Directive (EU) 2016/343). The possibility of shifting this burden to the defence has been included in recital 22 of the Directive, yet only for cases in which the prosecution resorts to presumptions of fact or law concerning the criminal liability, provided such presumptions are ‘confined within reasonable limits, taking into account the importance of what is at stake and maintaining the rights of the defence, and the means employed should be reasonably proportionate to the legitimate aim pursued. Such presumptions should be rebuttable and, in any event, should be used only where the rights of the defence are respected’. 

In times of automated law enforcement, this means determining whether such presumptions can be generated directly by AI systems and testing the use of AI-driven tools to establish these presumptions as to the extent the defendant still has adequate means to prove otherwise. Additionally, the EU lawmaker should acknowledge that the burden-of-proof rule and the right to information overlap to the extent that one-sided access to voluminous data that originates from various sources, which the defendant does not have access to by default, may increase the ‘innocence threshold’ significantly or even establish a presumption of guilt.[146]

In addition, Article 6(2) of Directive (EU) 2016/343 stipulates that ‘any doubt as to the question of guilt is to benefit the suspect or accused person, including where the court assesses whether the person concerned should be acquitted’. This provision may have a significant impact on national criminal justice systems, ‘particularly since the concept of doubt and its threshold is not defined in the Directive’.[147] AI adds an extra layer of complexity concerning the link between doubt and AI fallibility rates. In that sense, disclosure of false positive rates to protect the rights enshrined in Directive 2012/13/EU is to be seen in a different light, namely that of ensuring the effective protection of the in dubio pro reo principle by means of further delineating the notion of doubt.[148]

Given the significant differences national evidence admissibility rules present,[149] Article 10 of Directive (EU) 2016/343 obliges the EU MSs to ensure that suspects and accused persons have an effective remedy if their rights enshrined therein are breached. As with the right to information, these remedies are not further specified but left to the discretion of the national legislator.

The added value of Directive (EU) 2016/343 predominantly lies in addressing the challenges for fundamental rights protection that stem from ‘the blurring of the boundaries between the judicial determination of guilt or innocence on the one hand and labelling on the other hand’.[150] It does so by introducing not only rules of judgment (Articles 6–7) but also rules of treatment throughout the criminal proceedings with regard to public references to guilt (Article 4) and the presentation of suspects and accused persons (Article 5).[151] Today, labelling may be exacerbated when resort is taken to AI-driven profiling and risk assessment tools, not only as part of criminal proceedings but also outside criminal proceedings[152] (e.g., for predictive policing purposes). The scope of the term ‘criminal proceedings’ (and, thus, that of the Directive itself) has already caused confusion, particularly with regard to proceedings the legislator has not formally qualified as criminal. Thus, several scholars claim that the term ‘criminal proceedings’ is an autonomous notion of EU law in need of interpretation by the CJEU.[153] While confusion has arisen primarily with regard to punitive administrative or civil proceedings, the EU lawmaker (or at least the CJEU) should also take a stand on whether the questioning of innocence in predictive policing settings may justify protection under Directive (EU) 2016/343 and, if yes, to what extent.[154]

Equally important is the relation between the aforementioned EU secondary laws and national laws, particularly when taking into account the enlargement of the Directives’ scope and their applicability to domestic cases.[155] The enforcement of the safeguards they provide for can be enhanced in different ways, including: a) through their direct effect that allows affected individuals to invoke and claim rights directly before the national courts in the case of a non-timely transposition or inadequate implementation of the Directive; b) through the availability of effective remedies in domestic laws for the violation of the rights enshrined in the Directives; c) through monitoring duties of the Commission and the possibility of initiating infringement proceedings in case of a MS’s failure to transpose or implement the Directive; and d) through the power of the CJEU to interpret EU law under the preliminary reference procedure.[156]

Lastly, the defence rights should be further complemented by rules on the admissibility of AI-generated evidence. In this regard, one should acknowledge that civil-law jurisdictions usually lack express rules on evidence admissibility, including exclusionary rules. As a consequence, all relevant evidence is admissible as part of the courts’ truth-seeking mission, provided that the evidence has not been acquired in a severe breach of human rights.[157] The complexity of AI-generated evidence calls for revisiting existing frameworks to ensure that only evidence that is in compliance with human rights standards enters the courtroom.[158] This presupposes that AI-generated evidence is ‘built’ to ensure formal assessment. The latter should cover the whole lifecycle of this evidence, including, for instance, the training of the algorithm or past adverse incidents (e.g., false positives), and be based on meaningful information. This information must be accessible to and interpretable by the judiciary, the defence and, where necessary, external experts. The accessibility and interpretability of AI-generated output shall ensure that defence rights take the potential and capabilities of the machine into account. Further, due regard must be paid to the power imbalance and fallibility of the system, especially in criminal law matters, considering that the implications for affected individuals may be far-reaching. Lastly, such evidentiary rules should be coupled by stringent requirements for the reasoning of court decisions.[159]

4.   Conclusion

The EU set itself the ambitious goal of becoming the first regional actor worldwide to regulate AI though a dedicated legal instrument, namely the AIA. Within the latter, a typology of automated law enforcement systems has been crystalised as part of the risk-based approach to AI systems. In this context, certain systems have been deleted from the list of high-risk systems, including AI systems intended to be deployed for crime analytics. Worryingly, this choice neither acknowledges the risk of repurposing nor pays attention to the specificities of the law enforcement environment. On the whole, the AIA – though constituting a welcome initiative, especially when compared to the often too elastic ethical standards on AI – does not live up to its ambition of thoroughly and effectively protecting fundamental rights. Maybe the most prominent lacuna concerns the area of individual rights themselves – with the exception of Article 86 AIA. Against this backdrop and with a focus on the law enforcement context, this article argued that AI systems should be designed, developed and deployed in ways that meet the already established defence rights standards. To attain this goal, input is required from multiple stakeholders (including judges, prosecutors, defence lawyers and civil society).[160]Equally necessary is a twofold, mutually reinforcing learning strategy: On the one hand, tech designers, developers and engineers must set out to understand human rights protection standards and the underlying normative framework; on the other hand, human rights experts must acquire technical skills to be able to fulfil advisory and assessment duties.[161] The AIA should thus promote human rights consciousness at the design level. Further, EU legislative initiatives in the area of fundamental rights should complement the AIA. Regarding defence rights in particular, this would mean reshaping the protective scope of the respective EU secondary laws in light of novel AI realities. In a next step, the national legislator must then transpose and implement the enhanced defence rights standards contained in EU Directives and thus contribute to an enriched protective ensemble.