The AI Act fosters trustworthy AI in the EU. Learn about the challenges faced by AI today and how the AI Act can help.

The main purpose of the AI Act (EU 2024/1689) is to assign obligations and responsibilities for high-risk AI systems, and the organizations that operate them. While the recent AI boom is characterized by a “move fast and break things” / “buyer beware” mentality, there is a very real need to consider potential risks and build AI systems that are resilient against eventual misuse and abuse. The problem is that many developers (and users) of these systems aren’t aware of the issues at all!

Whether an AI system is a high risk one actually depends on how it is used. The AI Act covers a wide range of areas, including vehicles, medical devices, biometrics, critical infrastructure, education, employment, essential services, law enforcement and democratic processes. In practice, it is often easier to define what is not high risk: systems that pose no significant risk to health, safety or fundamental rights and perform only low-stakes procedural tasks, assist rather than replace human decision-making, or require human review anyway. Open-weight AI models, AI used for R&D, national security or purely personal purposes are also outside of the scope of the AI Act.

The AI Act has been rolled out in phases, with several provisions already in force. In February 2025, prohibitions on certain AI practices took effect, alongside a requirement for providers and deployers of AI systems to ensure a baseline level of AI literacy. In August 2025, rules governing general-purpose AI models, such as large language models, came into force. August 2026 marks the next major milestone, when transparency and watermarking requirements, together with the remaining rules for high-risk AI systems, are scheduled to apply.

However, the Digital Omnibus package on AI (proposed in November 2025 and accepted in June 2026) to strengthen Europe’s AI competitiveness, would relax some of these requirements. It would extend most compliance deadlines to December 2027 for standalone high-risk AI systems and to December 2028 for AI systems embedded in regulated products (such as medical devices), while also postponing the watermarking requirement for AI-generated content until December 2026.

Roles and obligations

The main roles are Provider (developers of AI systems and models), Deployer (users of AI systems, except personal use), Distributor (making AI systems available on the EU market), Importer (see Distributor, but for systems developed outside the EU), Authorized Representative (representative of a Provider to act as a go-between), and Product Manufacturer (Deployer or Distributor of their own product based on an existing AI system).

Going through all the obligations would be beyond the scope of this article (and compliance is not our focus here – there are numerous existing tools for that such as the compliance checker maintained by the Future of Life Institute), so just to sum it up:

• Providers, as per Article 16, must ensure their systems meet certain EU quality and safety requirements (CE marking) and keep necessary records / demonstrate conformity as needed.
• Deployers, as per Article 26, must provide human oversight, monitor the system’s operation, validate its input, and notify the authorities together with the Provider or Distributor in case the system presents a risk.
• Distributors, as per Article 24, must verify the presence of the CE marking and if they suspect that the AI system may not conform to requirements, don’t make it available on the market or, if already on the market, withdraw or recall the product (or ensure that the Provider or Importer does so); they must also inform the Provider or Importer along with the authorities in case a system presents a risk.
• Importers, as per Article 23, must verify the original Provider’s regulatory conformance obligations and the existence of an Authorized Representative, and follow similar rules as the Distributor with respect to non-compliance and risks.
• Providers and Deployers must also ensure their technical staff has ‘AI literacy’ (more on that later) as per Article 4.

The penalties for non-compliance vary depending on the infraction – as typical for similar regulations, it takes the larger number (or smaller, for SMEs) from a flat value or a percentage of the company’s total turnover in the previous year. Deploying or operating AI for explicitly prohibited purposes carries an extreme penalty (35M EUR or 7%), failing to comply with role-based obligations carries a high penalty (15M EUR or 3%), and providing incomplete or misleading information to authorities carries a significant penalty (7.5M EUR or 1%). Reduced versions of these penalties apply to EU institutions themselves, too!

Avoiding the weaponization of AI

When discussing AI risks, some people may think of doomsday rogue AI apocalypse scenarios and similar existential threats. However, it is much more realistic to consider the threat of criminals using AI to facilitate their harmful activities at scale, and authorities abusing AI to perform large-scale surveillance harming citizens’ privacy. Well-known AI security researcher Nicholas Carlini (Anthropic, formerly Google DeepMind) described these threats in his “Machines of Ruthless Efficiency” essay.

This is in line with malicious use of AI observed before the LLM craze: large-scale misinformation was already facilitated by AI in the 2010s and the DARPA Twitter Bot Challenge in 2015 experimented with the use of language models to detect automated misinformation campaigns at scale. So-called ‘deepfakes’ (very realistic synthesized audio or video of a real person) have also been around since then, and are now actively used for disinformation, scams and nonconsensual pornography.

The AI Act directly tries to fight this by forbidding certain uses of AI (that may not be recognized as criminal on their own) in Article 5: subliminal messaging and deception, exploiting the vulnerabilities of individuals or groups of people, biometric categorization, social scoring, predictive policing, scraping photos for facial recognition, emotion recognition in the workplace or education (except for medical or safety reasons), and real-time biometrics. The Digital Omnibus plans to extend this with a targeted ban on AI systems that generate sexual and intimate content without consent.

Transparency requirements are also strict: Article 50 says that (except if used for law enforcement) Deployers must indicate if any content (audio / video / image / text) was generated or manipulated with AI (with exceptions for artistic or satirical audio / video content, or text content that has undergone human editorial control), and must inform people for any emotion or biometric data processing taking place (which must also be GDPR-compliant). In addition, if these systems are deployed in a public context, the AI Office of the EU must perform a fundamental rights impact assessment on the system. At the same time, Providers must inform the users of AI systems whenever they are interacting with an AI (unless the interaction is obvious), and the outputs of their AI systems must be clearly marked as artificially generated or manipulated.

What’s this about ‘general-purpose’ AI?

Whenever people talk about AI today, they typically mean generative AI: systems using deep neural networks to generate content based on user input. Of these, the most significant are large language models that can generate text (including even source code) or multimedia content based on a text prompt. The AI Act calls such complex flexible models ‘general purpose AI’ (GPAI) in Article 51 and defines systemic risk in recital 110. These problems are generally recognized in the AI domain as the main areas of concern within responsible or trustworthy AI. Currently it is not possible to fully eliminate or prevent these issues, but solutions exist to help with detection and mitigation.

Using NIST’s categorization, responsible AI includes seven building blocks – aligning them with the AI Act, we get:

• Accountability and transparency: Establishing accountability is the main focus of the AI Act itself, and transparency towards end-users is also a significant concern as discussed above.
• Fairness with mitigation of harmful bias: Bias is mentioned in multiple recitals, and as per Article 15, reducing the risk of biased output is one of the conformance requirements for AI systems.
• Validity and reliability: In addition to being one of the systemic risks and one of the conformance requirements (Article 15), it is also brought up as a particular risk in the context of law enforcement (recital 59).
• Explainability: In the context of transparency towards deployers, it is one of the conformance requirements in the AI Act (Article 13).
• Privacy: Protecting the information of natural persons via transparency and prohibitions, as discussed above.
• Safety and Security: Article 15 focuses on the security of AI systems, derived from machine learning security issues.

100 (or at least ~40) ways to wreck your AI

Machine learning has faced (and is continuing to face) numerous security challenges. Our article on machine learning security goes in depth about the potential threats – in short, while there may not be exactly “100 ways to wreck you AI”, there is a wide variety of attacks against machine learning systems. To give a quick summary of the four main categories (all of which are mentioned in Article 15 – the last two are grouped together as ‘confidentiality attacks’):

• Poisoning: Putting a ‘backdoor’ in the model by training it with intentionally malicious data so it either becomes useless or behaves abnormally whenever it encounters the backdoor trigger. It was successfully demonstrated (Souly et al, 2025) that just 250 poisoned documents were sufficient to compromise an LLM, regardless of the size of the dataset or the complexity of the model.
• Evasion: Sending an input (an ‘adversarial sample’) to the model that gets misinterpreted and forces the model to make a mistake. Common examples in the LLM context are jailbreaking and prompt injection – and current research (Nasr et al, 2025) indicates that state-of-the-art anti-injection safeguards are essentially useless against a sufficiently skilled attacker.
• Model inversion: Forcing the model to reveal credit card numbers, sensitive information, or source code fragments that were included in the training data (Nasr et al, 2025).
• Model extraction: Also known as model stealing, or model distillation. By making (a huge number of) regular requests to the model and observing its responses, the attacker constructs a clone of the model that performs similarly to the original (Carlini et al, 2024).

Note that while the theory behind each of these attacks goes back to the 2010s, each of them has been successfully demonstrated against real-world large language models recently, highlighting that Providers and Deployers are not sufficiently prepared against these risks.

AI security is software security, in the end

Unfortunately, attackers tend to have a much easier time than defenders – while countermeasures exist to many of the aforementioned threats (e.g. adversarial training against evasion), they are limited in scope. It is not a coincidence that the European Commission has put out a General-Purpose AI Code of Practice document with a 40+ pages long chapter on safety and security. In particular, Appendix 4 lists security mitigation objectives and measures – and one thing that’s immediately apparent is that the recommendations don’t use AI. Mitigations include standard software security measures such as identity and access management, multi-factor authentication and strong access controls to limit access to the model; vulnerability and patch management to reduce the risk of vulnerability exploitation; making sure all software interfacing with the model is reviewed for vulnerabilities; verifying training data to protect against poisoning; and external security assurance via red teaming, bug bounties and security software.

A lot of these are well-known security measures that apply to any software – because AI systems are also software products themselves, and prone to regular software vulnerabilities and design mistakes like any other applications. As an example, security researcher Johann Rehberger demonstrated a ‘Month of AI Bugs’ in August 2025 where he found a different vulnerability or exploit in an agentic AI coding tool each day. Similarly, supply chain attacks against AI systems have been demonstrated, sometimes even using AI models hosted on open platforms like Hugging Face to deliver malware to unsuspecting developers – see e.g. Acronis’ write-up from April 2026.

The most important tool to ensure AI security, however, is awareness. This is why Article 4 of the AI Act requires AI literacy from the staff of Providers and Deployers. In fact, the Digital Omnibus plans to expand this as a larger-scale AI literacy initiative in society as well!

Check out our Machine learning security and Generative AI courses, in which we cover all of the above aspects of AI-assisted software development – and much more.