'Vibe coding' is seemingly taking (or has already taken) the world by storm. Let's see the security implications!

The use of AI has been dominating the discourse among software developers over the last three-four years, with a lot of different modes of usage – first as a chatbot giving suggestions and explanations, useful as an eager-but-bumbling virtual intern to take care of the ‘boring parts’. However, around the start of 2025, the tools (and the models behind them) became good enough to succeed without the need for a human to read, understand and adjust the generated code. This is where the story of ‘vibe coding’ starts, from a tweet by Andrew Karpathy (co-founder of OpenAI):

The term ‘vibe coding’ got so popular that Collins Dictionary named it 2025’s word of the year, even though that definition itself doesn’t match Karpathy’s. Sometimes people use the term to refer to the process of writing any code with AI assistance (even if the programmer is actively involved in the process). A crucial part of vibe coding is, however, blindly accepting AI output without reviewing, understanding, or even just reading it. Eventually AI tools became able to autonomously perform testing and bug fixing as well via so-called agent loops (the possibility was always there, but the models and harnesses weren’t good enough to do it in practice). Nowadays it is possible to prompt a tool with a specification (or even just a vague idea) and let it work on its own to iteratively write, build and debug the code until it’s ready – no human involvement necessary!

Who is vibing and why?

The implications of fully autonomous code generation are vast, and it’s easy to see it as the “endgame” of software development. To paraphrase Douglas McIlroy (and others), the job of a programmer is to program themselves out of their job. Clearly if there is no need for humans to do any programming anymore, programmers have finally succeeded!

Letting users develop custom tools with no or minimal knowledge of programming has been around forever as ‘end-user development’: batch files, spreadsheet macros and so on. Later, low-code and no-code WYSIWYG frameworks emerged to (for instance) allow the creation of personal web pages through a GUI without any detailed knowledge about HTML, JavaScript, or languages like PHP. From an optimistic ‘glass half full’ perspective, AI is a large step forward here to remove barriers and democratize software development. With vibe coding tools, a manager at a company could quickly prototype a tool to help with a particular problem without having to involve company IT or developers!

As for developers: sometimes it may be easier to solve a problem by writing a custom program for that specific purpose. Before the age of vibe coding however, the time and energy investment just wasn’t worth it (see relevant xkcd comic). But this is not the case anymore – a developer can just prompt an agent harness like Claude Code, Codex, or Pi to write, test, and run the program in the background while the developer can focus on much more important tasks.

When the vibes get uncanny

Vibe coding is built on large language models and thus inherits all their issues. Specifically, LLMs can hallucinate and produce incorrect or made-up results, but will be very confident about it – this is inherent to their operation and not something that can be ‘fixed’, as per OpenAI. This, combined with chatbot sycophancy, can inflate the users’ perceptions of their own abilities as well (see Rathje et al, 2025). This is the reason some people started to call AI chatbots jokingly “Dunning-Kruger as a service“. LLM-written text today has its own ‘uncanny valley’ effect too: overstructured articles filled with emoji-laden bullet points and constructs like “it’s not x – it’s y” trigger a mental immune reaction in many readers.

If we look at this overconfidence in the vibe coding context, we get a real problem of developers (or non-technical personnel) generating code they implicitly trust to ‘just work’ without even trying to understand it. Many vibe-coded apps are toy prototypes used by one person in a low-security environment (usually to do a one-off task), which inherently limits the damage they can do. But even then, giving agentic AI tools unfettered access to real systems can present serious risk – see the incident where the Meta’s Head of AI Safety used an agent to manage her email inbox and it ended up malfunctioning and deleting all of her emails. There is an endless stream of such cautionary tales involving OpenClaw and other agentic tools.

Where it is just asking for trouble, however, is taking a vibe-coded prototype and trying to turn it into a product (or handing it off to other developers and asking them to ‘productize’ it). It may be missing critical security features like authentication and authorization, it may unexpectedly fail if more than one client tries to connect to the server, and a lot of the functionality may just be “Potemkin code”: a nice-looking user interface produced entirely by some client-side JavaScript with a nonexistent backend. For a stark example, in January 2026, someone at Cloudflare published a blog post (that itself was suspected to have been LLM-generated – it has been edited since) about a ‘production-ready’ open-source Matrix home server. The code technically compiled and ran, but it was missing crucial security functionality and was littered with hallucinations and TODOs, as reported by JadedBlueEyes on Mastodon. Matthew Hodgson, a Matrix representative, also put out a (very kindly worded) statement that said “[…] to be clear, the code doesn’t yet implement any of Matrix’s core features which allow you to federate safely, and so doesn’t yet constitute a functional Matrix server, let alone a production-grade one which you should consider deploying. It doesn’t model rooms as a replicated graph of events; it doesn’t check permissions or uphold power levels: it’s the equivalent of a filesystem which ignores permissions, or a blockchain which doesn’t implement a consensus mechanism.” Not great.

Can’t we just let the robots ‘vibe it out’ among themselves?

But let’s try to outsource all that to the model, it should know better than us anyway, right? The Platonic ideal of programmers “coding themselves out of a job” is the dark factory – an environment where no humans even need to look at the process other than providing an initial specification. This is hyped up as the final stage of automation in e.g. Dan Shapiro’s blog post. This requires AI tools to (agentically) perform autonomous testing and validation repeatedly until the specification is satisfied. It is pretty well recognized at this point that test-driven development (TDD) is a good match for agentic AI. The Superpowers framework is a pretty popular implementation of something close to a ‘dark factory’. There are many guides for using deterministic methods (unit tests, integration tests) to verify the correctness of the implementation (e.g. the Spec-As-Test Feedback Loop, red/green TDD). So, assuming that we have a miraculous perfect code generating machine, what can go wrong?

The first problem is cognitive debt – a recognized neural and behavioral consequence of LLM use – applied to code: if the developer never has to look at or understand the code, they won’t, which seriously damages maintainability. Even if we (foolishly) assume the program works 100% correctly now and will never need bugfixing or debugging, developers won’t understand how it works, which makes it very hard to figure out future features or directions and make any changes to the system (even if the changes themselves are done by agentic AI). LLMs can do many things, but they cannot make strategic decisions, and they cannot magically speed up human understanding.

The second issue is GIGO (garbage in, garbage out) – it has been with us since the start of computing, and it especially applies here. AI tools will happily generate code based on any specification we give them, and if the specification is faulty, the program will inherit all its faults as design flaws, especially when it comes to non-functional requirements. Even if we (wrongly) assume that the model cannot hallucinate and was trained on bug-free code that is a perfect match for our use case, if we forget about authentication or input validation in the specification, the program will still be inherently vulnerable. And while it’s a good idea to use TDD concepts and deterministic tools to control and verify tool outcomes, if the tests themselves are AI-generated, there is always a risk of something going wrong.

But surely the models/tools/harnesses will get better…

Ever since the explosion of large language models (and everyone equating ‘AI’ with them), there has been a back-and-forth between AI skeptics and AI maximalists. The early years clearly favored the skeptics with language models fumbling even the simplest things, but since the second half of 2025, a lot of the models seem capable enough to succeed even at complex tasks like writing working code from scratch. This means we just need to wait for the next great model and all the problems we have now will eventually go away, right?

Well, not really. We’re not going to take a side in the ‘AGI is just around the corner’ vs ‘the potential of LLMs is dangerously overestimated’ fight – though, as software security people, we’re more sympathetic to the latter. This is because there are numerous proven weaknesses of neural networks – and thus large language models – that can cause massive security issues if we just leave everything to the models. These don’t get nearly enough attention because everyone must move fast and break things (for a summary of the potential impact of untrustworthy LLMs and how they’re quietly ignored, see Johann Rehberger’s The Normalization of Deviance in AI article). We have a more in-depth article about the underlying issues on machine learning security, but just a few quick examples of inherent risks:

• Poisoning: LLMs are pre-trained with self-supervised learning, and it was successfully demonstrated (Souly et al, 2025) that just 250 poisoned documents in the training dataset (which is assumed to be minimally curated data scraped from the public Internet) were sufficient to compromise an LLM, independent of the size of the dataset or the complexity of the model. If the models of tomorrow are compromised, this becomes a global scale supply chain attack able to cause denial of service, insert backdoors in code, spread misinformation, or even just subtly favor specific products over others…
• Evasion (jailbreaking, prompt injection): research (Nasr et al, 2025) indicates that state-of-the-art anti-injection safeguards only work against simplistic attacks and are essentially useless against a sufficiently skilled attacker (both human and adversarial ML). This is in line with the state of machine learning security when it comes to adversarial examples for over a decade: the best solutions work on the same principles as denylist-based input validation, which is – at best – a cat-and-mouse game.
• Model inversion: LLMs have been demonstrated to reveal credit card numbers, sensitive information, or source code fragments that were included in the training data (Nasr et al, 2025), which has significant implications for secrets and company IP sent to these models via prompts (such as API keys stored in a configuration file that’s sent as part of the context) that may get repurposed as training data and allow other users of the same coding tool.
• Model extraction (model stealing, distillation): By making (a huge number of) regular requests to the model and observing its responses, the attacker can construct a clone of the model that performs similarly to the original (Carlini et al, 2024). The reality of this is demonstrated by the conflicts and ‘arms race’ surrounding the distillation of frontier models by third parties.

But let’s focus on the risks of vibe coding in particular. Even before the emergence of agentic coding, vibe coding existed in concept: prompt the LLM either via chatbot or autocomplete and run the generated code without checking. As demonstrated back in 2023 (Greshake et al, 2023), it was possible to trick LLMs to autocomplete code with, say, os.system(“rm -rf /”) and if a human didn’t review the code, that was all it took.

The early vibe coding era was full of stories about vibe coded software failing catastrophically: either being trivial to hack (the infamous story of Enrichlead or the Tea app), or agents deciding to delete production databases and other critical data (SaaStr with Replit, AWS Kiro, Antigravity, etc.). This happened outside software development too, such as a musician blindly running a ChatGPT-generated script that permanently deleted a year’s worth of work from their Mac.

These issues with vibe coding are still with us in mid-2026 and better tools or models haven’t made them go away. For instance, Moltbook (the vibe-coded social media site for agents) shipped with a publicly accessible database along with other security issues,  rsync’s maintainer switched to vibe coding and broke a lot of functionality, developers carelessly using Claude Code and OpenClaw were found to be responsible for a huge portion of GitHub secrets leaks, and the list goes on. In the end what really matters is not how much the model or the harness knows, but rather how much the (human) developer knows.

Humans are still (and always) more important than tools

AI coding tools will always make mistakes – incorrectly using them can generate even more mistakes. You can’t fix AI-induced problems reliably by throwing more AI at them – practically that means always keeping a human eye on code, even if the code was partially or fully AI-generated. Even if a mythical new model turns out to identify vulnerabilities better than an expert penetration tester, it won’t magically write secure and bug-free code.

Cognitive debt by itself can directly lead to security issues. As an illustration: in early 2025, Cloudflare’s workers-oauth-provider was written with Claude Code, but it was not vibe coded. As per the readme: “[…] To emphasize, this is not “vibe coded”. Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs. I was trying to validate my skepticism. I ended up proving myself wrong.” Unfortunately, the library shipped with a vulnerability: it did not validate the redirect URI, which in some situations could allow an attacker to redirect the user to their own site and steal their credentials (see the advisory). This was a glaring problem since the first and most important best practice in the OAuth Best Practices RFC is to validate the redirect URI! The response of the developer was apologetic: “Readers who are familiar with OAuth may recognize that failing to check redirect URIs against the allowed list is a well-known, basic mistake, covered extensively in the RFC and elsewhere. The author of this library would like everyone to know that he was, in fact, well-aware of this requirement, thought about it a lot while designing the library, and then, somehow, forgot to actually make sure the check was in the code. That is, it’s not that he didn’t know what he was doing, it’s that he knew what he was doing but flubbed it.” There was a second, less dramatic vulnerability (see advisory) that allowed PKCE downgrade attacks due to a lack of validation logic. Both of these issues highlight some more subtle challenges of vibe coding: the code always ‘looks right’ (so it’s easy to skim it and go “LGTM!”), which can bias the human reviewers positively. It is also much easier to spot an obvious bug than a missing security control, especially if the developer isn’t familiar with the code. And this was close to a best-case scenario with a developer who was skeptical, acutely aware of security requirements and relevant RFCs, and reviewing the code every step of the way. Consider the same situation with the entire program being vibe coded – the developer would’ve just accepted everything. In that scenario, even if they decided to review the vibe coded program later, the developer wouldn’t have had the necessary knowledge about OAuth to check for these (and many more) issues!

As Adi Shamir (the S in RSA) famously said, “there are no secure systems, only degrees of insecurity”. When it comes to the security of software, it’s all about risk management. Vibe coding a proof-of-concept demonstration is one thing, but production code is (should be) different, especially if writing code in a security-critical domain like finance, healthcare, automotive, or industrial control systems. One of the recurring themes in the failures of vibe coding is security – specifically, vibe coded programs missing critical security controls. A developer can only assess these issues if they have the appropriate software security skills themselves (which, incidentally, also means they can write better and more specific prompts for security functionality, ultimately resulting in more secure code).

AI-generated code is only going to get more and more important. Arvind Narayanan said in his ICML 2026 keynote that an expected trajectory is in-house software development of highly customized and personalized (bespoke) software rather than off-the-shelf software from third-party vendors. As we’ve seen, large language models are still no substitute for human expertise in software development and software engineering expertise – including software security – is more important than ever.

No matter what the hype says, skilled developers are always going to be needed and are harder to replace than someone with little engineering experience who just prompts LLMs all day.

Check out our Machine learning security and Responsible AI courses, in which we cover all of the above aspects of AI-assisted software development and the necessary best practices to make sure you can use AI as the force multiplier it is supposed to be instead of a technical (and cognitive) debt generator!