The Illusion of Competence. AI-generated software risks.

Why AI-Generated Software is Like Hiring a Contractor Who Learned Architecture from YouTube

by Andrei Răileanu

The Friend's Dilemma

A friend called me last evening with a question that's becoming increasingly common in 2026: "Should I vouch for this software? My buddy built it with ChatGPT—it connects our sales app to the ERP. Works perfectly. Should I recommend it to the board?"

I asked him one question back: "Can your buddy explain what the code does and how it does it without asking ChatGPT?"

Silence.

Not the comfortable kind. The kind where someone realizes they've been operating on an assumption that just collapsed. "I... I don't think so," he finally said. "But does it matter? The integration works. Sales data flows into the ERP. Invoices generate automatically. The team loves it."

This is the conversation happening in boardrooms, Slack channels, and coffee meetings across every industry right now. Someone's nephew, or marketing manager, or operations analyst has built something "simple" using AI. It works. It's fast. It's free. And now someone with actual authority is being asked to vouch for it.

Here's what nobody's saying out loud: vouching for software you don't understand makes you liable for risks you can't see. And in 2026, those risks carry €35 million price tags.

The Democratization Promise (And What It Actually Delivered)

Let me be clear about something upfront: I build with AI every day. I use it to compress 12-month projects into 12 weeks. I've built businesses on AI-augmented development. This isn't a Luddite manifesto against progress. But there's a difference between using AI as a strategic partner and mistaking AI output for competence.

The promise of AI democratization was beautiful: anyone could build software. The barrier to entry would disappear. Small businesses could automate workflows without hiring expensive developers. Innovation would flourish. And in many ways, that promise was delivered. I've seen operations managers automate repetitive tasks in hours that would have taken weeks to spec and months to develop traditionally.

But here's what the democratization narrative conveniently omits: making coding accessible didn't make software engineering comprehensible.

AI removed the syntactic friction of programming—the semicolons, the bracket matching, the cryptic error messages. What it didn't remove was the conceptual complexity of building reliable, secure, maintainable systems. It's the difference between learning to use a saw and learning to build a house. YouTube can teach you the former in twenty minutes. The latter requires understanding load-bearing walls, electrical codes, plumbing regulations, and a thousand decisions that only reveal themselves when things go wrong.

AI-assisted development is the same. ChatGPT will write you a function that connects your sales app to your ERP. It will look professional. It will probably work on the first try. What it won't tell you is that it hardcoded your database credentials into a configuration file that's now committed to your Git repository. Or that it's using a deprecated API endpoint that will break when the vendor updates their system next quarter. Or that it's storing financial transaction data in plaintext logs because the AI optimized for making it work, not making it secure.

The AI democratized the ability to generate code. It didn't democratize the judgment required to evaluate whether that code should exist in production.

The Hidden Costs of AI-generated software risks. (Or: What Your AI Didn't Tell You)

When non-specialists use AI to generate software, they're operating in a dangerous blind spot. The code works—that's the problem. It works just enough to pass superficial testing, but fails catastrophically under conditions the creator didn't know to test for.

Let me walk you through what's hiding in that "simple middleware" integration my friend was being asked to vouch for. Here are some of the more obvious and easy-to-understand examples of AI-generated software risks.

Secret Sprawl: The 3.2% Problem

Here's a statistic that should terrify anyone deploying AI-generated code without review: commits assisted by AI expose credentials at a rate of 3.2%, compared to 1.5% in manually written code. That's not a rounding error. That's a doubling of the risk that your API keys, database passwords, OAuth tokens, or AWS credentials end up hardcoded in your source code.

Why does this happen? Because AI models are trained on public repositories where—let's be honest—developers frequently commit secrets they shouldn't. The AI learns this pattern and replicates it. When a non-specialist prompts the AI to "connect to the database," the AI helpfully generates code with a placeholder connection string... that looks suspiciously like a real connection string.

The non-specialist, not knowing better, replaces the placeholder with their actual credentials and commits it. Congratulations, your production database password is now in your Git history forever. Even if you change it tomorrow, the old one is still there, waiting for someone to find it.

The research from GitGuardian and Snyk (March 2026) shows this isn't theoretical. It's endemic. And it's getting worse as AI-assisted development scales.

Technical Debt 2.0: Architecture Chaos

Traditional technical debt is a conscious trade-off. An experienced engineer says, "I'm going to take a shortcut here because we need to ship, but I'm documenting this as something we'll refactor later."

AI-generated technical debt is unconscious. The person generating the code doesn't know they're creating it. The AI optimizes for solving the immediate problem in the prompt, not for long-term maintainability.

What does this look like in practice?

Your sales-to-ERP integration has worked perfectly for three months. Then your sales team grows. Suddenly, you're processing 10x the transaction volume. The integration starts timing out. Invoices take hours to generate instead of seconds. The team is furious.

You bring in an actual engineer to fix it. They look at the code and discover:

• The AI generated a synchronous process that blocks on every API call

• There's no error handling—if one invoice fails, the entire batch crashes

• It's re-fetching the same customer data from the sales app on every transaction instead of caching it

• The database queries aren't indexed, so every lookup does a full table scan

  
  

An experienced developer would have architected this differently from day one. The AI had no concept of scale, edge cases, or failure modes. It wrote code that worked for the prompt it received, not for the business problem it was solving.

Now you're paying an engineer $150/hour to rewrite the entire thing. That "free" integration just cost you $15,000 in remediation, plus the opportunity cost of three months of degraded performance.

81% of executives report that AI technical debt is blocking the success of their AI initiatives. This is why.

Model Poisoning: The Adversarial Injection You Didn't See Coming

Here's where it gets genuinely scary.

Sophisticated attackers have figured out how to poison AI training data or manipulate prompts to inject malicious code into AI-generated outputs. This isn't science fiction—it's documented in the CVE database.

In January 2026, there were 6 reported vulnerabilities directly attributable to AI-generated code. By March, that number was 35. Researchers estimate the real number is 5-10x higher because most organizations don't track whether their code came from AI or human developers.

One particularly nasty attack vector: adversarial injection in pull requests. An attacker submits a seemingly benign code suggestion to an open-source project. The AI-assisted developer doesn't understand the code well enough to spot the backdoor. It gets merged. Now thousands of projects downstream are compromised.

When your non-technical buddy used ChatGPT to build that integration, did they audit the dependencies? Did they verify the packages weren't typosquatting legitimate libraries? Did they check for obfuscated code paths that only activate under specific conditions?

Of course not. They don't know what any of those words mean.

EU AI Act: The €35 Million Timer Started in August 2026

Let's talk about the regulatory hammer that's about to fall.

The EU AI Act came into force in August 2024, but the compliance clock that really matters will start ticking in August 2026—just three months from now. That's when the general application provisions go live.

Right now, in May 2026, most companies are scrambling. Any AI system handling customer data or making automated decisions needs to be compliant. And "simple middleware" doesn't get a pass.

What does compliance mean?

• Transparency requirements (you must be able to explain how the system makes decisions)

• Risk assessment documentation

• Quality management systems

• Post-market monitoring

  
  

Your friend's sales-to-ERP integration? It's processing financial transaction data. It's making automated decisions about invoice generation. Under the EU AI Act, that's a regulated system.

If it's deployed without proper documentation, without auditability, without compliance verification? The fine is up to €35 million or 7% of global annual revenue, whichever is higher.

"But we're a small company!" Doesn't matter. The regulations don't have a revenue threshold for high-risk systems. If you're processing EU citizen data through an AI-assisted system without compliance documentation, you're liable. And when the regulator asks, "Who approved this deployment?" and someone points to you—the person who vouched for it—guess whose name is on the enforcement action?

ISO 5055: The Structural Quality Standard Nobody Knows About

In 2026, ISO/IEC 5055 became the de facto standard for structural software quality in enterprise procurement. It measures critical weaknesses across four dimensions: security, reliability, performance efficiency, and maintainability.

AI-generated code consistently fails ISO 5055 audits. Why? Because the AI optimizes for functional correctness (does it do what the prompt asked?) not structural quality (is it built in a way that won't collapse under pressure?).

The specific failures are predictable:

• Lack of error handling (reliability failure)

• Hardcoded configurations (security failure)

• Unoptimized algorithms (performance failure)

• Zero documentation (maintainability failure)

  
  

When your friend's integration goes into production and a client runs an ISO 5055 audit (which is increasingly standard in B2B contracts), it will fail. That failure becomes a breach of contract. The client can sue for damages. And you vouched for it.

The Reputational Damage Scenario (Or: What Happens When Your Name Is On It)

Let me paint you a picture of how this actually plays out.

Month 1: Your friend deploys the integration. It works beautifully. The sales team loves it. Your friend gets praised in the all-hands meeting. You get a thank-you email for the recommendation.

Month 2: Everything still works. You almost forget you were involved.

Month 3: A security researcher doing routine scanning of your company's GitHub repositories finds exposed AWS credentials in a commit from the integration. They responsibly disclose it to your security team.

Your security team investigates. They discover:

• The credentials have been public for 90 days

• They provide full access to your production database

• The access logs show unauthorized queries from IP addresses in Eastern Europe

• Customer financial data has been exfiltrated

  
  

Month 4: You're sitting in a conference room with the CEO, the General Counsel, your CISO, and a very expensive incident response firm. They're walking through the timeline. "Who approved this integration going into production?" the lawyer asks. Your friend points to the email thread. Your name is on it. "They vouched for it." The lawyer makes a note.

Month 5: The GDPR supervisory authority issues a preliminary finding. €20 million fine for inadequate security measures leading to a data breach.

Month 6: Your largest client, whose data was in the breach, terminates their contract. That's 30% of your annual revenue, gone.

Month 7: You're looking for a new job. But when potential employers Google your name, the first result is a trade publication article about the breach, with your quote from the email thread where you vouched for the integration.

This isn't a hypothetical scare story. This is a composite of real incidents I've consulted on in the past 18 months. The details change. The pattern doesn't.

Here are 3 public (in)famous examples. History is littered with 'it worked perfectly until it didn't' moments:

• Uber lost 57 million records because of hardcoded credentials in GitHub.

• Samsung accidentally leaked proprietary source code directly into ChatGPT’s training set.

• Knight Capital went bankrupt in 45 minutes because of unmanaged legacy code—the ultimate technical debt.

When you vouch for software, you're not just saying "this works." You're saying, "I've verified this won't cause catastrophic harm." And if you haven't actually verified that—if you're taking someone's word that the AI did a good job—you're signing a blank check on your reputation.

The Uncomfortable Truth (Or: Why the YouTube Surgeon Analogy Isn't Hyperbole)

You wouldn't recommend a surgeon who learned their craft from YouTube tutorials. Not because YouTube tutorials are bad—many are excellent educational resources. But because surgery requires judgment that can't be compressed into a tutorial.

It requires knowing when the textbook approach doesn't apply. When the anatomy is variant. When the patient's comorbidities change the risk calculus. When to call for help. A YouTube-trained surgeon might successfully perform a routine appendectomy on a healthy 25-year-old. They watched the video. They followed the steps. It worked.

But what happens when the appendix is retrocecal? When the patient has a bleeding disorder? When there's unexpected adhesions from prior surgery?

The YouTube surgeon doesn't know what they don't know. They can't make the judgment calls because they lack the foundational understanding of anatomy, physiology, and surgical principles that would tell them they're in over their head.

This is exactly what's happening with AI-generated software.

The AI can write code that works for the prompt it received. What it can't do is anticipate the edge cases, the security implications, the scalability constraints, or the regulatory requirements that weren't in the prompt.

And the person wielding the AI—if they're not a software engineer—can't evaluate whether the output is safe to deploy. They can verify that it works. They can't verify that it won't fail catastrophically under conditions they haven't thought to test for.

You wouldn't vouch for the YouTube surgeon operating on your mother. Why would you vouch for YouTube-grade software handling your company's financial data?

What This Actually Means (And What You Should Do About It)

I'm not telling you to stop using AI for software development. I use it extensively. It's transformative.

I'm telling you that AI-assisted development without AI-informed auditing is reckless. If someone asks you to vouch for software built by a non-specialist using AI, here's what you need to verify before your name goes anywhere near it:

Can the creator explain, in their own words, how the system works? Not "it connects the sales app to the ERP." I mean: How does it authenticate? How does it handle failures? What happens if the API is down? Where is the data stored? How is it secured?

If they can't answer these questions without consulting the AI, that's your red flag.

Has it been security audited? At a minimum, you need:

• SAST (Static Application Security Testing) scan results

• Confirmation that no credentials are hardcoded

• Verification of dependency versions (no known CVEs)

If the answer is "we didn't do that," don't vouch.

Has a senior engineer reviewed it? Not the person who generated it. Someone with the judgment to evaluate whether it should exist in production. If the answer is "we didn't think we needed that," absolutely don't vouch.

Is there compliance documentation? If it processes EU citizen data, handles financial transactions, or makes automated decisions, you need EU AI Act compliance verification.

If the answer is "what's the EU AI Act?", run.

The Real Cost of the Illusion

The democratization of programming is a net positive for the world. I genuinely believe that. It's enabling innovation at a pace we've never seen before. But democratization without accountability is just chaos with better UX. The illusion of competence—the belief that being able to generate code means understanding software engineering—is going to cost companies billions in technical debt, security breaches, regulatory fines, and reputational damage over the next few years.

The people who will pay the highest price aren't the AI companies or the non-specialists building things with AI. It's the people who vouched for the output without verifying it was safe.

Don't be that person.

Democratization is wonderful. Accountability is mandatory.

And if you can't tell the difference between code that works and code that's safe, don't vouch for it.

❓ Frequently Asked Questions (FAQ)

1. Is AI-generated code safe for production environments?

While AI can generate functional code quickly, it often lacks structural integrity and security awareness. Without expert human review, AI code carries a higher risk of credential leaks (3.2% vs 1.5% for manual code) and technical debt that can stall long-term scalability.

2. What are the legal risks of using AI in software development under the EU AI Act?

Starting August 2026, the EU AI Act imposes strict transparency and risk-management obligations. High-risk systems—such as those handling financial data or automated decisions—must have clear audit trails. Failure to comply can result in fines up to €35 million or 7% of global turnover.

3. How can companies mitigate "Technical Debt 2.0" from AI tools?

Mitigation requires moving from a "copy-paste" culture to a "review-first" framework. Implementing automated quality measures like ISO/IEC 5055 and requiring senior engineer sign-offs on all AI-assisted commits are essential steps to maintain system stability.

4. Does AI-generated code infringe on software quality standards like ISO/IEC 25010?

AI-generated code often fails ISO quality audits because it optimizes for the immediate "prompt" rather than long-term maintainability, reliability, and performance. Using AI without a governance layer typically results in architectural chaos that breaches enterprise-grade quality models.

📚 References & Technical Foundation

To ensure the highest standards of accuracy in the age of AI, this article synthesizes data from the following academic research and regulatory frameworks:

1. Key Industry Data & Metrics *GitGuardian & Snyk (2026): State of Secrets Sprawl in AI-Assisted Workflows. Analysis of 10M+ commits.

2. Wharton School (2025): The Productivity-Quality Paradox: A Longitudinal Study on AI-Assisted Development.

3. IBM Research (2026): Q1 Enterprise AI Survey: Technical Debt as the Primary Blocker.

   Academic Bibliography & Regulatory Standards

4. EU AI Act: Regulation (EU) 2024/1689 of the European Parliament and Council. [Official Source](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689)

5. Momcilovic et al. (2024): Developing Assurance Cases for Adversarial Robustness and Regulatory Compliance in LLMs. [arXiv:2410.05304v1](https://arxiv.org/pdf/2410.05304v1)

6. Uddin & Aziz (2026): Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection. [arXiv:2604.14231v1](https://arxiv.org/pdf/2604.14231v1)

7. ISO/IEC 5055: Automated Source Code Quality Measures. [ISO Standard](https://www.iso.org/standard/80623.html)

8. Governatori (2014) & Kosenkov et al. (2024): Frameworks for ICT Regulatory Compliance and Requirements Engineering. ---

Andrei Raileanu (founder of humanailabs.org) — The AI Whisperer Helping entrepreneurs build AI-powered businesses 10x faster without losing stability. When not compressing year-long projects into 12-week sprints, writing about the hard truths of AI democratization that most people prefer to ignore.

Connect: https://linkedin.com/in/andreiraileanu | andreiraileanu.eu Next: Part 2 — "The Governance Gap: How to Audit AI-Generated Software Without Killing Innovation" ---

Published on andreiraileanu.eu | May 2026