Can you trust AI-generated code?

Two peer-reviewed studies give a precise — and uncomfortable — answer.

1) Models write insecure code even when they can recognise it. CoV-Eval (Mou et al., 2025) benchmarked 20 proprietary and open-source LLMs across secure generation, vulnerability detection and repair. The pattern: most models detect vulnerabilities reasonably well, yet still generate insecure code by default — and their repair ability is weak, often failing even when you tell them the exact vulnerability type.

2) Their confidence is not a safety signal. A 2026 study on 'security calibration' (Siddiq et al.) found confidence is systematically inflated: for GPT-4o-mini, code the model rated high-confidence (p ≥ 0.8) was still insecure about 33% of the time. A review gate that trusts the model's own confidence would wave roughly one in three vulnerable snippets straight through.

3) Automated repair doesn't rescue it. The same study tested calibration-guided auto-repair and found only limited security gains while frequently introducing functional regressions — for some vulnerability classes, repair success was effectively 0% because models won't replace the insecure API they reached for.

None of this is 'stop using AI' — it's 'gate it like any other code.' Concretely:

▪️ Treat generated code as a fast first draft that must pass the same checks as human code: review, tests, and SAST + dependency scanning in CI.

▪️ Never auto-merge on model confidence; at most, use it to prioritise what a human looks at first.

▪️ Put a security reviewer on anything touching auth, input handling, crypto, deserialization or data access — the areas models most often get wrong.

How does your team gate AI-generated code today — and has it caught anything?