What the AI Audit is for

Most teams who ship an LLM feature never get asked a security question about it. Not because they are careless — because there is no obvious moment to ask. The feature goes out with the sprint, it works, and nobody owns the question of what it could be made to do.

We built the AI Audit to create that moment. This is what I would tell you if you asked whether it is worth your afternoon — including the parts where it falls short.

Whether this is for you

Skip the regulatory language for a second. There is one question that decides it:

Does your application call a language model, and can anything that model produces reach a person, a tool, a shell, a browser, or a database?

If yes, the audit applies to you. That is the whole qualifier. It does not matter whether you trained anything, whether you call it "AI", or whether you are a two-person team or a bank. A wrapper around someone else's API with a single tool attached can hold the full lethal trifecta, and most of the gaps the audit finds live in application code, not in the model.

It is not for you if nothing you ship touches a model, or if what you need is a certificate you can show a customer. More on that below.

What it actually does

Two assessments that happen to live in one page.

The security half scores you against the OWASP LLM Top 10 — 49 statements across the ten controls, from prompt injection and output handling through to excessive agency and unbounded consumption. These are the things an attacker works from.

The compliance half asks which EU AI Act obligations actually reach you, then scores you against only those. Answer that you are a deployer of a limited-risk system and you get the transparency and literacy duties. Answer that you are a high-risk provider and the set expands to the full Article 9–15 stack. Most people get far fewer questions than they fear.

Three details that matter more than the headline number:

• Nothing you type leaves your browser. There is no account, no upload, no server-side scoring. We did not want to tell you to be careful with your data and then collect it.
• It is a maturity scale, not a checklist. Every statement is scored 0 to 3 — not in place, ad hoc, documented, robust. Yes/no checklists let you claim a control exists because somebody once wrote a regex. The gap between "ad hoc" and "documented" is where most real risk lives, and a binary question cannot see it.
• The two scores are never averaged. Security and compliance have different owners, different deadlines, and different failure modes. One blended number would let a good security score paper over a missing Article 50 disclosure. That is a trade you should make on purpose, if you make it at all.

How to get a real answer out of it

The audit is only as honest as the person answering it, so most of the practical advice is about that.

Run it against one system, not your company. "Our AI" is not a scope. Pick the specific application, with its specific tools and its specific data sources. If you have three, run it three times — the scores will differ more than you expect.

Do it with whoever actually holds the credentials. The person who knows whether model output is escaped before rendering is usually not the person who knows whether there is a post-market monitoring plan. An hour with both in the room produces a result worth keeping. One person guessing produces a number.

Score what is true today, not what is designed. The most common way to get a useless report is to answer with the architecture diagram instead of the running system. "We intend to rate-limit" is a 0.

Reserve 3 for things you could evidence. If someone asked you to prove that control tomorrow — a test, a log, a dashboard — could you? If not, it is a 2.

Budget about half an hour if you know the system well, and longer if you have to go and ask. One warning, because I would rather say it here than have you discover it: there is no save yet. Answers live in the page, so a refresh loses them. Do it in one sitting, and print the report before you close the tab. We are fixing this.

What it is not

I would rather be clear about this than have you find out at a bad moment.

It is not a penetration test. It asks whether you test for indirect injection. It does not test for indirect injection. A self-assessment tells you where to look; it cannot tell you what is actually there. If the audit says your injection testing is at 0, the next step is to go and try the attack against something you own.

It is not legal advice, a certification, or a conformity assessment. It gives you an internal read on where you stand. For a high-risk system, and before any public compliance claim, that needs qualified counsel and, where required, a notified body. Nothing you print from our site is a substitute for either.

It is not a one-time exercise. Both halves move — your system picks up a new tool, the rules pick up a new deadline. A score from six months ago describes a system you no longer run.

What to do with the report

The report is deliberately boring: your scores, a per-control scorecard, a gap register, and a remediation roadmap. The gap register is the part that earns its keep. Anything you scored 0 is flagged Critical and anything at 1 is High, each one mapped to its OWASP control and, where relevant, the EU article it sits under.

Three things to do with it:

1. Turn the Critical rows into tickets, with names on them. A gap register that lives in a PDF is a document. One that lives in your tracker with an owner and a date is a plan.
2. Start with blast radius, not score. A 0 on output handling or agency usually matters more than a 0 somewhere the worst case is an ugly answer. The roadmap orders by urgency; your own system should break the tie.
3. Keep the printout and re-run it next quarter. The second report is more useful than the first, because the delta tells you whether anything you decided actually happened.

Where to go next

Run it against one system you own. If you only have twenty minutes, do the security half and stop — it stands on its own, and it is the half that describes what can be done to you today.

If the report turns up gaps you want to learn to fix and then verify by actually landing the attack, that is what Breaking AI and the range are for.

And if a question in the audit is ambiguous, or the report says something you cannot act on, tell me — [email protected]. A self-assessment that asks unclear questions produces confident, wrong answers, and I would rather hear about that from you than not hear about it at all.

— Komal