What Is a Security Ledger? Proof That Your AI did what it said it was going to do

Luke (hakluke) Stephens

Decorative abstract image of a log

AI is great at testing and auditing code… most of the time. Now that AI is helping to ship code faster, we need to also rely on AI more heavily for security reviews. This means that we also need a way to check that AI is doing those security reviews properly, because it’s very common for AI to perform checks that don’t make sense, or don’t prove anything, and then confidently assert that something is proven.

For example, let’s say your AI agent has finished testing the new invoice endpoint. It hands you a report: “Authorization checks passed. Users cannot access other users’ invoices.”

That’s great but… Which users? Which invoices? Did it actually send a request, or did it read the middleware and just decide everything looked sensible? AI models, even the best ones, hallucinate, and will confidently claim things that they didn’t properly test.

You might open the run details and discover that the AI ran every request using an administrator account. So all the agent did was successfully demonstrate that an administrator can access invoices. That’s not the test you were expecting, and it doesn’t prove anything useful. The useful question is whether an ordinary user can read somebody else’s invoice, and that still hasn’t been tested.

This is the problem a security ledger helps you spot. When an agent says it checked something, you need a record of the work that lets you decide whether its conclusion holds up. Especially if you’re going to let it work through more findings than you could reasonably investigate yourself.

What should go into a security ledger?

A security ledger is an auditable, running record of security tests and their results, linked to the issue or code change being investigated. For live application testing, that means retaining the tool calls, tool responses, targets, payloads, requests and responses behind a verdict.

Think about what makes a good bug report useful. You can follow the reproduction steps, inspect the evidence and understand why the behaviour is a vulnerability. A ledger keeps that level of detail about the investigation, including attempts that failed or found nothing.

Keep the recorded observations separate from the agent’s interpretations. The tool records an HTTP response. The agent explains what it thinks that response means. You should be able to inspect either of those independently. That separation becomes very useful when the two disagree!

The invoice test that actually answers the question

Let’s fix our imaginary invoice investigation. We’re working in an authorised test environment with two ordinary test users, Alice and Bob, and invoices created for the test. Alice should be able to read her own invoice. She should not be able to read Bob’s.

First, establish the setup. Alice can retrieve her invoice using her session. Bob can retrieve his invoice using his. Both sessions work, both objects exist, and we know who owns them.

Now request Bob’s invoice using Alice’s session.

This is a basic insecure direct object reference, or IDOR, test. OWASP’s testing guide recommends using multiple users with different owned objects so you can test the access boundary directly instead of guessing object identifiers.

Here’s a simplified record of that final request. These are fictional fields, just to make the point.

Field Recorded value
Target Staging deployment for commit abc123
Test Read another user’s invoice
Requesting identity Alice, ordinary user
Expected owner Bob, established by test fixture
Request GET /api/invoices/inv_bob_01
Authentication Alice’s test session, secret redacted
Response HTTP 200, body contains Bob’s invoice data
Evidence Stored request and response, linked to this run

The response body is a key piece of evidence. A 200 could contain an error message, a login page or an empty result. The observation supporting the finding is that Alice’s authenticated request returned data she was not allowed to read.

A reviewer can now check the claim. Was Alice really a normal user? Was the invoice private? Do the recorded request and response match the explanation? If the agent accidentally used Bob’s session, the record gives you a way to catch that too.

Now imagine the request receives a 403 and no invoice data. That supports a narrower statement: this particular cross-user read was blocked in this run. You still haven’t tested whether Alice can edit the invoice, download its PDF or access it through another endpoint.

And if Alice’s session expired before the test, the run is inconclusive. “The application rejected an expired session” tells you very little about whether its object-level permissions work.

Without all of this context, it’s easy for AI to falsely assume that authorization has been tested.

How to start

You don’t need an elaborate schema to start keeping useful evidence. You just need enough context to interpret it later. You can expand on it at any point.

For each investigation, I’d want at minimum:

  • The thing being investigated. Link the original finding or ticket, the target environment and, where available, the deployed build or commit at the time of testing.
  • The test conditions. Record the tool and version, relevant configuration, time, test identity and prerequisites. 
  • The action and observation. Keep the input or payload and what the tool actually observed.
  • The outcome and its limits. Distinguish a completed test with no finding from an error, a skipped test or an investigation that could not reach the required state.
  • The conclusion. Let the agent explain its verdict and point to the specific evidence supporting it. Keep that explanation identifiable as the agent’s contribution.

Record the unsuccessful attempts as well. If the first ten requests failed to authenticate and the eleventh succeeded, that history can explain why results have been misleading.

A little warning though: be careful with what you retain. Raw HTTP traffic can contain credentials and customer data. Prefer test accounts and fixtures, redact secrets, restrict access and set a retention policy.

Record test evidence the agent cannot rewrite

Start with the evidence your tools already produce: proxy histories, scanner output and CI artifacts. Save that output as tests run, linked to the finding and build under test. Keep the requests, responses and failed attempts, not just a “passed” status. A reviewer should be able to open a finding and follow what happened without asking the agent to reconstruct it.

Use a separate recording service to append the tool output, and give the testing agent read-only access to the stored evidence. It can reference those records in its verdict, but cannot overwrite or delete them. If a record needs correcting, add a correction and retain the original. You can then check the agent’s verdict against the recorded test results, even when its explanation is wrong.

Add security tests as features change

Linking tests to changes helps you spot gaps in coverage. Suppose a change introduces invoice exports. You can see that the ordinary read endpoint was tested, while the new export route has no recorded investigation. That gives the team a specific gap to close before relying on the result.

  • Write tests with the feature. For invoice exports, check that Alice can export her own invoice, cannot export Bob’s, and cannot export anything without signing in. Use integration tests that exercise the route and its permission checks.
  • Be specific when asking AI to write them. Give it the access rules, test users and expected outcomes. Ask it to check the returned data as well as the status code, then review the assertions yourself.
  • Run the tests and keep the results with the change. Record which build ran, what passed or failed, and whether any tests were skipped. Having a test in the repository does not tell you whether it ran.

Keep a short checklist of the checks each feature needs, then compare it with the recorded runs. The ledger shows what was tested, but it cannot tell you which tests you forgot to plan. If a planned check has no recorded result, find out whether it was skipped, failed to run or ran without saving its evidence.

When a test exposes a bug, keep it as a regression test. If Alice can export Bob’s invoice, the test should fail because it expected access to be denied. After fixing the permissions, run the same test against the new build and save both results. The developer can then compare the original failure with the passing result after the fix, and rerun the test whenever that code changes.

How Detectify approaches it

Detectify’s Agentic Tooling puts this separation into the workflow. A suspected issue becomes a ticket. Your agent investigates it against the running application using Detectify’s testing tools, accessed through MCP.

Detectify records the tool activity in the Security Ledger. The agent submits its verdict separately, linked to the entries behind it, and cannot edit the recorded evidence.

That is the part I’d pay attention to. The agent can explain why it believes it found reflected XSS, but you can inspect the recorded assessment and browser evidence rather than accepting its explanation alone. A claim without a supporting test remains unverified.

The scope is the activity recorded through the tooling. It should not be mistaken for a record of everything an agent might do elsewhere, or proof that every possible vulnerability has been checked.

You can book a demo or start a trial at Detectify to see this in action.

Start by trying this on one finding

Before scaling up your security agent, give it just one suspected issue in an authorised environment. Let it investigate, then review the evidence.

Ask the obvious questions, e.g. Can you tell what it tested? Can you identify the account and target? Did it reach the behaviour it was meant to exercise? Does the response support the verdict?

Fix those gaps while the workflow is small. Then make evidence capture part of the tool execution, so that you can do the same on any test in future.

Remember that the whole purpose of a ledger is not to tell you whether your application is secure. It lets you answer a deeper question: what exactly did we do to test whether it’s secure, and what happened?

 

Luke (hakluke) Stephens

I'm a dad, hacker and entrepreneur. I founded Haksec Group, a holding company spanning HackerContent (cybersecurity marketing agency), Ironbark Cyber (consultancy), and TRIAGERS™ (outsourced VDP triage).

Check out more content