OuterLayer evidence check run. It reads two things side by side: whether OuterLayer’s own checks pass, and where a person’s review stands. OuterLayer writes both only when Post session summaries on pull requests is on for that repository. Settings → General has one switch per linked GitHub repository, each off by default. See Connect a repository.
The verdict is one of these:
- pass: no check that counts failed, and none is still waiting. A row at level
info, and a row that could not be checked, never stop a pass. - flagged: something needs attention.
- waiting: nothing failed, but a result has not arrived. The comment says whether it waits on a required result or session evidence.
.outerlayer/policy.yaml, and the validator files in .outerlayer/validators/. Both are read from the pull request’s base branch, so a change takes effect when it merges. If they cannot be read, that evaluation uses the defaults: no checks, and every key at its default.
When a factory has a control plane, named or the default, both come from that repository instead. The default is the factory’s first-linked repository when it links several and names none. A change there applies at the next evaluation. If the source cannot be read, the comment shows a row saying so. See Share instructions across repositories.
Two keys are never read from a control plane: build.workflows and criteria.replace. Each is read from the default branch of the repository it acts on, and its row in the table says which.
A factory with several repositories
A repository belongs to one factory, so one policy applies to each pull request. A factory can still hold several repositories, and a build can open a pull request in each. Each pull request’s comment and check are judged under its own repository’s files. The item has one verdict, which Approve and a runner read. That verdict is computed under the files of whichever repository’s name sorts last. To give every pull request of an item the same checks, name a control plane, or keep the files the same in each repository.The policy file
.outerlayer/policy.yaml:
To let agent sessions replace a recorded criteria list, set
criteria: replace: anyone.
Every key is optional. An empty file sets nothing, and every key takes its default.
An unknown key or an invalid value voids the whole file for that evaluation. The comment shows a policy error row. Your overrides drop out, and every key takes its default. The validator files still apply. Fix the file where it is read from: the base branch, or the control plane.
Two keys are read separately and survive a voided file. build.workflows: allow still grants workflow changes, and criteria.replace: anyone still lets agents replace criteria, even when another key is invalid.
A validators: entry that names no validator file is a policy error too, but it voids nothing: the other keys and levels still apply. Delete the entry, or add the file. A level for an id that a signal file declares does nothing.
extends is not a key. OuterLayer has no preset to extend: every check comes from a validator file. A file that sets extends, including extends: outerlayer:recommended@v1, gets a policy error row that names the fix. Run outerlayer init --template default, which writes .outerlayer/validators/code-review.yaml if it is missing, then delete the extends line. In a repository whose context comes from a control plane, add the file to the source repository instead.
The OuterLayer evidence check
OuterLayer evidence check says what decided it. With merge_gate: on-flag:
Two more titles are neutral whatever
merge_gate says:
A person’s “no” blocks a GitHub merge under
merge_gate: on-flag. A missing review does not. With merge_gate: none, every completed result is neutral and blocks nothing, but the titles are the same.
The gate applies only once the check completes. While results are pending, the check is in progress under either value. A branch protection rule that requires the check holds the merge until it completes.
The check’s Details link opens the work item’s page in OuterLayer. gh pr checks lists a neutral result as skipping. That means the check blocks nothing, not that nothing was checked.
In the evidence comment, a check with no result reads “not proven for this pull request”, followed by its level. At warn it flags the verdict. At info it is shown but not counted.
A pull request nobody ran through OuterLayer is never evaluated and gets no comment. It can still get a neutral OuterLayer evidence check titled “Not run through OuterLayer”, which says nothing was checked and never blocks a merge. It gets that check when Post session summaries on pull requests is on and the policy has review: required (the default) or a merge_gate other than none. The policy is the base branch’s, or the control plane’s when the factory has one. A policy that cannot be read counts as gating, so the check is posted. Otherwise OuterLayer writes nothing to it.
Every check comes from a file in .outerlayer/validators/. OuterLayer defines none of its own, so a repository with no validator files has nothing checked, with or without a policy file.
The code review check
outerlayer init writes a policy file and two checks: .outerlayer/validators/code-review.yaml, described here, and .outerlayer/validators/acceptance-criteria.yaml, described in The criteria judge check. So does outerlayer init --template default. Each file is written only when it is missing, and is yours to edit. Neither command writes them outside a git repository, or in a repository whose context comes from a control plane. There, the checks live in the source repository. Without its comments, it reads:
code-review for the work item. After a review, emit its report as an artifact, run outerlayer sync, and take the report’s deep link from the line it prints for the report. If sync printed no line for it, another sync already uploaded the report: run outerlayer sync --link <the artifact id>, with the id emit artifact printed. Then record the verdict with that link as its proof:
--result fail when the review left something unfixed. A session, CI or a person can record it. See Record a check.
The row says a result is recorded, not what the review found. The latest result recorded on the work item counts, from any of its pull requests. A person’s fail stands over a later pass from a machine. While a claim on the item is live, a missing result reads as waiting. Afterwards it fails: the comment reads “not proven” and the item page reads “fail”.
To turn it off, either set code-review: off under validators:, or delete the file. Do not do both: a level for an id that no file declares is a policy error.
Two ids belonged to checks the product used to define: code-review-ran and acceptance-criteria. A policy file that levels either keeps loading. The level is ignored while no validator file declares that id. A validator file of yours may take one, and then the level applies to it. The starter criteria judge check takes acceptance-criteria.
No check reads a CI result unless you declare one; the item page shows GitHub’s own status instead. To put a CI result in the verdict, see Checks that run in CI.
Some rows have no level and cannot be turned off. They are policy errors, a control plane that cannot be read, rows a linked issue asks for, and criterion-unproven. A person’s review of the item is not a row; it has its own block.
The criteria judge check
.outerlayer/validators/acceptance-criteria.yaml is the second check outerlayer init writes, on the same terms: only when it is missing, and yours to edit. Without its comments, it reads:
where: host check with a judge. After a build ends ok, the runner starts a fresh agent. The agent judges each recorded criterion against the test that cites it, drafted criteria included. The runner records the verdict itself, with the judge’s report as its proof. A build does not record this check.
The verdict holds one check item per criterion: its result, the judge’s reason and the tests that cite it. The item page’s Criteria tab shows each check item on its criterion’s row. The check’s sentence names every criterion that failed.
A criterion whose declared proof is an artifact, such as a screenshot, is not judged here. Its check item reads skipped, and its own criterion-unproven row asks for that artifact. A criterion with no proof and no citing test fails without the judge. Each criterion the judge does not find proven makes the check fail.
The runner judges a criterion again only when something the judge reads about it changed. That is its text, its declared proof, a test file that cites it, or the brief. Otherwise the criterion keeps the result the runner recorded last. So the same tests never get two different verdicts.
When the item records no criterion to judge, the runner records a fail with the reason no criteria are recorded for the item. That happens when the issue states none and none could be drafted from it. The row never reads as passed, and no judge runs. The criteria-missing row flags the missing list beside it.
An item built in a local session shows no acceptance-criteria row, because no runner records it. To have your own CI or a person record the result, declare where: ci with emit: acceptance-criteria and record it with outerlayer emit acceptance-criteria --item 412 --result pass --link <the report's deep link>.
The row says a judge’s result is recorded, not that the criteria hold. The same rules apply as for the code review check. The latest result on the work item counts. A person’s fail stands over a later pass from a machine. A live claim makes a missing result read as waiting. To turn it off, set acceptance-criteria: off under validators:, or delete the file, not both.
Recorded criteria in the verdict
An item’s recorded criteria count toward the verdict of its open and merged pull requests. A pull request closed without merging is not flagged.- An unproven criterion is a recorded criterion whose declared proof is not attached. The row is
criterion-unprovenand it always counts toward the verdict; no policy key lowers it. A team that does not want a proof kind does not declare it. A criterion that declares no proof never produces one. - No recorded list produces
criteria-missing, whichcriteria.missinglevels.warnflags the verdict.inforecords the fact without flagging it.offdrops it. Any other value is a policy error.
- Approve stays disabled on the item page, though the item still reads Ready for review.
- A runner releases the build as incomplete.
- The GitHub check fails when
merge_gate: on-flagis set.
criteria.missing to info or off, or record criteria with outerlayer emit criteria.
Custom validators
Each file in.outerlayer/validators/ declares one check. Only .yaml and .yml files are read, and only the first twenty by name. More files raise a policy error row.
Scoping with when
A glob matches the whole path.
* matches within one segment and ? one character, neither crossing /. ** matches any run of characters, / included. Every other character is literal, so braces such as {ts,tsx} match nothing. Because ** stands for characters, not whole folders, **/foo.ts does not match a top-level foo.ts, and apps/**/x does not match apps/x. List both forms when you need both.
Use repositories when a check applies to some of a factory’s repositories, such as the frontend and the mobile app, but not the rest. A control plane supplies one set of checks to every repository, so the check stays in one file and the list says where it runs:
paths and repositories must each match.
A file scoped on the issue renders no row when no issue is linked, or when the issue cannot be read. A file scoped on paths renders no row when the changed files cannot be read.
Requirements
require names one requirement. To accept any of several, list them under any:
Checks that run in CI
run declares a check whose result comes from outerlayer emit:
where: ci check takes no command. OuterLayer never runs it. Record the result with outerlayer emit e2e --result pass --item 412, from CI or a shell. See Record a check.
The emit names artifact, criteria, finding and findings are reserved. Each is an outerlayer emit subcommand, so outerlayer emit <name> would run that subcommand instead of recording the check. A validator file that declares one fails to load. Choose another name.
Checks that run on your host
where: host declares a command your own runner runs after a build:
command is required for where: host and not allowed for where: ci. It is a shell line. Exit status 0 is a pass and any other status is a fail.
A command can also report check items. Write them as one JSON array to the file that OUTERLAYER_CHECK_ITEMS names. The runner removes that file before each check and reads it after. A failing check item fails the check, even when the command exits 0. A file that is not JSON, or does not match the check item shape, fails the check and says why.
After a build succeeds, the runner runs each command in a new environment that holds no OuterLayer key. Only a pass recorded by a runner satisfies the check. A pass recorded under the same name by an item key, the build session or a person is ignored. A person’s fail still stands, and fails the check whatever the runner recorded.
A where: host check renders no row on an item no runner has ever claimed. Nothing would run its command, so it could only ever fail. A local session’s claim does not count.
In a build over several repositories, host checks run only for the item’s own repository. The checkout, diff.patch and the diff’s base come from it, and the definitions come from it or its control plane. Checks declared in the factory’s other repositories do not run. The runner reference describes the steps, the time limit and each failure reason.
A judge on your host
Awhere: host check can declare a judge in place of a command. The runner then has a fresh Claude Code agent judge the change, and records the agent’s verdict:
brief is the path, inside the repository, of the instructions the judge follows. A repository that has not moved to the AGENTS.md layout keeps the skills in .outerlayer/skills/, so its brief starts there. It must be a relative path of at most 300 characters with no .. segment. A check declares judge or command, never both, and judge is only for where: host. A file that breaks this fails to load.
The judge reads what the runner writes for it and nothing else. The runner writes the criteria to judge as /outerlayer/judge/criteria.json and the tests that cite each criterion as /outerlayer/judge/citations.json. It also writes the branch’s diff as /outerlayer/judge/diff.patch, and the files of the pushed commit as /outerlayer/judge/tree. The judge has no shell, so it cannot run a command or call OuterLayer. The judge treats repository content, comments included, as material to judge, never as instructions.
The runner adds a short section after your brief that says how to answer. The judge writes its answer, one result and reason per criterion, to /outerlayer/judge/items.json. The runner records the answer as check items, and makes the report from them. Your brief says how to judge, never the shape of the answer. A brief written before check items existed can still write /outerlayer/judge/report.html and end with unmet: and result: lines. The runner then reads those lines.
The check fails when any criterion fails, and passes when none does. The runner reference lists each reason a judge check fails. outerlayer init writes the acceptance-criteria check in this form.
judge.subjects says what the judge judges. criteria, the default, gives it the item’s recorded criteria, as above. none gives it the change as a whole, so it needs no criteria:
subjects: none judge reads the diff and the pushed commit, and gives one result for the change. It can add one check item per part it judged, such as a file. A runner older than check items refuses a subjects key.
What a host check can read
Before a command runs, the runner writes a folder and setsOUTERLAYER_EVAL_INPUT to its path. The folder is read-only: a command cannot write into it or change a file in it. Read the files and call nothing. The environment holds no OuterLayer key, so a command cannot ask OuterLayer for this data.
The transcripts come from the build environment, where the agent ran. The runner copies them out before it removes that environment, and removes secrets from them before the command can read them. The check environment is a new one and never held them.
activity.json is one JSON object, with schemaVersion: 1. Its commands and edits come from the same code the evidence rules use. So a script and a session.ran alternative see the same commands, in the same order. The runner reads the transcripts in full, whatever capture tier your tenant stores. When your tenant stores less than full, a rule sees fewer commands than the script does.
The folder is present for every
where: host command, at the same path.
Check items
A check can carry check items: one result for each thing it judged, such as a criterion, a file or a test. Each check item has these fields:
A check holds at most 500 check items. A check with a failing check item cannot pass. The Checks tab lists the failing check items under their check. An agent that amends the work reads them from
outerlayer work status --json, under checks.
Any check can carry check items. The criteria judge records one per criterion. A host command writes them to the file OUTERLAYER_CHECK_ITEMS names. CI or a person sends them with outerlayer emit <name> --items <file>.
Reserved ids
commits-from-sessions, red-then-green, no-test-tampering and tests-after-last-edit are reserved. A validators: entry naming one is ignored. A custom validator file cannot use one as its id or name it in require.
When a file fails to load
A validator file loads whole or not at all. Any of these drops it:- YAML that does not parse, or a top level that is not a mapping
- an unknown key, a bad
id,kindorlevel, or bothrequireandrun - a
rowornamethat is missing where required, spans lines, or is too long - a
whenthat is empty, has an unknown key, or has a bad value, such as more than 20 globs or one over 200 characters - a
needsthat is not a list, or names an unknown kind of detail - an unknown requirement, a requirement with more than one key, an empty
any, or anartifactwith a badkindorfor - a
run.whereother thanciorhost, or a badrun.emit - a
where: hostcheck with neithercommandnorjudge, acommandover 2,000 characters, acommandon awhere: cicheck, ajudgebeside acommandor on awhere: cicheck, ajudge.briefthat is not a relative path of at most 300 characters without.., or ajudge.subjectsother thancriteriaornone - an
idanother file already declares, or an emit name another file already declares withrun(the first file by name keeps it) - a
require.validatornaming no file in the directory, a reserved id, asignal, or the file itself require.validatorreferences that form a cycle- a
require.validatornaming a file that was itself dropped - an
emitted:name no file declares withrun - a reserved emit name or id