Skip to main content
Use this page when a runner is running but a build did not go as expected.

​
Read the runner’s log

The runner prints one line per event, timestamp first. Failure lines also go to standard error.
#3 is the item number. A line ending continuing or retrying next tick is a failure the runner absorbed. unexpected error: <message> is one it did not expect; it keeps going. A repeating failure is logged once, then every tenth tick as still failing: <message> (N ticks), then recovered: when it stops. A drain says what it is waiting on, and repeats it every thirty seconds. To stop taking work, run outerlayer runner stop. It waits for running builds to finish. outerlayer runner stop --now ends them as stopped.

​
Find out why a build ended

Start from the work item, not the host. A host build records how it ended, the session that ran in it, and what the runner did around the agent.
  1. Run outerlayer work status --item <n> --json, or ask the MCP server for get_work_item. The item’s claims list every attempt, newest first. Each has an id, an outcome, the stage it ended in, an exitCode and a reason. Outcomes and reasons says what they mean.
  2. Find the attempt’s session. Each session on the item has a claimId. The one that matches the attempt’s id is the session that ran in it. It shows what the agent was doing when the attempt ended. A session linked by a person or a pull request has a claimId of null.
  3. Print the runner’s log for the attempt with outerlayer work logs --item <n>. It prints the newest attempt’s log. Add --attempt <id> for another attempt. The MCP tool is get_runner_log. When the attempt ended in provision, sync, cleanup or report, the log ends with the last 200 lines of that stage’s log.
  4. Read the item’s review threads with outerlayer work threads --item <n>, or get_work_item_threads, to see what a person asked for.
outerlayer work logs exits with an error when the attempt is still running, because the runner sends its log when it releases the attempt. It also exits with an error for an attempt that a runner older than this feature released. Log in to the host only when these do not answer your question. Never signal a build’s processes by hand on a runner host. A build ended that way is lost, and nothing on the item can show who did it.

​
When the runner did not end the command

A command can die of a signal the runner never sent. An operator’s kill, the kernel’s out-of-memory killer and a supervisor all do this. The runner then records one of two reasons:
  • the command exited 137, the status of a process killed by SIGKILL; the runner did not end it. The status is 128 plus the signal number. A process killed by a signal reports that status, and a program can exit with it on its own, so the runner cannot prove a kill.
  • the command was killed by SIGTERM on the host; the runner did not end it. The command’s process on the host died of a signal, and there is no exit code.
Either reason points to a cause outside the runner. A build the runner ended itself keeps its own outcome and reason: timed_out, idle, lease_lost or stopped. A build the kernel killed for memory keeps the out-of-memory reason.

​
Outcomes and reasons

Every build ends with an outcome. A failed build often also records a reason naming the cause. outerlayer runner logs <item> prints the newest build’s log. Other reasons are a sentence, or a line a hook wrote; see The cleanup hook.

​
When nothing is building an item

An item that is not being built can carry a waiting reason. The item page shows it under Not building, and the item reads Not picked up when a build request waits and no runner takes it. outerlayer work status --item <n> --json and get_work_item return it as waiting. It is null while the item is building, or queued with a runner to take it. A runner that has not heartbeated within three of its poll intervals is not counted.

​
When a lease expired

A host build can end without a release: the machine lost power, or the runner was killed. Nothing writes the end, so the lease runs out. The item page lists the attempt in Last build with the outcome lease expired, the runner’s name, and the time of the runner’s last renewal. claims in outerlayer work status --item <n> --json marks it with leaseExpired: true and lastRenewedAt. A lease that ran out uses up its build request like any other claim that was not released as interrupted or lease_lost. To retry, run outerlayer work build or select Rerun on a host.

​
When the gateway refuses a signature

Each refusal is a 401, logged to the organization audit log as runner_key_signature_refused.

​
When a call to the platform does not land

The runner retries the claim, renew and release for up to two minutes when the gateway does not answer. Refusals are never retried. A release that still fails is logged without released:
runner status --recent marks it (release failed), and the next runner start sends it again. The claim lapses fifteen minutes after the last renewal, and the item returns to the queue. If someone released the claim first, the runner logs lease was already released, with the outcome the platform stored and the outcome this attempt ended with. The platform keeps the outcome it stored first.

​
When the runner holds off for disk

A runner that takes no work and logs this has less free disk than it needs:
The runner logs it once, keeps polling and keeps renewing the leases of builds that already run. outerlayer runner status shows the same line, and outerlayer doctor warns about it. GET /v1/hosts lists the runner with lowDisk, the free and needed bytes. Its status is not changed by this. The runner measures the file system that holds its own directory, <config dir>/runner. Free space there, or lower runner.housekeeping.minFreeDisk. When there is room the runner logs free disk is back above the minimum · taking work again and claims on its next poll, with no restart. By default the minimum equals runner.build.disk: room for one more build. Housekeeping removes what builds leave behind, which is usually enough. If the disk is full of something else, such as Docker images a person pulled, the runner does not remove it. See Housekeeping.

​
When the runner is blocked

A runner that takes no work and logs claiming nothing: followed by a reason has failed one of its own checks. What the runner checks, and how it reports a failed check, is in When the runner claims nothing. A typical line:
Once the check passes, the runner logs the runner can run its work again · taking work and claims on its next poll. Two fixes take effect only after a service restart: /dev/kvm access and the Claude credential. The others take effect on the next poll. outerlayer doctor reports the KVM group and the credential of the running service, not of your shell.

​
When the Image build limit check fails

The runner builds every image in its own builder, a container with a memory limit of runner.build.memory, which needs Docker’s Buildx plugin. When Buildx or its docker-container driver is missing, or older than 0.18, outerlayer doctor fails the “Image build limit” check, and the runner claims nothing. It logs claiming nothing: with the reason, and its heartbeat carries the same reason. Install the plugin: docker-buildx-plugin from Docker’s apt repository, or docker-buildx from Ubuntu’s own packages. Then run outerlayer doctor again. The runner takes work again by itself, with no restart. A runner that has Buildx but is still starting its builder logs the runner is still preparing its image builder. The first start pulls the BuildKit image, which takes a moment. If the reason does not change, it names why the builder did not start, for example that Docker Hub cannot be reached. The builder container does not use a proxy on the host’s loopback address. So a daemon that reaches the network only through one needs a proxy the container can reach.

​
When an image build runs out of memory

A recipe step that uses more than runner.build.memory is stopped by the kernel, in the runner’s image builder. The build is released failed at the provision stage with this reason:
Only that build ends. The builder and the other builds carry on. The log named in the reason shows the step. Raise runner.build.memory, and check that concurrency times build.memory still fits in the host’s memory less 2 GiB. When the limit changes, the runner replaces its builder once the image builds it is running have ended. The next image build starts with an empty cache.

​
When a build runs out of memory

In a service that sets Delegate=yes, each microVM build’s Firecracker process runs in a cgroup of its own. When the kernel kills it at the cgroup’s limit, the build is released failed with this reason:
Only that build ends. The runner and the other builds carry on. Raise runner.build.memory, and check that concurrency times build.memory still fits in the host’s memory less 2 GiB. A build killed below its limit means this host needs more reserved memory. It needs more than the 1 GiB reserved for Firecracker or the 2 GiB reserved for the system. Lower runner.concurrency to leave more room. A service without Delegate=yes gives microVM builds no limit of their own, and outerlayer doctor warns. Run outerlayer runner install --service --replace to write the unit this CLI generates. A container build’s limit is Docker’s --memory at build.memory.

​
When an item comes back

A build request is used up by any claim after it: a host build released ok, incomplete, failed, timed_out or idle, a claim that lapsed, or a local session. So a failed build is not claimed again. To retry, run outerlayer work build, or select Rerun on a host in the Last build block of the item page. The control is offered when no build is running and no request is unused. It also needs either no open pull request on the item, or a review that waits on an agent. With an open pull request, the rerun starts an amend instead of a build. work build --local does not queue the item. interrupted, lease_lost and exit 75 leave the request unused, so the item stays available. So does stopped when the operator ended the build on the host. A build stopped through OuterLayer uses its request up. An item that still needs work comes back through the amend queue when a person sends notes or fails it, not through the implement queue. A failed amend build is not retried until a person fails the item again, or a member selects Rerun on a host. While no session is linked, the item’s Sessions list names who holds a live claim. It names a runner by name, or a person’s name followed by “(local session)”. When nothing is queued it names outerlayer work build. Run it yourself gives the command that starts a session by hand.