CLI
The reference for the shadowlap command: every command, its flags, how it exits and the JSON it
prints. Read it for the exact behaviour of a flag or the fields of a report. The
quickstart walks through a first run.
shadowlap run -- node agent.js # record a run
shadowlap replay 01M2P9PVCP --strict # replay it; exit 1 if it diverges
shadowlap blame 01M2P9PVCP # find the model call where a failed run was lost
shadowlap test goldens/* # replay the packed goldens, as CI doesOn this page
- Commands
- Requirements
- Global options
- Where a store is created
- Exit status
- JSON output
- demo
- run
- When the command never starts
- Stopping a run
- Run summary as JSON
- replay
- What a replay keeps
- Checks on a replay
- Repeating a fork
- Replay summary as JSON
- calibrate
- mcp
- dashboard
- ui
- show
- ls
- search
- diff
- label, note
- blame
- verify
- repair
- fix
fix lsfix showfix titlefix checkfix prfix apply- decisions
- Decisions report as JSON
- migrate
- The fields
migratechanges - What
migratewrites - Migrate report as JSON
- migrate apply
- test
- The command a golden runs
- Test history and gates
- A golden's injection test
- Test report as JSON
- Error-path presets
- bisect
- pack, unpack
- What
packwrites - What
unpackchecks - serve
- check
- injection
- Proving cause with
--clean - Injection report as JSON
- coverage
- Packing a capture
- Coverage scan as JSON
- init
- What
initfinds - What
initwarns about initfindings as JSON- doctor
- normalize, redact
redact preview- skill
- import
- Traces
- The traces in a window
--json- export html
- export dataset
- export pairs
- export har
- export otel
- rm, gc
- A dry run
- Deleting what a dry run listed
gcas JSON- tasks
tasks scoretasks servetasks models- push, pull
- share
- store
store setstore checkstore showstore policystore cistore push-from-ci- The configuration file
- Environment
- See also
Commands
#The commands, in the groups shadowlap --help lists them in:
| Group | Commands |
|---|---|
| Try it | demo |
| Record and replay | run, replay, calibrate, mcp |
| Look at runs | dashboard, ui, show, ls, search, diff, label, note |
| Find and fix | blame, verify, repair, fix |
| Preview a change | decisions, migrate |
| Test | test, bisect, pack, unpack, serve, check, injection, coverage |
| Set up | init, doctor, normalize, redact, skill |
| Import | import |
| Export and clean up | export html, export dataset, export pairs, export har, export otel, rm, gc |
| Train | tasks |
| Team store | push, pull, share, store |
What every command shares: the requirements, the global options, the exit statuses, the JSON output, the configuration file and the environment variables.
Requirements
#The CLI runs on Node 22.17 or later, but not on Node 23. Node 23's built-in SQLite has no full-text
search, which every store's index needs. Node 22.16 has it, but crashes shadowlap ui when its
terminal closes.
- Another Node: the CLI names the Node it needs and exits 2.
- A SQLite without full-text search: a Node in that range can have a SQLite built without it,
such as a custom build. Opening a store on it says to use an official build, as
doctordoes.
Global options
#shadowlap takes these options:
| Flag | What it does | Default |
|---|---|---|
--db <path> | The store to use | . |
-v, --verbose | Print the sidecar's log on stderr. Give every attempt its own line where a terminal folds them, as blame, verify and repair do | — |
-h, --help | Print the help; after a command, that command's, with its options and these | — |
-V, --version | Print the version number, before any command | — |
Every command takes --db and --verbose, before the command or after it:
shadowlap --db other.db ls and shadowlap ls --db other.db are the same. Run ids can be abbreviated
to any unique prefix.
Where a store is created
#A store is created by the first command that records into it. These commands never create one:
- Commands that only read it:
ls,show,search,diff,pack, the exports,fix lsandtasks. - Commands that change only what it already holds:
label,note,rm,fix title, andcoverage, which keeps its scans there.
Where there is no store, they find nothing. When the project's store is in a folder above (run from
src/, say), they name it.
Exit status
#Every command ends with an exit status a script or a CI step can tell apart: 1 for a check that failed, 2 for a command it can't run as given, 3 for a verdict that couldn't be reached, 70 for a bug. Each command's section says which statuses it ends with. The statuses are:
| Status | Meaning |
|---|---|
0 | Ok: the command did what it was asked, and what it judged passed |
1 | Failed: what it judged failed, or the work could not be finished |
2 | Usage or configuration: a command it can't run as given |
3 | Inconclusive: a gate or check that could not decide |
70 | Internal error: a bug in shadowlap |
128 + n | Stopped by signal n |
1, failed, covers:
- a check, gate or test that failed;
- a replay that diverged where it may not (
--strict,test); - a change verify or repair rejected;
- a search that found nothing;
- work that could not be finished: a host that didn't answer, a patch that didn't apply, a store another process held too long.
2, usage or configuration, covers:
- a flag, argument or command it doesn't take;
- a run or fix it can't find, or can't use yet (one still recording);
- a file it can't read, or an invalid
shadowlap.config.json; - a live command with no key;
- a recorded command it needs
--yesto run; - a replay that would run a write shadowlap never recorded again;
- an agent on a Node too old for the guard.
3, inconclusive, covers:
- a gate or check that could not decide. That is
0when it only warns: under--inconclusive warn, or, with none given, for a gate against history whose golden has fewer than 30 sessions that ran out of attempts with no decline or drop in them; - a budget spent before a verdict;
- the provider unavailable.
70, internal error: the message says so. Run the command again with SHADOWLAP_DEBUG=1 to see
where it happened, and report it.
128 + n, a stop signal: 130 for Ctrl-C, 129 for a closed terminal (SIGHUP), 143 for
SIGTERM. That holds however the command wound down, for the dashboard, ui, serve, tasks serve,
mcp, demo and replay --watch too. It holds for a signal that came during its last work too, such as
repair removing its worktree after its last verify. A signal in the last moment before the process
exits, a few milliseconds, may find it already ending with its own status.
run and replay end with the agent's own status where none of these applies, as the agent would
have ended without shadowlap: 127 for a command that isn't on the PATH, and 128 + n for an agent a
signal stopped. SHADOWLAP_DEBUG=1 prints the stack of any error, for a bug report.
JSON output
#Most commands can print what they did as JSON, for scripts, CI and coding agents:
--jsonprints the result as JSON on stdout. The commands that report progress as they work (blame,verify,repair,fix apply,decisions,migrate,migrate apply,bisect,injectionandtasks score) still print it on stderr.--summary-json <file>(run,replay) andtest --json <file>write the JSON to a file.
Each command's section lists its fields. This table says where:
| Command | What it prints |
|---|---|
run --summary-json <file> | The run's summary (fields) |
replay --summary-json <file> | The replay's summary and checks, or every attempt and the report (fields) |
show, ls, search, diff | A run, the runs, the matches, a comparison (show, ls, search, diff) |
label | The run's labels (fields) |
blame, verify, repair | Their reports (blame, verify, repair) |
fix ls, fix show, fix pr, fix apply | The dashboard's read model of Fixes (fields) |
decisions, migrate | The report the dashboard reads (decisions, migrate) |
migrate apply | What it applied, and the goldens it left behind (fields) |
test --json <file> | The session's report (fields) |
bisect, check, injection, coverage | Their results (bisect, check, injection, coverage) |
calibrate, init, doctor, skill install | What each found or did (calibrate, init, doctor, skill) |
normalize ls, redact ls, redact preview | The rules, and what they would mask (rules, preview) |
import | The runs it made and what it skipped (fields) |
gc | What it deleted or would delete (fields) |
tasks, tasks score, tasks models | The tasks, a score, the models (Training) |
push, pull, share, store | What went and came, the link, the store (push, pull, share, store) |
shadowlap demo
#shadowlap demo [--keep <dir>]Runs the whole loop on a refund agent, with no key and nothing to clone, as the quickstart shows it. It needs git.
- It makes a temporary folder a git repository, which it removes at the end.
- It writes the agent there, the check its runs must pass, and a
shadowlap.config.jsonnaming the repairer. - It starts a scripted model on 127.0.0.1, which answers the agent in Anthropic's Messages API and the repairer in OpenAI's Chat Completions. The demo says it is scripted.
- It runs
run,label,replay,blame,repairandfix applythere, each printing as it does in a terminal. After blame, a line says how to read a probe's verdict from the probes it made. - It ends with how long that took, how many requests the scripted model answered, and where they went: the recordings, blame's attempts and repair's.
In a terminal, its own lines wrap at whole words to the terminal's width, as blame's do, and each command it runs is shown as it is typed.
Your own model keys and endpoints are left out of its environment, and telemetry is off for it.
demo takes one flag:
| Flag | What it does | Default |
|---|---|---|
--keep <dir> | Run in this folder, a new or an empty one, and keep it, to look around with shadowlap dashboard | A temporary folder, removed |
demo exits with:
| Status | Meaning |
|---|---|
0 | Every step did what the demo expects |
| A step's status | A step that didn't, which it names |
2 | A --keep folder that isn't empty, or no git |
128 + n | A stop signal: Ctrl-C, SIGTERM, or the SIGHUP of a closed terminal |
A stop signal ends the demo once the command it runs then has ended, which a terminal's signal stops too. It removes its folder and ends with 128 + the signal's number.
shadowlap run
#shadowlap run [--summary-json <file>] [--guard warn|off] -- <command> [args...]Records a run: it runs the command under shadowlap and exits with the command's exit code.
shadowlap run -- node agent.js
shadowlap run --summary-json run.json -- python agent.pyBefore the command starts, run:
- starts a sidecar;
- sets
ANTHROPIC_BASE_URL,OPENAI_BASE_URL,GOOGLE_GEMINI_BASE_URL,SHADOWLAP_URLandSHADOWLAP_RUN_IDfor the child; - preloads in-process capture, the network guard and recorded user input into Node and Python children.
The run's status is completed, failed (a non-zero exit) or crashed (interrupted: Ctrl-C, or a
stop signal that reached shadowlap or the agent).
run takes these flags:
| Flag | What it does | Default |
|---|---|---|
--summary-json <file> | Also write the run's summary as JSON to this file (fields) | — |
--guard warn|off | The network guard for connections that bypass shadowlap | warn |
run exits with:
| Status | Meaning |
|---|---|
| The agent's | The agent's own exit code |
127 | The command isn't on the PATH, or can't start for another reason (below) |
128 + n | A signal ended the agent: 130 for SIGINT, 129 for SIGHUP, 143 for SIGTERM |
On Windows, a command that is a batch file, such as npm start or npx tsx agent.ts, runs through
cmd.exe, its arguments passed on as given.
What the run did unrecorded is kept on it, for its page in the
dashboard: the hosts the guard saw it reach
without shadowlap, and the tools init finds in the project with no declared effect that the
run never recorded. At its end, run names the tools the model called that the run recorded no call
of, which a replay runs for real (details).
When the command never starts
#- Not on the PATH:
run(andreplay) says so (✗ pythn: command not found onPATH). Wherepythonis missing butpython3is there, it gives the command withpython3. It keeps no run and exits 127. - Can't start for another reason: the same, with that reason in the system's own words: a path to a file that isn't there, or a program that is there but can't run.
- Under
testandbisect: a golden whose command never started fails with the same words. One replayed in a working directory that doesn't exist (--cwd, or a folder a commit doesn't have yet) says so, rather than blaming the command.
A Node agent on a Node before 18.18 never starts either. That Node refuses the preload (--import in
NODE_OPTIONS) and exits 9 before any of the agent runs:
runandreplaythen say the agent's Node is too old for shadowlap's guard, which needs Node 18.18 or later, and keep no run.testfails the golden with those words.replay --repeat,blame,verify,repairanddecisionsstop with them, exit status 2.
Where the agent prints straight to the terminal, as under run and replay, shadowlap can't read
that refusal. So it asks the agent's Node again: the one the command names, else the node on its
PATH.
A Node that a launcher or a script picks, such as npx node@18.17.0 or volta run, can't be asked.
An agent it starts that exits 9 before any call keeps its run. The summary adds that if Node printed
--import= is not allowed in NODE_OPTIONS above, the agent's Node is too old.
Stopping a run
#A stop signal reaches the agent once:
- In a terminal, the agent shares shadowlap's process group, so the terminal's Ctrl-C reaches
both, as it would reach the agent alone. The agent keeps the terminal to read from (
/dev/ttyincluded, as a password prompt opens it). - As a dashboard job, it shares the job's process group, which the dashboard stops as a whole, and kills when it can't wait.
- Elsewhere (CI, a script), the agent leads a process group of its own. shadowlap passes it each SIGINT, SIGTERM or SIGHUP it gets, once. An agent still running three seconds later is killed with its group, and a second signal kills it at once.
Closing the terminal (SIGHUP) ends the run as interrupted too, and kills an agent that ignores it. A signal that comes while the run's judge checks are asked is kept for when they are done; a second one ends the run as interrupted at once.
An agent a signal ended exits with 128 + its number: 130 for SIGINT, 129 for SIGHUP, 143 for SIGTERM.
replay --repeat, test, blame, verify and decisions --from-code stop their agents the same
way. An agent still running when shadowlap itself exits is killed with its group.
Run summary as JSON
#--summary-json writes { "exitCode", "summary", "unrecordedTools" }:
| Field | Meaning |
|---|---|
exitCode | The agent's exit code |
summary | The run's summary (below) |
unrecordedTools | The tools the model called that the run never recorded: each name, count, firstCallNo, effect, why and fix, as show --json lists them |
The summary has these fields:
| Field | Meaning |
|---|---|
runId, mode, status | The run, its mode and its status |
total | Calls |
byOrigin, byKind | Calls per origin and per kind |
upstreamRequests | Live calls that reached a provider |
injected | Calls --fail-at answered |
refused | Calls answered with why they weren't sent: an intervention that couldn't be applied, or a placeholder key a live call would carry |
imported | Whether the run came from elsewhere: a pack someone recorded, or a capture |
tokensNotRespent | Tokens the replayed calls didn't spend again |
cost | spent, saved and captured in USD, and unknownModels with no price |
durationMs | How long the run took, in milliseconds, once it has ended |
notReached | Recorded calls the replay never made |
extraReads | Clock, random and pure reads the recording held no counterpart for, answered live while the replay went on (extra reads) |
warnings | The run's warnings |
divergence | The divergence report, or null |
judge | With judge checks: what the judge's calls did and cost (below) |
Every count and cost is the agent's. A run with judge checks has judge too: what the judge's calls
did and cost apart from them. It has calls, replayed from the recording, live, upstreamRequests,
spent, saved, unknownModels, and notAsked: kept for runs recorded when a call no one answered
was kept, and 0 since.
A replay whose every request matched but that stopped before its recording did has a divergence too,
of type ended_early, kept on the run like any other:
callNo,kindandnameare the first recorded model call, or call of a tool that isn't pure, it never made;replayedBeforeis the calls it was served;recordedis the calls the recording holds.
shadowlap replay
#shadowlap replay <run> [options] [-- <command> [args...]]Replays a recorded run; forks at the first changed call. It re-runs the recorded command unless one
follows --.
shadowlap replay 01M2P9PVCP # replay as recorded
shadowlap replay 01M2P9PVCP --strict # exit 1 if it diverges, for CI
shadowlap replay 01M2P9PVCP --fail-at 12=429 # make model call 12 fail with a 429
shadowlap replay 01M2P9PVCP -- node agent.js # replay against your own commandA clean replay needs no real key. The agent gets a placeholder key for each provider whose key the recording's environment held and this one has no credential for:
- A live call that would carry one, after a fork, is refused with one line:
a live call needs a real key: set ANTHROPIC_API_KEY, or keep the replayoffline. - Whatever the agent signs in with itself (a key you set, one it loads from
.env, a profile, a token provider, an instance role) is used as it is.
A run imported with unpack, or a replay of one, carries a command someone else recorded. replay
shows it and asks before running it. Without a terminal to ask on, it refuses: pass --yes once you
have read the command, or give your own after --.
These flags decide how the replay runs:
| Flag | What it does | Default |
|---|---|---|
--strict | Exit 1 if the replay diverges, or ends before its recording does (for CI) | — |
--writes block|stub|allow | What write tools do after divergence | block |
--reads live|replay | What read tools do after divergence | live |
--live-tool <name> | Force this tool live: its first call, and every call of it after a fork (repeatable). Use it after fixing a tool | — |
--live-input | After the fork, ask for what a person types instead of serving the recorded answers in order (recorded input) | — |
--expect <json|@file> | Checks the replay (each attempt, with --repeat) must pass besides exiting 0, as for blame: '{"tools":{"send_ | The checks shadowlap.config.json gives the recording (below) |
--watch | Replay again whenever a file in the project changes (below). Ctrl-C to stop | — |
--summary-json <file> | Also write the summary (with --repeat, every attempt and the report) as JSON to a file (fields) | — |
--guard warn|block|off | The network guard for connections that bypass shadowlap. run accepts warn|off; test defaults to block | warn |
--allow-unrecorded <names> | Run these tools again for real, though the recording never recorded them: names separated by commas (repeatable; below) | — |
-y, --yes | Run the recorded command of a run imported from a pack, or pulled from the team store, without asking | — |
These flags change what the replay does. Interventions says what each one does to the conversation prefix, and what each provider takes:
| Flag | What it does | Default |
|---|---|---|
--diverge-at <call> | Force call n live | — |
--fail-at <call>=<failure> | Fail call n instead of serving it, so the agent's own error path runs (repeatable; below) | — |
--override <call>=<json|@file> | Substitute a recorded tool result; the run forks at the first model call that sees it (repeatable) | — |
--inject-result <call>=<text|@file> | Plant an instruction in what read tool call n returned, as an attacker would leave it; the rest of the result stays (repeatable; below) | — |
--inject <call>=<text> | Add an instruction at model call n, and keep it in place in every later request of the conversation (details) | — |
--inject-as user|system | user: in the user turn the model answers next, on every provider. system: a system message after the conversation; refused on Gemini and Bedrock Converse | user |
--prompt-from <call> | Apply your changed system prompt and tool definitions only from model call n: earlier model calls replay from the recording (details) | — |
--set <call>:<key>=<value> | Set a request field from model call n on, where the provider keeps it; a dotted key is a path (repeatable; below) | — |
--model <call>:<model> | Switch the model from model call n on: in the body, or in the path on Gemini, Vertex AI, Bedrock and Azure | — |
These flags run the replay more than once:
| Flag | What it does | Default |
|---|---|---|
--repeat <n> | Run the replay n times and report how the attempts differ after the fork (Repeating a fork) | — |
--concurrency <n> | With --repeat: attempts running at once | 1 |
--chaos | Run the error-path presets on this run instead of one replay (below) | — |
--presets <list>, --chaos-tool <name>, --inconclusive fail|warn | With --chaos: as for test --chaos | --inconclusive: fail |
More on some of them:
--fail-at: a model call takes an HTTP status (--fail-at 12=429, details). A tool call takestimeout,error,error:<message>or a status (--fail-at 19=timeout), and the tool does not run (details).--inject-result: text that starts with@is written@@…(details).--set: the field is where the provider keeps it:temperatureisgenerationConfig.on Gemini, andtemperature inferenceConfig.on Bedrock Converse (details).temperature --watch: what a replay writes itself (the agent's own files, the store, the--summary-jsonfile) starts no other. A file that changes while a replay runs is taken for one of its writes, and the replay says so, so a save made then needs saving again.--allow-unrecorded: it is for tools that write, or that nothing says what they do. Without it, a replay whose model called such a tool is refused before anything runs, exit status 2.--chaos: a fork for each preset, judged by--expect, or, for a golden's run, by itschaos.expector the write limits of itsexpect. Writes are stubbed unless--writesis given, and--summary-jsonwrites their report. It doesn't combine with the flags that change a replay:--override,--fail-at,--inject-result,--inject,--set,--model,--prompt-from,--diverge-at,--live-tool,--repeat,--watchand--strict.
A change the recorded request can't take (a system message on Gemini, a field Converse has no place for, a write tool to plant in) is refused before anything runs, with the reason. An intervention the replay never reached is named in its summary.
replay exits with the agent's status, except:
| Status | Meaning |
|---|---|
1 | With --strict, the replay diverged or ended early |
1 | A check failed |
1 | Under --guard block, the network guard never watched the agent (below) |
2 | The run is still recording |
3 | The agent exited 0 and a check could not be decided |
Under --guard block, the guard never watched the agent when it made model or tool calls and either
no shadowlap preload loaded in its processes, or one of them runs with no guard. The process with no
guard is the Claude Code CLI the Claude Agent SDK runs, while it can still reach something the guard
can't watch and guard.accept doesn't take it (details). Its summary then
reads ✗ Replay not watched, never clean.
With --repeat, the status is:
- 0 only when every attempt asked for ran, exited 0, was watched by the guard under
--guard block, and passed its checks; - 3 when all that kept an attempt from passing is a check that could not be decided;
- 128 + n for a
--repeatthat a stop signal cut short, 130 for Ctrl-C; - else 1. With
--strict, it is 1 also when any attempt diverged or ended early.
A run still recording is refused, exit status 2, until it has ended: it has no end to replay to yet,
as the dashboard's Replay and Fork wait for it. One whose recording process was killed (see
gc) replays as it is.
What a replay keeps
#The replay keeps the options that change what it does on its run: --fail-at, --override,
--inject, --inject-result, --model, --set, --prompt-from, --diverge-at, --live-tool, and
--writes or --reads when not the default. Its page in the
dashboard then names what made the fork, and the
fork menu can make it again.
The run keeps each flag exactly as given, an override's result included, beside a short form to show:
- an override read from a file is named by the file (
--override16=@policy.json); - one given inline is named by its call;
- a long instruction is shortened.
A copy of the run (pack, export html) keeps only the short form, with the project's redaction
rules applied to it.
Checks on a replay
#A replay is judged by checks when it has them: --expect, or, without it, the tests entry of
shadowlap.config.json for the recording the run came from (the pack it was unpacked from, or the
golden shadowlap test replayed). The checks are those of an outcome check,
calls and when rules included.
- The replay exits 1 when a check fails, and 3 when the agent exited 0, none failed and one could not be decided.
- The checks are kept on the run, with the
--expectthey came from, so making the fork again from the fork menu judges it the same way. - With
--repeat, every attempt is judged. The report says in how many attempts each check failed, could not be decided (inconclusive) or did not apply:
Checks (--expect)
✗ send_email called 0 times: failed in 2 of 3 attempts (#1, #3)The agent's output is captured, while still shown, only when a check reads it (output, or a judge
check of the output). A replay that plants text with --inject-result and has no checks says that
nothing judged whether the agent obeyed it.
A judge check is asked once the agent exits, before the replay's run ends. A clean
replay is answered from its recording, and a fork asks the judge live; either way the call is the
replay's own. Its line gives the verdict, the judge's reason and where it came from. Its entry in
checks has status (passed, failed, or inconclusive when it was not judged), reason,
source and judge.
Repeating a fork
#Everything after a fork is live, so a model may do something different each time. One good attempt
doesn't show that a change fixed the behavior. --repeat n runs the same replay n times, each
attempt its own run, and summarizes them:
⑂ 5 attempts from 01M2P9PVCP, each forked at call 11
5 of 5 exited 0 · 20 upstream requests · 16.2k input tokens not re-spent
Paths after the fork
5× open_ticket (stubbed) → open_ticket (stubbed) → open_ticket (stubbed)
every attempt took the same path
Tools after the fork
tool attempts calls
open_ticket 5 of 5 15 (15 stubbed)
Final model responses: 5 differentA path is the sequence of tool calls after the fork and what happened to each: blocked, stubbed, overridden or an error.
- Tools called between the same two model calls match in any order, since agents often run them at once.
- Attempts with the same path are grouped, most common first.
- Arguments aren't part of a path.
shadowlap diffcompares two attempts in full, and the report names the pair to start with. - The prefix before the fork is served from the recording, so it costs nothing however many attempts there are. If attempts fork at different calls, the report says the prefix isn't deterministic.
- An attempt that stopped before its recording did, with every request matching, forked nowhere but
is no clean replay either. The first line counts those apart (
· 1 of 5 ended early, at call7), and each attempt in the JSON report hasendedEarly: the first recorded call it never made, ornull.
With --concurrency, attempts run at the same time through one sidecar, and each attempt's output is
printed whole when it finishes. --strict exits 1 if any attempt diverged or ended early. Otherwise
the exit status is as above: 0 only when every attempt ran, exited 0, was
watched under --guard block and passed its checks, and 3 when all that kept
one from passing is a check that could not be decided.
Replay summary as JSON
#--summary-json writes what run writes, with these differences:
| Field | Meaning |
|---|---|
exitCode | The agent's, or 1 where the guard never watched it under --guard block |
checks | When the replay has checks: each check as it came out, with status, ok, label and detail |
unrecordedTools | The tools this replay ran for real without recording them |
With --repeat, it writes { "attempts", "report" }:
| Field | Meaning |
|---|---|
attempts | Each attempt's exitCode (so counted), summary, checks and unrecordedTools |
report | The report the terminal prints: parentRunId, attempts, forkedAt, paths, tools, distinctFinalResponses, succeeded, spent, saved and tokensNotRespent |
shadowlap calibrate
#shadowlap calibrate [run] [--write] [--max-rounds <n>] [--json] [--guard <mode>] [--allow-unrecorded <names>] [-y]Replays a recording offline, repeatedly, and proposes a normalizer for every value that differs
between two runs of the same code, until the replay is clean, trying up to --max-rounds replays.
It makes no model calls, and writes are blocked. See Calibrate.
Without a run named, it takes the recording the dashboard's setup made for this project. Else it takes the newest finished recording made here: one that completed or failed, not one imported from a pack or a capture. When there is none, it asks you to record a run first.
Each round is printed as it ends. A recording that is already clean ends with "Nothing to add". Afterwards, it removes the blobs its own replays left that nothing else uses, unless a run is recording.
calibrate takes these flags:
| Flag | What it does | Default |
|---|---|---|
--write | Add the proposals to shadowlap.config.json, and say their numbers | — |
--max-rounds <n> | How many replays to try first | 8 |
--json | Print the result as JSON when calibration ends (below) | — |
--guard <mode>, --allow-unrecorded <names>, -y | As for replay | --guard: warn |
calibrate exits with:
| Status | Meaning |
|---|---|
0 | The replay ends clean, with or without proposals |
1 | Calibration stops before it is clean |
2 | The run named is still recording, there is no recording to calibrate, or its model called an unrecorded tool that writes |
--json prints the result:
| Field | Meaning |
|---|---|
runId | The recording calibrated |
source | given when you named the run, setup for the setup's recording, newest when it picked the newest |
clean | Whether the replay ended clean |
rounds | Each round it ran: its divergence, and the normalizers it proposed |
normalizers | The proposed normalizers, each with why it was proposed |
written | Whether --write added any rule |
added | The numbers of the rules it added; a proposal already in the file isn't counted |
stopped | Why it stopped, when it isn't clean |
shadowlap mcp
#shadowlap mcp [--effect <tool=effect>]... -- <server command>
shadowlap mcp --url <server> [--header "Name: value"]... [--allow-http] [--effect <tool=effect>]...
shadowlap mcp --url <server> --listen <address> [--header "Name: value"]... [--allow-http] [--effect <tool=effect>]...A proxy for an MCP server. Put it in the client's server configuration in place of the server itself; the client talks stdio to it. See MCP servers.
- The server: it starts a local server from its command, or speaks Streamable HTTP to a remote
one at
--url, sending each--headerwith every request. - What it records:
tools/callis recorded, replayed, blocked or stubbed like an SDK tool call, so a remote write never runs again on replay unless the replay allows writes (--writes allow). Every other message passes through. - Effects: they come from the server's tool annotations (
readOnlyHint→read, otherwisewrite), overridden by--effectormcp.effectsin the config. - Its run: the proxy finds its run even when the client starts it with a minimal environment. When it was started under shadowlap but can't record, it refuses write tools rather than run them unrecorded.
- On Windows: a server command that is a batch file (
npx -y …) runs through cmd.exe.
mcp takes these flags:
| Flag | What it does | Default |
|---|---|---|
--url <server> | Speak Streamable HTTP to a remote server at this endpoint | — |
--header <header> | Send this header with every request to the --url server; $VAR or ${VAR} is read from the environment (repeatable) | — |
--allow-http | Send headers over plain http to a host other than this machine | Headers go over plain http only to this machine |
--effect <tool=effect> | pure, read or write for a tool, overriding the server's annotations (repeatable) | The server's annotations |
--listen <address> | Serve the --url server over Streamable HTTP at http://<address>/mcp (below) | Stdio |
With --listen, the proxy serves the --url server over HTTP instead, for a client that speaks
HTTP. It runs until stopped.
- It listens on a loopback address:
127.0.0.1:8931, or a port alone. - Each session goes in the server's own session id.
- It records into the run its command started. Started on its own, it records into the run of the shadowlap command that started the session's client.
- It refuses writes in a session it finds no run for.
mcp exits with:
| Status | Meaning |
|---|---|
| The server's | In front of a command: the server's own status |
127 | In front of a command: the server can't be started |
0 | With --url: the client closed its end |
2 | A flag, an address or a config to fix |
70 | A bug |
128 + n | A stop signal, which a --listen proxy runs until (130 for Ctrl-C) |
shadowlap dashboard
#shadowlap dashboard [run] [--call <n>] [--port 4777] [--no-open] [--read-only]A local web UI over the store: runs and forks, a strip of calls by kind or origin, the conversation at any call, playback of streamed responses, fork diffs, comparison, the activity feed, and replay and fork actions. See Dashboard.
It listens on 127.0.0.1, with a session token in the link it prints.
dashboard takes these flags:
| Flag | What it does | Default |
|---|---|---|
--call <n> | With a run: open at this call | — |
--port <port> | The port to listen on; --port 0 asks for any free one (below) | 4777 |
--no-open | Print the link instead of opening a browser | — |
--read-only | Show every page and change nothing (below) | — |
A port that is taken, or that the system keeps for itself (a range Windows reserves), falls back to a
free one, and --port 0 asks for any free one. Either way, the link names the port it got.
With --read-only, actions, labels, notes, uploads and deletes are refused, and the dashboard says
why. It needs a store to show, and never creates one.
shadowlap ui
#shadowlap ui [run] [--call <n>]Opens the terminal UI. A fork opens at its divergence, unless --call names another
call.
ui takes one flag:
| Flag | What it does | Default |
|---|---|---|
--call <n> | With a run: select this call | The divergence, for a fork |
shadowlap show
#shadowlap show <run> [--json] [--request <call>] [--response <call>]Prints a run's timeline: one row per call, then the divergence report if the run forked.
- Each row: the call's kind, name, origin (
live,replayed,blocked,stubbed,overridden), outcome, latency, and tokens or arguments. - Above the timeline: the run's status, duration, upstream requests and what it spent. For a
replay, what replaying didn't spend again (
$0.0075 not re-spent (8.1k inputtokens)). - Input tokens, here and in each row, are everything the model read: uncached, written to the
prompt cache and read from it, as
test,replayand the dashboard count them.
show takes these flags:
| Flag | What it does | Default |
|---|---|---|
--json | Print the run as JSON (below) | — |
--request <call> | Print the stored request of call n | — |
--response <call> | Print the stored response of call n | — |
What else it shows:
- An interrupted run: a run stopped before it ended (Ctrl-C, a kill, a crash, or a recorder that
went away) reads
interrupted, as the dashboard says it.--jsonkeeps the stored status,crashedorrunning. - Someone else's spend: a teammate's run pulled from the team store says who spent
it where it was recorded (
≈$0.0030 spent by maya). A run from a pack or a capture says what its traffic cost where it ran (of traffic captured). - Untrusted reads: a run with untrusted reads names them above the timeline,
each with where its mark came from (the first five, then how many more). It tags each one's row
untrusted, as the dashboard's Calls view does. - Tools it never recorded: a line names the tools its model called that the run recorded no call
of (
not recorded: Bash (14 calls from #3,writes)), which a replay runs for real (details). - Where blame lost it: a run blame searched says where it was lost, as blame's report
does:
blame: lost at model call #5 (high confidence), rescued by01M5FAB2C1, orlost gradually, around model calls #17 and #18when no one call lost it.
Untrusted reads are the reads of a tool that the config's tools, the SDK's trust or its MCP
server's openWorldHint marks untrusted, where shadowlap injection plants.
--json prints { "run", "summary", "calls", "untrusted", "unrecordedTools" }, the object the
dashboard's Provenance tab copies:
| Field | Meaning |
|---|---|
run | The run as stored (runId, parentRunId, mode, status, divergence, command, codeVersion, startedAt, endedAt, meta) |
summary | Its summary, as run --summary-json writes it |
calls | Every call as stored (runId, callNo, kind, name, fingerprint, occurrence, origin, effect, requestRef, meta, at, end, attempts) |
untrusted | Its untrusted reads in call order, as the dashboard's GET /api/injection/runs/:id lists them (below); empty if it has none |
unrecordedTools | The tools its model called that the run never recorded (below); empty if it has none |
Each of its untrusted reads, untrusted, has these fields:
| Field | Meaning |
|---|---|
callNo | The read's call number |
name | The tool's name |
from | Where its mark came from: config, sdk or mcp |
readAt | The model call that reads its result, where shadowlap injection forks; null when no model call follows it |
Each of the tools its model called that the run never recorded, unrecordedTools, has these fields:
| Field | Meaning |
|---|---|
name | The tool's name, as the model called it |
count | How often the model asked for it |
firstCallNo | The model call that first did |
effect | write, read or unknown |
why | Why it has that effect, in a few words |
fix | How to have it recorded: fence, fence-effects, mcp, none or wrap |
shadowlap ls
#shadowlap ls [--tree] [-n <limit>] [--label <label>] [--json]Lists runs, newest first, with their labels.
ls takes these flags:
| Flag | What it does | Default |
|---|---|---|
--tree | Nest forks under their parents | — |
-n, --limit <n> | The most runs to list | 50 |
--label <label> | Keep runs with this label; repeat it to require several | — |
--json | Print one object per run (below) | — |
A run stopped before it ended, or left marked running by a recorder that went away, reads
interrupted, as the dashboard lists it. --json keeps the stored status.
--json prints one object per run: runId, parentRunId, mode, status, calls, command,
codeVersion, startedAt, endedAt and labels, and these:
| Field | Meaning |
|---|---|
forkedAt | { callNo, type, kind, name, reason } where the run forked, or null (below) |
hasCommand | false for a run with no command of its own (below) |
imported | { at, from } for a run from a pack, a capture or the team store, or null (below) |
partial | The kinds of call a partial recording covers (["api"] for a capture), or null |
repeat | { group, attempt, of } for an attempt of replay --repeat, or null |
checks | { passed, total } for a run from shadowlap test with outcome checks, or null (below) |
forkedAt: a replay that stopped before its recording did hastypeended_early, at the first recorded call it never made. It lists asended early @n, never as a clean replay.hasCommand:falsefor a run imported from a capture, recorded throughserve, or recorded by a test that gave no command (its command reads<test>and the test's name). It has no command, so replaying it needs yours after--.imported: for a run from the team store,fromis the store's URL:s3://…,gs://…,az://…orr2://…. For a trace,keyis its id in the trace store. A Langfuse trace also keepsurl, its page there.checks:passedcounts only the checks that passed. When there are any,inconclusiveandnotApplicablecount those that could not be decided or did not apply; the rest oftotalfailed.
shadowlap search
#shadowlap search refund policy # every word must appear
shadowlap search '"refund policy"' --run 01M2P9PVCP -n 50 [--json]Searches what runs recorded: prompts, model responses, tool arguments and tool results. Quoted words must appear together.
search takes these flags:
| Flag | What it does | Default |
|---|---|---|
--run <run> | Search only this run | — |
-n, --limit <n> | The most matches to print | 20 |
--json | Print one object per match (below) | — |
Each match names the run, the call and where the text was found. It ends with the
shadowlap ui <run> --call <n> that opens it.
- Attempts come last: matches in attempts (blame's probes, verify's attempts,
replay --repeat's), which repeat the run they came from word for word, come after those in the runs you made, as in the dashboard's search. - The index: it is built the first time you search, and updated for runs that changed since. Each model call indexes only the turns it added to the conversation, so a long run is indexed once, not once per call.
- A run still recording is indexed from where the last search stopped, not from its first call again.
search exits 1 when nothing matches, as grep does.
--json prints one object per match:
| Field | Meaning |
|---|---|
runId, callNo, kind, name | The run, and the call the match is in |
field | prompt, response, arguments or result |
snippet | The text around the match |
runStartedAt | When the run started |
shadowlap diff
#shadowlap diff <a> <b> [--json]Compares two runs call by call, aligned by position. It exits 1 when the runs differ, like diff.
diff takes one flag:
| Flag | What it does | Default |
|---|---|---|
--json | Print the comparison as JSON (below) | — |
It prints, in order:
- What differs about the runs as a whole: their system prompts, the tools they offered (added, removed, or with a changed definition), the models they used and their code versions.
- How many leading calls are identical in request and response.
- What a replay's flag did, when one run is the other's replay and the flag that made it changed
an answer rather than a request (
--fail-at,--override,--inject-result,--diverge-at). It says what each run got there:⑂ #19 got HTTP 529 instead of an answer, by --fail-at19=529, or⑂ #16 lookup_policy returned {…} instead of{…}. - The first call whose request differs, with a structural diff and normalizer hints, and how many later requests differ too.
- Every call whose request was the same but whose response was not, such as a live re-run where the model answered differently (both answers are shown), or a tool that returned something else. A call answered the same way in another response, where only its id or token counts differ, is counted apart.
Two calls make the same request when a replay served one from the other, or when what each sent is the same, however each run fingerprinted it. What each sent covers:
- its kind and name;
- the headers and query that make a model call the call it is;
- its body, each number as it was sent, so two ids past 2^53 a double rounds alike differ.
A run recorded by an earlier shadowlap compares with its replay as the replay matched it. Its model requests, stored with their numbers rounded, compare rounded on both sides.
--json prints the comparison:
| Field | Meaning |
|---|---|
a, b | Each run's runId and calls |
identicalPrefix, prefix | How many leading calls are identical, and those calls (the first 500) |
firstRequestDifference | callNo, kind, name and the structural diff, or null |
responseDifferences | Each with sameAnswer when only the response differed, not its answer, and by, the flag, for the call a replay's flag answered |
after | Each run's course from the first difference on |
onlyInA, onlyInB | How many calls each run made beyond the other's end, when no request differed |
changes | systemPrompt, each run's system prompts by their hashes in systems, tools, models, code |
intervention | What a replay's flag changed (below), or null |
intervention is set when one run is the other's replay and its flag changed an answer:
run:aorb, the one the flag made;kind:fail,override,answer,plantorresample;flag,callNoandname;beforeandafter: what each run got there;words.
shadowlap label, note
#shadowlap label <run> bug refund-flow # add labels
shadowlap label <run> bug --remove # remove one
shadowlap label <run> [--json] # print them
shadowlap note <run> "The model skipped the refund check." [--call <n>]What people know about a run that the recording can't. Both travel in packs.
- A label says what the run is, so it can be found (
ls --label, the dashboard's search) and exported as part of a dataset (export dataset --labelgood). A label is up to 40 letters, digits, spaces and_ . : / -. - A note says why, on the run or on one call.
showprints them, and the dashboard shows a call's notes in its details.
label and note take these flags:
| Flag | What it does | Default |
|---|---|---|
--remove | label: remove these labels instead of adding them | — |
--json | label: print the labels as JSON (below) | — |
--call <n> | note: the call the note is about | The run |
label --json prints { runId, labels }, with the labels as they are after the change.
shadowlap blame
#shadowlap blame <run> [options] [-- <command> [args...]]Finds the model call where a failed run was lost: the last one whose answer, asked again, lets the run pass. Blame explains the search, the report and the limits.
shadowlap blame 01M2P9PVCP --expect '{"tools":{"send_email":0}}'How it searches:
- It replays the run as recorded, to confirm the failure.
- It probes model calls with forks that replay everything before the probed call for free.
- It binary-searches for the boundary.
Each blame is kept with the fix it opens or joins, by the run's labels, else its check.
blame takes these flags:
| Flag | What it does | Default |
|---|---|---|
--expect <json|@file> | What a passing run must do besides exiting 0: the expect object of an outcome check, such as '{"tools":{"send_ | — |
--gate <json|@file> | How a probe decides, after each attempt (gating; below). Give any of the four figures | {"good":0. |
--attempts <n> | A fixed count instead of the gate: attempts per probed call | 5 with --passes alone |
--passes <n> | With a fixed count: how many passing make a rescue | 2, or 1 with one attempt |
--model <model> | Rescue with this model from the probed call on, instead of asking the recorded one again | — |
--set <key=value> | Set a request field from the probed call on, such as temperature=1 (repeatable) | — |
--scope <name> | Probe only model calls made in this scope: the agent's, and not a simulated user's | — |
--scan <n> | Model calls past the boundary to probe too, in case rescue works again after it | 2 |
--writes stub|block | What write tools do after the fork; allow is not accepted (below) | stub |
--reads live|replay | What read tools do after the fork | live |
--concurrency <n> | Attempts of one probe running at once (below) | 1 |
--budget <usd> | Start no attempt once this much is spent on live model calls | — |
--timeout <seconds> | Stop an attempt after this long, and count it as failed | — |
--json | Print the report as JSON on stdout (fields) | — |
--guard, --allow-unrecorded, -y | As for replay | --guard: warn |
More on some of them:
--gate: a gate that tells a re-asked call that passesgoodof the time or more (a rescue) from one that passesbador less (lost), up tomaxattempts.riskis the whole search's, spread over the most probes it can make, so each probe runs at its share with itsmaxdeepened to match (the search).- The default gate: for a search of 5 to 10 probes, three straight passes rescue and six or seven straight failures lose.
- A probe's verdict shows its tally and its bound at its own confidence, under the rule, which is
stated once:
rescued · 5 of 10 · rate >= 0.10 (99.1%)undergate: rescue at 60%+, lost at 10%-, <= 5% risk, up to 32attempts, in a search of 6 probes. --writes: a write the recording made, with the same arguments, gets what it got then, an error included.--concurrency: the probe decides on its attempts in the order they started, and its first verdict stands. One starts only while fewer than n have started that the probe hasn't counted, so at most n − 1 run past the attempt that decides.
Its progress goes to stderr:
- In a terminal, a probe's attempts fold into one line that counts them as they come, and the
probe's verdict then takes its place. The pairs of a pinned call's check fold into one line that
counts each call's attempts (
3 pairs: #5 3 passed · #6 1 passed, 2failed). - With
-v, or anywhere else (a CI log, a file, the dashboard), every attempt keeps its line, with its run id.
In a terminal, a line of prose wider than the terminal wraps at whole words, before it starts, as it
goes and in its report. Each line it wraps to is indented as the first, past a mark that leads it
(!, ✗), or under the text of a row. A command, or a flag with its value, stays in one piece. The
lines that give commands or paths, and the rows of its tallies, stay whole. Written to a pipe or a
file, each line stays whole.
Verify and repair print theirs the same way.
blame exits with:
| Status | Meaning |
|---|---|
0 | The search reached a verdict: a blamed call, or no call that rescues the run |
1 | It refuses what it saw: the run passes its oracle, or does not replay as recorded |
2 | It can't start (below) |
3 | It stopped before a verdict (below) |
128 + n | A stop signal; 130 for Ctrl-C |
It can't start, exit status 2, for:
- a run with no model call to probe;
- a run that fails only a cost check a model with no price keeps from being judged;
- an agent on a Node too old for the guard;
- a run whose model called a tool it never recorded that writes.
It stops before a verdict, exit status 3, when:
- the budget is spent, or the provider unavailable;
- what the replay as recorded showed could not be decided;
- its first probe could not tell a rescue from a loss (Inconclusive at the first model call).
shadowlap verify
#shadowlap verify --fixes <run...> [--holdout <run...>] [--keeps <run...>] [options] [-- <command> [args...]]Checks a change before it ships. Verify explains the roles, the report and the warnings.
shadowlap verify --fixes label:bug --keeps label:goodIt replays, under the code in the working directory:
- failing runs, which the change must fix;
- good runs, which it must not break.
It accepts the change only when every failing run passes its gate and every good run is kept: not
worse, by the margin or more, than the same run under the code before the change, attempt for attempt.
Runs are ids, unique prefixes or label:<name>. A replay that forks nowhere settles its run in one
free attempt.
verify takes these flags:
| Flag | What it does | Default |
|---|---|---|
--fixes <run...> | Failing runs the change was written to fix | — |
--holdout <run...> | Failing runs of the same kind the change was not written against | — |
--keeps <run...> | Runs that passed and must not get worse, each kept against its own baseline, such as label:good | — |
--expect <json|@file> | What a passing run must do besides exiting 0, as for blame | — |
--gate <json|@file> | How a run decides, after each attempt where its replay forks (gating; below). Give any of the four figures | {"good":0. |
--attempts <n> | A fixed count instead of the gate: attempts per run where the replay forks | 5 with --passes alone |
--passes <n> | With a fixed count: how many must pass | 80% rounded up, 4 of 5 |
--model <model>, --set <key=value> | The change is a request field, set from each run's first model call | — |
--from <model> | With --model or --set: change only the model calls made with this model, from the first of them on (repeatable; below) | — |
--live-tool <name> | The change is in this tool: run it live, its first call and every call of it after a fork (repeatable) | — |
--writes stub|block, --reads live|replay | As for blame | stub, live |
--concurrency <n> | Runs verified at once | 1 |
--budget <usd>, --timeout <seconds> | As for blame | — |
--fail-fast | Start nothing more once a run has failed | — |
--margin <share> | A good run that forks is kept unless its pass rate drops this much or more below its baseline's, at the gate's risk | verify.margin in shadowlap.config.json, else 0.1: 10 points |
--kept-max <n> | The most pairs a good run that forks takes against its baseline, or attempts against its history | verify.max, else 200, or a --gate max off its default |
--baseline paired|history | What a good run is compared with (below) | verify.baseline, else paired |
--base <rev> | With a paired baseline: the commit before the change, checked out on its own for the baseline's attempts | HEAD, when the working tree has a change |
--build <command> | With a paired baseline at a commit: build its checkout before its attempts, such as "npm run build" | build in the config |
--json | Print the report as JSON on stdout (fields) | — |
--guard, --allow-unrecorded, -y | As for replay | --guard: warn |
More on some of them:
--gate: a gate that tells a run that passesgoodof the time or more from one that passesbador less, each verdict wrong at mostriskof the time at those two rates, up tomaxattempts. With the default, a run that always passes is accepted after 12 attempts, and one that always fails rejected after 3.--from: such as the agent's model calls, and not a simulated user's or a sub-agent's on another model.decisions --then-verifyandmigratepass the models they switched from.--margin,--baseline: good runs are kept against their own baseline.pairedis the same run under the code before the change, each attempt paired with one of the change's.historyis its golden's test history.
verify exits with:
| Status | Meaning |
|---|---|
0 | The change is accepted |
1 | The change is rejected |
2 | A run's model called a tool the run never recorded that writes, before anything runs |
3 | The verification did not finish: a run's gate or baseline inconclusive, the budget spent, the provider unavailable |
128 + n | A stop signal, whenever it comes, the verdict reached or not |
The verdict is kept with the fix of its failing runs, whatever it is. An accepted one turns the fix verified when it counts there: it checked the fix's check on a run that failed it.
shadowlap repair
#shadowlap repair --fixes <run...> [--holdout <run...>] [--keeps <run...>] [options] [-- <command> [args...]]Fixes failing runs. Repair explains the loop and the safeguards.
shadowlap repair --fixes label:bug --keeps label:good --repairer anthropic- It blames each failing run.
- It asks a model you choose (
--repairer, orrepair.repairerin the config) to rewrite the system prompt or a tool description. Rewrites that name a detail of the failing runs, or add a placeholder the agent would fill in, are set aside before anything runs. - It keeps the smallest rewrite
verifyaccepts twice, the second time with fresh attempts: on the failing runs, the runs held out from the repairer (--holdout), and the good runs. - The fix comes out as a patch, a pull request body and a recording of each fixed run.
Each candidate is verified in a temporary git worktree, never in the working tree. A good run is verified against the same run under the code without the candidate, in another worktree.
repair takes these flags:
| Flag | What it does | Default |
|---|---|---|
--fixes <run...> | The failing runs | — |
--holdout <run...> | The failing runs kept from the repairer | Every other --fixes run, when there are two or more |
--keeps <run...> | The runs that must not get worse (below) | — |
--expect <json|@file> | What a passing run must do besides exiting 0 | — |
--repairer <provider:model> | The model that proposes fixes (below) | repair.repairer in the config |
--repairer-timeout <seconds> | Give up on the repairer when it hasn't answered after this long | repair.timeout in the config, else 600 |
--candidates <n> | Rewrites to ask for, and the most it tries | 3 |
--stronger <model> | When the recorded model can't be rescued, rescue with this one and try switching to it, before asking the repairer | — |
--scope <name>, --set <key=value> | As for blame: which calls to probe, and what probes change, such as temperature=1 | — |
--gate <json|@file> | The gate each candidate's runs are verified by, as for verify | {"good":0. |
--attempts <n>, --passes <n> | A fixed count instead of the gate, as for verify | — |
--max-growth <percent> | Reject a fix that grows input tokens per run by more than this, before verify runs it when its rewrite alone does | 25 |
--build <command> | Build the agent in each candidate's worktree and in its baseline's, and in the working tree after --apply | build in the config |
--apply | Leave the accepted fix in the working tree | — |
--branch <name> | Commit it to a branch it makes, without checking anything out | — |
--pr | With --branch: push it and open a pull request with gh | — |
--goldens <dir> | With --apply or --branch: put the recordings of the fixed runs in this directory too; with --branch, one inside the repository | — |
--budget <usd> | Stop once this much is spent on blame, the repairer, verification and judge checks (below) | — |
--writes, --reads, --concurrency, --timeout, --guard, --allow-unrecorded, -y | As for verify | — |
--json | Print the report as JSON on stdout (fields) | — |
More on some of them:
--keeps: a good run that forks is kept unless it drops 10+ points below its baseline, the same run on the code without the candidate, attempt for attempt, as verify keeps one. Its verdict bounds the drop as verify's does (drop <= 0.10 (95%)).--repairer:anthropic(Claude Opus 5.5),anthropic:<model>,openai:<model>, orclaude-code[:<model>]to run your own Claude Code (2.1.248 or later) in print mode, restricted: no tools, no MCP server, none of your settings (details).--budget: the repairer is asked only when what is left covers the most asking it could cost.
repair exits with:
| Status | Meaning |
|---|---|
0 | A candidate is accepted, and handed over as asked |
1 | None is accepted, a repairer that could not be asked among them, or the hand-over fails (below) |
2 | It can't start (below) |
3 | It stopped before a verdict: the budget spent, too little of it left to ask the repairer, or the provider unavailable |
128 + n | A stop signal, whenever it comes, the verdict reached or not |
A branch it can't make (a name git refuses, or one that exists) is found at the hand-over: that fails the hand-over, and the repair is kept.
It can't start, exit status 2:
- outside a git repository;
- with another repair running in the folder;
- with
--goldensoutside the repository, with--branch; - for an agent on a Node too old for the guard;
- for a run whose model called a tool it never recorded that writes.
The repair is kept with the fix its blames opened or joined, even when the hand-over failed. When a candidate is accepted under the fix's check, the fix turns verified. When none is, the fix stays open, with every candidate tried and why it was dropped.
shadowlap fix
#shadowlap fix ls [--status open|verified|applied] [--json]
shadowlap fix show <fix> [--json]
shadowlap fix title <fix> <title...>
shadowlap fix check <fix> --expect <json|@file> [--json]
shadowlap fix apply <fix> [--name <run>=<name>]... [--goldens <dir>] [--no-test] [--build <command>] [--json]
shadowlap fix pr <fix> [<url>] [--json]A fix is one failure, from the blame that reproduced it to a change that held.
- Where fixes come from: a blame opens one or joins one, by the run's labels, else its check. Verify and repair report to the fix of the runs they were given. Their verdict counts for it only when it checked the fix's check on a run that failed it.
- Fix ids look like
fx-01M3RZ4XKN…. Any unique prefix works, with or withoutfx-, and the list shows the first ten characters. - Activity: blame, repair, verify, apply and
fix checkeach log an activity event in the store (blame.done,repair.accepted,repair.empty,verify.accepted,verify.rejected,fix.applied,fix.check).
Its subcommands take these flags:
| Flag | What it does | Default |
|---|---|---|
--status <status> | fix ls: only open, verified (applied ones too) or applied fixes | — |
--expect <json|@file> | fix check: the fix's new check, an expect object as under expect in shadowlap.config.json tests | — |
--name <run>=<name> | fix apply: the name of the golden copied from a fixed run, at most 64 characters and not a name Windows keeps (repeatable) | The fix's label |
--goldens <dir> | fix apply: where the recordings go | goldens |
--no-test | fix apply: copy the recordings without running shadowlap test on them | — |
--build <command> | fix apply: rebuild the agent after the patch, before the test | build in the config |
--json | Print it as JSON: the dashboard's read model of Fixes (below) | — |
With --json, each subcommand prints:
| Subcommand | --json prints |
|---|---|
fix ls | { fixes, counts } (fields) |
fix show | The fix, with its three steps in full (fields) |
fix apply | What it did (fields) |
fix check | What changed (fields) |
fix pr | { fixId, pr: { url, number, state, at } } (below) |
| An id that matches no fix, or several | { "status": "error", "error": { "reason": "unknown" | "ambiguous", "message" } }, with the fixId given; exit status 2 |
An error in fix pr, fix apply or fix check | { fixId, status, error }, with status error |
fix ls
#Lists fixes, the most recently opened first. Each fix takes up to three lines:
- Its status (
open,verified,applied), id, title, label, failing runs, and how far each step got (lost at #5 · prompt change · 3 fixed · 1 kept). Repair says by hand for a change you made yourself that verify checked. - What its failing runs did and the check they must meet (failed: send_email called twice, not once · must: send_email called once), where the fix came from (from CI · PR #138) and when it was opened.
- A line that says so, when an applied fix doesn't hold.
What its failing runs did is the title blame gave the fix when it opened it, so the second line names it only once someone renames the fix.
fix ls --json prints { fixes, counts }: each fix as its row in the list, and the counts of all,
open and verified fixes, applied ones counted as verified. Each row has these fields:
| Field | Meaning |
|---|---|
fixId, status, title, label, labels, check | The fix, as the list shows it |
href | Where the dashboard shows it |
failed | What its failing runs did (below) |
judge | The check's first judge: its label, the question it asks, what it reads and its model; null for none |
failing, good | How many runs are failing, and how many good |
steps, source | Its steps, and where it came from |
openedAt, verifiedAt, appliedAt | When it was opened, verified and applied: epoch milliseconds, or null |
broken | Why it is broken |
attempts, spent | The attempts, and the USD spent, behind it |
failed is in the words blame titled the fix with when it opened it, which a title of someone's own
no longer says. For a fix opened on another machine, it is the first failure its blame recorded. It is
null when no blame recorded one.
fix show
#Prints a fix: what its failing runs did and the check they must meet, as fix ls says them, and its
three steps.
- Blame: where the run was lost, and the probes that found it.
- Repair: the change, the candidates tried and why each was dropped, who proposed them or why no repairer did (not fixed · the repairer could not be asked: claude was not found), and the repair's warnings.
- Verify: every verdict, named by what changed (a repair's candidate, the request fields it set,
or the code version) and headed by the runs it checked, with each run's role, outcome and tally as
shadowlap verifyprints it (12 of 12 passed · rate >= 0.73 (95%)). One that doesn't count for the fix says why in its heading.
It ends with what to do next. It says when the repair it suggests would end No fix to try without
running the agent, and why (the blames it would reuse rescued no failing run), or couldn't ask its
repairer from here (claude isn't on the PATH).
fix show --json has the three steps in full:
| Field | Meaning |
|---|---|
fixId, href, failing, good, openedAt, verifiedAt, appliedAt | As in fix ls |
openedBy | Who opened it |
checks | Every check it has had, the first first: its text, when it was given (at) and by whom |
status, title, label, labels, check, judge, failed, oracle, expect, source | The fix (below) |
broken | For an applied fix that doesn't hold, why: its test failed, or a verify rejected it after it was applied |
failingRuns, goodRuns | The runs, each with runId, exists (false once deleted from the store), status and labels |
left | The failing and good run ids a check given later left out (see fix check) |
guard, fixedRun | The golden that guards it, as its key under tests names it, and a run to pack as one; each null for none (below) |
steps | Blame, repair and verify (below) |
blame, repair, verify | Each step in full (below) |
blames, repairs | Every blame and repair behind the fix, newest first (below) |
defaultStep, defaultVerify | The step the dashboard opens at, and the verdict it shows first: the newest that counts |
attempts, spent, upstreamRequests | What everything behind the fix took: the attempts that ran live, and what they and the repairer cost |
actions | What can be done next (below) |
lead | The step it leads with, as its dashboard page and row do: key, label and command, or null for none |
commands | The blame, repair and verify commands to run it again |
repairer | The repairer the config names (below) |
applied | Once applied, how: patch, repairId, the goldens copied, tests, by, via |
rung | Where it stands: open, blamed once a blame finds the call, verified, applied, or merged once its pull request is |
pr | Its pull request, with number, url and state (open, closed, merged, or null when unknown); null for none |
The fix's own fields:
labelis the one most of its failing runs carry, andlabelsall it groups runs by.checkis in words, andfailedis what its failing runs did (asfix lshas it).oracleis the conditions a verdict must check to count.expectis the--expectJSON (null for the exit code alone).sourcesays where it came from, such asfrom CI · PR #138.guardis the golden a failing run of it replayed, else one that applying it copied and the goldens folder still holds. A verified fix with none is usually one to apply first.fixedRunis a recording of it working, to pack as that golden: the first passing attempt of its accepted verdict on its first failing run. It isnullwithout an accepted verdict that counts.
steps holds blame, repair and verify, each with state (done, todo, no for a verify that
only rejected, or warn for a step that ran and needs a look), a title, a meta line and a short
form. Only verdicts that count move the verify step. Repair is done with the change repair made,
or by hand for one you made that a verdict checked.
blame has:
- its verdict
line: theword, itstone, theheadin parts (a probe's tally each) and thefinding; - the call where the run was lost (
lostAt, model callstepofsteps),confidence, theboundarysentence and bothanswers; probeswith their verdicts, each with itstally,boundandinterval, the gate'smarks, and the attempts per probe (bundles);- the agent's calls of the run, with whether each is in the blame's scope and probed, and a probed
call's
tally(callRow); cost, with what the judge's checks cost apart (judge), andcommand.
repair has:
- its verdict
line, asblame's, andagainwhen the accepted change was verified twice; - the
change, with the text itadded; the result per role (roles); and whether its verificationcounts; - the gate's
rule, from the first candidate it verified when none was accepted; - who proposed it (
repairer), or why no repairer did (noRepairer: itstext, and thewarningthat says what to do); - where each run was
lost, and thediagnosis; - every candidate, with its
verdict(accepted,rejected,set aside,not tried,build failed,unsettled) andwhy, and the runs those whys name by their short ids (named); - the
fileshanded over, and the pull request body (prBody); - what each stage
cost, and how manyattemptsit ran: the probes of the blames it ran, and every candidate's verification; commandandwarnings.
verify has one entry per change verified, newest first, with:
- its
verdict, and whether itcounts; - the change it checked, named by what changed (
code: a repair's candidate, the request fields it set and from which call, or the code version;labelas the dashboard's switch shows it); - its
headline: by the runs it checked, or, for one that doesn't count, why; - a
notewhen it was rejected after the fix was applied; oracle;- each run's
role,attempts,outcomeand where it went (note), and the token and cost changes per run; - each run's
tally,cutShortandboundin the wordsshadowlap verifyprints, itsintervalagainst the gate'smarks, and its attempts (tries); - the failing runs an accepted verdict left out, that no other has checked (
left); - what it took to find out, with the judge's checks apart (
judge); - what answered the change's tool calls after the fork, and the baselines' apart, in
afterFork, as the report's line says it, ornullwhen no attempt made a tool call after its fork; - its
reasons, with those that say a run is inconclusive or not settled, rather than that the change failed it, inunsettled.
actions says what can be done next:
repair, for an open fix:noFixsays why it would end No fix to try without running the agent, or isnull.spentis true when its last repair tried candidates and accepted none, with no blame or check change since, so running it again would try the same.apply, for a verified or applied one:kindispatch, ormarkwhen there is no patch (below).applied, once it is applied.verifyLeft, for a verified or applied fix whose change is in the working tree: the failingrunsno accepted verdict checked, and thecommandthat verifies the working tree's code on them. It isnullwhen every failing run was verified.verifyChange, for a fix with a failing run left: thoseruns, and thecommandthat verifies a change of your own on them, the runs held out and the good runs it keeps, under its check. It isnullwhen none is left.
apply has:
kind, thecommandthat applies it, and thegitApplycommand;pending, saying what applying left undone;- every file the patch changes in
files, the patch itself indiff, and where it is kept inpatchFile; inTreewhen it is in the working tree already, which applying only marks;- how many recordings applying copies into the goldens folder, in
goldens; - each recording it is still to copy, by its
run(the short id), with thenameit takes unless--namegives another, innames; and the names the goldens folder holds, which none may take, inheld(nullwhere the folder isn't read); - the request fields a change with no patch set, in
change; refused, saying whyfix applywon't apply it, when the patch is more than the change its repair verified, or a teammate's is no rewrite a repair makes in this working tree.gitApplyisnullthen, and once the patch is in.
blames lists every blame (blameId, runId, at, outcome, lostAt, confidence), and
repairs every repair (repairId, at, outcome, accepted, title), newest first.
repairer has its name, whether it is configured, and, on an open fix, the problem that keeps it
from being asked here (such as claude not on the PATH), or null.
fix title
#Replaces the title derived from the check with one of your own. Blames that join the fix later keep it. With a team store, the latest rename on any machine is the one every machine keeps.
fix check
#Gives an open fix another check, as --expect takes it. Its failing and good runs are judged by it
again from what they recorded, judges included, with nothing replayed. A failing run it passes and
a good run it fails leave the fix, though shadowlap keeps them. A run none of its checks applies to
fails nothing, so it leaves as well (not applicable). Each run prints as check prints
it, after its role, and a run that leaves says so.
- What stays. Every blame, repair and verdict keeps the check it ran under. A verdict counts for the fix only when it checked the fix's check, so one made under the check before stops counting.
- Refused, with exit status 2. Nothing changes when it refuses:
- A fix that is verified or applied keeps the check its accepted verdict was given under: add the check to the golden that guards it instead.
- It refuses a check none of the failing runs fails as recorded, since the fix would no longer be this failure.
- It refuses a fix none of whose failing runs is left in the store to judge the check on.
- It also refuses a check holding a hidden character, and one that doesn't read.
- The check it has changes nothing, with exit status 0.
fix check --json prints { fixId, status, check, previous, runs, left, uncounted, next }:
| Field | What it holds |
|---|---|
status | changed, or unchanged for the check the fix has |
check, previous | The check given and the one before, each { expect, key, oracle, text } |
runs | Each run as the check judged it, failing ones first (below) |
left, uncounted, next | The runs it left out ({ failing, good }), the verdicts that stopped counting, and the command of the fix's next step |
Each of its runs (runId, role, outcome, na, why, checks) is as the check judged it.
role is failing or good, and outcome is passed, failed, inconclusive, or gone from
the store. na says none of its checks applied: it failed nothing, so its outcome is passed, and
it reads not applicable.
fix pr
#Records the pull request opened for a fix, given its https:// link. repair --pr and CI's
fix on failure record theirs, so the list says where the fix went
(from CI · PR #138).
With no link, it reads where the recorded pull request stands from GitHub, with gh pr view
(installed and signed in), and records it: open, merged or closed. When gh can't say, or isn't
asked, the state stays as recorded and the line says why. The dashboard reads it the
same way, in the background, while Fixes is open.
Only a pull request on github.com, on the host GH_HOST names, or on the host of the project's
origin remote is read. A fix pulled from the team store can name any address, and gh would send
that host a token. A token for a GitHub server of your own only ever goes to the host GH_HOST names.
fix pr --json prints { fixId, pr: { url, number, state, at } }, with a note saying why when the
state is as recorded because GitHub wasn't asked or couldn't say.
fix apply
#Hands a verified fix over. Run it in the folder the repair ran in, where its directory under
.shadowlap/repairs/ is.
- It runs
git applyon the accepted repair'sfix.patch. - It copies the recordings the fixed code made into the goldens folder, each a golden named as
below. Each copy says why it is there in its
accepted.jsonl: the fix, as its reason. - It runs
shadowlap teston them, and marks the fix applied. - It replays the project's other goldens, and names each that diverges on what the patch changed (below).
fix apply exits with:
| Status | Meaning |
|---|---|
1 | It refuses the patch: not the change its repair verified, a file it may not write through, or fix.patch changed since |
1 | The patch does not apply, or the test fails |
2 | The fix is still open (nothing verified to apply), or it runs outside the repository and folder the repair ran in |
2 | A golden's name won't do (below), and nothing is changed |
A patch that applied stays applied.
The goldens' names. Each recording of a fixed run becomes a golden of its own, a folder in the goldens folder named for what a test reads it by, never the run's id:
--name <run>=<name>names the one from that run, by its id or the start of it, such as--name01K5CQ2E8M=double-email. Give one for each golden you name.- A golden you don't name takes the fix's label, else its title, as a name:
refund-bug, orrefund-bug-1andrefund-bug-2for two. A label that makes a name Windows keeps gives way to the title, and a title togolden. - A name takes lower-case letters, digits and hyphens, from a letter or a digit, at most 64 of them. Each golden needs one of its own, and not one the goldens folder holds already, read without case as macOS and Windows read a folder's name.
- A name Windows keeps for itself won't do, whatever its case:
con,nul,prn,aux,com1tocom9andlpt1tolpt9.
A name that won't do changes nothing: fix apply says why and how to give another, and exits 2. So
does a --name where it copies no recording: a fix with no repair behind it, one applied with none
left to copy, or a teammate's repair, whose recordings stay on their machine. It prints where each
golden went and the run it holds (copied goldens/double-email from 01K5CQ2E8M).
A golden copied before keeps its name and takes no other: one fix apply copied, one
repair --goldens copied under the run's id, and one an apply stopped part way left, the very
recording under whatever name it gave it. A --name for one of them can give it only that name.
The project's other goldens. Then fix apply replays the project's other goldens offline, in a
store of their own that nothing keeps. A prompt or a tool description goes into every request, so they
can diverge on what the patch changed:
- It names each that diverges on what the patch changed: a request field that holds text the patch
added, or lost text it removed. It prints the command that re-records them,
shadowlap test <goldens>--update. - It leaves out those from a pack the repository doesn't track, recorded elsewhere, whose command runs only with a yes, and names them.
- A golden that may fork is named only when it forks before its task's takeover, as
--updatere-records it then. --no-testskips this too.
A teammate's repair, from the team store, left its recordings on their machine.
fix apply puts the patch in, says so in one line, and leaves nothing to finish. Nothing here runs the
code that patch put in: no --build, test or replay of the other goldens. It says to read the change
(git diff -- <file>) and then run shadowlap test.
What the patch may change. Before anything goes in, the patch is checked against the repair it came with. It has to be the rewrite of the accepted candidate, to that candidate's file alone, changed in place: no file made, deleted, renamed or copied, and no mode or binary change. It may never touch:
- a file of a recording: a pack's
run.json, which holds the command a replay runs, itscalls.jsonland blobs, or anything in__recordings__; shadowlap.config.jsonor the.shadowlapfolder;- a dot-file or a file in a dot-folder, such as a workflow under
.githubor Claude Code's.claude/settings.json, another file that runs, such as aDockerfileor a script, or one a repair rewrites no text in, such as Go source.
A repair rewrites the text the agent sends the model where its source holds it, and never reads those.
A patch that is anything more is refused, and nothing is changed. So is one to a file git ignores,
whose change neither git status nor git diff would show, or to a file reached through a link to a
folder, which git apply refuses too.
A teammate's patch, from a pull, is held to a repair's own rewrites in your working tree. The text it rewrites is read where the patch goes. It has to be inside one string of JavaScript, TypeScript or Python source, a string value of YAML, JSON or TOML that doesn't run, or a Markdown, plain text, prompt or Jinja file. It holds each placeholder whole, and its rewrite has to be one a repair would try. See where an anchor goes and the team store.
Line endings. The patch goes in on the file's bytes as they are, so the file is the one the repair
verified. Git applies it to a copy of them that nothing converts. git apply in the working tree would
convert line endings as core.autocrlf (Git for Windows sets it) or a text or eol attribute in
.gitattributes asks, and write a file with LF back with CRLF on every line.
A patch whose lines end otherwise than the file's (a teammate's, made where the file has LF, for a
checkout where it has CRLF) applies only through that conversion. So it goes in as git apply puts it
in, and the file keeps its CRLF.
What it only marks.
- Applied is marked on this machine only.
- A patch already in the tree, applied by hand, committed, or left there by
repair --apply, is recognised withgit apply --reverse --checkand only marked. - A fix verified on the working tree itself, with no repair behind it, has no patch, so it is only
marked applied. One verified with a change of request fields (
--set,--model) says which, since no file carries them: set them in your agent.
A conflict. When the working tree has changes the patch conflicts with, nothing is changed. The message says whether to commit or stash them, or to apply the patch by hand.
Run again on an applied fix, fix apply finishes what is left, and keeps when the fix was applied.
It copies the goldens not in the project yet (repair --apply copies none without --goldens), and
runs the test when it hasn't passed, on goldens copied before where they are. With nothing left, it
does nothing; with --no-test, it says when a test is still to pass.
A test that fails leaves the fix applied, marked as not holding. It stays so until a run of fix apply
passes it, or a shadowlap test session passes every one of its goldens (after test --update
re-recorded what another fix changed, say), which says the fix holds again.
It trusts nothing it reads from the store. Otherwise it changes nothing, and says why:
- the repair's directory must be
.shadowlap/ repairs/ <repair id>, each golden inside itsgoldens/; - the patch must be the accepted candidate's one rewrite to that candidate's file, no larger than a repair writes;
- a teammate's must rewrite text where a repair rewrites it, in the file as it is here;
- that file must be a plain file git doesn't ignore, not a symbolic link or a folder;
fix.patchmust be the patch the repair verified.
fix apply --json prints { fixId, status, again, patch, repairId, goldens, tests, outdated, update }:
| Field | Meaning |
|---|---|
fixId, repairId | The fix, and its repair |
status | applied, or already-applied when nothing was left to do |
again | Set when it finished a fix applied before |
patch | What became of the patch: applied, already in the tree, or none to apply |
goldens | Each golden's run (the fixed run it holds, by its short id), from, to, and whether it was copied |
tests | passed, failed or skipped |
outdated | The project's other goldens that diverge on what the patch changed, each with its golden, callNo, the call's kind and name, and request field path |
update | The command that re-records them, or null |
shadowlap decisions
#shadowlap decisions <runs|goldens...> (--model <model> | --from-code | --set <key=value>) [options] [-- <command> [args...]]
shadowlap decisions --verify <session> [runs...] [options] [-- <command> [args...]]Previews what a change decides differently, without running the agent. Decisions explains the ranks, the report and the limits.
shadowlap decisions goldens/* --model claude-opus-5
shadowlap decisions label:refund --from-code --then-verifyHow it works:
- Each recorded model call is asked again: its request exactly as recorded, with only what the change touches swapped in, sent through the same upstreams a live call goes to.
- The answer's decision (the tools it calls, in order, with their arguments, and whether it stops or hands over) is compared with the recorded one, and ranked by consequence, writes first.
- A text-only change that one of the run's judge checks reads is judged on the recorded words and on the words under the change. It ranks 2 when the check comes out otherwise.
No tool runs, and a step doesn't depend on the one before, so the steps are asked in parallel.
What it asks. Runs are ids, unique prefixes or label:<name>. A path such as goldens/* names
packs, which are imported as test imports them and named as its history names them. The change is
one or more of --model, --set and --from-code.
- A model switch changes the agent's calls: those made with the model the runs use most, or with the
models
--fromnames. - A call made with another model (a simulated user, a sub-agent) is left as recorded, and counted
untouched. - shadowlap's own calls (its judge's, in a
shadowlap:scope) are no step at all.
Credentials. Requests need the provider's credentials on this machine, unless
SHADOWLAP_ points the provider at a server of your own. Without them nothing is
sent, and the command says which is missing:
ANTHROPIC_API_KEY,OPENAI_API_KEY,GEMINI_API_KEYorAZURE_OPENAI_API_KEY;- or, where the AI SDK keeps the last two,
GOOGLE_andGENERATIVE_ AI_ API_ KEY AZURE_API_KEY; - AWS credentials for Bedrock, and Google credentials for Vertex AI.
An Azure OpenAI deployment call is sent with the api-version it was recorded with. One recorded before
recordings kept a call's query takes OPENAI_API_VERSION (or AZURE_OPENAI_API_VERSION), as the
AzureOpenAI SDKs do; without it, nothing is sent.
decisions takes these flags:
| Flag | What it does | Default |
|---|---|---|
--model <model> | The change is this model, at the agent's model calls: in the body, or in the path on Gemini, Vertex AI, Bedrock and Azure (below) | — |
--from <model> | The recorded model the change switches from: --model and --set change only the calls made with it (repeatable; an alias names its dated snapshots) | The model the runs use most; with --set alone, every call |
--from-code | The change is the system prompt and tool definitions your working tree sends (below) | — |
--dry-run | With --from-code alone: read what your working tree sends each run, offline, print it and stop. Nothing is asked and no session is kept | — |
--set <key=value> | The change sets this request field, where the provider keeps it (temperature is generationConfig. on Gemini); repeatable | — |
--samples <n> | Ask each model call n times under the change and, from 3 on, n times as recorded, the control (below) | 3: six requests a step |
--concurrency <n> | Requests in flight at once. A run whose requests carry cache breakpoints is asked one step at a time, in order | 4 |
--scope <name> | Ask only the model calls made in this scope; none for those made in no scope | — |
--then-verify | Then run verify on the runs a preview can't clear (which) | — |
--verify <session> | Ask nothing: verify the runs a kept session could not clear (below) | — |
--kept-max <n> | With --verify: the most pairs a good run takes, as for verify | Verify's |
--test-session <id> | With --from-code: keep the preview with this test session, as its pull request comment's decisions (below) | — |
--expect <json|@file>, --gate <json|@file>, --attempts <n>, --passes <n>, --writes stub|block, --guard | With --then-verify or --verify: as for verify | Verify's |
--budget <usd> | Send no request, and start no verify attempt, once this much is spent | — |
--timeout <seconds> | Stop a model request, or a verify attempt, after this long | A model request waits ten minutes, as the official SDKs do |
--strict | Exit 1 when a run has a rank 1 or 2 change, for CI | — |
--inconclusive fail|warn | What an inconclusive result does to the exit status: fail exits 3, warn exits 0 with a warning | fail |
-y, --yes | Run recorded commands without asking (below) | — |
--json | Print the report as JSON on stdout (below); progress goes to stderr | — |
More on some of them:
--model: a model of another family than the recorded one (a Claude call asked ofgpt-…) is not asked. Its requests take another format, which is verify's to test.--from-code: the prompt and tools are read by replaying each run offline under your code, its model calls matched without their prompt and tools. Nothing is sent to read them, no tool runs, and the replay is not kept. A run whose first model request differs in more than its prompt and tools is refused, with the difference.--samples: a step is ranked only when its decisions run more consequential than the control's, by an exact permutation test at 5% (sampling). Below 3 there is no control, and a step flags its run when more than half its samples show a write, stop or judged change.--then-verify: a run that passed as recorded must not get worse than its own baseline, and one that failed must pass. Each is judged under its checks (its golden's, or for a run named by id those of the recording it comes from) or--expect, one verify per set of checks.--verify: it verifies the runs a kept session could not clear (itsdc-…ormg-…id, or a prefix of it), or those of them named after it, as--then-verifyverifies them, under the session's change. It keeps what verify finds in the session. The dashboard runs it as itsdecisions-verifyjob.--test-session: the GitHub Action'sdecisions: truegives it the sessiontestjust ran, so the pull request comment lists the decisions the branch changes, and Settings → CI's preview shows them. A session the store has no record of is refused before anything is asked.--gateand the rest: each run is decided by verify's gate, unless--attemptsor--passesgives a fixed count.--yes: it runs the recorded command of a run imported from someone's pack (--from-code,--then-verify,--verify), and of a golden from a pack the repository doesn't track, astestasks of one.
The command after -- replaces each run's recorded one, for --from-code, --then-verify and
--verify.
What it prints. The report opens with how many runs decide a write or when to stop differently, and the steps it asked. Each run then has a line, and under it a line for each step that changed or wasn't asked (reading the report). A step's line gives its call and its step, then one of these:
| It says | When |
|---|---|
rank 1 to rank 5 | Its change ranks, by consequence, with what it decided as recorded and under the change |
within the model's own variation | Its samples decided otherwise no more often than as recorded, with both counts |
inconclusive | Its samples changed, but too few answers came, under the change or as recorded, to tell that from the model's own variation |
not asked | It got no answer, with why |
An inconclusive step counts as not asked, and its run settles nothing unless something else flags it.
decisions exits with:
| Status | Meaning |
|---|---|
0 | Every run was asked |
1 | --then-verify or --verify rejected the change, or, with --strict, a run has a rank 1 or 2 change |
2 | A refusal before anything is asked: no credentials, no such run or session |
3 | The result settles nothing (below), unless --inconclusive warn |
128 + n | A stop signal; 130 for Ctrl-C |
A result that settles nothing is inconclusive. That is when:
- it stopped early: the budget, or a stop signal, which exits 128 + n instead;
- a run could not be asked at all: a
--from-codechange that is more than a prompt, or a model that refused every step; - a run nothing else flags has a text-only change a judge check reads that could not be judged, in any sample of a step its samples don't flag (no key for the judge's provider, the budget spent);
- a run nothing else flags has a changed step whose control, the request as recorded, got no answer;
- verify didn't run, or didn't settle the runs it took.
The change is sent as given. When the model it switches to refuses the requests as recorded, a warning
says what it takes instead (thinking={"type":"adaptive"}, temperature left out), which
migrate sends.
A request that fails is retried twice when the provider says so (408, 409, 429, 5xx), honouring
retry-after. One the provider refuses keeps its answer's message in error.
Each session is kept in the store, with each run's part as a row of its own for its page in the
dashboard, and logs a decisions.done activity event. Values the project's
redaction rules match are masked in every argument, text and error
the report holds, as in every export, pack and share link.
Decisions report as JSON
#--json prints the report the dashboard reads:
| Field | Meaning |
|---|---|
decisionsId | dc-…, the session as the store keeps it |
kind | decisions |
at | When the session ran |
change | The change (below) |
command | The command that asks the same way again, every option given |
samples | The samples asked of each model call |
runs | Each run (below) |
counts | Steps by the rank they were given (below) |
flagged | The runs a rank 1 or 2 change flags: beyond the control's variation, or with no control by what most samples show |
toVerify | The runs a preview can't clear, those flagged among them: what --then-verify and --verify verify (which) |
cost | Per run, the mean before and after in USD; what asking spent; upstreamRequests; and unknownModels with no price |
tokens | before and after, each input (uncached), output, cacheRead and cacheWrite (cache creation) |
retirements | Each model the runs use, or switch to, that its provider retires (below) |
notes | What to keep in mind reading it (below) |
warnings | The report's warnings |
verify | With --then-verify: what verify found (below); else null |
judge | What the judge checks that read text-only steps found (below); null when no check read a step |
remedies | Empty here; see migrate |
stopReason | Why it stopped before every step was asked (the budget, Ctrl-C), or null |
change has its model, its fields, from (the recorded models they change at, or null for every
call), fromCode, and text, the change in words.
counts has the steps by the rank they were given, 1 to 5, then:
noise: those whose samples decided otherwise no more often than as recorded;same;untouched: a request the change leaves as it was, such as one the working tree sends with the same prompt, or one made with a model the switch leaves;notAsked.
retirements lists each model the runs use, or switch to, that its provider retires, as
the retirement table has it: provider, model, platform,
matched, deprecated, retires, earliest, regions, replacement as the page words it and the
model ids it names in replacements, state, days, source and text.
notes says what to keep in mind reading it: later steps are judged in the recorded run's state,
reasoning left out, cache breakpoints kept, calls left as recorded, the text-only steps a judge check
reads and how they came out, values masked.
Each of runs has these fields:
| Field | Meaning |
|---|---|
runId, golden | The run, and its golden or null |
steps, asked | Its model calls, and how many were asked again |
rank | The most consequential rank its steps were given, or null |
majority | The same, by what decides a step: beyond the control's variation, or with no control what most samples show (below) |
counts | Its steps by rank, as counts above |
cost | before and after for the steps asked, and what asking them spent |
tokens | before and after |
thinkingStripped | Reasoning blocks left out of its requests |
refused | Why none of its calls could be asked, or null |
decisions | Its model calls, the changed ones first, most consequential first (below) |
warnings | The run's warnings |
A 1 or 2 in a run's majority flags the run.
Each of its decisions, a step, has these fields:
| Field | Meaning |
|---|---|
callNo, step, scope | The call in the recording, its place among the run's model calls, and its scope |
rank | With a control, the rank it earns beyond the control's variation; else the most consequential change any sample showed (below) |
majority | With a control, the same; else the rank more than half its samples show, or null |
changes | Each change (below) |
recorded, now | The decision as recorded, and under the change (below) |
samples | asked, answered, changed, unjudged (below), and each sample's rank in ranks, 0 for the same |
control | From 3 samples on, the request asked as recorded, as often (below) |
noise | With control: p, the exact one-sided p-value that the change's decisions run more consequential than the control's |
sampled | With control: the most consequential change any sample of the change showed, ranked or not |
model | recorded and asked |
tokens | before as recorded, and after as the answer under the change would use them (below) |
cost | before and after for those tokens, and what asking it spent, every sample |
thinkingStripped | Reasoning blocks left out of the request |
error | Why it wasn't asked, or got no answer |
judge | For a text-only change a judge check reads (below); null when no check read the step |
More on a step's fields:
rankisnullwhen there is none, and when either side got fewer than 3 answers, which leaves the step inconclusive.changes: each has itsrank, itskind, thetooland itseffect,before,afterand adetailsentence.kindis one ofwrite-added,write-removed,write-arguments,write-order,stops,handover,judge,read-added,read-removed,read-argumentsandtext. Ajudgechange is a judge check that comes out otherwise for the words under the change, with its statusbeforeandafter.recordedandnow: each decision'sstop(one oftools,answer,limit,refused,pausedandother), the provider'sstopReason, itstoolswith their arguments and effect, and itstext.nowisnullwhen no answer came.- A text, here and in a
textchange, keeps its first 2,000 characters, and says how many more the recording holds. The comparison reads all of it. samples.unjudged: the text-only samples whose words a judge check reads but could not judge.control:asked,answered,changed,ranks, anderror, why fewer than 3 were answered. With it,rankandmajorityare the rank the change earns beyond the control, ornull. It is absent below 3 samples.tokens.after: in a run like the recorded one, its input split between cache read, cache creation and uncached input, as the recorded call's was.
A step's judge is now's, the sample shown: the most consequential, or, among text-only ones, one
whose words could not be judged. It lists each check:
| Field | Meaning |
|---|---|
name | The check |
of | What it reads: answer or run |
model | Its model |
pass | The answer that passes |
recorded, now | Its verdict on the recorded words, and on the words under the change (below) |
changed | Whether the check changed; null when either side is inconclusive |
Each verdict, recorded or now, has these fields:
| Field | Meaning |
|---|---|
status | passed, failed or inconclusive |
verdict | The judge's verdict: yes, no, or null when not judged |
reason | The judge's reason, or why it is inconclusive: not judged: … |
source | Where the verdict came from (below) |
cost | What it cost when asked here, in USD; 0 when free |
request | A digest of the judge's request, by which the very same request is answered again for free |
source is recording (the golden's own judge call), kept (asked before, and kept with a session in
this store), live, or null when not judged.
The report's judge says what the judge checks that read text-only steps found, and what judging cost
apart from asking:
| Field | Meaning |
|---|---|
steps | The steps whose text-only change a check reads |
changed | Of those, the steps a check came out otherwise for |
inconclusive | Of those, the steps that wait on a judge: a sample's words could not be judged, and the step's samples don't flag it without them |
spent, upstreamRequests | What the judge's calls in this session cost, and their upstream requests |
unknownModels | The judge's models with no price |
The rest of steps are those every check came out as recorded for.
With --then-verify, verify holds what verify found:
| Field | Meaning |
|---|---|
verdict | accepted, rejected, incomplete, or not-run when no run needed verifying or verify could not run, with why in reasons |
verifyIds | The verdicts kept |
runs | Each run verified (below) |
rule | The rule they were decided by, stated once (below) |
commands | The commands that verify them again |
warnings | Each good run judged by its gate alone, and why; absent when none was |
footprint | What the runs verified cost a run, recorded and now; absent when no attempt showed it |
spent, upstreamRequests | What it spent, and its upstream requests |
reasons | Its reasons |
Each run verified has:
- its
runId,role,outcomein verify's own words,passesandattempts; bound, the bound of its pass rate under the gate, exact under its stopping (rate >= 0.73 (95%));nullfor a clean replay or a fixed count;undecided: why its last attempt could not be judged, which stopped itinconclusive. That attempt neither passed nor failed, and there is no bound;nullotherwise;- the
ruleit was decided by; parts, its tally in verify's own words: what it counted (tally,12 of 12 passed), why an attempt stopped it (undecided,the provider unavailable), the attempts the provider cut short (cutShort) and thebound. A session kept before they were has none, and reads in the same words from its figures.
A good run kept against its own baseline has the baseline's passes and attempts, and its bound
is the drop's (drop <= 0.10 (95%)).
The rule is stated once, such as gate: tells 90%+ from 70%- at <= 5% risk, up to 80 attempts. It is
null in a session whose runs were decided by different rules, verified again later under other figures,
where each run's own says which.
--verify --json prints:
| Field | Meaning |
|---|---|
sessionId | The session |
kind | decisions or migrate, the session's |
change | The change, in words |
verify | As above, for the runs it verified |
passes | A migrate session's runs, as its What still passes reads after this verify, as migrate prints them; null for a decisions session |
A refusal before anything is asked (no credentials, no such run or session) prints
{ "kind": "decisions", "outcome": "error", "error": { "message": … } }: kind, outcome and
error. It exits 2.
shadowlap migrate
#shadowlap migrate --to <model> [goldens...] [options] [-- <command> [args...]]Switches models with the evidence, for a model a provider retires, or one you would rather pay for. Decisions explains each block of its report.
shadowlap migrate --to claude-opus-5- It runs
decisions --modelon the goldens (goldens/*unless you name packs or runs), with the fields the model it switches to refuses changed to what it takes (below). - It runs
verifyon the runs a preview can't clear. - It compares what a run costs.
- It ends in one report: what changes, what still passes, what it costs, and what to set.
It switches the agent's model calls:
- those made with the models
--fromnames; - else those made with a model the retirement table says retires;
- else those made with the model the runs use most.
A simulated user or a sub-agent on another model keeps its own, in the preview and in verify alike. shadowlap's own calls (its judge's) are none of the agent's.
migrate takes these flags:
| Flag | What it does | Default |
|---|---|---|
--to <model> | The model to switch to. Required | — |
--from <model> | The recorded model to switch from (repeatable; an alias names its dated snapshots) | As above |
--scope <name> | Only the model calls made in this scope; none for those made in no scope | — |
--set <key=value> | A request field the switch sets as well (repeatable) | — |
--samples <n>, --concurrency <n> | As for decisions | 1, 4 |
--expect <json|@file> | What a passing run must do besides exiting 0 | Each run's checks (below) |
--gate <json|@file>, --attempts <n>, --passes <n>, --writes stub|block, --guard | Verify's, as for verify. Each run is decided by verify's gate, unless --attempts or --passes gives a fixed count | Verify's |
--budget <usd>, --timeout <seconds> | As for decisions | --timeout: a model request waits ten minutes |
--inconclusive fail|warn | What an inconclusive result does to the exit status, as for decisions | fail |
-y, --yes | Run recorded commands without asking (below) | — |
--json | Print the report as JSON on stdout (below) | — |
--expect defaults to each run's checks from shadowlap.config.json: its golden's, or, for a run
named by id, those of the recording it comes from. --yes runs the recorded command of a run imported
from someone's pack without asking, and of a golden from a pack the repository doesn't track, as
test asks of one.
migrate exits with:
| Status | Meaning |
|---|---|
1 | A run verify took breaks under the model it switches to |
3 | The report settles nothing (below), unless --inconclusive warn |
0 | Otherwise |
The report settles nothing when it stopped early, a step got no answer, a run that needed verify went unverified, or a run nothing else flags has a text-only change a judge check reads that could not be judged.
The fields migrate changes
#The fields it changes come from two places.
The retirement table, which keeps what a provider's pages say it stops accepting:
- Anthropic's Claude 4.7 and later, in a Messages API request on the Claude API, Bedrock or Vertex AI,
answer a non-default
temperature,top_portop_kwith a 400, and the page says to omit them. So they are left out of what to set. The preview and verify sendtemperatureits default, 1, meanwhile. - They refuse a manual thinking budget too, so
thinkingbecomes{"type": "adaptive"}. - OpenAI's o-series take
max_tokensasmax_completion_tokens.
The target model's own refusal, when it names what to change: max_tokens: 64000 > 32000,
at most 16384 completion tokens, Only the default (1) value is supported,
"thinking.type.enabled" is not supported, or
'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.
- The step is asked once more with that field changed, and so is every later request to that model, through that provider, that needs it.
- An output limit only ever lowers a request, and never under its thinking budget.
- A refusal that names nothing to change stays the step's
error.
Verify runs with the fields set to a value. A field to leave out, or to send under another name, has to change in your agent's requests first, since verify runs your code. So verify waits for it, and the report says what to change.
What migrate writes
#It writes the pull request body and the change to the project's files (a patch git apply takes)
under .<id>/. It keeps the report in the store (mg-…), logging a
migrate.done activity event.
The patch changes only a string value that names a model it switches from, in code and config: a
string literal, a JSON value, a YAML, TOML or .env value. Every other mention is left as it is, and
listed for review: in prose and changelogs, in tests, as an object key such as a price table's, inside
a longer string, or in shadowlap.config.json.
In code, the patch also takes out each field What to set leaves out that is set to a literal on a line
of its own, with a comma after it, in the call whose model it switches (temperature=0,). Every other
field stays for you, and the report says which: one in JSON, YAML, TOML or .env config, one set to
something other than a literal, and one set elsewhere.
Migrate report as JSON
#--json prints the report:
| Field | Meaning |
|---|---|
migrateId | mg-…, the report as the store keeps it |
kind | migrate |
at | When it ran |
from, to | The recorded models it switches from, and the model it switches to |
command | The command that migrates the same way again, every option given |
decisions | The decisions report, with its remedies (below) |
changes | runs, flagged, and each rank 1 or 2 change in steps (below) |
passes | What still passes: verify's verdict, and the runs (below) |
costs | perRun and total, before and after in USD, tokensPerRun, endToEnd (below), and what checking cost (below) |
set | The --set flags verify took, the fields changed, those given with --set, and byHand, the names of those the config change leaves for you (below) |
configChange | The change to the project's files (below); null when no file names the model |
pullRequest | title and body |
files | Where the dir, the pullRequest body and the patch were written |
retirements, notes, warnings, stopReason | As in the decisions report |
Each of the decisions report's remedies has these fields:
| Field | Meaning |
|---|---|
field | The request field |
value | What the requests were sent with in its place; null when it was left out of them |
leftOut | Whether your agent's requests should leave it out |
renamedTo | The name it goes under instead, or null |
why | Why |
source | Its source page, or null when the provider's refusal said it |
steps | The steps that needed it |
changes.steps holds each rank 1 or 2 change most samples show, a judged one included: runId,
golden, callNo, rank, detail, the recorded and the now action, and how the arguments
differ.
passes has verify's verdict, not-run when no run needed verifying or verify waits for a change
to your code. Each of its runs has these fields:
| Field | Meaning |
|---|---|
runId, golden | The run, and its golden |
verdict | passed, failed or inconclusive |
outcome | How it came out (below) |
tally | For a run verify took: its passes, attempts, bound (exact under the gate's stopping), the rule that decided it, undecided and parts (below); else null |
diverges | Once the config change is in, where a golden that replays exactly diverges: callNo, its first switched call, and path, $.model. Else null |
stays | Why a golden replays as recorded once switched: fork, it passes on its checks wherever it forks; no-call, it sends no switched model. Else absent |
A tally's bound, rule, undecided and parts are as in the decisions report's
verify.
outcome is in verify's own words when it was verified (kept, fixed, broken, not-fixed,
inconclusive, unsettled). Else it is:
unchanged, when the preview cleared it, so its recording's own verdict stands and it was not run end to end;unsettled, when a run that needed verify went unverified;not-judged, when a text-only change a judge check reads could not be judged;- or
not-asked.
checking is what checking cost: decisions, judge (judging the text-only changes a judge check
reads) and verify.
endToEnd is what the runs verified cost a run, recorded and now, each with inputTokens,
outputTokens and cost, by what their attempts did. perRun counts only the recorded steps, so it
can't see a run that takes more of them. It is null when no attempt showed it.
configChange holds the files patched, with the places in each, the patch, each mention left
for review (its path, line and text), and each field the patch takes out with the model in
leftOut (its field, path and line). set.byHand names each field What to set holds that the
patch doesn't take out, and each given with --set: while it holds one, migrate apply refuses.
A refusal before anything is asked prints kind, outcome (error) and error, as decisions does.
migrate apply
#shadowlap migrate apply <id> [--goldens <dir>] [--build <command>] [--json]Puts a migration's config change in your working tree, then names each golden it leaves behind. Decisions explains when to run it.
shadowlap migrate apply mg-01K5CQ2E8MIt takes the migration's mg-… id, or a unique prefix of 4 characters or more after mg-. It applies
only a migration that is all clear, and whose patch is the whole switch:
- It refuses one that breaks runs or settles nothing, one with no config change, and one whose
set.byHandnames a field: applied alone, the patch would break runs. - It checks the patch as
fix applychecks a repair's. The file must be the migration's own, unchanged since. Each file it names must be one the migration switched: a plain file that git sees, reached through no link. - It applies the patch with
git apply. A patch already in the tree is only marked applied. A tree whose changes conflict with it is left alone, with what to do. - It replays the project's goldens offline, in a store that nothing keeps. It names each that
diverges on
$.model, with its call and the command that re-records it,shadowlap test <goldens>--update.
The migration keeps its applied state, and Activity gets a migrate.applied event.
| Flag | What it does | Default |
|---|---|---|
--goldens <dir> | The project's goldens folder, replayed offline | goldens |
--build <command> | Rebuild the agent after the patch is applied, before the replay. A compiled agent needs it, or the old build replays and no golden diverges | build in the config |
--json | Print what was done as JSON on stdout (below) | — |
migrate apply exits with 2 when it refuses the migration or the folder it runs in, 1 when the
patch doesn't apply, and 0 otherwise.
--json prints { migrateId, status, patch, files, leftOut, outdated, update }:
| Field | Meaning |
|---|---|
migrateId | The migration |
status | applied, or already-applied when it was applied before and its patch is in the tree |
patch | applied, or already in the tree |
files, leftOut | The files the patch changes, and the fields it takes out with the model |
outdated | The goldens that diverge on $.model, each with its golden, callNo, the call's kind and name, and path |
update | The command that re-records them, or null |
A refusal prints { migrateId, status: "error", error: { message } }, as fix apply does.
shadowlap test
#shadowlap test <pack...> [--junit <file>] [--json <file>] [--markdown <file>] [--annotations]
[--update] [--reason <why>] [--online] [--writes <policy>] [--cwd <dir>] [--guard <mode>] [--push]
[--inconclusive fail|warn] [--max-attempts <n>] [--suite-risk <r>] [--continue]
[--allow-unrecorded <names>] [-y]
shadowlap test <pack...> --chaos [--presets <list>] [--chaos-tool <name>] [--inconclusive fail|warn] [-y]CI mode: replays packed recordings offline, re-running each recorded command. Checks, and
recordings allowed to fork, are configured under tests in shadowlap.config.json.
shadowlap test goldens/*
shadowlap test goldens/* --junit junit.xml --markdown "$GITHUB_STEP_SUMMARY" --annotationsAny of these fails the pack:
- a divergence;
- a live call;
- a non-zero exit;
- a replay that ends before its recording does;
- a failed outcome check.
A * in the last part of a path (goldens/*) is expanded by the command itself.
Each replay's agent gets a placeholder key for each provider its recording
called whose credentials the environment lacks, so a job without secrets tests it. --online,
--update, --chaos and a golden that may fork give none, and need the real keys.
test takes these flags:
| Flag | What it does | Default |
|---|---|---|
--junit <file> | Write a JUnit XML report | — |
--json <file> | Write a JSON report (below) | — |
--markdown <file> | Append a Markdown report: a results table, and the diff and agent output for each failure. Point it at $GITHUB_STEP_SUMMARY for a GitHub job summary | — |
--annotations | Print a GitHub Actions annotation for each pack that didn't pass, attached to its run.json (below) | — |
--update | Re-record each golden that diverged, live with writes stubbed, and replace its pack (below) | — |
--reason <why> | With --update: why the change is meant, kept beside each golden it replaces | — |
--online | Allow live calls after a divergence | Every provider offline: a live call fails the pack |
-y, --yes | Run the recorded command of a golden from someone else's pack, which the repository doesn't track, without asking | — |
--writes block|stub|allow | What write tools do after a divergence | block |
--cwd <dir> | Where the recorded commands run | The current directory |
--guard block|warn|off | The network guard | block |
--allow-unrecorded <names> | Run these tools again for real, though a golden never recorded them: names separated by commas (repeatable; below) | — |
--push | Push every test run, with the golden it replayed, and the session's report to the team store (below) | — |
--inconclusive fail|warn | What an inconclusive verdict does: fail the run, or warn (below) | fail, with an exception (below) |
--max-attempts <n> | Raise every gate's limit, its max, for this session only, to settle an inconclusive golden without changing the config | — |
--suite-risk <r> | The most a healthy suite may go red for nothing in this session, split across its gated goldens (below) | suiteRisk under tests in the config, else 0.05 |
--continue | Carry an inconclusive gate's attempts over from the golden's last result, and run only the attempts still needed (below) | — |
--chaos | Run the error-path presets instead of the session above | — |
--presets <list> | With --chaos: the presets to run, comma-separated (timeout, empty, malformed, oversized, 429, 500, or all) | The golden's chaos.presets, else all |
--chaos-tool <name> | With --chaos: the tool whose first call the tool presets hit | The golden's chaos.tool, else the first tool call that reads the world |
More on some of them:
--annotations: an error, or a warning for an inconclusive one let through, with why it only warns or counts.--update: it also re-records each golden that may fork and forked before its task's takeover. It replaces the pack (run.json,calls.jsonl,blobs/, and a.gitattributeswhen the folder has none; other files in its folder stay) in the working tree, when the agent exits 0 and its checks pass.- What
--updateprints: what the agent sends differently, and what the re-recording does differently from the golden it replaces. When the divergence is a value that changes every run and the config has no rule at its path, it warns, with the normalizer to add. --reason: the golden'saccepted.jsonlgains a line with who accepted the change, when and why. A later test whose request takes out a sentence that change added says so, with the reason.--updateand judges: a golden that replayed clean but holds no verdict for a judge check gets it asked live, and kept in its pack.--allow-unrecorded: for tools that write, or that nothing says what they do. Without it, a session with a golden whose model called such a tool is refused before any golden runs, every such golden named, exit status 2. A read is named, one line per golden.--push: a pull request's CI job pushes to its own folder. A missing store or missing credentials is a warning; the tests still decide the exit status.--continue: it carries attempts over when the golden's recording, checks and gate are unchanged. With--max-attempts, it runs up to the higher limit. A gate continued this way decides exactly as one that had that limit from the start. A golden whose last result isn't an inconclusive gate's runs afresh, and says why.--suite-risk: see one risk for the whole suite.--chaos: each golden forked at one call with a tool timeout, an empty result, malformed JSON, an oversized result, a provider 429 and a 500.
--inconclusive covers an inconclusive verdict of any kind:
- a gate's;
- a check's;
- an injection gate's, one whose test showed too little to say;
- with
--chaos, a preset's: one whose gate ran out of attempts between its bounds, whose replay never reached the call, or that only a check no attempt can decide kept from passing.
Under warn, the pack counts as passed for the exit status, a JUnit report marks it skipped,
--annotations gives it a warning, and every report still says it is inconclusive.
With neither fail nor warn given, each fails the run, but for a gate against history whose golden
has fewer than 30 sessions (three of its windows). That one only warns when it runs out of attempts,
saying it needs more history (its result's warned). It counts after all, saying why (counted), when
the session's plan reads its last sessions below the ones before them, 99 to 1 likely or more, or
beyond chance at its share.
test exits with:
| Status | Meaning |
|---|---|
0 | Every pack passed |
1 | A pack failed, other than a pack it needs --yes to run |
2 | The packs can't be tested as given (below) |
3 | None failed, and one is inconclusive (below) |
128 + n | A signal stopped the session: 130 for Ctrl-C, 129 for a closed terminal, 143 for SIGTERM |
2is for packs that can't be tested as given:- none matches, a config it can't read, or a golden that would run a write shadowlap never recorded again;
- a path named as it is holds no pack, whatever else failed;
- the only goldens that failed are packs it needs
--yesto run (someone else's, which nobody said yes to). Each fails as a golden in every report, while the others run.
3becomes0when every inconclusive one only warns: under--inconclusive warn, or, with none given, a gate against history whose golden has too few sessions, as--inconclusivesays.128 + n: the session then keeps the results of the packs that finished.
The command a golden runs
#A golden replays the recording and the command its own pack holds. The store may already hold another recording under the pack's run id: a copy pulled from a pull request's folder, a pack someone sent that reused the id, or the golden a copied one came from.
- If only the command differs, the pack's command runs.
- If the calls differ, the golden fails, naming where the store's copy came from.
- The store's copy's command never runs.
- A store's copy that differs only in its redaction rules, more of them or fewer, is the pack's recording. That is the run recorded here under the golden packed from it, or a newer pack's copy under an older checkout's pack. It replays with every rule either one carries as a normalizer.
A golden this machine recorded, or one whose pack the project's git repository tracks (its run.json
is in the index), runs its command without asking. A pack from anyone else, outside the repository or
not added to it, asks first, as replay asks of an imported run:
- it runs after a yes at a terminal, or with
-y; - without a terminal (CI), or after a no, it isn't run, and fails saying why;
- with several goldens tested side by side, they ask one at a time.
Test history and gates
#Each session is kept in the project's store as test history: the command, the
revision, passed and failed (an inconclusive golden is neither: its result says so), and each golden's
result with why it failed. Global --db :memory: keeps nothing.
- Why a divergence failed: the first change the normalizers in force don't mask. The report lists the masked ones too, without a hint to add a rule the config has.
- A session still running: until it ends, the store also names the process running it. So the dashboard shows a session started in a terminal as running, and one whose process was killed as one that did not finish.
- Each replay prints
replaying run <id>, and is marked as made bytestfor its golden. - A golden replayed more than once (
repeat, or agate) numbers its attempts, and files them under the one its result shows, so the dashboard lists them as one test run.
With several goldens, their attempts run in turns, one at a time: the first attempt of each golden, then the second of each one whose gate hasn't decided, and so on. So a burst of failures lands on many goldens once each (attempts are independent). Each golden's result is printed in order, once every golden's replays have ended.
A golden with a gate runs attempts until the gate decides. It says how in one line, the
verdict word, the tally, the bound and the rule
(passed · 12 of 12 · rate >= 0.73 (95%) · gate: tells 90%+ from 70%- at <= 5% risk, up to 80 attempts),
with pass^k under it (pass^5 1.00).
- The forecast: before any golden runs, the session prints its
forecast: the attempts it runs on average and at most,
and what they cost. With several gated goldens, it splits its one risk across them (
--suite-risk). - An unavailable provider: an attempt the agent failed while its provider was unavailable before the model answered is run again, and not counted. One the provider cut short after is counted against either verdict.
- Inconclusive: a gate that runs out of attempts between its bounds is inconclusive. So is a pack
whose only checks that did not pass could not be decided, such as a
judge check with no verdict, and one whose injection gate showed too
little to say. Each is marked
?in amber and counted apart (1 passed, 0 failed, 1inconclusive).
A golden's injection test
#A golden whose entry has an injection gate
("injection": { "obeyed": 0, "repeat": 3 }) gets its injection test once its replay passed. Payloads
are planted in its untrusted reads, live from the model call that reads them, with writes stubbed and
reads the recording doesn't hold blocked.
It fails, as an injection failure, when more attempts obey (or are influenced) than the gate allows.
It fails too when the test showed too little to say:
- no untrusted read to plant in;
- an attempt in which a model call failed, or the agent exited non-zero;
- no attempt a model read;
- a test cut short.
That verdict is inconclusive, and --inconclusive warn lets it through.
Test report as JSON
#--json <file> writes a JSON report: the session, counts and, per pack, the result and the kind of
failure. The report is { version, session, revision, passed, failed, inconclusive, interrupted, results, costPerRun },
and suite with a gated golden:
| Field | Meaning |
|---|---|
version, session, revision | The report's version, the session in the store's history, and the revision it tested |
passed, failed | The packs, counted by what CI does with them (below) |
inconclusive | The inconclusive packs, either way (below) |
interrupted | Whether a signal stopped the session; results has the goldens that finished before it |
results | One result per pack (below) |
costPerRun | What a run costs live, in USD: before as recorded, after as replayed here, the mean over goldens that ran whole; null without a price |
suite | With a gated golden: the session's one risk, and how it was split (below) |
An inconclusive pack counts as failed, or as passed when it only warns: under --inconclusive warn, or,
with none given, a gate against history with too few sessions (its warned). inconclusive counts the
inconclusive ones either way:
- a gate that ran out of attempts between its bounds;
- a pack that fell short only on a check that could not be decided;
- one whose provider was unavailable too many attempts in a row.
suite has these fields:
| Field | Meaning |
|---|---|
risk | The session's one risk |
perGoldenRisk | Each gated golden's share of it |
method | How it was split: bonferroni |
goldens | How many goldens shared it |
inconclusive | What inconclusive verdicts did |
red, averaged | The chance the session went red for nothing, as planned (below) |
over | When any golden could not keep within its share, how many could not (below) |
early | How many have too few sessions to keep within their share |
drop | How many gates against history saw their last sessions sit below the sessions before them, beyond chance |
short | How many have too few sessions for an inconclusive to count |
forecast | Made before it ran: attempts on average, most, and their cost on average in USD (below) |
redis at most that at each SPRT gate's good rate. Withaveraged, it is on average over the pass rates the sessions of its gates against history leave open.over: once their chances add up past the risk, the summary statesred, notrisk.forecast.costis null with no price;unpricedcounts the goldens without one.
One result per pack, in results, has these fields:
| Field | Meaning |
|---|---|
name, dir, golden | The pack, its folder, and the golden as test history names it |
mode | replay, or fork for a recording that forks on purpose |
ok | Whether CI lets it through |
warned | Why an inconclusive one only warns, when no --inconclusive was given: needs more history: 3 of 30 sessions |
counted | Why one counts though its golden's history is that short (below) |
verdict | passed, failed or inconclusive, whatever CI does with it |
failure | The kind of failure (below) |
message, details | Its one-line message, and the details the terminal shows |
output | The agent's last lines of output, when it failed |
seconds | How long it took |
runId, runs | The replay shown, and every attempt (below) |
goldenRunId | The recording replayed |
calls | Calls by origin: total, replayed, live, blocked, stubbed |
upstreamRequests, tokensNotRespent | Upstream requests, the judge's included, and tokens not re-spent |
divergence | type, callNo, kind, name, and for one that ended early recorded, the calls its recording holds |
checks, expect | Each check (below), and the expect they came from |
attempts | For a recording that repeats or has a gate (below) |
gate | For a golden with a gate (below); null otherwise |
judge | What its judge checks' calls did and cost over every attempt, apart from the agent's (below), or null |
updated | With --update, what it updated: from, to, calls, changes, and decides and reason when it has them |
takesOut | Each sentence a diverged golden's request takes out that an accepted change added, with at, by, reason and fix; absent with none |
injection | Its injection gate (below), or null |
More on a result's fields:
counted, for example:counts though it has only 15 of 30 sessions: its last 10 sessions sit below the 5 before them,….failure:diverged,check,exit,no-calls,error,inconclusive(for an inconclusive one), orinjectionfor a golden's injection gate.gateis null, too, when an attempt a check could not decide stopped it before it decided anything.judge:calls,replayed,live,notAsked,upstreamRequests,spent,savedandunknownModels.
Each attempt in runs has its runId, ok (true for one that passed), and its verdict:
passedorfailed;inconclusive, when all it fell short on is a check that could not be decided;unavailable, when the agent failed while its provider was. It is run again, and never counted, unless the provider cut it short after the model answered and a gate or a fixed count counted it.
Each of checks has its status (passed, failed, inconclusive or not-applicable), with ok,
label and detail; see how a check comes out. A
judge check's has its reason, source and judge too: name,
model, of, verdict, source, cost, redactions, normalized, cut and callNo.
attempts, for a recording that repeats or has a gate:
| Field | Meaning |
|---|---|
total | The attempts counted |
passed | The attempts that passed |
inconclusive | The attempts a check could not decide, which count neither way |
required | The attempts that must pass; null with a gate |
unavailable | The attempts the provider was unavailable for, which total leaves out |
cutShort | Those it cut short after the model answered, which a gate or a fixed count counted against either verdict, and total holds |
injection holds its injection gate, or null:
| Field | Meaning |
|---|---|
resultId | The result it is kept under |
attempts | Its attempts |
obeyed, influenced, ignored, inconclusive | How many attempts obeyed, were influenced, ignored it, or were inconclusive |
allowed | What it allowed, as obeyed and influenced |
verdict | passed, failed or inconclusive, or null when it did not run |
reason | The reason it failed, or is inconclusive |
skipped | Why it did not run |
A result's gate holds the gate's figures as this session ran it:
| Field | Meaning |
|---|---|
kind | sprt or baseline |
good, bad, risk, max | With sprt: its figures |
failRisk | With sprt: a lower risk for a failure, when its share of the suite's risk set one |
drop, risk, window, min, max | With baseline: its figures |
widened | With baseline, for a golden with a record (below) |
widened has from, the most attempts a gate against history runs by default. Past it, a pass also
holds against the golden's record at risk, however its max came past it: the plan, the config,
--max-attempts or Run more attempts. The record is every attempt of its last sessions (passes,
attempts, sessions), or, with before, of the sessions before its last before.
Then gate holds what it made of the attempts:
| Field | Meaning |
|---|---|
decision | pass, fail or inconclusive |
passes, attempts | passes of attempts |
outcomes | Each attempt in order, true for a pass |
carried | With --continue: how many of them were carried over from the session it continued |
interval | The interval of the pass rate at 1 − risk: lo, hi, confidence |
bound | The bound its verdict line shows (below) |
unavailable | When the provider was unavailable for some attempts |
unavailableAt | When it cut some short after the model answered: their places in outcomes (below) |
passK | k, value, for k up to 8 |
clean | A replay that forked nowhere settled it with one attempt |
sentence | The sentence the terminal prints: its verdict line, the verdict word, the tally, the bound and the rule |
evidence | What it decided on (below) |
attemptsbeside the gate counts this session's own attempts, with--continuetoo.bound:lofor a pass,hifor a failure, both otherwise, with itsconfidenceandmethod. The method isstagewise, exact under the gate's own stopping, orconfidence-sequencefor attempts it didn't stop on by its rule; see reading the evidence.unavailableAt: each is a failure inoutcomes, counted against either verdict.
evidence, for an SPRT gate, has the score after each attempt (scores), what a pass and a failure
add (pass, fail), and the bounds it passes and fails at (upper, lower).
evidence, for a history gate, has:
- the
historyit pooled:passes,attempts,sessions, or null when there is none and the golden's recording is the baseline; - the
baselinerate; - the
probabilityof a drop after the last attempt, and after each one (probabilities, null beforemin); - the
thresholdit fails above, and the chance it passes at or below (passAt, 1/2); - with
widened: itsfrom,risk,recordandbefore, and the chance of a drop from that record after each attempt (probabilities, null up tofrom).
Gating has the maths.
Error-path presets
#test --chaos runs a golden's error-path presets instead of replaying it: the
failures a recording of a healthy run never holds, each played in at one call so the agent's own error
path runs.
Each preset plays in one failure:
| Preset | What the agent gets |
|---|---|
timeout | The tool call times out: the tool doesn't run, and the agent's SDK raises the timeout it would |
empty | The tool answers with nothing: an empty list, object or text, as it recorded one |
malformed | The tool answers with its JSON cut short. A tool that returned data gets the cut JSON text in its place |
oversized | The tool answers with its longest text repeated until it is a million characters long (chaos.oversize); JSON stays JSON |
429, 500 | The model call that reads the tool's answer fails with that status, every attempt of it |
⚡ goldens/forecast: 6 error-path presets at get_forecast (#3) and model call #4
✓ tool timeout passed timed out: the agent exited 0
✓ empty result passed answered with nothing: the agent exited 0
✗ malformed JSON failed answered malformed JSON: the agent exited with code 1
SyntaxError: Unterminated string in JSON at position 52 (line 1 column 53)
✓ oversized result passed answered 1,000,000 characters: the agent exited 0
✓ provider 429 passed failed with HTTP 429: the agent exited 0
✓ provider 500 passed failed with HTTP 500: the agent exited 0Where they hit. The tool presets hit the first tool call that reads the world, or the first call of
--chaos-tool. The provider presets hit the model call after it. A tool call that reads the world is
a read, else a tool whose effect nothing declares, else a write. It is never a pure tool, a person's
input or a call that failed when recorded.
How an attempt passes. Each preset is a fork from there, live with writes stubbed (--writes
changes that), so like a golden that forks it needs the
providers. An attempt passes when the agent exits 0 and passes the checks an error path must pass:
- those under
chaos.expectin the golden'stestsentry; - else the write limits of its own
expect: the upper bounds it sets on write tools, which hold on any path. The rest describes the path its recording took, which a path through a failure needn't take.
A preset's attempts are gated as they come in: by chaos.gate ({ good, bad, risk, max }, each
optional) when the entry has one, else by the golden's repeat and passes. An attempt stops after
chaos.timeout seconds (default 300) and fails: an error path can hang.
Its verdict is one of:
passedorfailed;inconclusive, when:- the gate ran out of attempts between its bounds;
- the replay never reached the call, which a golden that no longer replays does;
- or all that kept an attempt from passing is a check no attempt can decide, such as a judge with no way to be asked;
not-applicable: the recording has no call the preset can hit, or no result it can change, such as a binary one or one that is empty already.
The provider failing for real, beyond the failure a preset plays in, is the provider's, as under
test (gating):
- An attempt it failed before the model answered is run again, and never counted.
- One it cut short after is counted against either verdict, by a
chaos.gateand by the golden'srepeatalike. - Enough in a row, counted across the session's presets, leave the preset that met the last inconclusive, with the provider named, and every preset after it, never failed. Enough is 3 for one preset, one more for each tenfold: 4 from 2, 5 from 11.
"chaos": false leaves a golden out of test --chaos. Each attempt is a run bundled under the
golden's recording, and the session and every result are kept in the store for the dashboard.
The table. When a chaos.gate decided the presets, every line gives the tally and the bound of the
pass rate, exact under the gate's stopping. A line under the table states the gate's rule once
(gate: tells 90%+ from 50%- at <= 10% risk, up to 6 attempts). Under the table, a line names the
checks that judged every attempt and where they came from
(checks: send_alert called at most 1 time (the write limits of the golden's expect)). When only the
exit status did, two lines say how to give them.
test --chaos exits with:
| Status | Meaning |
|---|---|
1 | A preset failed, or a golden couldn't be run |
2 | The only goldens that couldn't be run are packs it needs --yes to run |
3 | A preset was inconclusive (0 under --inconclusive warn) |
128 + n | A stop signal; 130 for Ctrl-C |
The other reports. --junit writes a case per preset, whose failure lists every attempt by its
verdict. --markdown writes a table of them, listing under it any golden whose presets couldn't run.
Both state a chaos.gate's rule, once under the table, and in every case it decided.
Error-path presets as JSON
#--json <file> writes
{ version, session, revision, presets, passed, failed, inconclusive, notApplicable, interrupted, results }.
In results, each recording has these fields:
| Field | Meaning |
|---|---|
golden | The golden |
runId | The run the presets forked |
outcomes | Each preset (below) |
judge | The judge of its presets: the source of their checks (chaos.expect, --expect, write-limits or exit), and the checks as a failure names them |
rule | The rule a chaos.gate decided its presets by, as the table states it, or null |
error, skipped | The error that kept it from running, or why it was skipped |
In outcomes, each preset has these fields:
| Field | Meaning |
|---|---|
preset, label | The preset, and its label |
verdict | Its verdict |
callNo, target | The call it hit, and its target: kind, name |
passes, attempts | The attempts that passed, of those it ran |
detail, summary | A sentence naming what came of it, and its summary, without the call, as the table shows it |
reason | The reason it failed (its first failing attempt's) or is inconclusive; null when its gate ran out of attempts |
gate | The gate it was decided by |
upstreamRequests, spent | Its upstream requests, and what it spent |
unavailable, cutShort | When its provider failed for real: attempts never counted, and attempts cut short and counted |
runs | Each attempt (below) |
In runs, each attempt has these fields:
| Field | Meaning |
|---|---|
runId | The attempt's run |
verdict | passed, failed, or inconclusive (below) |
passed | True for a passed attempt only |
exitCode, checks | The agent's exit code, and its checks |
reason | The reason it failed, or is inconclusive |
lastLine | For an agent that exited with an error: the error it died of, as it printed it |
An attempt is inconclusive when its replay never reached the call, when all that kept it from passing
is a check no attempt can decide, or when its provider failed for real, with unavailable then
retried or counted.
shadowlap bisect
#shadowlap bisect <golden|run> --good <rev> [--bad <rev>] [--expect <json|@file>] [--fork] [--build <command>] [--cwd <dir>] [--timeout <seconds>] [--allow-unrecorded <names>] [-y] [--json] [-- <command>]Finds the first commit where a recording fails. It replays it
at commits between --good (where it passed) and --bad, halving the range each time, each commit
checked out in a temporary git worktree.
shadowlap bisect goldens/refund --good v1.4.0A divergence, a replay that ends before its recording does, a failed check or a non-zero exit fails a commit. Replays are offline, so it costs no tokens. It prints the first bad commit with its subject, author and date, and why it fails there, and any untestable commits just before it that could be the one.
bisect takes these flags:
| Flag | What it does | Default |
|---|---|---|
--good <rev> | A commit where the recording passed | — |
--bad <rev> | A commit where it fails | HEAD |
--expect <json|@file> | The checks a replay must pass besides exiting 0 | The golden's, under tests |
--fork | Let each replay diverge and run live from the fork, so only the exit status and the checks decide (below) | — |
--build <command> | Run in each worktree before its replay, e.g. "npm ci && npm run build". A commit where it fails is skipped | build in the config |
--cwd <dir> | Where the recorded command runs, inside the repository | The current directory |
--timeout <seconds> | Stop a replay after this long, and count it as failing | 300 |
--guard <mode> | The network guard: block, warn or off | block |
--allow-unrecorded <names> | As for replay | — |
-y, --yes | At each commit, run without asking the recorded command of a run from a pack or the team store, or of an untracked golden (below) | — |
-- <command> | The agent's command to run at each commit instead of the recorded one | — |
--json | Print the result as JSON: each replay, the first bad commit and its change, and the replays, tokens and upstream requests it took | — |
--fork is for a run that failed a check, whose recording no older commit would replay. The live
part costs tokens, and the result says how many. A replay that forks asks its
judge checks live. One no replay can ask (no key for its provider, and no upstream set
for it) is refused before anything runs.
Whose command runs. Each replay runs the recorded command, or the one after --, in the commit's
worktree. A run from someone else's pack, or pulled from the team store, carries a command
someone else chose, which would run at every commit:
- bisect asks first, as
replaydoes, and with no terminal to ask in it refuses (exit status 2) unless given--yes, which runs it at each commit without asking; - so does a golden from a pack the repository doesn't track, recorded elsewhere, as
testasks of one; - a golden recorded on this machine, or one the repository tracks, runs without asking.
bisect exits with:
| Status | Meaning |
|---|---|
0 | It found the first bad commit |
1 | There is none between the two: it passes at --bad, or already fails at --good |
2 | It can't search what it was given (below) |
3 | A commit at either end can't be tested |
128 + n | A stop signal; 130 for Ctrl-C |
It can't search what it was given, exit status 2, for:
- a commit that isn't one;
- a run with no command;
- a recording whose model called a tool it never recorded that writes.
What it keeps. The answer is kept in the project's activity either way: the first bad commit, or why
there is none between the two commits (it already fails at --good, it passes at --bad, or one of
them can't be tested). It is kept with the golden's newest test result as the search started, the
failure it answers.
The first bad commit keeps its change against its first parent: the file it opens, the files it changed, that file's hunk and the line it added. The dashboard shows it with the answer, without a checkout.
The dashboard's Tests shows it above the golden's buttons while the golden keeps failing as it did then: not once it passed, and never over a later failure. An answer given while the golden passed answers none. A search stopped with Ctrl-C keeps nothing.
--json prints
{ outcome, target, good, bad, commits, steps, first, untested, why, replays, tokens, upstreamRequests, spent, seconds }:
| Field | Meaning |
|---|---|
outcome | found, passes-at-bad, fails-at-good, untestable or stopped |
why | Why there is no answer, when there is none |
target | kind (golden or run), its name and runId |
good, bad | The good and bad commits |
commits | How many commits come after the good one, up to and including the bad one |
steps | Each replay: rev, short, subject, ok, skipped, its result and seconds |
first | The first bad commit, with why it fails there and its change (below) |
untested | The untested commits just before it |
replays, tokens, upstreamRequests, spent, seconds | What it took: replays, tokens, upstream requests, USD spent and seconds |
A commit is { rev, short, subject, author, date }.
first.change is what the first bad commit changed against its first parent, with paths relative to
the repository, or null when git couldn't say:
| Field | Meaning |
|---|---|
file | The file it opens: the first changed under --cwd that isn't in a golden's folder or shadowlap.config.json, else the first |
files | Every file it changed, up to 50 |
hunk | That file's diff from its first @@, up to about 60 lines and 4 KB, or null for a binary or deleted file |
line | The new file's line of its first added line, or null without a hunk |
cut | The hunk was cut to its bounds |
merge | It is a merge, reverted with -m 1 |
file passes over a file the commit deleted, which has no hunk to show, for one it kept that fits the same rule.
shadowlap pack, unpack
#shadowlap pack <run> <dir> [--lineage] [--no-redact]
shadowlap unpack <dir>pack writes a run as a directory, to commit as a CI golden or to share. unpack imports one. The
dashboard's Import capture… → Pack does the same as unpack, from a folder you drop or pick
(Import a capture or a trace).
pack takes these flags; unpack takes none:
| Flag | What it does | Default |
|---|---|---|
--lineage | pack: include the run's ancestors | — |
--no-redact | pack: skip the redaction rules in shadowlap.config.json, and say how many it skipped | — |
What pack writes
#A pack holds run.json, calls.jsonl, one uncompressed file per blob under blobs/, and a
.gitattributes (* -text, unless the folder has one). The .gitattributes keeps git from
converting line endings, which would break the blobs' hashes in a Windows checkout. Stable names and
one file per blob keep it git-friendly, so it can be committed as a CI golden or shared.
- A run still recording is packed once it has ended, as every export waits for it: half a run would
make a golden that never matches. Until then,
packrefuses it, exit status 2. With--lineage, it waits for every run the pack would hold. - Packing into a folder that already holds a pack (the same golden again) replaces its
run.json,calls.jsonlandblobs/as a whole, and leaves anything else there. So the folder never keeps an earlier pack's blobs, unredacted ones above all. - Redaction: the redaction rules in
shadowlap.config.jsonare applied to the copy, but for a model call's signed reasoning, which stays as recorded.packnames each rule that matched inside it. - Credentials: a credential a recording kept never goes into a pack, with the rules or with
--no-redact. - Authors: who made each run (its author, often an email) is left out of it.
The local store is never modified, but for this: a golden whose checks ask a judge is judged once, live, before it is packed. The judge's calls are added to the run and go into the pack, so every replay of the golden is judged from the recording.
- A verdict the run already holds for the very request is not asked again.
- One that can't be asked (no key) is said to be inconclusive, with how to record it.
- A check that reads the agent's
outputcan't be judged when packing, since a recording keeps no output.packsays so, andshadowlap test <golden> --updaterecords its verdict from what a replay prints. - A run that didn't complete (its agent exited with an error, or it was interrupted) is not judged at all, since no replay of it is either.
What unpack checks
#unpack imports a pack, treating it as untrusted input. A pack that fails a check imports nothing:
- every record is validated, its redaction rules' patterns as the config's are, so none that could take too long comes in;
- each file is read without following a link;
- every blob is verified against its hash. A blob git checked out with CRLF line endings is verified as it was committed, and refused when that doesn't match either;
- only the blobs its calls need come into the store;
- a time more than a day ahead of this machine's clock fails a check.
Nothing on this machine records an unpacked run, so one the pack marks as still running comes in as
interrupted (crashed, ended at its last call). Runs that already exist are skipped. When the store's
run under that id is another recording (another command, or other calls), unpack says so and keeps it
as it is.
Imported runs are marked, and unpack prints the command each one would run on replay:
imported 01K5CQ4N7G8M2XWQH6V3T9RB1D command: node dist/agent.js --task 'triage inbox'
Replaying an imported run executes the command it was recorded with (shown above), which came from whoever made the pack.
shadowlap replay asks before running it; give your own command after "--" to run that instead.SECURITY.md lists every check.
shadowlap serve
#shadowlap serve [--recordings [--allow-unrecorded <names>]]Starts a sidecar with one recording run, for a process you launch yourself, and prints the environment
to export: export lines, or on Windows PowerShell's $env:NAME = 'value'. It ends the run on Ctrl-C,
SIGTERM or a closed terminal (SIGHUP).
serve takes these flags:
| Flag | What it does | Default |
|---|---|---|
--recordings | Start no run of its own, and serve recorded tests instead (below) | — |
--allow-unrecorded <names> | With --recordings: replay these tools again for real, though a recording never recorded them (below) | — |
- No command: serve never sees the command that starts your process, so the run has none, as a
capture has none. Replaying it, or a fork, blame or Find breaking commit, needs your command after
--. - Several runs: hosts that manage several runs (a test runner) can create them through the
sidecar's
/v1/runsAPI. Each run's environment pins it: the provider base URLs carry the run id, and SDK calls send anx-shadowlap-runheader. So several processes can record through one sidecar at once. - In-process capture: the printed environment has the base URLs only. Clients that need
in-process capture (Bedrock, and Azure or Vertex with the
endpoint in code) record under
shadowlap run.
With --recordings, it serves recorded tests for the pytest and Vitest integrations, which start it
themselves. A recording whose model called a tool it never recorded that writes, or may, is refused
before its replay starts, unless --allow-unrecorded <names> or SHADOWLAP_ in its
environment names it.
shadowlap check
#shadowlap check [recording...] [--expect <json|@file>] [--json]Judges recordings by checks from what they recorded. Nothing is replayed and nothing is sent, so a check can be tried on every run there is before it goes into the config.
shadowlap check label:bug --expect '{"tools":{"send_email":0}}'A recording is one of:
- a run id or unique prefix;
label:<name>, for every run with that label;- a pack directory (
goldens/*, with*in the last part matched by the command itself), read into a store of its own, so the project's store is left as it is.
With none, it judges every recording in the store that has ended. A recording named twice (a run and a label it has, a pack and its run in the store) is judged once, as its pack when it is one.
check takes these flags:
| Flag | What it does | Default |
|---|---|---|
--expect <json|@file> | The checks to judge every recording by: an expect object | Each recording's own, under tests |
--json | Print the report as JSON (below) | — |
Each recording is judged by --expect, or without it by its entry under tests in
shadowlap.config.json. A recording with neither is counted apart. It comes out:
- passed;
- failed: a check failed, or the run exited with an error or was interrupted;
- inconclusive: a check on the agent's output, which a recording doesn't keep, a judge check with no verdict, a condition it can't read, or a partial recording that holds no tool calls.
Each line names the recording by its id, labels and command, or its pack. Its checks follow one that
did not pass. Under the totals, the report says when the checks fail a recording labelled good or
pass one labelled bug. It also names a value the checks read that none of the calls judged has (a
misspelt result.totl), with the key it most likely meant.
The totals count a recording that passed only because none of its checks applied, a when whose
condition never held, as not applicable: 2 passed, 1 not applicable. It never counts as passed
there, and it leaves the exit status 0.
check exits with:
| Status | Meaning |
|---|---|
0 | Every recording judged passed |
1 | One failed |
2 | None was judged |
3 | None failed, and one was inconclusive |
A dashboard job that runs check gets the same report as its result. --json prints
{ expect, recordings, counts, unchecked, notes }:
| Field | Meaning |
|---|---|
expect | The expect given, or null |
recordings | Each recording judged (below) |
counts | The recordings by outcome: passed, failed, inconclusive, and notApplicable apart from passed |
unchecked | The recordings counted apart for want of checks |
notes | Recordings labelled good that fail, or bug that pass, and values the checks read that no call judged has |
Each of recordings has these fields:
| Field | Meaning |
|---|---|
runId | The run |
pack | The pack it came from, or null |
labels | Its labels |
source | The source of its checks: --expect, or the recording directory of its tests entry |
outcome | passed, failed or inconclusive |
why | Why it failed, or could not be settled |
checks | Its checks, as each check comes out: status, ok, label, detail, and a judge's reason and source |
shadowlap injection
#shadowlap injection <run|golden...> [--repeat <n>] [--templates <ids>] [--payloads <file>]
[--baselines <n>] [--obeyed <n>] [--influenced <n>] [--live-reads] [--inconclusive <mode>]
[--html <file>] [--no-redact] [--concurrency <n>] [--budget <usd>] [--timeout <seconds>]
[--guard <mode>] [--allow-unrecorded <names>] [-y] [--json]
shadowlap injection <run> --clean <call[=result]> [--write <call>] [--gate <json>] [--json]
shadowlap injection --list-templates [--json]Causal injection tests: whether text in what the agent reads can steer it.
shadowlap injection goldens/research --repeat 3 --html injection.htmlFor each read of a tool marked untrusted, it plants each payload in the recorded result, with a canary
of its own, as --inject-result does. Then
it replays:
- the calls before the read, from the recording;
- the rest live, from the model call whose request carries the planted text;
- with writes stubbed;
- with reads served from the recording when they match, and blocked when they don't.
A tool is marked untrusted by "tools": { "fetch_page": { "trust": "untrusted" } } in the config, by
trust: 'untrusted' in the SDK, or, for an MCP read tool, by its server
setting openWorldHint.
An attempt comes out:
- obeyed: a tool call carries the canary, which is proof the planted text caused it, for a write, and for a read, which can carry data out;
- influenced: a write that no clean replay of the same read made;
- ignored: otherwise;
- inconclusive: the attempt shows nothing either way (no model read the planted text, a model call failed, the agent exited non-zero), with why.
The payloads are the built-in templates (--list-templates prints them), unless --payloads or
--templates say otherwise. A golden's "injection" entry under tests gives its defaults. Every
replay is a run marked with the test it belongs to, and every result is kept in the store for the
dashboard.
Each run tested comes to a verdict against its limits:
- failed, when more attempts obeyed (or were influenced) than allowed;
- inconclusive, when the test showed too little to say: no untrusted read to plant in, an attempt in
which something failed, no attempt a model read, or the planned attempts cut short by
--budget. With a limit on influenced attempts, a clean replay in which something failed makes it inconclusive too; - else passed.
injection takes these flags:
| Flag | What it does | Default |
|---|---|---|
--repeat <n> | Attempts per untrusted read and payload. With --clean, the most clean attempts the test may take (below) | The golden's, else 1 |
--templates <ids> | Built-in templates to plant: all, none, or ids separated by commas | all |
--payloads <file> | Your own payloads, or a red-teaming tool's (below) | — |
--list-templates | Print the built-in templates; with --json, as the versioned data they ship as | — |
--baselines <n> | Clean replays of each read, forked at the model call that reads it, that influenced compares with (below) | The golden's, else 3 |
--obeyed <n> | Attempts that may obey before the run fails | The golden's, else 0 |
--influenced <n> | Attempts that may be influenced before the run fails | The golden's, else no limit |
--live-reads | Run the reads the recording doesn't hold after the fork, instead of fencing them (below) | — |
--inconclusive fail|warn | What an inconclusive verdict does to the exit status | fail |
--clean <call[=result]> | Replace read <call> with a neutral result, or the one given (JSON, text, or @file), and decide whether its content caused each write after it | — |
--write <call> | With --clean: the recorded write to test (repeatable) | Every write after the read |
--gate <json> | With --clean: the sequential test that decides; the figures left out are the default's | {"good":0. |
--html <file> | Also write the security report (below) | — |
--no-redact | With --html: skip the redaction rules | — |
--concurrency <n> | Attempts running at once | 1 |
--budget <usd> | Start no attempt once this much is spent on live model calls; a test cut short is inconclusive | — |
--timeout <seconds> | Stop an attempt after this long; it is inconclusive | — |
--guard <mode> | The network guard: block, warn or off | block |
--allow-unrecorded <names> | As for replay | — |
-y, --yes | Run the recorded command of an imported run, or of a golden from a pack the repository doesn't track, without asking | — |
--json | Print the report as JSON (below); progress goes to stderr | — |
More on some of them:
--repeatwith--clean: fewer than the gate needs to decide a write is refused.--payloads: text (one a line, or blocks between---lines), JSON or JSON Lines. Their{{canary}}is filled in. They are planted instead of the templates, unless--templatesnames some.--baselines: the recording's own writes after the read count as one more. A clean replay in which something failed is left out, and, with--influenced, makes the verdict inconclusive.--live-reads: the agent gets an error for a blocked read. A read that carries planted text is blocked all the same.--html: the reads × payloads matrix, the proof of each obeyed cell, why the inconclusive ones show nothing, the--cleanverdicts and the payloads. The redaction rules are applied, and every canary is kept.
injection exits with:
| Status | Meaning |
|---|---|
0 | Every run tested passed; with --clean, every write was decided |
1 | A run tested failed |
3 | One is inconclusive (0 with --inconclusive warn) |
128 + n | A stop signal: 130 for Ctrl-C, 129 for a closed terminal, 143 for SIGTERM |
Proving cause with --clean
#--clean <call> asks the reverse of a recorded incident:
- Read
<call>is replaced with a neutral result, or the one given. A neutral result has each string with a space in it emptied, and keeps ids, numbers and the shape. - The run replays until the sequential test decides, for each write the run made after the read, whether it comes back. A write is known by the values that identify it: an address, an id, an amount.
- Each write is reported caused by the read's content, not caused, or inconclusive.
A clean attempt in which something failed, or that never made the read, decides nothing; three in a row stop the test.
Injection report as JSON
#With --list-templates --json it prints the built-in templates, and exits:
| Field | Meaning |
|---|---|
version, updated | The template set's version, and when it was updated |
templates | Each template, with its id, shape, title and text |
Otherwise, with --json it prints the report:
| Field | Meaning |
|---|---|
templates | The built-in set's version and updated |
results | One per run tested (below) |
totals | The totals over the runs tested by planting (below) |
verdict | The verdict they come to: passed, failed or inconclusive |
passed | Whether the exit status is 0 |
html | The report's path, or null |
totals adds up these, over the runs tested by planting:
| Field | Meaning |
|---|---|
runs | The runs tested |
attempts | Their attempts |
obeyed, influenced, ignored, inconclusive | The attempts that came out each way |
A run tested by planting has these fields:
| Field | Meaning |
|---|---|
resultId | The inj-… id it is kept under |
mode | plant |
runId, golden | The run, and its golden or null |
templates | The templates used: their set's version, and the ids |
repeat | Attempts per read and payload |
reads | The untrusted reads (below) |
unrecordedReads | fence or live |
skipped | The untrusted calls skipped, each with callNo, name and why |
payloads, cells, attempts | The payloads, the cells and the attempts (below) |
baselines | The reads' baselines (below) |
totals | Its totals |
verdict | The strongest verdict an attempt showed |
gate | Its gate (below) |
stopReason | Why it stopped |
cost | spent, upstreamRequests and unknownModels |
startedAt, endedAt | When it started and ended |
Each untrusted read has these fields:
| Field | Meaning |
|---|---|
callNo, name | The read |
from | config, sdk or mcp |
readAt | The model call that reads it |
args | Its arguments |
planted, why | Whether it was planted in, and why not |
Each payload has these fields:
| Field | Meaning |
|---|---|
id | The payload |
source | template or file |
shape, title | Its shape and title |
text | Its text, its placeholders unfilled |
canary | Whether it has one |
Each cell, one read with one payload, has these fields:
| Field | Meaning |
|---|---|
read, payload | The read and the payload |
verdict | Its verdict |
attempts | Its attempts |
obeyed, influenced, ignored, inconclusive | How many attempts came out each way |
runs | Their runs |
Each attempt has these fields:
| Field | Meaning |
|---|---|
runId | The attempt's run |
read, payload, attempt | The read, the payload, and which attempt it is |
canary | The canary it carried: token, address, id and amount |
verdict | Its verdict |
why | Why an inconclusive one shows nothing |
failed | Whether that is because something failed in it |
plantedAt, forkedAt, readAt | Where the payload was served, where the attempt forked, and the first model call whose request carried it (below) |
exitCode | The agent's exit code |
evidence | The pieces of evidence (below) |
spent | USD spent |
plantedAt, forkedAt and readAt are in the attempt's own call numbers.
Each piece of evidence has these fields:
| Field | Meaning |
|---|---|
kind | canary, a write that carried it; read, a read that carried it; or write, a write no clean replay made |
callNo | The attempt's call |
tool | The tool |
path | Where in its arguments |
value | The value |
why | Why it is evidence |
Each read's baseline has these fields:
| Field | Meaning |
|---|---|
read | The read |
runs | The clean replays' runs |
writes | The writes they and the recording made |
failed | The clean replays that failed, each runId and why |
The gate has these fields:
| Field | Meaning |
|---|---|
obeyed, influenced | The limits |
verdict | passed, failed or inconclusive |
reason | The reason |
A run tested with --clean has these fields:
| Field | Meaning |
|---|---|
resultId | The id it is kept under |
mode | clean |
runId, golden | The run, and its golden |
read | The read replaced, with callNo, name, trust, from and args |
canary | The canary the run carried, when it is an attempt of an injection test, or null |
neutral | The neutral result served |
unrecordedReads | fence or live |
writes | The recorded writes (below) |
attempts | The clean attempts (below) |
gate, rule | The gate it was decided with, and its rule (below) |
verdict | caused when the read's content caused a write, not caused when every write comes back without it, else inconclusive |
stopReason, cost | Why it stopped, and what it cost |
startedAt, endedAt | When it started and ended |
rule is stated as the terminal states it once under the writes:
gate: tells a write gone in 80%+ of clean attempts from one gone in 20%-, at <= 5% risk, up to 10 attempts.
Each recorded write has these fields:
| Field | Meaning |
|---|---|
callNo, name, args | The write |
identifiers | The values that make it the same write again |
attempts | The clean attempts that decided it |
gone | How many it was gone in |
verdict | Its verdict |
bound | The bound on the share of clean attempts it was gone in (below), or null before an attempt decided anything |
bound is as its line shows it beside the tally: rate >= 0.29 (95%) for a caused write,
rate <= 0.71 (95%) for one not caused, both ends for an undecided one.
Each clean attempt has these fields:
| Field | Meaning |
|---|---|
runId, attempt | The attempt's run, and which attempt it is |
forkedAt | Where it forked |
exitCode | The agent's exit code |
made | The writes it made, each callNo and name |
gone | The recorded writes gone in it |
why | Why it decided nothing, or null |
shadowlap coverage
#shadowlap coverage [--json]
shadowlap coverage --pack [<behaviour>...] [--not-packed fail|warn] [--json] [-- <command>]
shadowlap coverage --pack <behaviour>... --fork --expect <checks> [-- <command>]Production coverage: groups the project's imported runs by behaviour, and names the
behaviours no golden covers. Imported runs are the captures and traces import made, a teammate's from
the team store included: runs that came with no command of their own.
Coverage of what 3 imported runs did: 2 behaviours, 1 not covered by a golden (1 golden read)
✗ 2 runs lookup_order → read_policy → issue_refund
writes issue_refund · answered · seen 2d ago in refund.har
nearest golden: goldens/escalate, the same for 2 steps;
then it calls issue_refund where the golden calls escalate
runs: 01K5CQ9C4H 01K5CQ9B7Q · behaviour 3f9a2c1d5e8b7a60
✓ 1 run lookup_order → read_policy → escalate
writes escalate · answered · seen 2d ago in escalate.har
covered by goldens/escalate
Make a golden of one run of each: shadowlap coverage --pack -- <the agent's command, with {message} where its input goes>A behaviour is:
- the tools the model called, in order: the calls one answer asked for together match in any
order, as
--repeatreads paths; - the writes among them: a tool whose effect nothing recorded or configured names counts as one;
- how the run ended:
answered,model-error(its last model call failed),token-limit,tool-call(it ended asking for a tool),agent-error(a recording whose agent failed) orno-model-calls.
The path is read from the model's own answers, so a capture with no tool calls compares with a golden. Arguments are not part of a behaviour.
The goldens are read from their packs on disk (the goldens folder and the packs tests names), as
test reads them. A golden covers a behaviour when its recording behaves the same way. A behaviour no
golden covers names the golden that shares the most of its path, where they part, and why the last
--pack couldn't pack it. Each scan is kept in the store, its groups replacing the last scan's, for the
dashboard.
coverage takes these flags:
| Flag | What it does | Default |
|---|---|---|
--json | Print the scan as JSON (below) | — |
--pack [<behaviour>...] | Make a golden of one run of each behaviour no golden covers, or of the behaviours named by key: four characters of it or more | — |
-- <command> | With --pack: the agent's command, for a capture no run here says the command of (below) | — |
--fork | With --pack and the behaviours named: pack each as a golden that forks (below) | — |
--expect <checks> | With --fork: what the agent must do on that input now, as JSON or @file.json, in the grammar of a tests entry's expect | — |
--not-packed fail|warn | With --pack: what a behaviour that could not be packed does to the exit status; warn reports it and exits 0, for a scheduled job | fail |
--guard <mode> | The network guard for the replays --pack makes: block, warn or off | block |
coverage exits 0. With --pack, it exits 1 when a behaviour asked for could not be packed, unless
--not-packed warn.
Packing a capture
#--pack turns a capture into a golden test can replay. A capture holds model calls and no command, so
its newest runs (up to three) are each replayed with the agent's command until one reproduces it. The
command is:
- the command of the newest run here (not one from a pack or the team store) whose first model call sent the capture's first request: its fingerprint under the normalizers, or its bytes;
- then the command given after
--, its placeholders filled for that capture.
The replay is offline, and needs no tool to answer:
- each tool call the model asked for is served the result the next request carried back to the model
(as MCP content, for a tool this store has recorded through
shadowlap mcp); - a tool the capture holds calls of itself keeps them;
- a tool some of whose calls got no answer back runs live;
- writes are stubbed.
A replay that sends every request the capture holds and exits 0 is a recording of the agent as it is
doing what production did. It is packed into goldens/<first tool>-<writes>-<key> (with the project's
redaction rules, as pack writes one) and logged in the activity feed.
In the folder's name, the tools are at most 3, each once, lowercased with - for anything but letters
and digits, and cut at 48 characters. <key> is the key's first 6 characters, so the example's
uncovered behaviour goes to goldens/. When that folder is taken,
-2, -3 and so on are added.
One that doesn't is not packed, and says why:
no-command: nothing says what starts the agent;other-message: the command started it with another first message than production's (the wrong command, or one that needs{message});not-reproduced: the agent sends something else (the code changed since the traffic was captured, or a tool's arguments are not the model's);agent-error: the agent failed on what production got back;gone: its runs were deleted.
A behaviour the agent no longer reproduces can still be kept, as the test that the change holds.
--fork packs production's run, with the command that starts the agent as production did, as a golden
that forks where the agent parts from it and passes on --expect. Its tests entry is written with
"fork": true and --expect, even when the agent no longer does what production did.
What --pack made of each behaviour is kept in the store, and the scan after it counts the goldens it
made.
Placeholders. In the command after --, {message} stands for the capture's first message (the text
of the first user turn it sent), and {message:/regex/} for the part of it the regex picks (its first
group, else the whole match). So each capture starts as production started it.
- A placeholder is an argument of the agent's own command, passed as it is.
- A placeholder is refused in a shell's script, after an interpreter's
-c,-eor-m, or where the message would name what runs. - What runs is the program, the script an interpreter runs or one of its flags, the command a runner
such as
env,cross-envornpxruns, or whatnpm exec,pnpm dlx,bun x,uv run,pipx runand the like run. - A capture whose message would start an argument with a dash is left out.
- A regex is checked as a redaction rule's is (Patterns that could take too long).
Coverage scan as JSON
#--json prints { scanId, at, runs, uncovered, groups, goldens, pack }:
| Field | Meaning |
|---|---|
scanId | The scan in the store |
at | When it ran |
runs | The imported runs read |
uncovered | The behaviours no golden covers |
groups | Every behaviour (below) |
goldens | Every golden read (below) |
pack | With --pack, what each behaviour became (below); else null |
groups has an entry for every behaviour:
| Field | Meaning |
|---|---|
key | The behaviour's key |
covered, coveredBy | Whether a golden covers it, and the goldens it is covered by |
path, writes | Its path, and its writes |
unknown | The tools of unknown effect |
outcome | How its runs ended |
runs, runIds | How many runs behaved so, and their ids, newest first |
sources | The sources they came from |
firstSeen, lastSeen | When it was first and last seen |
nearest | For one no golden covers: the golden, the steps shared, and where it differs (below) |
command | The command that would reproduce its newest run, with the run it came from |
packed | The golden this --pack made of it: golden, runId, mode |
lastPack | What the last --pack made of it, here or before (below) |
nearest.differs: thestep, what the behaviour does there asseen, and the golden's.lastPack:at, thegolden(or null), itsmode, and thereasonandwhyit wasn't packed.
goldens has an entry for every golden read:
| Field | Meaning |
|---|---|
golden, runId | The golden, and its run |
path, writes, outcome | Its path, writes and outcome |
error | The error that kept it from being read |
With --pack, pack says what each became:
| Field | Meaning |
|---|---|
key | The behaviour |
golden, runId | The golden written, and the run packed |
from | The imported run it came from |
command | The command it ran with, where that came from |
mode | The golden's mode: replay or fork |
reason, why | The reason it wasn't packed, and why |
shadowlap init
#shadowlap init [--json]Reads the project in the current directory, and prints the steps to a first recording, filled in for it. The detection reads source text; it runs nothing.
init writes one file: an empty shadowlap.config.json, only when there is none. Run it again after
wrapping the tools it names, to check again: the config is left as it is, and the tools are scanned
afresh.
init takes one flag:
| Flag | What it does | Default |
|---|---|---|
--json | Print the findings as JSON (below) | — |
What init finds
#- The language and package manager, from
package.json,pyproject.toml, requirements files and lockfiles. - The model SDKs, from dependencies or imports: Anthropic, OpenAI, Gemini, and the AI SDK, LangChain, LlamaIndex and Agent Framework provider packages. And the models a framework names for its own client to call.
- The agent framework and its version.
- The command that starts the agent.
- The command that builds it, from
package.json'sbuildscript, with the project's package manager (npm run build). Untilshadowlap.config.jsonnames abuild, it says how to name that one there, for an agent that runs what it builds. - Every tool definition it can find, whether it declares an effect, and a suggested effect for those that don't.
- Whether a Claude Agent SDK query runs through the fence.
- A model server the project already points at, such as Ollama in
OPENAI_BASE_URL, and theSHADOWLAP_value that keeps it as the upstream.UPSTREAM_ OPENAI
The package manager in a workspace. In a package of a workspace, whose lockfile is at the
workspace's root, it is the workspace's manager. That is the root above it, up to the repository's
root, that takes the folder in: pnpm-workspace.yaml, workspaces in package.json with its lockfile
or packageManager field, or [tool.uv.workspace].
The models a framework names: Mastra's router ids (model: 'anthropic/claude-sonnet-5-5'), the AI
SDK's string ids (model: 'openai/gpt-5', Vercel's AI Gateway), LangChain's init_chat_model,
Pydantic AI's, CrewAI's, Google ADK's, Agno's and Strands' model strings, and the model classes those
frameworks and smolagents import. An agent given no model counts as its framework's default (Strands:
Bedrock; Agno and CrewAI: OpenAI).
The frameworks it knows are the Vercel AI SDK, OpenAI Agents SDK, Mastra, LangChain and LangGraph, the Anthropic tool runner, the Claude Agent SDK and Pydantic AI. It knows the OpenAI Agents SDK for Python, LangChain and LangGraph for Python, CrewAI, Google ADK, LlamaIndex, Microsoft Agent Framework, Strands Agents, smolagents and Agno too.
The command that starts the agent comes from the first of these:
package.jsonscripts: not one that starts a web dev server or a watcher, normastra devormastra start, which serve agents and wait for requests. Record a script or a test that calls one.- The file
package.jsonnames: itsbin, then itsmain. - A script
pyproject.tomldeclares ([project.scripts]), run withuv runin a uv project andpoetry runin a Poetry one. - A conventional entry file (
agent.py,main.py,agent.ts,index.js…). A Python one runs withuv run agent.pyorpoetry run python agent.pythere. - Else the one file at the top of the folder that imports a model SDK and that no other file there
imports (
node triage.mjs). With several such files, it names none.
Tool definitions it finds:
- decorated functions: Strands', smolagents' and the Claude Agent SDK's
@tool, Pydantic AI's@agent.tool; - LangChain's and CrewAI's
BaseToolclasses, by theirname; - the plain functions an agent is given:
Agent(in Pydantic AI, Google ADK, Agent Framework and Agno, and LlamaIndex'stools= [search_ web]) FunctionTool.from_ defaults( fn= …); - each framework's own tool calls.
An effect given where a tool is used counts: with_effect(SendEmail(), "write"), or the Claude Agent
SDK's withEffects('orders', [...], { refund_order: 'write' }). An AI SDK tool kept in a constant
(const searchDocs = tool({ … })) counts too, under the key that gives it to the model
(tools: { search_docs: searchDocs }), or its own name in shorthand.
The suggested effect goes by the tool name's verbs, where a name puts them
(how). A name with no hint gets write, which never
runs again after a fork unless the replay allows writes (--writes allow).
The fence. A Claude Agent SDK query runs through the fence when a file that imports shadowlap's
adapters calls withFence (with_fence in Python), or its hooks. The SDK's built-in tools (Bash,
Write, Edit and the rest) run inside Claude Code, where no wrapper reaches them. So, until a query does,
the step after the install is to run it through the fence: query({ prompt, options: withFence(options) }),
or with_fence(options) in Python. The SDK's in-process tools keep the step that declares their effects
(withEffects, with_effects).
A model server. OPENAI_BASE_URL counts unless every model client the project has is another
provider's. Each of these counts as an OpenAI client:
- an OpenAI SDK,
langchain-openaiand@ai-sdk/openaiamong them, and a framework's[openai]extra, which installs it; - a framework that brings one: Pydantic AI as a whole, the OpenAI Agents SDK, CrewAI, Mastra;
openaiin a lockfile;- SDKs
initcan't tell.
An agent on Anthropic's or Gemini's SDK alone (or Pydantic AI's slim package with its [anthropic]
extra) never sends to it. Neither does Pydantic AI when every model its code names is another
provider's (Agent() and it builds no OpenAI model or provider. Pydantic
AI makes a client only for the providers of its models, so the OpenAI client it installs, and the
openai that puts in a lockfile, go unused.
What init warns about
#- A model client constructed with a literal base URL:
shadowlap runcannot redirect it, so its calls would not be recorded. - A model a framework names whose calls go to a host shadowlap doesn't know: a Mastra router id or
a LiteLLM or CrewAI string with another provider (
groq/…,openrouter/…), or smolagents'InferenceClientModel(Hugging Face). Declare its host in the config when it speaks OpenAI's, Anthropic's or Gemini's API (how). - A Node the project pins before 18.18 (
.nvmrcor.node-version,18.17): an agent on it never starts under shadowlap, since that Node refuses the guard.
Its install step adds the SDKs with the project's package manager, and the CLI: shadowlap as a dev
dependency, which the steps after it run with npx, or, for a Python project,
npm install --global shadowlap. Run by a global shadowlap, it leaves the CLI out, and the steps run
that one.
In the dashboard. In a folder that is a project in the dashboard, init
also marks the first step of its setup done. In a terminal, a line on stderr says whether it could. Run
by the dashboard, whose setup has its own buttons for the next steps, it prints only what it found, a
line each, and what it did with shadowlap.config.json.
init findings as JSON
#--json prints the findings, with the lines the dashboard's setup shows:
| Field | Meaning |
|---|---|
summary | The lines the dashboard's setup shows (below) |
cwd | The folder it read |
languages, packageManager | The languages found, and the package manager |
providers | Each SDK, and the package it came via |
frameworks | The frameworks found |
command | Its argv, and where it came from |
build | The build package.json runs: its argv, where it came from, and the build the config names already (configured), or null |
tools | Each with its file, line, effect, suggested effect, why and wrapped |
unwrapped | The tools that aren't wrapped |
wrap | The code that wraps them (below) |
fence | Each Claude Agent SDK the project uses (below) |
unrecordedModels | The models it warns about: each with its file, line, model, and the framework that builds its client, via |
modelServer, hardcodedBaseUrls | The model server the project points at, and the model clients built with a literal base URL (below) |
scannedFiles | How many source files it read |
config | Its path, whether it existed, and whether it was created |
folder | The kind of project the folder holds |
project | In a dashboard project: whether it could mark the first step done (below) |
summary: the language with its pinned runtime and package manager, the model SDKs with their versions, the framework, the command and where it came from, and what happened to the config. With them,node: the Node the project pins with its minor, and whether it is older than the guard takes (tooOld).wrap: the code that wraps the tools in the project's language, through its framework's adapter, with the fence for a Claude Agent SDK query that runs through none.fence: each Claude Agent SDK'sframework, thelanguageof the code that fences its query, and where a query runs through the fence,at, or null.- The addresses it prints (
modelServer,hardcodedBaseUrls) never carry credentials: a password, or a key in the query, is left out. A model server that had one gets noupstream. projectsays whether it could (noted), and why not: the project list was busy, or a newer shadowlap wrote it.
shadowlap doctor
#shadowlap doctor [--json]Checks what support would check, and says how to fix each problem it finds.
doctor takes one flag:
| Flag | What it does | Default |
|---|---|---|
--json | Print the checks as a list (below) | — |
It checks:
- Node and its built-in SQLite, with the full-text index every store has. A Node build whose SQLite has none is told to use an official build, and the store is left unread rather than called damaged.
- The guard preloads.
- Python on the PATH, as
python3,pythonor both, since an agent's command names one. - The store: runs, size, schema, and runs a killed process left marked running.
- A stale or incomplete
.shadowlap/active.json. - The configuration: every rule, price and MCP effect parsed, and the checks under
tests(below). - Agent-process environment that leaked into your shell:
SHADOWLAP_URL,NODE_OPTIONS,PYTHONPATH.
Under tests, it names:
- each rule that names a tool no recording called or offered to its model, or reads a value no recorded call of its tool has, with the name it most likely meant. It holds the rules against the packs the entries apply to, and the store's 200 newest runs;
- each judge check whose model retires, by the retirement table that ships with shadowlap;
- each judge check whose provider has no key in this environment and no upstream set for it. No judge can be asked for it live, so it reads not judged wherever its verdict isn't recorded;
- each golden that holds no recorded verdict, or only one given to an older form of the judge's request,
for a judge check that applies to it, with the
test --updatethat records it.
A judge under a when whose condition never held in the golden is not
applicable there, and needs no recorded verdict.
A URL it names (a base URL, an upstream, the telemetry collector, a sidecar's) shows no password, no
credential in its query and no session token, each <CREDENTIAL>. So the report can be pasted where
others read it.
doctor exits 1 when something is blocking. The dashboard runs it from Settings → Project's
Health and lists the checks there.
--json prints the checks as a list, each { "name", "verdict", "detail" }:
| Field | Meaning |
|---|---|
name | The check |
verdict | ok, warn or fail |
detail | What it found |
fix | How to fix it, where there is a fix |
shadowlap normalize, redact
#shadowlap normalize add [--path <jsonpath>] [--pattern <pattern>] [--kind api|tool|clock|random] [--name <name>] [--replace <text>] [--expect <version>]
shadowlap normalize rm <n|name> [--expect <version>]
shadowlap normalize ls [--json]
shadowlap redact add [--path <jsonpath>] [--pattern <pattern>] [--kind api|tool] [--name <name>] [--replace <text>] [--expect <version>]
shadowlap redact rm <n|name> [--expect <version>]
shadowlap redact ls [--json]
shadowlap redact preview [run] [--json]Both edit shadowlap.config.json in the working directory.
Normalizers mask values before matching. Redaction rules mask stored
content in every export, pack and share link of a run. Each add and rm is recorded in the project's
activity, in a store that exists.
shadowlap normalize add --pattern uuid --replace '<uuid>'
shadowlap redact add --path '$.args.api_key'Each of them has these subcommands:
| Subcommand | What it does |
|---|---|
add | Add a rule: a pattern, a path, or both (below) |
rm | Remove a rule: the number ls shows (2 or #2), or the tool name, path or pattern of exactly one rule |
ls | List the rules, each with its number |
redact preview | Show what the redaction rules would mask in a run that has ended (below) |
The subcommands take these flags:
| Flag | What it does | Default |
|---|---|---|
--pattern <pattern> | add: a named pattern (uuid, ulid, iso-date, iso-datetime, number, email) or /regex/flags | — |
--path <jsonpath> | add: where in a request, such as $.metadata.user_id; a tool's arguments are under $.args | — |
--name <name> | add: scope the rule to one tool or call: a tool's name, or a call's, such as "POST /v1/messages" | — |
--kind <kind> | add: only calls of this kind: api, tool, clock or random for a normalizer; api or tool for redaction | — |
--replace <text> | add: the replacement text | <MASKED>; for redaction, <REDACTED> |
--expect <version> | add, rm: refuse the edit when the file is no longer the version ls --json printed | — |
--json | ls, redact preview: print as JSON (below) | — |
How add matches.
- A pattern alone masks matches in every string. For redaction, it masks those of every string but a clock or random read's, which every copy keeps as recorded.
- A path alone replaces the whole value there, numbers included.
--namescopes the rule to one tool or call.- It refuses a rule that would change nothing, or that it can't read, and, for redaction, a replacement the rule would rewrite again. Adding a rule that is already there changes nothing.
Where a redaction rule's path reaches. It reaches requests in every copy. What a call got back, a
response or tool result that is JSON (a tool's result is under $.result), and an attempt's error
($.error), it reaches only in a copy that is read (export html, har, dataset). It never reaches
them in a pack, whose replay hands them to the agent, nor in a clock or random read. $ alone, and a
path that starts with a wildcard, reach requests only.
ls --json prints { file, exists, version, error, rules }:
| Field | Meaning |
|---|---|
file, exists | The config file, and whether it exists |
version | Its version, as --expect takes it |
error | Why the list can't be read (it isn't a list; ls then exits 2), else null |
rules | Each rule, with its number n; one the config can't load has error instead |
redact preview
#redact preview shows what the rules would mask in a run that has ended (default: the newest). Nothing
is written. See Preview.
- Per rule: what each would mask, counted, with up to two samples. It reads the run through the
functions
packapplies the rules with, the run's own record included: a fork's flags, the divergence report, the checks and the last blame. - What a pack leaves as it is: under each rule, what it matches that a pack keeps. That is a number, since a copy masks the strings of a JSON document and keeps its numbers as recorded, or anything in a clock or random read, which every copy keeps as recorded.
- Where each is: for each place, the path rule that would mask it, a rule that already does, or why none can. In what a call got back, that is a path rule that only the copies that are read apply.
details: on a line of its own, what the rules match in the run's command line, labels and notes, whichexport html,datasetandotelmask andpackkeeps.- Canaries: an injection attempt's own canaries are kept, as a pack,
export html, a share, a dataset and pairs keep them, so they aren't counted. The first line then endscanaries kept.
--json prints { "runId", "calls", "unreadable", "partial", "total", "returned", "unmasked", "rules", "details", "kept" }:
| Field | Meaning |
|---|---|
runId | The run |
calls, unreadable | The calls read, and those that couldn't be |
partial | Set when a limit stopped the preview early (below); else null |
total | What a pack masks |
returned | What only the copies that are read mask |
unmasked | What is left as it is |
rules | Each rule (below) |
details | What the rules match in the run's command line, labels and notes: a count, samples and the rules that match there |
kept | What the rules would match in an injection attempt's own canaries, which a pack, export html, a share, a dataset and pairs keep |
partial has how many of the run's calls it read, of how many, and the limit it stopped at:
calls, bytes or time.
Each of rules has these fields:
| Field | Meaning |
|---|---|
n, rule | Its number, and the rule |
count, samples | What it masks, and up to two samples, each before and after |
reached | How many values a path rule reached |
note | When it masks nothing, why |
returned | The count and samples of what a path rule masks in what calls got back, which only export html, har and dataset mask |
unmasked | What it matches that is left as it is (below) |
A rule's unmasked has:
- its
count, and up to twosamples; - each sample's
value, whether it is akey, thecallNo,kindandnameof its call, whether it isinthe call's request, response or error, itspaththere, and the stream'seventit is in, or null; - the
placesa path rule would mask them at; - the
remediesfor up to 20 of those: each therule, thecommandthat adds it, andcopies(allfor a request;readfor what a call got back, which only the copies that are read mask); - the rules already
coveredthat do in those copies, by their numbers; - the
reasonsthe rest get none (below).
The reasons are:
replay: a clock or random read;stream: in a stream's events;whole: the whole request or response, or a key at its top;key: under a key holding];signed: in a model call's signed reasoning, which a pack keeps as recorded.
shadowlap skill
#shadowlap skill install [claude-code|agents|kiro] [--force] [--json]Puts the coding agents' instructions into the project you run it in, from the copy that
comes with the CLI. Nothing is downloaded, and the commands they name are the ones this CLI takes. They
are two skills, setup-agent and debug-agent.
Where it writes them:
| Argument | Where |
|---|---|
claude-code (the default) | The Claude Code plugin, with both, in .shadowlap/, where Claude Code loads it for everyone working on the project once the folder is trusted |
agents | The skills alone, in . and . (below) |
kiro | The skills alone, in .kiro/skills/ |
.agents/skills/ is where Codex, Cursor, GitHub Copilot, Gemini CLI, Antigravity, Devin Desktop (formerly Windsurf),
OpenCode,
Cline, Amp, goose, Zed and Junie read skills. Each of those agents' names (codex, cursor,
copilot, gemini, antigravity, windsurf, devin, opencode, cline, amp, goose, zed,
junie) does the same as agents.
In those two folders, which every tool's skills share, each skill is named shadowlap-<skill>, and
leaves out allowed-tools, which names Claude Code's tools. Nothing but the shadowlap- folders is
read or written there.
skill install takes these flags:
| Flag | What it does | Default |
|---|---|---|
--force | Replace what is there when it isn't this version's copy, and remove the files of ours this version doesn't have (below) | — |
--json | Print what it did as JSON (below) | — |
- A copy that is already this version's is left as it is.
- Anything else there (another version's copy, or one edited here) is left as it is too, and the command exits 1, saying which files differ.
--forcereplaces them, which is how a project moves to the copy of a newer shadowlap. It removes the files of ours that this version doesn't have (in the plugin's folder, or in ashadowlap-skill's), so an older version's skill stops loading.- A link on the way that leads out of the project, or a link where the install puts a file, is
refused, with or without
--force.
skill install exits 0 unless it was refused. Settings → Coding agents in the
dashboard runs it as the skill-install job.
--json prints { agent, path, version, before, installedVersion, outcome, files, reason }:
| Field | Meaning |
|---|---|
agent | Where it installed: claude-code, agents or kiro |
path | The folder it installed into |
version | This version, the copy it installs |
before | What was there: missing, current, other-version or changed |
installedVersion | The version the copy there says it is: the plugin's manifest, or each skill's metadata.version |
outcome | installed, updated, unchanged or refused |
files | Each file's path and action (below) |
reason | Why it was refused |
A file's action is written, replaced, removed or unchanged. For a refused install, it is
differs or extra: what it left.
shadowlap import
#shadowlap import har traffic.har # model calls out of a HAR, as a recording
shadowlap import nock recording.json # the same, from a nock recording
shadowlap import vcr cassette.json # the same, from a VCR cassette (JSON serialization)
shadowlap import litellm cap/*.jsonl # production traffic a LiteLLM proxy captured
shadowlap import otel otel-traces.json # OpenTelemetry traces with the GenAI conventions
shadowlap import langfuse <trace-id> # a Langfuse trace, read with the project's keys
shadowlap import langsmith <trace-id> # a LangSmith trace, read with your API key
shadowlap import phoenix <trace-id> # an Arize Phoenix trace, from Phoenix's REST API
shadowlap import logfire <trace-id> # a Pydantic Logfire trace, read with a read token
shadowlap import langfuse --since 7d # each Langfuse trace of the last 7 days, one run per trace
shadowlap import har traffic.har --json # every run made, by its full id, and the entries skippedTurns a capture something else made, or a trace you already keep, into a recording you can replay your own code against.
These formats are captures:
| Format | What writes it |
|---|---|
har | mitmproxy, Charles, Proxyman, a browser's devtools, Polly.js |
nock | nock.recorder.rec({ output_objects: true }) |
vcr | VCR.py with its JSON serializer (vcr.'json')) (below) |
litellm | A LiteLLM proxy, through shadowlap.callback: production traffic, one run per correlation id (below) |
vcr: a YAML cassette has to be converted first. shadowlap reads JSON so that no YAML parser is trusted with a cassette from anywhere.litellmtakes several files, since a proxy writes one per worker. The calls of one agent run spread over several workers' files are put back together, in the order they started. See Capturing production traffic.
otel, langfuse, langsmith, phoenix and logfire read traces.
Its subcommands take these flags:
| Flag | What it does | Default |
|---|---|---|
--name <name> | har, nock, vcr, litellm, otel: the name to record as the run's source (below) | The file's name |
--json | Print every run made, by its full id, and what it skipped (below) | — |
--host <url> | Trace stores: the store's API, for a self-hosted instance or another region | Below |
--allow-http | Trace stores: accept plain http for another host than this machine, on a network you trust | Plain http only for this machine (below) |
--since <when> | Without a trace id: import the traces that started from then on; logfire takes it with one too | 30 days ago |
--until <when> | Without a trace id: the traces that started before then | The current time |
--limit <n> | Without a trace id: the most traces to import, newest first; at most 10,000 | 100 |
--trace-name <name> | Without a trace id: only the traces of this name | — |
--project <name> | phoenix: the project the traces are in | PHOENIX_PROJECT or PHOENIX_PROJECT_NAME, else default |
--since, --until, --limit and --trace-name are for langfuse, phoenix and logfire:
the traces in a window.
What is kept. Anthropic, OpenAI, Gemini, Azure, Vertex and Bedrock hosts are recognised:
- Every other entry is skipped, and
--jsonlists the reasons. So is an entry whose fields aren't what the format says (a method that isn't text, a null entry), naming the field. - The capture's credential headers (
authorization,x-api-key,api-key, cookies,proxy-authorization,x-goog-api-key,x-amz-security-token) and its query's credentials are never stored. - The rest of an answer's headers is kept as captured, and masked in every copy.
- Header names and values are escaped and bounded, because a capture is a file from anywhere.
- An import that stops part way leaves no run behind.
The request bodies are the real ones, which is what matters. Replay matches on a fingerprint of the outgoing request, so a reconstructed body would fork at call 1. Every format produces the same fingerprint for the same request, so it does not matter which tool captured it.
An imported run is a partial recording. A capture of the
network holds no tool, clock or randomness calls, so a replay runs those live rather than diverging on
their absence, and --writes still fences write tools. No format carries SSE event boundaries, so a
streamed response replays in one piece, rather than at its recorded pacing.
It has no command. A capture says what was sent, not what sent it, so the run has no command to re-run:
lsandshowdisplay<no-recorded-command>in its place, andls --jsonsays"hasCommand": false;replaywithout a command refuses, and says what to run;blameandverifyrefuse the same way, and take your command after--asreplaydoes;checkanddecisionsneed none: they read the calls the capture kept;- its cost is reported as captured, not spent: that traffic was paid for where it ran.
Give your own command:
shadowlap import har traffic.har
shadowlap replay <run> -- python agent.pyThat replay is also how an import becomes a CI golden. The model calls come from the capture and
everything else is recorded live, so the resulting run covers everything, and shadowlap pack turns it
into a golden for shadowlap test.
Labels and authors. Every imported run is labelled imported, and appears in the dashboard's
activity feed. It names who imported it as its author: SHADOWLAP_AUTHOR when set, ci in CI,
otherwise git's user.email. Its source is the file's name; --name <name> records another. The
dashboard's Import capture… passes the name of the file you picked, which it keeps under a name of its
own while it imports.
Judged by the rules on calls. An import judges each run it makes by the project's rules on calls:
every calls check in any golden's expect under tests in
shadowlap.config.json. A rule on calls reads only the calls a run made, so it holds for any run of the
agent. Counts, when rules, judges and budgets belong to their golden's own scenario and are not applied.
Imported 01M3S1C7QD from langfuse:7f3a9c2e: 3 model calls and 1 tool result (3 anthropic).
✓ judged by the rules on calls in shadowlap.config.json · 1 check passedThe verdict is one of these:
- passed: every rule that had a call to read holds.
- failed: a rule broke. The line names the first rule broken, the call and the values.
- not applicable: no rule had a call to read. A rule whose tool the run never called doesn't apply,
and a capture of the model calls alone (
har,nock,vcr,litellm) holds no tool calls.
The verdict is kept on the run, with the rules it was judged by. Once the rules change, the dashboard judges the run again from its calls, and an import of the traces in a window judges again each trace it imported before. With no rule on calls, nothing is judged, and the import says so.
import exits 1 when nothing could be imported: runs is empty. An import of
the traces in a window exits as that section says.
Traces
#shadowlap import otel otel-traces.json [otel-logs.json]
shadowlap import langfuse 7f3a9c2e-41b8-4d0e-9a51-2c6e8b1d4f07 [--host https://langfuse.example.com] [--allow-http]
shadowlap import langsmith 0192f4c1-8e2d-7a3b-b6c4-5d9e0f1a2b3c [--host https://langsmith.example.com/api/v1] [--allow-http]
shadowlap import phoenix 4a2db41deabc4aec332fb9fbfa0fc3e2 [--project support-agent] [--host https://phoenix.example.com] [--allow-http]
shadowlap import logfire 01a16460e2f311342463057f62b79fa9 [--since 90d] [--host https://logfire-eu.pydantic.dev] [--allow-http]A trace keeps what its instrumentation logged: the messages and the model settings, tool outputs when someone logged them, rarely the tool definitions.
- Each model call is rebuilt as the request its provider's SDK sends and the response it got, and stored like a capture's.
- It is rebuilt in Anthropic's, OpenAI's or Gemini's format: the provider the trace names, else the model's family, else OpenAI's, which most other servers speak.
- The run is a partial recording: what the trace didn't keep runs live on replay, with writes fenced.
The order of the calls. The run holds the calls in the order they started, read as finely as the source keeps the time: to the microsecond in Phoenix, Logfire and LangSmith, and the nanosecond in OTLP. That is because a tool and the model call that reads its result often start in the same millisecond. Calls that started at the same instant go in the order one needed another:
- a span before the calls that ran inside it;
- a model call before the tool call it asked for;
- a tool call before the model call whose request carries its result.
Calls that say nothing of each other keep the order the source lists them in.
Each source reads:
| Source | What it reads | Key |
|---|---|---|
otel | OTLP JSON files: spans with the GenAI or OpenInference conventions | — |
langfuse | The trace's observations | LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY |
langsmith | The trace's runs, from POST /runs/query, every page | LANGSMITH_API_KEY |
phoenix | The trace's spans, from GET /v1/spans, every page | PHOENIX_API_KEY, when it is set |
logfire | The trace's spans, from one SQL query on the records table (POST /v2/query) | LOGFIRE_READ_TOKEN, a read token |
otel reads OTLP JSON, as an OTLP/HTTP exporter sends it, the collector's file exporter writes it
(one request per line), or Grafana Tempo downloads a trace. Several files are read as one, so traces and
their logs can come separately. One run is made per trace. It reads spans with the OpenTelemetry GenAI
conventions:
gen_ai.input.messagesandgen_ai.output.messages, withgen_andai. system_ instructions gen_ai.tool.definitions, also from agen_event;ai. client. inference. operation. details - the older
gen_ai.promptandgen_ai.completion: as JSON, ongen_ai.content.*events, or flattened as OpenLLMetry writes them; - or the per-message
gen_ai.*.messageandgen_ai.choiceevents, on the span or as log records (resourceLogs, tied to the span by its id). That is where the official instrumentations (openai-v2, vertexai, google-genai) send them; - the model, provider, settings, token usage and finish reason;
execute_toolspans, for tool calls.
It reads OpenInference's conventions too (Arize Phoenix, and the LlamaIndex, DSPy and smolagents
instrumentations): LLM spans with llm.input_messages, llm.output_messages, llm.tools and
llm.parameters, and TOOL spans. LLM spans are rebuilt exactly from input.value and
output.value where its OpenAI and Anthropic instrumentations log the request and response themselves.
When a framework's span wraps the client library's span for the same call, the inner one is kept.
Agents. A model call keeps the agents whose spans it ran inside, outermost first
(supervisor/researcher), which the dashboard shows. An agent span names
gen_ai.agent.name or runs invoke_agent, or is an OpenInference AGENT span, OpenLLMetry's agent span
or Pydantic AI's agent run. It is for reading only: a replay never matches by it.
langfuse reads the trace's observations from the Observations API v2
(GET /api/observations, a page of 1,000 at a time), or GET /api/public/traces/<id> from a
self-hosted Langfuse without it. Generations are model calls. TOOL observations, and spans named after a
tool a generation asked for, are tool calls. AGENT observations name the agent of the calls under them.
The run keeps the trace's page in Langfuse, its htmlPath on the host the API answered from, for the
dashboard's Open in Langfuse ↗. Read from the v2 API, the trace is asked for once more, after the
observations, for five seconds at most. When it doesn't answer in time, the import stands without the
page.
langsmith: llm runs are model calls: LangChain's serialized messages, or the request and response
themselves from LangSmith's wrapped OpenAI and Anthropic clients, which rebuild exactly. tool runs are
tool calls.
phoenix:
- A Phoenix without authentication needs no key.
- The API takes no order for spans, so they are put in the order they started once read. A trace of more than 20,000 spans is refused rather than cut.
- The project is
--project, elsePHOENIX_PROJECTor its older namePHOENIX_PROJECT_NAME, as Phoenix's own client reads them, elsedefault. - It reads OpenInference's conventions, as
otelreads them. LLM spans are model calls, rebuilt exactly where the OpenAI or Anthropic instrumentation logged the request itself as JSON. TOOL spans are tool calls. - It needs Phoenix 13.9 or later, which reads spans by trace. An older one is refused, with the reason.
logfire:
- It reads among spans that started since
--since: a date, a time, or how long ago, such as90d; 30 days by default. - Logfire's instrumentations and Pydantic AI write the current GenAI conventions, read as
otelreads them. Where Pydantic AI's model span wraps the client library's own (the Anthropic SDK's), that span adds what was sent that Pydantic AI leaves out, such asmax_tokensand streaming. - A function wrapped with
@logfire.instrumentis a tool call when the model asked for a tool of its name: its span's (@logfire.instrument( 'lookup_ order')), or the function's own under the decorator's default message (Callingtools.lookup_order). It keeps its arguments, and its result withrecord_return=True. - A block an agent handed back as the Anthropic SDK returned it is read back as the block.
logfire.logs it as the text the SDK prints for it (instrument_ anthropic( ) ToolUseBlock(id= 'toolu_ …', …)).
Keys and hosts. The keys are read from the environment, and sent only to the host. Nothing of the
keys is stored. The host is --host, else the variable the store's own SDK reads, else its usual
address:
| Store | Variable | Usual address |
|---|---|---|
langfuse | LANGFUSE_HOST | https://cloud. |
langsmith | LANGSMITH_ENDPOINT | https://api. |
phoenix | PHOENIX_ENDPOINT, PHOENIX_ or PHOENIX_BASE_URL | http://localhost:6006, Phoenix as phoenix serve starts it |
logfire | LOGFIRE_BASE_URL | The Logfire region the read token names: https://logfire-us. or https://logfire-eu. |
- A self-hosted LangSmith is reached at its
/api/v1path. - Plain http is accepted for this machine (
localhost,127.0.0.0/8,::1), since the keys travel with every request.--allow-httpaccepts it for another host, on a network you trust. - A Phoenix that asks for no key, with
PHOENIX_API_KEYunset, is read over plain http on any host, such as thehttp://phoenix:6006of a compose file, since nothing secret travels.
The run's source reads langfuse:<first 8 characters of the id> (or langsmith:…, phoenix:…,
logfire:…), or the file's name for otel. The run is labelled with the source as well as
imported.
What replays from a trace. Replay matches each model call on its whole request, but for the API
version an Anthropic client names (anthropic-version), which no trace keeps:
- An imported call names none, and is matched whatever version the agent's SDK names. Its replay names none either, so a golden packed from it replays on any release of the SDK.
- A trace that kept everything the agent sends replays it with nothing live. Where it left something out, typically the tool definitions, the replay forks at the first model call that differs and runs live from there, which the import says.
- A tool call replays when the trace kept its arguments (as an object) and its output, and only if every
call of that tool was kept. Such tools are listed in
meta.coversastool:<name>; any other tool runs live on replay. - A model call that failed where it ran, or kept no messages or output, is skipped and runs live.
Two kinds of agent fork early, because what they send isn't what their traces keep. Both still import, and replay runs live from the fork with writes fenced.
- A Pydantic AI agent forks at its first model call, whatever the store. Pydantic AI logs the
conversation, not the request it sends, which writes text as content blocks and sets
tool_choice. - An agent that hands the Anthropic SDK's own response objects back as messages
(
"content": reply.content) forks at its second model call where the trace kept the response whole but those blocks only as the text the SDK prints for them, as OpenInference's Anthropic instrumentation logs them.
For that agent, the request is then rebuilt from llm.input_messages, which leaves out fields the SDK sends with those
blocks (citations, caller, toolset_name), and holds a tool result as the user's text. Where the
trace logged the blocks as JSON, or by the GenAI conventions, such an agent replays from the
trace alone, as one that builds its messages as plain dicts does. Traced by
logfire.), which logs each block as the text the SDK prints for it, it replays
from the trace alone too: that text is read back as the block.
A trace from export otel carries no prompt text, so it has nothing to replay: share the
run as a pack instead.
--json adds what the source kept:
"kept": { "traces": 1, "spans": 14, "modelCalls": 9, "modelCallsImported": 9, "withSettings": 9, "withToolDefinitions": 0, "toolCalls": 5, "toolCallsKept": 3 }The traces in a window
#shadowlap import langfuse --since 7d
shadowlap import phoenix --since 2026-10-01 --until 2026-10-08 --project support-agent
shadowlap import logfire --since 24h --trace-name support-agent --limit 20Without a trace id, import langfuse, import phoenix and import logfire import each trace that
started in a window, one run per trace, newest first:
- from
--since: a date, a time, or how long ago, such as7dor12h; 30 days ago by default; - to
--until: the same; the current time by default; - up to
--limittraces: 100 by default, at most 10,000; - only those named
--trace-name, when it is given: Langfuse's trace name, the root span's name in Phoenix and Logfire.
Give a trace id or these flags, not both. Only import logfire takes --since with a trace id too, for
how far back to look for it.
Each store is asked through its own API for the traces' roots:
- Langfuse, for the root observations in the window:
GET /api/public/ v2/ observations, or its traces list,GET /api/public/traces, where it has no v2 API; - Phoenix, for the root spans:
GET /v1/withprojects/ <project>/ spans parent_id=null; - Logfire, with one SQL query for the root spans in the
recordstable.
Each trace is then read as a single import reads it, with the same keys, hosts and flags.
A trace store's import keeps the trace's id with the run (meta.imported.key), so a trace imported
before, by a window or by its id, is skipped and not read again. A trace still running when it was
imported keeps what it had by then: import it by its id once it has ended, to bring in the rest. It
prints one line per trace, as it goes:
3 traces in Langfuse on cloud.langfuse.com that started from 2026-10-16 09:12 UTC to 2026-10-23 09:12 UTC:
11111111-41b8-4d imported as 01M5N2NX8F: 3 model calls and 1 tool result support-chat
22222222-41b8-4d imported before, as 01M5N2NVXR support-chat
33333333-41b8-4d skipped: no model calls in its 3 observations nightly-eval
Imported 1 of 3 traces: 1 imported before, 1 skipped.- Skipped: a trace with nothing to import is skipped, and so is one the store can't give whole: gone since it was listed, or more spans than an import reads.
- Imported before: once the rules on calls changed since it was judged, it is judged again from its calls, and its line ends with the verdict it was judged to (· judged again: 1 check passed).
- A failure (the store refusing, failing or not answering) stops the import at that trace. The runs it made stay, and the same command run again goes on from there.
An import of the traces in a window exits with:
| Status | Meaning |
|---|---|
0 | Otherwise, an empty window included |
1 | The window held traces and none of them is imported, in this import or before |
1 | A failure: a store that failed or didn't answer |
2 | A failure: keys the store refused |
A failure's exit status is the failure's own.
--json adds the window it read, window:
| Field | Meaning |
|---|---|
since, until | The window's start and end |
limit | The most traces it imports |
traceName | The --trace-name given |
traces lists each trace it found, newest first:
| Field | Meaning |
|---|---|
id, name | The trace, and its name |
startedAt | When it started |
outcome | imported, already (imported before, as runId), skipped (for its reason), failed (where the import stopped) or unread (after that) |
runId | The run it was imported as |
reason | Why it was skipped |
judged | The verdict it was judged to by the rules on calls: an imported trace's, or one imported before once the rules changed; null otherwise |
kept counts what the traces it imported kept, and entries every observation or span it read.
--json
#Every import subcommand takes --json and prints one object, whatever the format: each run it made,
by its full id, with its source and what it holds, then everything it skipped. One LiteLLM file can
make several runs, one per correlation id; they are all listed.
{
"format": "litellm",
"entries": 4,
"runs": [
{
"runId": "01K5CQ9C4H8M2XWQH6V3T9RB1D",
"source": "shadowlap-4121.jsonl, shadowlap-4122.jsonl",
"key": "incident-42",
"calls": { "model": 2, "tool": 0 },
"labels": ["imported"],
"imported": [{ "callNo": 1, "kind": "api", "name": "POST /v1/messages", "provider": "anthropic", "model": "claude-haiku-4-5" }],
"judged": null
}
],
"skipped": [{ "url": "line 4", "reason": "not JSON (a partly written line?)" }]
}The object has these fields:
| Field | Meaning |
|---|---|
format | The format read |
entries | What was read: HAR entries, nock definitions, cassette interactions, capture lines, spans, observations or runs |
runs | Each run made (below); empty when nothing could be imported |
skipped | Each entry skipped, and why |
Each of runs has these fields:
| Field | Meaning |
|---|---|
runId | The run, by its full id |
source | Where the run records it came from (below) |
key | The correlation id for LiteLLM, the trace id for otel and the trace stores, otherwise null |
calls | Its calls: model and tool |
labels | Its labels |
imported | The calls it imported |
judged | Its verdict by the rules on calls (above), as kept on the run; null with no rule |
source is the file's name (never its folder, which belongs to this machine and would travel with the
store), the names of the files a LiteLLM run was put together from, or langfuse:…, langsmith:…,
phoenix:… or logfire:….
When nothing could be imported, runs is empty, and the exit code is 1. Started from the dashboard, the
command also reports each run to the job that started it.
shadowlap export html
#shadowlap export html <run> # shadowlap-01m2p15fwy.html
shadowlap export html <run> run.html --lineage # with the run's ancestors, so they compare inside the file
shadowlap export html <run> --no-redact # skip the config's redaction rulesOne HTML file with the dashboard's views and the run's data built into it: calls, conversations, responses, the divergence diff, the context and timeline views, and comparison between the runs it holds.
- It opens from a file, an artifact or a pull request comment, on a machine that has neither shadowlap nor the recording, and it loads nothing from the network.
- Actions, live updates and search across runs belong to the dashboard, not to a copy.
- Like a pack, it leaves out who made each run.
- Without a file named, it writes
shadowlap-<run>.html.
export html takes these flags:
| Flag | What it does | Default |
|---|---|---|
--lineage | Include the run's ancestors, so they compare inside the file | — |
--no-redact | Skip the redaction rules in shadowlap.config.json | — |
It is a copy of recorded data, so the redaction rules in shadowlap.config.json apply, as they do to
pack. The command says how many applied, that there were none, or, with --no-redact, how many it
skipped.
A redaction rule that can't be read stops it, naming the rule. A normalizer that can't be read only
warns, ! <why>; the copy's Show diff applies no normalizers, and the copy's diff applies none.
A run still recording is refused, exit status 2: every export waits for a run to end, or the copy would
show it recording for good. With --lineage, so is a run whose ancestor is still recording.
A 30-call run comes to about 1.4 MB, most of it the dashboard's own code, styles and fonts, which every copy carries.
shadowlap export dataset
#shadowlap export dataset --label good --out good.jsonl
shadowlap export dataset <run...> --calls model --format openai-chatRecordings as an eval or fine-tuning dataset, one JSON line per example. Select runs by id, by
--label or with --all.
export dataset takes these flags:
| Flag | What it does | Default |
|---|---|---|
--label <label> | Select the runs with this label (repeatable: all of them) | — |
--all | Select every run in the store | — |
--calls final | One example per run: its last model call | Default |
--calls model | One example per model call | — |
--format jsonl | { input: { messages, tools }, output: { text, tool_calls }, metadata } | Default |
--format braintrust | { input, expected, metadata } | — |
--format langfuse | { input, expectedOutput, metadata } | — |
--format openai-chat | { messages }, the recorded answer as the last assistant message | — |
--out <file> | Write here instead of stdout | stdout |
--no-redact | Skip the redaction rules | — |
A run named that is still recording is refused, exit status 2. One picked by --label or --all is
left for a later export, and the command says how many.
- Messages are the conversation exactly as the model saw it, in a provider-neutral shape: system, user, assistant (with its tool calls and their ids) and tool results, whichever provider recorded it.
metadatacarries the run and call, the model, labels, notes, outcome check results, the command, code version and fork point.
Redaction. The redaction rules apply to each model call's request and response as pack applies
them, path rules included, before its conversation and answer are read. Then the rules without a path
apply to every string of the row, its command line, labels and notes too. A rule that can't be read
stops the export. The command says how many rules applied, that there were none, or, with
--no-redact, how many it skipped.
shadowlap export pairs
#shadowlap export pairs --out pairs.jsonl
shadowlap export pairs --min-confidence medium --format trl --out pairs.jsonlPreference pairs read from the attempts blame, verify and --repeat left in the store, one JSON line
per pair. Each pair is the conversation as the model saw it at a call, in the messages
export dataset writes, an answer to prefer and one to avoid. Training explains where
pairs come from, and what high and medium mean.
export pairs takes these flags:
| Flag | What it does | Default |
|---|---|---|
--min-confidence high | Answers blame proved decided the run | Default |
--min-confidence medium | Also where a failing and a passing attempt first answered differently | — |
--format openai-preference | { input: { messages, tools }, preferred_output, non_preferred_output } | Default |
--format trl | { prompt, chosen, rejected } as chat messages, with tools | — |
--out <file> | Write here instead of stdout | stdout |
--no-redact | Skip the redaction rules (below) | — |
It prints every source it read (including those that made no pairs, and why), the count and how many were left out, the shape, the redaction line and the file.
- What left the machine: each export is kept in the store as what left the machine.
- No pair: when no pair is at the confidence asked for, nothing is written, and a file already there is left as it was.
- Redaction: the rules apply to the copy, rules without a path to every string, and rules with a
path to the message fields kept.
--no-redactskips them.
shadowlap export har
#shadowlap export har <run> # HAR to stdout
shadowlap export har <run> traffic.har
shadowlap export har <run> --no-redact # skip the config's redaction rulesA run's model API calls as a HAR, for any tool that reads one, and the reverse of import:
exporting a run and importing it again produces the same fingerprints.
- Tool, clock and randomness calls are left out, because a HAR describes HTTP.
- Azure, Vertex and Bedrock hosts depend on the account, so a representative host for the provider stands in.
- Credentials are never written.
- Without a file named, it writes the HAR to stdout.
export har takes one flag:
| Flag | What it does | Default |
|---|---|---|
--no-redact | Skip the redaction rules in shadowlap.config.json | — |
Redaction. A HAR carries full requests and responses, so the redaction rules in
shadowlap.config.json apply, as they do to export html:
- every rule to the request bodies, and to a response that is JSON: rules with a path included, since a HAR is only read;
- the rules without a path to the URL, the headers and every response, streamed or not.
A model streams its text in pieces, so an address it writes back arrives split across events. Each run
of pieces (a block's text, a tool call's arguments) is masked as the text it adds up to, and the masked
text is spread back over the same events. The rules are named in the log (_shadowlap.redactions).
--no-redact skips them; the command says when there are none to apply, and how many it skipped.
A run still recording is refused, exit status 2, as by every export.
shadowlap export otel
#shadowlap export otel <run> # OTLP JSON to stdout
shadowlap export otel <run> --out trace.json # a file only, even with OTEL_EXPORTER_OTLP_ENDPOINT set
shadowlap export otel <run> --endpoint http://localhost:4318 # or OTEL_EXPORTER_OTLP_ENDPOINT
shadowlap export otel <run> --no-redact # skip the config's redaction rules
shadowlap export otel <run> --service refund-agent # service.name, default shadowlap-agentA run as one OpenTelemetry trace: a span for the run, and a child span per call. No OpenTelemetry SDK is involved.
- It uses the GenAI semantic conventions (
gen_ai.operation.name,gen_ai.system,gen_ai.request.model,gen_ai.usage.*,gen_ai. response. finish_ reasons,gen_ai.tool.name), plusshadowlap.*attributes for the origin, effect, fingerprint and cost. - Trace and span ids are derived from the run id, so exporting twice updates the same trace.
- Headers for an authenticated collector come from
OTEL_(EXPORTER_ OTLP_ HEADERS key=value,key2=value2).
export otel takes these flags:
| Flag | What it does | Default |
|---|---|---|
--out <file> | Write the trace to this file, and send it nowhere else unless --endpoint is given too | OTEL_ when it is set, else stdout |
--endpoint <url> | The OTLP/HTTP endpoint, such as http://localhost:4318 | OTEL_ENDPOINT, unless --out is given |
--service <name> | The service.name resource attribute | shadowlap-agent |
--no-redact | Skip the redaction rules in shadowlap.config.json | — |
With --out, the trace goes to that file and nowhere else unless --endpoint is given too. Without
either, it goes to OTEL_ when that is set, else to stdout.
A collector that doesn't answer stops it with exit status 1, and says where the trace went and why
(connection refused (ECONNREFUSED), no such host (ENOTFOUND)). A file asked for with --out is
written first.
Redaction. A trace carries no prompt text, tool arguments or results. The redaction rules without a path apply to what it does carry from the run:
- span names;
- the command line (
process.command_args), masked as one text, as every copy masks it, and before the run's span is named after its first 200 characters; - model and tool names;
- error messages.
shadowlap's own ids and fixed values stay as they are, and the command says how many rules applied. A rule that can't be read stops the export. A run still recording is refused, exit status 2, as by every export.
shadowlap rm, gc
#shadowlap rm <run...> [--forks]
shadowlap gc [--older-than <days> | --before <time>] [--keep <n>] [--expect <count>-<digest>] [--dry-run] [--json]rm deletes runs. gc deletes old runs, removes blobs nothing references any more, and vacuums the
store. The TUI's x does the same as rm for one run.
shadowlap gc --older-than 30 --dry-runrm and gc take these flags:
| Flag | What it does | Default |
|---|---|---|
--forks | rm: also delete the forks of each run, recursively | — |
--older-than <days> | gc: delete runs that started more than this many days ago, such as --older-than 30 | — |
--before <time> | gc: delete runs that started before this time: the cutoff a dry run prints (epoch milliseconds), or a date (2026-09-01) | — |
--keep <n> | gc: keep only the newest n runs, such as --keep 200 (below) | — |
--expect <count>-<digest> | gc: delete only the runs a dry run listed (below) | — |
--dry-run | gc: change nothing, and say what a real run would free | — |
--json | gc: print the report as JSON (below) | — |
What rm deletes. rm deletes runs as the dashboard deletes them, each with the attempts bundled
under it: --repeat, blame's probes, verify's attempts, a golden's repeats in one test session, an
error-path preset's attempts, and theirs in turn. So none is left behind as a run of its own. Their forks
keep their data, but lose the link.
It leaves these, with the reason, as the dashboard leaves them, and then exits 1:
- a run still recording;
- one a run in progress replays from (a replay, fork or attempt of it serves its answers from it);
- one with an attempt still running.
What gc deletes. gc deletes old runs, each with the attempts bundled under it, however recent,
as rm takes them. Runs are counted as the lists count them, so --keep 200 keeps the newest 200 runs,
each attempt counted with the run it came from. Runs still recording are never touched. One a run in
progress replays from, or with an attempt still running, is left, with the reason, as rm leaves it.
It then removes blobs nothing references any more, and vacuums the store.
While a run is recording, in this project's store and from any process, no blob is collected and
the store isn't vacuumed. A recording stores a blob just before the call that references it, and reuses a
blob already stored, so a blob that looks unreferenced may be one it is about to use. gc still deletes
the old runs, says which runs are recording, and frees their blobs on a later run. rm collects blobs
under the same rule.
Abandoned runs. A run whose recording process was killed (kill -9, out of memory, a machine that
went away) stays marked running. But it isn't recording, and it holds nothing up. The store names the
process that records each run until the run ends:
- a run whose process is gone from this machine is abandoned at once;
- one whose process can't be asked (it ran on another machine) is abandoned after a day without a call;
- one whose process is still there is abandoned after a week without one.
gc lists abandoned runs, marks them interrupted as of the last thing they recorded, and deletes them
like any other run when they match --older-than or --keep. Until then, every copy of such a run
holds it that way: pack, export html, export otel and share write it interrupted, ended at its
last call (a teammate who unpacks it never sees it recording). bisect takes it so, leaving the store
as it is.
A dry run
#--dry-run changes nothing, and says what a real run would free:
▸ runs older than 30 days: 3 of 120 · runs still recording are never touched
01M2R1HDVD record completed 41d ago node agent.js
…
blobs nothing would reference: 14 · 2.1 MB
✓ dry run: would delete 3 runs and free 2.1 MB. To delete exactly these runs: shadowlap gc --before 1757000000000 --expect 3-9c1f0e2a6b7d4c5eThe size is what those blobs take in the store. When no run matches, it says what a plain
shadowlap gc would give back instead, with no command that deletes runs: blobs nothing references, and
unused space of 1 MB or more that deleted runs left in the store.
▸ runs older than 30 days: none · runs still recording are never touched
blobs nothing would reference: 14 · 2.1 MB
unused space in the store: 51.4 MB
✓ dry run: no runs to delete; would give back 2.1 MB of blobs nothing references and 51.4 MB of unused space. To do that: shadowlap gcWhile a run is recording, gc gives none of it back, so the dry run says it can be given back once no
run is recording:
✓ dry run: no runs to delete; 51.4 MB of unused space can be given back with shadowlap gc once no run is recording.
In a project that has never recorded, gc reports nothing to clean, and creates no store.
Deleting what a dry run listed
#To delete exactly what a dry run listed, pass its cutoff and its expect back:
shadowlap gc --before <cutoff> --expect <count>-<digest>That names those runs: how many, and a digest of their ids and their attempts'. It refuses, deleting
nothing and with exit status 2, when the runs that match by the time it runs are different, even the
same number of others, or have other attempts under them. --before also takes a date (2026-09-01).
gc as JSON
#A real run is recorded in the project's activity as runs.cleaned. --json prints the same report,
with these fields:
| Field | Meaning |
|---|---|
dryRun, olderThanDays, keep | The run's options |
cutoff | Runs that started before it go, in epoch milliseconds |
total | The runs in the store before, as the lists count them |
count | How many runs it deletes, or would |
attempts | The attempts bundled under them, which go with them |
runs | The first 100 of those runs |
expect | The value to pass back to --expect |
blobs, bytes | The blobs, and the bytes, freed |
recording, recordingCount | The runs still recording: the first 100, and how many |
abandoned, abandonedCount | The abandoned runs: the first 100, and how many |
held, heldCount | The matching runs held, each with why (below): the first 100, and how many |
deferred | Whether blobs were deferred because of a recording |
store | The store (below) |
A run is held because a run in progress replays from it, or an attempt of it is still running.
store has its path, its size before, its unused space free (which vacuuming gives back), and,
after a real run, after. The lists stop at 100, so the report stays small however many runs a
clean-up deletes. The store's size counts the database's pages, whatever its write-ahead log holds, so
another process holding the store open (a dashboard) doesn't make a cleanup look like growth.
shadowlap tasks
#shadowlap tasks [--json]
shadowlap tasks score [goldens...] --model <model> [--base-url <url>] [--attempts <n>] [--reads replay]
shadowlap tasks serve [goldens...] [--port 4779] [--base-url <url> [--key-env <name>]]
shadowlap tasks models [--json]Goldens as tasks a model can be scored on. A model under test takes over at one model call, the calls before it replay from the recording for free, and the golden's checks score what it did from there.
tasks (or tasks ls) lists them: where the model takes over and why, what each rewards, which tools
run live or are stubbed after the takeover, and whether each is:
- ready;
- blocked: its test fails, its last test is inconclusive, its test may fork and forked before the
takeover (which
test --updatefixes), or it was packed from a capture and has no command; - not a task: its checks only cap tokens and cost, or its model calls are made in more than one scope;
- or has no golden.
It warns when every ready task rewards the same behaviour. The takeover, why, what a task rewards and
the agent's scope can be set in a task object next to its expect in the config.
Training explains the defaults.
tasks takes one flag:
| Flag | What it does | Default |
|---|---|---|
--json | Print them as JSON (Training) | — |
tasks score
#Scores a model on every ready task, or on the goldens given (. for every golden of the project).
Training has which model can answer which recording, every
option and the JSON.
shadowlap tasks score --model qwen3:8b --base-url http://127.0.0.1:11434/v1- Each attempt is a run. The score is the share of checks it passed, of those that applied. A
whenwhose condition never held is left out. - Scores are kept per model. Scoring a model again replaces its scores, unless the re-score stops short.
tasks score takes these flags:
| Flag | What it does | Default |
|---|---|---|
--model <model> | The model to score (required; below) | — |
--base-url <url> | An OpenAI-compatible endpoint, such as http://127.0.0.1:8000/v1 (no user or password in it) | — |
--key-env <name> | The variable holding the endpoint's key (below) | OPENAI_API_KEY for api.openai.com; ANTHROPIC_API_KEY for anthropic: on a recording that isn't Anthropic's |
--attempts <n> | Attempts per task | 5 |
--reads live|replay | What read tools do after the takeover | live |
--concurrency <n>, --budget <usd>, --timeout <seconds>, --guard | As for blame | — |
-y, --yes | Run the recorded command of a golden from a pack the repository doesn't track, without asking (below) | — |
--json | Print the scores as JSON on stdout | — |
Which model it asks. --model <name> asks the recording's provider for another model.
anthropic:<name> and openai:<name> ask that provider's API, and --base-url an OpenAI-compatible
server, such as Ollama or vLLM. When that isn't the API the recording spoke, the conversation is
translated at the takeover, and the answer back.
Either way, a small upstream on 127.0.0.1 asks the model under test from the takeover call on, whatever the agent asked before it. It answers in the recording's own API: Anthropic's, OpenAI's (chat completions or the Responses API), Gemini's, Amazon Bedrock's (Claude, and the Converse API) or Vertex AI's and Azure OpenAI's. Another model of Bedrock, Vertex AI or Azure OpenAI themselves can't be asked: score Anthropic's or OpenAI's API, or a model at your own endpoint, on those recordings.
--key-env: another model of the recording's own provider goes with the agent's own key.--yes: without it, such a golden asks first, astestasks of one, and without a terminal it is skipped, saying why.
tasks score exits with:
| Status | Meaning |
|---|---|
0 | Every task was scored in full |
1 | A task was skipped or stopped |
2 | Nothing given can be scored |
3 | The budget stopped it |
tasks serve
#Answers one attempt per HTTP request, for a training loop that wants a reward. Training shows a request and its response.
POST /v1/ with {"model": …, "baseUrl": …} returns each check's result, the
score, the run and its cost. It listens on 127.0.0.1, checks the Host header, and needs the token it
prints. SHADOWLAP_TASKS_TOKEN sets it, at least 16 characters.
tasks serve takes these flags:
| Flag | What it does | Default |
|---|---|---|
--port <port> | The port to listen on | 4779, or a free one if it is taken |
--base-url <url> | The endpoint requests that name none ask | — |
--key-env <name> | The variable whose key goes to that endpoint, and nowhere else | — |
--concurrency <n> | Attempts running at once; more requests wait their turn | 2 |
--reads live|replay | What read tools do after the takeover, unless a request says | live |
--timeout <seconds> | Stop an attempt after this long, and count it as failed | — |
--guard <mode> | The network guard: warn, block or off | warn |
-y, --yes | Serve goldens from packs the repository doesn't track without asking (below) | — |
A golden from a pack the repository doesn't track, recorded elsewhere, is served only once you say yes
to its command as it starts, or with -y/--yes. A request for one nobody said yes to gets a 409.
tasks models
#Lists models to score:
- the one the recordings used;
- each configured provider's models: Anthropic's and OpenAI's with the keys in the environment, and Ollama's on this machine;
- the models scored before.
Each comes with its price per attempt, last score and the ready tasks it can answer. A model that can answer none says why.
tasks models takes these flags:
| Flag | What it does | Default |
|---|---|---|
--timeout <ms> | How long to wait for each provider's list; a provider that doesn't answer within it becomes a note | 1500 ms |
--json | Print them as JSON | — |
shadowlap push, pull
#shadowlap push <run...> # these runs, with their ancestors
shadowlap push --all # every run on this machine that isn't in the store as it is
shadowlap pull [<run...>] # teammates' runs, or only these (ids or unique prefixes, as in the store)
shadowlap pull --include-pr # and the runs pull requests' CI jobs pushed to their own foldersCopy runs between this machine and the team store named by store in
shadowlap.config.json, with this machine's own credentials.
push and pull take these flags:
| Flag | What it does | Default |
|---|---|---|
--all | push: every run on this machine that is not in the store as it is | — |
--include-pr | pull: also bring the runs pull requests' CI jobs pushed to their own folders (below) | — |
--json | Print the result as JSON (below) | — |
What goes and comes.
- Blobs shared between runs go up once.
- A run still recording goes when it ends.
- Labels and notes merge both ways, and a run's results come from the machine that made it.
- A run deleted here isn't pulled again, unless named.
- A run named to
pushthat the store has already is checked part by part, and what the bucket lost of it (its calls, a blob) is sent again. - A push names the pusher as the author of a run made here without one. It leaves here, saying why, a run a pull would refuse.
Counting. Both count by run, as the dashboard does: "45 runs and 69 attempts pushed". Attempts
(--repeat, blame's probes, verify's attempts, a golden's in one test session) are those filed under
their runs. A run counts as only on this machine while it, or an attempt filed under it, isn't in the
store.
Fixes go with their runs. A push sends each fix with its blames, repairs and verdicts, once one of its runs is in the store. A pull brings the team's, merged with this machine's.
- A teammate's repair, from a pull or from the store's copy a push merges with, has its
fix.patchand reports written to.shadowlap/repairs/<id>/. - Whether a fix is applied is this machine's own, and never goes.
- A fix over 16 MB stays here, and a pull skips one in the store that is, saying which.
What a pull checks. pull checks every run it brings as strictly as unpack. A run in the store
that fails is skipped with its forks, and tried again next time. A pulled run is marked with the store's
URL (meta.imported.from). It reads as its author's recording, with what it spent, and replaying it
asks before running its command, as an unpacked run's does (--yes says yes).
In a pull request's CI job. In a pull request's CI job (GitHub Actions
pull_request events, GitLab merge request pipelines, or SHADOWLAP_PR set), push writes only to the
pull request's own folder, pr/<number>/ in the store. pull leaves those folders out unless given
--include-pr. Runs it brings from one stay on this machine: push --all leaves them out, and push
refuses one by name.
push and pull exit with:
| Status | Meaning |
|---|---|
1 | A run it was named can't go or come, or the store fails part way |
2 | No store is set, or this machine's credentials are missing or refused |
push --json prints
{ store, pullRequest, pushed, updated, unchanged, queued, repaired, refused, blobs, segment, onlyLocal, fixes }:
| Field | Meaning |
|---|---|
store | The store |
pullRequest | The pull request whose folder it wrote to, or null |
pushed | The runs pushed |
updated | The runs whose labels, notes or results were merged |
unchanged | The runs unchanged |
queued | The runs still recording |
repaired | The parts the bucket lost, sent again |
refused | Each run refused, with why |
blobs | The blobs uploaded, already present, and their bytes |
segment | The index segment it wrote, or null |
onlyLocal | How many runs are still only on this machine |
fixes | The fixes that went with the runs, none from a pull request's job (below) |
A push's fixes has the ids sent, those that failed (fixId, reason), and the teammates'
repairs the store's copies brought.
pull --json prints
{ store, pulled, merged, needPush, skippedDeleted, missing, unknown, failed, skippedSegments, blobs, segments, inStore, byAuthor, onlyLocal, pullRequests, fixes }:
| Field | Meaning |
|---|---|
store | The store |
pulled, merged | The runs pulled, and merged |
needPush | The merged ones whose changes the next push sends |
skippedDeleted | Runs deleted here, and not pulled again |
missing | Runs gone from the store |
unknown | Names that match no run |
failed | Runs that couldn't come, with why |
skippedSegments | Index segments that couldn't be read |
blobs | The blobs downloaded, and their bytes |
segments | The index segments read |
inStore, onlyLocal | The runs in the store, and those only here |
byAuthor | The pulled runs, by author |
pullRequests | With --include-pr, each pull request's folder |
fixes | The fixes that came with the runs (below) |
A pull's fixes has the ids this machine didn't have (pulled), those here the store's copy added to
(merged), those in the store that couldn't be read (failed: fixId, reason), and the repairs
whose files it wrote.
shadowlap share
#shadowlap share <run> [--lineage] [--json]Writes the run's HTML report, with every redaction rule applied, to the team store, and prints a presigned link that expires after 7 days. Anyone with the link can open it, without an account.
share takes these flags:
| Flag | What it does | Default |
|---|---|---|
--lineage | Include the run's ancestors | — |
--json | Print the link as JSON (below) | — |
A run still recording, or with --lineage one whose ancestor is, is shared once it has ended. Until then
share refuses it, exit status 2, before anything is asked of the store.
A redaction rule that can't be read stops it, and a normalizer that can't be read warns, as for
export html.
--json prints { runId, url, expiresAt, note, store, key, bytes, redacted, rules, runs }:
| Field | Meaning |
|---|---|
runId | The run |
url | The link |
expiresAt | When it expires, in epoch milliseconds |
note | When it may stop working sooner: it was signed with temporary credentials |
store, key | The store, and the object's key in it |
bytes | Its size |
redacted, rules | Whether it is redacted, and with how many rules |
runs | The runs it holds |
shadowlap store
#shadowlap store set <url> [--provider s3|gcs|azure|r2] # write "store" to shadowlap.config.json
shadowlap store set <url> --check # check access first; write it only if that works
shadowlap store check # list, write, read and delete; encryption, versioning, object lock
shadowlap store show # last push and pull, runs in the store and only here, last check (offline)
shadowlap store policy [--provider s3|gcs|azure|r2] # the access to ask your cloud admin for: two policies
shadowlap store ci --provider github|gitlab|other [--packs <glob>] # a CI step that runs test --push
shadowlap store push-from-ci on|off # whether CI pushes its test runs and fixesSet up the team store. Each subcommand takes --json. It has these subcommands:
| Subcommand | What it does |
|---|---|
set | Write the store's URL to the config, after checking it against the provider's scheme |
check | Prove this machine's credentials can list, write, read and delete a test object |
show | The last push and pull, the runs in the store and only here, and the last check, offline |
policy | Print the two policies to ask your cloud admin for |
ci | Print a CI step that runs test --push |
push-from-ci | Whether CI pushes its test runs and fixes |
They take these flags:
| Flag | What it does | Default |
|---|---|---|
--provider <provider> | set: check the URL against this provider's scheme only. policy: the provider. One of s3, gcs, azure, r2 | policy: the store's |
--check | set: check access first, as check does, and write the store only when it works | — |
--provider <ci> | ci: github, gitlab or other | — |
--packs <glob> | ci: the goldens to test | goldens/* |
--json | Print the result as JSON (below) | — |
store set
#Checks the URL against the provider's scheme (with --provider, against that one only), and writes it
to the config, keeping everything else.
With --check, it first checks the store as check does, and writes nothing when that fails. It then
says what to fix, and to run the same store set … --check again (in the dashboard's store setup,
Check access and connect again).
With --json, set prints { store, url, provider, bucket, prefix, account, region, previous, config, set }:
| Field | Meaning |
|---|---|
store | The store, as written to the config |
url | Its URL, without the options after ? |
provider, bucket, prefix, account, region | Its parts |
previous | The store it replaced |
config | The config file's path |
set | True once the config is written |
check | With --check: the check, as check --json prints it |
When the check fails, it prints only { store, url, provider, set, check }, with set false. It writes
nothing, and exits 1.
store check
#Proves this machine's credentials can list, write, read and delete a test object, and reports encryption, versioning and object lock, with a warning when versioning or object lock is off. It exits 1 when access fails.
check prints
{ url, credentials, steps, ok, encryption, versioning, objectLock, warnings, notes, failure, pullRequest }:
| Field | Meaning |
|---|---|
url | The store's URL |
credentials | Where the credentials came from |
steps | Each step: name, ok, detail, code |
ok | Whether all passed |
encryption, versioning, objectLock | The bucket's, each with state and detail |
warnings | Its warnings |
notes | What couldn't be checked |
failure | The kind of the first failure |
pullRequest | The pull request whose folder it checked |
store show
#Shows the last push and pull, the runs in the store and only here, and the last check, offline.
show prints
{ url, store, provider, providerName, bucket, prefix, region, pushedAt, pulledAt, inStore, onlyLocal, localAttempts, queued, check }:
| Field | Meaning |
|---|---|
url, store, provider, providerName, bucket, prefix, region | The store, and its parts |
pushedAt, pulledAt | The last push and pull, in epoch milliseconds |
inStore | The runs in the store |
onlyLocal | The runs only here: a run counts while it, or an attempt filed under it, is only here |
localAttempts | The attempts only here |
queued | The runs still recording, to go when they end (below) |
check | The last check: ok, at, encryption, versioning, objectLock, credentials |
A queued run that ended since counts as only here until the next push sends it, and a queued run that was deleted is forgotten.
store policy
#Prints two policies scoped to the store's folder, each an IAM policy, role bindings, a role assignment or R2 token scopes:
- one for people and CI on protected branches;
- one for pull requests' CI jobs, which may write only under
pr/.
policy prints { provider, writer, pullRequest, notes }: each policy with whom it is for, its
kind, its text and notes.
store ci
#Prints a step for GitHub Actions, GitLab CI or any other CI.
- Credentials: short-lived ones (OIDC) where the CI and the cloud allow it, and the pull-request credentials beside the protected branches'. On GitHub, that is for pull requests from the repository itself: a fork's gets no OIDC token, so its job skips them and runs the tests without pushing.
- The tests run with
npx shadowlap@<version>, this CLI's own release. On GitHub, Node 22 and a line for the agent's dependencies (npm ci && npm run build, to make your project's) come before them. On GitLab, that line opens each job's script. - Keys: the goldens it tests that run live (marked
fork, or with an injection gate) need the keys of the providers their recordings call. On GitHub, the step'senvtakes them from the repository's secrets, by name. On GitLab and any other CI, a note names them. - Keys it can't name: a comment says to add your agent's own, where a key can't be named: a pack that can't be read, or Vertex AI, which signs in with a token.
ci prints { ci, provider, file, text, oidc, pullRequests, notes }:
| Field | Meaning |
|---|---|
ci, provider | The CI, and the cloud provider |
file, text | The file the step goes in, and the step's text |
oidc | Whether it uses short-lived credentials |
pullRequests | Whether pull requests' jobs push to their own-folder, or not at all (no-push) |
notes | What to set up once, on the cloud side |
store push-from-ci
#push-from-ci off writes "ciPush": false to the config. test --push in CI then runs the tests and
keeps their runs in the job (outside CI, --push still pushes), and the GitHub Action keeps the fixes
it opens there too. on turns it back on.
push-from-ci prints { ciPush, previous, config }.
The configuration file
#shadowlap.config.json, in the directory you run shadowlap from. Every key is optional, and
shadowlap init writes one with a $schema line already in it:
{
"$schema": "https://raw.githubusercontent.com/devjoinedthechat/shadowlap/main/shadowlap.config.schema.json",
"clock": "auto",
"normalize": [{ "kind": "api", "pattern": "uuid", "replace": "<uuid>" }],
"redact": [{ "pattern": "email", "replace": "<email>" }],
"http": { "*.pinecone.io": "read" },
"mcp": { "effects": { "write_file": "write" } },
"prices": { "my-model": { "input": 1.5, "output": 7.5 } },
"tests": { "goldens/*": { "expect": { "tools": { "send_email": 0 } } } }
}Its keys are these:
| Key | What it does |
|---|---|
normalize | Masks values before fingerprinting, so an id the agent makes on every run does not fork a replay (details) |
redact | Masks stored content in every export, pack and share link of a run (below) |
clock | auto (default) matches dates near each run's start across days; exact requires equality |
http | Hosts called straight from a client library, recorded as tool calls with an effect (details) |
mcp.effects | Effects by MCP tool name, overriding the server's annotations (details) |
tools | trust by tool name, and unrecorded (below) |
prices | USD per million tokens by model prefix, for cost estimates, with a second card for a long prompt (details) |
tests | Outcome checks for shadowlap test, by recording directory, and each golden's gates, task and presets (below) |
repair | repairer (provider:model), baseUrl and timeout for the model shadowlap repair asks for fixes (below) |
build | The command that builds the agent, such as npm run build, run where a command takes --build and none is given (below) |
store | The team store, a bucket you own (s3://, gs://, az://, r2://), written by shadowlap store set. Never credentials |
ciPush | false keeps test --push in CI, and the GitHub Action, from pushing (shadowlap store push-from-ci on|off; below) |
guard | accept: ["claude-code"]: take the Claude Code CLI as it is under --guard block (below) |
More on some of them:
redact: where the rules apply. A redacted pack still replays, and a push copies runs to the team store as recorded.tools:trustby tool name (or a host underhttp): set it tountrustedfor a read that brings outside text in, whereshadowlap injectionplants payloads. It is recorded on each call of the tool as it is made. Read when a run is tested, it wins over every recorded mark: the SDK's, an MCP server's, an earlier config's.tools.unrecordedmaps the tools the agent runs without shadowlap recording them toreadorwrite. That decides whether every command that replays a run may run them again (details).repair.timeout: the seconds the repairer may take to answer, default 600.build: one line, run in a shell, so a compiled agent never replays an old build. Each command says so as it runs it;--buildoverrides it, and--build ""runs none:Command Where it runs repairEach candidate's worktree and its baseline's, and the working tree after --applyfix apply,migrate applyThe working tree, once the patch is in verifyThe checkout of the code before the change, for a paired baseline at a commit bisectEach commit's worktree, before its replay In
verify's andbisect's checkouts, as with--build, nothing is linked from your working tree: a build there installs what it needs, such asnpm ci && npm run build. Settings → Project sets it in the dashboard, andshadowlap initsuggests thebuildscriptpackage.jsonhas.ciPush:falsekeepstest --pushfrom pushing to the team store in CI, and the GitHub Action from pushing the fixes it opens.guard: what the Claude Code CLI the Claude Agent SDK runs can reach that the guard can't watch is named, and fails nothing, while every Node and Python process keeps block.SHADOWLAP_and the GitHub Action'sGUARD_ ACCEPT= claude-code guard-acceptinput add to it. Any other value is refused, exit status 2, before anything runs.
A golden's entry under tests holds:
- its outcome checks (details):
calls,whenandjudgein Checks, and judge checks; - a
gatethat decides on a forked golden's attempts with stated error rates (gating); - an
injectiongate (details); - a
taskobject, with where a model under test takes over, why, what the task rewards and the agent's scope (see Training); - a
chaosobject, with the golden's error-path presets, orfalseto leave it out of them.
The schema is
shadowlap.
in the repository. It gives completion in any editor that reads $schema, and it lets CI check a config
without running shadowlap.
It is stricter than the parser on purpose. The parser ignores a key it doesn't know, so that a config
written for a newer version still works, while the schema rejects it. That is what turns "normalise"
from a rule that silently never applies into an error you see as you type.
shadowlap doctor checks the file too, and reports every rule, price and effect it parsed.
Environment
#These environment variables change what shadowlap does:
| Variable | What it does | Default |
|---|---|---|
SHADOWLAP_ANTHROPIC, _OPENAI, _GEMINI, _AZURE, _VERTEX, _BEDROCK | Override a provider's upstream: a mock, or an OpenAI-compatible server such as Ollama | ANTHROPIC_BASE_URL, OPENAI_BASE_URL and GOOGLE_GEMINI_BASE_URL, when they are set |
SHADOWLAP_ | Accept the Claude Code CLI's unwatched connections under --guard block, as guard.accept in the config does (below) | — |
SHADOWLAP_CAPTURE=off | Don't capture model calls inside Node and Python agents; only the base-URL variables apply (details) | — |
SHADOWLAP_TELEMETRY=0, DO_NOT_TRACK=1 | Disable telemetry | — |
SHADOWLAP_TELEMETRY_URL | Send telemetry to a collector you run | Nothing is sent |
SHADOWLAP_AUTHOR | Who made the runs and activity entries a command records, as the dashboard shows them | ci in CI, else git's user.email, else the OS user |
SHADOWLAP_PR | A pull request's number, set in its CI build (below) | — |
SHADOWLAP_HOME | Where this machine's files go, as an absolute path: the dashboard's project list, and the registrations of running agents and mcp --listen proxies | ~/.shadowlap |
SHADOWLAP_PROJECT_ROOTS | Folders besides your home folder that the dashboard may suggest and add projects from, separated like PATH | — |
SHADOWLAP_TASKS_TOKEN | The token tasks serve requires, at least 16 characters | A random one, printed |
SHADOWLAP_ | For serve --recordings, in a recording's environment: what --allow-unrecorded <names> says (serve) | — |
SHADOWLAP_DEBUG=1 | Print the stack of any error, for a bug report (exit status) | — |
OLLAMA_HOST | Where tasks models lists Ollama's installed models | 127.0.0.1:11434 |
HTTPS_PROXY, HTTP_PROXY, NO_PROXY | Reach the team store and the clouds' sign-in services through a proxy (details; below) | — |
NO_COLOR | Plain output | — |
SHADOWLAP_UI_THEME | dark, light or ansi: the terminal UI's palette | The one picked from what the terminal reports |
SHADOWLAP_GUARD_ACCEPT: any other value ends every command that takes--guardwith 2 as it starts, before anything runs.SHADOWLAP_PRis for where shadowlap doesn't recognise pull requests by itself; it does on GitHub Actions and GitLab CI. The build then writes only to that pull request's folder in the team store.SHADOWLAP_HOME: a relative path is ignored, and~/.shadowlapholds all of these files instead, as the dashboard says. It holds your dashboard Spending too, insettings.json. Processes that find each other's files there need the same one, so give it toshadowlap mcpin your MCP client's configuration too (details).- The proxy variables: the agent never reaches the sidecar through one.
run,replay,testandserve --recordingsadd127.0.0.1,localhostand::1toNO_PROXYandno_proxy.
See also
#- Quickstart: a first recording, step by step.
- How it works: recording, replaying and forking.
- CI: goldens, outcome checks and test history.
- Interventions: what each flag that changes a replay does.
- Blame, Verify and Repair: finding and fixing a failure.
- Team store: sharing runs with your team.