CLI reference (tinspec)

The headless runner is a separate download from the desktop app and versions separately. This page describes tinspec v0.1.1 — macOS only, one universal binary. Install steps and a worked pipeline: Run a suite in CI.

The binary is called tinspec, not tinspec-cli.

tinspec <command> [args] [global flags]

Commands

Command	What it does
tinspec run chain <name>	Validate, resolve, pre-flight, run and report one chain. <name> is the chain’s name:, not its file stem.
tinspec run request <collection> <name>	Send one saved request and evaluate its own tests, one case per assertion. <collection> is the path relative to the workspace, or just the file name.
tinspec lint [chain]	Check the workspace — or one chain — without sending anything.
tinspec ls <target>	List what the workspace declares. Targets: collections, chains, requests, providers, environments, senders.
tinspec secrets list	Every secretRef the workspace declares, its environment variable, whether a value is available, and what declared it. Never a value.
tinspec snapshot spec <provider> -o <file>	Resolve an API provider’s description and write it as a committable JSON baseline.
tinspec check contract <provider> --baseline <file>	Resolve the live description and diff it against that baseline.

lint, ls and secrets list are offline by construction — they never open a socket, so they work as a pre-commit hook and inside a build container with no egress.

run chain

Flag	Meaning
--target <NODE_ID>	Stop after this node.
--data <PATH>	.csv, .tsv, .json or .jsonl: one run per row, each column binding a declared input node. A column naming no declared input is a hard error before the first request.
--fail-fast	Stop at the first failing row instead of running the whole matrix.

Rows run sequentially, each starting from the base environment; variables never bleed from one row to the next. Each row becomes its own suite, named chain [row 2], or chain [row 2: acme] when the file has a name or id column to label it with. An absent cell falls back to the input’s own default, a required input must come from a column or a --var, and --var beats a column beats the default.

check contract

Flag	Meaning
--baseline <PATH>	The snapshot written by tinspec snapshot spec. Required.
--fail-on breaking|any|never	What counts as a failure. Default breaking.
--allow <RULE>	Allow a change even when breaking, by its stable rule id. Repeatable.

Rule ids are the machine-readable half of the classifier, so an allow-list never string-matches prose: operationRemoved, operationAdded, operationDeprecated, shapeChanged, requiredInputAdded, inputAdded, inputRemoved, inputBecameRequired, inputBecameOptional, inputBecameNullable, inputBecameNonNullable, outputAdded, outputRemoved, outputBecameRequired, outputBecameOptional, outputBecameNullable, outputBecameNonNullable, typeChanged, typeNarrowed, typeWidened, fieldDeprecated.

snapshot spec and check contract are OpenAPI-only. A provider of any other kind is refused by name, with the reason, rather than failing obscurely.

Global flags

Every one of these is accepted on any command.

Flag	Meaning
--workspace <DIR>	The project folder. Default: the nearest ancestor containing .tinspec/, found the way git finds a repository.
--env <NAME>	Environment by name. Default: the first one any collection declares. Only collection-file environments are visible.
--var <K=V>	Variable override. Repeatable, and the highest precedence — it beats a --data column, a pre-run chain’s output and the environment.
--sender <ID>	Authenticate as this sender for every request that resolves to a provider and names no sender of its own. An unknown id is refused at startup.
--secrets-file <PATH>	A .env-style file of secretRef=value lines. Refused if it lives inside the workspace.
--secrets-command <CMD>	A command run once with the wanted reference names on stdin, returning NAME=value lines.
--reporter pretty|junit|json	Output format. Repeatable.
--out <PATH>	Where to write each reporter, in the same order. - means stdout.
--timeout <MS>	Per-request timeout floor; a request’s own settings wins.
--insecure	Skip TLS certificate verification.
--no-strict-variables	Allow an unresolved {{var}} to go out as a literal.
--no-color	Never colour the output. Also honoured: a non-tty stdout, and NO_COLOR.

Strict variables are on by default here, unlike the engine’s own default. On a laptop the request is in front of you; in CI an unresolved placeholder comes back as an opaque 400 that nobody can diagnose from a log.

Exit codes

The contract a pipeline is written against.

Code	Meaning
0	Everything passed.
1	A test or assertion failed. The only “your API is wrong” code.
2	Usage: a bad flag, an unknown chain, sender or environment, a data column naming no declared input, two secret references colliding on one environment variable.
3	Workspace: unparseable YAML, a dangling chain reference, a chain that does not validate. The repo is broken, not the API.
4	Transport or engine: connect failed, DNS, TLS, an unresolved variable. The “infrastructure, retry me” code.
5	Secret or auth: a missing secretRef, a failed token dance.
6	Contract: a breaking change against the committed baseline. check contract only.
7	The run exceeded its own timeout. Partial results were still reported.
130	Interrupted. A chain’s teardown region does not run.

Two rules hold the table together. Exit 1 is reserved for a failed expectation and nothing else — a runner that returns 1 for both a wrong response and an unreachable host makes || retry impossible to write correctly. And a failed expectation beats an error when both happen in one run: an API that is demonstrably wrong must not be reported as “retry me”, or the regression hides behind the retry.

A pruned chain branch is skipped, never a failure.

Secret resolution

A secretRef in a committed file never carries a value. In CI the value comes from the first of these that has it:

1. --secrets-file — keys are the reference names verbatim, not the mangled environment spelling. Never auto-discovered; refused inside the workspace root.
2. --secrets-command — the wanted names arrive on stdin, one per line; NAME=value comes back on stdout. Non-zero exit is fatal, and the command’s stdout is never echoed into an error.
3. The environment — TINSPEC_SECRET_<NAME>, where <NAME> is the reference uppercased with every character outside A–Z0–9 replaced by _. provider.CLERK_SECRET_KEY reads TINSPEC_SECRET_PROVIDER_CLERK_SECRET_KEY.

The mangling is lossy, so two distinct references can want the same variable. That is refused at startup rather than resolved — picking one silently would authenticate the run as the wrong principal. tinspec secrets list diagnoses it, and prints the variable name for every reference so you never have to work the mangling out by hand.

Missing secrets stop the run before the first request, with exit 5.

Variable precedence

Lowest to highest:

collection defaults  <  environment  <  path variables  <  pre-run chain output  <  --var

Inside run chain --data, a row’s columns sit between the environment and --var. A --var that names a declared input rewrites that input node too, not just the scope — otherwise the node would run and overwrite the row.

Reporters and report shape

No --reporter at all means pretty on a terminal and json when stdout is not one.

--out takes one path per reporter in the same order, or none at all; a partial pairing is refused rather than silently pairing the file with the wrong reporter.

JUnit is the Surefire dialect. Every case carries name, classname and time — GitLab builds a test’s identity from the first two, and a case missing either is one it cannot address. A chain’s contract findings, when a report has them, arrive as their own contract suite: a breaking change is a <failure>, a compatible one a passing case that still names its rule.

<testsuites name="checkout-smoke" tests="4" failures="1" errors="0" skipped="0" time="0.480">
  <testsuite name="checkout-smoke [row 1]" tests="4" failures="1" errors="0" skipped="0" time="0.480">
    <testcase name="login" classname="checkout-smoke [row 1]" time="0.120">
      <properties>
        <property name="tinspec.httpStatus" value="200"/>
        <property name="tinspec.kind" value="request"/>
      </properties>
    </testcase>
    <testcase name="check · body.id" classname="checkout-smoke [row 1]" time="0.000">
      <failure message="body.id: expected &quot;7&quot;, got nothing" type="assertion">body.id
expected: 7
actual:   (not present)</failure>
    </testcase>
  </testsuite>
</testsuites>

JSON carries a reportVersion from day one, because a CI script written against it is a contract the moment it exists. Version 1 today; it is bumped for any change a reader could trip over, never for a purely additive one.

{
  "reportVersion": 1,
  "name": "checkout-smoke",
  "startedAtUnixMs": 1757714400000,
  "durationMs": 480,
  "suites": [
    {
      "name": "checkout-smoke [row 1]",
      "durationMs": 480,
      "cases": [
        { "name": "login", "classname": "checkout-smoke [row 1]", "durationMs": 120,
          "outcome": { "kind": "passed" } },
        { "name": "check · body.id", "classname": "checkout-smoke [row 1]", "durationMs": 0,
          "outcome": { "kind": "failed", "message": "body.id: expected \"7\", got nothing",
                       "detail": "body.id\nexpected: 7\nactual:   (not present)" } }
      ]
    }
  ],
  "specs": [],
  "drift": []
}

failed and errored are kept apart deliberately: failed is an expectation that did not hold, errored is a check that could not be performed. Collapsing them would throw away the one bit that says whether the API is wrong or the test is.

Masking

No request header, no request body and no credential ever enters a report — not as a render-time filter, but because the builder never collects them. A failing case attaches its response body, truncated to a few kilobytes and with every resolved secret value replaced by <redacted>. A passing case attaches nothing.

What the runner reads

The same committed files the app writes — see Project layout. Collections, .tinspec/chains/, .tinspec/auth/, .tinspec/providers/ and .tinspec/route-matchers/ are all read; tinspec.db is not. Nothing is written back: the only files that appear are the ones you asked for — a baseline from snapshot spec -o, and whatever --out points at.

A file anywhere in the workspace that does not parse stops the command with exit 3, naming the file — before anything is resolved, so a chain is never blamed for a collection’s typo.

Not in v0.1.1

Named rather than left to be discovered:

• run pipeline and run file. A pipeline run stops at the first failure and returns nothing about it, so a runner built on it would always report an empty failure.
• tinspec auth login and auth token. The first is real work; the second prints a live credential into a CI log.
• Any parallelism. --data rows, chain branches and split all run sequentially. There is no --data-concurrency.
• --offline, TAP output, --json-errors, -q/-v.
• A spec binding on run chain. It reports no contract drift; that is what the separate check contract job is for. An op: schema assertion consequently fails with “no declared response schema in scope” rather than passing vacuously.
• gRPC-reflection and GraphQL contract baselines. snapshot spec is OpenAPI-only.
• Collection- and provider-level send settings inside a chain. The chain runner does not take them, so a provider-configured proxy works in the editor and fails in a chain. run request does apply them.
• A machine-readable report from a streaming request. --reporter junit|json is refused for one, because the streaming path evaluates no request-level tests and the file would claim a clean run. WebSocket is refused outright: no single response to report on.
• Teardown after an interrupt. Ctrl-C abandons the in-flight request and exits 130; a killed job leaves its fixtures behind.
• Linux and Windows binaries. macOS only at this version.