Run a suite in CI

tinspec is a second program: the headless runner. It reads the same project folder the desktop app writes — collections, chains, auth providers, environments — and sends through the same engine. The CLI at v0.1.1 is built on the same engine release the desktop’s v0.1.0-preview.10 ships, so a request that works in the app behaves the same in a pipeline.

It does one job: run something you already saved, report on it, and exit with a code CI can gate on. It authors nothing in your project — the only file it writes is a contract baseline you explicitly ask for — and it never writes a secret anywhere.

Full flag-by-flag surface: CLI reference.

Install

macOS only at v0.1.1. There is no Linux binary and no Windows binary: the Homebrew formula has a macOS branch and nothing else, and the download page lists one file. A CI job therefore needs a macOS runner today.

Homebrew — two commands, because brew tap user/repo resolves to GitHub and this tap is on GitLab, so the URL has to be given the first time:

brew tap tinspec/tap https://gitlab.com/tinspec/homebrew-tap.git
brew install tinspec/tap/tinspec

Homebrew 6 may answer Error: Cannot tap tinspec/tap: invalid syntax in tap!. Nothing is wrong with the formula — Homebrew 6 refuses a tap it has not been told to trust and reports the refusal as a syntax error, which sends you looking in the wrong place. Trust the formula, then repeat the two commands above:

brew trust --formula tinspec/tap/tinspec

--formula is not decoration: brew trust tinspec/tap prints Trusted tap and the tap still fails. Older Homebrew has no brew trust and needs none.

Or take the tarball, which is what a CI image with no Homebrew wants:

curl -LO https://tinspec.dev/releases/tinspec_0.1.1_darwin_universal.tar.gz
tar -xzf tinspec_0.1.1_darwin_universal.tar.gz
sudo mv tinspec /usr/local/bin/

The file list, sizes and SHA-256 checksums live on tinspec.dev/download, so this page does not repeat them — one copy cannot go stale.

The macOS build is a single universal binary, Developer ID–signed and notarized. A notarization ticket cannot be stapled to a bare executable, so a tarball downloaded in a browser pays one online Gatekeeper check the first time it runs; a brew install or curl download is not quarantined and skips that entirely.

Check it landed:

tinspec --version
tinspec ls chains

What it runs

The runner finds your project the way git finds a repository: the nearest ancestor directory containing .tinspec/, starting from the working directory. So a CI job’s working directory does not have to be exact. --workspace DIR names it explicitly.

Two things can be run:

tinspec run chain <name>                    # the chain's `name:`, not its file stem
tinspec run request <collection> <name>     # one send

run request evaluates the request’s own tests as one case per assertion, which is what makes a CI UI say “status is 200” passed, “body.id exists” failed rather than just naming the request. If the request has a pre-run chain, it runs first, and a pre-run failure aborts the send instead of turning into a confusing 401 further down.

Nothing is sent until the run is known to be sendable. In order, before the first byte: every file in the workspace must load, the chain must validate, --sender must name a sender that exists, no two secret references may collide, and every secret the run will reach for must have a value. Each of those has its own exit code.

Secrets, on a machine with no keychain

On your laptop a secretRef resolves through the OS keychain. A CI runner has no keychain, no app-data directory, and nobody to approve a prompt — so the value comes from one of three places, in this precedence:

1. --secrets-file <path> — a .env-style file, one NAME=value per line, # comments, an optional pair of quotes unwrapped. The keys are the reference names verbatim, not the mangled environment spelling. It is never auto-discovered, and it is refused outright if it lives inside the workspace — a secrets file that gets picked up implicitly is a secrets file that gets committed.
2. --secrets-command "<cmd>" — the vault escape hatch. The command runs once with the wanted reference names on stdin, one per line, and returns NAME=value lines on stdout. A non-zero exit is fatal. Its stdout is never echoed, on success or failure, because the natural error message would be a list of credentials heading into a build log.
3. Environment variables, which need no flag and which every CI system can do. The name is TINSPEC_SECRET_ followed by the reference uppercased with every character outside A–Z0–9 replaced by _ — so provider.CLERK_SECRET_KEY reads TINSPEC_SECRET_PROVIDER_CLERK_SECRET_KEY.

You do not have to work the mangling out by hand:

tinspec secrets list

prints every reference the workspace declares, the environment variable it reads, whether a value is currently available, and what declared it — never a value.

That mangling is lossy: a.b, a-b and a_b all become A_B. If two distinct references in one workspace would read the same variable, the run refuses to start rather than picking one. Silently picking is an authorization bug — the suite would authenticate as somebody, just not necessarily the somebody the file named. Rename one reference, or supply both through --secrets-file, whose keys have no such restriction.

A secret with no value stops the run with exit 5 before a single request is sent, naming every reference that is missing.

Authenticate with the real login, not a stored token

This is the part worth changing a pipeline for.

Every other runner replays a static token pasted into a CI secret. The pipeline then both stores a credential and never tests the login — the flow most likely to break is the one thing nobody checks, and the token expires on a Friday.

tinspec run chain checkout-smoke --sender qa-admin

--sender performs the real token dance on each run, through the same auth provider machinery the app uses — the OIDC round trip, the Keycloak password grant, the script you wrote. Nothing is stored, and the login enters test coverage.

What it does exactly, because the scope matters:

• It fills the identity for every request that resolves to an auth provider and names no sender of its own. A request that names its own sender keeps it.
• It never creates authentication. A request that resolves to no provider still sends unauthenticated.
• An unknown sender id is refused at startup (exit 2) with the ids that do exist. It is never a silent downgrade to the default identity — that would produce two identical green pipelines and a false conclusion.

Inside a chain, a node’s credential comes from the request’s own auth or from its collection’s defaults.auth. A chain has no API-provider context in the runner, so an auth binding that lives on a spec source — the rules in the Auth Map — is not applied to a chain node. If a chain authenticates in the app and sends bare in CI, that is the reason: put the binding on the collection’s defaults or on the request.

tinspec ls senders prints the ids, grouped by provider.

Reporters

tinspec run chain checkout-smoke --reporter junit --out report.xml

Three reporters, all repeatable:

--reporter	What it is
pretty	One line per case and a tally. The format a person reads while waiting.
junit	JUnit XML in the Surefire dialect GitLab, Jenkins and GitHub Actions all parse.
json	A versioned JSON document, stable enough to write a script against.

With no --reporter at all: pretty when stdout is a terminal, json when it is not — because “not a terminal” is a pipeline capturing stdout.

--out takes one path per reporter, in the same order (- means stdout), or none at all. A partial pairing is refused rather than resolved: the obvious reading of --reporter pretty --reporter junit --out report.xml is “the JUnit one goes to the file”, and positional pairing would hand the file to pretty — a mismatch nobody notices until they open an empty report.xml in a CI artifact browser.

Every case carries both a name and a classname, which is the pair GitLab’s JUnit parser builds a test’s identity from. A pruned chain branch is reported as skipped and is never counted as a failure — a branch that was not taken is the graph working.

What a report is allowed to contain

No request header, no request body and no credential ever enters a report. That is not a filter applied while rendering; it is what the report builder collects in the first place.

A failing case attaches its response body — a CI failure whose body you cannot see costs somebody an afternoon — truncated to a few kilobytes and passed through the resolved secret values, so a login response containing a live token reaches the file with the credential replaced by the same <redacted> marker the app shows in a request preview. A passing case attaches nothing.

Exit codes

The whole table is in the CLI reference. The rule that makes it useful:

Exit 1 means a test or assertion failed, and nothing else does. A runner that returns 1 for “the response was wrong” and 1 for “couldn’t reach the host” makes || retry impossible to write correctly and makes a flaky network indistinguishable from a real regression in a pipeline’s history. Transport failures are 4, secrets and auth are 5, a contract break is 6.

When both happen in one run, a failed expectation beats an error: a run where one assertion failed and one request could not connect is a run whose API is demonstrably wrong, and reporting it as “infrastructure, retry me” would let the regression hide behind a retry.

So a retry wrapper can be written honestly:

tinspec run chain checkout-smoke --reporter junit --out report.xml
status=$?
case $status in
  0) ;;                       # passed
  1) exit 1 ;;                # the API is wrong — do not retry
  4|7) echo "infrastructure; retrying" ;;
  *) exit $status ;;          # usage, workspace, secrets, contract: fix the repo or the job
esac

Run the same chain over a table of rows

tinspec run chain checkout-smoke --data rows.csv --fail-fast

.csv, .tsv, .json and .jsonl are accepted. Every column must name an input node the chain declares — a column that names nothing is a hard error at row 0, before any request, listing what the chain does declare. That is the whole difference from the magic-global model, where a typo is silent and a 200-row matrix runs green having tested nothing.

• A missing cell falls back to the input’s own default; a required input must come from a column or a --var.
• --var beats a data column beats the default.
• Each row is its own suite in the report, named chain [row 2] — or chain [row 2: acme] when the file has a name or id column to label it with.
• Rows run sequentially, each starting from the base environment. There is no concurrency flag, and variables never bleed from one row into the next.

tinspec ls chains prints each chain’s declared inputs, which is the header row you need.

The honest CI shape is two jobs

run chain sends requests; it carries no spec binding, so it reports no contract drift. The API-changed-underneath-us half is a separate command, and therefore a separate job:

# once, committed to the repo
tinspec snapshot spec billing -o .tinspec/baselines/billing.json

# in CI
tinspec run chain checkout-smoke --reporter junit --out reports/chain.xml
tinspec check contract billing --baseline .tinspec/baselines/billing.json \
  --reporter junit --out reports/contract.xml

Both write the same JUnit dialect, so one artifacts: reports: junit: glob covers them.

check contract resolves the provider’s live description, diffs it against the committed snapshot, and exits 6 when something breaking changed. --fail-on any fails on any change at all, --fail-on never reports without failing. Individual changes can be allowed by their stable rule id rather than by matching prose — --allow outputAdded, --allow operationDeprecated, repeatable. Breaking and compatible changes both appear in the report; only a breaking one fails the job.

The baseline is the API’s own bundled description, so it reviews and diffs in git by itself. snapshot spec is OpenAPI-only: a gRPC-reflection baseline and a GraphQL one fit the same shape and are not built, and the command says so rather than failing mysteriously.

Before the pipeline: check it offline

tinspec lint            # or: tinspec lint checkout-smoke
tinspec ls chains
tinspec secrets list

lint, ls and secrets list never open a socket. That is a promise, not an accident: a lint that needs egress cannot run as a pre-commit hook or inside a locked-down build container, which is exactly where it earns its keep. lint reports missing request references, variables consumed but never produced, non-chainable protocols and dangling edges, and exits 3 if any of them is an error. It does not check extraction paths against the spec’s documented responses — that needs a live spec binding, i.e. network, so it says so in its own summary rather than quietly skipping it.

Limits worth knowing before you write the job

• Only environments declared in a collection file are visible. The desktop also keeps per-project environments in a machine-local, uncommitted store; those deliberately cannot affect CI. tinspec ls environments prints the collection each visible one came from.
• Collection- and provider-level send settings do not apply inside a chain. run request applies them; the chain runner does not take them at all, so a suite behind a proxy configured at the provider layer succeeds in the editor and fails in a chain. Put the setting on the request itself for now.
• A schema assertion cannot pass in the runner. The CLI resolves no spec endpoints, so an assertion using op: schema fails with “no declared response schema in scope” rather than passing vacuously. Keep those assertions for the app until run chain carries a spec binding.
• Streaming and WebSocket. run request refuses a WebSocket request outright — it has no single response to report on — and refuses --reporter junit|json for a streaming request, because the streaming path evaluates no request-level tests and a machine-readable report from it would claim a clean run.
• Ctrl-C does not run a chain’s teardown region. An interrupted job leaves its fixtures behind and exits 130. Pretending otherwise — racing a cleanup against a second Ctrl-C — would be worse than the honest surprise.
• run pipeline and run file do not exist, and there is no tinspec auth login or auth token. See the reference for the full list of what is deliberately absent.