Agentic testing

Environments, specs & suites

Point Agentic Testing at a running instance of your API, get a spec, and generate coverage.

1. Add an environment

An environment is a target to run tests against — typically a staging deployment. From the repository's Testing section, add one with:

Base URL — where the API is reachable.
Auth — none, a bearer token, an API-key header, or basic auth. Secrets are encrypted at rest and are never returned decrypted by the API, even to you — reads show a redacted marker plus the auth type.
Requests per second — a hard cap, enforced across the whole run so testing never hammers a real deployment. Defaults to 2 and can't be unset.
Destructive opt-ins — specific mutating operations (e.g. DELETE /orders/{id}) you explicitly allow for this environment.
Disposable — flip this on for a throwaway/ephemeral environment where destructive operations are fine to run without opting each one in individually.

2. Get a spec

Agentic Testing needs an OpenAPI spec to compile tests against. Goosy looks for one already committed in the repository (openapi.json, swagger.yaml, and similar); if none exists, it derives one from your route and handler code instead. Either way the spec is stored versioned by commit SHA and a content hash — re-extracting identical content at the same commit is a no-op, and a new, different spec version marks tests compiled against the old one stale rather than silently leaving them wrong.

Generated specs are labeled

A spec Goosy derived from code rather than found in the repo is marked generated, and inferred operations carry a confidence label — high, medium, or low — so you can see at a glance which parts of the spec came from clear code and which were inferred.

3. Generate a smoke suite

With a spec in place, generating a smoke suite is one click and costs no tokens — it's deterministic, built directly from the spec:

1
One test per operation
Every non-destructive operation in the spec gets a test asserting a successful status class and basic response shape.
2
Destructive operations are excluded, with a reason
A mutating operation not opted in for the environment is listed as excluded rather than silently dropped, so you can see exactly what isn't covered and why.
3
Regeneration is idempotent
Running it again after a spec change updates existing tests in place, keyed by operation — a test you manually disabled stays disabled even if the underlying operation is no longer flagged destructive.

4. Write a custom test

For a flow smoke coverage doesn't capture, write the intent in plain English and compile it:

Example intent

Create an order with two line items, then fetch it back
and expect the returned total to match the sum of the items.

Compiling drafts a test artifact with a fast model and verifies/repairs it with a stronger one before it's saved — you get a human-readable step preview (each HTTP request, what gets extracted from the response, and what's asserted) before the test is ever run. A test that fails to compile is saved disabled with the model's explanation rather than half-written; edit the intent and recompile.

Secrets never reach the model

The compile step only ever sees your spec and the intent text — never environment credentials. Where auth is needed, the compiled artifact references a reserved placeholder that's substituted from the environment's stored credentials at run time, outside the compile step entirely.

Variable chaining

A value extracted from one step's response — an order ID, say — can be referenced in a later step's path, query, headers, or body. This is how “create, then fetch” tests work: the create step extracts the new resource's ID, and the fetch step's path uses it.

← OverviewRuns, self-heal & monitoring →