Conformance checks

Status: draft. A hosted checker and supplier registration UI do not exist yet. The downloadable examples describe expected results; they do not certify a live service.

Start a bounded run

The supplier explicitly starts a run against an immutable configuration and accepts its displayed limits. The platform records the configuration revision, model revision, profile, capabilities, suite version, limit-policy version, start time, and immutable deadline. Limits cover HTTP request count, input size, output token ceiling, response bytes, connect/idle timeouts, and total run time. Production values and recheck cadence are not selected by these examples.

Endpoint control must be verified before conformance checks can consume supplier resources. The production ownership-verification method remains undecided.

Checks execute real inference and consume supplier resources. They do not reserve or debit a buyer wallet, accrue supplier income, or promise platform reimbursement. They are outside marketplace transactions. The run must be bounded even if a supplier ignores its requested token ceiling.

Required cases

The Chat suite assigns stable IDs to the checks below. The Responses profile defines its separate required cases and suite. The manifest selects a suite by explicit profile ID; passing one never qualifies the other. Several assertions may share a bounded request; each network request still counts against the run limit.

Case ID	Evidence	Passing condition
destination_security	Parsed URL, DNS and TLS/connection checks	Destination satisfies the egress policy
authentication	Missing and invalid authentication	401 or 403 before generation; no redirect
model_mapping	Valid mapping and an unknown model request	Returned model matches declaration; unknown model rejected
json_completion	Bounded non-stream request	Valid one-choice text completion and usage
incremental_stream	Bounded stream observation	Multiple content events arrive before terminal completion
terminal_usage	Parsed complete stream	Consistent IDs, one finish, one usage summary, then [DONE]
invalid_request	Unsupported field and malformed input	400 or 422, with a structured error
bounded_io	Checker transport and deadline behavior	Limits enforced, no redirect/retry or success fabricated on abort

bounded_io includes platform-client tests with controlled local fault servers; it does not prove supplier cancellation. The run records the verified client policy version and its own bounded execution. A short answer with insufficient events cannot pass incremental_stream from a buffered final result. The checker may use another probe within the original limits; otherwise the case remains not_run with an insufficient-evidence reason.

Run states and results

Runs move from queued to running to passed, failed, timed_out, or cancelled. There is no state waiting for a person. Persist each probe’s identity and budget consumption before network I/O; a worker lease expiring cannot grant permission to repeat a potentially sent probe. A worker restart resumes only persisted safe work within the original run’s limits; a potentially sent inference probe is not replayed when its outcome is unknown. Lost probe results count against the request budget. The deadline cannot be extended by retries or restarts. A deadline sweep terminates abandoned queued or running jobs.

The report schema defines terminal reports. Each case includes expected, sanitized observed, and remediation text. Outcomes are passed, failed, or not_run; omitted, unknown, or duplicate cases cannot produce a passing report. When the overall outcome is passed, every required case must be passed before the deadline.

Reports do not contain raw credentials, authorization headers, prompts, response bodies, or customer data. The example report is synthetic evidence for documentation tests only. Suppliers cannot submit a report to grant themselves eligibility; backend accepts authenticated platform worker results for a previously registered run.

Eligibility and configuration changes

Eligibility is backend-owned derived state. Its scope is the exact endpoint ID, configuration revision, model ID/revision, profile ID, and capability set. Backend also verifies the expected suite and limit-policy versions, run identity and deadline, and that the configuration remains current and enabled.

A current passing report satisfies the protocol check for that scope only. Changes invalidate the relevant result before new traffic can be admitted. Reject stale results with stale_configuration; retain them for audit. Removal or suspension blocks new admission. Existing admitted attempts keep their frozen configuration and original financial deadlines; configuration changes do not authorize replacement attempts. Credential revocation can still make them fail.

Schema validation does not grant eligibility. Passing once does not establish future health, actual model identity, honest usage, or quality. Operational rechecks, expiry, cooldowns, and capacity policy remain to be defined without manual approval.