Open specs
Eval format
Portable JSON for eval datasets + run results. Tool-agnostic.
Subpath: @agentskit/core/eval-format.
#Dataset
{
"name": "triage-v1",
"version": "1.0.0",
"cases": [
{
"id": "refund",
"input": "How do I get a refund?",
"expect": { "kind": "regex", "value": "refund policy" }
}
]
}#Expectation kinds
| Kind | Match rule |
|---|---|
literal | exact string equality |
regex | RegExp test |
normalized | whitespace + case-insensitive |
similarity | cosine ≥ threshold (needs embedder) |
#API
import { matchesExpectation, parseEvalSuite } from '@agentskit/core/eval-format'
const suite = parseEvalSuite(json)
const ok = matchesExpectation(output, suite.cases[0].expect)