--- title: "Definition-Attached Testing" scope: "core" kind: "guide" category: "features" aliases: - "calcit.test" - "calcit test" - "affected tests" - "test metadata" --- # Definition-Attached Testing Calcit stores strict tests beside a definition's code, documentation, examples, schema, and tags. Tests are named metadata entries rather than ordinary definitions, so tools can discover, select, and run them without relying on naming conventions. Examples remain executable documentation. Tests are CI guardrails: an assertion or preprocessing failure makes `calcit test` exit unsuccessfully, while other selected tests continue unless `--fail-fast` is used. ## 通过条件与显式断言 每项测试在预处理和执行都正常完成时通过；测试函数的返回值不参与通过判定。 单独写 `= 4 (+ 1 2)` 虽然返回 false，仍可能显示 PASS。返回 nil 或 Unit 也不是失败信号。 比较预期值时使用显式断言： ```cirru assert= 3 $ + 1 2 ``` 布尔条件可用 `assert= true` 或已导入的 `calcit.test/is`。`throws?` 只返回布尔值， 需要断言其结果；预期抛错时可直接使用已导入的 `is-throws`。 `match` / `if` 的检查也要在分支内断言，意外结果分支应抛错，而不是只返回 false： ```cirru match (get ({} (:answer 42)) :answer) (:some value) (assert= 42 value) (:none) (raise |Expected-answer) ``` ## Add a Test Import the built-in assertions where the test expression will run: ```bash calcit edit add-import app.main --code 'quote $ calcit.test :refer $ is is= is-not= is-throws throws? fail' ``` Attach a stable, named test to a definition: ```bash calcit edit add-test app.main/add adds-two-numbers \ --tags unit,fast \ --code 'quote $ is= 3 (add 1 2)' ``` The expression is compiled in the owning definition's namespace. It can use that namespace's imports and local definitions. Use a file or stdin for a multiline expression. As with other AST edit commands, the input must contain exactly one quoted node. ### 测试中的显式宿主能力 测试共享所属 namespace 的名字解析，但不继承所属定义 schema 的 `:features`。 直接读取宿主字段、调用宿主方法或执行 `js-cast` 的测试，把这些操作放进现有 `hint-fn` 显式标注的局部函数，再调用它；普通断言不需要增加 feature。 ```cirru.no-check let check-host! $ fn () hint-fn $ Fn (-> Unit) (:features :js-ffi) let host $ checked-counter-host $ make-counter-host 3 assert= 3 $ host :total , &unit check-host! ``` 这里的 `checked-counter-host` 和 `make-counter-host` 来自测试所在 namespace； 这个例子展示 `calcit/js-ffi-module/calcit.cirru` 中实际回放的测试结构。 `calcit test` 与跨后端定义测试运行器都保留这个显式词法边界，不为生成的测试函数 自动开启 `:js-ffi`。删除局部注解后，直接宿主操作仍报 `E_JS_FFI_FEATURE_REQUIRED`。 feature 声明不会增加 backend 能力：JS-only 的测试仍需选择 JS 后端，文件片段、 模块依赖和 browser/node target 仍须有效；原生测试可以调用显式标注的普通函数， 但不会因此获得 JavaScript 宿主实现。不要用测试 tag、排除项或拥有者的 feature 代替测试体所需的显式声明。 ## Inspect and Maintain Tests ```bash calcit query tests app.main/add calcit query context app.main/add --format json calcit edit add-test app.main/add adds-two-numbers --overwrite --code 'quote $ is= 5 (add 2 3)' calcit edit rm-test app.main/add adds-two-numbers ``` Test names must be non-empty, have no surrounding whitespace, and be unique within one definition. Replacing one requires explicit `--overwrite`; removal uses the stable name rather than an array index. The persisted shape is equivalent to: ```cirru.no-check :tests $ [] %{} 'TestEntry :name |adds-two-numbers :code $ quote $ is= 3 (add 1 2) :tags $ #{} :unit :fast ``` Snapshots written before this field existed load with an empty test list. ## Run Tests ```bash # All project-owned tests calcit test # One namespace or definition calcit test app.main calcit test app.main/add # Exact name and tags calcit test app.main --name adds-two-numbers calcit test --tag unit --tag fast # Exclude slow or integration guards, and reject an empty selection calcit test --tag unit --exclude-tag slow --exclude-tag integration --require-match # Inspect selection without executing calcit test app.main --list # Emit a compact machine-readable report for large CI or agent runs calcit test --tag unit --summary-only --format json ``` Each test is compiled as its own synthetic function and executed independently. Reports use the stable identifier `namespace/definition#test-name`. Without a scope, `calcit test` discovers tests only in namespaces defined by the input snapshot. Tests bundled with `calcit-core.cirru` or loaded modules are excluded, so an external project does not accidentally run Calcit's own test suite. Pass an explicit namespace such as `calcit test calcit.test` when maintaining the core assertions. In JSON mode, runner output produced by `println`/`echo` is redirected to stderr so stdout remains one parseable report envelope. Full reports contain one row per selected test, including its execution duration. `--summary-only` suppresses per-test program output and emits counts and total duration only, with `detail: "summary"`; failures are still reported by the test runner. This is useful when an agent needs a reliable gate without loading hundreds of passing test rows. The report also distinguishes `selected` tests from `executed` tests when `--fail-fast` stops early. `--exclude-tag` removes a test carrying any supplied tag, after required `--tag` matching. `--require-match` turns an empty selection into a failure; use it in CI whenever a tag, scope, or affected-test command is expected to protect a non-empty suite. ## Run Affected Tests ```bash calcit test --affected app.math/add calcit test --affected app.math/add --affected app.math/subtract calcit test --affected app.math/add --list ``` For affected selection, Calcit preprocesses every candidate test and follows the compiled `DefId` dependency graph transitively. It always includes tests attached directly to a changed definition. If dependency analysis for a test fails, that test is conservatively selected and reported as failed, so a static-analysis problem cannot silently hide a failing guardrail. Normal `calcit` execution does not run tests implicitly. CI and coding agents should invoke `calcit test` explicitly. ## Built-in Assertions The `calcit.test` namespace is embedded in `calcit-core.cirru` and requires no external module: - `is` asserts a truthy expression. - `is=` and `is-not=` compare values. - `throws?` reports whether an expression raises. - `is-throws` requires an expression to raise. - `fail` raises immediately with a message. All assertion macros evaluate each supplied expression once. ## Execution Cost Normal `calcit test` runs compile each selected test only when it is about to run. This keeps `--fail-fast` responsive and avoids preprocessing tests after the first failure. `--affected` intentionally compiles its candidate tests first: it needs their static dependency graph to select a safe, transitive subset. Tests execute the selected test and its runtime dependencies; documentation and examples are not themselves executed as tests. Prefer a short definition-local test for one API contract, and reserve integration fixtures for behavior that actually crosses definitions or backends. ## Target Coverage ### 仓库 CI 与快速本地回归 CI 的 Core/CLI 与文档检查共用一次 `ci` 配置构建的 CLI：开启优化，同时保留 debug assertions 与整数溢出检查。同一 workflow 内按提交 SHA 命名的 artifact 供两个任务下载使用，不跨提交寻找旧二进制。脚本通过 `CALCIT_BIN` 选择 CLI； 未指定时保留各脚本原有的本地构建查找方式。 ```bash cargo build --locked --profile ci --bin calcit yarn install --immutable yarn compile yarn procs-link export CALCIT_BIN="$PWD/target/ci/calcit" node scripts/run-core-tests.mjs node scripts/check-strict-default.mjs node scripts/check-known-assertion.mjs bash scripts/test-wasm.sh bash scripts/check-docs-md.sh ``` 这些命令用于对应范围的回归。普通 core 语义由统一运行器覆盖； `CALCIT_LINT_CORE=1` 的直接执行另行保留，用于检查改写后的树。 `post_lowering_cli` 和 namespace-import 集成测试在 Rust 测试任务中运行，后者 同时实际执行生成的 JS。 断言检查脚本保留 fixture 专属回放、非法类型程序的逐后端诊断以及失败时不产出 代码的检查；core 定义回放统一由 `run-core-tests.mjs` 承担。 PR 推送新提交时，CI 取消同一 PR 旧提交上尚未完成的 Test workflow；main 的每次提交仍独立验证。减少执行成本时，应先移除重复构建与重复回放，保留 原始 `:tests`、零匹配失败检查和独立的后端边界验证。 ### 统一后端回放 `calcit test` 在 native 上运行 definition `:tests`。仓库用一个统一运行器把同一批 `:tests` 在 native、生成的 JS 与 WASM 上各执行一次： ```bash # 默认：全部 core :tests，native / JS / WASM node scripts/run-core-tests.mjs # 按 tag、名称或定义缩小范围；零个测试被选中时失败 node scripts/run-core-tests.mjs --tag unicode --backend native,js node scripts/run-core-tests.mjs --target 'calcit.core/integer?' --name distinguishes-integers # 其它 Snapshot 的 :tests 也可重放；WASI 0.3 command 需显式选择并提供 Wasmtime node scripts/run-core-tests.mjs --snapshot calcit/test-traits.cirru --target test-traits.main/test-qualified-contains-boundary --backend native,js WASMTIME_CLI=wasmtime node scripts/run-core-tests.mjs --backend wasi --target 'calcit.core/integer?' ``` 运行器为每个测试生成一个零参数函数，写入临时 Snapshot，再分别执行：native 与 WASI 逐个打印标记后运行，JS 导入生成的模块逐个调用，WASM 在 Node 中实例化后逐个调用导出函数。native 是参照结果，JS 与 WASM 的 `println` 输出必须与 native 一致。失败报告列出测试 id、后端、期望与实际值，以及该测试在其它后端的状态。 回放外部项目 Snapshot 时，运行器在私有临时目录保留同级、上级及传递文件模块的 路径关系，沿用选定入口的 modules 配置；JS 文件片段和对应 `deps.cirru` 也保留。 已安装的 `.calcit/modules` 图只读复用，让临时 Snapshot 使用相同的模块与 CLI 版本约束。cursor、错误状态和写锁留在临时目录，不写回源项目。 从该项目目录运行并指定固定 `CALCIT_BIN`，生成 JS 使用该目录的 Node 依赖； 运行器不安装依赖或写入模块缓存。真正缺失的文件模块在准备回放时明确失败， 不会对每个测试重复二分同一个加载错误，也不记作 unsupported 或空通过。 应用已有 browser/Node 等命名入口时，用同一个运行器的 `--entry` 选择原入口， 再用 tag 或 definition scope 选择该宿主上的测试： ```bash node scripts/run-core-tests.mjs --snapshot calcit.cirru --entry server --tag node --backend js ``` 入口的 target、modules、type slots 和 feature policy 沿用 Snapshot 配置，不按测试 内容猜测，也不自动授予 FFI 能力。省略 `--entry` 仍使用 default；未知入口失败。 选择 browser 入口执行 Node-only 测试仍报告原 target 错误。JS 构建明确引用全部 选中测试，单项编译失败不会变成缺少导出或静默跳过其它 namespace 的测试。 新增的 core `:tests` 自动进入三个后端，无需登记。某个后端暂时无法运行的测试写入 `scripts/core-tests-exclusions.cirru`，按后端列出测试 id 或 tag，并写明原因，原因以类别开头：`unsupported`（后端明确拒绝该构造）、`parity`（结果与 native 不同，另开 issue 修复）、`host`（需要回放宿主未提供的能力）、`replay`（因不在所属 namespace 回放而产生）。清单中引用不存在的测试会使运行失败；`--report-unexpected-pass` 会额外运行被排除的测试，列出已经通过、可以移出清单的条目。 新增测试需要核对它实际使用的后端能力。例如 `try`、局部 `ref` 和开放值的 `data-view` 目前在 WASM/WASI 明确报告 unsupported；对应测试仍完整运行于 native/JS，排除项必须指向具体测试并记录实际诊断。一个测试同时覆盖已支持和未支持的构造时，保留原测试，并给已支持的断言单独附加 `:tests`，避免因整项排除而丢失这部分跨后端覆盖。不能用排除项掩盖本应支持的行为回归或改变原断言。 只测试编码、ABI、memory layout、宿主注入或 unsupported 诊断的场景继续使用 `scripts/test-wasm*.mjs`、`scripts/check-*.mjs` 等专用脚本；新的跨后端回放需求给 `:tests` 加 tag 并用统一运行器选择，不再新增 `scripts/check-*.mjs`。 运行器拒绝空的或重复的后端列表。native 与 WASI 的每个测试必须完整输出开始和结束标记；进程提前正常退出不会被计为通过。`--report-unexpected-pass` 自动补上 native 参照，只有测试成功且输出与 native 一致时才建议移除排除项。 排除清单只作用于这个运行器。`calcit test` 在 native 上不读清单，所以新增的 core `:tests` 必须先在 native 通过；native 尚不符合的规则留在对应 issue 中，修复时再补测试。 ### 差分随机测试 `scripts/fuzz-primitives.mjs` 用固定随机种子生成数值、字符串与列表原语的调用，输入覆盖 NaN、±inf、`-0`、i32 与安全整数边界、非 ASCII 文本、空值和越界下标，再交给上面的运行器在各后端执行。数值与计数结果经 `to-string` 转成文本，其余标量按共享协议输出，再与 native 比较：native 返回时其它后端必须返回相同文本，native 报错时其它后端必须报错或 trap。 ```bash # PR 中的小规模运行：每个原语 6 次调用 node scripts/fuzz-primitives.mjs --seed 1 --cases 6 # 大规模运行；失败信息包含种子、调用与各后端结果，用同样参数即可复现 node scripts/fuzz-primitives.mjs --seed 42 --cases 150 ``` 已由 issue 跟踪的差异写在脚本的 `known` 表中，按原语与后端汇总输出，不使运行失败；修复后从表中删除。新发现的差异先固化为 core `:tests` 或开 issue，再决定是否加入 `known`。`Primitive fuzz` workflow 每周以运行编号为种子执行一次大规模运行，也可以手动触发并指定种子。 ### 限制 - 快速本地回归命令只验证对应范围，不代替完整 CI。 - WASM 在独立的回放 namespace 中执行 core 测试，依赖 `calcit.core` 内部豁免的测试列为 `replay`。 - WASM 宿主只提供 `io.log_*` 与 `math` 导入，其它宿主调用会使测试失败。 - WASI backend 不在默认集合中，需要 `WASMTIME_CLI`。 - 差分随机测试只覆盖返回标量的原语，不覆盖 FFI 与宿主 IO。 ## Choose the Test Surface (Calcit-first) Place user-observable behavior in definition `:tests`; keep Rust tests for the low-level boundaries a Calcit program cannot reasonably construct. - Prefer `:tests` for language semantics, type inference, macro expansion, core APIs, runtime behavior, and cross-backend contracts. Write the same expressions a user would, so reviewers see the input and expected result directly. - Keep Rust tests for parsers/serializers, internal data structures, host/FFI boundaries, memory or concurrency invariants, error recovery, and conditions without a stable Calcit surface. - A backend-specific implementation may keep precise Rust tests, but should also have a Calcit-side test for the shared semantic. Add target execution or codegen evidence when JS, IR, or WASM is involved. - When behavior changes, migrate a Rust test to `:tests` if it only verifies semantics Calcit can express; remove the duplicate once Calcit coverage is equivalent or stronger. - Do not add Rust/Calcit test-count ratios, coverage percentages, or a statistics analyzer. Placement follows reviewable semantic boundaries. 例如，严格 Tag 语义把用户可观察的 Tag/String 区分写在 `calcit.core/&compare` 的 `:tests` 中。WASM 若复用这项语义，应执行同一份 Calcit case；只有编码、ABI、 memory layout 或 unsupported lowering 这类 backend 内部边界才保留 Rust/脚本 fixture。 ### PR checklist - Why does this coverage belong in `:tests`, or what concrete low-level reason keeps it in Rust? - Do the Calcit tests assert the relevant success, failure, or boundary branch with user-level expressions? - If a Rust test only restates the same external semantics, was it migrated or removed? - Do Native and the relevant JS/IR/WASM checks still pass? ## Core Test Placement For a core function, macro, or builtin whose behavior can be expressed in one namespace, attach the test directly to that definition in `calcit-core.cirru`: ```bash calcit src/cirru/calcit-core.cirru edit add-test calcit.core/range creates-half-open-range \ --tags unit,core \ --code 'quote $ assert= ([] 2 3 4) (range 2 5)' ``` The repository runs this suite with `yarn try-core-tests`, which explicitly loads `calcit-core.cirru` and runs its `:unit` tests in summary mode. When the input snapshot itself contains `calcit.core`, `calcit` preserves that source namespace instead of replacing it with the binary's embedded copy. This makes the command reliable even when the globally installed `calcit` predates the source snapshot being tested. Keep `calcit/test-*.cirru` when a case verifies several definitions together, parser syntax, stateful behavior, JavaScript/WASM code generation, or a full program flow. Those files are integration fixtures rather than a substitute for definition-local tests. ## 主仓库开发回归 开发 Calcit 编译器和 runtime 时，在仓库根目录执行： ```bash cargo test CALCIT_PROOF_AUDIT=0 CALCIT_LINT_CORE=1 yarn check-all ``` 第二条命令使用与现有 CI 相同的通用语言 fixture 模式。CI 的入口是 [`scripts/ci-check-all.mjs`](../../scripts/ci-check-all.mjs)，它将同一套 `package.json` 中的 `check-all` 步骤分配到多个 runner。 - `CALCIT_PROOF_AUDIT=0` 让通用 fixture 保留其有意测试的开放写法，严格类型诊断继续生效。 - `CALCIT_LINT_CORE=1` 开启与 CI 一致的改写后节点校验，检查 lowering 是否保留类型证据。 默认证明审计仍由完整回归中的 [`scripts/check-strict-default.mjs`](../../scripts/check-strict-default.mjs) 验证。该脚本主动清除 `CALCIT_PROOF_AUDIT`，执行默认审计的合法用例与错误拒绝用例； 通用 fixture 模式不会替代这一门禁。对比本地与 CI 的结果时，先记录完整命令和上述两个环境变量。 CI 的 Core and CLI checks 还统计 core Snapshot 中未归类的弱类型位置，数量只能持平或下降： ```bash calcit src/cirru/calcit-core.cirru analyze weak-types --summary-only --format json \ | jq '.data.summary.intents.unresolved' ``` 上限写在 `.github/workflows/test.yaml` 的 `UNRESOLVED_LIMIT`。数量下降时，在同一个 PR 里把上限改成新的数量； 新增的位置应写出具体类型，确实开放的边界归入已有的 intent 分类。 普通业务项目继续使用默认证明审计： ```bash calcit calcit.cirru --check-only calcit fix --workflow strict --verify --format edn ``` 上面的 fixture 环境变量仅用于 Calcit 主仓库回归，不应写入业务项目脚本或全局 shell 配置。