Testing

Choose tests by the behavior you changed, not just the nearest filename. Setup prepares the default suite; selected integrations own additional prerequisites. Verification layers explain what each kind of evidence proves.

Choosing tests

A diagnostic-count test does not prove runtime support. Fixture lanes cannot exercise constructs absent from their inputs; add a targeted emitted regression. Framework suites may cover only a named subset (Rust CI uses the inflector subset); a suite's green result is not a claim about everything in that target.

Fixtures

• fixtures/tiny-blog/ and fixtures/tiny-api/ are checked in. Use them for small shape/absent-feature gates; extend them deliberately.
• fixtures/real-blog is generated by bin/rh fixture using scripts/create-blog.
• fixtures/store is generated by (cd fixtures && ../scripts/create-store store); its generator runs the Rails guide tests too.
• External corpora have their own pins and setup. For example, Writebook is an inventory, not runtime conformance.

The generated trees are ignored by Git; a fresh clone has neither. Accessors in src/fixtures.rs name the missing tree and command. bin/rh verify lists missing fixtures but never installs or generates them. Missing Ruby, gems, or SDKs can cause a harness to skip or fail; read the executed test output rather than assuming exit zero means coverage.

Skips and early exits

A test that skips passes, and Nextest does not show a passing test's output, so a skip printed with eprintln! never reaches a CI log. A test whose prerequisite (a gem for a native Rails control, Ruby on PATH, a native compiler) is missing calls prerequisite::missing(what) from tests/support/prerequisite.rs and returns. ROUNDHOUSE_SKIP_LOG=<file> appends each skip to that file; the CI unit job lists them in its summary. ROUNDHOUSE_REQUIRE_PREREQUISITES=1 fails such a test instead, for a run that provides everything. PostgreSQL keeps its own pair, ROUNDHOUSE_PG_TEST_URL and ROUNDHOUSE_PG_TEST_REQUIRED.

A run_ruby or run_spinel* script reports a failed check by raising, so exit status 0 alone would pass a script that stopped before its checks. The emit-and-run harness adds a completion line after the script and counts the run a success only if that line was printed. Do not end a passing script with exit; let it reach its last line.

Native probes

A Spinel test that checks one construct rather than the app can compile the probe alone: run_spinel_slim(&["runtime/active_support_ext"], script) in emit_and_run.rs builds only the files the overlay writes and the runtime entries named, which takes seconds where run_spinel's whole-blog boot takes minutes. Name too few entries and spinel refuses or the run raises; it does not pass. The probe no longer shares whole-program inference with the app, so scheduled Full validation sets ROUNDHOUSE_SPINEL_SLIM=0 and compiles the same tests inside boot. Keep run_spinel for tests whose subject is the app, its models or its HTTP surface.

The spinel-framework lane runs spinel at -O0: these suites judge behavior, and cc at -O2 was over a third of their CPU. spin build, spinel-smoke, spinel-compare and the Campfire lanes keep -O2. Locally, CI_SPINEL_OPT= keeps the default for scripts/ci-spinel-framework.sh, and CI_SPINEL_FRAMEWORK_BUDGET sets the seconds after which it stops a shard's suites and names them.

Native controls

A test that claims emitted behavior matches Rails runs the same program under CRuby with real Rails and compares. rails_oracle.rb picks that Rails in one place: by default the installed release the suite pins, RH_RAILS_ORACLE=<version> for another release, or RH_RAILS_ORACLE=<path> for a Rails checkout. Call RailsOracle.activate("activerecord", sqlite3: ">= 2.9") before requiring Rails; other gems take a requirement, not one exact build. When RailsOracle.available? (same arguments) is false, skip with a reason rather than fail. Some older controls still pin 8.1.4 inline and ignore RH_RAILS_ORACLE.

Fork specs

Some suites were ported from the Shopify/roundhouse-1 fork's tests to serve as specs for behavior the fork's compiler has and this one does not yet. A test the compiler does not pass is ignored with the group of behavior it waits on, #[ignore = "fork spec: <group>"] (load order, autoload, conditional declarations, class-body loops and hooks, gem closures, rails_dependent_by_name, postgresql). Run a group's tests with --ignored to see what remains; a change that makes one pass removes its ignore. Where a spec also asserts the fork's emitted layout (a file path, a generated line), the change that makes it pass rewrites that assertion for this tree's layout and keeps its comparison with native Ruby.

The PostgreSQL specs compare the emitted program with native Rails on the same server (tests/support/pg_probe.rs). They need ROUNDHOUSE_PG_TEST_URL and the pg gem, and skip without them unless ROUNDHOUSE_PG_TEST_REQUIRED is set. Two of their groups are not yet compiler work: postgresql native control, whose native side fails on the Rails release the oracle loads, and postgresql layout, which still asserts the fork's emitted files.

Complete local verification

1. Install cargo-nextest once:

   cargo install cargo-nextest --locked
   

2. Run the complete default suite:

   bin/rh verify --all
   

This command runs library and binary tests plus every integration suite with Nextest. It then runs cargo test --locked --doc, because Nextest does not run documentation tests. Ignored tests remain ignored. It does not use a result cache, and it bypasses Nextest's default filter to prevent omitted tests.

The full run compiles integration sources into one executable. build.rs discovers every tests/*.rs suite; no second list needs maintenance. Only full verification enables this target. Normal Cargo commands and bin/rh verify --test NAME keep the individual targets.

Rust tests that start their own executable use the name! helper for --exact selection. It preserves the test name in both layouts. tests/test_subprocess.rs checks that the child executes its test body, not just that the child exits successfully.

--all cannot be combined with --test, --toolchain, --ignored, --lib, or --contract. NEXTEST_TEST_THREADS controls test processes; --jobs controls Cargo workers. Full verification defaults incremental compilation on and preserves an explicit CARGO_INCREMENTAL value. --all --plan remains read-only.

An ARM64 macOS run produced these measurements with the optimized test profile:

The complete combined command took 265.837 s with a warm build, including discovery and documentation checks. The first combined build in this work took 188.84 s. That build cost is not included in the warm-run time. Switching between full and focused modes can rebuild the library and test target. The final run passed 5,985 tests. All 6,334 existing test identities and their ignored status matched the separate targets; one new regression test checks child selection. The same 350 tests remained ignored.

The macOS verification used TMPDIR=/private/tmp. The default /var/... temporary path caused existing assertions against canonical /private/var/... paths to fail.

Focused verification

Without --all, bin/rh verify runs the integration suites you select, sequentially and fail-fast. The library suite runs only with --lib, after those suites. --contract runs ingests_without_errors and type_analysis_coverage in tests/real_blog.rs, then every_runtime_method_body_is_fully_typed in tests/runtime_src_integration.rs, and does not build the other integration binaries. A --contract --exact filter that matches no test fails that check. Neither mode is a hosted CI emulator or merge approval. A green focused run does not replace the commit bar below.

bin/rh verify --plan --base main --test ingest
bin/rh verify --test ingest --test real_blog
bin/rh verify --contract
bin/rh verify --lib --test ingest
bin/rh verify --test framework_tests_ruby --ignored
bin/rh verify --toolchain ruby --json > /tmp/verification.json

• Repeat --test or --toolchain to select more suites. Default --test excludes ignored tests; an ignored-only suite otherwise executes nothing. --ignored runs only ignored tests in selected --test suites, not the library suite or the contract filters. --toolchain already selects ignored tests in <target>_toolchain.
• --plan is read-only: no Cargo, installs, tests, or verification lock. With no --test, --toolchain, --contract, or --lib, it is only a hosted-coverage preview and reports no local checks. A run without --plan, --all, or one of those selectors exits with an error.
• --base REF compares the current working tree with that exact commit (default HEAD), including non-ignored untracked files. It neither fetches nor finds a merge base. The hosted selection preview is informational, not GitHub's actual PR merge-tree/draft/label context.
• Execution uses locked Cargo commands. Every suite this loop launches keeps --test-threads=1, because suites share SQLite or the process-global ingest::survey. Cargo workers default to CARGO_BUILD_JOBS or the machine's processor count. --jobs overrides workers. Explicit Cargo environment settings are preserved. Verify does not set CARGO_INCREMENTAL; Cargo's own dev/test default applies, and a caller-supplied value is passed through. Hosted unit shards do not use this script.
• The worktree lock coordinates verifier callers only. Do not run unrelated Cargo builds/tests concurrently in that checkout.
• --json separates executed, failed, and not-run checks; child output goes to stderr. It includes filesystem-space snapshots, with advisory low-space warnings. It is not a reusable receipt or content fingerprint.

For all options, use bin/rh verify --help. The implementation and verifier contract tests are bin/rh and tests/rh_verify_test.rb.

Direct Cargo checks

cargo test --locked --lib
cargo test --locked --test ingest
cargo test --locked --test framework_tests_ruby -- --ignored --nocapture
cargo test --locked --test rust_toolchain -- --ignored --nocapture
cargo test --locked                 # default suite before commit
cargo test --locked --all-targets   # milestones

CI repeats the CRuby gates (framework_tests_ruby, uuid_key_ruby, tiny_api cruby_gate_) on the next MRI line, env.MRI_RUBY_NEXT, in compare-ruby-next. To reproduce that locally, put that Ruby first on PATH and install the scaffold bundle for it too (bundle resolves ruby from PATH), then run the same commands.

Focused checks do not replace the default-suite commit bar. Native toolchain and framework integrations are generally ignored so local default tests do not require every SDK. They are not all separate GitHub jobs; see CI.

The local test profile uses opt-level = 1 to reduce compiler pipeline costs across the integration suites. It keeps debug assertions, overflow checks, and file/line backtraces. No test selection or ignored-test rule changes. The first build costs more; later runs reuse the optimized artifacts.

Unit CI overrides optimization only for the roundhouse package. It uses opt-level=0 for archive and documentation builds while dependencies stay optimized. See the CI execution guide.

On an ARM64 macOS checkout, three runs of each built test binary gave these median times. These measurements exclude compilation:

For debugger locals, use CARGO_PROFILE_TEST_DEBUG=2. To step through unoptimized test code, also use CARGO_PROFILE_TEST_OPT_LEVEL=0. Suites that assert file/line backtraces need symbols. Neither the verifier nor a passing subset cleans old build artifacts. If space is exhausted, stop builds and inspect the dedicated target directory before removing disposable outputs. Never strip live programs or clean a cache shared with another checkout.