Testing System & Verification Gates

The compiler test suite is managed by cppbolt-testrunner. Tests are discovered automatically by directory structure, categorized by feature area, and validated against multiple independent gates.

The expected passing baseline is defined in tests/PASSING.txt. To run the test suite and verify current counts:

cargo build --release
cargo run --bin cppbolt-testrunner -- target/release/cppbolt-cli

1. Test Execution & Exit Codes

The test runner discovers tests by directory:

Positive Tests: Contain <name>.cpp and <name>.expected files (with optional <name>.input providing stdin).
Runtime Trap Tests: Assert that execution produces a non-zero WebAssembly unreachable trap (tests/runtime_trap/).
Negative Tests: Contain <name>.cpp and <name>.expected_err files, asserting exact diagnostic output (tests/negative/).

The command syntax is:

cppbolt-testrunner [compiler|ref] [group]

The first argument selects the compiler binary (or ref to run host g++). The optional second argument filters execution to a single feature group.

The Passing Manifest (`tests/PASSING.txt`)

A failing test that is absent from tests/PASSING.txt is treated as an expected failure (e.g. planned template and STL container tests): the runner executes it, reports the failure under Expected failures, but does not cause the process to exit with an error.

The runner exits 0 if and only if:

Every test listed in tests/PASSING.txt passes.
Every passing test is listed in tests/PASSING.txt.
Every discovered test executes without unhandled errors.
The structural variant gate passes.
The independent test manifest matches.

Unexpected test failures or manifest discrepancies exit 1.

2. Test Categories

Main Tests (`tests/<feature>/`)

Contains hand-written and generated tests covering supported C++ features. Every supported language or library construct must have a corresponding positive test in this directory.

Anti-Overfitting Variants (`tests/<feature>_variant/`)

Every main test has a counterpart in a matching _variant group. Variants guard against the compiler overfitting to specific test inputs. The runner compares each same-named test pair and enforces:

Fail: Sources that are byte-identical or differ only in comments/whitespace fail the gate.
Warn: Sources that differ only in identifier names or numeric literals emit a warning.
Pass: Sources that differ structurally (e.g. using helper functions, loops, or scoped blocks) pass cleanly.

Cleanroom Independent Tests (`tests/independent/`)

Tests in tests/independent/ are authored without inspecting compiler implementation code, main tests, or variant tests. Authors work strictly from the C++17 ISO specification and public compiler documentation.

Expected outputs are generated with g++ -std=c++17 -O0 and verified with clang++ -std=c++17 -O0.
All files in the directory are cryptographically pinned in tests/independent/MANIFEST.sha256. The test runner fails if any file changes or disappears without owner approval.

Generated Differential Tests

Large parameter matrices (such as lvalue_matrix.cpp generated by tools/generate_lvalue_matrix.py and printf_table.cpp generated by tools/generate_printf_table.py) are generated mechanically to exhaustively verify operator and formatting combinations against native compilers.

3. Golden Baseline & Native Oracle Gates

Golden Hashes (`tests/golden.txt`)

Before any behavior-preserving refactor, verify the golden baseline:

cargo run --release --bin cppbolt-testrunner -- --check-golden tests/golden.txt target/release/cppbolt-cli

tests/golden.txt records the SHA-256 hashes of both the emitted .wasm binary and standard output for every compiling positive test. A pure refactor must leave every hash unchanged. Regenerate intentionally with --golden only when compiler behavior or binary encoding changes deliberately.

Native Oracle Gate (`--oracle`)

The test runner can replay every positive test through native host compilers:

cargo run --release --bin cppbolt-testrunner -- --oracle
python3 tools/verify_mini_stl_gxx.py

The oracle compiles each test with g++ -std=c++17 -O0 and clang++ -std=c++17 -O0 and asserts that native stdout matches the checked-in .expected files. This gate also runs in CI via the integration test tests/oracle_gate.rs.

4. Size & Complexity Census (`cppbolt-census`)

To maintain long-term codebase health, src/bin/cppbolt-census.rs scans Rust source files and reports function lengths, parameter counts, and file sizes:

cargo run --release --bin cppbolt-census

Verified by tests/census_fixture.rs, the project strictly enforces:

No function may exceed 80 lines.
No function may accept more than 5 parameters (excluding self).
No source file in src/ may exceed 500 lines.

cppbolt Documentation