differential-fuzzer

Information about the differential fuzzer tool, how to run it and use it catch bugs in Turso. Always load this skill when running this tool

By tursodatabase · 1,142 installs

npx skills add tursodatabase/turso --skill differential-fuzzer

Source repository · Upstream listing

Differential Fuzzer Always load [Debugging skill for reference](../debugging/) The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs. Location testing/differential oracle/fuzzer/ Running the Fuzzer Single Run Continuous Fuzzing (Loop Mode) Docker Runner (CI/Production) Environment variables for docker runner: TIME LIMIT MINUTES Total runtime (default: 1440 = 24h) PER RUN TIMEOUT SECONDS Per run timeout (default: 1200 = 20min) NUM STATEMENTS Statements per run (default: 1000) LOG TO STDOUT Print fuzzer output (default: false) GITHUB TOKEN For auto filing issues SLACK WEBHOOK URL For notifications Output Files All output goes to simulator output/ directory: File Description test.sql All executed SQL statements. Failed statements prefixed with FAILED: , errors with ERROR: schema.json Database schema at end of run (or at failure) test.db Turso database file (only with keep files ) test sqlite.db SQLite database file (only with keep files ) Reproducing Errors Always follow these steps 1. Find the seed and profile in the error output: 2. Re run with that seed and profile (a seed only replays under the same profile): 3. Read the minimized reproduction first. On an oracle failure the fuzzer writes these files to simulator output/ : minimized.sql a shrunken state script plus the shrunken failing statement, produced automatically. Start here. turso state.sql / sqlite state.sql each engine's full state as a replayable script, when you need more than the minimized version kept. test.sql every executed statement (the failing one is marked FAILED: ). The minimizer falls back to replaying this history when the failure depends on how the state was built, not just its contents. schema.json table structure at failure time. 4. Probe the reproduction with differential probe . It runs a statement per line script on Turso and SQLite side by side, prints both outcomes for every statement, marks divergences, and compares the final table contents. Exit code 1 means something diverged. Use it instead of piping SQL into the two shells: the tursodb shell cannot ATTACH ':memory:' AS aux , so fuzzer reproductions with an aux schema only run correctly through the probe. Reading from stdin also works: echo "SELECT ~X'96';" cargo run q p differential fuzzer bin differential probe . 5. Bisect by editing the script. Copy minimized.sql , simplify one thing at a time (replace an expression with a constant, drop a column, drop a state line), and re run the probe after each edit. The divergence marker tells you immediately whether the edit kept the bug. This loop usually ends at a one line kernel you can hand to EXPLAIN on both engines. 6. Create a regression test in .sqltest (preferred) or .rs from the kernel. Always load the [Debugging skill for reference](../debugging/). Understanding Failures Oracle Failure Types 1. Row set mismatch Turso returned different rows than SQLite 2. Turso errored but SQLite succeeded Turso rejected valid SQL 3. SQLite errored but Turso succeeded Turso accepted invalid SQL 4. Schema mismatch Tables/columns differ after DDL Warning (non fatal) Unordered LIMIT mismatch LIMIT without ORDER BY may return different valid rows Key Source Files File Purpose main.rs CLI parsing, entry point runner.rs Main simulation loop, executes statements on both DBs oracle.rs Compares Turso vs SQLite results schema.rs Introspects schema from both databases memory/ In memory IO for deterministic simulation Tracing Set RUST LOG for more detailed output: