codspeed-setup-harness
Set up performance benchmarks and CodSpeed harness for a project. Use this skill whenever the user wants to create benchmarks, add performance tests, set up CodSpeed, configure codspeed.yml, integrate a benchmarking framework (criterion, divan, pytest-benchmark, vitest bench, go test -bench, google
By codspeedhq · 434 installs
npx skills add codspeedhq/codspeed --skill codspeed-setup-harness
Source repository · Upstream listing
Setup Harness
You are a performance engineer helping set up benchmarks and CodSpeed integration for a project. Your goal is to create useful, representative benchmarks and wire them up so CodSpeed can measure and track performance.
Step 1: Analyze the project
Before writing any benchmark code, understand what you're working with:
1. Detect the language and build system : Look at the project structure, package files ( Cargo.toml , package.json , pyproject.toml , go.mod , CMakeLists.txt ), and source files.
2. Identify existing benchmarks : Check for benchmark files, codspeed.yml , CI workflows mentioning CodSpeed or benchmarks.
3. Identify hot paths : Look at the codebase to understand what the performance critical code is. Public API functions, data processing pipelines, I/O heavy operations, and algorithmic code are good candidates.
4. Check CodSpeed auth : Ensure codspeed auth login has been run.
Step 2: Choose the right approach
Based on the language and what the user wants to benchmark, pick the right harness:
Language specific harnesses (recommended when available)
These integrate deeply with CodSpeed and provide per benchmark flamegraphs, fine grained comparison, and simulation mode support.
Language Framework How to set up
Rust divan (recommended), criterion, bencher Add codspeed <framework compat as dependency using cargo add rename
Python pytest benchmark Install pytest codspeed , use @pytest.benchmark or benchmark fixture
Node.js vitest (recommended), tinybench v5, benchmark.js Install @codspeed/<framework plugin , configure in vitest/test config
Go go test bench No packages needed — CodSpeed instruments go test bench directly
C/C++ Google Benchmark Build with CMake, CodSpeed instruments via valgrind codspeed
Exec harness (universal)
For any language or when you want to benchmark a whole program (not individual functions):
Use codspeed exec m <mode <command for one off benchmarks
Or create a codspeed.yml with benchmark definitions for repeatable setups
The exec harness requires no code changes — it instruments the binary externally. This is ideal for:
Languages without a dedicated CodSpeed integration
End to end benchmarks (full program execution)
Quick setup when you just want to track a command's performance
Choosing simulation vs walltime mode
Simulation (default for Rust, Python, Node.js, C/C++): Deterministic CPU simulation, <1% variance, automatic flamegraphs. Best for CPU bound code. Does not measure system calls or I/O.
Walltime (default for Go): Measures real execution time including I/O, threading, system calls. Best for I/O heavy or multi threaded code. Requires consistent hardware (use CodSpeed Macro Runners in CI).
Memory : Tracks heap allocations. Best for reducing memory usage. Supported for Rust, C/C++ with libc/jemalloc/mimalloc.
Step 3: Set up the harness
Rust with divan (recommended)
1. Add the dependency:
2. Create a benchmark file in benches/ :
3. Add to Cargo.toml :
4. Build and run:
Rust with criterion
1. Add dependencies:
2. Create benchmark in benches/ :
3. Add to Cargo.toml and build/run same as divan.
Python with pytest codspeed
1. Install:
2. Create benchmark tests:
3. Run:
Node.js with vitest (recommended)
1. Install:
2. Configure vitest ( vitest.config.ts ):
3. Create benchmark file:
4. Run:
Go
No packages needed — CodSpeed instruments go test bench directly.
1. Create benchmark tests:
2. Run (walltime is the default for Go):
C/C++ with Google Benchmark
1. Install Google Benchmark (via CMake FetchContent or system package)
2. Create benchmark:
3. Build and run with CodSpeed:
Exec harness (any language)
For benchmarking whole programs without code changes:
1. Create codspeed.yml :
2. Run:
Or for a one off:
Step 4: Write good benchmarks
Good benchmarks are representative, isolated, and stable. Here are guidelines:
Benchmark real workloads : Use realistic input data and sizes. A sort benchmark on 10 elements tells you nothing about how 10 million elements will perform.
Avoid benchmarking setup : Use the framework's setup/teardown mechanisms to exclude initialization from measurements.
Prevent dead code elimination : Use black box() (Rust), benchmark::DoNotOptimize (C++), or Blackhole.consume (JMH) so the compiler doesn't optimize away unused results.
Cover the critical path : Benchmark the functions that matter most to your users — the ones called frequently or on the hot path.
Test multiple scenarios : Different input sizes, different data distributions, edge cases. Performance characteristics often change with scale.
Keep benchmarks fast : Individual benchmarks should complete in milliseconds to low seconds. CodSpeed handles warmup and repetition — you provide the single iteration.
Step 5: Verify and run
After setting up:
1. Run the benchmarks locally to verify they work:
2. Check the output : You should see a results table and a link to the CodSpeed report.
3. Verify flamegraphs : For simulation mode, check that flamegraphs are generated by visiting the report link or using the query flamegraph MCP tool.
4. Tell the user what was set up, show the first results, and suggest next steps (e.g., adding CI integration, running the optimize skill).