result-to-claim

Use when experiments complete to judge what claims the results support, what they don't, and what evidence is still missing. Codex MCP evaluates results against intended claims and routes to next action (pivot, supplement, or confirm). Use after experiments finish — before writing the paper or runni

By wanshuiyin · 405 installs

npx skills add wanshuiyin/auto-claude-code-research-in-sleep --skill result-to-claim

Source repository · Upstream listing

Result to Claim Gate 🔒 Do not wrap this skill in /loop , /schedule , or CronCreate . It is verdict bearing — it judges whether results support a claim. Re running that verdict on a wall clock timer adds no new signal (the verdict changes only when the results change, not when the clock ticks). What you actually want to schedule is the external wait that precedes it — experiments done → then run this gate once . See [ shared references/external cadence.md ](../shared references/external cadence.md). Experiments produce numbers; this gate decides what those numbers mean . Collect results from available sources, get a Codex judgment, then auto route based on the verdict. Context: $ARGUMENTS When to Use After a set of experiments completes (main results, not just sanity checks) Before committing to claims in a paper or review response When results are ambiguous and you need an objective second opinion Workflow Step 1: Collect Results Gather experiment data from whatever sources are available in the project: 1. W&B (preferred): wandb.Api().run("<entity /<project /<run id ").history() — metrics, training curves, comparisons 2. EXPERIMENT LOG.md : full results table with baselines and verdicts 3. EXPERIMENT TRACKER.md : check which experiments are DONE vs still running 4. Log files : ssh server "tail 100 /path/to/training.log" if no other source 5. idea stage/docs/research contract.md (legacy fallback: docs/research contract.md ): intended claims and experiment design Assemble the key information: What experiments were run (method, dataset, config) Main metrics and baseline comparisons (deltas) The intended claim these experiments were designed to test Any known confounds or caveats Step 1.5: Deterministic evidence pre check (before spending a Codex call) For every claim that cites a specific number + a source file, verify the evidence exists mechanically — no model call — to catch hallucinated evidence before the jury runs (see [ shared references/evidence precheck.md ](../shared references/evidence precheck.md)). 1. Build the claims list. From the cited numbers and their result files, write [{"id", "value", "source"}, ...] to .aris/claims.json ( source is the result file/glob relative to the project root; value is the cited number or string). 2. Run the pre check — this is a real step, not a suggestion. Execute the block below (resolver per integration contract §2, Policy B : warn and skip if the helper is unresolved — never block the audit): The output is {"results": [{id, value, source, status, ...}], "summary": {status: n}} with status ∈ {verified, value not found, path missing, unparseable} . 3. Act on the statuses. Any claim returned value not found or path missing is hallucinated evidence — mark it claim supported: no with integrity status: evidence not found immediately; do NOT spend a Codex call defending a number that isn't in the data. unparseable claims (no usable value/source) just go to the jury normally. 4. Carry the per claim status into Step 2. Feed a small evidence pre check: <id → verified value not found path missing unparseable table (from .aris/evidence precheck.json ) into the Step 2 Codex prompt so the jury knows which claims have real evidence to read. If the pre check was skipped (helper unresolved), say so in that slot rather than omitting it. verified here means only that the cited evidence exists — whether it supports the claim is still the Codex jury's call in Step 2 (a deterministic gate DRIVES, it does not ACQUIT). Step 2: Codex Judgment Send the collected results to Codex for objective evaluation. Include ONLY claims that passed the Step 1.5 pre check — claims already terminally rejected ( evidence not found ) keep their deterministic verdict and are NOT re litigated here: Step 3: Parse and Normalize Extract structured fields from Codex response: Step 3.5: Check Experiment Integrity (if audit exists) Skip this step if EXPERIMENT AUDIT.json does not exist. See shared references/experiment integrity.md for the full integrity protocol. Step 4: Route Based on Verdict no — Claim not supported 1. Record postmortem in findings.md (Research Findings section): What was tested, what failed, hypotheses for why Constraints for future attempts (what NOT to try again) 2. Update CLAUDE.md Pipeline Status 3. Decide whether to pivot to next idea from IDEA CANDIDATES.md or try an alternative approach partial — Claim partially supported 1. Update the working claim to reflect what IS supported 2. Record the gap in findings.md 3. Design and run supplementary experiments to fill evidence gaps 4. Re run result to claim after supplementary experiments complete 5. Multiple rounds of partial on the same claim → record analysis in findings.md, consider whether to narrow the claim scope or switch ideas yes — Claim supported 1. Record confirmed claim in project notes 2. If ablation studies are incomplete → trigger /ablation planner 3. If all evidence is in → ready for paper writing Step 5: Update Research Wiki (if active) Skip this step entirely if research wiki/ does not exist. If research wiki/ exists, resolve $WIKI SCRIPT per the canonical chain documented in [ shared references/wiki helper resolution.md ](../shared references/wiki helper resolution.md) (Variant B — warn and skip for caller skills). The verdict / idea outcome page edits below run on raw markdown and don't need the helper, but edges, query pack rebuild, and the log line do. This skill never edits a claim's status field and never creates a claim node — claims are born (and their proof status set) by /proof checker ; here we only attach experiment edges. Rules Codex is the judge, not CC. CC collects evidence and routes; Codex evaluates. This prevents post hoc rationalization. Do not inflate claims beyond what the data supports. If Codex says "partial", do not round up to "yes". A single positive result on one dataset does not support a general claim. Be honest about scope. If confidence is low, treat the judgment as inconclusive and add experiments rather than committing to a claim. Fail closed if the reviewer is unavailable. If the Codex call fails, first walk the capability fallback chain in shared references/reviewer routing.md ( gpt 6 astra + ultra → gpt 6 astra + xhigh → gpt 5.5 + xhigh , capability errors only). If no allowed pair succeeds: write CLAIMS FROM RESULTS.md containing ONLY the first line verdict: REVIEW UNAVAILABLE (a machine checkable gate for pipeline callers), record the same in findings.md, and STOP — CC never substitutes its own claim judgment (a loop can drive, never acquit; acceptance gate.md ). Downstream steps (wiki add experiment edges, ablation planner, paper claims) must not consume a run without a Codex verdict. Exception: the deterministic evidence pre check (Step 1.5) may still terminally mark a claim claim supported: no for hallucinated evidence — a deterministic rejection needs no reviewer; only SUPPORTIVE or ambiguous outcomes require one. Always record the verdict and reasoning in findings.md, regardless of outcome. Review Tracing After each mcp codex codex or mcp codex codex reply reviewer call, save the trace following shared references/review tracing.md (Policy C — forensic; never silently skip). Use save trace.sh (resolved per the chain in shared references/integration contract.md §2) or write files directly to .aris/traces/<skill /<date run<NN / . Respect the trace: parameter (default: full ).