Research Paper Writing Pipeline End to end pipeline for producing publication ready ML/AI research papers targeting NeurIPS, ICML, ICLR, ACL, AAAI, and COLM . This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission. This is not a linear pipeline — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops. <! ascii guard ignore <! ascii guard ignore end When To Use This Skill Use this skill when: Starting a new research paper from an existing codebase or idea Designing and running experiments to support paper claims Writing or revising any section of a research paper Preparing for submission to a specific conference or workshop Responding to reviews with additional experiments or revisions Converting a paper between conference formats Writing non empirical papers — theory, survey, benchmark, or position papers (see [Paper Types Beyond Empirical ML]( paper types beyond empirical ml)) Designing human evaluations for NLP, HCI, or alignment research Preparing post acceptance deliverables — posters, talks, code releases Core Philosophy 1. Be proactive. Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate. 2. Never hallucinate citations. AI generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as [CITATION NEEDED] . 3. Paper is a story, not a collection of experiments. Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready. 4. Experiments serve claims. Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative. 5. Commit early, commit often. Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history. Proactivity and Collaboration Default: Be proactive. Draft first, ask with the draft. Confidence Level Action High (clear repo, obvious contribution) Write full draft, deliver, iterate on feedback Medium (some ambiguity) Write draft with flagged uncertainties, continue Low (major unknowns) Ask 1 2 targeted questions via clarify , then draft Section Draft Autonomously? Flag With Draft Abstract Yes "Framed contribution as X — adjust if needed" Introduction Yes "Emphasized problem Y — correct if wrong" Methods Yes "Included details A, B, C — add missing pieces" Experiments Yes "Highlighted results 1, 2, 3 — reorder if needed" Related Work Yes "Cited papers X, Y, Z — add any I missed" Block for input only when : target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first. Phase 0: Project Setup Goal : Establish the workspace, understand existing work, identify the contribution. Step 0.1: Explore the Repository Look for: README.md — project overview and claims results/ , outputs/ , experiments/ — existing findings configs/ — experimental settings .bib files — existing citations Draft documents or notes Step 0.2: Organize the Workspace Establish a consistent workspace structure: Step 0.3: Set Up Version Control Git discipline : Every completed experiment batch gets committed with a descriptive message. Example: Step 0.4: Identify the Contribution Before writing anything, articulate: The What : What is the single thing this paper contributes? The Why : What evidence supports it? The So What : Why should readers care? Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?" Step 0.5: Create a TODO List Use the todo tool to create a structured project plan: Update this throughout the project. It serves as the persistent state across sessions. Step 0.6: Estimate Compute Budget Before running experiments, estimate total cost and time: Track actual spend as experiments run: When budget is tight : Run pilot experiments (1 2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs. Step 0.7: Multi Author Coordination Most papers have 3 10 authors. Establish workflows early: Workflow Tool When to Use Overleaf Browser based Multiple authors editing simultaneously, no git experience Git + LaTeX git with .gitignore for aux files Technical teams, need branch based review Overleaf + Git sync Overleaf premium Best of both — live collab with version history Section ownership : Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency. LaTeX conventions to agree on early : \method{} macro for consistent method naming Citation style: \citet{} vs \citep{} usage Math notation: lowercase bold for vectors, uppercase bold for matrices, etc. British vs American spelling Phase 1: Literature Review Goal : Find related work, identify baselines, gather citations. Step 1.1: Identify Seed Papers Start from papers already referenced in the codebase: Step 1.2: Search for Related Work Load the arxiv skill for structured paper discovery: skill view("arxiv") . It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation. Use web search for broad discovery, web extract for fetching specific papers: Additional search queries to try: Recommended : Install Exa MCP for real time academic search: Step 1.2b: Deepen the Search (Breadth First, Then Depth) A flat search (one round of queries) typically misses important related work. Use an iterative breadth then depth pattern inspired by deep research pipelines: When to stop : If a round returns 80% papers already in your collection, the search is saturated. Typically 2 3 rounds suffice. For survey papers, expect 4 5 rounds. For agent based workflows : Delegate each round's queries in parallel via delegate task . Collect results, deduplicate, then generate the next round's queries from the combined learnings. Step 1.3: Verify Every Citation NEVER generate BibTeX from memory. ALWAYS fetch programmatically. For each citation, follow the mandatory 5 step process: If you cannot verify a citation: Always tell the scientist : "I've marked [X] citations as placeholders that need verification." See [references/citation workflow.md](references/citation workflow.md) for complete API documentation and the full CitationManager class. Step 1.4: Organize Related Work Group papers by methodology, not paper by paper: Good : "One line of work uses X's assumption [refs] whereas we use Y's assumption because..." Bad : "Smith et al. introduced X. Jones et al. introduced Y. We combine both." Phase 2: Experiment Design Goal : Design experiments that directly support paper claims. Every experiment must answer a specific question. Step 2.1: Map Claims to Experiments Create an explicit mapping: Claim Experiment Expected Evidence "Our method outperforms baselines" Main comparison (Table 1) Win rate, statistical significance "Effect is larger for weaker models" Model scaling study Monotonic improvement curve "Convergence requires scope constraints" Constrained vs unconstrained Convergence rate comparison Rule : If an experiment doesn't map to a claim, don't run it. Step 2.2: Design Baselines Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?" Standard baseline categories: Naive baseline : Simplest possible approach Strong baseline : Best known existing method Ablation baselines : Your method minus one component Compute matched baselines : Same compute budget, different allocation Step 2.3: Define Evaluation Protocol Before running anything, specify: Metrics : What you're measuring, direction symbols (higher/lower better) Aggregation : How results are combined across runs/tasks Statistical tests : What tests will establish significance Sample sizes : How many runs/problems/tasks Step 2.4: Write Experiment Scripts Follow these patterns from successful research pipelines: Incremental saving — save results after each step for crash recovery: Artifact preservation — save all intermediate outputs: Separation of concerns — keep generation, evaluation, and visualization separate: See [references/experiment patterns.md](references/experiment patterns.md) for complete design patterns, cron monitoring, and error recovery. Step 2.5: Design Human Evaluation (If Applicable) Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automated experiments — human eval often has longer lead times (IRB approval, annotator recruitment). When human evaluation is needed: Automated metrics don't capture what you care about (fluency, helpfulness, safety) Your contribution is about human facing qualities (readability, preference, trust) Reviewers at NLP venues (ACL, EMNLP) expect it for generation tasks Key design decisions: Decision Options Guidance Annotator type Expert, crowdworker, end user Match to what your claims require Scale Likert (1 5), pairwise comparison, ranking Pairwise is more reliable than Likert for LLM outputs Sample size Per annotator and total items Power analysis or minimum 100 items, 3+ annotators Agreement metric Cohen's kappa, Krippendorff's alpha, ICC Krippendorff's alpha for 2 annotators; report raw agreement too Platform Prolific, MTurk, internal team Prolific for quality; MTurk for scale; internal for domain expertise Annotation guideline checklist: Reporting requirements (reviewers check all of these): Number of annotators and their qualifications Inter annotator agreement with specific metric and value Compensation details (amount, estimated hourly rate) Annotation interface description or screenshot (appendix) Total annotation time See [references/human evaluation.md](references/human evaluation.md) for complete guide including statistical tests for human eval data, crowdsourcing quality control patterns, and IRB guidance. Phase 3: Experiment Execution & Monitoring Goal : Run experiments reliably, monitor progress, recover from failures. Step 3.1: Launch Experiments Use nohup for long running experiments: Parallel execution : Run independent experiments simultaneously, but be aware of API rate limits. 4+ concurrent experiments on the same API will slow each down. Step 3.2: Set Up Monitoring (Cron Pattern) For long running experiments, set up periodic status checks. The cron prompt should follow this template: Silent mode : If nothing has changed since the last check, respond with [SILENT] to suppress notification to the user. Only report when there's news. Step 3.3: Handle Failures Common failure modes and recovery: Failure Detection Recovery API rate limit / credit exhaustion 402/429 errors in logs Wait, then re run (scripts skip completed work) Process crash PID gone, incomplete results Re run from last checkpoint Timeout on hard problems Process stuck, no log progress Kill and skip, note in results Wrong model ID Errors referencing model name Fix ID and re run Key : Scripts should always check for existing results and skip completed work. This makes re runs safe

research-paper-writing