Generate tech news digests with unified source model, quality scoring, and multi-format output. Six-source data collection from RSS feeds, Twitter/X KOLs, GitHub releases, GitHub Trending, Reddit, and web search. Pipeline-based scripts with retry mechanisms and deduplication. Supports Discord, email

Tech News Digest Evidence led tech digest with unified collection and one editorial selection shared across formats. Follow references/digest prompt.md as the authoritative mode policy. Editorial Contract Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never filler. Optimize content within sections: a request to shorten GitHub releases must not become a whole report redesign or one message digest. Weekly ≤35 unique items: overview; configured topic sections in configured order (normally LLM, AI Agent, Crypto, Frontier Tech), with 2–4 substantial syntheses per topic as an advisory aim. Each synthesis uses 2–4 concise sentences joining week progress, evidence and implications, not concatenated headlines. Retain empty topic, key release, project and viewpoint/reading headings with honest non bullet coverage notes. Key releases ≤5 repositories (Chinese explanation ≤80 characters); projects ≤3; viewpoints/deep reading ≤4; optional next week watch ≤3 pending releases, fixes or uncertainties. Do not force experiments; factual watch claims need citations, clearly proposed/not performed experiments retain the narrow citation exemption. Caps are not quotas. Count evidence events inside syntheses and standalone watch items. Preserve topic sections in both modes; no global three theme constraint. Body limits: daily ≤6500 characters, weekly ≤10000, excluding URL targets; never fill to the limit. Mandatory security coverage: follow the primary alert/search and pre delivery checks in references/digest prompt.md . Preserve existing topics; use optional 🛡️ 安全警报 after the overview only for credible urgent incidents in either mode; weekly may also use an optional dedicated security/risk section for consequential nonurgent risks. Omit when empty and count/deduplicate within existing caps. Log checks and decisions; failed collection is not evidence of no incidents. Relevance, impact and evidence before internal scores. Exceptional claims need original technical evidence and independent assessment where possible, otherwise exclude/downgrade. No public scores, social metrics or operational statistics. Daily keeps substantive, nonduplicative KOL viewpoints. A material coverage gap may receive one concise caveat without raw counts; details stay in the final operational log. Deduplicate events and canonical URLs across the whole report. Daily repeats require incremental developments; weekly may synthesize daily coverage. Daily keeps blog picks; weekly viewpoints/deep reading may include original opinions, papers, docs, postmortems or essays. Project discovery is not verified trending; no lifetime derived growth or repeated mature repos without meaningful changes. Preserve the prompt's consequential release rules, exact versions and official links. Plain bullets with bold titles; concise inline Discord links [来源](<URL ) . Section aware chunks ≤1700 characters including numbering. Apply the requested language, including Simplified Chinese explanations when configured. Validate the canonical archive with python3 scripts/validate digest.py input FILE mode daily or mode weekly before delivery; fix all errors. Weekly production runs append one topic heading 'EXACT LOCALIZED HEADING' per configured topic in order; without these flags the gate cannot infer custom headings. Missing/failing validation blocks delivery. Manually verify unique event counts inside syntheses, required non topic headings/overview, security placement, evidence/coverage honesty and cross format parity too. Discord, Markdown, email and PDF use identical selections/judgments. Operational stats/runtime attribution belong in the final log, not content. Legacy ITEMS PER SECTION , BLOG PICKS COUNT and EXTRA SECTIONS are advisory parameters superseded by mode policy: ignore their counts/extra sections, especially duplicate weekly trends. Quick Start 1. Configuration Setup : Default configs are in config/defaults/ . Copy to workspace for customization: 2. Environment Variables : TWITTERAPI IO KEY twitterapi.io API key (optional, preferred) X BEARER TOKEN Twitter/X official API bearer token (optional, fallback) TAVILY API KEY Tavily Search API key, alternative to Brave (optional) WEB SEARCH BACKEND Web search backend: auto brave tavily (optional, default: auto) BRAVE API KEYS Brave Search API keys, comma separated for rotation (optional) BRAVE API KEY Single Brave key fallback (optional) GITHUB TOKEN GitHub personal access token (optional, improves rate limits) 3. Generate Digest : 4. Select and Validate : Follow references/digest prompt.md ; validate the canonical Markdown before rendering the same selections to Discord, email and PDF. Configuration Files sources.json Unified Data Sources topics.json Enhanced Topic Definitions Scripts Pipeline run pipeline.py Unified Pipeline (Recommended) Features : Runs all 6 fetch steps in parallel, then merges + deduplicates + scores Output : Final merged JSON ready for report generation (~30s total) Metadata : Saves per step timing and counts to .meta.json GitHub Auth : Auto generates GitHub App token if $GITHUB TOKEN not set Fallback : If this fails, run individual scripts below Individual Scripts (Fallback) fetch rss.py RSS Feed Fetcher Parallel fetching (10 workers, max 2 concurrent per host), retry with jittered backoff, feedparser + regex fallback Timeout: 30s per feed, ETag/Last Modified caching in $TECH NEWS DIGEST STATE DIR fetch twitter.py Twitter/X KOL Monitor Backend auto detection: uses twitterapi.io if TWITTERAPI IO KEY set, else official X API v2 if X BEARER TOKEN set Rate limit handling, engagement metrics, retry with backoff fetch web.py Web Search Engine Auto detects Brave API rate limit: paid plans → parallel queries, free → sequential Without API: generates search interface for agents fetch github.py GitHub Releases Monitor Parallel fetching (10 workers), 30s timeout Auth priority: $GITHUB TOKEN → GitHub App auto generate → gh CLI → releases.atom Without a token (or on a 403/429 rate limit) it reads github.com/<repo /releases.atom , which needs no auth, consumes no REST quota, and also surfaces tag only repos such as torvalds/linux The Atom feed lists every tag, so entries are kept only when the tag carries a <major .<minor version ( v2.13.0 , v6.19 rc4 ). This drops CI tags such as PyTorch's trunk/<sha and viable/strict/<epoch , which would otherwise crowd out real releases fetch github.py trending Project Discovery (legacy flag) Searches GitHub API for discovery candidates; search results do not establish genuine trends. Neutral internal base score 5, not a growth signal. No daily stars est or lifetime derived growth. Daily discovery ≤3, weekly ≤3; require a current reason to try the project. fetch reddit.py Reddit Posts Fetcher Parallel fetching (4 workers), public JSON API (no auth required) 13 subreddits with score filtering enrich articles.py Article Full Text Enrichment Fetches full article text for high scoring articles Cloudflare Markdown for Agents (preferred) → HTML extraction (fallback) → Skip (paywalled/social) Blog domain whitelist with lower score threshold (≥3) Parallel fetching (5 workers, 10s timeout) merge sources.py Quality Scoring & Deduplication Quality scoring, URL dedup (tracking params stripped, query preserved), title similarity dedup (75%), previous digest penalty Multi source bonus: +5 per additional source type corroborating the same story Topic priority is derived from topics.json order; pass defaults / config to keep it in sync Output: topic grouped articles sorted by score validate config.py Configuration Validator JSON schema validation, topic reference checks, duplicate ID detection generate pdf.py PDF Report Generator Converts markdown digest to styled A4 PDF with Chinese typography (Noto Sans CJK SC) Emoji icons, page headers/footers, blue accent theme. Requires weasyprint . sanitize html.py Safe HTML Email Converter Converts markdown to XSS safe HTML email with inline CSS URL whitelist (http/https only), HTML escaped text content source health.py Source Health Monitor Tracks per source success/failure history over 7 days in $TECH NEWS DIGEST STATE DIR/source health.json Reports unhealthy sources ( 50% failure rate, minimum 2 samples) summarize merged.py Merged Data Summary Human readable summary of merged data for LLM consumption Shows top articles per topic with scores and metrics User Customization Workspace Configuration Override Place custom configs in workspace/config/ to override defaults: Sources : Append new sources, disable defaults with "enabled": false Topics : Override topic definitions, search queries, display settings Merge Logic : Sources with same id → user version takes precedence Sources with new id → appended to defaults Topics with same id → user version completely replaces default Example Workspace Override Templates & Output Canonical Markdown ( references/templates/markdown.md ) Daily/weekly structure governed by references/digest prompt.md ; one validated selection for all formats. Discord Template ( references/templates/discord.md ) Plain bullets, bold titles, inline [来源](<URL ) links; section aware chunks ≤1700 characters including numbering. Email Template ( references/templates/email.md ) Convert validated Markdown with sanitize html.py , send the HTML file with send email.py . Same selections, no extra metadata sections or operational statistics. PDF Template ( references/templates/pdf.md ) Same canonical report, A4 Chinese typography and page numbers. Generated via scripts/generate pdf.py (requires weasyprint ); no PDF only content. Default Sources (216 enabled) RSS Feeds (93) : AI labs, arXiv, research blogs, tech news, crypto news, Chinese tech media, YouTube channels Twitter/X KOLs (76) : AI researchers, model labs, crypto leaders, tech executives GitHub Repos (47) : Major open source projects (llama.cpp, vLLM, SGLang, LangGraph, MCP, etc.) Reddit (13, disabled by default) : Reddit blocks datacenter IPs; set REDDIT CLIENT ID / REDDIT CLIENT SECRET and re enable in your overlay Web Search (4 topics) : LLM, AI Agent, Crypto, Frontier Tech A further 7 RSS/GitHub defaults ship disabled because the upstream feed is dead; each carries a note saying why. All sources pre configured with appropriate topic tags and priority levels. Dependencies Optional but Recommended : feedparser =6.0.0 Better RSS parsing (fallback to regex if unavailable) jsonschema =4.0.0 Configuration validation All scripts work with Python 3.8+ standard library only. Monitoring & Operations Health Checks Archive Management Digests automatically archived to <workspace /archive/tech news digest/ Previous digest titles used for duplicate detection Old archives cleaned automatically (90+ days) Error Handling Network Failures : Retry with exponential backoff Rate Limits : Automatic retry with appropriate delays Invalid Content : Graceful degradation, detailed logging Configuration Errors : Schema validation with helpful messages API Keys & Environment Set in ~/.zshenv or similar: Twitter : TWITTERAPI IO KEY preferred ($3 5/m

tech-news-digest