Generate tech news digests with unified source model, quality scoring, and multi-format output. Six-source data collection from RSS feeds, Twitter/X KOLs, GitHub releases, GitHub Trending, Reddit, and web search. Pipeline-based scripts with retry mechanisms and deduplication. Supports Discord, email

Tech News Digest Automated tech news digest system with unified data source model, quality scoring pipeline, and template based output generation. Quick Start 1. Configuration Setup : Default configs are in config/defaults/ . Copy to workspace for customization: 2. Environment Variables : TWITTERAPI IO KEY twitterapi.io API key (optional, preferred) X BEARER TOKEN Twitter/X official API bearer token (optional, fallback) TAVILY API KEY Tavily Search API key, alternative to Brave (optional) WEB SEARCH BACKEND Web search backend: auto brave tavily (optional, default: auto) BRAVE API KEYS Brave Search API keys, comma separated for rotation (optional) BRAVE API KEY Single Brave key fallback (optional) GITHUB TOKEN GitHub personal access token (optional, improves rate limits) 3. Generate Digest : 4. Use Templates : Apply Discord, email, or PDF templates to merged output Configuration Files sources.json Unified Data Sources topics.json Enhanced Topic Definitions Scripts Pipeline run pipeline.py Unified Pipeline (Recommended) Features : Runs all 6 fetch steps in parallel, then merges + deduplicates + scores Output : Final merged JSON ready for report generation (~30s total) Metadata : Saves per step timing and counts to .meta.json GitHub Auth : Auto generates GitHub App token if $GITHUB TOKEN not set Fallback : If this fails, run individual scripts below Individual Scripts (Fallback) fetch rss.py RSS Feed Fetcher Parallel fetching (10 workers, max 2 concurrent per host), retry with jittered backoff, feedparser + regex fallback Timeout: 30s per feed, ETag/Last Modified caching in $TECH NEWS DIGEST STATE DIR fetch twitter.py Twitter/X KOL Monitor Backend auto detection: uses twitterapi.io if TWITTERAPI IO KEY set, else official X API v2 if X BEARER TOKEN set Rate limit handling, engagement metrics, retry with backoff fetch web.py Web Search Engine Auto detects Brave API rate limit: paid plans → parallel queries, free → sequential Without API: generates search interface for agents fetch github.py GitHub Releases Monitor Parallel fetching (10 workers), 30s timeout Auth priority: $GITHUB TOKEN → GitHub App auto generate → gh CLI → releases.atom Without a token (or on a 403/429 rate limit) it reads github.com/<repo /releases.atom , which needs no auth, consumes no REST quota, and also surfaces tag only repos such as torvalds/linux The Atom feed lists every tag, so entries are kept only when the tag carries a <major .<minor version ( v2.13.0 , v6.19 rc4 ). This drops CI tags such as PyTorch's trunk/<sha and viable/strict/<epoch , which would otherwise crowd out real releases fetch github.py trending GitHub Trending Repos Searches GitHub API for trending repos across 4 topics (LLM, AI Agent, Crypto, Frontier Tech) Quality scoring: base 5 + daily stars est / 10, max 15 fetch reddit.py Reddit Posts Fetcher Parallel fetching (4 workers), public JSON API (no auth required) 13 subreddits with score filtering enrich articles.py Article Full Text Enrichment Fetches full article text for high scoring articles Cloudflare Markdown for Agents (preferred) → HTML extraction (fallback) → Skip (paywalled/social) Blog domain whitelist with lower score threshold (≥3) Parallel fetching (5 workers, 10s timeout) merge sources.py Quality Scoring & Deduplication Quality scoring, URL dedup (tracking params stripped, query preserved), title similarity dedup (75%), previous digest penalty Multi source bonus: +5 per additional source type corroborating the same story Topic priority is derived from topics.json order; pass defaults / config to keep it in sync Output: topic grouped articles sorted by score validate config.py Configuration Validator JSON schema validation, topic reference checks, duplicate ID detection generate pdf.py PDF Report Generator Converts markdown digest to styled A4 PDF with Chinese typography (Noto Sans CJK SC) Emoji icons, page headers/footers, blue accent theme. Requires weasyprint . sanitize html.py Safe HTML Email Converter Converts markdown to XSS safe HTML email with inline CSS URL whitelist (http/https only), HTML escaped text content source health.py Source Health Monitor Tracks per source success/failure history over 7 days in $TECH NEWS DIGEST STATE DIR/source health.json Reports unhealthy sources ( 50% failure rate, minimum 2 samples) summarize merged.py Merged Data Summary Human readable summary of merged data for LLM consumption Shows top articles per topic with scores and metrics User Customization Workspace Configuration Override Place custom configs in workspace/config/ to override defaults: Sources : Append new sources, disable defaults with "enabled": false Topics : Override topic definitions, search queries, display settings Merge Logic : Sources with same id → user version takes precedence Sources with new id → appended to defaults Topics with same id → user version completely replaces default Example Workspace Override Templates & Output Discord Template ( references/templates/discord.md ) Bullet list format with link suppression ( <link ) Mobile optimized, emoji headers 2000 character limit awareness Email Template ( references/templates/email.md ) Rich metadata, technical stats, archive links Executive summary, top articles section HTML compatible formatting PDF Template ( references/templates/pdf.md ) A4 layout with Noto Sans CJK SC font for Chinese support Emoji icons, page headers/footers with page numbers Generated via scripts/generate pdf.py (requires weasyprint ) Default Sources (213 enabled) RSS Feeds (93) : AI labs, arXiv, research blogs, tech news, crypto news, Chinese tech media, YouTube channels Twitter/X KOLs (73) : AI researchers, model labs, crypto leaders, tech executives GitHub Repos (47) : Major open source projects (llama.cpp, vLLM, SGLang, LangGraph, MCP, etc.) Reddit (13, disabled by default) : Reddit blocks datacenter IPs; set REDDIT CLIENT ID / REDDIT CLIENT SECRET and re enable in your overlay Web Search (4 topics) : LLM, AI Agent, Crypto, Frontier Tech A further 7 RSS/GitHub defaults ship disabled because the upstream feed is dead; each carries a note saying why. All sources pre configured with appropriate topic tags and priority levels. Dependencies Optional but Recommended : feedparser =6.0.0 Better RSS parsing (fallback to regex if unavailable) jsonschema =4.0.0 Configuration validation All scripts work with Python 3.8+ standard library only. Monitoring & Operations Health Checks Archive Management Digests automatically archived to <workspace /archive/tech news digest/ Previous digest titles used for duplicate detection Old archives cleaned automatically (90+ days) Error Handling Network Failures : Retry with exponential backoff Rate Limits : Automatic retry with appropriate delays Invalid Content : Graceful degradation, detailed logging Configuration Errors : Schema validation with helpful messages API Keys & Environment Set in ~/.zshenv or similar: Twitter : TWITTERAPI IO KEY preferred ($3 5/mo); X BEARER TOKEN as fallback; auto mode tries twitterapiio first Web Search : Tavily (preferred in auto mode) or Brave; optional, fallback to agent web search if unavailable GitHub : Auto generates token from GitHub App if PAT not set; unauthenticated fallback (60 req/hr) Reddit : Disabled by default — Reddit returns 403 to datacenter IPs. Set REDDIT CLIENT ID / REDDIT CLIENT SECRET and re enable the sources to use the OAuth API Cron / Scheduled Task Integration OpenClaw Cron (Recommended) The cron prompt should NOT hardcode the pipeline steps. Instead, reference references/digest prompt.md and only pass configuration parameters. This ensures the pipeline logic stays in the skill repo and is consistent across all installations. Daily Digest Cron Prompt Weekly Digest Cron Prompt Why This Pattern? Single source of truth : Pipeline logic lives in digest prompt.md , not scattered across cron configs Portable : Same skill on different OpenClaw instances, just change paths and channel IDs Maintainable : Update the skill → all cron jobs pick up changes automatically Anti pattern : Do NOT copy pipeline steps into the cron prompt — it will drift out of sync Multi Channel Delivery Limitation OpenClaw enforces cross provider isolation : a single session can only send messages to one provider (e.g., Discord OR Telegram, not both). If you need to deliver digests to multiple platforms, create separate cron jobs for each provider: Replace DISCORD CHANNEL ID delivery with the target platform's delivery in the second job's prompt. This is a security feature, not a bug — it prevents accidental cross context data leakage. Security Notes Execution Model This skill uses a prompt template pattern : the agent reads digest prompt.md and follows its instructions. This is the standard OpenClaw skill execution model — the agent interprets structured instructions from skill provided files. All instructions are shipped with the skill bundle and can be audited before installation. Network Access The Python scripts make outbound requests to: RSS feed URLs (configured in tech news digest sources.json ) Twitter/X API ( api.x.com or api.twitterapi.io ) Brave Search API ( api.search.brave.com ) Tavily Search API ( api.tavily.com ) GitHub API ( api.github.com ) Reddit JSON API ( reddit.com ) No data is sent to any other endpoints. All API keys are read from environment variables declared in the skill metadata. Shell Safety Email delivery uses send email.py which constructs proper MIME multipart messages with HTML body + optional PDF attachment. Subject formats are hardcoded ( Daily Tech Digest YYYY MM DD ). PDF generation uses generate pdf.py via weasyprint . The prompt template explicitly prohibits interpolating untrusted content (article titles, tweet text, etc.) into shell arguments. Email addresses and subjects must be static placeholder values only. File Access Scripts read from config/ and write to workspace/archive/ . No files outside the workspace are accessed. Support & Troubleshooting Common Issues 1. RSS feeds failing : Check network connectivity, use verbose for details 2. Twitter rate limits : Reduce sources or increase interval 3. Configuration errors : Run validate config.py for specific issues 4. No articles found : Check time window ( hours ) and source enablement Debug Mode All scripts support verbose flag for detailed logging and troubleshooting. Performance Tuning Parallel Workers : Adjust MAX WORKERS in scripts for your system Timeout Settings : Increase TIMEOUT for slow networks Article Limits : Adjust MAX ARTICLES PER FEED based on needs Security Considerations Shell Execution The digest prompt instructs agents to run Python scripts via shell commands. All script paths and arguments are skill defined constants — no user input is interpolated into commands. Two scripts use subprocess : run pipeline.py orchestrates child fetch scripts (all within scripts/ directory) fetch github.py has two subprocess calls: 1. openssl dgst sha256 sign for JWT signing (only if GH APP env vars are set — signs a self constructed JWT payload, no user content involved) 2. gh auth token CLI fallback (only if gh is installed — reads from gh's own credential store) No user supplied or fetched content is ever interpolated into subprocess arguments. Email delivery uses send email.py whic

tech-news-digest