Sources of research: what to monitor and how

Research Question

What are the best sources for AI/ML research, and what is the right monitoring strategy for each — RSS, YouTube channels, arXiv, newsletters, GitHub?

Findings

Executive Summary

The most reliable automated AI/ML research sources for a GitHub Actions–based pipeline are RSS feeds from practitioner blogs and lab blogs — all confirmed accessible from runner IPs — plus the already-configured arxiv-mcp-server for targeted paper queries during agent research sessions. YouTube channel Atom feeds are blocked from GitHub Actions cloud IPs (confirmed independently from prior research) and cannot drive automated monitoring. arXiv RSS feeds work from runner IPs and deliver 300–430 papers per category per weekday, but this volume requires keyword filtering before automated ingestion is practical. Seven RSS feeds should be added to config/sources.yaml immediately using existing fetcher infrastructure; YouTube channels should remain empty until a channel-discovery mechanism that works from cloud IPs is built.

Key Findings

1. arXiv RSS feeds at export.arxiv.org are accessible from GitHub Actions runner IPs and deliver 300–430 new papers per weekday per category (cs.AI: ~348, cs.LG: ~307, combined cs.LG+cs.CL: ~431 on 2026-03-05). (confidence: high)
2. YouTube channel Atom feeds (www.youtube.com) return HTTP 404 from GitHub Actions runner IPs, confirming the prior-research finding — YouTube blocks cloud provider IP ranges for this endpoint. (confidence: high)
3. Seven practitioner RSS feeds confirmed accessible from runner IPs — Hugging Face Blog, Lil'Log, DeepMind Blog, The Gradient, BAIR Blog, Simon Willison, Sebastian Raschka — all return valid XML and are suitable for immediate addition to config/sources.yaml. (confidence: high)
4. The arxiv-mcp-server (v0.3.2) is already configured in .github/mcp.json but is not installed; it is designed for agent-driven, query-on-demand paper access rather than bulk daily ingestion, making it the right tool for targeted arXiv research within the research loop. (confidence: high)
5. Hugging Face Papers (Hugging Face Daily Papers) has no RSS endpoint and cannot be monitored via the existing RSS fetcher without building a scraper; the HF Blog RSS is a partial substitute covering HF-produced content but not the community paper voting. (confidence: high)
6. OpenAI and Anthropic do not expose working RSS feeds for their research blogs (OpenAI returns HTML at /blog/rss.xml; Anthropic returns 404); primary-source monitoring of these labs requires either scraping or manual tracking. (confidence: high)
7. Unfiltered arXiv RSS ingestion is not practical for a single-owner personal research system: cs.AI alone produces ~350 papers/weekday, yielding ~1,750/week that would need to be processed or discarded. A keyword-filter step in the pipeline, or the query-on-demand arxiv-mcp-server model, is required. (confidence: high)
8. GitHub trending has no official RSS or API endpoint; release monitoring for specific tracked repos is possible via github.com, which is a viable source type for tracking specific framework versions (e.g., transformers, vllm). (confidence: high)
9. The existing rss.sources section of config/sources.yaml is empty and can be populated immediately using the existing RSS fetcher without any code changes. (confidence: high)
10. youtube.channels entries in config/sources.yaml cannot support automated video discovery from GitHub Actions; they would only be functional when run locally or via a runner without cloud IP restrictions. (confidence: high)

Assumptions

• Assumption: The single-day arXiv paper count (348, 307, 431) is representative of typical weekday volume. Justification: arXiv submission volumes are stable over months, with minor variation; a single sample is sufficient for order-of-magnitude planning. The March 2025 estimate is not expected to change materially within the scope of this repo's monitoring.
• Assumption: The companion repo davidamitchell/Latest-developments- monitors a set of YouTube channels that would be relevant to add here. Justification: The companion repo's stated purpose is YouTube channel monitoring; however, since config/sources.yaml was inaccessible (GitHub API auth failure), the specific channels are unknown. This does not affect the recommendation: populate channels in config/sources.yaml for any known channel IDs, accepting that automated monitoring from GitHub Actions won't work until the runner IP issue is resolved.
• Assumption: The seven accessible RSS feeds represent stable, ongoing publications that will continue to produce AI/ML–relevant content. Justification: All are established sources (Lil'Log: 2018–present; HF Blog: 2021–present; DeepMind Blog: ongoing; etc.) with track records of years. Frequency varies but all have posted within 2025.

Analysis

The source selection trade-off is between coverage and manageability. arXiv provides the broadest academic coverage but at volumes that require either AI-assisted filtering or a query-on-demand model. The practitioner blogs (Lil'Log, Sebastian Raschka, Chip Huyen) produce far less content but at a higher concentration of relevance for this owner's research themes (AI strategy, agents, consciousness, ML engineering). The practical recommendation is: populate RSS with low-to-medium-volume high-quality feeds immediately; address arXiv volume through the arxiv-mcp-server (query-driven) rather than RSS ingestion.

YouTube channel monitoring's failure mode is not signal quality (YouTube practitioner content is high-value) but infrastructure: the Atom feed endpoint is blocked at the IP level. This is a solvable problem (yt-dlp can list channel videos; the YouTube Data API returns channel video lists), but the solution belongs in a separate backlog item.

The two missing primary sources (OpenAI, Anthropic) lack RSS endpoints. This is unlikely to change. The correct monitoring strategy for these is: monitor their GitHub repos for paper releases, or follow key researchers' arxiv submissions directly.

Risks, Gaps, and Uncertainties

• The companion repo's source list is unknown — it may already include channels or feeds that should be carried over. This remains a gap until the repo is accessible.
• arXiv RSS volume management is deferred — no keyword filter exists yet. Until it is built, adding arXiv RSS to config/sources.yaml would flood the pipeline.
• YouTube channel monitoring remains broken from GitHub Actions. A credible fix path exists (yt-dlp --flat-playlist or YouTube Data API v3 for channel listing) but has not been prototyped.
• Some feeds (DeepMind Blog, BAIR Blog) may have low posting frequency for extended periods if lab research cycles slow. The RSS fetcher will simply return no new items; no failure, but also no coverage during gaps.
• Hugging Face Papers community votes represent the ML community's current focus, which is a high-signal indicator not captured by any RSS-accessible feed. This gap is structural — HF does not expose Papers as RSS.

Open Questions

1. What YouTube channel IDs are monitored in davidamitchell/Latest-developments-? (Low priority to answer directly; medium priority to carry over any relevant channels to youtube.channels once runner IP issue is addressed.)
2. Should arXiv RSS monitoring be implemented with a keyword filter, or is query-on-demand via arxiv-mcp-server sufficient for this repo's scale? (Depends on how the research loop evolves; becomes relevant when the loop processes 10+ items/week.)
3. Is there a reliable method to extract the HF Daily Papers list via a web scrape, given no RSS exists? (Low priority; the HF Blog RSS partially covers this.)
4. Can github.com/<owner>/<repo>/releases.atom feeds be added to config/sources.yaml and processed by the existing RSS fetcher? If so, which repos are worth tracking (e.g., huggingface/transformers, vllm-project/vllm, openai/openai-python)?

sources

• [ ] davidamitchell/Latest-developments-/config/sources.yaml — inaccessible (GitHub API auth unavailable in this session)
• [x] Hugging Face Daily Papers — HTML page, no RSS; 200 OK
• [x] arXiv RSS feeds — verified working; export.arxiv.org/rss/cs.AI returns 348 papers/day; cs.LG returns 307; combined cs.LG+cs.CL returns 431
• [x] arxiv MCP server: arxiv-mcp-server PyPI package v0.3.2 by @blazickjp — MCP interface for arXiv search; already configured in .github/mcp.json