Local index vs reference

Research Question

For each type of research content, should we store a local copy / index, or just maintain a reference (URL, citation)? What are the right trade-offs between storage cost, offline access, durability, and searchability?

Findings

Executive Summary

For a git-based, GitHub-hosted research corpus with a single owner reviewing diffs via the GitHub website, the correct policy is: store YouTube transcripts locally as plain text files, keep arXiv papers as reference-only (URL/ID), paste key passages from web pages inline into research items, and always commit notes and synthesis locally. This policy is derived from three criteria applied to each content type: whether on-demand fetching is reliable (transcripts are blocked on cloud IPs), whether the source URL is stable (arXiv is permanent; general web has a 2–14 year half-life depending on content type), and whether the file format is git-friendly (text yes, binary no). Git LFS is not needed: no anticipated content type approaches GitHub's 50 MiB warning threshold, and LFS adds operational complexity that conflicts with the owner's GitHub-website-only workflow.

Key Findings

1. YouTube transcripts must be stored locally as .txt files because cloud IP blocking makes on-demand re-fetching unreliable, and text files are fully git-diffable and accessible from the GitHub website without any special tooling. The existing Research/transcripts/ directory and fetch-transcript workflow implement this policy correctly.

2. arXiv papers must remain reference-only because PDFs are binary files that produce unreadable git diffs, arXiv URLs are permanently stable (the service has operated since 1991 with institutional backing), and the relevant content can be represented as text excerpts within research items rather than full PDFs. Key excerpts should be pasted inline when they are primary evidence.

3. Web pages that are primary sources for key findings should have their relevant 1–3 paragraph excerpt pasted directly into the research item's Context or Findings section at the time of research, because 38% of web pages from 2013 were inaccessible by 2023 (Pew Research, 2023) and 23% of news articles already contained dead URLs by 2023. The URL is still recorded for attribution; the inline text guards against evidence disappearing.

4. General web URLs that are peripheral sources (background reading, not direct evidence for a specific finding) should be recorded as references only, because the overhead of snapshotting every source is not proportionate to the benefit at current corpus scale.

5. Git LFS is not warranted for this repository: text transcripts at 40–150 KB per file and 1,000 transcripts would total 150 MB — well within GitHub's practical repository limits — and LFS adds complexity (separate client, quota management) that conflicts with the owner's GitHub-website-only workflow.

6. Binary files (PDFs, audio, video) must never be committed to the repository regardless of size, because they produce unreadable git diffs and this constraint is already established by ADR-0003 and the local database item; the policy documented here is consistent with that existing constraint.

7. The storage-vs-reference decision is fully determined by three criteria in order: (1) is on-demand fetching reliable? (2) does the source have meaningful link rot risk within a 5-year horizon? (3) is the format text (git-friendly) or binary (git-hostile)? Applying these criteria leaves no ambiguous cases for the four content types in scope.

8. A separate Research/snapshots/ directory for web page archives is not warranted at current corpus scale; the inline-paste pattern already observable in completed research items (e.g., 2026-02-28-ai-strategy.md) is sufficient and does not require new tooling or directory structure.

Assumptions

• Assumption: Average text transcript is 40–150 KB per file. Justification: Standard English word density for spoken transcripts is approximately 8,000–15,000 words per hour of content, and plain text encodes at ~5 bytes per word. No measured sample from this repository was available (the Research/transcripts/ directory is currently empty). If transcripts are consistently shorter (sub-30-minute videos) the size drops proportionally; this does not change the storage recommendation.

• Assumption: The YouTube video deletion rate represents a "medium" link rot risk for this content type. Justification: No systematic study of YouTube video deletion rates was found. The "medium" label is conservative relative to general web content (high) and arXiv (very low), and is consistent with the observation that YouTube channels referenced in research tend to be established content creators. The storage recommendation for transcripts is driven by IP blocking, not by this risk level, so the assumption does not affect the policy outcome.

Analysis

The three-criteria framework (re-fetchability, link rot risk, format) was sufficient to determine unambiguous storage policies for all four content types without requiring case-by-case judgment. The framework's strength is that the criteria are largely independent: a content type can fail on one criterion (e.g., binary format) while passing on another (e.g., stable URL) and the format criterion alone is sufficient to rule out local storage for PDFs. The re-fetchability criterion alone is sufficient to require local storage for transcripts.

The Git LFS question was answered quantitatively rather than heuristically. The 40–150 KB per transcript estimate, combined with GitHub's 50 MiB threshold, shows that the repo would need approximately 333–1,250 transcripts before any single file approached the warning limit, and the total storage concern only becomes real at 5,000–25,000 transcripts (375 MB – 3.75 GB). This is orders of magnitude beyond the anticipated scale. The operational cost of LFS (incompatible with GitHub-website-only workflow) makes it doubly unattractive.

The inline-paste approach for web pages resolves the tension between durability and storage overhead: it stores exactly the evidence needed to support a specific claim (the relevant passage) without committing a full HTML snapshot of every web page ever consulted.

Risks, Gaps, and Uncertainties

• Transcript size assumption is unverified: The 40–150 KB estimate has not been measured against actual fetched transcripts in this repository. If the repo begins fetching very long videos (>3 hours) from content-dense channels, individual transcripts could reach 400–500 KB. This remains well below GitHub's thresholds.
• YouTube video deletion rate is unmeasured: The "medium" link rot classification for YouTube is an inference without a specific quantitative source. A future item could investigate this if transcript durability becomes a concern.
• No snapshot strategy for web pages: The inline-paste approach is adequate for key sources but does not provide full snapshots of peripheral references. If the research evolves to require full-page snapshots (e.g., for regulatory or legal documentation), a separate archive workflow (Wayback Machine API, or a snapshots/ directory with stripped-HTML .txt files) would be needed.

Open Questions

• Q1: Should the fetch-transcript workflow be updated to tag transcripts with metadata (fetch date, video title, duration) to support future lifecycle management (e.g., re-fetching stale transcripts)? This is an implementation question for the tooling backlog, not a research question.
• Q2: Is there a systematic data source for YouTube video deletion rates that could replace the "medium" assumption with a measured rate? This is low priority: the storage decision for transcripts is driven by IP blocking, not by deletion risk.

sources

• [x] Git LFS docs
• [x] Link rot research
• [x] GitHub file size limits
• [x] arXiv identifier permanence
• [ ] Similar decisions in knowledge management tools (Obsidian, Logseq)