Similarity algorithms and growth policy for a file-based controlled theme…

Research Question

Which similarity algorithms are appropriate for detecting near-synonym themes in a controlled vocabulary of 20–40 slug-based labels, and what growth policy prevents both vocabulary explosion and collapse in a file-based corpus of approximately 300 items growing at roughly 5 items per week?

Findings

Executive Summary

For a file-based corpus of ~300–400 items with slug-based theme labels, the appropriate near-duplicate detection algorithm pair is Levenshtein edit distance ≤ 2 (for character-level typos and plural variants) combined with token Jaccard similarity ≥ 0.6 (for multi-token near-synonyms). Both are pure Python with no external dependencies. A vocabulary of 20–40 canonical slugs is well-matched to the corpus size — the existing 16-item ai_themes vocabulary covers 86% of items, confirming that 16–40 is the pragmatic range. A growth policy requiring ≥3 items to justify adding a new theme is consistent with controlled vocabulary literature and prevents the singleton explosion observed in the uncontrolled tags: field (798 unique values). The SKOS (Simple Knowledge Organization System) prefLabel/altLabel pattern — canonical slug as preferred label, aliases as alternative labels — is directly applicable without requiring RDF or external tooling.

Key Findings

1. Token Jaccard similarity on hyphen-split tokens is the primary algorithm for near-synonym detection in slug-based vocabularies. (high confidence; source: Manning, Raghavan, Schütze: Introduction to Information Retrieval, Chapter 19 — Hierarchical Clustering (Cambridge University Press, 2008)
2. Levenshtein edit distance ≤ 2 complements Jaccard by catching character-level typos and singular/plural variants such as knowledge-graph vs knowledge-graphs. (high confidence; source: Manning, Raghavan, Schütze: Introduction to Information Retrieval, Chapter 19 — Hierarchical Clustering (Cambridge University Press, 2008)
3. A vocabulary of 20–40 canonical theme slugs is appropriate for a corpus of ~300–400 items; the existing 16-theme ai_themes field at 86% coverage confirms the lower bound is sufficient. (high confidence; source: Hedden Information Management: SKOS Taxonomies DARIAH-Campus: Controlled Vocabularies and SKOS
4. A growth policy requiring ≥3 items for a new theme prevents singleton explosion and is consistent with information science best practice. (high confidence; source: Hedden Information Management: SKOS Taxonomies
5. The SKOS prefLabel/altLabel/scopeNote pattern is applicable to a YAML file-based vocabulary without requiring RDF serialisation or external tooling. (high confidence; source: W3C SKOS Primer (2009)
6. Cosine TF-IDF, BM25, and embedding similarity are not appropriate for pairwise slug-label deduplication at vocabulary scales of 20–40 terms. (medium confidence; inference from algorithm characteristics and scale analysis)

Assumptions

• Assumption: No model service is available for embedding similarity. Justification: The pipeline is file-based and GitHub-Pages-compatible; no external model service has been approved (credentials table in repo instructions).
• Assumption: Corpus growth rate of ~5 items/week is stable. Justification: Based on observed corpus history; could vary but does not materially affect the algorithm or threshold recommendations.

Analysis

The uncontrolled tags: field demonstrates the explosion failure mode concretely: 798 unique values after ~400 completed items, with the majority appearing only once. The controlled ai_themes 16-item vocabulary, introduced via Gemini enrichment, demonstrates the correction: 86% coverage with 16 themes. The research findings converge on three actionable design parameters for W-0077: (1) 20–40 canonical slugs, (2) a synonym/alias map using the SKOS altLabel pattern, and (3) a ≥3-item growth threshold. The Levenshtein + token Jaccard pair provides the algorithmic backbone for the monthly review workflow (W-0080) to surface candidates for human confirmation.

Risks, Gaps, and Uncertainties

• The Jaccard ≥ 0.6 threshold is not empirically calibrated against this specific corpus vocabulary. The monthly review workflow should track false positives and false negatives to refine it.
• Fifty-five completed items currently lack ai_themes: data. These items may contain emerging themes not captured in the initial 16-theme set. The ≥3-item growth policy accommodates this: themes emerge naturally as items accumulate.
• If the pipeline gains a model service in future, embedding-based semantic similarity would be superior for detecting lexically dissimilar but semantically close slugs (e.g. cost-performance vs economic-efficiency). This is a known capability gap, not a defect in the current recommendation.

Open Questions

• Should the monthly theme-review workflow (W-0080) also compute pairwise Jaccard across all themes: values observed in the corpus — not just the vocabulary definition file — to detect synonym drift introduced by the enrichment pipeline?
• Is a brief scopeNote per canonical slug necessary at vocabulary launch, or can it be deferred to a later iteration?

sources

• [x] W3C SKOS Primer (2009) — authoritative reference for SKOS label types (prefLabel, altLabel, scopeNote) and concept scheme structure
• [x] W3C SKOS Reference (2009) — normative specification for SKOS property semantics and integrity conditions
• [x] DARIAH-Campus: Controlled Vocabularies and SKOS — practitioner guide covering vocabulary quality, orphaned concepts, and periodic review
• [x] Hedden Information Management: SKOS Taxonomies — professional vocabulary-management guidance; covers vocabulary size, growth, and deprecation
• [x] Manning, Raghavan, Schütze: Introduction to Information Retrieval, Chapter 19 — Hierarchical Clustering (Cambridge University Press, 2008) — authoritative treatment of Jaccard and edit-distance similarity in information retrieval clustering contexts