1# Content Curation Rubric
2 
3Use this rubric to decide whether an external source deserves entry into the research pipeline. It extracts the reusable policy from `researcher/llm-as-a-judge.md` and resolves the gate ambiguity: any failed gate rejects the source unless a human explicitly overrides it.
4 
5## Gatekeeper Triage
6 
7All gates must pass before dimensional scoring.
8 
9| Gate | Pass | Fail |
10| --- | --- | --- |
11| G1 Mechanism Specificity | Defines a concrete mechanism, pattern, metric, workflow, or architecture | Uses vague advice like "improve prompts" without explaining how |
12| G2 Implementable Artifacts | Includes code, schemas, prompt templates, diagrams, API contracts, configs, or enough procedure to implement | Pure commentary with no artifact or reproducible procedure |
13| G3 Beyond Basics | Covers advanced context, harness, memory, tool, evaluation, multi-agent, or research operations patterns | Introductory content only |
14| G4 Source Verifiability | Author or organization is identifiable and technically credible | Anonymous, unverifiable, or marketing-only source |
15 
16If any gate fails, record `REJECT` with the failed gate and stop.
17 
18## Dimensional Scoring
19 
20Score each dimension as 0, 1, or 2.
21 
22| Dimension | Weight | Score 2 | Score 1 | Score 0 |
23| --- | --- | --- | --- | --- |
24| D1 Technical Depth and Actionability | 35% | Directly implementable with artifacts or exact procedure | Useful but requires interpretation | No path to implementation |
25| D2 Repo Relevance | 30% | Directly maps to context engineering, harness engineering, or skill authoring | Adjacent to repo scope | Out of scope |
26| D3 Evidence and Rigor | 20% | Quantitative evidence, baselines, ablations, public logs, or reproducible method | Plausible experience report | Unsupported claim |
27| D4 Novelty and Insight | 15% | New mechanism, counterintuitive finding, or high-value failure mode | Useful synthesis of known ideas | Common knowledge |
28 
29Formula:
30 
31```text
32weighted_total = D1*0.35 + D2*0.30 + D3*0.20 + D4*0.15
33```
34 
35## Decisions
36 
37| Decision | Condition |
38| --- | --- |
39| APPROVE | All gates pass and weighted total >= 1.4 |
40| HUMAN_REVIEW | All gates pass and 0.9 <= weighted total < 1.4 |
41| REJECT | Any gate fails or weighted total < 0.9 |
42 
43## Overrides
44 
45- O1: D1 = 0 forces `REJECT`.
46- O2: D2 = 0 forces `REJECT`.
47- O3: D3 = 1 and total >= 1.4 forces `HUMAN_REVIEW`.
48- O4: D4 = 2 and total < 1.4 forces `HUMAN_REVIEW`.
49 
50## Required Evidence
51 
52Every approved or reviewed source must include:
53 
541. Retrieved source URL and retrieval status.
552. Specific quoted or paraphrased evidence for every passing gate.
563. A mechanism summary in one paragraph.
574. Candidate skill impact: new skill, update existing skill, reference-only, or reject.
585. Known limitations, missing evidence, or assumptions.
59 
60## Output Shape
61 
62Use `../templates/source-evaluation.json` for machine-readable output. If the model cannot produce valid JSON, log the raw output and route to human review instead of silently fixing the evaluation.
63