1# Case Studies: LLM Project Development
2 
3This reference contains detailed case studies of production LLM projects that demonstrate effective development methodology. Each case study analyzes the problem, approach, architecture, and lessons learned.
4 
5## Case Study 1: Karpathy's HN Time Capsule
6 
7**Source**: https://github.com/karpathy/hn-time-capsule
8 
9### Problem Statement
10 
11Analyze Hacker News discussions from 10 years ago and grade commenters on how prescient their predictions were with the benefit of hindsight.
12 
13### Task-Model Fit Analysis
14 
15This task is well-suited for LLM processing because:
16 
17| Factor | Assessment |
18|--------|------------|
19| Synthesis | Combining article content + multiple comment threads |
20| Subjective judgment | Grading predictions against known outcomes |
21| Domain knowledge | Model has knowledge of what actually happened |
22| Error tolerance | Wrong grade on one comment does not break the system |
23| Batch processing | Each article is independent |
24| Natural language output | Human-readable analysis is the goal |
25 
26### Development Methodology
27 
28**Step 1: Manual Prototype**
29 
30Before building any automation, Karpathy copy-pasted one article + comment thread into ChatGPT to validate the approach. This took minutes and confirmed:
31- The model could produce insightful hindsight analysis
32- The output format worked for the intended use case
33- The quality exceeded what he could do manually
34 
35**Step 2: Agent-Assisted Implementation**
36 
37Used Opus 4.5 to build the pipeline in approximately 3 hours. The agent handled:
38- HTML parsing for HN frontpage
39- Algolia API integration for comments
40- Prompt template design
41- Output parsing logic
42- Static HTML rendering
43 
44**Step 3: Batch Execution**
45 
46- 930 LLM queries (31 days × 30 articles)
47- 15 parallel workers
48- ~$58 total cost
49- ~1 hour execution time
50 
51### Pipeline Architecture
52 
53```
54fetch → prompt → analyze → parse → render
55```
56 
57**Stage 1: Fetch**
58- Download HN frontpage for target date
59- Fetch article content via HTTP
60- Fetch comments via Algolia API
61- Output: `data/{date}/{item_id}/meta.json`, `article.txt`, `comments.json`
62 
63**Stage 2: Prompt**
64- Load article metadata and content
65- Load comment tree
66- Generate markdown prompt from template
67- Output: `data/{date}/{item_id}/prompt.md`
68 
69**Stage 3: Analyze**
70- Submit prompt to GPT 5.1 Thinking API
71- Parallel execution with ThreadPoolExecutor
72- Output: `data/{date}/{item_id}/response.md`
73 
74**Stage 4: Parse**
75- Extract grades from "Final grades" section via regex
76- Extract interestingness score via regex
77- Aggregate grades across all articles
78- Output: `data/{date}/{item_id}/grades.json`, `score.json`
79 
80**Stage 5: Render**
81- Generate static HTML with embedded JavaScript
82- Create day pages with article navigation
83- Create Hall of Fame with aggregated rankings
84- Output: `output/{date}/index.html`, `output/hall-of-fame.html`
85 
86### Structured Output Design
87 
88The prompt template specifies exact output format:
89 
90```
91Let's use our benefit of hindsight now in 6 sections:
92 
931. Give a brief summary of the article and the discussion thread.
942. What ended up happening to this topic?
953. Give out awards for "Most prescient" and "Most wrong" comments.
964. Mention any other fun or notable aspects.
975. Give out grades to specific people for their comments.
986. At the end, give a final score (from 0-10).
99 
100As for the format of Section 5, use the header "Final grades" and follow it 
101with simply an unordered list in the format of "name: grade (optional comment)".
102 
103Please follow the format exactly because I will be parsing it programmatically.
104```
105 
106Key techniques:
107- Numbered sections for structure
108- Explicit format specification with examples
109- Rationale disclosure ("because I will be parsing it")
110- Constrained output (letter grades, 0-10 scores)
111 
112### Parsing Implementation
113 
114The parsing code handles variations gracefully:
115 
116```python
117def parse_grades(text: str) -> dict[str, dict]:
118    # Match "Final grades" with optional section number or markdown
119    pattern = r'(?:^|\n)(?:\d+[\.\)]\s*)?(?:#+ *)?Final grades\s*\n'
120    match = re.search(pattern, text, re.IGNORECASE)
121    
122    # Handle both ASCII and Unicode minus signs
123    line_pattern = r'^[\-\*]\s*([^:]+):\s*([A-F][+\-−]?)(?:\s*\(([^)]+)\))?'
124```
125 
126### Lessons Learned
127 
1281. **Manual validation first**: The 5-minute copy-paste test prevented hours of wasted development.
129 
1302. **File system as state**: Each article directory contains all intermediate outputs, making debugging trivial.
131 
1323. **Idempotent stages**: Re-running only processes items that lack output files.
133 
1344. **Agent-assisted development**: 3 hours to working code by focusing on requirements, not implementation details.
135 
1365. **Parallel execution**: 15 workers reduced execution time without increasing token costs.
137 
138---
139 
140## Case Study 2: Vercel d0 Architectural Reduction
141 
142**Source**: https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools
143 
144### Problem Statement
145 
146Build a text-to-SQL agent that enables anyone at Vercel to query analytics data through natural language questions in Slack.
147 
148### Initial Approach (Failed)
149 
150The team built a sophisticated system with:
151- 17 specialized tools (schema lookup, query validation, error recovery, etc.)
152- Heavy prompt engineering to constrain reasoning
153- Careful context management
154- Hand-coded retrieval for schema information
155 
156**Results**:
157- 80% success rate
158- 274.8 seconds average execution time
159- ~102k tokens average usage
160- ~12 steps average
161- Constant maintenance burden
162 
163### The Problem
164 
165The team was solving problems the model could handle on its own:
166- Pre-filtering context
167- Constraining options
168- Wrapping every interaction in validation logic
169- Building tools to "protect" the model from complexity
170 
171Every edge case required another patch. Every model update required re-calibrating constraints. More time was spent maintaining scaffolding than improving outcomes.
172 
173### Architectural Reduction
174 
175The hypothesis: What if we just give Claude access to the raw files and let it figure things out?
176 
177**New architecture**:
178- 2 tools total: ExecuteCommand (bash) + ExecuteSQL
179- Direct file system access via sandbox
180- Semantic layer as YAML/Markdown/JSON files
181- Standard Unix utilities (grep, cat, find, ls)
182 
183```javascript
184const agent = new ToolLoopAgent({
185  model: "anthropic/claude-opus-4.5",
186  tools: {
187    ExecuteCommand: executeCommandTool(sandbox),
188    ExecuteSQL,
189  },
190});
191```
192 
193### Results
194 
195| Metric | Before (17 tools) | After (2 tools) | Change |
196|--------|-------------------|-----------------|--------|
197| Avg execution time | 274.8s | 77.4s | 3.5x faster |
198| Success rate | 80% | 100% | +20% |
199| Avg token usage | ~102k | ~61k | 37% fewer |
200| Avg steps | ~12 | ~7 | 42% fewer |
201 
202The worst case before: 724 seconds, 100 steps, 145k tokens, and still failed.
203Same query after: 141 seconds, 19 steps, 67k tokens, succeeded.
204 
205### Why It Worked
206 
2071. **Good documentation already existed**: The semantic layer files contained dimension definitions, measure calculations, and join relationships. The tools were summarizing what was already legible.
208 
2092. **File systems are proven abstractions**: The model understands file systems deeply from training. grep is 50 years old and works perfectly.
210 
2113. **Constraints became liabilities**: With better models, the guardrails were limiting performance more than helping.
212 
213### Key Lessons
214 
2151. **Addition by subtraction**: The best agents might be ones with the fewest tools. Every tool is a choice you are making for the model.
216 
2172. **Build for future models**: Models improve faster than tooling. Architectures optimized for today may be over-constrained for tomorrow.
218 
2193. **Good context over clever tools**: Invest in documentation, clear naming, and well-structured data. That foundation matters more than sophisticated tooling.
220 
2214. **Start simple**: Model + file system + goal. Add complexity only when proven necessary.
222 
223---
224 
225## Case Study 3: Manus Context Engineering
226 
227**Source**: Peak Ji's blog "Context Engineering for AI Agents: Lessons from Building Manus"
228 
229### Problem Statement
230 
231Build a general-purpose consumer agent that can accomplish complex tasks across 50+ tool calls while maintaining performance and managing costs.
232 
233### Core Insight
234 
235KV-cache hit rate is the single most important metric for production agents. It directly affects both latency and cost.
236 
237- Claude Sonnet cached: $0.30/MTok
238- Claude Sonnet uncached: $3.00/MTok
239- 10x cost difference
240 
241With an average input-to-output ratio of 100:1 in agentic workloads, optimizing for cache hits dominates the cost equation.
242 
243### Key Patterns
244 
245**1. Append-Only Context**
246 
247Never modify previous actions or observations. Ensure deterministic serialization (JSON key ordering must be stable). A single token difference invalidates the cache from that point forward.
248 
249Common mistake: Including a timestamp at the beginning of the system prompt kills cache hit rate entirely.
250 
251**2. Mask, Do Not Remove**
252 
253Do not dynamically add or remove tools mid-iteration. Tool definitions live near the front of context - any change invalidates the KV-cache for all subsequent content.
254 
255Instead, use logit masking during decoding to constrain tool selection without modifying definitions. This maintains cache while still controlling behavior.
256 
257**3. File System as Context**
258 
259Treat the file system as unlimited, persistent, agent-operable memory. The model learns to write and read files on demand.
260 
261Compression strategies should be restorable:
262- Web page content can be dropped if URL is preserved
263- Document contents can be omitted if file path remains available
264 
265**4. Recitation for Attention**
266 
267Manus creates a todo.md file and updates it step-by-step. This is not just organization - it pushes the global plan into the model's recent attention span.
268 
269By constantly rewriting objectives at the end of context, the agent avoids "lost in the middle" issues and maintains goal alignment.
270 
271**5. Keep Errors In Context**
272 
273Do not hide failures. When the model sees a failed action and the resulting error, it implicitly updates beliefs and avoids repeating mistakes.
274 
275Erasing failures removes evidence the model needs to adapt.
276 
277### Multi-Agent for Context Isolation
278 
279The primary goal of sub-agents in Manus is context isolation, not role division. For tasks requiring discrete work:
280- Planner assigns tasks to sub-agents with their own context windows
281- Simple tasks: pass instructions via function call
282- Complex tasks: share full context with sub-agent
283 
284Sub-agents have a submit_results tool with constrained output schema. Constrained decoding ensures adherence to defined format.
285 
286### Layered Action Space
287 
288Rather than binding every utility as a tool:
289- Small set (<20) of atomic functions: Bash, filesystem access, code execution
290- Most actions offload to sandbox layer
291- MCP tools exposed through CLI, executed via Bash tool
292 
293This reduces tool definition tokens and prevents model confusion from overlapping descriptions.
294 
295### Iteration Expectation
296 
297Manus has refactored their agent framework five times since launch. The Bitter Lesson suggests structures added for current limitations become constraints as models improve.
298 
299Test across model strengths to verify your harness is not limiting performance. Simple, unopinionated designs adapt better to model improvements.
300 
301---
302 
303## Case Study 4: Anthropic Multi-Agent Research
304 
305**Source**: Anthropic blog "How we built our multi-agent research system"
306 
307### Problem Statement
308 
309Build a research feature that can explore complex topics using multiple parallel agents searching across web, Google Workspace, and integrations.
310 
311### Architecture
312 
313Orchestrator-worker pattern:
314- Lead agent analyzes query and develops strategy
315- Lead spawns subagents for parallel exploration
316- Subagents return findings to lead for synthesis
317- Citation agent processes final output
318 
319### Performance Insight
320 
321Three factors explained 95% of performance variance in BrowseComp evaluation:
322- Token usage: 80% of variance
323- Number of tool calls: additional factor
324- Model choice: additional factor
325 
326Multi-agent architectures effectively scale token usage for tasks exceeding single-agent limits.
327 
328### Token Economics
329 
330- Chat interactions: baseline
331- Single agent: ~4x more tokens than chat
332- Multi-agent: ~15x more tokens than chat
333 
334Multi-agent requires high-value tasks to justify the cost.
335 
336### Prompting Principles
337 
3381. **Think like your agents**: Build simulations, watch step-by-step, identify failure modes.
339 
3402. **Teach delegation**: Subagents need objective, output format, tools/sources guidance, and clear boundaries.
341 
3423. **Scale effort to complexity**: Explicit guidelines for agent/tool call counts by task type.
343 
3444. **Tool design is critical**: Distinct purpose and clear description for each tool. Bad descriptions send agents down wrong paths entirely.
345 
3465. **Let agents improve themselves**: Claude 4 models can diagnose prompt failures and suggest improvements. Tool-testing agents can rewrite tool descriptions to avoid common mistakes.
347 
3486. **Start wide, then narrow**: Broad queries first, evaluate landscape, then drill into specifics.
349 
3507. **Guide thinking process**: Extended thinking mode as controllable scratchpad for planning.
351 
3528. **Parallel tool calling**: 3-5 subagents in parallel, 3+ tools per subagent in parallel. Cut research time by up to 90%.
353 
354### Evaluation Approach
355 
356- Start with ~20 representative queries immediately
357- LLM-as-judge with rubric: factual accuracy, citation accuracy, completeness, source quality, tool efficiency
358- Human evaluation catches edge cases automation misses
359- Focus on end-state evaluation for multi-turn agents
360 
361---
362 
363## Cross-Case Patterns
364 
365### Common Success Factors
366 
3671. **Manual validation before automation**: All successful projects validated task-model fit with simple tests first.
368 
3692. **File system as foundation**: Whether for state management (Karpathy), tool interface (Vercel), or memory (Manus), the file system provides proven abstractions.
370 
3713. **Architectural simplicity**: Reduction outperformed complexity in multiple cases. Start minimal, add only what proves necessary.
372 
3734. **Structured outputs with robust parsing**: Explicit format specifications combined with flexible parsing that handles variations.
374 
3755. **Iteration expectation**: No project got architecture right on the first try. Build for change.
376 
377### Common Failure Patterns
378 
3791. **Over-constraining models**: Guardrails that helped with weaker models become liabilities as capabilities improve.
380 
3812. **Tool proliferation**: More tools often means more confusion and worse performance.
382 
3833. **Hiding errors**: Removing failures from context prevents models from learning.
384 
3854. **Premature optimization**: Adding complexity before basic functionality works.
386 
3875. **Ignoring economics**: Token costs compound quickly; estimation and tracking are essential.
388 
389