1---
2name: baoyu-url-to-markdown
3description: Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.
4version: 1.61.0
5metadata:
6  openclaw:
7    homepage: https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown
8    requires:
9      anyBins:
10        - bun
11---
12 
13# URL to Markdown
14 
15Fetches any URL via `baoyu-fetch` CLI (Chrome CDP + site-specific adapters) and converts it to clean markdown.
16 
17## User Input Tools
18 
19When this skill prompts the user, follow this tool-selection rule (priority order):
20 
211. **Prefer built-in user-input tools** exposed by the current agent runtime — e.g., `AskUserQuestion`, `request_user_input`, `clarify`, `ask_user`, or any equivalent.
222. **Fallback**: if no such tool exists, emit a numbered plain-text message and ask the user to reply with the chosen number/answer for each question.
233. **Batching**: if the tool supports multiple questions per call, combine all applicable questions into a single call; if only single-question, ask them one at a time in priority order.
24 
25Concrete `AskUserQuestion` references below are examples — substitute the local equivalent in other runtimes.
26 
27## CLI Setup
28 
29**Important**: The CLI source is vendored in `{baseDir}/scripts/lib`. `scripts/package.json` installs only third-party runtime dependencies.
30 
31**Agent Execution Instructions**:
321. Determine this SKILL.md file's directory path as `{baseDir}`
332. Resolve `${BUN}` runtime: if `bun` installed → `bun`; else suggest installing Bun
343. If `{baseDir}/scripts/node_modules` does not exist, run `${BUN} install --cwd {baseDir}/scripts`
354. `${READER}` = `{baseDir}/scripts/baoyu-fetch`
365. Replace all `${READER}` in this document with the resolved value
37 
38## Preferences (EXTEND.md)
39 
40Check EXTEND.md in priority order — the first one found wins:
41 
42| Priority | Path | Scope |
43|----------|------|-------|
44| 1 | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Project |
45| 2 | `${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | XDG |
46| 3 | `$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | User home |
47 
48| Result | Action |
49|--------|--------|
50| Found | Read, parse, apply settings |
51| Not found | **MUST** run first-time setup (see below) — do NOT silently create defaults |
52 
53**EXTEND.md supports**: download media by default, default output directory.
54 
55### First-Time Setup ⛔ BLOCKING
56 
57When EXTEND.md is not found, you **MUST** use `AskUserQuestion` to gather preferences before creating EXTEND.md. **NEVER** create EXTEND.md with silent defaults. Generation is BLOCKED until setup completes. Batch all three questions into a single call:
58 
59- **Q1 — Media** (header "Media"): "How to handle images and videos in pages?"
60  - "Ask each time (Recommended)" — Prompt after each save
61  - "Always download" — Download to local `imgs/` and `videos/`
62  - "Never download" — Keep remote URLs
63- **Q2 — Output** (header "Output"): "Default output directory?"
64  - "url-to-markdown (Recommended)" — Save to `./url-to-markdown/{domain}/{slug}.md`
65  - User may pick "Other" and type a custom path
66- **Q3 — Save** (header "Save"): "Where to save preferences?"
67  - "User (Recommended)" — `~/.baoyu-skills/` (all projects)
68  - "Project" — `.baoyu-skills/` (this project only)
69 
70After answers, write EXTEND.md, confirm "Preferences saved to [path]", then continue.
71 
72Full template: [references/config/first-time-setup.md](references/config/first-time-setup.md).
73 
74### Supported Keys
75 
76| Key | Default | Values | Description |
77|-----|---------|--------|-------------|
78| `download_media` | `ask` | `ask` / `1` / `0` | `ask` = prompt each time, `1` = always, `0` = never |
79| `default_output_dir` | empty | path or empty | Default output directory (empty = `./url-to-markdown/`) |
80 
81**EXTEND.md → CLI mapping**:
82 
83| EXTEND.md key | CLI argument | Notes |
84|---------------|-------------|-------|
85| `download_media: 1` | `--download-media` | Requires `--output` to be set |
86| `default_output_dir: ./posts/` | Agent constructs `--output ./posts/{domain}/{slug}.md` | Agent generates path, not a direct flag |
87 
88**Value priority**: CLI arguments → EXTEND.md → skill defaults.
89 
90## Usage
91 
92```bash
93# Default: headless capture, markdown to stdout
94${READER} <url>
95 
96# Save to file
97${READER} <url> --output article.md
98 
99# Save with media download
100${READER} <url> --output article.md --download-media
101 
102# Wait for interaction (login/CAPTCHA) — auto-detect and continue
103${READER} <url> --wait-for interaction --output article.md
104 
105# Wait for interaction — manual control (Enter to continue)
106${READER} <url> --wait-for force --output article.md
107 
108# JSON output
109${READER} <url> --format json --output article.json
110 
111# Force specific adapter
112${READER} <url> --adapter youtube --output transcript.md
113```
114 
115## Options
116 
117| Option | Description |
118|--------|-------------|
119| `<url>` | URL to fetch |
120| `--output <path>` | Output file path (default: stdout) |
121| `--format <type>` | Output format: `markdown` (default) or `json` |
122| `--json` | Shorthand for `--format json` |
123| `--adapter <name>` | Force adapter: `x`, `youtube`, `hn`, or `generic` (default: auto-detect) |
124| `--headless` | Force headless Chrome (no visible window) |
125| `--wait-for <mode>` | Interaction wait mode: `none` (default), `interaction`, or `force` |
126| `--wait-for-interaction` | Alias for `--wait-for interaction` |
127| `--wait-for-login` | Alias for `--wait-for interaction` |
128| `--timeout <ms>` | Page load timeout (default: 30000) |
129| `--interaction-timeout <ms>` | Login/CAPTCHA wait timeout (default: 600000 = 10 min) |
130| `--interaction-poll-interval <ms>` | Poll interval for interaction checks (default: 1500) |
131| `--download-media` | Download images/videos to local `imgs/` and `videos/`, rewrite markdown links. Requires `--output` |
132| `--media-dir <dir>` | Base directory for downloaded media (default: same as `--output` directory) |
133| `--cdp-url <url>` | Reuse existing Chrome DevTools Protocol endpoint |
134| `--browser-path <path>` | Custom Chrome/Chromium binary path |
135| `--chrome-profile-dir <path>` | Chrome user data directory (default: `BAOYU_CHROME_PROFILE_DIR` env or `./baoyu-skills/chrome-profile`) |
136| `--debug-dir <dir>` | Write debug artifacts (document.json, markdown.md, page.html, network.json) |
137 
138## Agent Quality Gate
139 
140**CRITICAL**: treat default headless capture as provisional. Some sites render differently in headless mode and can silently return low-quality content without failing the CLI.
141 
142After every headless run, inspect the saved markdown. See [references/quality-gate.md](references/quality-gate.md) for the full checklist, recovery workflow, and capture-mode table. Read it whenever a run looks suspicious or the user asks about login/CAPTCHA handling.
143 
144## Output Path Generation
145 
146The agent must construct the output file path — `baoyu-fetch` does not auto-generate paths.
147 
148**Algorithm**:
1491. Determine base directory from EXTEND.md `default_output_dir` or default `./url-to-markdown/`
1502. Extract domain from URL (e.g., `example.com`)
1513. Generate slug from URL path or page title (kebab-case, 2-6 words)
1524. Construct: `{base_dir}/{domain}/{slug}/{slug}.md` — each URL gets its own directory so media files stay isolated
1535. Conflict resolution: append timestamp `{slug}-YYYYMMDD-HHMMSS/{slug}-YYYYMMDD-HHMMSS.md`
154 
155Pass the constructed path to `--output`. Media files (`--download-media`) are saved into subdirectories next to the markdown file, keeping each URL's assets self-contained.
156 
157## Adapters & Media
158 
159See [references/adapters.md](references/adapters.md) for the adapter catalog (X, YouTube, Hacker News, generic), per-adapter notes, the media download flow (`ask` / always / never), and the JSON output schema. Read it before answering adapter-specific questions or handling media prompts.
160 
161## Environment Variables
162 
163| Variable | Description |
164|----------|-------------|
165| `BAOYU_CHROME_PROFILE_DIR` | Chrome user data directory (can also use `--chrome-profile-dir`) |
166 
167**Troubleshooting**: Chrome not found → use `--browser-path`. Timeout → increase `--timeout`. Login/CAPTCHA → `--wait-for interaction`. Debug → `--debug-dir` to inspect captured HTML and network logs.
168 
169## Extension Support
170 
171Custom configurations via EXTEND.md. See **Preferences** section above for paths and supported keys.
172