baoyu-url-to-markdown

# Adapters & Media

Read when choosing an adapter, handling media, or answering adapter-specific questions.

## Built-in Adapters

| Adapter | URLs | Key Features |
|---------|------|-------------|
| `x` | x.com, twitter.com | Tweets, threads, X Articles, media, login detection |
| `youtube` | youtube.com, youtu.be | Transcript/captions, chapters, cover image, metadata |
| `hn` | news.ycombinator.com | Threaded comments, story metadata, nested replies |
| `generic` | Any URL (fallback) | Defuddle extraction, Readability fallback, auto-scroll, network idle detection |

Adapter is auto-selected based on URL. Override with `--adapter <name>`.

### YouTube

- Extracts transcripts/captions when available
- Transcript format: `[MM:SS] Text segment` with chapter headings
- Availability depends on YouTube exposing a caption track; videos with captions disabled or restricted playback may produce description-only output
- Use `--wait-for force` if the page needs time to finish loading player metadata

### X/Twitter

- Extracts single tweets, threads, and X Articles
- Auto-detects login state; if logged out and content requires auth, JSON output shows `"status": "needs_interaction"`
- Use `--wait-for interaction` for login-protected content

### Hacker News

- Parses threaded comments with proper nesting and reply hierarchy
- Includes story metadata (title, URL, author, score, comment count)
- Shows comment deletion/dead status

## Media Download Workflow

Driven by `download_media` in EXTEND.md:

| Setting | Behavior |
|---------|----------|
| `1` (always) | Run CLI with `--download-media --output <path>` |
| `0` (never) | Run CLI with `--output <path>` (no media download) |
| `ask` (default) | Follow the ask-each-time flow below |

### Ask-Each-Time Flow

1. Run the CLI **without** `--download-media` with `--output <path>` → markdown saved
2. Check the saved markdown for remote media URLs (`https://` in image/video links)
3. **If no remote media found** → done, no prompt needed
4. **If remote media found** → ask via `AskUserQuestion`:
   - header: "Media", question: "Download N images/videos to local files?"
   - "Yes" — Download to local directories
   - "No" — Keep remote URLs
5. If the user confirms → run the CLI **again** with `--download-media --output <same-path>` (overwrites markdown with localized links)

### Media Layout

When `--download-media` is enabled:

- Images → `imgs/` next to the output file (or `--media-dir`)
- Videos → `videos/` next to the output file (or `--media-dir`)
- Markdown media links are rewritten to local relative paths

## Output Format

Markdown to stdout (or file with `--output`).

JSON output (`--format json`) returns structured data:

- `adapter` — which adapter handled the URL
- `status` — `"ok"` or `"needs_interaction"`
- `login` — login state detection (`logged_in`, `logged_out`, `unknown`)
- `interaction` — interaction gate details (kind, provider, prompt)
- `document` — structured content (url, title, author, publishedAt, content blocks, metadata)
- `media` — collected media assets with url, kind, role
- `markdown` — converted markdown text
- `downloads` — media download results (when `--download-media` used)

---
name: first-time-setup
description: First-time setup flow for baoyu-url-to-markdown preferences
---

# First-Time Setup

## Overview

When no EXTEND.md is found, guide user through preference setup.

**BLOCKING OPERATION**: This setup MUST complete before ANY other workflow steps. Do NOT:
- Start converting URLs
- Ask about URLs or output paths
- Proceed to any conversion

ONLY ask the questions in this setup flow, save EXTEND.md, then continue.

## Setup Flow

```
No EXTEND.md found
        |
        v
+---------------------+
| AskUserQuestion     |
| (all questions)     |
+---------------------+
        |
        v
+---------------------+
| Create EXTEND.md    |
+---------------------+
        |
        v
    Continue conversion
```

## Questions

**Language**: Use user's input language or saved language preference.

Use AskUserQuestion with ALL questions in ONE call:

### Question 1: Download Media

```yaml
header: "Media"
question: "How to handle images and videos in pages?"
options:
  - label: "Ask each time (Recommended)"
    description: "After saving markdown, ask whether to download media"
  - label: "Always download"
    description: "Always download media to local imgs/ and videos/ directories"
  - label: "Never download"
    description: "Keep original remote URLs in markdown"
```

### Question 2: Default Output Directory

```yaml
header: "Output"
question: "Default output directory?"
options:
  - label: "url-to-markdown (Recommended)"
    description: "Save to ./url-to-markdown/{domain}/{slug}.md"
```

Note: User will likely choose "Other" to type a custom path.

### Question 3: Save Location

```yaml
header: "Save"
question: "Where to save preferences?"
options:
  - label: "User (Recommended)"
    description: "~/.baoyu-skills/ (all projects)"
  - label: "Project"
    description: ".baoyu-skills/ (this project only)"
```

## Save Locations

| Choice | Path | Scope |
|--------|------|-------|
| User | `~/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | All projects |
| Project | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Current project |

## After Setup

1. Create directory if needed
2. Write EXTEND.md
3. Confirm: "Preferences saved to [path]"
4. Continue with conversion using saved preferences

## EXTEND.md Template

```md
download_media: [ask/1/0]
default_output_dir: [path or empty]
```

## Modifying Preferences Later

Users can edit EXTEND.md directly or delete it to trigger setup again.

# Quality Gate & Recovery

Headless Chrome can silently return low-quality content — layout shells, login walls, or framework payloads — without the CLI returning a non-zero exit code. Read this after every headless run so you can catch and recover from those cases.

## Checks the Agent Must Run

1. Confirm the markdown title matches the target page, not a generic site shell
2. Confirm the body contains the expected article/page content, not just navigation, footer, or a generic error
3. Watch for obvious failure signs:
   - `Application error`
   - `This page could not be found`
   - Login, signup, subscribe, or verification shells
   - Extremely short markdown for a page that should be long-form
   - Raw framework payloads or mostly boilerplate content
4. Do NOT accept a run as successful just because the CLI exited `0`

**Tip**: run with `--format json` to get structured signals including `status`, `login.state`, and `interaction`. `"status": "needs_interaction"` means the page requires manual interaction.

## Recovery Workflow

1. Start headless (default) unless there is already a clear reason to use interaction mode
2. Review markdown quality immediately after the run
3. If the content is low quality or indicates login/CAPTCHA:
   - `--wait-for interaction` for auto-detected gates (login, CAPTCHA, Cloudflare)
   - `--wait-for force` when the page needs manual browsing, scroll loading, or complex interaction
4. If `--wait-for` is used, tell the user exactly what to do:
   - Login required → sign in in the browser
   - CAPTCHA visible → solve it
   - Slow loading → wait until content is visible
   - `--wait-for force` → press Enter when ready
5. If JSON output shows `"status": "needs_interaction"`, switch to `--wait-for interaction` automatically

## Capture Modes

| Mode | Behavior | Use When |
|------|----------|----------|
| Default | Headless Chrome, auto-extract on network idle | Public pages, static content |
| `--headless` | Explicit headless (same as default) | Clarify intent |
| `--wait-for interaction` | Opens visible Chrome, auto-detects login/CAPTCHA gates, waits for them to clear, then continues | Login-required, CAPTCHA-protected |
| `--wait-for force` | Opens visible Chrome, auto-detects OR accepts Enter keypress to continue | Complex flows, lazy loading, paywalls |

**Interaction gate auto-detection**: Cloudflare Turnstile / "just a moment" pages, Google reCAPTCHA, hCaptcha, custom challenge / verification screens.