sigilbox/AGENTS.md

Agent notes for local-page-archiver

Project overview

This tool renders web pages in Chromium (via Playwright) and saves them as fully self-contained HTML files. All external assets (images, fonts, stylesheets) are inlined as data URIs so the resulting file works offline.

The pipeline is:

URL ──► set request filters ──► Playwright render ──► inject cosmetic filters/userscripts ──► inline assets ──► write HTML

Source layout

• src/cli.mjs — CLI entrypoint. Supports archive and help. Accepts --archive-path, --id, and --headful flags.
• src/archiver.mjs — Core archiving logic. Loads privacy filters, steers the browser, injects adblockers/userscripts, and calls the inliner.
• src/asset-inliner.mjs — Fetches and inlines external resources (images, CSS, iframes). Also strips <script> and <noscript> tags for a static archive.
• privacy-filters/ — Third-party filter lists and userscripts used to strip paywalls, trackers, and ad banners before the snapshot is taken.

Privacy filters (privacy-filters/)

bpc-paywall-filter.txt

An AdBlock Plus / uBlock Origin filter list. It contains three kinds of rules:

1. Network rules (||tracker.com^, /regex/) — block specific third-party paywall / tracking scripts.
2. Exception rules (@@||example.com^) — whitelist requests that a global block rule would otherwise hit.
3. Cosmetic rules (example.com##.paywall) — inject CSS to hide DOM elements (e.g. subscription banners, blurred overlays).

At module load time archiver.mjs parses this file into three arrays (blockRules, allowRules, cosmeticRules). Network rules are enforced at the Playwright level with page.route(...). Cosmetic rules are injected as a <style> tag after the page reaches domcontentloaded.

Cosmetic rule caveats

Some advanced cosmetic syntax is unsupported and is silently discarded or downgraded:

• Terminal :remove() — converted to CSS hiding; we can't actually remove DOM nodes from the CSS layer.
• :style(...) — converted to real CSS using the style content from the filter rule.
• :xpath(...), :upward(...), :matches-css(...), :matches-media(...), :matches-path(...) — discarded during parsing.

userscript/ directory

Contains Greasemonkey-style userscripts (bpc.*.user.js) plus a shared library bpc_func.js. They do heavy lifting: decrypt paywalls, reconstruct article text from JSON data embedded in the page, remove blur overlays, etc.

Selective injection (important)

Each userscript declares @match and @exclude metadata. Only matching scripts are injected. For example, on bloomberg.com only bpc.en.user.js is injected. The shared bpc_func.js helper is injected first, then the matching userscript files.

Malformed asset fetches such as quoted Stripe or Google Pay script URLs usually mean escaped markup inside srcdoc or another HTML attribute is being parsed as top-level HTML. The inliner should only read attributes from real opening tags, and it sanitizes srcdoc iframe HTML recursively.

The matching logic is a simple glob parser for userscript @match patterns:

• *://*.com/* matches any .com domain
• *://example.com/path/* matches that path prefix
• @exclude patterns take precedence and skip the script

GM.xmlHttpRequest mock

The userscripts rely on GM.xmlHttpRequest to fetch article text from archive mirrors or API endpoints. In a Playwright context this doesn't exist, so we inject a tiny mock that wraps the browser's native fetch() and presents the same callback interface (onload, onerror).

Timing

Userscripts are injected after domcontentloaded but before networkidle. We then wait an extra 2 s (page.waitForTimeout) so any setTimeout(..., 1000) callbacks inside the scripts have time to fire before we snapshot the DOM.

Stealth / anti-detection

Stealth package naming trap

The package names that look like Playwright-specific stealth plugins are placeholders:

npm install playwright-extra playwright-extra-plugin-stealth

playwright-extra-plugin-stealth, playwright-extra-stealth, and playwright-stealth are placeholder packages (version 0.0.1) that literally throw on require():

Error: Wrong package, please see this:
https://github.com/berstend/puppeteer-extra/issues/454

If we revisit package-based stealth, the working route to evaluate is playwright-extra with puppeteer-extra-plugin-stealth. This project currently uses manual evasions instead, keeping package.json limited to plain Playwright.

Manual stealth evasions

Instead we apply the same core evasions manually via context.addInitScript() and browser launch flags.

Launch flags:

• --disable-blink-features=AutomationControlled
• --disable-infobars
• --disable-web-security
• --no-sandbox, --disable-setuid-sandbox
• --disable-dev-shm-usage
• Removed --enable-automation via ignoreDefaultArgs

Init script (injected into every page before any scripts run):

Object.defineProperty(navigator, 'webdriver',
  { get: () => undefined, configurable: true, enumerable: true });
window.chrome = window.chrome || { runtime: {} };
window.navigator.permissions.query = (/* patched for notifications */);

CRITICAL: Avoid delete navigator.webdriver + iframe trick

An earlier version used a more elaborate stealth snippet that did delete navigator.webdriver and then created an <iframe> to steal the real navigator descriptor. This crashed the Chromium renderer process on tab creation with:

Protocol error (Page.addScriptToEvaluateOnNewDocument): Target crashed

The current init script is minimal and safe — it only overrides the getter via Object.defineProperty and avoids DOM mutation during page init.

Browser context & headful mode

renderPage() auto-detects whether a display is available ($DISPLAY / $WAYLAND_DISPLAY). If neither is set it defaults to headless. The caller can override via options.headless.

• Viewport: 1366×768 (standard laptop resolution, not the default Playwright 1280×720)
• Locale: en-US
• Timezone: America/New_York
• User-Agent: macOS Chrome 130. This is pinned for site compatibility and is not automatically synchronized with the installed Chromium version.

Docker / Podman support

Dockerfile

• Base: mcr.microsoft.com/playwright:v1.60.0 (must stay in sync with the playwright npm version)
• Installs Node 22 (the base image may ship an older Node)
• Runs npx playwright install chromium so the browser binary is baked into the image

podman-run.sh

Helper for local runs. Two modes:

1. ./podman-run.sh archive <URL> — headless, mounts ./archives
2. ./podman-run.sh headful-archive <URL> — headful with internal VNC

Headful mode details: The container's ENTRYPOINT is node src/cli.mjs. To run a shell command inside the container (setting up Xvfb + x11vnc) we must override the entrypoint:

podman run --rm --entrypoint sh <image> -c "...setup Xvfb... && node src/cli.mjs archive <URL>"

Port 5900 inside the container maps to 5901 on the host to avoid conflicts with macOS's built-in VNC.

docker-compose.yml

Includes a headful profile that can be run with:

URL=https://example.com docker compose --profile headful up archiver-headful

Unlike podman-run.sh, Compose currently maps VNC to host port 5900.

Known limitations

Site-specific blocking

Some publishers can still return bot walls, consent walls, or region-specific variants depending on IP reputation and timing. Treat these as site/network-sensitive failures and reproduce with the exact URL, mode, and environment before assuming the browser stealth layer is the root cause. Bloomberg has recently archived successfully in local verification.

Unsupported adblock syntax

• Advanced procedural cosmetic filters (:upward(), :xpath(), :matches-css(), etc.) are silently ignored.
• Terminal :remove() cosmetic filters are downgraded to CSS hiding.
• Scriptlet injection (##+js(...)) is not supported by the filter parser. The BPC userscripts still run separately when their metadata matches the page.
• Preprocessor directives (!#if) are skipped.

Adding a new site to privacy filters

If you add a new filter rule or userscript:

1. Filter rules: Edit privacy-filters/bpc-paywall-filter.txt. archiver.mjs reloads the file on every process start, so no code changes are needed.
2. Userscripts: Drop a new .user.js into privacy-filters/userscript/ and add its filename to the userScriptFiles array inside loadPrivacyFilters() in archiver.mjs.
3. Test with node src/cli.mjs archive <URL> and inspect the generated HTML.

Rebuilding the Docker image

Playwright npm version image tags must match:

# Check what npm installed
node -e "console.log(require('playwright/package.json').version)"

# Update the FROM line in Dockerfile if needed
# Then rebuild
podman build -t local-page-archiver .

Development quick reference

# Install deps
npm install

# Install browser binaries
npm run install-browsers

# Archive a page (headless)
node src/cli.mjs archive https://example.com

# Archive a page (headful on macOS)
node src/cli.mjs archive https://example.com --headful

# Archive inside container (headless)
./podman-run.sh archive https://example.com

# Archive inside container (headful + VNC)
./podman-run.sh headful-archive https://example.com
# Then open vnc://localhost:5901