206 lines
8.8 KiB
Markdown
206 lines
8.8 KiB
Markdown
# Agent notes for local-page-archiver
|
||
|
||
## Project overview
|
||
|
||
This tool renders web pages in Chromium (via Playwright) and saves them as fully self-contained HTML files. All external assets (images, fonts, stylesheets) are inlined as data URIs so the resulting file works offline.
|
||
|
||
The pipeline is:
|
||
|
||
```
|
||
URL ──► Playwright render ──► inject privacy filters ──► inline assets ──► write HTML
|
||
```
|
||
|
||
## Source layout
|
||
|
||
- `src/cli.mjs` — CLI entrypoint. Supports `archive` and `help`. Accepts `--archive-path`, `--id`, and `--headful` flags.
|
||
- `src/archiver.mjs` — Core archiving logic. Loads privacy filters, steers the browser, injects adblockers/userscripts, and calls the inliner.
|
||
- `src/asset-inliner.mjs` — Fetches and inlines external resources (images, CSS, iframes). Also strips `<script>` and `<noscript>` tags for a static archive.
|
||
- `privacy-filters/` — Third-party filter lists and userscripts used to strip paywalls, trackers, and ad banners before the snapshot is taken.
|
||
|
||
## Privacy filters (`privacy-filters/`)
|
||
|
||
### `bpc-paywall-filter.txt`
|
||
|
||
An AdBlock Plus / uBlock Origin filter list. It contains three kinds of rules:
|
||
|
||
1. **Network rules** (`||tracker.com^`, `/regex/`) — block specific third-party paywall / tracking scripts.
|
||
2. **Exception rules** (`@@||example.com^`) — whitelist requests that a global block rule would otherwise hit.
|
||
3. **Cosmetic rules** (`example.com##.paywall`) — inject CSS to hide DOM elements (e.g. subscription banners, blurred overlays).
|
||
|
||
At module load time `archiver.mjs` parses this file into three arrays (`blockRules`, `allowRules`, `cosmeticRules`). Network rules are enforced at the Playwright level with `page.route(...)`. Cosmetic rules are injected as a `<style>` tag after the page reaches `domcontentloaded`.
|
||
|
||
#### Cosmetic rule caveats
|
||
|
||
Some advanced cosmetic syntax is unsupported and is silently discarded:
|
||
|
||
- `:remove()` — we can't actually remove DOM nodes from the CSS layer; only hide them.
|
||
- `:style(...)` — converted to real CSS `display: none`.
|
||
- `:xpath(...)`, `:upward(...)`, `:matches-css(...)` — discarded during parsing.
|
||
|
||
### `userscript/` directory
|
||
|
||
Contains Greasemonkey-style userscripts (`bpc.*.user.js`) plus a shared library `bpc_func.js`. They do heavy lifting: decrypt paywalls, reconstruct article text from JSON data embedded in the page, remove blur overlays, etc.
|
||
|
||
#### Selective injection (important)
|
||
|
||
Each userscript declares `@match` and `@exclude` metadata. **Only matching scripts are injected.** For example, on `bloomberg.com` only `bpc.en.user.js` is injected. Injecting all scripts into every page caused literal JavaScript source expressions to leak into DOM attributes, which the asset inliner then tried to fetch as URLs, producing garbage `HTTP 403` warnings.
|
||
|
||
The matching logic is a simple glob parser for userscript `@match` patterns:
|
||
- `*://*.com/*` matches any `.com` domain
|
||
- `*://example.com/path/*` matches that path prefix
|
||
- `@exclude` patterns take precedence and skip the script
|
||
|
||
#### `GM.xmlHttpRequest` mock
|
||
|
||
The userscripts rely on `GM.xmlHttpRequest` to fetch article text from archive mirrors or API endpoints. In a Playwright context this doesn't exist, so we inject a tiny mock that wraps the browser's native `fetch()` and presents the same callback interface (`onload`, `onerror`).
|
||
|
||
#### Timing
|
||
|
||
Userscripts are injected **after** `domcontentloaded` but **before** `networkidle`. We then wait an extra 2 s (`page.waitForTimeout`) so any `setTimeout(..., 1000)` callbacks inside the scripts have time to fire before we snapshot the DOM.
|
||
|
||
## Stealth / anti-detection
|
||
|
||
### The `playwright-extra` stealth packages are broken
|
||
|
||
The common recommendation is:
|
||
|
||
```bash
|
||
npm install playwright-extra playwright-extra-plugin-stealth
|
||
```
|
||
|
||
**This does not work.** All three packages on npm (`playwright-extra-plugin-stealth`, `playwright-extra-stealth`, `playwright-stealth`) are **placeholder packages** (version `0.0.1`) that literally throw on `require()`:
|
||
|
||
```
|
||
Error: Wrong package, please see this:
|
||
https://github.com/berstend/puppeteer-extra/issues/454
|
||
```
|
||
|
||
No functional Playwright stealth plugin exists in those package names. We therefore removed them from `package.json`.
|
||
|
||
### Manual stealth evasions
|
||
|
||
Instead we apply the same core evasions manually via `context.addInitScript()` and browser launch flags.
|
||
|
||
**Launch flags:**
|
||
- `--disable-blink-features=AutomationControlled`
|
||
- `--disable-infobars`
|
||
- `--disable-web-security`
|
||
- `--no-sandbox`, `--disable-setuid-sandbox`
|
||
- `--disable-dev-shm-usage`
|
||
- Removed `--enable-automation` via `ignoreDefaultArgs`
|
||
|
||
**Init script (injected into every page before any scripts run):**
|
||
```js
|
||
Object.defineProperty(navigator, 'webdriver',
|
||
{ get: () => undefined, configurable: true, enumerable: true });
|
||
window.chrome = window.chrome || { runtime: {} };
|
||
window.navigator.permissions.query = (/* patched for notifications */);
|
||
```
|
||
|
||
#### CRITICAL: Avoid `delete navigator.webdriver` + iframe trick
|
||
|
||
An earlier version used a more elaborate stealth snippet that did `delete navigator.webdriver` and then created an `<iframe>` to steal the real navigator descriptor. **This crashed the Chromium renderer process on tab creation** with:
|
||
|
||
```
|
||
Protocol error (Page.addScriptToEvaluateOnNewDocument): Target crashed
|
||
```
|
||
|
||
The current init script is minimal and safe — it only overrides the getter via `Object.defineProperty` and avoids DOM mutation during page init.
|
||
|
||
## Browser context & headful mode
|
||
|
||
`renderPage()` auto-detects whether a display is available (`$DISPLAY` / `$WAYLAND_DISPLAY`). If neither is set it defaults to headless. The caller can override via `options.headless`.
|
||
|
||
- **ViewPort:** 1366×768 (standard laptop resolution, not the default Playwright 1280×720)
|
||
- **Locale:** `en-US`
|
||
- **Timezone:** `America/New_York`
|
||
- **User-Agent:** macOS Chrome 130 (matches the Playwright image's Chromium version)
|
||
|
||
## Docker / Podman support
|
||
|
||
### Dockerfile
|
||
|
||
- Base: `mcr.microsoft.com/playwright:v1.60.0` (must stay in sync with the `playwright` npm version)
|
||
- Installs Node 22 (the base image may ship an older Node)
|
||
- Runs `npx playwright install chromium` so the browser binary is baked into the image
|
||
|
||
### `podman-run.sh`
|
||
|
||
Helper for local runs. Two modes:
|
||
|
||
1. **`./podman-run.sh archive <URL>`** — headless, mounts `./archives`
|
||
2. **`./podman-run.sh headful-archive <URL>`** — headful with internal VNC
|
||
|
||
**Headful mode details:**
|
||
The container's `ENTRYPOINT` is `node src/cli.mjs`. To run a shell command inside the container (setting up Xvfb + x11vnc) we must override the entrypoint:
|
||
|
||
```bash
|
||
podman run --rm --entrypoint sh <image> -c "...setup Xvfb... && node src/cli.mjs archive <URL>"
|
||
```
|
||
|
||
Port `5900` inside the container maps to `5901` on the host to avoid conflicts with macOS's built-in VNC.
|
||
|
||
### `docker-compose.yml`
|
||
|
||
Includes a `headful` profile that can be run with:
|
||
|
||
```bash
|
||
docker compose --profile headful up archiver-headful
|
||
```
|
||
|
||
## Known limitations
|
||
|
||
### Bloomberg bot wall
|
||
|
||
Bloomberg detects our requests as automated. Both headless and headful mode return **"Are you a robot?"** from this IP. We verified the same page text is returned by `curl` with identical headers, confirming the block is network-level (IP / TLS fingerprint / rate-limit reputation), not browser-fingerprint-level. To archive Bloomberg you currently need a residential proxy or to use an archive mirror service.
|
||
|
||
### Unsupported adblock syntax
|
||
|
||
- Advanced procedural cosmetic filters (`:remove()`, `:upward()`, `:xpath()`) are silently ignored.
|
||
- Scriptlet injection (`##+js(...)`) is not supported — only cosmetic CSS injection works.
|
||
- Preprocessor directives (`!#if`) are skipped.
|
||
|
||
## Adding a new site to privacy filters
|
||
|
||
If you add a new filter rule or userscript:
|
||
|
||
1. **Filter rules:** Edit `privacy-filters/bpc-paywall-filter.txt`. `archiver.mjs` reloads the file on every process start, so no code changes are needed.
|
||
2. **Userscripts:** Drop a new `.user.js` into `privacy-filters/userscript/` and add its filename to the `userScriptFiles` array inside `loadPrivacyFilters()` in `archiver.mjs`.
|
||
3. Test with `node src/cli.mjs archive <URL>` and inspect the generated HTML.
|
||
|
||
## Rebuilding the Docker image
|
||
|
||
Playwright npm version image tags must match:
|
||
|
||
```bash
|
||
# Check what npm installed
|
||
node -e "console.log(require('playwright/package.json').version)"
|
||
|
||
# Update the FROM line in Dockerfile if needed
|
||
# Then rebuild
|
||
podman build -t local-page-archiver .
|
||
```
|
||
|
||
## Development quick reference
|
||
|
||
```bash
|
||
# Install deps
|
||
npm install
|
||
|
||
# Install browser binaries
|
||
npm run install-browsers
|
||
|
||
# Archive a page (headless)
|
||
node src/cli.mjs archive https://example.com
|
||
|
||
# Archive a page (headful on macOS)
|
||
node src/cli.mjs archive https://example.com --headful
|
||
|
||
# Archive inside container (headless)
|
||
./podman-run.sh archive https://example.com
|
||
|
||
# Archive inside container (headful + VNC)
|
||
./podman-run.sh headful-archive https://example.com
|
||
# Then open vnc://localhost:5901
|
||
```
|