wayback-orchestrator

Parallel Wayback Machine downloader for power users. Crawl any number of domains at the same time, rotate through a pool of cheap datacenter proxies on every request (without paying for a "rotating" plan), spill completed files to S3 so your disk stays flat, and let a 10-minute optimization loop notice when a worker gets throttled and replace its IP automatically.

Measured: ~1,200 files/minute aggregate across 4 parallel splits on a single MacBook, using a 100-IP Webshare backbone plan + the in-tree downloader. That's ~620× faster than running pywaybackup directly against the same proxy.

Wayback Machine's documented per-IP rate limit is ~15 req/min. In practice they fingerprint sustained traffic and throttle a single IP from ~240 req/min down to ~20 req/min within hours. Downloading a mid-size archive site (e.g. a UGC dictionary or news archive with 80k unique URLs) from one IP takes 20-30 hours.

You can buy a "rotating residential" plan from any proxy vendor for $1.40+/GB, but it's overkill for archive.org (low-value target, mostly text). If you already have a "Backbone" or "Static Datacenter" plan with N usernames where each user is sticky-IP, this tool turns that pool into per-request rotation for free.

You also can't easily run two pywaybackup (the obvious off-the-shelf tool) processes against the same output dir (sqlite races), and pywaybackup's workers hang on dropped connections because they bypass HTTPS_PROXY entirely (it calls raw http.client.HTTPSConnection which ignores the env var). That made it impractical for serious throughput.

Two-part design:

1. wayback_downloader.py — a single-file 300-line scratch-built downloader. Uses requests.Session (native HTTPS_PROXY), enforces hard per-request timeouts (no socket hangs), writes an append-only JSONL ledger (no sqlite lock contention), and is resume-safe. Drop-in CLI-compatible with pywaybackup so it slots into the orchestrator with one config-line change.
2. The orchestrator — splits each domain into N disjoint date ranges, spawns one downloader per split, all going through a local rotating proxy forwarder (random Webshare user per TCP CONNECT = effective per-request IP rotation). A 10-min monitor loop watches throughput and rotates/escalates throttled splits. An S3 sync daemon continuously uploads finished files and (optionally) deletes locals so disk stays flat.

If you'd rather use upstream pywaybackup, the orchestrator still supports it — see patches/Worker.py for the HTTPS_PROXY patch you'll need. But the in-tree downloader is the recommended default since v0.2.0.

                            +----------------------------+
sites.yaml ─→ orchestrator ─┤  spawn pywaybackup × N     │
                            │  per domain (date splits)  │
                            └──────────────┬─────────────┘
                                           │ HTTPS_PROXY=http://127.0.0.1:8888
                                           ▼
                            +----------------------------+
                            │ proxy_forwarder (stdlib)   │
                            │ port 8888 → random Webshare │   per-request IP rotation
                            │ port 8889 → scrape.do      │   (fallback / opt-in)
                            └──────────────┬─────────────┘
                                           ▼
                                    web.archive.org
                                           │
                                           ▼
                            +----------------------------+
                            │  pywaybackup workers       │
                            │  write to splits/NNN/      │
                            └──────────────┬─────────────┘
                                           │
                            +──────────────▼─────────────+
                            │  s3_sync (every 5 min)     │
                            │  rclone copy → Scaleway    │   cleanup_local=true
                            │  delete local on success   │   keeps disk flat
                            └────────────────────────────┘

                            monitor (every 10 min):
                              parse pywaybackup progress logs,
                              compute rate_5m + peak,
                              ROTATE if rate < 0.4 * peak,
                              ESCALATE to scrape.do after 2 rotates

Four daemons (proxy_forwarder, orchestrator, monitor, s3_sync), one YAML sites file, one config file. ~1,200 lines of Python + bash total. macOS + Linux.

It is CDX-bounded, not a recursive crawler. On startup it issues one CDX query (https://web.archive.org/cdx/search/cdx?url=<domain>&matchType=domain) which returns every archived URL Wayback knows for the domain — including subpages. Those rows go into a sqlite DB; worker threads claim each row and fetch the snapshot. No HTML parsing, no link discovery. Implications:

• Subpages Wayback never archived → invisible to us (and to anyone using Wayback).
• Subpages Wayback did archive → all in scope automatically.
• --last mode → fetches the newest snapshot per unique URL.
• Infinite loop is technically impossible — counter is monotonic, sqlite row count is finite.

git clone https://github.com/YigitKonur/wayback-orchestrator
cd wayback-orchestrator
./install.sh

That script: checks for python3/pipx/rclone, installs pywaybackup via pipx, patches its Worker.py to honor HTTPS_PROXY, installs Python deps, seeds config.yaml from the example, and makes scripts executable.

Manual steps after:

Open config.yaml and replace the YOUR_* placeholders with your Webshare credentials, scrape.do token, and Scaleway bucket name.

Add a remote named whatever you set in config.yaml → s3.rclone_remote (default scw):

rclone config
# n) new remote
# name> scw
# type> s3
# provider> Scaleway
# access_key_id> SCW...
# secret_access_key> ...
# endpoint> s3.fr-par.scw.cloud (or your region)
# region> fr-par
# acl> private

Verify:

rclone lsd scw:

Bucket creation is automatic on first sync (or scw object bucket create name=YOUR_BUCKET region=fr-par).

python3 start_daemons.py

That launches proxy_forwarder.py, orchestrator.py, monitor.py, s3_sync.py as detached sessions (uses subprocess.Popen(start_new_session=True) — the portable equivalent of Linux setsid, since macOS doesn't ship setsid).

Verify:

ps -axo pid,command | grep -E 'wayback-orchestrator/(proxy_forwarder|orchestrator|monitor|s3_sync)\.py' | grep -v grep
# should list 4 processes

for i in {1..10}; do curl -sS -x http://127.0.0.1:8888 https://api.ipify.org; echo; done | sort -u | wc -l
# should print 8-10 (= unique exit IPs from your Webshare pool)

The everyday workflow is one command:

./add_site.sh example.com

This (a) appends an entry to sites.yaml, (b) sends SIGHUP to the running orchestrator, which reloads on its next 60s tick. Within ~1 minute orchestrator will:

1. Query CDX once for example.com to discover total snapshot count and earliest/latest timestamps.
2. Decide splits automatically: ceil(total / 12000), capped at 8.
3. Carve the time range into N disjoint date windows (e.g. 4 splits across 2002-2026 = ~6 years each).
4. For each split: mkdir state/example.com/splits/NNN/, spawn waybackup -u example.com --last -o splits/NNN/ --start <ts1> --end <ts2> --workers 8 --retry 5 --wait 30 --log with HTTPS_PROXY=http://127.0.0.1:8888.
5. Persist plan.json so the next tick knows what's planned vs running.

Common flags:

./add_site.sh huge-site.com --splits 8       # force splits count
./add_site.sh some-site.com --proxy scrapedo # use scrape.do instead of Webshare for this site
./add_site.sh blog.example  --mode all       # fetch every snapshot of every URL (not just latest)

To adopt an existing pywaybackup output dir (e.g. you already ran it manually for hours and don't want to lose progress), edit sites.yaml directly:

sites:
  - domain: example.com
    splits: 4
    status: running
    adopted_from: ~/Downloads/example-archive   # this dir will be moved into state/<domain>/splits/000

Then before launching the orchestrator: move the dir yourself and run the sqlite path-fix:

mkdir -p state/example.com/splits
mv ~/Downloads/example-archive state/example.com/splits/000
DB=$(ls state/example.com/splits/000/waybackup_*.db)
sqlite3 "$DB" "UPDATE waybackup_snapshots SET file = REPLACE(file, '$HOME/Downloads/example-archive', \
  '$(pwd)/state/example.com/splits/000') WHERE file LIKE '$HOME/Downloads/example-archive%';"

pywaybackup stores absolute paths in sqlite's file column; the SQL UPDATE keeps the resume cache valid after a move. Orchestrator's next tick spawns splits 001..NNN with the new architecture; split 000 picks up where it left off.

Live audit for a domain:

./audit.sh example.com

Sample output:

example.com — audit @ 2026-05-27T06:53:01Z
  CDX snapshots:             200,000
  DB rows total:             177,843  (000:87,797  001:60,195  002:18,115  003:11,736)
  Downloaded:                 10,251  (5.13%)
  HTTP 2xx files:             10,234  (5.12%)
  Failed/Other:                2,093
  Disk used:               621.8 KB across 32 files
  S3 uploaded:             242.5 MB across 9,853 files
  Cross-split duplicates:  3,477  (INVESTIGATE)
  ETA:                     ~6h12m
  Per-split rates (5m):    000:142/min  001:38/min  002:0/min(in ROTATE)  003:62/min

Cross-split duplicates should be small (single-digit %) — high values mean date boundaries are catching the same URL in two ranges. Acceptable up to ~5%.

Optimization decisions (one line per 10-min tick):

tail -f state/example.com/optimization.log

2026-05-27T06:35:10Z split=002 rate=212/min peak=237 done=1127/18115 (6.2%) err=2% status=OK
2026-05-27T06:45:10Z split=002 rate=18/min  peak=237 done=1175/18115 (6.5%) err=11% status=DECAY
2026-05-27T06:55:10Z split=002 ROTATE reason=decay rate=18 peak=237
2026-05-27T06:55:42Z split=002 RESPAWN pid=42112
2026-05-27T07:05:10Z split=002 rate=198/min peak=237 done=2120/18115 (11.7%) err=3% status=RECOVERED

Live S3 size:

rclone size scw:wayback-archive-YOURNAME

Stop everything:

./stop_daemons.sh             # stops the 4 daemons, does NOT touch running waybackup splits
pkill -TERM -f 'waybackup -u' # to stop the splits themselves (resume-safe via sqlite)

If you're considering scrape.do for a hot domain, run the bench:

mkdir -p /tmp/scrapedo-bench
python3 bench_scrapedo_api.py example.com /tmp/scrapedo-bench --max-seconds 600 --threads 4
cat /tmp/scrapedo-bench/_summary.json

It hits Wayback CDX once (via Webshare to be polite), then GETs each snapshot through https://api.scrape.do/?token=X&url=Y with N threads. Reports OK/err counts, throughput, and credit burn. scrape.do bills per successful request (~$0.0008 each on the Hobby tier).

Proxy mode (http://TOKEN:@proxy.scrape.do:8080) needs --insecure because scrape.do MITMs TLS and re-signs certs with their own CA, which requests rejects by default. The orchestrator auto-appends --insecure to wayback_downloader.py for any split routed through scrape.do; if invoking the downloader directly, add it yourself.

scrape.do proxy mode is SLOW for bulk — measured ~6 files/min vs ~500-700/min/site via the Webshare forwarder (MITM handshake + proxy-stack latency per request dominates). Use Webshare for crawling; reserve scrape.do for targeted/blocked URLs or use its API mode (bench_scrapedo_api.py), which is fast. Don't route a 50k-URL domain through scrape.do proxy mode — it'll take days.

Splits stay at the same counter for hours. Almost certainly your pywaybackup isn't actually using the proxy. Verify the patch:

grep -q "_build_connection" "$(pipx environment --value PIPX_LOCAL_VENVS)/pywaybackup/lib/python*/site-packages/pywaybackup/Worker.py" && echo OK || echo MISSING

If MISSING, re-run ./install.sh. Then check active TCP connections of a split — they should target 127.0.0.1:8888, not 207.241.x.x directly:

lsof -nP -p $(cat state/<domain>/splits/000/_pid) | grep TCP

Webshare rotation isn't actually random. 10 sequential curls through 127.0.0.1:8888 should give 8-10 distinct IPs. If they all match: forwarder process is using a single user for some reason. Check logs/proxy_forwarder.log for errors; restart with ./stop_daemons.sh && python3 start_daemons.py.

CONNECTION REFUSED -> could not query cdx server in a split log means pywaybackup's CDX query (uses requests) couldn't reach the proxy. Confirm 127.0.0.1:8888 is listening: lsof -nP -iTCP:8888 -sTCP:LISTEN.

S3 sync reports rc=N stderr=...InvalidAccessKeyId... — your rclone remote uses S3 API keys (separate from the API token in ~/.config/scw/config.yaml). Generate them in Scaleway console → Object Storage → API Keys.

address already in use when starting forwarder — old forwarder still running. ./stop_daemons.sh first, or lsof -nP -iTCP:8888 -sTCP:LISTEN to find and kill the holder.

pywaybackup ships http.client.HTTPSConnection("web.archive.org") in Worker.py. This Python stdlib class does not read HTTPS_PROXY from the environment — that's only respected by urllib.request and higher-level libraries like requests. The result: pywaybackup's CDX query (which uses requests) goes through your proxy, but every snapshot fetch (the bulk of traffic) connects directly from your home IP.

Our patch replaces that line with:

self.connection = _build_connection()

Where _build_connection() reads the env var and, if set, opens a CONNECT tunnel through the proxy with Proxy-Authorization: Basic header. See patches/Worker.py. We're upstreaming this; until merged, install.sh applies it locally.

MIT. See LICENSE. Credit to bitdruid/python-wayback-machine-downloader upstream for the actual downloading.

PRs welcome, especially:

• Linux / Docker packaging
• Cloudflare R2, AWS S3, Backblaze B2 sync alternatives (the s3_sync.py uses rclone so any remote should work — needs testing)
• Better cross-split dedup (currently date-disjoint splits can fetch the same URL with different timestamps)
• Native scrape.do proxy mode integration (needs SSL verify exception or custom CA install)