Policy-aware · Distributed · Written in Go
Crawl docs.
Keep their history.
DocWeave coordinates stateless workers through a durable PostgreSQL frontier—respecting host boundaries, recovering work, suppressing duplicates, and preserving content versions for change detection.
robots.txt honoredSSRF defendedPostgreSQL coordinated
SOURCEdocs.example.comexplicit boundary
host budget · 100ms
WORKER 01leasedpage/842
WORKER 02fetchingpage/843
WORKER 03parsingpage/841
WORKER 04readyqueue: 154
durable frontier
POSTGRESQLleases · versions · searchsingle source of crawl truth
1,000 fetched 0 duplicates 0 errors
Coordination without another queue
Workers scale out.
The rules stay global.
PostgreSQL leases provide crash-safe, at-least-once work distribution. A shared host-limit table ensures adding replicas never multiplies a site’s request rate.
Built for responsible collection
A crawler, not a scraping-evasion toolkit.
Boundaries are explicit.
Allowed hosts are validated up front. Private-network targets are rejected by default and redirects are checked again.
01One shared host budget.
Every worker transacts against the same rate limit, so horizontal scale never becomes accidental overload.
02Leases make crashes ordinary.
If a worker disappears, expired work returns to the frontier for another worker—without an extra queue service.
03Changes stay inspectable.
Conditional requests avoid unnecessary downloads while distinct content versions remain available for comparison.
04Measured host-politeness benchmark
1,000 pages.
Zero shortcuts.
A four-worker, two-process local crawl held one shared 100 ms host budget. The result measures coordination correctness—not maximum network throughput.
Read the benchmark methodology ↗POST /api/v1/crawls
{
"root_url": "https://docs.example.com",
"allowed_hosts": ["docs.example.com"],
"max_depth": 3,
"crawl_delay_ms": 500
}
→ live progress over server-sent eventsObservable by default
API, dashboard, metrics, and live progress.
Create and recrawl jobs through a versioned HTTP API. Search captured pages with PostgreSQL full text search, watch server-sent events, and inspect provisioned Prometheus and Grafana telemetry.
- Readiness and health checks
- Structured JSON logs
- Prometheus crawl metrics
- Provisioned Grafana dashboard
Open source · MIT