Most of a web crawler is handling rejection

ID/EN

August 3, 2026

robots.txt, Cloudflare challenges, circuit breakers, and a queue that never drops a job: the 90% of crawling that isn't fetching a page.

A crawler's hard problem isn't fetching pages. It's everything servers do to avoid being fetched.

I learned that slowly, mostly at night, watching a log fill with 403s and soft-200s: responses that return 200 OK but hand back a Cloudflare challenge page instead of content. Fetching a page is one line of code. The other 90% of the work is handling the dozen different ways a server can say no.

This post is about that 90%: reading robots.txt, pacing requests per host, escalating past bot defenses only when forced, spotting challenge pages that lie about succeeding, and knowing when to stop retrying. That last one took me longest to get right. Every mechanism below is something I got wrong at least once first.

Politeness first, because rudeness gets you blocked

Early on I treated this as an adversarial problem: the sites blocking me must be detecting some trick I needed to hide better. That's backwards. Most blocks aren't clever. They're a rate limiter reacting to the fact that I sent a hundred requests in a second. The fix isn't stealth, it's not being rude in the first place.

Three things do most of the work.

robots.txt, with a cache that separates definitive failures from transient ones. I parse it with temoto/robotstxt and cache the result for 24 hours. The care is in the failure handling. A 404 means "no rules exist," a definitive answer, so I cache allow-all. A 5xx or a timeout means "I couldn't ask," so I cache nothing and retry next time. Getting this wrong in either direction hurts: cache a transient 5xx as a block and you lock yourself out of a site that was down for ten seconds; treat a real disallow as transient and you keep hitting a path you were asked to avoid.

A per-host delay clock. Each host gets max(my 1000ms default, the site's own Crawl-delay), capped at 60 seconds so a site advertising Crawl-delay: 3600 can't freeze a worker. The only non-obvious part is pacing without holding a lock while you sleep:

// One "next allowed time" per host. Reserve your slot under the lock,
// then release it and sleep. Concurrent goroutines aimed at the same
// host still get spaced correctly, and none block on the sleep.
func (p *politeness) pace(host string, delay time.Duration) time.Duration {
    p.mu.Lock()
    hp := p.hosts[host]
    now := time.Now()
    if hp.next.Before(now) {
        hp.next = now
    }
    wait := hp.next.Sub(now)
    hp.next = hp.next.Add(delay) // claim the next slot before releasing
    p.mu.Unlock()
    return wait // caller sleeps for `wait` outside the lock
}

Partitioning the frontier by domain. Each URL's domain is hashed with FNV-1a and routed to a fixed partition, so all of example.com always lands on the same worker. That makes per-host pacing a purely local, in-memory concern. No worker has to coordinate with any other to know when a host was last hit. The alternative, a shared token bucket in Redis, adds a network round-trip and a dependency for something that's really just a timestamp in a map. (The map is LRU-capped at 50,000 hosts.)

None of this is sophisticated, but it's the difference between a crawler that runs for months and one that gets its IP range blocked in an afternoon.

The escalation ladder

Some sites need more than politeness. They sit behind Cloudflare or a similar bot defense. The instinct is to reach for the heaviest tool immediately: a full headless browser that renders everything. But that's slow and resource-heavy, and challenge systems are tuned to notice automation, so leading with a browser isn't even more reliable. It's just more expensive.

So the fetcher escalates in tiers, cheapest first.

Tier 1: plain HTTP. A net/http request, 20-second timeout, body capped at 16 MiB. Every redirect hop is revalidated through an SSRF guard that blocks non-HTTP schemes and, after DNS resolution, refuses to connect to loopback, private, link-local, and the cloud metadata address 169.254.169.254. Most requests never leave this tier.

Tier 2, in principle: TLS fingerprint impersonation. The standard middle rung is making your TLS handshake (JA3/JA4) match a real Chrome and caching the cf_clearance cookie so a cleared host stops challenging you. I'll be straight that this tier is designed but not built yet. Today the fetcher jumps from tier 1 to tier 3. I'm mentioning it because leaving it out would misrepresent how the ladder is meant to work, and because anyone claiming a "finished" Cloudflare bypass is selling something.

Tier 3: a real browser, out of process. When a request comes back challenged, the URL goes to an external browser sidecar (a FlareSolverr-style service) over HTTP, not an in-process headless Chrome. Keeping it in a separate service means a browser crash or memory leak can't take the crawler down with it. Calls are bounded by a semaphore of 2 and a 30-second timeout.

The rule that matters: escalation happens on exactly two signals. An HTTP 403, or a response that returned 200 but is actually a challenge page (next section). A plain 5xx doesn't escalate. A server that's down is a retry-and-backoff problem, and spending a scarce browser slot on it is waste.

Detecting responses that lie

The failure mode that cost me the most time is the soft-200: the server returns 200 OK, but the body is a Cloudflare interstitial, not the page. Trust the status code and you'll store a million challenge pages and think the crawl is working.

Detection has two parts:

var challengeMarkers = []string{
    "just a moment", "challenge-platform", "cf_chl",
    "/cdn-cgi/challenge", "attention required",
    "enable javascript and cookies", "cf-browser-verification", "turnstile",
}

func needsEscalation(body []byte) bool {
    lower := bytes.ToLower(body)
    for _, m := range challengeMarkers {
        if bytes.Contains(lower, []byte(m)) {
            return true
        }
    }
    // Density check: a non-trivial page whose bytes are almost all
    // script/markup and almost no visible text. A JS shell that
    // hasn't rendered yet.
    return len(body) > 2048 && visibleTextLen(body) < 200
}

The first part is a list of strings that reliably appear in challenge pages. The second is a crude density check: strip <script> and <style>, strip the remaining tags, count what's left. If the raw HTML is over 2 KB but fewer than 200 bytes of visible text survive, it's almost certainly a JavaScript shell that hasn't rendered. Hand it to the browser tier, which can wait for the page to fill in.

The density check is imprecise on purpose. It occasionally misfires on a legitimately sparse page, but it's cheap and right the large majority of the time, which is the trade I want when the alternative is rendering every page in a browser just to be certain.

Knowing when to stop retrying

This took me longest, and it isn't really a technical problem. Sometimes the browser tier still can't get past a challenge, not because it's misconfigured, but because that site's defense can't be beaten today. My instinct was to retry. That instinct is wrong, and I have an incident to prove it.

One night a browser sidecar genuinely couldn't solve modern Cloudflare challenges. It wasn't crashed; it was up, working, and failing, burning the full timeout on every URL: 48 of 49 attempts timing out at 60-plus seconds each. Those calls piled up against the queue's in-flight limit and the whole crawl consumer stalled. By 02:45, 722 jobs were dead-lettered. One unbeatable host had stopped everything, because every worker was stuck waiting on it.

The fix is a circuit breaker, but the interesting part is the trip condition. The naive version counts failures per target and trips on a hard site. That's wrong: a hard site that still returns a real page isn't something you want to blacklist. So the breaker distinguishes backend health from target difficulty:

// Trip on the sidecar being sick, not on a target being hard.
// A dead backend: timeout, connection refused, unsolved challenge (Status == 0).
// A working backend that rendered a genuinely tough page (Status != 0)
// does NOT count against the breaker.
func browserBackendFailure(err error, status int) bool {
    return err != nil || status == 0
}

Five consecutive backend failures open the breaker for two minutes; after that it lets exactly one probe through to test recovery. A working backend that merely rendered a tough page doesn't count against it. I also cut the browser timeout from 90 seconds to 30. A stuck call that fails fast returns its slot to a URL that might actually resolve.

The queue never drops a job

Under all of this is a NATS JetStream work queue: one durable stream per pipeline stage, with WorkQueuePolicy retention so a message disappears once it's acknowledged. A failed job retries on a client-side schedule of 2s, 10s, 30s, 60s, 120s, up to 8 deliveries, then lands in a dead-letter state.

One detail here is a trap worth flagging. JetStream has a per-consumer BackOff field that looks like exactly what you want for retry spacing. I leave it unset deliberately, because setting it makes BackOff[0] override AckWait: a 2-second first backoff quietly shrinks your acknowledgment window to 2 seconds, and any handler that runs longer gets redelivered while it's still working. You end up running the same job several times over and it's genuinely hard to see why. The backoff lives in my own code instead, and AckWait stays a full 30 seconds.

Because NATS delivers at-least-once but I want each effect to land once, every worker writes its output and the next stage's job into Postgres in a single transaction (the transactional outbox pattern), and a relay polls that table every 500ms and publishes with a deterministic message ID so a race can't produce duplicates. A job that exhausts its retries isn't discarded. It's marked failed and left in place, and there's an operator command to redrive a batch once whatever blocked them has changed.

Finding new pages without scraping search engines

To discover new URLs, the obvious move is to query a big search engine. It's also the fastest way to get blocked. Search engines are the strictest of all about automated traffic, and a datacenter IP is the first thing they throw out. So instead of scraping results, I query official APIs built to be queried: a paid web-search API plus a few keyless encyclopedic and scholarly ones, fanned out in parallel, each returning its own batch, merged and deduped by normalized URL. If one is down the others still work. Leads discovered this way get a lower relevance prior (0.5) than curated seeds (0.9).

This is also where the worst bug of the project lived, and the fix is a small lesson about ordering. My richest source of new links is encyclopedic category pages, pages that aren't content themselves, just lists of links. The bug: I was running the relevance filter on a page before harvesting its links. A category page in a regional language, whose title and body don't match my single-language relevance vocabulary, would fail that filter and get discarded along with every link on it. I was quietly killing multilingual discovery at the source.

The fix was to decide what a page is before deciding whether it's relevant:

// Decide what the page IS before deciding if it's relevant.
// Lead/listing pages always get harvested for links, in any language.
// Only genuine content candidates go through the relevance gate.
func routePage(isDoc, isLead, isListing, relevant bool) disposition {
    switch {
    case isDoc:      return dispExtract      // a real document (even a PDF)
    case isLead:     return dispHarvestOnly  // e.g. an encyclopedia hub
    case isListing:  return dispHarvestOnly  // an index/pagination page
    case relevant:   return dispExtract
    default:         return dispReject
    }
}

To make regional-language hubs parseable I also had to teach the harvester the local word for "Category" across a dozen languages: tumbung in Banjar, kawan in Acehnese, dalala in Gorontalo, plus the obvious cognates for the neighbors. A whole set of pages that had been invisible because I couldn't read the label on them became reachable.

Why the failure handling is the point

A crawler doesn't have a finish line. It runs continuously, with no state where it's "done." What's worth optimizing isn't how often it succeeds but how it behaves when it fails. Most of what I've described is failure handling: a cache that fails safe, a breaker that gives up on one host without giving up on the rest, a queue that defers work instead of dropping it. None of it is clever. It's the part that decides whether the system runs unattended for months or falls over the first time a site fights back.