How We Cut Crawler Costs by Slowing Googlebot Down

Author: Nikolay Ishmametyev, Engineering Manager at Joom

Search crawlers generate a significant share of traffic to large marketplaces. At Joom, Googlebot is the largest source, although the same infrastructure handles other search crawlers as well.

This traffic is important: crawlers revisit millions of product pages to discover new items and refresh information about existing ones. But Googlebot traffic did not arrive at a steady rate. It repeatedly dropped for short periods and then caught up. At first, we treated this as an external traffic pattern that our infrastructure simply had to absorb. It turned out that our own serving behavior was part of the reason.

We introduced a request queue that dynamically delays crawler requests before releasing them to the product backend. This made the load more predictable and reduced the compute cost per million processed crawler requests by approximately 20%.

Why Googlebot Traffic Came in Waves

The request rate followed a repeating pattern: periods of high activity were interrupted by short drops, after which Googlebot quickly caught up.

The reason was the interaction between Googlebot's parallel requests and our own response time. Googlebot does not send an unlimited number of requests in parallel. Google describes its crawl capacity limit in exactly those terms: the total time a server spends holding connections open, factoring in both the number of parallel connections and their duration.

That makes the crawler and the server a single closed system. When some requests took slightly longer to complete, more of Googlebot's connections stayed occupied, so fewer new requests arrived. As those requests finished, connections freed up and the rate rose again, loading the backend a little more — and the cycle repeated. Because we passed incoming requests almost directly to the backend, this feedback loop showed up in backend load as well.

The metrics supported this explanation. When incoming RPS dropped, the number of in-flight requests increased rather than decreased: Googlebot had not stopped crawling, its active requests were still waiting to complete. We also saw the same pattern inside individual service instances, which made a monitoring artifact unlikely.

Individual drops lasted a few seconds and the cycle repeated roughly every minute - far too short for autoscaling to react to. We therefore had to provision enough compute for the high-rate parts of the cycle, even though that capacity was underused during every drop.

We considered two straightforward ways to handle this pattern.

• Provisioning for the peaks would protect the backend, but it meant keeping that extra capacity online continuously.
• A rate limiter that rejects excess requests could suppress the fluctuations, but only by setting the limit low enough to reduce the total number of requests processed successfully.

We needed a third option: smooth the short-term fluctuations without reducing the total crawler throughput. As Stripe notes in its rate-limiting article, pacing requests is a good fit when they can be spaced out without changing their outcome. That was exactly the property we could use for crawler traffic.

Smoothing crawler traffic with a request queue

We added a request queue in front of the product backend. Instead of deciding only whether to process or reject a request, it could also delay it:

1. process now
2. delay
3. reject

The queue sits in front of the product backend and assigns each crawler request a dynamic delay. When the incoming rate is below the configured target, requests proceed almost immediately. When it rises above the target, the queue adds a dynamic delay and spreads requests over time.

We calculate this delay using the Generic Cell Rate Algorithm (GCRA), a standard rate-limiting algorithm that represents accumulated load as time rather than as an explicit queue length. A bounded leaky-bucket queue could provide similar behavior. We chose GCRA because our main constraint was waiting time rather than queue size.

The delay is bounded. If a request can be released within the maximum waiting time, it remains open and proceeds to the backend later. If the calculated delay is too large, we return 429.

The goal is not to make the traffic perfectly flat. It is to absorb short-term fluctuations before they reach the backend and turn them into a steadier workload. This reduces the amount of compute capacity needed for the high-rate parts of the cycle without reducing the total number of requests processed over longer periods.

But bounded delay has a limit. Once the required waiting time exceeds the latency budget, we need a different failure mode.

Explicit throttling is better than accidental failure

Google recommends returning 429 or 503 when a server is approaching its serving limit. We use 429 Too Many Requests as an explicit fallback when the queue can no longer absorb the load, rather than letting requests fail deeper in the stack with 5xx.

However, 429 still has a business cost. Each rejected request means that Googlebot failed to fetch a product page on that attempt, while its price or availability may already have changed. If the same URL keeps returning 429 over several days, Google may keep stale information for it and eventually drop it from the index.

To reduce the impact of these rejections, we plan to make them recoverable. The idea is to asynchronously persist product URLs for which we return 429 and give them higher priority when Googlebot retries them. Once a page is served successfully, it can be removed from the recovery list.

What changed after the rollout

The queue did two things at once.

First, it did what we designed it for: it absorbed short-term fluctuations before they reached the product backend, turning them into a steadier workload.

Second, and less expected, Googlebot's own request rate became steadier. The queue made our response time predictable, so its connections stopped freeing up in bursts. The oscillation did not disappear — it moved out of the request rate and into the queue's waiting time.

This came with a trade-off: the smoother request rate was achieved by adding a few seconds of waiting time to most crawler requests. We monitor this delay separately and plan to test different limits across locales to quantify how it affects Googlebot crawling.

Google also documents that increased latency can lower the crawl capacity limit. We did not observe that: instead of crawling less, Googlebot kept its throughput by using more parallel connections. We keep an eye on this, since a bounded, deliberate delay and a genuinely slow server are not the same signal.

Stable backend load meant we no longer had to keep capacity online for the high-rate parts of the cycle. We track this as the compute allocated to serving and rendering crawler traffic, divided by the number of crawler requests processed — so the figure reflects efficiency, not traffic volume. After the rollout it dropped by approximately 20%.

Importantly, this reduction did not come from processing less crawler traffic. The number of crawler requests processed per day after the rollout remained at or above its pre-rollout level.

What we learned

The main lesson was that average request rate was not enough for capacity planning. Short-term oscillations forced us to provision for the high-rate parts of the cycle even though that capacity was underused during every drop.

A bounded request queue gave us a better trade-off: spend a few seconds of crawler latency to absorb those fluctuations, keep crawler throughput stable, and use compute resources more efficiently.