Cut your proxy bill before you switch provider

Say your job fetched a million pages last month and the invoice came to 2,300 GB. That is close to what you would get from the 2025 Web Almanac's median desktop page, which moves 2,412 KB over the network, so it is a fair stand-in for a crawler nobody has tuned.

I work for ResiFactory, which sells proxy bandwidth by the gigabyte, and I would still leave the rate until last. Before it comes a duller question, the one that pays: of those 2,300 gigabytes, which did the job need?

On 2026-11-06 I read the Web Almanac page-weight chapter, web.dev's page on text compression and RFC 9110. We then fetched eight public pages with curl, twice each: once asking for compression, once refusing it. One GET per page, direct rather than through a proxy, no images or scripts. Eight pages on one day from one machine is a sample, not a survey.

A gigabyte on the bill is not a gigabyte of pages

Our FAQ puts the meter in one line: "We meter outbound + inbound bytes through the proxy." So the request you send counts, the headers that come back count, and the body counts. Your scraper, if it logs anything, logs the body it saved.

For a 240 KB article that gap is noise, but an API call is a different animal, and the one we measured shows why: we asked GitHub's API for the metadata of a single repository and got 1,584 bytes of compressed JSON, which arrived inside 1,345 bytes of response headers and was preceded by our own 224-byte request. Of the 3,153 bytes a meter would count, the JSON you would actually store was barely half.

GitHub sends a lot of rate-limit fields, and another API might send a third as many headers, so I would not quote that ratio at anyone. I would quote the order of work it implies, which is that on a small-payload job you measure the overhead before you start shaving the JSON.

Then the divisor. Our meter divides by 1,073,741,824. If your logs divide by 1,000,000,000, your total comes out 7.4% above ours on identical traffic, and both are correct: they are two units called GB. Ask your provider which one it uses before you reconcile anything.

Blocks are billed too, and our own usage report does not soften it: "a challenge page is bytes delivered." When a news site refused our fetch on 2026-11-06, the 403 it sent had a 771-byte body under 3,612 bytes of headers, so with the request the refusal weighed 4,607 bytes. You would never notice one. A million of them is about 4.3 GB spent on pages that never arrived.

Find the expensive layer before you change anything

A monthly total tells you how much, never where. For where, you need bytes and request counts per target over a few weeks, enough to see a line move.

I know our dashboard best, so I will use it. The Usage tab is labelled "Data used, by pool and domain," and the domains table has eight columns: Domain, Traffic, Share, Requests, KB/req, Trend, Δ prev and Est. cost. Two of them do most of the work.

KB/req is described in the Insights panel as "Average payload weight per request. Rising means heavier pages or media." A target that sat at 50 KB per request in October and sits at 400 in November has either redesigned its pages or your job has started pulling something new. You want to know which.

Requests read against Traffic catches retry loops. The challenge-detection card covers traffic to known challenge endpoints, and it carries this note: "A high request count with low bytes usually means challenges are firing on your targets." Thousands of requests, a few megabytes: that is a scraper hammering a wall.

Est. cost carries its own warning, "Estimated from your blended purchase price, not a bill," and it means it: the column is for ranking domains against each other, while the money is in the ledger we invoice from.

History is the other limit. Per-domain detail stops at 90 days, so anyone who wants to compare this November with last November has to export as they go, through "Domains (CSV)" or "Full report (XLSX)" in the menu, or GET /usage/domains, which takes a date range, a pool filter and a cursor (API docs).

No per-domain breakdown at your current provider? Then the first job is yours, not theirs. Log host, response body bytes, header bytes and status code for every request your client makes, and you have the table above in a spreadsheet by the end of the week.

Compression is the large lever, but only on documents

Across the eight pages, asking for gzip or Brotli cut the total bytes per request, counting body, headers and request together, by between 61.6% and 88.9%; on the body alone the range was 74.2% to 89.7%. I went in expecting a large number, because web.dev already puts larger text files at up to 70-90%. What I had not expected was the floor. The worst page of the eight, the GitHub JSON, still lost three-fifths of its bytes, and that was the one where headers and the request, which we could not compress, made up half the total.

Three of the pages:

• A Wikipedia article went from 238,980 body bytes to 47,324, an 80.2% cut.
• An Allbirds product page went from 706,261 to 106,933, 84.9%.
• The Steam storefront went from 1,011,245 to 142,783, 85.9%.

web.dev gives "as high as 70-90% for larger files" of text, and all three land in that band. The Almanac's CDN chapter puts Brotli at "20-26% better compression than Gzip," so ask for br as well as gzip.

Run the same test on your own targets. Without --compressed, curl leaves the body encoded, so size_download is what crossed the wire:

URL='https://en.wikipedia.org/wiki/Web_scraping'
curl -s -o /dev/null -H 'Accept-Encoding: gzip, br' -w 'compressed   body=%{size_download} headers=%{size_header}\n' "$URL"
curl -s -o /dev/null -H 'Accept-Encoding: identity' -w 'uncompressed body=%{size_download} headers=%{size_header}\n' "$URL"

The saving is real only if your client is fetching uncompressed today. HTTP Archive defines its sizes as "the number of bytes sent over the network, which may be compressed," so the 2,300 GB I opened with already counts compressed bytes wherever the origin compressed them. Look at what your client sends in Accept-Encoding before you put this saving in a forecast.

Images are where this lever stops working. web.dev is direct about it: recompressing formats that are already compressed, "such as most image formats," yields "little to no improvement." Now set that next to the Almanac's breakdown of the median desktop home page, where images account for 1,058 KB and HTML, CSS and JavaScript added together come to 801 KB. If you run a browser fleet, the largest single part of each page you pay for sits where compression cannot touch it.

So: an HTTP scraper pulling documents should check one header first. A browser fleet should look at images.

Retries and duplicates, as arithmetic

I cannot tell you your retry rate. We did not measure customer traffic for this post, and a "typical" retry rate from anyone is a guess about targets they have not seen. The formula, though, fits in a line.

If a share r of your requests are retried, and each retry downloads the full body again, your bytes rise by r. On the million median pages:

• 10% full-body retries: about 230 GB on top of 2,300.
• 20% full-body retries: about 460 GB.

Multiply by the rate on your invoice. Ours is on the pricing page; the arithmetic does not care whose rate it is.

One objection deserves a full answer, because it is correct: retries are not all waste. A retry that returns the right page has bought you the page.

What you lose is narrower than the retry count suggests, and it comes in two sizes. A hard block can be small, 4.6 KB in our one measurement, which is why it shows up in Requests long before anyone sees it in Traffic. A soft failure is the expensive kind: the origin returns a full-sized page with the wrong content, your parser rejects it, and you have paid for the whole page once without getting it before paying again for the retry.

Duplicates are the same sum with a different cause. Suppose 200,000 of your fetches are URLs you already hold, at 500 KB apiece. Multiply, 200,000 × 512,000, divide by 1,073,741,824: about 95 GB. A set of canonical URLs you have already fetched, checked before each request, takes all 95 back.

For pages you revisit on purpose, prices or stock levels checked daily, conditional requests are the lever. You store the ETag or Last-Modified value from the last fetch and send it back as If-None-Match or If-Modified-Since; if nothing has changed, the origin answers 304, and RFC 9110 §15.4.5 says what that costs: "A 304 response is terminated by the end of the header section; it cannot contain content or trailers." Here is what the standard says, and here is what it means. It defines the response. It does not oblige any site to send validators, and if your target does not, there is nothing for your client to send back.

The same page can cost 50 GB or 274 GB a million fetches

Take the Wikipedia article we measured and fetch it a million times.

Compressed, no retries: 53,957 bytes a request, about 50 GB. Uncompressed: 245,590 bytes, about 229 GB. Uncompressed with 20% full-body retries: about 274 GB.

274 against 50 is not a promise. It is one page from one day with two of the problems in this post stacked on it. Your figure comes from running the curl lines above on your top three domains by Share and multiplying by your own request counts.

Start with the header if you scrape documents, and with images if you render

If you scrape documents over plain HTTP, check Accept-Encoding, then deduplicate, then read requests against bytes per domain to find the retry loops; if you render in a browser, start with images. Either way, get per-domain numbers first, which our pay-as-you-go pool gives you from the same ledger we invoice from, though every sum here works at the provider you already have.