Scrapy + Mobile Proxies: Middleware, Rotation and Retry Handling

Scrapy has proxy support built in, which is why the integration looks like a one-liner and then fails in three specific ways. The one-liner is real. Everything around it — middleware ordering, credential handling, retries, and above all concurrency — is where a mobile proxy diverges from the datacenter pool Scrapy's defaults assume.

The working setup comes first, then the parts that bite.

The shortest setup that is actually correct

Set request.meta["proxy"] from a downloader middleware ordered below 750, put URL-encoded credentials in the proxy URL, and bring concurrency down to single digits. Scrapy's built-in HttpProxyMiddleware does the rest — it extracts the credentials and attaches Proxy-Authorization to ordinary requests and to CONNECT alike.

The middleware itself is about ten lines:

# myproject/middlewares.py
from urllib.parse import quote

PROXY_USER = "your_http_user"
PROXY_PASS = "your_http_pass"          # exactly as issued, unencoded
PROXY_HOST = "proxy.example.com"
PROXY_PORT = 30001

PROXY_URL = (
    f"http://{quote(PROXY_USER, safe='')}:{quote(PROXY_PASS, safe='')}"
    f"@{PROXY_HOST}:{PROXY_PORT}"
)


class MobileProxyMiddleware:
    """Applies one sticky mobile exit to every outgoing request."""

    def process_request(self, request, spider):
        request.meta["proxy"] = PROXY_URL
        return None

And the settings that go with it:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.MobileProxyMiddleware": 350,
    "myproject.middlewares.RetryAfterMiddleware": 560,
}

# Sized for one physical device, not a datacenter pool
CONCURRENT_REQUESTS = 2
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 2.0
RANDOMIZE_DOWNLOAD_DELAY = True

# Cellular round trips are slower than datacenter ones
DOWNLOAD_TIMEOUT = 90
RETRY_TIMES = 4

Note the quote(..., safe='') calls. Credentials inside a proxy URL are percent-decoded before use, so a literal @, :, %, / or # in the password lands in the wrong field and produces a 407 with a perfectly correct password. Encode each field separately, never the assembled URL.

You can skip the middleware and set the http_proxy and https_proxy environment variables instead — HttpProxyMiddleware reads them at startup. That works, but it is process-wide and invisible in your project's code.

What HttpProxyMiddleware does with your credentials

HttpProxyMiddleware runs at order 750 and is enabled by default. When it sees request.meta["proxy"] carrying userinfo, it removes the credentials from the URL, base64-encodes them, and stores the result as a Proxy-Authorization header on the request. The proxy value that remains is the bare host and port. This all happens silently.

That split matters for one reason: the header is cached alongside the proxy it was built for. If you switch between proxies by rewriting request.meta["proxy"], the credentials must be present in each URL you set so the middleware can rebuild the header. Setting Proxy-Authorization yourself and then changing the proxy underneath it is the classic way to send one endpoint's credentials to another.

The other trap is ordering, which runs in opposite directions depending on the hook. Request hooks run low number to high. Response hooks run high number to low.

Get one of those numbers on the wrong side and the symptom is not an error. It is a middleware that appears to do nothing.

The same middleware is the hook for per-spider exits. What you should not do is randomise the exit per request out of habit — see dedicated vs rotating proxies.

SOCKS5, HTTP, and where Scrapy resolves DNS

Scrapy's downloader speaks HTTP proxies only. There is no SOCKS5 support in the standard download handler, so socks5:// in request.meta["proxy"] does not quietly work — it fails. To use SOCKS5 you need a local forwarder that accepts HTTP proxy connections and speaks SOCKS5 upstream, or a replacement download handler.

For most Scrapy work that costs you nothing, and DNS is the reason. Fetching an HTTPS URL through an HTTP proxy means issuing CONNECT host:443 carrying the hostname, so the proxy performs the lookup. Resolution happens on the far side by default — on a mobile proxy that means on the device, over the carrier's resolvers, matching the carrier IP the destination sees.

That is the property browser automation has to configure by hand. With SOCKS5 the client chooses: socks5:// resolves on your machine and hands the proxy an address, socks5h:// sends the name. If you bridge Scrapy to SOCKS5, make sure the bridge is in hostname mode, or your DNS queries leave your own network while your TCP connections leave a phone. Background in SOCKS5 vs HTTP proxies; the case where hostnames break while literal IPs work is its own diagnostic.

A useful consequence: if your Scrapy crawl works and the same target fails from a SOCKS5-configured browser, suspect resolution mode before you suspect the proxy.

Concurrency: a device is not a pool

This is the section that decides whether the integration works. A mobile proxy exits through one physical handset with one radio and one uplink. Parallel requests do not add capacity — they split a fixed resource, and past a point they degrade the device itself. That is a property of the hardware, not a limit in Scrapy.

Three mechanisms stack, and they present differently enough to separate.

The uplink is the ceiling. Roughly 2-3 MB/s per device, set by the cellular upload path — everything the destination sends back has to be pushed up from the phone before it reaches you, and carriers allocate far less capacity upward than downward. More workers do not move it. Slow speeds in detail.

High fan-out can wedge the device. All of your concurrent connections share one multiplexed tunnel, and under very heavy parallel fan-out that multiplexer can wedge: the phone goes quiet while still appearing connected, and a watchdog recovers it in about three minutes. The trigger is simultaneous streams rather than request rate, so cutting the worker pool helps more than adding delays.

Bursty parallelism looks like automation. Thirty-two simultaneous connections from a consumer carrier address is not a shape a phone produces, and it is a reliable way to start collecting 429s.

Practical starting point per device: CONCURRENT_REQUESTS between 1 and 4, DOWNLOAD_DELAY between 1 and 5 seconds, and RANDOMIZE_DOWNLOAD_DELAY left on so the gaps are not metronomic. Then let AutoThrottle find the rest:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0   # keep it near one request in flight
AUTOTHROTTLE_DEBUG = False

AutoThrottle adjusts delay from observed latency, so it responds to a struggling device instead of hammering it harder, but it tunes the gap between requests, not the number of them, so it is no substitute for a sane CONCURRENT_REQUESTS. If you need throughput, add devices, not workers.

Retries, Retry-After, and the seven-second rotation gap

Scrapy's default RETRY_HTTP_CODES includes 429, so the built-in RetryMiddleware will retry a rate-limited request. What it will not do is read Retry-After. It applies its own schedule, so the retry frequently lands inside the window the server just told you to wait out, which is how a short throttle escalates into a long one.

Retry-After comes in two formats: a delay in seconds, or an HTTP-date. A parser that only handles integers breaks on the second, so handle both.

# myproject/middlewares.py
import time
from email.utils import parsedate_to_datetime

from scrapy.downloadermiddlewares.retry import get_retry_request

MAX_WAIT = 120.0


def parse_retry_after(value):
    """Retry-After is either delay-seconds or an HTTP-date. Handle both."""
    value = value.strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, when.timestamp() - time.time())


class RetryAfterMiddleware:
    """Honours Retry-After on 429/503, then re-schedules the request."""

    def process_response(self, request, response, spider):
        if response.status not in (429, 503):
            return response

        raw = response.headers.get("Retry-After")
        wait = parse_retry_after(raw.decode("latin-1")) if raw else None
        if wait is None:
            attempts = request.meta.get("retry_times", 0)
            wait = min(MAX_WAIT, 2.0 ** (attempts + 1))
        wait = min(wait, MAX_WAIT)

        # Slow the whole slot down, not just this request.
        # AutoThrottle adjusts the same field.
        slot = spider.crawler.engine.downloader.slots.get(
            request.meta.get("download_slot")
        )
        if slot is not None:
            slot.delay = max(slot.delay, wait)

        new_request = get_retry_request(
            request, spider=spider,
            reason=f"{response.status} retry-after {wait:.0f}s",
        )
        return new_request if new_request is not None else response

Two details are load-bearing. get_retry_request re-queues with the retry counter intact and returns None once RETRY_TIMES is exhausted — hence the fallback rather than an infinite loop. And raising slot.delay instead of sleeping keeps the reactor free, which is what AutoThrottle does internally.

Rotation needs the same treatment for a different reason. It is an airplane-mode toggle on a real handset: the radio detaches, the bearer is torn down, and on re-attach the carrier assigns a new address. That is about seven seconds of genuine interruption, almost all of it carrier-side radio work. Requests in flight fail, and an immediate retry lands in the same hole. Back off a few seconds and treat the first failure after a rotation call as expected. If a rotation returns success in roughly two seconds, nothing happened; that is a wedged radio.

Symptom, cause, fix

Most Scrapy proxy failures are one of a short list, and they are separable by symptom rather than by log spelunking. Place yours here before changing settings, because the most expensive mistake is applying a fleet-wide change to a single-device fault, or trying to tune around the uplink ceiling, which cannot be done.

When this becomes an infrastructure problem

Everything above is client-side and free. It stops being a Scrapy question when the settings are correct, the middleware ordering is right, Retry-After is honoured, and the crawl still degrades, because then the variable is the exit device, and Scrapy has no visibility into it.

The distinction that matters is shared pool versus dedicated device. On a shared pool, a stall or a run of 429s could be your concurrency or another tenant's, and nothing you can run separates them. On a dedicated device the crawl is the only traffic on that radio, so the chain from CONCURRENT_REQUESTS to observed behaviour is legible, and rotation is an endpoint you call rather than a ticket you file. That is the shape QuantumProxy sells — one real 4G/5G handset per customer, both protocols exposed, rotation as an API, and the reasoning holds whoever you buy from. The mechanism is in how mobile proxies work.

The honest limits. A mobile exit does not raise a quota that was never about IP addresses. It does not make a crawl undetectable, because address reputation is one input among pacing, header consistency and TLS fingerprint. And it does not lift the throughput ceiling — a scraper needing sustained bulk transfer is on the wrong tier entirely.

Two checks before you file a bug

Two commands tell you which half of the problem you are in. The first compares Scrapy against a client with no middleware stack at all; the second removes concurrency as a variable. Between them they separate "Scrapy is misconfigured" from "the proxy is unhealthy" faster than reading any log, and both take under a minute.

Make the same request with curl -v through the same proxy, from the same machine. If curl works and Scrapy does not, the proxy is fine and the fault is middleware ordering or credential handling.

Then drop CONCURRENT_REQUESTS to 1 and DOWNLOAD_DELAY to 5 and re-run a short crawl. If the failures disappear, you had a concurrency problem, and the fix is a device count rather than a setting.

If both pass and the target still refuses you, the transport was never the issue — that is a detection question, and the general diagnosis order beats more middleware. For browser-driven work under the same constraints, see Puppeteer and Playwright.

Frequently asked questions

How do I set a proxy in Scrapy?

Set request.meta["proxy"] to a proxy URL such as http://user:pass@host:port. The built-in HttpProxyMiddleware, enabled by default at order 750, reads that value, strips the credentials into a Proxy-Authorization header, and routes the request. You can set it in a spider callback, but a downloader middleware ordered below 750 is the maintainable place because it applies to retries and redirects too.

Does Scrapy support SOCKS5 proxies?

Not natively. Scrapy's downloader supports HTTP proxies and tunnels HTTPS through them with CONNECT. There is no SOCKS5 handling in the standard download handler. The usual workarounds are a local forwarder that accepts an HTTP proxy connection and speaks SOCKS5 upstream, or replacing the download handler with one that does. If you go the SOCKS route, make sure hostnames are resolved on the proxy side.

What should CONCURRENT_REQUESTS be with a mobile proxy?

Low — usually somewhere between 1 and 4 per device, with a real DOWNLOAD_DELAY on top. A mobile proxy exits through one physical handset with one radio and one uplink, so parallel requests split a fixed resource rather than adding capacity. High fan-out also degrades the device itself, which is a hardware property rather than a software limit.

Why does Scrapy return 407 when my credentials are correct?

Usually one of three things: the credentials contained characters that were not URL-encoded and got parsed into the wrong field, a cached Proxy-Authorization header from a previous proxy is being reused, or the credentials belong to a different protocol's port. Some providers issue separate username and password pairs for SOCKS5 and HTTP, and crossing them produces a 407 with credentials that are genuinely valid.

How do I make Scrapy honour Retry-After?

Add a downloader middleware whose process_response reads the Retry-After header on 429 and 503, parses both the delay-seconds and HTTP-date forms, raises the download slot's delay, and re-schedules the request with get_retry_request. Give it an order above 550 so it sees the response before the built-in RetryMiddleware does.

Should I rotate the IP inside a Scrapy run?

Only if your workload has no session identity to protect. Rotation is a physical radio re-attach and costs about seven seconds of interruption, so requests in flight will fail and need a short backoff before retrying. For crawls that carry cookies or a login, rotating mid-run destroys the continuity that made the session credible in the first place.