Order a server

Web scraping on a VPS with rotating proxies

Run a Scrapy crawler on a Linux VPS on a schedule, send its requests through rotating proxies, and keep it polite enough that nobody needs to block it.

updated Rack4#guides #python

Most scrapers start life on a laptop. That's fine until you want the data every morning, the laptop is asleep, and the site you're collecting from has started answering your home address with a CAPTCHA.

Moving the scraper to a small VPS solves the first problem: it runs on a schedule, all the time, on a stable connection. A rotating proxy gateway can help with the second, by changing the address your requests come from, on sites that allow the crawl. It doesn't guarantee access: a site can still refuse you for your request rate, your session or its own rules, and it's entitled to. This post sets up both, with Scrapy on Ubuntu, and ends with a crawler that runs itself once a day.

Why a VPS, and why proxies on top

A VPS gives a scraper what a laptop can't: it's always on, and it has a stable connection in a data centre. Pick a server location that suits the sites you're allowed to crawl. A small server is a good start for plain HTTP crawling: Scrapy is light, and how far one vCPU and 2 GB of RAM go depends on page sizes, concurrency, parsing and export. Measure memory on a real run before you raise concurrency.

What a VPS doesn't give you is a pool of addresses. A server has a small, fixed set: on Rack4, one IPv4 and one IPv6 address, and you can add a few more IPv4 addresses as add-ons. Many sites don't serve IPv6 at all, so most of a crawl comes from one or two IPv4 addresses, and sites notice when one address fetches a few thousand pages. The usual answers are rate limits, CAPTCHAs and blocks, sometimes for a whole range rather than just you.

A rotating proxy service sits between your crawler and the site. You send every request to one gateway, and the gateway sends it on from an address picked from its pool, by its own rules. How often the address changes, and whether one comes round again, is up to the provider. Your own server's address stays out of it, so it isn't the one that ends up on a blocklist.

One distinction, because it matters for our Acceptable Use Policy: this is about using a proxy service from your server. Running an open proxy on a Rack4 server, one anyone can relay through, isn't allowed.

Scrape politely first

Proxies spread your requests out. They don't make a badly behaved crawler acceptable, and a crawl that hammers a small site is a problem whichever address it comes from. Before you write any code:

  • Read robots.txt and the site's terms. Scrapy obeys robots.txt by default. Leave it on.
  • Go slowly. A few requests a second to one site is a lot for a small business's server. Scrapy's AutoThrottle backs off when the site slows down.
  • Collect what you need. Public prices, listings and articles are the usual targets. Personal data brings legal duties with it (GDPR in Europe, for example) whatever its source.
  • Say who you are. Set a user agent with a way to contact you. Site owners are far more patient with a crawler they can email.

Install Scrapy on the server

These steps use Ubuntu 24.04; Debian 12 is the same. Each block says who runs it: your own login with sudo, or the scraper user. Connect over SSH (see Connect with SSH if this is your first time). As your own login, create a user for the scraper so it doesn't run as root, then switch to it:

bash
sudo apt update
sudo apt install -y python3-venv
sudo adduser --disabled-password --gecos "" scraper
sudo -iu scraper

As the scraper user (the prompt now shows scraper@), create a project with its own Python environment. The project lands in /home/scraper/quotes:

bash
python3 -m venv ~/venv
~/venv/bin/pip install scrapy
~/venv/bin/scrapy startproject quotes
cd ~/quotes

Recent Ubuntu and Debian releases refuse a system-wide pip install, and the virtual environment is the clean way around that.

For practice we'll scrape quotes.toscrape.com, a site built for exactly this. Save this as /home/scraper/quotes/quotes/spiders/quotes.py:

python
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("a.tag::text").getall(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Still as scraper, run it once without proxies to check it works:

bash
cd ~/quotes
~/venv/bin/scrapy crawl quotes -O quotes.json

You should get a file with 100 quotes.

Route requests through the proxy gateway

A rotating proxy service gives you a gateway host, a port, a username and a password. Test them from the server with curl before touching Scrapy (as scraper or your own login; it doesn't matter here):

bash
curl https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo

The first line prints your server's own address. The other two should print addresses from the proxy pool. With a rotating gateway they often differ, but they don't have to: that depends on the provider. A 407 answer means the gateway didn't accept the username or password.

Keep the credentials out of your code. Type exit to leave the scraper user, and as your own login put them in a file only root can read:

bash
sudo tee /etc/quotes-scraper.env > /dev/null <<'EOF'
PROXY_GATEWAY=GATEWAY:PORT
PROXY_USER=USERNAME
PROXY_PASS=PASSWORD
EOF
sudo chmod 600 /etc/quotes-scraper.env

Scrapy already knows how to use a proxy: set request.meta["proxy"] to a URL with credentials in it and its built-in proxy middleware does the rest. So all we need is a small middleware that fills that in for every request.

One thing to check first: the middleware names a session in the proxy username, as USERNAME-session-<id>. That format is common but not universal, and some gateways ignore a session they don't understand. Look up your provider's format and change the user = line to match before you rely on it.

Switch back to the scraper user with sudo -iu scraper and add this to /home/scraper/quotes/quotes/middlewares.py:

python
import os
import secrets
from urllib.parse import quote

class RotatingProxyMiddleware:
    """Sends every request through the proxy gateway, each on its own session."""

    def __init__(self):
        self.gateway = os.environ["PROXY_GATEWAY"]
        self.user = os.environ["PROXY_USER"]
        self.password = quote(os.environ["PROXY_PASS"], safe="")

    def process_request(self, request, spider):
        # A request can ask for a sticky session; otherwise it gets a new one.
        session = request.meta.get("proxy_session") or secrets.token_hex(8)
        user = quote(f"{self.user}-session-{session}", safe="")
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"

Then switch it on and slow the crawl down in /home/scraper/quotes/quotes/settings.py:

python
DOWNLOADER_MIDDLEWARES = {
    "quotes.middlewares.RotatingProxyMiddleware": 350,
}

ROBOTSTXT_OBEY = True
USER_AGENT = "quotes-bot (+mailto:you@example.com)"
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True

The number 350 places our middleware before Scrapy's own proxy middleware (750), so the proxy URL is set by the time that one reads it.

What the session ID is for

The -session-... part of the username is the bit most guides skip, and it's why a "rotating" setup sometimes doesn't seem to rotate.

A rotating gateway picks a new exit address per connection, not per request. For HTTPS sites, your client opens a tunnel through the proxy and sends requests inside it, and the whole tunnel leaves from one address. Scrapy, like most HTTP clients, keeps connections open and reuses them, so without anything else a long run of requests can come from the same address.

Many gateways let you name a session in the username. The same session name asks for the same exit; a new name asks for a new one. Our middleware gives every request a fresh random name, so each request asks the gateway for a new session. Whether that means a different address every time is the provider's decision: pools reuse addresses, cap sessions, and may ignore a format they don't support. Scrapy's own connection handling also varies with its version and download handler, so check on a test run rather than assume. A new session costs a new TLS handshake, which is fine at polite crawl speeds.

Sometimes you want the opposite. Logging in, or paging through search results the site ties to one visitor, works better from one address. Set the session yourself for those requests:

python
yield scrapy.Request(url, meta={"proxy_session": "login1"})

Gateways also cap how long a sticky session holds its address, so long jobs should move to a new session name now and then.

Run it every day with a systemd timer

Type exit to get back to your own login; the rest of this section runs there, with sudo. Make a folder for the output, owned by the scraper user:

bash
sudo -u scraper mkdir -p /home/scraper/data

Create the service, /etc/systemd/system/quotes-scraper.service (for example with sudo nano):

ini
[Unit]
Description=Quotes scraper
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=scraper
WorkingDirectory=/home/scraper/quotes
EnvironmentFile=/etc/quotes-scraper.env
ExecStart=/home/scraper/venv/bin/scrapy crawl quotes -O /home/scraper/data/quotes.json

And the timer that starts it, /etc/systemd/system/quotes-scraper.timer:

ini
[Unit]
Description=Run the quotes scraper daily

[Timer]
OnCalendar=*-*-* 06:00:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

systemd reads the environment file before it drops to the scraper user, so the credentials can stay root-only. Start the timer and run the job once by hand to check it:

bash
sudo systemctl daemon-reload
sudo systemctl enable --now quotes-scraper.timer
sudo systemctl start quotes-scraper.service
journalctl -u quotes-scraper.service -n 30

The log ends with Scrapy's stats: requests made, responses by status code, items scraped. systemctl list-timers shows when the next run is due. Persistent=true means a run missed while the server was rebooting happens as soon as it's back.

Choosing the proxies

Which kind you need depends on the sites you're collecting from.

  • Rotating datacenter proxies are the place to start. They're fast and cheap per gigabyte, and most sites are happy with polite traffic from them.
  • Residential proxies exit through home connections. They cost more, and they're worth it only for sites that turn away data-centre addresses outright.
  • Unmetered plans, priced by concurrent connections instead of by gigabyte, make sense once your crawls move a lot of data.

Text pages are small, so a crawl of a few thousand pages a day uses little proxy traffic. Skip images and other files you don't need and it uses even less. Check your provider's usage page after the first runs rather than estimating.

We use rotating proxies from Node4 in our own examples. In fairness, we should say Node4 is run by Tekari, the same company that runs Rack4. Any provider with a rotating gateway works with the middleware above.

How big a server you need

Scrapy itself needs little. The smallest plan is a sensible place to start a plain HTTP crawl like this one. Watch free -h and the crawl's peak memory on real runs, and raise concurrency one step at a time; the limits you'll usually hit first are the site's patience and your proxy allowance.

That changes with headless browsers. If the site builds its pages with JavaScript and you need Playwright or Selenium to render them, each browser uses far more memory than a plain HTTP request, and how much depends on the pages. Measure one browser on your real pages before you run several. Choosing a plan covers the sizes.

When a run goes wrong

The journal (journalctl -u quotes-scraper.service) and Scrapy's closing stats tell you most of what happened. Some common cases:

  • 407 Proxy Authentication Required: the gateway refused the username or password. Check /etc/quotes-scraper.env, and the session format in the middleware's user = line.
  • Forbidden by robots.txt in the log: the site's robots.txt doesn't allow that path. Leave it out of the crawl.
  • Lots of 429 or 503 answers: you're going faster than the site wants. Lower CONCURRENT_REQUESTS_PER_DOMAIN and raise DOWNLOAD_DELAY. Scrapy's RETRY_HTTP_CODES and RETRY_TIMES decide what gets retried; AutoThrottle adjusts to response times and doesn't handle every refusal for you.
  • Timeouts: the gateway or the site is slow. Scrapy's DOWNLOAD_TIMEOUT decides how long it waits.

Don't paste logs with the proxy URL in them anywhere public: it contains your credentials.

Either way, the shape stays the same: a small server that never sleeps, a crawler that takes its time, and a gateway that changes the address your requests come from, used only where the site allows it.

Ready when you are

Order a server