Skip to content

ProxyFall proxies in Scrapy

Scrapy is a Python framework for crawling and scraping websites. Here's how to set ProxyFall up for large crawls with a new residential IP per request, in a couple of minutes.

Setup

Set up ProxyFall in Scrapy

  1. 1

    Scrapy's built-in proxy middleware reads the proxy from each request's meta, credentials included.

  2. 2

    Set meta={'proxy': ...} on your requests (or in a small downloader middleware for all of them).

  3. 3

    Rotate by leaving the session out; pin a site's crawl to one IP with -session-NAME.

Example

import scrapy

PROXY = "http://YOUR_LOGIN-country-us:YOUR_PASSWORD@gw.proxyfall.com:59731"

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request("https://ipinfo.io/json", meta={"proxy": PROXY})

    def parse(self, response):
        yield response.json()

Replace YOUR_LOGIN and YOUR_PASSWORD with the proxy login and password in your dashboard. The dashboard's generator writes these lines for any country, state, city or ISP.

Tips

Getting the most out of it

  • Keep CONCURRENT_REQUESTS reasonable: each request opens its own connection through the proxy.
  • Datacenter IPs (-type-datacenter) are cheaper for sites without strict anti-bot protection.

FAQ

Scrapy and ProxyFall: questions, answered

Do ProxyFall proxies work with Scrapy?

Yes. ProxyFall is a standard HTTP(S) and SOCKS5 proxy at gw.proxyfall.com:59731 with a login and password, which Scrapy supports. Targeting goes in the login, so nothing else needs configuring.

How do I rotate IPs?

Leave the session out of the login and every new connection gets a new IP. Add -session-NAME-lifetime-30 to keep one IP for 30 minutes (anything from a few minutes to 24 hours).

How much does it cost?

Residential traffic costs $0.99/GB for 1 to 49 GB and down to $0.86/GB from 1 TB, with every targeting option included and GB that never expire. Your first deposit of $5 or more, by card or crypto, adds 250 MB free.

More integrations

Other languages and tools

Every integration

Your first 250 MB are on us.

Free with your first $5 deposit, with every targeting option and sticky IPs up to 24 hours. Then grow at the lowest price per GB, with no expiry.

Create your account Compare prices

250 MB free trial

With your first $5 deposit

Get started