The complexity of web scraping is directly related to the project’s needs. That’s why if you are serious about web scraping you know that an important and critical component of this process is proxy management.
My purpose with this article is to cover everything you need to know about proxies and their connection to web scraping. This will make your life easier. At least I can hope.
If we are talking about web scraping and proxies I will assume that what is an IP and how it works is familiar to you. If not, try to go through this information here for a more complex start in web scraping knowledge.
What are (free) proxies
A proxy is a third-party server that enables you to route your request through their servers and use their IP address in the process.
In other words, a proxy works as a postman that delivers your correspondence and brings back other information for you.
When using proxies, your IP address is not available for the website you are making the request. Instead, it sees the address of the proxy, which gives you the ability to scrape websites anonymously. The more proxies you have, the greater the data available for you to scrape and fewer possibilities to be blocked when scraping.
Proxies’ benefits
1. Forget about blocking: Proxies enable you to crawl a website much more easily. This reduces significantly the chances that your spider will get banned or blocked.
2. Be wherever you need to be: Proxies help you to make your request from a specific geographical region or device (mobile IPs for example) which allows you to see the specific content that the website displays for that given location or device (this is extremely valuable when scraping product data from online retailers).
3. Sky’s the limit: Proxies list allows you to make a higher volume of requests to a target website without being banned.
4. When one door closes, another opens: Proxies allow you to get around blanket IP bans some websites impose.
5. Your own personal army: Proxies enable you to make unlimited concurrent sessions to the same or different websites.
When talking about free proxy servers, the overall definition is that you can connect to them without needing any special credentials. Luckily, on the Internet, there are plenty of free proxy servers from which you can choose.
However, there are some aspects you need to consider when choosing a proxy server.
The most important thing to look at is the source of the proxy. Since proxies take your information and re-route it through a different IP address, they still have access to any online requests you make.
As the Internet offers a lot of solutions for any problem, choosing the right one can be quite a challenge. Luckily, we have created a list of 10 free proxies and proxy lists suitable for web scraping.
1. WebScrapingAPI
2. ProxyScrape
3. Proxy Nova
4. Spys.one
5. Open Proxy Space
6. Free Proxy Lists
7. SSL Proxies
8. Proxy-List
9. Free proxy
10. ProxyScan
Free proxy (lists) options and where to find them
1. WebScrapingAPI
If you’re considering using proxies while web scraping, my first recommendation is WebScrapingAPI. Being part of the freemium tools category, you can always create a free account and receive a number of 1000 free requests. With the help of the free plan, as well as the paid ones, you will have access to a free list of features to help you with web scraping.
The WebScrapingAPI features available through the free credits are beyond compare with free proxies described below. Some of them are JavaScript rendering and headless Chrome. What makes WebScrapingAPI stand out is that it offers access to support even to users with a free account. Moreover, the IP addresses are more secure and reliable.
Another great advantage is that you can customize geolocation, forwarded headers, and the cookies sent in the requests. For returned status code different than 200, you won’t be charged. Only successful requests need to be paid.
The downside of every free plan is that, if you need to scrape a lot of data, you will need to upgrade at some point to a premium version, no matter the provider. It offers you several advantages and features to achieve your goals easier and faster. Usually, the premium pack comes with customer support. If you have a problem, you can get in touch with the provider’s team. By using a free list instead of a provider, you won’t receive any support with your technical issues.
2. ProxyScrape
ProxyScrape offers three types of free proxy lists: HTTP proxies, Socks4 proxies, and Socks5 proxies. For all of them, there are sorting options like country, anonymity, and SSL. Sorting by country can be a bit confusing because they use only two-character country codes, rather than the full name or another common coding for countries.
In terms of anonymity, they offer 4 options: elite, anonymous, transparent, and all.
A feature that stands-out for the free version is the “timeout” slider, measured in milliseconds. This allows a user to limit proxy results to those which meet or exceed a certain timeout threshold. For free users, the lists update every five minutes, while for premium ones every 1 minute.
Advanced features like rotating proxies and many others are available in the premium service. However, ProxyScrape doesn’t have a free trial. Thus, users will need to pay for the benefits, which defeats the purpose of getting free proxies to begin with.
3. Proxy Nova
Proxy Nova provides a list of free proxies with detailed information. You can see Proxy IP, Proxy Port, Last Check, Proxy Speed, Uptime, Proxy Country, and Anonymity. The list can be filtered down by several attributes such as the port number of a proxy, the country of origin of a proxy, and the level of anonymity of a proxy. Also, the proxy list is updated every 60 seconds, but despite the frequent updates, there is no way of knowing how large the pool of free proxy addresses is.
One nice technical feature is the need for manual refreshing of the page, which helps the users take their time when searching for the right proxy. Because there’s nothing more annoying than finding a great lead and losing it because the page auto-refreshed and it’s impossible to find it again.
The data regarding each proxy is showcased in different ways — for Proxy speed, we have both a color-coded bar and a number for milliseconds, and for Uptime, we have a percentage.
4. Spys.one
Spys.one is one of the largest lists of free proxies. At the moment of my research, it offers proxies lists from 166 countries all over the world. Also, you can narrow your search by city, choosing from a list of 3350 available options.
Users can further refine their search with multiple sorting and filtering options: anonymous free proxy, HTTPS/SSL proxy, SOCKS proxy, HTTP, and transparent proxy so users can narrow down their searches. Each address is rated for latency (number indicator), speed (graph bar), uptime (percentage), and check date (this indicates when a proxy was last checked to be live).
At the moment, there were at least 500 proxies checked to be live in the past 9 hours.
5. Open Proxy Space
Open Proxy Space users can find proxies with HTTP, HTTPS (SSL), SOCKS4, SOCKS5 protocols organized in batches. Each one has a label for type, creation date, and the number of proxies. They are ordered by the time of creation, with the newer ones at the beginning. Users can explore lists that were created months ago. The older the list, the more dead proxies it will contain, and newer batches are going to contain the active proxies from those past lists anyway.
After the user selects a batch, they can choose which country or countries to include or exclude from the list, then export the IPs in a text document. There aren’t many sorting options, almost none for free users, but there is the possibility to access custom filters with the paid version.
Bonus point: friendly UI/UX, which in this context is a rare finding.
6. Free Proxy Lists
As the name suggests, Free Proxy Lists allows you to find them, smooth and easy. That’s because it has a simple and friendly layout. Here you can find only HTML and HTMLS proxies exclusively, which means that if you are looking for SOCKS proxies will need to find another provider.
You can specify search criteria like country, ports, and anonymity options. For some free proxy lists, you can sort by region or city, or alphabetically. So, if you are looking for a specific city/region that starts with Z, you will probably have to scroll through a lot of pages.
Each address has two color-coded bar graphs next to it representing the response and transfer levels. Unfortunately, there is no number indicator for these graphics, so they’re hard to compare. The only numerical data you will get is the uptime, which is represented as a percentage.
7. SSL Proxies
SSL Proxy offers around 8000 HTTPS free proxies, checked and updated every 10 minutes. As the name suggests, this list contains only HTTPS proxies, HTTP, and SOCKS proxies being offered in exchange for an amount of money.
The free proxies come from various countries, which you can sort. Additionally, you can choose a single one from a picklist. Anonymity sorting option is available too with two alternatives: anonymous and elite.
A premium version offers multiple features like rotating and SOCKS proxies.
8. Proxy-List
Proxy-List contains over 2,000 free proxies. The main lists are sorted into four options: HTTP, HTTPS, SOCKS4, and SOCKS5 which update every hour and contain only working proxies. Elite proxies are available at cost. After choosing the proxy type, you have the standard sorting functions: country and anonymity (for HTTP and SSL proxies).
A nice feature is the ability to export the proxy lists as a text file. However, the data can be also copied into one’s clipboard with the press of a button.
9. Free proxy
Fancy a time travel and finding over 8000 proxies for free? There you have it.
As usual, users can select from different protocols like HTTP, HTTPS, SOCKS4, SOCKS5, and anonymity levels like elite and transparent. The available proxies are showcased with information like country, region, city, speed (graph bar and number kB/s), uptime (graph bar and percentage), response (graph bar and number — milliseconds), and last checked. The users receive all the information they need to decide upon the best proxy.
A cool feature is the option to select “Proxies by category”, a button that opens a different page with proxies by port, by region, and by city. These sub-lists are alphabetized, but unfortunately cannot be sorted in other ways.
Another bonus point is that the website is available in different languages like English, Czech, French, Polish, Chinese, and Japanese.
10. ProxyScan
ProxyScan offers around 5000 proxies, all checked every 10 minutes before making the list. As most of the providers in this list, ProxyScan gives the possibility to filter and sort by different criteria. You can pick by country (which can be chosen from a picklist or written), max ping (response), proxy type (HTTP, HTTPS, SOCKS4, SOCKS5), and anonymity (transparent, anonymous, elite).
The search results are showcased in a table, where essential information is displayed. The ping (response rate) appears as a color-coded graph bar and numbers, while the uptimes appear as a percentage. The last checked proxies are the first in the list, which helps the users take an informed decision when choosing the right proxies for their needs.
What have we learned from all this?
There are numerous solutions for a simpler web scraping process and all come with their particular advantages and disadvantages.
Using proxies will help you secure your privacy and avoid adding your true IP address to any blocklists. So you’re free to scrape the web to get competitor information, find email addresses, or get other data from a website. For as long as you like, proxy scrapers help you keep your bots and crawling pages protected.
Although there are various online lists of free proxies, not all of them rise to the same standards. Be mindful of the threats that come with the use of free proxies. That’s why using free proxy services from providers you trust can be a better alternative.
