Last Updated:

Sending an http request through a proxy server with Python authorization

Sending an http request through a proxy server with Python authorization

Proxy server, theory

A proxy server is an intermediate link in your communication with the World Wide Web.

You send a request to the server, it goes to the proxy server, from there the request is sent on and on until a sufficient response is received. After the proxy server has received a quality response, it sends it to the client who made the request.

There are different types of proxy servers. Some additionally provide anonymity, some simply serve as "postmen" forwarding requests and responses. Some use high-performance machines for their activities, some weaker ones. However, there are also such proxy servers that ask for monetary contributions for their services, while free ones can use your network traffic for their own purposes: to sell information about your Internet activities to advertisers to analyze the improvement of marketing policy, for example. Be vigilant and attentive when choosing which proxy server to use, there is no such thing as a perfectly secure network.

Using a proxy for your surfing on the Internet is not necessary, but sometimes there is a need to protect yourself because of the frequent hacker attacks, you are extremely concerned about anonymity on the Internet or you are simply tired of a homogeneous search results in connection with the recent numerous targeted searches for some gizmo.

There are proxies with authorization, that is, in order to interact with this intermediate link, knowledge of the login and the corresponding password is required. Also, there are in the world and do not require authorization and they can be used absolutely freely.

Let's look at how to send a request using both methods.


Software

For the Python programming language, one extremely useful module called requests has been created. Its task is modest and extremely significant – to help in interaction with the global world wide web. It does not come out of the box for Python 3.x.x, and therefore requires a separate installation. You can read about how to install the module here.


Proxy without authorization

Let's consider an example of using a random proxy server that does not require authorization. Import the requests module, then define the proxy server to use with the transmission protocol to use (SOCKS4HTTPHTTPS, and so on) and the specified portcreate a request with the proxy server data specified earlier, send the request, and receive a response. In the response, we will be interested only in the status - the request was successful or failed.

The schema for creating a proxy data record and sending a request is as follows, the required data is provided by the proxy provider:

  1. An associative array is created, where the key is the protocol used. It is necessary to prescribe only a protocol suitable for your connection;
  2. The value of the associative array key is the ip address or domain name of the proxy server;
  3. At the end of the value, a port is prescribed. After the address, a colon without any spaces is placed and the port used is prescribed;
  4. Create a request specifying the resource you need in the form of a domain name or ip-address, adding an associative array containing vital data about the proxy connection;

 

The server returned a response, it contains a status code, if it is equal to 200, then this means that everything is fine and the request was executed successfully. If your request returned a response with a different code, then you need to deal with the malfunction, because the problem can be anything from a server failure and continuing with your connection to the network.


Proxy with authorization

Proxies requiring authorization are often used in production, in companies to create a protected and closed Internet environment for their employees. In such closed networks, there may be official information, access to databases necessary for the work of the company and much more interesting.

As mentioned earlier, for this kind of connection you need to know the correct login and password. Let's say we have it.

The schema for creating a proxy data record and sending a request is as follows, the required data is provided by the proxy provider:

  1. An associative array is created, where the key is the protocol used. It is necessary to prescribe only a protocol suitable for your connection;
  2. The value of the associative array key is the ip address or domain name of the proxy server;
  3. At the end of the value, a port is prescribed. After the address, a colon without any spaces is placed and the port used is prescribed;
  4. Separately from the proxy server data, you prescribe up-to-date and reliable authorization information using the HTTPProxyAuth class, which will form everything you need based on your two passed arguments: login and password;
  5. Create a request by specifying the resource you need in the form of a domain name or ip-address, adding an associative array containing vital data about the proxy connection, as well as authorization data;

 

Check the status code. If the code is 200, then everything is great, if not, then we look at the list of error codes and analyze the cause of failure.