Selective blocking of curl and wget
Wget is a pretty handy tool that lets users download files from any HTTP/HTTPS or FTP server. Curl, on the other hand is an amazing tool that helps you build a new request with plethora of protocols such as HTTP, HTTPS, FTP, Telnet, SMTP etc. These tools can be used to test protocols or your server configurations, but at the same time they can also be misused by a malicious attacker to exploit any website.
In order to thwart these malicious attacks, we need to put specific measures in place. One such measure is to block all requests from wget and curl. However, there are specific use cases where you require that external users can’t use these tools but specific IPs or servers can.
So the question is how can we achieve this functionality with Nginx?
One simple way is to block all requests from wget and curl using the sample code given below.
### Block wget user agent ###
if ($http_user_agent ~* (wget|curl) ) {
return 403;
}
This code block will disallow all requests coming from wget or curl.
In our deployment scenarios, we have some specific servers, which access our website with these tools. Hence, we need a custom solution, where we can restrict curl or wget requests from outside network but not from specific servers or IP addresses.
One simple way of achieving this functionality is using Nginx modules. For our solution, we will be using Map and Geo module and some internal variables defined in them.
1. Map module is used to set or create variables depending on the value of source variables.
For example:
[js]
map $http_user_agent $disallow {
default 0;
~*(curl|wget) 1;
}
[/js]
In the code shown above, we have created a variable named disallow whose value will depend upon the http_user_agent source variable. Now if the user agent is curl or wget, then disallow variable will take value 1 else it will remain 0, which is the default value.
Reference: ngx_http_map_module
2. Geo module is used to set the value of variable depending on the value of client IP address.
For example:
[js]
geo $myserver {
default 1;
x.x.x.x 0;
}
[/js]
Here, we have declared a variable myserver, whose value depends upon the IP address (x.x.x.x) mentioned in the block. If the value of IP address is x.x.x.x then myserver value will be 0 else it will remain 1, which is the default value.
Reference: ngx_http_geo_module
Now, based on the value of disallow and myserver variable, we will be able to reject requests coming from unknown IPs with a 403.
We can define these two modules in nginx.conf file under http block as shown below.
[js]
http {
geo $myserver{
default 1;
x.x.x.x 0;
y.y.y.y 0;
}
map $http_user_agent $disallow{
default 0;
~*(curl|wget) $myserver;
}
}
[/js]
The specific IPs (x.x.x.x and y.y.y.y) from which we want to allow wget and curl are defined in geo block of code. These IPs will set the value of the variable myserver to 0. Based on myserver value, the variable disallow in map block will be set. In the example shown above, curl or wget user agents used by whitelisted IPs ( x.x.x.x/y.y.y.y) will set the disallow variable value to 0.
Now, the decision to accept or reject the curl/wget request is based upon the value of disallow variable. This can be defined in the virtual host file of the nginx.
Sample Code
[js]
server {
location / {
if($disallow) {
return 403;
}
}
}
[/js]
For whitelisted IPs, this code will be bypassed and for any other external users 403 error will be given as a result.
Hi Gunjan, Have you considered the scenario that in curl and wget you can easily modify the user agent to be any other value? For instance in curl you can use -H ‘User-Agent=Poop’. This way the request will not be blocked by the method described in your post? Thanks