cURL Set Agent – Everything You Need To Know

What is cURL?

cURL (client URL) is a popular command line tool which is used to send requests to between servers to obtain information/data via HTTP requests, HTTPS and other protocols.

On most every operating system the cURL command line tool comes installed by default and can be used in terminal (command prompt for windows).

When using the cURL utility you set the target url also known as the destination. The destination can be a website or server as example and from that command, depending on the flags used, you’re able to acquire quite a bit of information.

Using cURL can be very helpful for debugging various issues on a website. Some basic use cases for cURL would include following redirects to test newly added rewrite rules, download files, or making POST/GET requests to test a web page.

Another useful feature is setting the user agent string with cURL to make a request seem as if its coming from a different browser or different device.

For debugging purposes if you’re making a lot of requests to a website while testing, it can be handy to change the user agent string. Mostly this is done to bypass security software you may have installed in case of a rate limit or block due to amount of requests.

What is a user-agent string?

In this article, we’ll be covering how to set the user agent string using the cURL command and some other use cases of the utility.

A user agent string is an identifier section within an HTTP header that shows where a request is coming from.

Such information provided in the HTTP header on a request includes the application from which the request originated from and the web browser type.

As an example of why you would want to change the user agent string, lets say you’ve had reports of clients receiving errors in the Mobile Safari version of your site. In this case you could set the user agent string to Safari Mobile and try to replicate the error for debugging purposes.

Also if you’re making a large amount of requests to a site with cURL while debugging it may be beneficial to change the user agent string to avoid being blocked by security software you have in place.

To expand a bit further on the user agent string and how it is used to identify a request we’ll see in the following example a snippet from an web server traffic log or transfer.log that shows the following pieces of data:

Note the below section which was taken from the screenshot above:

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

The snippet shown above displays the cURL user agent string of my request to a website. From this we see the browser type being Google Chrome and the operating system as Linux for where the request originated from.

Common user agents strings

Need to change user agents in a cURL request but unsure of the most common agents?

Below is a list of common user agent strings from popular browsers that can be used with cURL.

Later in the article we will show how a specific user agent can be specified with cURL and the following agent strings can be referenced later.

Google Chrome – Windows

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

Mozilla Firefox – Windows

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0

Microsoft Edge – Windows

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/4

Safari – Mac OS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0.3 Safari/602.3.12

How to change the cURL user agent string

Below is the basic syntax for how to set cURL user agent string. With it we can add in a specific user agent string of our choice.

To do so you’d execute the following within your terminal but adding in the desired user agent string and website as the destination:

$ curl -A "user agent example" http://example.com

From the above user agent example we’re using the -A flag in order to set user agents within the specifying the user agent with cURL and making the request to a website called example.com.

Common use cases for changing the user agent string

Downloading a file

As an example, lets say we want to download a file from “example.com” using cURL but we want the user-agent to be set to Safari.

To do so we would execute the following command to make a cURL request as the Safari user agent string to download a file from the example.com website:

$ curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0.3 Safari/602.3.12" -O https://example.com/sample-file.tar.gz

Making a post request

Without any flags a default cURL request executes a GET request type to a destination or target website/web server.

If you want to submit a POST request type using cURL and specify a user agent string you would use the example below:

$ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/4" -d "variable=data" https://example.com/blog

As shown above we are using the -d flag to make a POST request using the Microsoft Edge user agent string.

Within the command we are sending data to the target URL which is specified with “variable=data”.

The “variable=data” section of the command would be the information we’re trying to send with the POST request.

Follow redirects

Let’s say you recently made a rewrite edit in your websites htaccess file and you want to make sure the redirect is working properly.

In this case we could use the Safari user agent string in a cURL command with the -L flag to follow the redirect.

We’ll want to execute the following cURL request to do so and use HTTP in this example to follow all redirects, such as if we were making sure an HTTP to HTTPS redirect was working properly:

$ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/4" -L http://example.com/blog

Also just to note that by default cURL will use an HTTP request if not specified.

With that being said the same can be accomplished with the following and it will use http://example.com/blog as the target:

$ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/4" -L example.com/blog

Conclusion

cURL is an extremely useful tool for debugging issues on a website or web server and there may be instances where you’ll want to change the user agent string within a request.

In this article we went over some common use cases to get you started with setting the user agent header and provided example usage such as downloading files, making POST requests and following redirections with cURL.