Curl vs Wget: Choosing the Right Tool for the Job

If you’ve worked with Linux systems for even a short while, you’ve surely heard of the two tools we use for interacting with remote data – curl and wget. While for simple tasks, we can use them interchangeably, they occupy quite distinct spots in the workflow. While curl is used to transfer data to and from servers and interact with remote servers as part of an overall workflow, wget is more of a dedicated tool that can replicate entire websites and has specialized options for recursive folder traversal, error recovery, and bulk downloads.

How to Use Curl

The syntax to use curl is simple. If you want to display the contents of a website, simply use

curl https://example.com

Note that this will display the output inside your terminal as shown here:

At first glance, this might seem useless. What’s the point of curl outputting its content into the stdout? After all, who wants to see the contents of their downloaded file in the terminal? But this behavior is deliberate, and it’s what makes curl so powerful.

But it can also be dangerous. For example, let’s create a small zip file in our current directory and use curl to “download” it. Here’s the output:

The zip file doesn’t contain regular human-readable text. So when you download it, it displays it in its raw, binary form. This can actually be dangerous, and so curl warns you when it’s about to do this kind of thing, and I’ve deliberately overridden it with the “–output -” parameter. Stuff like this can corrupt your terminal, so I don’t recommend it!

Why Curl Displays Output on stdout

What might seem like a nuisance is exactly what makes curl so useful. By redirecting the output of the response body to stdout, we can use curl in scripts with commands that pipe their output to other commands. For example, if the URL in question returns not an HTML page, but some JSON, then you can pipe the output of curl into the jq command, which takes JSON and manipulates it or extracts the data into human human-readable format.

Something like this, for example:

curl -s https://api.github.com/repos/curl/curl | jq '.stargazers_count'

Of course, it’s also useful if you just want to inspect the raw data for debugging purposes. For example, see the following command:

curl -I https://example.com

This gives us:

While debugging server connections, it’s super useful to be able to see the headers like this.

Useful Options to Use with Curl

Here are some of the commonly used options with curl. There are many others, but these are the ones I find most useful:

Save a curl download to a file instead of displaying the output in stdout:

curl -o file.txt https://example.com/data

If you use a small “o”, then you can specify your own file name. If you want to use the server’s file name instead, use a capital “O” like this:

curl -O https://example.com/file.zip

Follow Redirects:

By default, curl doesn’t follow redirects and will show you the response from the un-redirected page. For example, my personal blog redirects from HTTP->HTTPS. If I just use curl on the HTTP version, I’ll get something like this:

The response header shows that the URL has been permanently moved, and it also gives the location to which it is being redirected. But curl won’t follow the redirect on its own.

However, if I want curl to follow the redirects, I include the “-L” parameter like this:

curl -I -L http://www.bhagwad.com/blog

Now this gives me:

Now curl has followed the redirect until it hits a file code – in this case, the 200 response code shows that it’s reached its destination.

Sending POST Data with curl

Curl can do more than simply retrieve data. One of its biggest benefits is that it can send data to a server via a POST request. For example, let’s say we want to send the following      URL query parameters:

name=Alex
score=42

If this were a regular URL in the browser, it would look like this:

https://example.com/form?name=Alex&score=42

But to send it via curl, then you would use:

curl -X POST -d "name=Alex&score=42" https://example.com/form

Here’s an explanation of the parameters:

-X POST – Ensures that the HTTP method is “POST”.

-d “name=Alex&score=42” – sends the query data to the server

As you can see, we use curl for a lot more than just downloading files.

How to Use wget

In contrast to curl, wget’s essential features are geared towards downloading entire websites and storing them locally. On the face of it, wget’s syntax is pretty similar to curl’s:

wget [option]... [URL]...

But the actual options are very different. The easiest way to explain how wget works is to look at a full example. Let’s say I want to download an entire website onto my server. I can use:

wget --mirror --convert-links --adjust-extension --page-requisites \
     --no-parent --wait=1 --random-wait --user-agent="Mozilla/5.0" \     https://www.example.com/

The above command will create a mirror image of the website “https://www.example.com” and store it in a folder on the local server. Here’s an explanation of the options.

–mirror or -m

This is one of the most powerful features of wget, and it’s something that has no analogue in curl. It’s a shorthand for “-r -N -l inf –no-remove-listing” and it performs the following functions.

The “-r” parameter turns on recursive downloading. When wget encounters a link, it follows that link and downloads the target resource, then follows the links on that resource, and so on. By default, wget follows links to a depth of 5, but you can override it as shown below.

The “-N” parameter dramatically improves the efficiency of wget. Websites can take a long time to download, so there’s a good chance that it won’t be done in a single go. Either through an accidental disconnection or some other constraint, you’ll often have to pause the download in the middle and resume. The “-N” parameter tells wget to check the timestamp of the target resource on the remote server and see if the local copy (if it exists) has a newer timestamp. If so, then wget will skip that resource since it already exists. If you’re mirroring a site, then this is a very important parameter.

If you use “-l inf” wget will keep descending the tree of links forever. Use with caution for large websites!

Together, all these options make up the “–mirror” or “-m” parameter

–convert-links

This option changes the links on the local copy so that it becomes browsable offline. Otherwise, whenever you click a link on a local webpage, it’ll try and get the resource from the server. Remove this if you just want to create backups (though there are better tools for that).

–page-requisites

This option pulls not only the target webpage, but also the associated CSS and JS files.

Other Options

The other options, like “–no-parent”, prevent wget from moving above the starting directory, and “–wait=1 / –random-wait” ensures that wget pauses regularly to avoid overloading the server.

As you can see, the most powerful options for wget are all geared towards the mass downloading of resources. That’s all there is to it!

Conclusion

While both curl and wget can be used to download resources from a remote URL, the focus is very different. Curl is most often used as part of another script, feeding its output into another command, or being used to send POST requests, whereas wget is about mirroring remote resources and picking up where it left off. Wget is more of a dedicated browsing tool, and curl functions best as an interaction mechanism.