How to Use the xargs Command in Linux

In Linux, we often need to feed one command’s output into another. The most well-known example is grep, which uses the pipe (|) operator to receive input and then search that input for a specific pattern. I’ve written earlier about using the pipe command with Linux, and we’re all familiar with how it works. Unfortunately, this approach doesn’t work with all commands. The pipe operator converts the output of a command into a “standard input” or “stdin” stream. Many commands will treat the output differently compared to arguments.

The following sections will make this clear.

Arguments vs Stdin

Let’s take the well-known grep command. We have a file called “file.txt” containing the word “apple”. Consider the following example:

grep apple file.txt

The above command will search the contents of the file “file.txt” for the pattern “apple”. This is because “file.txt” isn’t raw data from stdin, but an argument for the grep command. Now consider the following command, instead:

echo file.txt | grep apple

In this new command, “grep” doesn’t look inside file.txt but searches the filename file.txt. This is because grep is treating “file.txt” as raw data from stdin, and it assumes that you’re looking for the word “apple” in the filename, not the contents.

The screenshot above shows that grep didn’t find anything with the second command, which makes sense, because the word “apple” doesn’t exist in “file.txt”!

Using xargs to Convert Stdin Input into Arguments

When you need to convert the output from one command into arguments in a second command, we use the xargs command. This can completely change the output of a command. Here’s the syntax of xargs:

xargs command

This reads from standard input (stdin) and passes it to the command as arguments. Time for a practical example. Let’s say you have a folder full of files, and you want to count the number of lines in all text files that end in “.txt”.

Here, “wc -l” is the command that counts the number of lines in a file and prints the filename next to it. So, for example, the command:

wc -l file.txt

Would return something like this:

You can see that the file contains three lines, and so wc outputs the number “3”, followed by the file name. What if we now want this output for all the files in the folder, as well as the subdirectories?

Consider the following command:

find . -name "*.txt" | wc -l

On first glance, it looks as if this might be what we want. The “find” command searches for all files ending in “.txt”, and then we pipe the command to wc -l. But instead of the expected output, we get this:

Oops! What happened? Instead of counting the number of lines in each file one by one, the command “wc” has treated the output of “find” as stdin, instead of an argument. As a result, it has counted the number of files in the output of the “find” command! Since the find command returned 10 files ending with “.txt”, that’s what the wc -l command shows us.

How do we get wc -l to treat the output for the “find” command as an argument instead of raw data from stdin? Enter xargs!

Now, look at the following command:

find . -name "*.txt" -print0 | xargs -0 wc -l

This command is pretty much the same as the previous one, but now we’ve appended “xargs -0” to the front of the “wc -l” command. We’ve also added a new argument, “-print0”, to find, which I’ll explain later. No,w when we run this command, here is what we see:

Now you see that “wc” is counting the lines of all the files that “find” found, because we passed the results as an argument using xargs, instead of as raw stdin input. This example perfectly illustrates the use case of xargs over simply piping.

Safely Piping stdin as Arguments

In the above example, you might have noticed two differences between when I used “wc” just with the pipe, and when I used it with xarges. The differences are:

1. The “find” command has a new parameter -print0
2. xargs has an option -0

This is a common practice to avoid mistakes in the way xargs parses input. By default, xargs uses a blank space to “split” the input and feed the arguments one by one to the command. So, for example, if xargs receives:

apple banana cherry

Then it will send three arguments to the target command because each of them is separated by a space. However, if a word contains two names, then it can have unexpected consequences. For example, what if the arguments are like this:

apple musk melon banana cherry

It’s obvious to the reader that “musk melon” is a single fruit, but xargs obviously doesn’t know that. So it will send four arguments to the target command instead of three. For situations like this, we need to use a different delimiter and tell xargs to respect that. Hence:

xargs -0

The “-0” parameter tells xargs to split the input based on the “null character” instead of a whitespace. Filenames aren’t allowed to have null characters, so there’ll never be a danger of a single file accidentally getting split into two arguments when we’re using xargs with find.

The second part of the equation is that we need to tell “find” to separate its list of filenames using the null character instead of a newline (that gets sent to xargs as whitespace). But also, Linux allows files to have newlines in their name! So we need an unambiguous character, and that can never be present in a file name. That character is conveniently the null character. Hence, we use:

find -print0

The “-print0” argument tells the find command to separate the list of files with a null character instead of a newline. So when this goes to xargs, the latter can correctly split the list as expected.

Without these precautions, you can do a lot of damage. For example, if using the xargs command to mass delete files, you could accidentally remove the wrong file, which would be disastrous.

Using xargs to Overcome the “ARG_MAX” Limit

Linux systems have a limit on the length of arguments that can be passed to a command. You can check it using:

getconf ARG_MAX

Here’s what you can see:

It’s unlikely, but if you’re running a script and passing a long list of arguments to a command, then you could conceivably run into this limit, long as it is. But if you pass the arguments using xargs, then xargs takes care of this limitation for you by batching the arguments so that you don’t hit this limit. It’s a safe way to ensure that your code doesn’t break unexpectedly if you don’t foresee large inputs.

Conclusion

Xargs can transform the way a command works and allow you to combine existing commands in new ways by converting what would have been the stdin stream into arguments. You can use it to remove thousands of files in a single command. Just make sure that you parse the stdin properly, since xargs defaults to using whitespace as a delimiter. If available, consider splitting individual items using a null character instead.