How To Use The Bash Heredoc Syntax In Linux

Linux allows you to use the bash prompt to provide the input for a variety of commands. For example, let’s say you want to quickly create a file with some text. The following will work nicely:

echo "Test file" > testfile.txt

This will create a file called “testfile.txt” as shown here:

You can see that we’ve created a new file with the text. This works well for small inputs. But as soon as you need to do something more, it becomes complicated. With line breaks, escaping special characters, and complications with expanding variables, we need a better way for large, complex inputs. And here’s where heredoc shines with bash.

Heredoc Syntax

Heredoc has a simple syntax:

command <<DELIMITER
text or script block
DELIMITER

In the above syntax, the “DELIMITER” can be any sequence of characters that you choose for the block. To end the block, you must use the same sequence on a separate line by itself with no leading or trailing spaces or tabs. Unless you’re careful, the heredoc block won’t end and you’ll get an error. This error is particularly difficult to debug because spaces and tabs aren’t visible, so make sure you pay special attention to avoid it!

By informal consensus, we generally use the delimiter “EOF” when using heredoc. It’s just a community standard to make it easy for us to more easily recognize heredoc statements. But you don’t have to use it and can choose something else instead.

Here’s an example of a heredoc statement to achieve the same thing as above:

cat <<EOF > testfile.txt
line 1
line 2
line 3
EOF

Like the previous “echo” statement, this too will create a file called “testfile.txt”. Note, however, that the input is spread across multiple lines. This allows you to create a text file using input that feels more natural. If you had to do this with echo statements, you would need it to look like this:

echo "line 1" > testfile.txt
echo "line 2" >> testfile.txt
echo "line 3" >> testfile.txt

This is considerably less readable than the previous code block. To be fair, you can also replicate the above by using echo with double quotes as shown here:

echo "line 1
line 2
line 3" > testfile.txt

While the above can replicate the output of heredoc, you need to be careful with escape sequences and variable expansion as discussed below.

Variable Expansion with Heredoc

Let’s say you want to include variables with your input. For example, we want to create a file with the current user and the current date. With heredoc, you can simply include the variables in the input and they’ll automatically expand. Like this:

cat <<EOF > testfile.txt
Welcome $USER!
Today is $(date).
EOF

Here’s what it looks like:

In this screenshot, I’ve included two variables in the heredoc statement – the date and the username. As you can see, I can just use them normally and the system expands them while inserting the text into the file. Pretty cool, right?

Once again though, to be fair, you can replicate this functionality with echo and “double quotes” like this:

echo "Welcome $USER!
Today is $(date)." > testfile.txt

The above code will replicate the heredoc functionality above. As you can see, echo is quite a powerful command. But I find that the real benefit of heredoc comes with the next feature – no need to escape characters.

No Need to Escape Special Characters

This is where heredoc shines. Special characters are used in many different types of inputs. For example, say you’re trying to send a JS block like this:

<script type="text/javascript">
    function processData() {
        let data = {
            "user": "John Doe",
            "age": 30,
            "address": "123 Main St\nCity, Country",
            "balance": 1000.50
        };
        alert(`Name: ${data.user}\nAge: ${data.age}\nAddress: ${data.address}\nBalance: $${data.balance}`);
    }
</script>

As for why you might be trying to use this as an input, there could be many reasons. I’ve done it myself when I want to dynamically insert some Javascript into my HTML page. But if I tried to provide this text as input to any command using the bash prompt, it would be a disaster. There are too many special characters like dollar signs, double quotes, and backslashes. If I had to escape every single symbol in the above code, it would look like this:

<script type=\"text/javascript\">
    function processData() {
        let data = {
            \"user\": \"John Doe\",
            \"age\": 30,
            \"address\": \"123 Main St\\nCity, Country\",
            \"balance\": 1000.50
        };
        alert(\`Name: \${data.user}\\nAge: \${data.age}\\nAddress: \${data.address}\\nBalance: \$\${data.balance}\`);
    }
</script>

As you can see, the above is quite disastrous. Not only is it hard to create, it’s also hard to read and debug! While it’s true that there are several tools that can escape you code automatically, it’s not sustainable in the long term.

Here’s what would happen if I tried to echo the Javascript code block without first escaping everything:

As you can see, none of the special characters made it into the final output. The only way to “echo” this is to escape all the dollar signs, newline characters, and double quotes.

But if we wanted to do the same thing with heredoc from bash, it’s super easy. Here’s the command:

cat <<EOF > script.js
<script type="text/javascript">
    function processData() {
        let data = {
            "user": "John Doe",
            "age": 30,
            "address": "123 Main St\nCity, Country",
            "balance": 1000.50
        };
        alert(\`Name: \${data.user}\nAge: \${data.age}\nAddress: \${data.address}\nBalance: \$\${data.balance}\`);
    }
</script>
EOF

And here’s the output:

As you can see, the code is entered into the “testfile.txt” file perfectly. There was no need to escape all the special characters like with did with the echo command. This simple example brings together all the benefits of HEREDOC and shows its power. No other tool can so easily accept arbitrary data without restrictions.

Particularly Useful for SQL Queries

One super useful scenario for HEREDOC is SQL queries. Here’s why.

First, SQL queries are often multi-line. With heredoc, you don’t need to worry about using escape characters to separate lines. Second, SQL queries often use bash variables like the date or username. With heredoc, you can use variables in the text directly. A third reason is that SQL queries are full of special characters like single and double quotes, semi-colons, dollar signs, and more. SQL queries are hard enough to debug as is. But if you had to read them through a haze of escape sequences, it would be an impossible chore.

Ability to Directly Execute Command with HEREDOC

One of the most useful features of HEREDOC is the ability to directly pass strings of text to commands. For example, here’s an SQL query:

mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" <<EOF
SELECT first_name, last_name, age
FROM users
WHERE age > $AGE_THRESHOLD
ORDER BY age DESC;
EOF

Here we have a command – mysql – and we’re using the HEREDOC syntax on bash to send a query directly. There’s no need to pipe the output from an echo command after escaping the characters. As a result, the final output is more readable. Note how I was able to use variables in the HEREDOC string without any problems.

Conclusion

HEREDOC is a superb tool for complex and long inputs for all the reasons above. Particularly regarding SQL queries with their long, multi-line inputs, variables, and special characters, HEREDOC allows you to input them in a very readable manner, compared to cruder tools like echo. Once you learn to use HEREDOC effectively like I did, you’ll find yourself using it everywhere!