How To Use rsync With “exclude” To Sync Files In Linux

The rsync command’s roots go back to 1996 – though not nearly as far back as some of the other Linux commands like awk. It was designed to efficiently sync files between two locations. At the time, the technology it used – the delta transfer algorithm – was a new one, but it has since become the standard for synchronization of this sort. I’ll explain further down how rsync maintains efficient transfers.

The “exclude” parameter in rsync is used to specify which files and folders you don’t want to include in the sync. I’ll give a complete overview below about what options you can use.

Basic rsync Syntax with “exclude”

Here’s the basic rsync syntax while ensuring that some files and directories are left out:

rsync [options] --exclude 'pattern' source destination

The options are self-explanatory. “pattern” is used to match the files and folders. “Source” and “Destination” refer to the locations from which you want to sync to the location you want to sync.

Detailed Syntax of Exclude with Use Cases

There are many ways to exclude items from rsync. Here’s a breakdown.

Excluding a Single File

To exclude a single file from an rsync operation, we use the following syntax:

rsync -av --exclude 'file.txt' /source/directory/ /destination/directory/

In the above example, we’re excluding the file “file.txt”. The options “-av” refer to “archive” and “verbose” that I’ll explain later.

Excluding Multiple Files

If you want, you can exclude multiple files at the same time by using several “–exclude” statements like this:

rsync -av --exclude 'file1.txt' --exclude 'file2.txt' /source/directory/ /destination/directory/

In this example, we exclude “file1.txt” and “file2.txt”.

Excluding a Directory

To exclude a directory, we use a slash (/) at the end of the pattern like this:

rsync -av --exclude 'dir/' /source/directory/ /destination/directory/

Here, “dir” is the name of the directory and the slash at the end indicates that it’s not a file.

Batching Files with Wildcards

It’s not feasible to use more than a couple of excluded files, so rsync allows us to specify a clutch of files using wildcards. For example:

rsync -av --exclude '*.tmp' /source/directory/ /destination/directory/

The above example excludes all files with the extension “.tmp” from the rsync operation. This is a very common use case since temporary files are not necessary to preserve the state of a project, and often only increase the bloat of the data transfer.

Including Multiple Patterns in a Reusable File

Sometimes, you might have created a detailed list of patterns that you want to exclude. Perhaps dozens of directories, or a huge list of files that are irrelevant to the synchronization process. If you were to include each of these patterns in a rsync statement, it would end up looking unwieldy. So rsync allows you to put them all in a single file and then specify that file while writing the command. We use the “exclude-from=” flag.

So a command like this:

rsync -av --exclude-from='exclude-list.txt' /source/directory/ /destination/directory/

Specifies that all the patterns for the exclusion will be found in the file “exclude-list.txt”. The file itself could contain the following text:

*.tmp
cache/
temp/

In this, I’ve included three patterns – all “.tmp” files, the “cache” directory, and the “temp” directory. Once again, these are common exclusions in projects that are backed up, since there’s no need to backup cache or temporary files.

How it Works

Let’s say I have two directories – a “source”, and a “destination” as shown here:

In the “source” directory, I’ve created two files and a single directory. As you can see, the “destination” directory is empty. Now I’m going to use the rsync command to copy everything from the source to the destination while excluding the “file1” file. The command is:

rsync -av --exclude 'file1' ./source/ ./destination/

And here’s the output:

You can see that rsync sends two items to the destination – file2 and the directory, while also excluding “file1” as specified in the command.

Useful Flags to Use with Rsync and Exclude

Two of the most common flags used with rsync and exclude are “a” and “v”. Combined, we write them as “-av”. Here’s what they mean.

Archive Flag for rsync

The archive flag, or “a”, is a conglomeration of many useful flags that we use to most closely replicate the status of files when performing a backup. Without these, it wouldn’t be a perfect copy. Here’s a breakdown of the various components.

-r: This recursively copies directories from the source to the destination. Without this functionality, rsync would copy only the files directly in the source directory and leave out any others in subdirectories.

-l: The “l” flag, or “–links” flag, copies symbolic links as they are between the two locations. Without this, rsync would, instead, copy the target files themselves to the destination location.

-p: This option preserves file permissions, ensuring that executable scripts, for example, retain their executability once copied. Otherwise, you’d have to redo all the permissions for all the files from scratch.

-t: This flag preserves the modification times for the files. Normally, when you copy and paste a file or directory to a new location, the “modified” time for the destination file changes to the time when it was copied. Using the “-t” option, rsync also copies the modification time, so that information isn’t lost and you can still sort them by date.

-g: This option preserves the group ownership of each file. Without this, each copied file will take on the ownership of the user running the rsync command. This way, you don’t lose information relating to who is allowed to access which file.

-o: Like with “–group”, this option preserves the owner of the original file. This requires super admin users since it’s a command that requires a user to be part of the sudoers group.

-D: This option preserves special files like device files, etc.

In all, the set of above options is meant to create as perfect a replica of the source location as possible. Hence, put together, they all work with the “archive” flag.

Verbose

The “–verbose” flag is just to let the end user know what’s going on as the rsync proceeds. Without this flag, if the syncing process takes a long time, it can be hard to just sit while nothing happens and the input prompt is missing. Even for background operations, you can use the verbose flag to output the results into a file so you can view them later.

Rsync is One Way by Default

An important point to remember is that rsync will not delete files from the destination directory. So if it’s not empty, it won’t be an exact copy of the source directory. If you want rsync to remove these files, you should use the “–delete” flag like this:

rsync -av --delete /source/directory/ /destination/directory/

When rsync completes, the source and destination directory will be the same.

Conclusion

As you can see, rsync is easy to use and has a universal addressing system that can transfer files between local and remote systems. The “exclude” flag is a powerful addition that can dramatically reduce the time it takes to synchronize, by excluding temporary and cached files. It’s a complete backup solution for any project along with the archive and verbose flags.