16 Practical rsync examples

Rsync is a fast and versatile file synchronization tool that allows to copy and sync files locally and from rsync service or any device that supports remote shell (Rsync does not support synchronization between remote devices). Rsync offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the set of files to be copied.

Rsync implements a very efficient transfer algorithm based on deltas which reduces the amount of data sent over the network due it sends only the differences between the source and the destination file.

Rsync is widely used for backups and mirroring.

Rsync finds the files that need to be transferred using a quick check algorithm (by default), looking for files that have changed in size or in last modified date.

Other features:

+ Allows you to copy links, devices (In GNU/Linux all devices are identified as special file under /dev DIR), owner, group and permissions

+ Exclude and exclude-from options similar to GNU tar

+ Exclude files in CVS style

+ Can use any transparent remote shell, including ssh or rsh

+ Does not require super privileges

+ Support pipelining to minimize transfer time

+ Supports anonymous authentication via rsync service (ideal for mirroring and backup)

Rsync can access the remote host using a remote shell mechanism or contacting directly the rsync service via TCP. If the source or destination contains a colon (:) after the host name then the remote shell will be used as the transport mechanism. If the source or destination contains a double colon (::) after the host name or if the Rsync protocol (rsync://) is used then Rsync will contact the rsync service directly.

If you only specify the origin Rsync will be similar to ls -l.

If both the origin and the destination are local paths then Rsync behaves like an improved copy (cp) command.

Synopsis

$ rsync [OPCIONES] SRC [DEST]

SRC and DEST can have this format:

user@host:path

if you do not specify user then Rsync will use the current user.

Examples

1. Transfer/Sync using wildcards

Transfer/Sync all files from the current DIR that match *.c pattern to the src DIR (relative to $HOME for current user) of the helios host. If some of the files to be transferred exists in the remote DIR then Rsync only sends the differences. The modification date is preserved.

$ rsync -t *.c helios:src/

2. Transfer/Sync in archive mode

Transfer/Sync all files from the src/bar DIR (relative to $HOME for current user) of the helios host to the local DIR: /data/tmp.

$ rsync -avz helios:src/bar /data/tmp

The option -a is a shortcut to:

-r: recursive transfer
-l: transfer/copy symlinks
-p: preserves permissions
-t: preserves the modification date of the file
-g: preserves the group that owns the file
-o: preserves the owner to which the file belongs
-D: transfer/copy devices and special files (for the root user only)

3. Transfer/Sync content under a DIR

If you specify a / at the end of src/bar instead of creating the DIR bar under /data/tmp the files (and dirs) under bar is copied. A slash at the end of the origin can be interpreted as "copy the contents of this directory" instead of "copy this directory"

$ rsync -avz helios:src/bar/ /data/tmp

4. Transfer/Sync certain files

Specify several files at the same time. Copy the files: fich1, fich2, fich3 and fich4 from helios to the local DIR /dest. All commands specified below are equivalent.

$ rsync -av helios:fich1 :fich2 :fich3 :fich4 /dest/
$ rsync - av helios:fich1 helios:fich2 helios:fich3 helios:fich4 /dest/
$ rsync - av helios:fich1 :fich2 helios:fich{3,4} /dest/
$ rsync - av helios:fich{1,2,3,4} /dest/

5. Transfer/Sync files with white spaces

To copy a file that contains white spaces, use the argument --protect-args (-s)

$ rsync -avs helios:'I am a file with white space.txt' /dest/

6. Disable an option

To disable a particular option which is implicitly active in a previous option we prefix it a no, if in the example 2 we would like that the owner of the file is the user who receives the files on the destination server then execute.

$ rsync -avz --no-o root@helios:/root/src/bar /data/tmp

If you omit --no-o then all files copied under /data/tmp will be owned by root while if you specify it then all files will be owned by the current user.

7. Set specific permissions on files to transfer

To set specific permissions use the option --chmod for example if we want the following settings:

– Do not preserve the sticky bit for the owner and group
– Group has read and execute perms on DIR
– Group has read perm on the FICH
– World does not have any permissions on DIR and FICH

We execute

$ rsync -avz -chmod=ug-s,Dg=rx,Fg=r,o-rwx helios:src/bar /data/tmp

8. Mapping the owner and group to a specific user and group

$ rsync -avz --chown=root:librebyte /data/tmp helios:src/bar

The above command sets as owner root and group librebyte to the files sent to the remote server.

If the user who receives the files does not have permissions to change the owner and group then rsync will issue an error message.

The user and group that receives must exist or rsync throws an error message.

For greater control over user and group mapping use the options --usermap and --groupmap respectively. Do not use previous options together with --chown.

9. Using relative paths

Means that the specified path is completely sent to the remote server instead of the last part of the file name for example:

$ rsync -av /foo/bar/baz.c helios:/tmp/

The above command creates the file baz.c under the DIR tmp in the helios server while

$ rsync -avR /foo/bar/baz.c helios:/tmp/

creates the file /foo/bar/baz.c under the DIR tmp. As you will see the above command includes the option -R which specifies the use of relative paths, this option has its equivalent in the long form: --relative.

If you want to send only a part of the path then insert a ./, for example the following command will create the file /tmp/bar/baz.c note that DIR foo will be not created on the target.

$ rsync - avR /foo/./bar/baz.c helios:/tmp/

10. Backup

You can make backups by using the --backup option.

$ rsync -avz --delete --backup conf helios:conf

The above command only keeps 2 versions of files (the original and a copy) if you want to keep multiple copies then you must use the options --suffix and --backup-dir, for example the following command keeps several backup on “backup” DIR, the –suffix option adds to the name of the file the date in which the backup is made.

$ rsync - avz --delete --backup  --backup-dir =../backup --suffix=_ $(date +%Y-%m-%d.%H.%M) conf/ helios:conf/

The “backup” DIR will be created in the parent of conf DIR (it is possible to specify an absolute path).

11. Partial copies

If the connection is interrupted Rsync (by default) will eliminate any partial transfer, to change this behavior, use the options --partial or --partial-dir, it is recommended to use --partial-dir instead of --partial. Rsync will begin from where it was at the time that the connection was interrupted. Rsync will create/delete automatically the DIR specified by --partial-dir.

$ rsync -avz --partial-dir=../tmp-partial documents/ helios:mydoc

The tmp-partial DIR is created under mydoc DIR of the helios server (it is possible to specify an absolute path),--partial-dir has different behavior than --backup-dir option regarding the place where the specified DIR will be created if relative paths are established in both.

12. Transferring ACLs and extended attributes

-A and -X allow to synchronize extended attributes and ACLs (access control list) respectively. Keep in mind that the destination file system must support both options and be compatible with the source file system.

$ rsync -aAXvz ~/documents/ helios:~/documents/

13. Deleting extraneous files

The option --delete allows you to delete the files found in the destination but are not in the origin, this option is useful for mirroring. For example if we would like to mirror our websites:

$ rsync -avz --delete /var/www helios:/var/www

14. Filters

Rsync can include, exclude files to sync/transfer using the options:

*--cvs-exclude: is a shortcut to exclude files that usually are not synchronized or transferred (*.old *.bak *.BAK *.orig *.re j.of-* *.olb *.obj *.so *.exe *.o *.a *.Z *.elc *.ln core .svn / .git/ .hg/ .bzr/…).

*--exclude = PATTERN: exclude files that match with PATTERN.

*--exclude-from = FICH: read exclude patterns from FILE. Each pattern must be on a separate line, the lines that start with # or ; are considered comments.

*--include = PATTERN: don’t exclude files matching PATTERN.

*--include-from = FICH: read include patterns from FILE. Each pattern must be on a separate line, the lines that start with # or; they are considered to be comments.

*--files-from = FICH: allows you to specify an exact list of files to transfer, useful if you only want to transfer at the same time some FICH from different DIR.

*--filter= RULE : this is the most advanced, complex and flexible option. The above options are simplifications of --filter.

Some examples:

Exclude common files

$ rsync - avz --cvs-exclude documents/ helios:doc

It is possible to exclude specific files by creating a .cvsignore file under the DIR that you want to transfer also if you create a .cvsignore file under $HOME both rules will be mixed, .cvsignore patterns must be separated by a whitespace, and not by a line break. We have the following structure.

documents/Notes /
├── .git
│ ├── branches
│ ├── hooks
│ ├── info
│ ├── logs
│ ├── objects
│ └── refs
├── .gitignore
├── Internet
├── MultiMedia
├── Security
│ └── adminstradores_de_contrasenas
├── SO_Tipos_UNIX
│ └── Debian_-_Ubuntu
├── VIM
└── .zim

And we want to exclude the .gitignore and.zim DIR then we create .cvsignore file under Notes DIR with the following patterns:

.cvsignore .gitignore .zim

If we want to exclude all the hidden files it is enough to specify the following pattern

.*

Using –exclude and –exclude-from

We can obtain the same result before using the options –exclude and/or –exclude-from

$ rsync -avz --exclude='.zim'; --exclude='.git' --exclude-from='documents/exclude-list.txt'  documents helios:doc

The file exclude-list.txt:

*.exe
*.old
*.bak
*.BAK
*.orig
exclude-list.txt

15. Using non standard port for ssh service (Added)

If your network administrator has changed the port in which the ssh is “listening on” then you should use the -e option

$ rsync -avz -e 'ssh -p 2222' /var/www helios:/var/www

In the above command rsync instructs to the ssh shell that uses port 2222 for connection and data transfer.

16. Using rsync-path option

This option is useful if rsync is not found in the $PATH of receiving side or if you need to run some command before rsync, for example sudo.

$ rsync --rsync-path='sudo rsync' -az  --no-o --no-g --no-p --delete /var/www sedlav@helios:/var/www

A typical case of applying the above command is when:

– ssh root authentication is disabled.

– user www can authenticate via ssh and has access to dir /var/www/ via sudo (for more information see: Manual Sudoers)