HowTo Use rsync For Transferring Files Under Linux

How do you install and use rsync to synchronize files and directories from one location (or one server) to another location? - A common question asked by new sys admin.

rsync is a free software computer program for Unix and Linux like systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.

So what is unique about the rsync command?

It can perform differential uploads and downloads (synchronization) of files across the network, transferring only data that has changed. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection.

How do I install rsync?

Use any one of the following commands to install rsync. If you are using Debian or Ubuntu Linux, type the following command:
# apt-get install rsync
OR
$ sudo apt-get install rsync
If you are using Red Hat Enterprise Linux (RHEL) / CentOS 4.x or older version, type the following command:
# up2date rsync
RHEL / CentOS 5.x or newer (or Fedora Linux) user type the following command:
# yum install rsync

Always use rsync over ssh

Since rsync does not provide any security while transferring data it is recommended that you use rsync over ssh session. This allows a secure remote connection. Now let us see some examples of rsync command.

Comman rsync command options

• --delete : delete files that don't exist on sender (system)
• -v : Verbose (try -vv for more detailed information)
• -e "ssh options" : specify the ssh as remote shell
• -a : archive mode
• -r : recurse into directories
• -z : compress file data

Task : Copy file from a local computer to a remote server

Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in
$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~
Output:

Password:
sent 19099 bytes  received 36 bytes  1093.43 bytes/sec
total size is 19014  speedup is 0.99

Please note that symbol ~ indicate the users home directory (/home/jerry).

Task : Copy file from a remote server to a local computer

Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer's /tmp directory:
$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot

Task: Synchronize a local directory with a remote rsync server or vise-versa

$ rsync -r -a -v --delete rsync://rsync.nixcraft.in/cvs /home/cvs
OR
$ rsync -r -a -v --delete /home/cvs rsync://rsync.nixcraft.in/cvs

Task: Mirror a directory between my "old" and "new" web server/ftp

You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)
$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd

Read related previous articles

• How do I sync data between two Load balanced Linux/UNIX servers?
• How do I sync data between two Load balanced Windows 2003 servers?

Other options - rdiff and rdiff-backup

The rdiff command uses the rsync algorithm. A utility called rdiff-backup has been created which is capable of maintaining a backup mirror of a file or directory over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. Next time I will write about these utilities.

rsync for Windows Server/XP/7/8

Please note if you are using MS-Windows, try any one of the program:

1. DeltaCopy
2. NasBackup

Further readings

=> Read rsync man page
=> Official rsync documentation

from http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html
---------------------------------------------
How to Copy Files Across a Network/Internet in UNIX/LINUX (Redhat, Debian, FreeBSD, etc) - scp tar rsync

One of the many advantages of Linux/UNIX is how many ways you can do one thing. This tuturial is going to show you some of the many ways you can ttransfer files over a network connection.

In this article/tutorial we will cover rsync, scp, and tar. Please note that there are many other ways these are just some of the more common ones. The methods covered assume that SSH is used in all sessions. These methods are all much more secure and reliable than using rcp or ftp. This tutorial is a great alternative for those looking for an FTP alterative to transfering files over a network.

scp

scp or secure copy is probably the easiest of all the methods, its is designed as a replacement for rcp, which was a quick copy of cp with network funcationability.

scp syntax

scp [-Cr] /some/file [ more ... ] host.name:/destination/file

-or-

scp [-Cr] [[user@]host1:]file1 [ more ... ] [[user@]host2:]file2

Before scp does any copying it first connects via ssh. Unless proper keys are in place, then you will be asked for usernames. You can test if this is working by using ssh -v hostname

The -r switch is used when you want to recursively go through directories. Please note you must specify the source file as a directory for this to work.

scp encrypts data over your network connection, but by using the -C switch you can compress the data before it goes over the network. This can significantly decrease the time it takes to copy large files.

Tip: By default scp uses 3DES encryption algorithm, all encryption algorithms are slow, but some are faster than others. Using -c blowfish can speed things up.

What scp shouldn't be used for:
1. When you are copying more than a few files, as scp spawns a new process for each file and can be quite slow and resource intensive when copying a large number of files.
2. When using the -r switch, scp does not know about symbolic links and will blindly follow them, even if it has already made a copy of the file. The can lead to scp copying an infinite amount of data and can easily fill up your hard disk, so be careful.

rsync

rsync has very similar syntax to scp:

rsync -e ssh [-avz] /some/file [ more ... ] host.name:/destination/file

-or-

rsync -ave ssh source.server:/path/to/source /destination/dir

rsync's speciality lies in its ability to analyse files and only copy the changes made to files rather than all files. This can lead to enormous improvements when copying a directory tree a second time.

Switches:

-a Archive mode, most likely you should always keep this on. Preserves file permissions and does not follow symlinks.

-v Verbose, lists files being copied

-z Enable compression, this will compress each file as it gets sent over the pipe. This can greatly decrease time depending on what sort files you are copying.

-e ssh Uses ssh as the transport, this should always be specified.

Disadvantages of using rsync:
1. Picky syntax, use of trailing slashes can be confusing.
2. Have to remember that you are using ssh.
3. rsync is not installed on all computers.

tar

tar is usually used for achiving applications, but what we are going to do in this case is tar it then pipe it over an ssh connection. tar handles large file trees quite well and preserves all file permissions, etc, including those UNIX systems which use ACLs, and works quite well with symlinks.

the syntax is slightly different as we are piping it to ssh:

tar -cf - /some/file | ssh host.name tar -xf - -C /destination

-or with compression-

tar -czf - /some/file | ssh host.name tar -xzf - -C /destination

Switch -c for tar creates an archive and -f which tells tar to send the new archive to stdout.

The second tar command uses the -C switch which changes directory on the target host. It takes the input from stdin. The -x switch extracts the archive.

The second way of doing the transfer over a network is with the -z option, which compresses the stream, decreasing time it will take to transfer over the network.

Some people may ask why tar is used, this is great for large file trees, as it is just streaming the data from one host to another and not having to do intense operations with file trees.

If using the -v (verbose) switch, be sure only to include it on the second tar command, otherwise you will see double output.

Using tar and piping can also be a great way to transfer files locally to be sure that file permissions are kept correctly:

tar cf - /some/file | (cd /some/file; tar xf -)

This may seem like a long command, but it is great for making sure all file permissions are kept in tact. What it is doing is streaming the files in a sub-shell and then untarring them in the target directory. Please note that the -z command should not be used for local files and no perfomance increase will be visible as overhead processing (CPU) will be evident, and will slow down the copy.

Why tar shouldn't be used:
1. The syntax can be hard to remember
2. It's not as quick as to type scp for a small number of files
3. rsync will beat it hands down for a tree of files that already exist in the destination.

There are several other ways of copying over a network, such as FTP, NAS, and NFS but these all requre specialised software installed on either the receiving or sending end, and hence are not as useful as the above commands.

from  http://www.crucialp.com/resources/tutorials/server-administration/how-to-copy-files-across-a-network-internet-in-unix-linux-redhat-debian-freebsd-scp-tar-rsync-secure-network-copy.php
------------------------------------------

SCP: Howto, tips & tricks

The most simple case

In the most simple case, you can connect to a server that supports ssh with a syntax as short as this:

[pineehad@localhost ~]$ ssh yourserver

Note: If you do not have any ssh server nearby that you can access, you can also try this command with your own computer as a server. To do this, replace “yourserver” with “localhost”.
Of course, yourserver should be replaced by a hostname or an ip address of the server you want to connect to. As you can see in the terminal snippet, I am logged in as pineehad. If you do not specify a username (I’ll explain how to do that later in this tutorial), SSH will assume that you want to login with the username you’re currently logged in with. So, in this case, SSH will try the username pineehad.
Of course, you need to be sure that the server supports ssh connections. The ssh client tries to connect to port 22 defaultly. This means that, if you want to connect to a remote host with the default settings, you should make sure that, if applicable, port 22 is forwarded to the server you’re trying to connect to. You will find more regarding the SSH port further in this tutorial.
Now, back to the command we ran. If the server supports SSH connections and you can reach it by port 22, you should be prompted for a password (if this is the first time you try to connect to the server, ssh will first ask the question if you want to continue connecting, which can generally just be answered with a ‘yes’). If you type a password here, you won’t see asterisks appearing. Don’t panic, this is ssh’s normal behaviour. It makes connecting using ssh even more safe, because any accidental spectators won’t be able to see the length of the password. After entering the password, if the username and the password were correct, you should be running a shell on the server. If not, make sure you are connecting to a server of which you know that you should be able to login with your username and the specified password. You could try connecting to your own computer (see the note beneath the terminal quote) or read on to learn how to specify an other username.
Once you’re done trying the ssh shell, you can exit it by pressing Ctrl + D.

Specifying a username

It’s actually quite simple to specify a different username. You might even already be familiar with it. See the following example:

[pinehead@localhost ~]$ ssh yourusername@yourserver

The above will make ssh try to connect with the username “yourusername” instead of (in my case) pineehad. This syntax is also used by a lot of other protocols, so it’ll always come in handy to know it. By the way, you will still be asked for a password. For security reasons, it is not even possible to directly specify the password in the syntax. You will always be asked interactively, unless you start configuring the server in an advanced way (which is exactly why that topic is out of this tutorials scope: this tutorial documents how to use the clients, not how to configure the server).

Specifying a port

There are many reasons to move the ssh service to an other port. One of them is avoiding brute-force login attempts. Certain hackers try to get access to ssh servers by trying a lot of common usernames with common passwords (think of a user “john” with password “doe”). Although it is very unlikely that these hackers will ever get access to the system, there is an other aspect of the brute-force attacks that you’ll generally want to avoid: the system and connection load. The brute-force attacks usually are done with dozens or even thousands of tries a second, and this unnecessarily slows down the server and takes some bandwidth which could’ve been used a lot better. By changing the port to a non-default one, the scripts of the hackers will just be refused and most of the bandwidth will be saved.
As the ssh command can’t just guess the port, we will have to specify it if it’s not the default 22 one. You can do that this way:

[pineehad@localhost ~]$ ssh -p yourport yourusername@yourserver

Of course, you will have to replace “yourport” with the port number. These is an important difference between ssh and scp on this point. I’ll explain it further on.

Running a command on the remote server

Sometimes, especially in scripts, you’ll want to connect to the remote server, run a single command and then exit again. The ssh command has a nice feature for this. You can just specify the command after the options, username and hostname. Have a look at this:

[pineehad@localhost ~]$ ssh yourusername@yourserver updatedb

This will make the server update its searching database. Of course, this is a very simple command without arguments. What if you’d want to tell someone about the latest news you read on the web? You might think that the following will give him/her that message:

[pineehad@localhost ~]$ ssh yourusername@yourserver wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”

However, bash will give an error if you run this command:

bash: !”: event not found

What happened? Bash (the program behind your shell) tried to interpret the command you wanted to give ssh. This fails because there are exclamation marks in the command, which bash will interpret as special characters that should initiate a bash function. But we don’t want this, we just want bash to give the command to ssh! Well, there’s a very simple way to tell bash not to worry about the contents of the command but just pass it on to ssh already: wrapping it in single quotes. Have a look at this:

[pineehad@localhost ~]$ ssh yourusername@yourserver ‘wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”‘

The single quotes prevent bash from trying to interpret the command, so ssh receives it unmodified and can send it to the server as it should. Don’t forget that the single quotes should be around the whole command, not anywhere else.

SCP

The scp command allows you to copy files over ssh connections. This is pretty useful if you want to transport files between computers, for example to backup something. The scp command uses the ssh command and they are very much alike. However, there are some important differences.
The scp command can be used in three* ways: to copy from a (remote) server to your computer, to copy from your computer to a (remote) server, and to copy from a (remote) server to another (remote) server. In the third case, the data is transferred directly between the servers; your own computer will only tell the servers what to do. These options are very useful for a lot of things that require files to be transferred, so let’s have a look at the syntax of this command:

[pineehad@localhost ~]$ scp examplefile yourusername@yourserver:/home/yourusername/

Looks quite familiar, right? But there are differences. The command above will transfer the file “examplefile” to the directory “/home/yourusername/” at the server “yourserver”, trying to get ssh acces with the username “yourusername”. That’s quite a lot information, but scp really needs it all. Well, almost all of it. You could leave out the “yourusername@” in front of “yourserver”, but only if you want to login on the server with your current username on your own computer. Let’s have a closer look at the end of the command. There’s a colon over there, with a directory after it. Just like Linux’s normal cp command, scp will need to know both the source file(s) and the target directory (or file). For remote hosts, the file(s)/directory are given to the scp command is this way.
You can also copy a file (or multiple files) from the (remote) server to your own computer. Let’s have a look at an example of that:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile .

Note: The dot at the end means the current local directory. This is a handy trick that can be used about everywhere in Linux. Besides a single dot, you can also type a double dot ( .. ), which is the parent directory of the current directory.
This will copy the file “/home/yourusername/examplefile” to the current directory on your own computer, provided that the username and password are correct and that the file actually exists.
You probably already guessed that the following command copies a file from a (remote) server to another (remote) server:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile yourusername2@yourserver2:/home/yourusername2/

Please note that, to make the above command work, the servers must be able to reach each other, as the data will be transferred directly between them. If the servers somehow can’t reach each other (for example, if port 22 is not open on one of the sides) you won’t be able to copy anything. In that case, copy the files to your own computer first, then to the other host. Or make the servers able to reach each other (for example by opening the port).
Well, those are the main uses of scp. We’ll now go a bit more in-depth about the differences between ssh and scp.
*: Actually you can also use it just like the normal cp command, withhout any ssh connections in it, but that’s quite useless. It requires you to type an extra ‘s’ =).

Specifying a port with scp

The scp command acts a little different when it comes to ports. You’d expect that specifying a port should be done this way:

[pineehad@localhost ~]$ scp -p yourport yourusername@yourserver:/home/yourusername/examplefile .

However, that will not work. You will get an error message like this one:

cp: cannot stat `yourport’: No such file or directory

This is caused by the different architecture of scp. It aims to resemble cp, and cp also features the -p option. However, in cp terms it means ‘preserve’, and it causes the cp command to preserve things like ownership, permissions and creation dates. The scp command can also preserve things like that, and the -p option enables this feature. The port specification should be done with the -P option. Therefore, the following command will work:

[pineehad@localhost ~]$ scp -P yourport yourusername@yourserver:/home/yourusername/examplefile .

Also note that the -P option must be in front of the (remote) server. The ssh command will still work if you put -p yourport behind the host syntax, but scp won’t. Why? Because scp also supports copying between two servers and therefore needs to know which server the -P option applies to.

Another difference between scp and ssh

Unlike ssh, scp cannot be used to run a command on a (remote) server, as it already uses that feature of ssh to start the scp server on the host. The scp command does have an option that accepts a program (the -S option), but this program will then be used instead of ssh to establish the encrypted connection, and it will not be executed on the remote host.

Tips & Tricks with ssh and scp

Quite a handy thing about scp is that it supports asterisks. You can copy all files in a remote directory in a way like this:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/* .

And you can also just copy a whole directory by specifying the -r (recursive) option:

[pineehad@localhost ~]$ scp -r yourusername@yourserver:/home/yourusername/ .

Both of these also work when copying to a (remote) server or copying between a (remote) server and another (remote) server.
The ssh command can come in handy if you don’t know the exact location of the file you want to copy with scp. First, ssh to the (remote) server:

[pineehad@localhost ~]$ ssh yourusername@yourserver

Then browse to the right directory with cd. This is essential Linux terminal knowledge, so I won’t explain it here. When you’re in the right directory, you can get the full path with this command:

[pineehad@localhost ~]$ pwd

Note: pwd is an abbreviation of Print Working Directory, which is a useful way to remember the command.
You can then copy this output, leave the ssh shell by pressing Ctrl + D, and then paste the full directory path in your scp command. This saves a lot of remembering and typing!
You can also limit the bandwidth scp may use when copying. This is very useful if you’re wanting to copy a huge amount of data without suffering from slow internet for a long time. Limiting bandwidth is done this way:

scp -l bandwidthlimit yourusername@yourserver:/home/yourusername/* .

The bandwidth is specified in Kbit/sec. What does this mean? Eight bits is one byte. If you want to copy no faster than 10 Kbyte/sec, set the limit to 80. If you want to copy no faster than 80 Kbyte/sec, set the limit to 640. Get it? You should set the limit to eight times the maximum Kbyte/sec you want it to be. I’d recommend to set the -l option with all scp’ing you do on a connection that other people need to use, too. A big amount of copying can virtually block a whole 10 Mbit network if you’re using hubs.

from http://tuts.pinehead.tv/2012/09/03/ssh-and-scp-howto-tips-tricks/