Wednesday, May 4, 2016

Locally backup your data with rsync

Locally backup your data with rsync
  After 1 year later HDD data transfer speed was 70~100 kbps . How should you do with useless HDD ?  I try with following command:

rsync -r -t -v -progress --ignore-existing -s [ source] [ destination ]

rsync                       a fast, versatile, remote (and local) file-copying tool

-r, --recursive
       This tells rsync to copy directories recursively.  See also --dirs (-d).
Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full file list, so these options disable the incremental
recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and
--delay-updates. Because of this, the default delete mode when you specify --delete is now
--delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during
to request this improved deletion mode explicitly). See also the --delete-delay option that is a
better choice than using –delete-after. Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r

-t, --times
This tells rsync to transfer modification times along with the files and update them on the remote
system. Note that if this option is not used, the optimization that excludes files that have not
been modified cannot be effective; in other words, a missing -t or -a will cause the next transfer
to behave as if it used -I, causing all files to be updated (though rsync’s delta-transfer
algorithm will make the update fairly efficient if the files haven’t actually changed, you’re much
better off using -t).

-v, --verbose
This option increases the amount of information you are given during the transfer. By default,
rsync works silently. A single -v will give you information about what files are being transferred
and a brief summary at the end. Two -v options will give you information on what files are being
skipped and slightly more information at the end. More than two -v options should only be used if
you are debugging rsync.

Note that the names of the transferred files that are output are done using a default --out-format
of "%n%L", which tells you just the name of the file and, if the item is a link, where it points.
At the single -v level of verbosity, this does not mention when a file gets its attributes
changed. If you ask for an itemized list of changed attributes (either --itemize-changes or
adding "%i" to the --out-format setting), the output (on the client) increases to mention all
items that are changed in any way. See the --out-format option for more details.

-p, --perms
This option causes the receiving rsync to set the destination permissions to be the same as the
source permissions. (See also the --chmod option for a way to modify what rsync considers to be
the source permissions.)

When this option is off, permissions are set as follows:

o Existing files (including updated files) retain their existing permissions, though the
--executability option might change just the execute permission for the file.

o New files get their "normal" permission bits set to the source file’s permissions masked
with the receiving directory’s default permissions (either the receiving process’s umask,
or the permissions specified via the destination directory’s default ACL), and their
special permission bits disabled except in the case where a new directory inherits a setgid
bit from its parent directory.

Thus, when --perms and --executability are both disabled, rsync’s behavior is the same as that of
other file-copy utilities, such as cp(1) and tar(1).

In summary: to give destination files (both old and new) the source permissions, use --perms. To
give new files the destination-default permissions (while leaving existing files unchanged), make
sure that the --perms option is off and use --chmod=ugo=rwX (which ensures that all non-masked
bits get enabled). If you’d care to make this latter behavior easier to type, you could define a
popt alias for it, such as putting this line in the file ~/.popt (the following defines the -Z
option, and includes --no-g to use the default group of the destination dir):

rsync alias -Z --no-p --no-g --chmod=ugo=rwX

You could then use this new option in a command such as this one:

rsync -avZ src/ dest/

(Caveat: make sure that -a does not follow -Z, or it will re-enable the two "--no-*" options
mentioned above.)

The preservation of the destination’s setgid bit on newly-created directories when --perms is off
was added in rsync 2.6.7. Older rsync versions erroneously preserved the three special permission
bits for newly-created files when --perms was off, while overriding the destination’s setgid bit
setting on a newly-created directory. Default ACL observance was added to the ACL patch for rsync
2.6.7, so older (or non-ACL-enabled) rsyncs use the umask even if default ACLs are present. (Keep
in mind that it is the version of the receiving rsync that affects these behaviors.)
-o, --owner
This option causes rsync to set the owner of the destination file to be the same as the source
file, but only if the receiving rsync is being run as the super-user (see also the --super and
--fake-super options). Without this option, the owner of new and/or transferred files are set to
the invoking user on the receiving side.

The preservation of ownership will associate matching names by default, but may fall back to using
the ID number in some circumstances (see also the --numeric-ids option for a full discussion).

-g, --group
This option causes rsync to set the group of the destination file to be the same as the source
file. If the receiving program is not running as the super-user (or if --no-super was specified),
only groups that the invoking user on the receiving side is a member of will be preserved.
Without this option, the group is set to the default group of the invoking user on the receiving

The preservation of group information will associate matching names by default, but may fall back
to using the ID number in some circumstances (see also the --numeric-ids option for a full
-e, --rsh=COMMAND
This option allows you to choose an alternative remote shell program to use for communication
between the local and remote copies of rsync. Typically, rsync is configured to use ssh by
default, but you may prefer to use rsh on a local network.

If this option is used with [user@]host::module/path, then the remote shell COMMAND will be used
to run an rsync daemon on the remote host, and all data will be transmitted through that remote
shell connection, rather than through a direct socket connection to a running rsync daemon on the

Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a
single argument. You must use spaces (not tabs or other whitespace) to separate the command and
args from each other, and you can use single- and/or double-quotes to preserve spaces in an
argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string
gives you a single-quote; likewise for double-quotes (though you need to pay attention to which
quotes your shell is parsing and which quotes rsync is parsing). Some examples:

-e 'ssh -p 2234'
-e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'

(Note that ssh users can alternately customize site-specific connect options in their .ssh/config

You can also choose the remote shell program using the RSYNC_RSH environment variable, which
accepts the same range of values as -e.

See also the --blocking-io option which is affected by this option.
-s, --protect-args
This option sends all filenames and most options to the remote rsync without allowing the remote
shell to interpret them. This means that spaces are not split in names, and any non-wildcard
special characters are not translated (such as ~, $, ;, &, etc.). Wildcards are expanded on the
remote host by rsync (instead of the shell doing it).

If you use this option with --iconv, the args related to the remote side will also be translated
from the local to the remote character-set. The translation happens before wild-cards are
expanded. See also the --files-from option.
This tells rsync to skip updating files that already exist on the destination (this does not
ignore existing directories, or nothing would get done). See also --existing.

This option is a transfer rule, not an exclude, so it doesn’t affect the data that goes into the
file-lists, and thus it doesn’t affect deletions. It just limits the files that the receiver
requests to be transferred.

This option can be useful for those doing backups using the --link-dest option when they need to
continue a backup run that got interrupted. Since a --link-dest run is copied into a new
directory hierarchy (when it is used properly), using --ignore existing will ensure that the
already-handled files don’t get tweaked (which avoids a change in permissions on the hard-linked
files). This does mean that this option is only looking at the existing files in the destination
hierarchy itself.

Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:

Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]

rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST::DEST

rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST