Skip to content

Image or drive

If you want to move a sparse file cp, tar, rsync and cpio has options to sparse files.

Is it even called "sparse"?

cp --sparse=always source_file new_file
rsync --sparse source_file new_file
cpio --sparse
tar --sparse

Clone a drive

This will try to save a little space by sparsing the image but if there is residue (deleted blocks which are not deleted) in the drive it will not work.

dd sparses by block size, all sectors must be zero to be parsed.

Create the image

dd if=original.img conv=sparse | pigz -6 > backup.img.gz

Restore the image

gunzip -c backup.img.gz original.img

TRIM drive

Normal drive using fstrim

fstrim is applied to mounted filesystems.

sudo fstrim -v /mnt

Then clone the image.

Normal drive using blkdiscard

blkdiscard is applied to devices. It may wipe some things like LUKS.

sudo blkdiscard -v /dev/sda1

Then clone the image.

USB drive

USB drives (or SD cards) do not support TRIM, so if you want to save space just write a file with zeros (0) to clean the "empty" space.

You can also clone the image, mount it in your computer, write zeros, unmount and then create another image. Saves writes to SD cards.

Mount the drive. Write a file with all zeros.

dd if=/dev/zero of=zeros ; rm zeros

Then clone the image.

LUKS drive

Open the drive.

sudo cryptsetup open discard.img encrypteddrive --allow-discards

Mount.

sudo mount /dev/mapper/encrypteddrive /mnt

Trim. blkdiscard destroys LUKS.

sudo fstrim -v /mnt

Unmount.

sudo umount /mnt

Then clone the image.

tar

Source

Make the backup.

tar --acls --xattrs -cpf file.tar fileOrFolder
tar --acls --xattrs -I pbzip2 -cpf file.tar fileOrFolder
tar --acls --xattrs -I pigz -cpf file.tar fileOrFolder

The options are used to keep:

  • p - Permissions
  • acls - ACLs
  • xattrs - Extended atributes

Restore.

tar --acls --xattrs -xpf file.tar

List the contents.

tar tf file.tar

Recommended way for a directory.

cd folder
find . -type f -not -name files.sha256 -exec sha256sum "{}" >> files.sha256 \;
sha256sum --quiet -c files.sha256
cd ..
tar --acls --xattrs -cpf folder.tar folder
tar tf folder.tar | wc -l
find folder | wc -l
sha256sum folder.tar > folder.tar.sha256
sha256sum --quiet -c folder.tar.sha256

rsync

One way

Copy the files from source/ that have been modified.

rsync --archive -- source/ destination/
rsync -a -- source/ destination/
rsync -a -- source/ destination/

Options:

  • archive or a Archive mode. Equals to -rlptgoD (no -H,-A,-X).
  • hard-links or H Preserve hard links.
  • acls or A Preserve ACLs (implies --perms).
  • xattrs or X Preserve extended attributes.
  • perms or p Preserve permissions.
  • links or l copy symlinks as symlinks.
  • copy-links or L transform symlink into referent file/dir.
  • copy-unsafe-links Only "unsafe" symlinks are transformed.

Considerations:

  • If a file is added to destination/, it will not be available in source/
  • If a file is modified in destination/, it will be overwritten on sync. Use --update or --backup
  • If a file is deleted in destination/, it will "reappear" on sync.
  • If a file is removed in source/, it will stay on destination/. Use --delete
  • Hardlinks are no longer hardlinks. Use -H
  • Softlinks still are softlinks. Use -L

Update

Skip files that are newer on the receiver (destination/).

rsync --archive --update -- source/ destination/
rsync -au -- source/ destination/

Merge directories

Source

rsync -abviuzP -- source/ destination/

Options:

  • backup or b Preexisting destination files are renamed (suffix ~) as each file is transferred or deleted. You can control this suffix with --suffix .suf. The destination/ file will be renamed unconditionally, not the oldest.
  • itemize-changes or i turns on the itemized format, which shows more information than the default format.
  • update or u makes rsync transfer skip files which are newer in destination/ than in source/
  • compress or z turns on compression, which is useful when transferring easily-compressible files over slow links.
  • P turns on --partial and --progress
  • --partial makes rsync keep partially transferred files if the transfer is interrupted.
  • --progress shows a progress bar for each transfer, useful if you transfer big files.

SSH

rsync user@host.domain.local:/path/to/source/ destination/
rsync -azP user@host.domain.local:/path/to/source/ destination/

Incremental Backup

rsync -aHAXp --link-dest="/path/to/backup/previous" source/ "server:/path/to/backup/current/"
  • link-dest hardlink to files in DIR when unchanged

Find if it works. du show the files only once per link.

find . -type f -links +1

sha256sum

Verification sum for a file

sha256sum file.txt > file.txt.sha256

Check the verification sum for one or multiple files

sha256sum -c file.txt.sha256

Make one verification file for every file in a directory and subdirectories.

find . -type f -not -name files.sha256 -exec sha256sum "{}" >> files.sha256 \;

Split file

We have the file original.ext.

The first thing is get the checksum.

sha256sum original.ext > original.ext.sha256

Compress it if wanted.

pigz -9 -c original.ext > original.ext.gz
sha256sum original.ext.gz > original.ext.gz.sha256

Split it.

split -b 10M original.ext.gz "original.ext.gz_"

Get the checksum. Use .xx. because it will fail if you use something like gz_xx.sha256 because of the cat wildcard.

sha256sum original.ext.gz_a* > original.ext.gz.xx.sha256

Move them to where you want. Check the files.

sha256sum -c original.ext.gz.xx.sha256

Merge the split files into one.

cat original.ext.gz_* > original.ext.gz

Check the file.

sha256sum -c original.ext.gz.sha256

Extract the file.

gunzip original.ext.gz
sha256sum -c original.ext.sha256

Knowledge

Difference between discard and trim

Source: https://en.opensuse.org/SDB:SSD_discard_%28trim%29_support

Discard is the linux term for telling a storage device that sectors are no longer storing valid data and applies equally to both ATA and SCSI devices. ie. For ext4 filesystems, there is a discard mount option, not a trim or unmap option.. Historically this feature has not existed, but recently SSD manufacturers have requested this ability to increase the performance capability of their designs and SCSI array manufacturers have requested similar functionality to better support thin provisioning.

Thus it would be typical to discard small sector ranges with a SSD, but only large ranges with a SCSI array. ie. Some SCSI arrays may clip ranges to 4 MB boundaries. The implementation of array specific as to how ranges are clipped.

TRIM is the actual ATA-8 command that is sent to a SSD to cause a sector range or set of sector ranges to be discarded. As such it should only apply to ATA devices, but is often used generically. Given the prevalence of ATA devices, trim is often the most used of these terms.

Sources

TODO

  • LVM
  • Bitlocker