`rsync` - experienced advice, please

As Oliver Hardy would say to Stanley Laurel, "Well, that's another fine mess you've gotten us into!"

In chasing down the 'routined quit unexpectedly' error message, I may have made a discovery. An unpleasant one at that.

Quick background: I have a Mac that has three HDD attached (necessary because of connectors) and these are shared to my Ubuntu 22.04.5 LTS server via SMB. Or I should say "WERE" connected. After running my diagnostic and repair attempts, they no longer attach (I have a cron @reboot entry that automounts them). Nor can I manually attach them. I get either a permissions error, or some other indecipherable message.

I decided to open a terminal on the Mac and run fsck against a 5TB volume. Oops, errors reported. fsck can't repair them but reports the volume as clean. Mac's Disk Utility program is useless, as it barfs when it encounters an error.

This problem may be related to the routined crashing, so the problem now is how to copy off 4.5TB of data. Worse, my MATE server has only about 1.5TB free. I first thought rsync might be the answer, but that won't free up any space. More than likely, I'll have to buy a fresh drive and copy from one volume to another. I've not used rsycnc before, but since it has a resume feature, that could be handy, as transferring that much data between drives could take hours, and one >burp< kills the whole thing.

Thoughts?

You might want to look at these:




For usage of rsync, I can only offer up references posted regarding my own usage and coding for rsync-based backups. In my case, I have logic to perform that as independent jobs on each of the partitions on my 2 TB drive (each partition about 300 GB).

Note that the posting has a full English translation at the bottom of the posting, after the code listings. :slight_smile:

LOG of backup session:

root@OasisMega1:/Oasis/bin# quickRSYNC.sh

	 NOTE:   This script will ONLY perform synchronization backup (applying differences).
		 It does NOT perform a FULL COPY backup.

		 For FULL COPY backup, use 'OS_Admin__partitionSecondary_Mirror.sh' responding 'y' at the appropriate prompt ...


	 Forcing backup of all files.       Option '--keepNewer' is available to keep backup version of files if newer ...

	 Checksum comparisons is disabled.  Option '--compareFileData' is available to force this check by rsync ...

	 Following block devices have been identified:

		 sda      8:0    0   1.8T  0 disk 
		 sdb      8:16   0 465.8G  0 disk 
		 sdc      8:32   0   3.6T  0 disk 
		 sr0     11:0    1  1024M  0 rom  

	 External BACKUP disk is online ...

 ==============================================================================================
 ==================================  PARTITION BACKUP START  ==================================
 ==============================================================================================

	 SOURCE=               /DB001_F1
	 MIRROR= /site/DB005_F8/DB001_F1 ...

	 Proceed with backup of 'DB001_F1' ? [y|N] => 
		 'DB001_F1' skipped ...

 ==============================================================================================
 ==================================  PARTITION BACKUP START  ==================================
 ==============================================================================================

	 SOURCE=               /DB001_F2
	 MIRROR= /site/DB005_F2/DB001_F2 ...

	 Proceed with backup of 'DB001_F2' ? [y|N] => 
		 'DB001_F2' skipped ...

 ==============================================================================================
 ==================================  PARTITION BACKUP START  ==================================
 ==============================================================================================

	 SOURCE=               /DB001_F3
	 MIRROR= /site/DB005_F3/DB001_F3 ...

	 Proceed with backup of 'DB001_F3' ? [y|N] => 
		 'DB001_F3' skipped ...

 ==============================================================================================
 ==================================  PARTITION BACKUP START  ==================================
 ==============================================================================================

	 SOURCE=               /DB001_F4
	 MIRROR= /site/DB005_F4/DB001_F4 ...

	 Proceed with backup of 'DB001_F4' ? [y|N] => y



 Using previously determined bandwidth limit for rsync buffer setting ...

 Will apply parameter to limit flooding of I/O, memory and swap ==>>  --bwlimit=65536

	 Mon 20 Jan 2025 06:54:07 PM EST |rsync| Start DB001_F4 ...
	 Background 'rsync' working ...


 Expected Log files:
	 /site/DB005_F4/Z_backup.DB001_F4.DateSize.out
	 /site/DB005_F4/Z_backup.DB001_F4.DateSize.err


 Use 'OS_Admin__partitionMirror_Monitor.sh' to monitor rsync process.


 Imported LIBRARY:  INCLUDES__TerminalEscape_SGR.bh ...


	 Mon 20 Jan 2025 06:54:17 PM EST

	 PID 19287 is RSYNC child process ...
	 PID 19286 is RSYNC child process ...
	 PID 19285 is RSYNC MASTER process ...
	 RSYNC backup process under way ...

	 root       19285   19281  0 18:54 pts/0    00:00:00 rsync 
				--bwlimit=65536 
				--one-file-system 
				--recursive 
				--outbuf=Line 
				--links 
				--perms 
				--times 
				--group 
				--owner 
				--devices 
				--specials 
				--verbose 
				--out-format=%t|%i|%M|%b|%f| 
				--delete-during 
				--whole-file 
				--human-readable 
				--protect-args 
				--ignore-errors 
				--msgs2stderr ./ /site/DB005_F4/DB001_F4/


	 Scanning at 10 second intervals ...
	 ...............

	 RSYNC process (# 19285) has completed.

	 Mon 20 Jan 2025 06:56:50 PM EST

root@OasisMega1:/Oasis/bin# 

Images of tail end of log with code-driven colouring:


5 Likes

To incrementally sync my backup drive with my system's drive at file level I use the following command:

rsync -a --delete /home/ugnvs/ /mnt/backup/ugnvs

4 Likes

The allure of using the short-form is seductive, but I prefer to explicitly enumerate all the flags used so that I don't have to remember, or guess at, what the features of "-a" would be. :slight_smile:

Having just gone back to review the "-a" option, I've decided to "upgrade" my script command to include

  • --atimes
  • --crtimes

I guess I missed those during my initial script development because I thought they were implicitly included already. :frowning:

Personally don't know of the value of --acls for Desktop. I can see that being used more in a Corporate environment.

man rsync excerpt:

--archive, -a archive mode is -rlptgoD (no -A,-X,-U,-N,-H)

aka
recursive, symlinks, permissions, modification times, group, owner, device files

without ACLs, xattrs, access/create times, hard links

4 Likes

Thank you Eugene. I am aware of that being outlined, as you've described, in the man pages. For some reason, my brain thought those two were already included. As I said earlier, reviewing the man page again today, I realized they weren't.

3 Likes

If you already rsync'ed that data onto the server once before, and your files are not monster files/directories, then you should be able to safely perform your rsync, as long as your largest directory does not contain more than 1.3 TB to be on the safe side. If not, things will get tricky. Just make sure you are using the "--delete-during" flag, and NOT the "--delete-after" flag.

maybe you can clean up the 4.5TB data.

If cost is not an issue, a Western Digital 4TB drive is $99 on amazon.

2 Likes

Thank you. I am familiar with SIP and disabling it (which I need to do again after resetting PRAM). I am also familiar with launchd and launchctl but my Catalina OS is older than the Ventura mentioned in your links. But I'm going to give it a try. I'm still trying to figure out how I'm going to offload the data on the HDD. It's the old "trying to stuff 10 pounds of manure into a 5 pound bag." :face_with_raised_eyebrow:

Is triage a consideration or feasible ?

  • How much is temporary files vs good stuff ?
  • How much is multiple iterations when only need to keep last (i.e. system log files) ?
  • How much of that is data is unique, user-created vs publicly-available imported ?
  • How much is duplicated directories at different locations on disk (i.e. a copy of a copy of a copy ...) ?
  • How much of that is neglected "fluff" ?

All good questions. This goes back to my question regarding fdupes (and rdfind and jdupes, which all have exhibited problems sniffing out duplicate files.

2 Likes