I've been running into problems transferring large amounts of data successfully inasmuch that I first had problems with my drives crashing. I mean to a standstill and freezing the system. I did some work and was convinced that it was down to one particular nvme and the enclosure it was in. I swapped out the drive to a different make of enclosure and copying was fine. I wasn't convinced that the system should have froze the way that it did so i did a fresh install of 24.10. Things appeared to be going well till today. This time I was using gparted to copy one drive to another of the same make, model number, etc. Basically identical drives. Both are ssd from 2017 that have not been used extensively. Both are Adata 250GB and have given no trouble till today. For some reason one of the drives could not be read. It simply would not work with the system and I had to use gparted to sort out the formatting and partitioning. At least that's what I hoped. About one third of the way into copying about 130GB or so the process seemed to stall. No progress was made for over two hours so I had to kill the program. I am putting this latest incident down to the drive going faulty and it remains in the dust pile by itself for the moment. So here's my dilemma. Do I persist in using Ubuntu Mate 24.10 with what is possibly a bug in the system when it comes to copying large amounts of data or do I move away from it altogether. I am veering toward the latter. The main reason being that I don't trust Ubuntu Mate 24.10. I am used to copying 40 to 50GB at a time but now that I want to copy whole drives of data of 100GB and more I am having hiccups. I can copy over a few folders with around the 40 to 50GB mark but I am doubting that any more won't give problems and will just waste my time. I have also tried dd to copy whole drives but, after one stupid mistake in copying the wrong way round and actually deleting the drive with data I don't trust myself not to do this again. So, I may be stepping away from Linux after so many years...
Sorry to hear about the troubles with the large transfer between SSDs.
8 years old is pretty old for an SSD. Unpowered or powered, those cells definitely degrade over time. Because of the intensive 130 GB / 250 GB copy, one of the drives could have overheated or ran into bad blocks. S.M.A.R.T. data should have more details on their health.
The kernel logs should have answers at the time. It can be checked with a terminal:
sudo dmesg
The drive likely encountered an I/O error, usually indicating a hardware fault or bad cable. The kernel usually knows what happened.
I can't see how it's Ubuntu's fault, but if you did want to rule it out, there are tools like GParted Live (based on Debian). There might be non-Linux tools that can do full disk clones, but I'd bet they'd equally freeze/crash if they're nearing the end of their life. SSDs do have finite cycles before they start acting up like this.
dd and copying via GParted is really inefficient and more suited for hard disks. Because both methods are doing bit-by-bit copies, it will wear out the drive faster as it has to read and write everything - even if it's zero. That's just as bad as defragmenting SSDs.
If it was me, I'd use a method to "NAND clear the memory cells" (trim/sanitise) to essentially factory reset the drive to a fresh state; create the partitions (and attributes) identical to the original, and then either copy files or sudo rsync -av the lot from A to B if it needs to preserve file attributes/permissions.
However, some of these SSDs don't sound in good health, so I would recommend backing up the files onto a reliable spinning metal plate (HDD).
Thanks very much for your insight Luke. I am aware of the chips fragility. I used to be electronics supervisor in a university department and worked a lot with computers. I never quite got to grips with programming but that's by the by. The SSDs have never been heavily used and I've been doing the copying over and between various drives including mechanical. To be honest I can't think if any problems were purely SSD. The attempted gparted copy was SSD to SSD. I must try a couple of HDDs when I get a moment and check my notes from previous. You've given me food for thought. Thanks.
Just for a case, I'd like to mention that entire drives are better copied unmounted. Say, backing up system drive takes loading computer from live CD/USB.
P.S. Surely, file-level copying of data drives is done with drives mounted.
If copying goes well except for large files, it might be a good idea to check your RAM.
We need more informations on your setup.
What are the applications used to transfert these files ?
How many Go per files and whole?
Can you give us the output of:
lsblk -o name,fstype,size,fsused,fsuse%,fsavail | grep -Ev "loop"
On my setup, if i download very large file with bittorrent to ntfs partition, then it freeze, no way to finish. But, works fine if ext4 partition used. Take a look if a similar issue on your side.
lsblk -o name,fstype,size,fsused,fsuse%,fsavail | grep -Ev "loop"
NAME FSTYPE SIZE FSUSED FSUSE% FSAVAIL
nvme0n1 931.5G
├─nvme0n1p1 vfat 1G 6.1M 1% 1G
└─nvme0n1p2 ext4 930.5G 52G 6% 816.2G
nvme1n1 931.5G
└─nvme1n1p1 exfat 931.5G 268.8G 29% 662.7G
nvme2n1 476.9G
└─nvme2n1p1 exfat 476.9G 268.8G 56% 208.1G
Philippe for info:
I did some testing several days ago using the drives as I have posted using lsblk...
The SSDs used exfat for the main storage because I want to be able to copy files across various drives and they will use the same file system as much as possible. That will also include being able to be read by Microsoft Windows at some point in the future but I refuse to use NTFS if I don't have to. At least exfat is compatible with most systems and drives whether they are HDD, SSD or flash drives.
I had various folders containing various amounts and types of files. The smallest folder copied was 23.4GB and took 36.3 seconds using USB4 to read and write the drives. The largest folder copied 102.1GB which took 4m44.17s. I copied five folders with a total of 278.6GB which took 10m4s when copied individually. When the folders were copied enmasse the time taken was 16m8s. In my experiment I copied the largest folder with a fan blowing onto the aluminium enclosure and without. There was a small difference of approximately eight seconds reduction when using the fan. I have no HDDs capable of benefitting from using USB4 and I have not tried them out yet using USB 3.2 Gen 2 ports - this will be my next little project.
So, to summarize: it would seem that copying one folder at a time when storing on SSD is the preffered method if I am tight on time and don't need to copy a whole drives' contents. When thinking about the use of SSDs for data storage I am sure that heat dissipation is paramount to good performance and I will look more closely at the build construction of my next pc, laptop and enclosures for the SSDs. There is no doubt still a place for HDDs as far as reliability is concerned but having said that, they are moving parts as opposed to not moving and logically my brain can't get around to thinking that something that does not physically move other than at the molecular level should surely be more reliable and last longer physically than something that moves/wears out. For the comparatively low amount of data that I'm moving about my drives should not 'age' to the point that I should be worried about their contents after five years or so.
Nothing wrong with exFAT, but just wanted to emphasise it isn't a journaled file system. Data corruption is more likely to happen if the drive isn't "ejected" or "unmounted" properly. That's probably more a worry if the cable goes bad, got accidentally unplugged or the system crashes.
Journaled filesystems (like ext4, NTFS) keep track of changes before they are committed, so a little less prone to corruption, exFAT more like FAT32 in that regard.
Also, for your benchmark, time taken to copy could be skewed if the source files were already "cached" in RAM memory from the previous run. To tell the kernel to clear caches, run in a terminal:
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
If you are doing backups using rsync, make sure that you do not have the --max-size option specified.
Beyond that, you might want to consider the --bwlimit option:
--bwlimit=RATE limit socket I/O bandwidth
Also, although I haven't used this one myself, yet, there is also the --max-alloc option which could offer some assistance in minimizing OS sluggishness while rsync is running, especially if the file source is your SSD, you could reduce that "--max-alloc":
--max-alloc=SIZE
By default rsync limits an individual malloc/realloc to about 1GB in size. For most people this limit
works just fine and prevents a protocol error causing rsync to request massive amounts of memory. How‐
ever, if you have many millions of files in a transfer, a large amount of server memory, and you don't
want to split up your transfer into multiple parts, you can increase the per-allocation limit to some‐
thing larger and rsync will consume more memory.
Keep in mind that this is not a limit on the total size of allocated memory. It is a sanity-check
value for each individual allocation.
See the --max-size option for a description of how SIZE can be specified. The default suffix if none
is given is bytes.
Beginning in 3.2.3, a value of 0 specifies no limit.
You can set a default value using the environment variable RSYNC_MAX_ALLOC using the same SIZE values
as supported by this option. If the remote rsync doesn't understand the --max-alloc option, you can
override an environmental value by specifying --max-alloc=1g, which will make rsync avoid sending the
option to the remote side (because "1G" is the default).
Another option, again for rsync, --whole-file might also give some OS "focus" to ensure integrity of single file transfers.
If none of those help, then you are looking at tweaking kernel parameters, for which I am not an expert.
[edit]
Another thing I wanted to point out. "Disk imaging" is by definition
- a high-intensity activity,
- with extended duration, and
- little opportunity for recovery from failure point.
For those reasons, I highly recommend minimizing disk-to-disk imaging, and resort to processes like rsync, which offer much flexibility in terms of limiting OS/hardware/bandwidth abuse, with the attractive benefit of recovering from point of failure.
You can even program a max time, using
--stop-after=MINS or--stop-at=y-m-dTh:m
for "imaging" and perform a sleep at intervals to allow hardware to cool down.
Thanks. That's very informative. I've never used rsync. I tried once but Terminal did not recognize the command or something. I can't remember now.
Can you give an example of using rsync to copy several folders or even a complete drive contents to en external drive, etc. I will google the command too but, I've found from experience that a real world example of the command can be much more precise and easier to work out than trial and error with manual examples.
Thanks again.
On the use of rsync, I will point you to some earlier postings:
- This one points to where you can download my backup scripts. Essentially, if the target partition is empty, you are doing a full disk mirroring of data only. The target partition could be almost any filesystem type.
- Note: ALL of my public scripts are posted here:
- This one gives you a "visual" of what the terminal session looks like (other good stuff in that discussion).
- (note English translation of text at bottom of posting) This one lets you take a look at the script source-code before downloading, to give you a sense of my coding style. Again lots of good stuff in that discussion.
Hi, I copy 600+ GB regularly. I get 500+ MB/s speeds, ok for my system, but that can sometimes be slower and I found that using fstrim -v -a before large data copies seems to help.