Safe to remove USB device?

It’s not anything I’m doing, I do not remove the device for a good few seconds after the icon has disappeared once ejected. The simple fact is that the progress meter is not accurate and data is still being transfered once the meter has finished and vanished from the desktop and the device has been ejected and the icon has disappeared.

#Background
Safely remove mechanism was changed since Udisks were upgraded from 1.0 to 2.0.
Previously safe remove switched LED on flash off and removed device from the system.
For now after Ejecting the LED on flash remains on. But it depends on flash model.

We have reported bug 1067876 about having Eject instead of Safely remove.

#Software

Ancient

Previous versions of Ubuntu had nice looking application called Ejecter:

Description: application to unmount easily and safely external devices
Ejecter is a simple menu that sits in the system notification area, providing
you a quick way to unmount an external peripheral such as USB pendrive,
CD/DVD disk, external hard disk and so.
.
Ejecter will sleep behind the scenes and show an icon in the system tray when
one or more devices are connected to your computer.

It places its icon in notification area and allow to detach USB-device as in MS Windows. See screenshots (from here):

Ejecter active
Ejecter did safely remove

But is was removed from repositories since 14.04 LTS.
With some workaround we can get it working again on Ubuntu 16.04 LTS MATE and even on 18.04 LTS MATE (see this Q&A on AskUbuntu). But it does not spin-down USB HDDs.

##Current

On my systems I do the following: If I see only Eject option in the Caja window, then I launch GNOME Disks (gnome-disks or from Applications→Accessories→Disks), select appropriate device and click Power off the disk button:

After doing so I see that LED on flash is off or USB-HDD is spinned down.
But it is very likely that it is paranoid solution since unmounting with ejecting is enough to keep data safe :slight_smile:

The problem is I am doing everything right. I am waiting until the progress meter indicates that the job has apparently completed, I am ejecting the drive correctly, waiting a few seconds and removing the drive.

The problem is that even after the progress meter has indicated that data transfer has completed and even after I eject the drive correctly, data transfer is still in progress - Even though the icon has disappeared from the desktop after ejecting correctly. Therefore the issue is that on a drive that has no data transfer LED you have no idea when data transfer has actually completed and the drive is safe to remove.

It is for this very reason that a pop up indicating that the drive is safe to remove, or a more accurate data transfer indicator is really needed.

On a ~5GB transfer I can wait a full minute after ejecting and seeing the drives icon leave the desktop, pull the drive out and get an error that data transfer has not completed and have a garanteed corrupt file system. Guaranteed, every time.

1 Like
  1. Me too! I hate USB-flashes and disks without LEDs. I'm trying hard to buy devices with LED. But it is too difficult to find them nowadays.
  2. The second problem is that some flashes incorrectly report status of their write buffer.
  3. The third problem is that some external harddisk manufactures invented their own interface boards to make devices cheaper. Western Digital MyPassport is an illustration of this problem. Such devices have USB controller which is directly connected to the drive electronics (so it is not 2.5" SATA drive with USB-SATA bridge, it is 2.5" drive with direct USB connection and self-invented reduced buggy ATA-commandset).

I completely got Ejecter working on all LTS releases (see my answer on AskUbuntu):


You can try to it with your disks and report any success here :slight_smile:

All done my friend, I'm going to give this a shot. Thank you.

[EDIT] Just tried copying a large file and ejecting it, Ejecter works perfectly! Thanks Norbert_X!

1 Like

I got the same problem as @Bulletdust i transfert on a regular basis 2.7 GB on a usb thumb drive and discover many problems with “eject” or “remove safely” as well as the transfert bar being far to be accurate.

1 When formated as fat or dos the transfert bar take around 20 minutes to be full => but the thumb drive is still blinking (lucky to have a blinking light on the thumb drive) for at least 10 more minutes even when i right click then Eject and the icon disappear from the desktop… still blinking

2 When formated as ext4 (because fat or dos does not take in count .htaccess or any file starting with a period as well as lower/upper case it ask me to rename or overwrite thinking it’s the same file name) the progress bar goes a lot faster than in fat. it takes just 5 minutes but i need to wait 25 minutes before i can remove the usb thumb drive as it is still bliking during 25 minutes even after right click and eject.

For me it’s a Bug where the progress bar should be at least adjusted… And when we Eject, if the transfert is not finished we should have an alert saying that we cannot eject yet.

BTW i killed my phone micro SD card 2 years ago because of that (not knowing this, and with no possible way for me to know that the transfert was not finished yet)

1 Like

Hallo Bulletdust

You might find it useful to look at the “nmon” output in a small terminal window while you make such transfers.

To install:

sudo apt-get install nmon

To run:

nmon

Just an idea. :slight_smile:

2 Likes

Thanks alpinejohn, I’ve downloaded nmon and will definitely make use of it - Interesting software.

The problem is, people’s data is being lost, this issue really needs to be resolved at OS level. A proper ‘Your USB device can now be removed safely’ pop up is a must where it takes so long for the buffer to clear and the progress meter is so inaccurate.

So is it possible to integrate the Ejecter code into the OS in such a way as to make it work like that? (Resists urge to begin coding in language does not know…) Because I had a “cool” idea once to integrate Wine or some such emulator into the OS to improve performance, not that I would know how to do that.

OK, so there may be an issue with Ejecter on large file transfers to USB stick.

Yesterday I transferred a large movie file to USB stick. I waited for the file to transfer (the process took around 12 mins), once the file had completely transferred I waited about 15 seconds and ejected the USB stick using Ejecter and was informed the device was safe to remove. I removed the drive and went to watch the movie on my laptop. Half way through watching the movie playback failed, no matter what I did I could not get playback to resume even on my desktop PC using the same USB stick.

The original movie file plays back just fine off the hard drive of my desktop PC, the USB stick that was ejected correctly stops half way through, obviously the file is corrupted somehow.

Is there any way the devs can do something about the massive inaccuracies regarding the progress indicator in relation to file transfers to USB devices? It’s wildly inaccurate making it impossible to safely eject media.

Even better still, is there a way we can completely resolve the ejecting issue without using Ejecter? It’s a fair oversight that media is being corrupted due to an issue with file transfers to USB devices and ejecting media.

I believe that it’s due to how the kernel “caches” and “buffers” the drive. It completes the operation sooner (to the program) while the kernel is still writing it to disk. It seems there is a way to turn it off, but I haven’t tested this:

I’m no developer when it comes to hardware, kernel or the file manager, but seeing as 18.04 has the feature, I’m not sure if 16.04 will receive any updates in this area without risking stability as a Long Term Release (LTS).

It could be the kernel version – v4.4 is the one from 16.04 originally, but can be upgraded to v4.10 (and later) with the LTS Enablement Stack.


That said, I’ve yet to have any corruption issues with USB drives, and I’m still rocking 16.04. Instead I observe my system’s resources using indicator-multiload (MATE has a similar applet called System Monitor too) and so if I eject from the file manager, I can usually see there is still some disk write activity and then the RAM (buffer/cache) usage goes down.

If it just happens to one drive, there is a possibility the medium itself is corrupted. I’ve known this with a bunch of old floppy disks :floppy_disk: Even after successfully letting it write and flushing the cache/buffer (without ejecting):

sudo bash -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

The file ended up corrupted, checksums didn’t match.

The link you provided regarding turning off write cache states that you have to enter the command every time the system is rebooted, this isn’t really practical. In relation to using system monitor to monitor the disk activity and ram buffer cache, I cannot see this under system monitor anywhere?

It’s not an issue with the USB stick as the device is brand new, it’s definitely due to the progress indicator that’s effectively useless.

I’m also running kernel 4.17.19. Updating the kernel hasn’t made a difference and as a long term Ubuntu MATE user I can assure you this has been an issue for quite some time, it was present on 15.04 and every time someone complains about it people claim they haven’t experienced it so it’s a non issue - Yet they use methods to monitor buffer status before removing USB storage devices?

LTS or not, this is a fairly sizeable issue resulting in personal data corruption and I’m one of two people to complain of it in a week.

Can you see if this helps File copy to USB appears to finish too fast
Instead of adding the command applet to the mate panel, you could set up Conky.

Mostly as a suggestion to check if disk caching is indeed the culprit. If turning off write cache "solves" the problem on your next writes, then it will help isolate where the problem is for further drilling down. If it improves the situation, they'll be a way to make it permanent in the meantime.

With stock MATE System Monitor's applet, you can only observe graphs with basic tooltips. With indicator-multiload, you can click and configure it to show buffer/cache values and colour code the RAM graph like so:

When ejecting, you can see there's disk writing until it goes back to 0 kB/s. Some disks continue writing 100 MB/s after ejecting and others, 32 MB/s.

Certainly wasn't dismissing this as "non-issue". Considering 16.04 is mostly for security updates, 18.04 contains the latest developments with the new "safe to remove" notification. Thinking about it... I'm not even sure which package triggers that notification.

For all we know, specific hardware causes this behaviour, especially USB controllers and hubs, which may explain why some people experience it and others don't.

The generic USB Storage Device specification leaves a lot to be desired - for example, although essentially all USB thumb drives are flash based, there is no way to apply an Erase or Trim command, which totally screws up the drives ability to do proper wear leveling, and the protocols for controlling and reporting internal cache status are not well standardized .

So even if we have MATE tap into the Linux USB stack, and trip a notification to indicate that the device is no longer being streamed data from Linux’s disk cache (so the USB interface is no longer sending any data to the drive), that drive could still be caching data internally.

To get around these issues I always buy card readers and USB thumb drives which have a write indicator LED, then use the ‘safe remove’ and wait 10 to 20 seconds after the LED stops flickering.

2 Likes

The problem is that even considering the issues relating to flash storage, every other OS seems to be able to adequately detect when data transfer has completely finished and inform the user that the USB stick is definitely safe to remove.

As it stands at the moment, the progress indicator under Ubuntu MATE is totally and utterly out of whack by about 10 mins regarding USB flash drives.

As of today I think that Ubuntu solved the problem, see my post there:


OK it might need more testing BUT the speed and the behavior of the loading bar was very different than usual (both at home and at the office) and matched the blinking of the USB dongle.

1 Like

Very interesting, I’ll have to keep an eye on this myself as I feel it’s a very serious concern.

Take a look at this workaround – you can tweak a config file so the kernel will physically finish writing to disks sooner.

1 Like

Very interesting. Thanks.

I’ll give it a go.