7

I am running Ubuntu 20.04 on an old Dell Optiplex 760 box and the transfer speeds while copying files to USB flash drives are quite slower than what they should be, furthermore, at the end of the transfer the transfer program (I've tried copying files with nautilus, cp, dd, rsync) hangs even after the whole file has been transferred to the USB drive - this hanging makes scripted file transfers difficult as execution can't proceed until the transfer program returns 0. I checked the file size and even did an md5sum checksum to check the files' integrity and saw that the files had been copied correctly and were intact and yet the transfer program hangs around for around 5 minutes after 100% of the file has been transferred.

I thought the issue might be the device being recognized as a USB-1.0 device. I checked the output of lsusb and cross-referenced that with the output of usb-devices and found out that the flash drives are always being recognized as having USB2.0 speeds i.e. 480Mbps - so no problem with USB versions.

user@box:~$ lsusb


Bus 006 Device 059: ID 0781:5567 SanDisk Corp. Cruzer Blade
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 10f1:1a19 Importek USB 2.0 Camera
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

the relevant part of the usb-devices output

T:  Bus=06 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 59 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0781 ProdID=5567 Rev=01.26
S:  Manufacturer=SanDisk
S:  Product=Cruzer Blade
S:  SerialNumber=200530450005728000EF
C:  #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=200mA
I:  If#=0x0 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usb-storage

which shows nothing amiss with the USB version being employed. I checked around and others have found some workarounds that have to do with setting vm.dirty_bytes and vm.dirty_background_bytes configuration in sysctl to $((15*1024*1024)) bytes. This didn't improve the transfer speeds but the transfer programs are now allowed smaller buffer sizes and the lingering after 100% has stopped.

While working in Windows 7 on the same machine - using the same flash drives, I can consistently transfer with average speeds of 7-8MBps. However when I reboot and switch back to Ubuntu, average transfer rates onto USB flash drives don't ever top 2MBps.

The next apparent solution might have to do with the sync mount option being used while mounting the drive. I checked my /etv/fstab file and it has only entries pertaining to my HDDs - both SATA and USB connected drives are listed. But, none of my flash drives get listed in there. I use udiskctl --mount -b /dev/sdXX to mount my USB drives - or nautiluts automounts them for me - and don't know how I can check if sync is a mount option.

How can I check if sync is enabled, if so how can I disable it in Ubuntu 20.04?

endrias
  • 647
  • From what you say about the transfer hanging, sync is not enabled. Your transfer program writes to cache, from where it gets transferred to the usb drive; your program hangs when it closes the file (and presumably calls fsync at that time). This will wait until everything from cache is written to the drive. Reading the file or running md5sum will not check if the files is ok on the drive, it'll just read the cache. If you remove the drive during this time, plug it in somewhere else, and check the file, you'll see the file is corrupted. – Guntram Blohm May 02 '21 at 11:20
  • And by the way, the mount command won't show you options that remain in default; cat /proc/mounts or grep /dev/sdXX /proc/mounts should show you all. – Guntram Blohm May 02 '21 at 11:22
  • can I suggest when comparing different OSes the time taken should be the time from start to completion where "completion" includes "safely remove device". Also, have you tried any other memory sticks or USB mass storage devices by way of comparison? – Rodney May 02 '21 at 13:07

2 Answers2

8

When I have lots of writing to do to external devices, I disable the write cache for the device with hdparm. Here’s how I do it:

  1. Open Terminal
  2. Determine the device name with fdisk: 
    sudo fdisk -l
  3. Disable the write cache for the device: 
    sudo hdparm -W 0 /dev/sdX
    Note: Be sure to replace sdX with the appropriate value.

Notes:

  • This setting does not persist across reboots
  • This may make the machine appear to “freeze” during file copies
  • Transfer speeds will generally be about 70% of theoretical bus throughput (based on experience)
  • Some people suggest using the -K option in hdparm, but I’ve not seen this improve write throughput
matigo
  • 24,860
  • tnx. this is kinda like what I did by setting the vm.dirty_bytes and vm.dirty_background_bytes attributes using sysctl - in simple terms , and as I understand it - setting these values limits the overhead caching and buffering allowed for any program. The effects such measures have has to do with forcing them to show a more honest transfer rate rather than boosting the actual transfer rate - which remains quite lower, much lower than the 70% you've hinted at.. – endrias May 01 '21 at 17:07
6

When is it completed?

When e.g. Nautilus tells you that the transfer finished, it is usually not true (especially if the drive is mounted async), there is a significant difference between the Nautilus "completed" prompt and the end of actual write process.

The system monitor gnome shell extension will nicely complement copying with Nautilus, allowing you to keep an eye on the actual write process in real time; including USB devices.

About fstab:

The /etc/fstab file typically contains only such drives / partitions that you want to be automatically mounted upon each system start.

Since you use the USB drive only occasionally, typically you wouldn't want it in your fstab.

About sync:

You are correct about the sync mount option: it slows down things incredibly (I have experience with only ext4 file systems in this regard, and I can attest to the slow-down.)

Mounting:

I would suggest not using udiskctl (documentation) (because I don't know it well) (and also because it was not trivial for you to find out its arguments)

Instead, I would suggest using the mount command (documentation), with wich you can clearly state your async option. (Though it's worth noting that async mounting is mount's default behaviour, so normally it doesn't need stating (only if you want to be extra sure).)

As preparation, create a mountpoint, e.g.:

sudo mkdir /mnt/mystuff
sudo chown YOURUSERNAME:YOURUSERNAME /mnt/mystuff

And then:

sudo mount -o async /dev/sdXX /mnt/mystuff
Levente
  • 4,637
  • I get that mounting using mount could prove to be a solution but I was rather hoping for an alternative that doesn't require using passwordless sudo lines - since I've already opted for using udisksctl which does auto-mounting of hotpluggable devices without creds - in my /etc/sudoers file which seems like the only sensible way to daemonise the mount command. or do u know a safe daemonised implementation of mount that doesn't require passwordless sudo or leaving your password lying around in a script somewhere? – endrias May 01 '21 at 15:29
  • No, I'm not well versed neither in passwordless sudoing, nor in daemonising. – Levente May 01 '21 at 15:38
  • There are many questions on this site about "slow copy", not even limited to USB. Anytime the write is slower than the read, system buffers tend to fill up, and everything slows to a crawl. Try nocache to eliminate the unnecessary read buffering, and try copy programs (tar, cpio,...) that allow you to set a big buffer size to limit memory fragmentation. – ubfan1 May 01 '21 at 16:57
  • @ubfan1 "Try nocache to eliminate the unnecessary read buffering" — is that an option of something? What is it an option of? (I don't see the term in udiskctl's or mount's manual.) Regarding "try copy programs (tar, cpio,...) that allow you to set a big buffer size" I don't see buffer mentioned in tar's manual either... Would you want to add an answer about this info? – Levente May 01 '21 at 17:07
  • @levente nocache is a command from the from the nocache package, use is nocache cpio... etc. tar (tape archive) use "-b for "blocking factor) to increase the chunk size being written. All this is already in many answers, nothing new here. – ubfan1 May 01 '21 at 17:13
  • for those who have suggested fiddling with the caches and buffers inherent in any of the copy programs you've stated, I've checked and if the problem had to do with too small buffer sizes, the result would only be the buffer filling up quickly there by causing a pause until subsequent read and re-population of the buffer - a rate which will be limited by the speed with which the buffer can be emptied - a rate which will be limited by the rate at which the USB drive can be written unto. so adjusting the buffer sizes doesn't seem to help the low throughput, – endrias May 01 '21 at 17:15
  • @endrias, the difference between system buffers and user-space buffers is reducing the memory allocation thrashing. I find giving tar a 200MB buffer (with the appropriate blocking factor fro your block size) gives me a 3x copy speed improvement (10MB/s to 30MB/s, still a fraction of what it should be, but better). – ubfan1 May 01 '21 at 17:35