11:21 p.m.
I have a script that mounts an MP3 device, copies files and umounts. As
I have counted on the umount to hang until the data transfer is
complete. That has worked fine until recently. Now the umount finishes,
leaving much of the data uncopied.
Fedora 32, up-to-date at end-of-life.
Has some parameter been changed inadvertently? Or was I just lucky, and
my luck has run out? If so, is there a way to test for the real end of
transfer?
11:26 p.m.
On 8/25/21 9:21 PM, Geoffrey Leach wrote:
I have a script that mounts an MP3 device, copies files and umounts.
As
I have counted on the umount to hang until the data transfer is
complete. That has worked fine until recently. Now the umount finishes,
leaving much of the data uncopied.
You're saying that the copy command has finished and then it runs
umount? umount will certainly wait until the data is flushed. What's
missing on the destination? Is the filesystem corrupted?
Has some parameter been changed inadvertently? Or was I just lucky,
and
my luck has run out? If so, is there a way to test for the real end of
transfer?
You could add a "sync" after the copy and before the "umount".
12:01 a.m.
On Wed, 25 Aug 2021 21:26:19 -0700
Samuel Sieb <samuel(a)sieb.net> wrote:
On 8/25/21 9:21 PM, Geoffrey Leach wrote:
> I have a script that mounts an MP3 device, copies files and
> umounts. As I have counted on the umount to hang until the data
> transfer is complete. That has worked fine until recently. Now the
> umount finishes, leaving much of the data uncopied.
You're saying that the copy command has finished and then it runs
umount? umount will certainly wait until the data is flushed.
What's missing on the destination? Is the filesystem corrupted?
> Has some parameter been changed inadvertently? Or was I just lucky,
> and my luck has run out? If so, is there a way to test for the real
> end of transfer?
You could add a "sync" after the copy and before the "umount".
Just what I needed. Thanks.
2:11 a.m.
On 8/26/21 7:01 AM, Geoffrey Leach wrote:
On Wed, 25 Aug 2021 21:26:19 -0700
> You could add a "sync" after the copy and before the
"umount".
Just what I needed. Thanks.
Absolutely not.
If umount is failing to do a complete flushing by itself it is a serious
(press-release level) bug and must be investigated.
Which filesystem are you using?
Regards.
--
Roberto Ragusa mail at robertoragusa.it
2:40 a.m.
On 8/26/21 1:11 AM, Roberto Ragusa wrote:
If umount is failing to do a complete flushing by itself it is a
serious
(press-release level) bug and must be investigated.
I agree that an issue like this needs to be reported and investigated,
but there's no reason not to use such a simple mitigation as syncing
before umount while waiting for a permanent fix.
7:57 a.m.
On Thu, 26 Aug 2021 09:11:34 +0200
Roberto Ragusa <mail(a)robertoragusa.it> wrote:
On 8/26/21 7:01 AM, Geoffrey Leach wrote:
> On Wed, 25 Aug 2021 21:26:19 -0700
>> You could add a "sync" after the copy and before the
"umount".
>
> Just what I needed. Thanks.
Absolutely not.
If umount is failing to do a complete flushing by itself it is a
serious (press-release level) bug and must be investigated.
Which filesystem are you using?
ext4 -> vfat
12:34 p.m.
On 8/26/21 2:57 PM, Geoffrey Leach wrote:
On Thu, 26 Aug 2021 09:11:34 +0200
Roberto Ragusa <mail(a)robertoragusa.it> wrote:
> On 8/26/21 7:01 AM, Geoffrey Leach wrote:
>> On Wed, 25 Aug 2021 21:26:19 -0700
>
>>> You could add a "sync" after the copy and before the
"umount".
>>
>> Just what I needed. Thanks.
>
> Absolutely not.
> If umount is failing to do a complete flushing by itself it is a
> serious (press-release level) bug and must be investigated.
>
> Which filesystem are you using?
ext4 -> vfat
The destination is the importat part: vfat.
Are you sure the umount is successful?
Maybe umount fails immediately because of something else locking the disk
(e.g. a cd into the mounted path is enough to make umount fail),
the script finishes and you remove the disk assuming it has been unmounted
but it has not been.
Regards.
--
Roberto Ragusa mail at robertoragusa.it
12:45 p.m.
On Thu, 26 Aug 2021 19:34:55 +0200
Roberto Ragusa <mail(a)robertoragusa.it> wrote:
On 8/26/21 2:57 PM, Geoffrey Leach wrote:
> On Thu, 26 Aug 2021 09:11:34 +0200
> Roberto Ragusa <mail(a)robertoragusa.it> wrote:
>
>> On 8/26/21 7:01 AM, Geoffrey Leach wrote:
>>> On Wed, 25 Aug 2021 21:26:19 -0700
>>
>>>> You could add a "sync" after the copy and before the
"umount".
>>>
>>> Just what I needed. Thanks.
>>
>> Absolutely not.
>> If umount is failing to do a complete flushing by itself it is a
>> serious (press-release level) bug and must be investigated.
>>
>> Which filesystem are you using?
>
> ext4 -> vfat
The destination is the importat part: vfat.
Are you sure the umount is successful?
Maybe umount fails immediately because of something else locking the
disk (e.g. a cd into the mounted path is enough to make umount fail),
the script finishes and you remove the disk assuming it has been
unmounted but it has not been.
Good thought, but no. It's a laptop, and nothing else is happening.
12:56 p.m.
On 2021-08-26 10:34 a.m., Roberto Ragusa wrote:
On 8/26/21 2:57 PM, Geoffrey Leach wrote:
> On Thu, 26 Aug 2021 09:11:34 +0200
> Roberto Ragusa <mail(a)robertoragusa.it> wrote:
>
>> On 8/26/21 7:01 AM, Geoffrey Leach wrote:
>>> On Wed, 25 Aug 2021 21:26:19 -0700
>>
>>>> You could add a "sync" after the copy and before the
"umount".
>>>
>>> Just what I needed. Thanks.
>>
>> Absolutely not.
>> If umount is failing to do a complete flushing by itself it is a
>> serious (press-release level) bug and must be investigated.
>>
>> Which filesystem are you using?
>
> ext4 -> vfat
The destination is the importat part: vfat.
Are you sure the umount is successful?
Maybe umount fails immediately because of something else locking the disk
(e.g. a cd into the mounted path is enough to make umount fail),
the script finishes and you remove the disk assuming it has been unmounted
but it has not been.
Are there any messages from the umount command? If you run "mount", is
the drive still mounted?
4:27 p.m.
On Thu, 26 Aug 2021 10:56:47 -0700
Samuel Sieb <samuel(a)sieb.net> wrote:
On 2021-08-26 10:34 a.m., Roberto Ragusa wrote:
> On 8/26/21 2:57 PM, Geoffrey Leach wrote:
>> On Thu, 26 Aug 2021 09:11:34 +0200
>> Roberto Ragusa <mail(a)robertoragusa.it> wrote:
>>
>>> On 8/26/21 7:01 AM, Geoffrey Leach wrote:
>>>> On Wed, 25 Aug 2021 21:26:19 -0700
>>>
>>>>> You could add a "sync" after the copy and before the
"umount".
>>>>
>>>> Just what I needed. Thanks.
>>>
>>> Absolutely not.
>>> If umount is failing to do a complete flushing by itself it is a
>>> serious (press-release level) bug and must be investigated.
>>>
>>> Which filesystem are you using?
>>
>> ext4 -> vfat
>
> The destination is the importat part: vfat.
> Are you sure the umount is successful?
> Maybe umount fails immediately because of something else locking
> the disk (e.g. a cd into the mounted path is enough to make umount
> fail), the script finishes and you remove the disk assuming it has
> been unmounted but it has not been.
Are there any messages from the umount command? If you run "mount",
is the drive still mounted?
No messages, but findmnt confirms that the device is still mounted,
whereupon umount finished clean.
12:48 p.m.
On 8/25/21 9:26 PM, Samuel Sieb wrote:
You could add a "sync" after the copy and before the
"umount".
https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSync
We're getting into myths and legends territory. Running sync before
umount isn't harmful, but umount will flush any dirty buffers, so it
isn't necessary.
I'm not able to reproduce the problem described, though I'm running
Fedora 34. When I attach a USB drive, it is mounted with the "flush"
option. I can copy arbitrarily large file sets to the drive, and the
"cp" command appears to be nearly synchronous. The sequence "time cp
... ; time umount ..." indicates nearly all of the time spent is in the
cp command and umount requires between .2 and .3 seconds.
What are the indications that the copy operation originally reported was
incomplete? Does the "umount" command have any flags? Are there any
errors in "dmesg" after the script runs?
12:54 p.m.
On 2021-08-26 10:48 a.m., Gordon Messmer wrote:
On 8/25/21 9:26 PM, Samuel Sieb wrote:
> You could add a "sync" after the copy and before the "umount".
https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSync
We're getting into myths and legends territory. Running sync before
umount isn't harmful, but umount will flush any dirty buffers, so it
isn't necessary.
I'm not suggesting any myths. I agree that umount is supposed to sync
before finishing, but if something is going wrong with the umount, then
the sync will at least make sure everything is written.
4:16 p.m.
On Thu, 26 Aug 2021 10:48:15 -0700
Gordon Messmer <gordon.messmer(a)gmail.com> wrote:
On 8/25/21 9:26 PM, Samuel Sieb wrote:
> You could add a "sync" after the copy and before the "umount".
https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSync
We're getting into myths and legends territory. Running sync before
umount isn't harmful, but umount will flush any dirty buffers, so it
isn't necessary.
I'm not able to reproduce the problem described, though I'm running
Fedora 34. When I attach a USB drive, it is mounted with the "flush"
option. I can copy arbitrarily large file sets to the drive, and the
"cp" command appears to be nearly synchronous. The sequence "time cp
... ; time umount ..." indicates nearly all of the time spent is in
the cp command and umount requires between .2 and .3 seconds.
What are the indications that the copy operation originally reported
was incomplete? Does the "umount" command have any flags? Are there
any errors in "dmesg" after the script runs?
I was not aware of the mount flush option. Just what was needed. Thanks.
The MP3 files that were being transferred were empty.
The problem emerged only recently on a Fedora 32 system that is
untouched since end-of-life, so its not entirely surprising that its
not reproducible, let alone on 34.
dmesg says
FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run
fsck.
That says it all. Pity I never looked.
Again, thanks.
4:50 a.m.
On Wed, 2021-08-25 at 21:21 -0700, Geoffrey Leach wrote:
I have a script that mounts an MP3 device, copies files and umounts.
As I have counted on the umount to hang until the data transfer is
complete. That has worked fine until recently. Now the umount
finishes, leaving much of the data uncopied.
Just to be very clear:
Have you issued a string of commands, so that you expect all the copy
commands to be executed in series before the unmount command is issued?
Write-caching can be a problem, though the system shouldn't be
unmounting before the cache is flushed.
Or have you issued the copy and unmount commands, and hope that the
actual copying will hold the unmounting at bay until finished?
Parallel commands can be a problem, if you expect them to be issued in
one order, but they don't quite get queued up in the way you expect.
This sounds rather similar to the problem I've seen with with USB
flashdrives, and SD cards, where I've seen the message pop up on the
screen telling me it's okay to unplug the drive, but clearly the data
transfers actually haven't yet finished. In my case, that would be
some time after dragging and dropping files onto the destination, then
waiting what seemed a reasonable amount of time before I right-clicked
to eject or unmount the device
--
uname -rsvp
Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64
Boilerplate: All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.
8 a.m.
On Thu, 26 Aug 2021 19:20:06 +0930
Tim via users <users(a)lists.fedoraproject.org> wrote:
On Wed, 2021-08-25 at 21:21 -0700, Geoffrey Leach wrote:
> I have a script that mounts an MP3 device, copies files and umounts.
> As I have counted on the umount to hang until the data transfer is
> complete. That has worked fine until recently. Now the umount
> finishes, leaving much of the data uncopied.
Just to be very clear:
Have you issued a string of commands, so that you expect all the copy
commands to be executed in series before the unmount command is
issued? Write-caching can be a problem, though the system shouldn't be
unmounting before the cache is flushed.
Or have you issued the copy and unmount commands, and hope that the
actual copying will hold the unmounting at bay until finished?
Parallel commands can be a problem, if you expect them to be issued in
one order, but they don't quite get queued up in the way you expect.
This sounds rather similar to the problem I've seen with with USB
flashdrives, and SD cards, where I've seen the message pop up on the
screen telling me it's okay to unplug the drive, but clearly the data
transfers actually haven't yet finished. In my case, that would be
some time after dragging and dropping files onto the destination, then
waiting what seemed a reasonable amount of time before I right-clicked
to eject or unmount the device
copy ... copy ...umount
8:09 a.m.
maybe ... rsync... rsync ... sync ... unmount?
John Westerdale
Sr Container Consultant * RHCSA-8.2*
Red Hat <https://www.redhat.com/> NYC Office/WFH
john.westerdale(a)redhat.com
M: 201-376-9993 IM: jwesterd He / Him / His
<https://www.redhat.com/>
I respect your Life-Work balance.
No need to respond immediately if you receive this outside your normal
working hours.
On Thu, Aug 26, 2021 at 9:01 AM Geoffrey Leach <geoff(a)hughes.net> wrote:
On Thu, 26 Aug 2021 19:20:06 +0930
Tim via users <users(a)lists.fedoraproject.org> wrote:
> On Wed, 2021-08-25 at 21:21 -0700, Geoffrey Leach wrote:
> > I have a script that mounts an MP3 device, copies files and umounts.
> > As I have counted on the umount to hang until the data transfer is
> > complete. That has worked fine until recently. Now the umount
> > finishes, leaving much of the data uncopied.
>
> Just to be very clear:
>
> Have you issued a string of commands, so that you expect all the copy
> commands to be executed in series before the unmount command is
> issued? Write-caching can be a problem, though the system shouldn't be
> unmounting before the cache is flushed.
>
> Or have you issued the copy and unmount commands, and hope that the
> actual copying will hold the unmounting at bay until finished?
> Parallel commands can be a problem, if you expect them to be issued in
> one order, but they don't quite get queued up in the way you expect.
>
>
>
> This sounds rather similar to the problem I've seen with with USB
> flashdrives, and SD cards, where I've seen the message pop up on the
> screen telling me it's okay to unplug the drive, but clearly the data
> transfers actually haven't yet finished. In my case, that would be
> some time after dragging and dropping files onto the destination, then
> waiting what seemed a reasonable amount of time before I right-clicked
> to eject or unmount the device
copy ... copy ...umount
_______________________________________________
users mailing list -- users(a)lists.fedoraproject.org
To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
8:48 a.m.
On Thu, 26 Aug 2021 09:09:30 -0400
John Westerdale <jwesterd(a)redhat.com> wrote:
maybe ... rsync... rsync ... sync ... unmount?
On Thu, Aug 26, 2021 at 9:01 AM Geoffrey Leach <geoff(a)hughes.net>
wrote:
> On Thu, 26 Aug 2021 19:20:06 +0930
> Tim via users <users(a)lists.fedoraproject.org> wrote:
>
> > On Wed, 2021-08-25 at 21:21 -0700, Geoffrey Leach wrote:
> > > I have a script that mounts an MP3 device, copies files and
> > > umounts. As I have counted on the umount to hang until the data
> > > transfer is complete. That has worked fine until recently. Now
> > > the umount finishes, leaving much of the data uncopied.
> >
> > Just to be very clear:
> >
> > Have you issued a string of commands, so that you expect all the
> > copy commands to be executed in series before the unmount command
> > is issued? Write-caching can be a problem, though the system
> > shouldn't be unmounting before the cache is flushed.
> >
> > Or have you issued the copy and unmount commands, and hope that
> > the actual copying will hold the unmounting at bay until finished?
> > Parallel commands can be a problem, if you expect them to be
> > issued in one order, but they don't quite get queued up in the
> > way you expect.
> >
> >
> >
> > This sounds rather similar to the problem I've seen with with USB
> > flashdrives, and SD cards, where I've seen the message pop up on
> > the screen telling me it's okay to unplug the drive, but clearly
> > the data transfers actually haven't yet finished. In my case,
> > that would be some time after dragging and dropping files onto
> > the destination, then waiting what seemed a reasonable amount of
> > time before I right-clicked to eject or unmount the device
>
> copy ... copy ...umount
maybe ... rsync... rsync ... sync ... unmount?
Would certainly work, but would
rsync solve the problem?
11:03 a.m.
The rsync wouldn't fix anything, but copy/copy would -at best- overwrite
the same data ... rsync would compare and update if different.
Either case, purely a workaround.
Umount of a file system should include a sync (for open files on that file
system).
John Westerdale
Sr Container Consultant * RHCSA-8.2*
Red Hat <https://www.redhat.com/> NYC Office/WFH
john.westerdale(a)redhat.com
M: 201-376-9993 IM: jwesterd He / Him / His
<https://www.redhat.com/>
I respect your Life-Work balance.
No need to respond immediately if you receive this outside your normal
working hours.
On Thu, Aug 26, 2021 at 9:49 AM Geoffrey Leach <geoff(a)hughes.net> wrote:
On Thu, 26 Aug 2021 09:09:30 -0400
John Westerdale <jwesterd(a)redhat.com> wrote:
> maybe ... rsync... rsync ... sync ... unmount?
>
>
> On Thu, Aug 26, 2021 at 9:01 AM Geoffrey Leach <geoff(a)hughes.net>
> wrote:
>
> > On Thu, 26 Aug 2021 19:20:06 +0930
> > Tim via users <users(a)lists.fedoraproject.org> wrote:
> >
> > > On Wed, 2021-08-25 at 21:21 -0700, Geoffrey Leach wrote:
> > > > I have a script that mounts an MP3 device, copies files and
> > > > umounts. As I have counted on the umount to hang until the data
> > > > transfer is complete. That has worked fine until recently. Now
> > > > the umount finishes, leaving much of the data uncopied.
> > >
> > > Just to be very clear:
> > >
> > > Have you issued a string of commands, so that you expect all the
> > > copy commands to be executed in series before the unmount command
> > > is issued? Write-caching can be a problem, though the system
> > > shouldn't be unmounting before the cache is flushed.
> > >
> > > Or have you issued the copy and unmount commands, and hope that
> > > the actual copying will hold the unmounting at bay until finished?
> > > Parallel commands can be a problem, if you expect them to be
> > > issued in one order, but they don't quite get queued up in the
> > > way you expect.
> > >
> > >
> > >
> > > This sounds rather similar to the problem I've seen with with USB
> > > flashdrives, and SD cards, where I've seen the message pop up on
> > > the screen telling me it's okay to unplug the drive, but clearly
> > > the data transfers actually haven't yet finished. In my case,
> > > that would be some time after dragging and dropping files onto
> > > the destination, then waiting what seemed a reasonable amount of
> > > time before I right-clicked to eject or unmount the device
> >
> > copy ... copy ...umount
> maybe ... rsync... rsync ... sync ... unmount?
Would certainly work, but would rsync solve the problem?
_______________________________________________
users mailing list -- users(a)lists.fedoraproject.org
To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
974
days inactive
974
days old
17 comments
7 participants
participants (7)
-
Geoffrey Leach -
Gordon Messmer -
Joe Zeff -
John Westerdale -
Roberto Ragusa -
Samuel Sieb -
Tim