On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski <belegdol(a)gmail.com&gt; wrote: > my local kernel rebuilds have started failing for no apparent reason - I > was using a similar command successfully for several months. > /mnt/openmediavault is a samba share. This is what gets output into the log: [snip] > /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: > (39, fsync failed: Permission denied) I suspect that ^^^^ is the real problem, and the incorrect checksum is a result of not being able to read or write the filesystem. Can you verify correct ownership and permissions on every directory in that path?

On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: > On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski <belegdol(a)gmail.com&gt; wrote: >> my local kernel rebuilds have started failing for no apparent reason - I >> was using a similar command successfully for several months. >> /mnt/openmediavault is a samba share. This is what gets output into the log: > > [snip] > >> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >> (39, fsync failed: Permission denied) > > I suspect that ^^^^ is the real problem, and the incorrect checksum is > a result of not being able to read or write the filesystem. Can you > verify correct ownership and permissions on every directory in that > path? If fsync(2) failed then that would be happening after the file descriptor was open, so it wouldn't be filesystem permissions but probably SELinux. Rich.

On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski wrote: > W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: >> On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: >>> On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski <belegdol(a)gmail.com&gt; wrote: >>>> my local kernel rebuilds have started failing for no apparent reason - I >>>> was using a similar command successfully for several months. >>>> /mnt/openmediavault is a samba share. This is what gets output into the log: >>> >>> [snip] >>> >>>> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >>>> (39, fsync failed: Permission denied) >>> >>> I suspect that ^^^^ is the real problem, and the incorrect checksum is >>> a result of not being able to read or write the filesystem. Can you >>> verify correct ownership and permissions on every directory in that >>> path? >> >> If fsync(2) failed then that would be happening after the file >> descriptor was open, so it wouldn't be filesystem permissions but >> probably SELinux. >> >> Rich. >> > > I am seeing no errors in setroubleshoot and setting selinux to > permissive does not help either. Can it be the selinux of the > bootstrapped instance, and if so, how can I control it? There is > nothing obvious in the logs of the host: AFAIK mock only ever uses one kernel (the host) so disabling SELinux on the host rules out my SELinux theory. Rich.

W dniu 22.09.2021 o 18:34, Julian Sikorski pisze: > W dniu 21.09.2021 o 23:12, Richard W.M. Jones pisze: >> On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski wrote: >>> W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: >>>> On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: >>>>> On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski >>>>> <belegdol(a)gmail.com&gt; wrote: >>>>>> my local kernel rebuilds have started failing for no apparent >>>>>> reason - I >>>>>> was using a similar command successfully for several months. >>>>>> /mnt/openmediavault is a samba share. This is what gets output >>>>>> into the log: >>>>> >>>>> [snip] >>>>> >>>>>> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >>>>>> >>>>>> (39, fsync failed: Permission denied) >>>>> >>>>> I suspect that ^^^^ is the real problem, and the incorrect >>>>> checksum is >>>>> a result of not being able to read or write the filesystem.  Can you >>>>> verify correct ownership and permissions on every directory in that >>>>> path? >>>> >>>> If fsync(2) failed then that would be happening after the file >>>> descriptor was open, so it wouldn't be filesystem permissions but >>>> probably SELinux. >>>> >>>> Rich. >>>> >>> >>> I am seeing no errors in setroubleshoot and setting selinux to >>> permissive does not help either. Can it be the selinux of the >>> bootstrapped instance, and if so, how can I control it? There is >>> nothing obvious in the logs of the host: >> >> AFAIK mock only ever uses one kernel (the host) so disabling SELinux >> on the host rules out my SELinux theory. >> >> Rich. >> > This is all super strange. If I delete the pesign result dir, it is > built without issues. Moreover, repodata folder is also created, with > the sha256 checksum in the file matching the one of the pesign rpm. > What is odd though, is that even after mock fails, gedit is claiming > that repomd.xml and the data file extracted from primary.xml.gz keep > changing on disk. Can it be some odd system clock issue between my > desktop an my NAS? > > Best regards, > Julian This "file changed on disk" issue affects every single file in the folders the last mock run touched, so e.g. /mnt/openmediavault/kernel/results/fedora-34-x86_64/kernel-5.14.6-300.s0ix03.fc34 is affected, but /mnt/openmediavault/kernel/results/fedora-34-x86_64/kernel-5.14.5-300.s0ix01.fc34 is not. What does gedit use to determine that the file changed? Timestamps shown by ls -l stay constant. Best regards, Julian

W dniu 21.09.2021 o 23:12, Richard W.M. Jones pisze: > On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski wrote: >> W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: >>> On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: >>>> On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski <belegdol(a)gmail.com&gt; wrote: >>>>> my local kernel rebuilds have started failing for no apparent reason - I >>>>> was using a similar command successfully for several months. >>>>> /mnt/openmediavault is a samba share. This is what gets output into the log: >>>> >>>> [snip] >>>> >>>>> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >>>>> (39, fsync failed: Permission denied) >>>> >>>> I suspect that ^^^^ is the real problem, and the incorrect checksum is >>>> a result of not being able to read or write the filesystem. Can you >>>> verify correct ownership and permissions on every directory in that >>>> path? >>> >>> If fsync(2) failed then that would be happening after the file >>> descriptor was open, so it wouldn't be filesystem permissions but >>> probably SELinux. >>> >>> Rich. >>> >> >> I am seeing no errors in setroubleshoot and setting selinux to >> permissive does not help either. Can it be the selinux of the >> bootstrapped instance, and if so, how can I control it? There is >> nothing obvious in the logs of the host: > > AFAIK mock only ever uses one kernel (the host) so disabling SELinux > on the host rules out my SELinux theory. > > Rich. > This is all super strange. If I delete the pesign result dir, it is built without issues. Moreover, repodata folder is also created, with the sha256 checksum in the file matching the one of the pesign rpm. What is odd though, is that even after mock fails, gedit is claiming that repomd.xml and the data file extracted from primary.xml.gz keep changing on disk. Can it be some odd system clock issue between my desktop an my NAS?

Am 22.09.21 um 19:38 schrieb Fabio Valentini: > On Wed, Sep 22, 2021 at 6:35 PM Julian Sikorski <belegdol(a)gmail.com&gt; > wrote: >> >> W dniu 21.09.2021 o 23:12, Richard W.M. Jones pisze: >>> On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski wrote: >>>> W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: >>>>> On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: >>>>>> On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski >>>>>> <belegdol(a)gmail.com&gt; wrote: >>>>>>> my local kernel rebuilds have started failing for no apparent >>>>>>> reason - I >>>>>>> was using a similar command successfully for several months. >>>>>>> /mnt/openmediavault is a samba share. This is what gets output >>>>>>> into the log: >>>>>> >>>>>> [snip] >>>>>> >>>>>>> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >>>>>>> >>>>>>> (39, fsync failed: Permission denied) >>>>>> >>>>>> I suspect that ^^^^ is the real problem, and the incorrect >>>>>> checksum is >>>>>> a result of not being able to read or write the filesystem.  Can you >>>>>> verify correct ownership and permissions on every directory in that >>>>>> path? >>>>> >>>>> If fsync(2) failed then that would be happening after the file >>>>> descriptor was open, so it wouldn't be filesystem permissions but >>>>> probably SELinux. >>>>> >>>>> Rich. >>>>> >>>> >>>> I am seeing no errors in setroubleshoot and setting selinux to >>>> permissive does not help either. Can it be the selinux of the >>>> bootstrapped instance, and if so, how can I control it? There is >>>> nothing obvious in the logs of the host: >>> >>> AFAIK mock only ever uses one kernel (the host) so disabling SELinux >>> on the host rules out my SELinux theory. >>> >>> Rich. >>> >> This is all super strange. If I delete the pesign result dir, it is >> built without issues. Moreover, repodata folder is also created, with >> the sha256 checksum in the file matching the one of the pesign rpm. What >> is odd though, is that even after mock fails, gedit is claiming that >> repomd.xml and the data file extracted from primary.xml.gz keep changing >> on disk. Can it be some odd system clock issue between my desktop an >> my NAS? > > Oh my, that filesystem is mounted over a network? What filesystem is > backing that directory? SMB? NFS? > I assume that the issue you're seeing is caused by something like "NFS > doesn't support the fsync call properly" ... > > Fabio > It is a local network samba share, from a host running armbian and openmediavault. This used to work until last week or so, which is why I am starting to suspect some strange clock problem. I have tried rebooting the NAS but it did not help. Best regards, Julian

W dniu 28.09.2021 o 09:32, Julian Sikorski pisze: > W dniu 22.09.2021 o 21:07, Julian Sikorski pisze: >> Am 22.09.21 um 19:38 schrieb Fabio Valentini: >>> On Wed, Sep 22, 2021 at 6:35 PM Julian Sikorski <belegdol(a)gmail.com&gt; >>> wrote: >>>> >>>> W dniu 21.09.2021 o 23:12, Richard W.M. Jones pisze: >>>>> On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski wrote: >>>>>> W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: >>>>>>> On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James wrote: >>>>>>>> On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski >>>>>>>> <belegdol(a)gmail.com&gt; wrote: >>>>>>>>> my local kernel rebuilds have started failing for no apparent >>>>>>>>> reason - I >>>>>>>>> was using a similar command successfully for several months. >>>>>>>>> /mnt/openmediavault is a samba share. This is what gets output >>>>>>>>> into the log: >>>>>>>> >>>>>>>> [snip] >>>>>>>> >>>>>>>>> /mnt/openmediavault/kernel/results/fedora-34-x86_64/pesign-113-16.fc35/pesign-113-16.fc34.x86_64.rpm: >>>>>>>>> >>>>>>>>> (39, fsync failed: Permission denied) >>>>>>>> >>>>>>>> I suspect that ^^^^ is the real problem, and the incorrect >>>>>>>> checksum is >>>>>>>> a result of not being able to read or write the filesystem. >>>>>>>> Can you >>>>>>>> verify correct ownership and permissions on every directory in >>>>>>>> that >>>>>>>> path? >>>>>>> >>>>>>> If fsync(2) failed then that would be happening after the file >>>>>>> descriptor was open, so it wouldn't be filesystem permissions but >>>>>>> probably SELinux. >>>>>>> >>>>>>> Rich. >>>>>>> >>>>>> >>>>>> I am seeing no errors in setroubleshoot and setting selinux to >>>>>> permissive does not help either. Can it be the selinux of the >>>>>> bootstrapped instance, and if so, how can I control it? There is >>>>>> nothing obvious in the logs of the host: >>>>> >>>>> AFAIK mock only ever uses one kernel (the host) so disabling SELinux >>>>> on the host rules out my SELinux theory. >>>>> >>>>> Rich. >>>>> >>>> This is all super strange. If I delete the pesign result dir, it is >>>> built without issues. Moreover, repodata folder is also created, with >>>> the sha256 checksum in the file matching the one of the pesign rpm. >>>> What >>>> is odd though, is that even after mock fails, gedit is claiming that >>>> repomd.xml and the data file extracted from primary.xml.gz keep >>>> changing >>>> on disk. Can it be some odd system clock issue between my desktop >>>> an my NAS? >>> >>> Oh my, that filesystem is mounted over a network? What filesystem is >>> backing that directory? SMB? NFS? >>> I assume that the issue you're seeing is caused by something like "NFS >>> doesn't support the fsync call properly" ... >>> >>> Fabio >>> >> It is a local network samba share, from a host running armbian and >> openmediavault. This used to work until last week or so, which is why >> I am starting to suspect some strange clock problem. I have tried >> rebooting the NAS but it did not help. >> >> Best regards, >> Julian > > I have now managed to generate logs on the NAS, see attached. There > are two errors which stand out: NT_STATUS_NO_EAS_ON_FILE and > NT_STATUS_ACCESS_DENIED later. Does this give any indication as to > what might be happening? > > Best regards, > Julian I have been investigating this further. When building from a different client, I am getting most interesting results: $ mock --chain --localrepo=/mnt/openmediavault/kernel -r fedora-34-x86_64 goffice/goffice-0.10.49-1.fc36.src.rpm gnumeric/gnumeric-1.12.49-1.fc36.src.rpm will keep working from the laptop as goffice-0.10.49 was never rebuilt from the desktop, as long as goffice-0.10.50 results folder is not present in /mnt/openmediavault/kernel. $ mock --chain --localrepo=/mnt/openmediavault/kernel -r fedora-34-x86_64 goffice/goffice-0.10.50-2.fc36.src.rpm gnumeric/gnumeric-1.12.50-2.fc36.src.rpm will fail when run from the laptop, even if the goffice-0.10.50-2.fc36 folder does not exist when the build is started. Moreover, building to a different folder does not work either: $ mock --chain --localrepo=/mnt/openmediavault/kernel1 -r fedora-34-x86_64 goffice/goffice-0.10.50-2.fc36.src.rpm gnumeric/gnumeric-1.12.50-2.fc36.src.rpm This is beyond strange. To me it looks as if either the permissions set from the desktop PC are somehow wrong and additionally persist the folder deletion, or as if createrepo needs different permissions to generate checksum for goffice-0.10.50-2 than for goffice-0.10.49-1. Neither of the above makes too much sense. What does createrepo use to generate the checksum exactly? It would be good to have a simpler reproducer. Best regards, Julian

W dniu 20.10.2021 o 20:09, Julian Sikorski pisze: > W dniu 29.09.2021 o 10:08, Julian Sikorski pisze: > > W dniu 28.09.2021 o 09:32, Julian Sikorski pisze: > > > W dniu 22.09.2021 o 21:07, Julian Sikorski pisze: > > > > Am 22.09.21 um 19:38 schrieb Fabio Valentini: > > > > > On Wed, Sep 22, 2021 at 6:35 PM Julian Sikorski > > > > > <belegdol(a)gmail.com&gt; wrote: > > > > > > > > > > > > W dniu 21.09.2021 o 23:12, Richard W.M. Jones pisze: > > > > > > > On Tue, Sep 21, 2021 at 10:16:17PM +0200, Julian Sikorski > > > > > > > wrote: > > > > > > > > W dniu 21.09.2021 o 11:00, Richard W.M. Jones pisze: > > > > > > > > > On Mon, Sep 20, 2021 at 11:45:39AM -0600, Jerry James > > > > > > > > > wrote: > > > > > > > > > > On Mon, Sep 20, 2021 at 10:49 AM Julian Sikorski > > > > > > > > > > <belegdol(a)gmail.com&gt; wrote: > > > > > > > > > > > my local kernel rebuilds have started failing for > > > > > > > > > > > no apparent > > > > > > > > > > > reason - I > > > > > > > > > > > was using a similar command successfully for > > > > > > > > > > > several months. > > > > > > > > > > > /mnt/openmediavault is a samba share. This is > > > > > > > > > > > what gets > > > > > > > > > > > output into the log: > > > > > > > > > > > > > > > > > > > > [snip] > > > > > > > > > > > > > > > > > > > > > /mnt/openmediavault/kernel/results/fedora-34- > > > > > > > > > > > x86_64/pesign-113-16.fc35/pesign-113- > > > > > > > > > > > 16.fc34.x86_64.rpm: > > > > > > > > > > > > > > > > > > > > > > (39, fsync failed: Permission denied) > > > > > > > > > > > > > > > > > > > > I suspect that ^^^^ is the real problem, and the > > > > > > > > > > incorrect > > > > > > > > > > checksum is > > > > > > > > > > a result of not being able to read or write the > > > > > > > > > > filesystem. > > > > > > > > > > Can you > > > > > > > > > > verify correct ownership and permissions on every > > > > > > > > > > directory in > > > > > > > > > > that > > > > > > > > > > path? > > > > > > > > > > > > > > > > > > If fsync(2) failed then that would be happening after > > > > > > > > > the file > > > > > > > > > descriptor was open, so it wouldn't be filesystem > > > > > > > > > permissions but > > > > > > > > > probably SELinux. > > > > > > > > > > > > > > > > > > Rich. > > > > > > > > > > > > > > > > > > > > > > > > > I am seeing no errors in setroubleshoot and setting > > > > > > > > selinux to > > > > > > > > permissive does not help either. Can it be the selinux > > > > > > > > of the > > > > > > > > bootstrapped instance, and if so, how can I control it? > > > > > > > > There is > > > > > > > > nothing obvious in the logs of the host: > > > > > > > > > > > > > > AFAIK mock only ever uses one kernel (the host) so > > > > > > > disabling SELinux > > > > > > > on the host rules out my SELinux theory. > > > > > > > > > > > > > > Rich. > > > > > > > > > > > > > This is all super strange. If I delete the pesign result > > > > > > dir, it is > > > > > > built without issues. Moreover, repodata folder is also > > > > > > created, with > > > > > > the sha256 checksum in the file matching the one of the > > > > > > pesign > > > > > > rpm. What > > > > > > is odd though, is that even after mock fails, gedit is > > > > > > claiming that > > > > > > repomd.xml and the data file extracted from primary.xml.gz > > > > > > keep > > > > > > changing > > > > > > on disk. Can it be some odd system clock issue between my > > > > > > desktop > > > > > > an my NAS? > > > > > > > > > > Oh my, that filesystem is mounted over a network? What > > > > > filesystem is > > > > > backing that directory? SMB? NFS? > > > > > I assume that the issue you're seeing is caused by something > > > > > like "NFS > > > > > doesn't support the fsync call properly" ... > > > > > > > > > > Fabio > > > > > > > > > It is a local network samba share, from a host running armbian > > > > and > > > > openmediavault. This used to work until last week or so, which > > > > is > > > > why I am starting to suspect some strange clock problem. I have > > > > tried rebooting the NAS but it did not help. > > > > > > > > Best regards, > > > > Julian > > > > > > I have now managed to generate logs on the NAS, see attached. > > > There > > > are two errors which stand out: NT_STATUS_NO_EAS_ON_FILE and > > > NT_STATUS_ACCESS_DENIED later. Does this give any indication as > > > to > > > what might be happening? > > > > > > Best regards, > > > Julian > > > > I have been investigating this further. When building from a > > different > > client, I am getting most interesting results: > > > > $ mock --chain --localrepo=/mnt/openmediavault/kernel -r > > fedora-34-x86_64 goffice/goffice-0.10.49-1.fc36.src.rpm > > gnumeric/gnumeric-1.12.49-1.fc36.src.rpm > > > > will keep working from the laptop as goffice-0.10.49 was never > > rebuilt > > from the desktop, as long as goffice-0.10.50 results folder is not > > present in /mnt/openmediavault/kernel. > > > > $ mock --chain --localrepo=/mnt/openmediavault/kernel -r > > fedora-34-x86_64 goffice/goffice-0.10.50-2.fc36.src.rpm > > gnumeric/gnumeric-1.12.50-2.fc36.src.rpm > > > > will fail when run from the laptop, even if the goffice-0.10.50- > > 2.fc36 > > folder does not exist when the build is started. Moreover, building > > to > > a different folder does not work either: > > > > $ mock --chain --localrepo=/mnt/openmediavault/kernel1 -r > > fedora-34-x86_64 goffice/goffice-0.10.50-2.fc36.src.rpm > > gnumeric/gnumeric-1.12.50-2.fc36.src.rpm > > > > This is beyond strange. To me it looks as if either the permissions > > set from the desktop PC are somehow wrong and additionally persist > > the > > folder deletion, or as if createrepo needs different permissions to > > generate checksum for goffice-0.10.50-2 than for goffice-0.10.49-1. > > Neither of the above makes too much sense. > > What does createrepo use to generate the checksum exactly? It would > > be > > good to have a simpler reproducer. > > > > Best regards, > > Julian > > > > I have looked into this further - running createrepo alone works > fine: > > $ createrepo_c /mnt/openmediavault/kernel/results/fedora-34-x86_64/ > Directory walk started > Directory walk done - 28 packages > Temporary output repo path: > /mnt/openmediavault/kernel/results/fedora-34-x86_64/.repodata/ > Preparing sqlite DBs > Pool started (with 5 workers) > Pool finished > $ createrepo /mnt/openmediavault/kernel/results/fedora-34-x86_64/ > Directory walk started > Directory walk done - 28 packages > Temporary output repo path: > /mnt/openmediavault/kernel/results/fedora-34-x86_64/.repodata/ > Preparing sqlite DBs > Pool started (with 5 workers) > Pool finished > > What is going on? Is mock not using createrepo? > > Best regards, > Julian Upgrading to F35 did not fix this sadly.

Best regards, Julian _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure