churchyard reported a new issue against the project: `releng` that you are following: `` The https://fedoraproject.org/wiki/Fails_to_build_from_source policy says:
If a FTBFS bug remains in NEW state, a weekly reminder will be sent.
This never happened.
(Opened also as https://pagure.io/fesco/issue/1973 - feel free to close the less appropriate one.) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been assigned to `mohanboddu` by mohanboddu.
mohanboddu added a new comment to an issue you are following: `` We dont have the tooling ready for it, I will work on it asap. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
till added a new comment to an issue you are following: `` @zbyszek is also working on this and currently uses the Bugzilla NEEDINFO reminders for this. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: ``
The package will be orphaned after the FTBFS bug is in NEW state for 8 weeks.
This should happen this week. No remainders have been sent so far.
Is the orphaning still happening or is it blocked by the reminders? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` I am sorry that I poke at this again and again, but is there anything I can do to move this forward?
**I can send the reminders myself**, if this would make the policy happen. Let me know.
There are 1042 FTBFS bugzillas in NEW state. Those packages should have been orphaned already by this policy, yet nothing happened yet.
Getting unmaintained packages orphaned (and later retired) is important to get rid of old cruft, for me personally this is relevant for the "get rid of python2" effort, where plenty of packages that use python2 (yet nobody updates them) would go away. I also believe this is needed for the general health of the project.
This appears to be the consensus of FESCo anyway https://pagure.io/fesco/issue/1890
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
mohanboddu added a new comment to an issue you are following: `` @churchyard Can you create a script to do this and send it as PR. As I dont have time for this, I would appreciate your help.
Thanks ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` I'll have a look in couple days.
Just to make sure, the script will:
* get all NEW bugs blocking the FTBFS tracker * post a comment (to all fo them) from https://pagure.io/fesco/issue/1890#comment-520503 (except it says F29) * set a needinfo flag (assignee) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` Here's a script: https://github.com/hroncok/ftbfs_bugzilla/blob/master/ftbfs_weekly_reminder.... - it is public domain.
It updates bugz one by one and maintains a list of already updated bugz (at filesystem level) = it can be interrupted and run again.
I guess updating in bulk would be faster, but:
* I found no way to set needinfo for assignee in bulk * it tends to error and there is no (easy) way to tell which bugs got updated
So this is slow but quite reliable.
There is also bugzilla.log that gives you all the debug information needed.
The script has a TESTING boolean, set to True if you really want to run this, keep at False to use partner-bugzilla and fake bugz. Uses click and python-bugzilla (available from PyPI or RPMs). Tested on Python 3.7 only, should work with 3.6 as well.
Let me know if this is not what you wanted. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` Is the script OK for what you needed? How can I help? Shall I run it myself? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` https://pagure.io/releng/pull-request/7881 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: ``
A week before the mass branching any packages which still have open FTBFS bugs from the previous release will be retired.
That shall be 2019-02-12.
We have 1043 open F29 FTBFS bugs. Are we going to retire 1043 packages in February? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
syeghiay added a new comment to an issue you are following: `` @humaton will talk to @mohanboddu about this ticket. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` We have a new FTBFS tracker for a new release. Can we get this started please? How can I help? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
kevin added a new comment to an issue you are following: `` So, we need to set this up in infrastructure.
It looks like it uses an interactive login? Can we have it take a user/token for running in cron?
I guess we also need to determine what user we want to use here... releng? bodhi? something new for just this use? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: ``
It looks like it uses an interactive login? Can we have it take a user/token for running in cron?
Do you have an example of such script so I don't need to reinvent the wheel?
I guess we also need to determine what user we want to use here...
Releng files those bugs, so I guess we can reuse that user as well. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
kevin added a new comment to an issue you are following: `` There is this one:
https://github.com/fedora-infra/fas/blob/develop/tools/export-bugzilla.py
Yeah, releng user makes the most sense I guess... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` `from turbogears import config`, yummy :(
Do you have an example config I can see to rewrite this without turbogears? I guess I can use plain ConfigParser. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` I'm going to assume this for now:
``` [bugzilla] username = releng password = hunter2 ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` Adjusted https://pagure.io/releng/pull-request/7881 to use a config file. Adjust the config file paths as desired. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` Can we get the cron set up please? I'll try to talk to @kevin on IRC on Monday, so we can work on it together. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
smooge added a new comment to an issue you are following: `` Kevin is out on vacation until the 17th. I will try and look at what needs to be done on Monday. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Closed as Fixed by smooge.
smooge added a new comment to an issue you are following: `` Mohan and I deployed this this morning. It should start bugging people on the weekly cron.weekly run ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` Thanks. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Open by churchyard.
churchyard added a new comment to an issue you are following: `` While I have seen some of the comments, last time they have happened at 2019-06-17. This is apparently not happening weekly.
Take for example this [randomly selected F30FTBFS bug](https://bugzilla.redhat.com/show_bug.cgi?id=1675170):
The comments happened:
* 2019-04-27 (possibly a test run) * 2019-06-03 * 2019-06-17 (2 weeks after the previous, not 1 week)
If more comments appear today, we probably just run it bi-weekly instead of weekly. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
smooge added a new comment to an issue you are following: `` Cron job script was hosed. I have fixed the script and I am now running the script by hand. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
smooge added a new comment to an issue you are following: `` OK running the script, it gave the following output:
``` [root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron Cloning into 'releng'... remote: Counting objects: 7869, done. remote: Compressing objects: 100% (3487/3487), done. remote: Total 7869 (delta 4422), reused 7712 (delta 4286) Receiving objects: 100% (7869/7869), 1.36 MiB | 1.89 MiB/s, done. Resolving deltas: 100% (4422/4422), done. Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from happening. Gathering bugz, this can take a while... There are 755 NEW bugz, will send a reminder Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED Will update 25 bugz [####################################] 100% Finished! Updated 0 bugz Failed to update 25 bugz https://bugzilla.redhat.com/show_bug.cgi?id=1674669 https://bugzilla.redhat.com/show_bug.cgi?id=1674680 https://bugzilla.redhat.com/show_bug.cgi?id=1674753 https://bugzilla.redhat.com/show_bug.cgi?id=1674783 https://bugzilla.redhat.com/show_bug.cgi?id=1674817 https://bugzilla.redhat.com/show_bug.cgi?id=1675022 https://bugzilla.redhat.com/show_bug.cgi?id=1675036 https://bugzilla.redhat.com/show_bug.cgi?id=1675097 https://bugzilla.redhat.com/show_bug.cgi?id=1675108 https://bugzilla.redhat.com/show_bug.cgi?id=1675364 https://bugzilla.redhat.com/show_bug.cgi?id=1675407 https://bugzilla.redhat.com/show_bug.cgi?id=1675415 https://bugzilla.redhat.com/show_bug.cgi?id=1675454 https://bugzilla.redhat.com/show_bug.cgi?id=1675586 https://bugzilla.redhat.com/show_bug.cgi?id=1675593 https://bugzilla.redhat.com/show_bug.cgi?id=1675684 https://bugzilla.redhat.com/show_bug.cgi?id=1675784 https://bugzilla.redhat.com/show_bug.cgi?id=1675892 https://bugzilla.redhat.com/show_bug.cgi?id=1675899 https://bugzilla.redhat.com/show_bug.cgi?id=1675918 https://bugzilla.redhat.com/show_bug.cgi?id=1676017 https://bugzilla.redhat.com/show_bug.cgi?id=1676164 https://bugzilla.redhat.com/show_bug.cgi?id=1676181 https://bugzilla.redhat.com/show_bug.cgi?id=1676205 https://bugzilla.redhat.com/show_bug.cgi?id=1679955
```
I am not sure what is causing bzapi.update_bugs([bug.id], update) to give an exception. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` rerunning the script on failure should do. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
smooge added a new comment to an issue you are following: `` It seemed to fail identically on rerun
``` [root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron Cloning into 'releng'... remote: Counting objects: 7869, done. remote: Compressing objects: 100% (3487/3487), done. remote: Total 7869 (delta 4422), reused 7713 (delta 4286) Receiving objects: 100% (7869/7869), 1.36 MiB | 1.63 MiB/s, done. Resolving deltas: 100% (4422/4422), done. Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from happening. Gathering bugz, this can take a while... There are 755 NEW bugz, will send a reminder Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED Will update 25 bugz [####################################] 100% Finished! Updated 0 bugz Failed to update 25 bugz https://bugzilla.redhat.com/show_bug.cgi?id=1674669 https://bugzilla.redhat.com/show_bug.cgi?id=1674680 https://bugzilla.redhat.com/show_bug.cgi?id=1674753 https://bugzilla.redhat.com/show_bug.cgi?id=1674783 https://bugzilla.redhat.com/show_bug.cgi?id=1674817 https://bugzilla.redhat.com/show_bug.cgi?id=1675022 https://bugzilla.redhat.com/show_bug.cgi?id=1675036 https://bugzilla.redhat.com/show_bug.cgi?id=1675097 https://bugzilla.redhat.com/show_bug.cgi?id=1675108 https://bugzilla.redhat.com/show_bug.cgi?id=1675364 https://bugzilla.redhat.com/show_bug.cgi?id=1675407 https://bugzilla.redhat.com/show_bug.cgi?id=1675415 https://bugzilla.redhat.com/show_bug.cgi?id=1675454 https://bugzilla.redhat.com/show_bug.cgi?id=1675586 https://bugzilla.redhat.com/show_bug.cgi?id=1675593 https://bugzilla.redhat.com/show_bug.cgi?id=1675684 https://bugzilla.redhat.com/show_bug.cgi?id=1675784 https://bugzilla.redhat.com/show_bug.cgi?id=1675892 https://bugzilla.redhat.com/show_bug.cgi?id=1675899 https://bugzilla.redhat.com/show_bug.cgi?id=1675918 https://bugzilla.redhat.com/show_bug.cgi?id=1676017 https://bugzilla.redhat.com/show_bug.cgi?id=1676164 https://bugzilla.redhat.com/show_bug.cgi?id=1676181 https://bugzilla.redhat.com/show_bug.cgi?id=1676205 https://bugzilla.redhat.com/show_bug.cgi?id=1679955 ftbfs_weekly_reminder.py had an error condition Look in /tmp/ftbfs_reminder.8AGrhM for more info
```
Here is the script ``` #!/bin/bash # A weekly cron job which actually does the items needed. TMPDIR=`mktemp -d /tmp/ftbfs_reminder.XXXXXX` GITREPO=https://pagure.io/releng.git SCRIPT=ftbfs_weekly_reminder.py if [ $? -eq 0 ]; then cd ${TMPDIR} git clone ${GITREPO} if [ $? -eq 0 ]; then cd releng/scripts ./${SCRIPT} if [ $? -ne 0 ]; then echo "${SCRIPT} had an error condition" echo "Look in ${TMPDIR} for more info" # Do not clean up trash exit 1 fi else echo "Unable to clone ${GITREPO}" echo "Look in ${TMPDIR} for more info" # Do not clean up trash exit 1 fi cd /tmp/ rm -rf $TMPDIR else echo "Unable to create ${TMPDIR}" exit 1 fi
``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: ``
You can't ask Marek Mahut mmahut@redhat.com because that account is disabled.
It seems that the script needs to distinguish temporary BZ nonsense failures against permanent ones :( ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` do you happen to have /tmp/ftbfs_reminder.8AGrhM ? the script loggs the exceptions. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` correction, the exception is logged in bugzilla.log
``` Traceback (most recent call last): File "ftbfs_weekly_reminder.py", line 90, in send_reminder bzapi.update_bugs([bug.id], update) File "/usr/lib/python3.7/site-packages/bugzilla/base.py", line 1321, in update_bugs return self._proxy.Bug.update(tmp) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__ return self.__send(self.__name, args) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 104, in _ServerProxy__request self)._ServerProxy__request(methodname, params) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request verbose=self.__verbose File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 201, in request return self._request_helper(url, request_body) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 179, in _request_helper return self.parse_response(response) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 154, in parse_response return unmarshaller.close() File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault 32000: "You can't ask Martin Preisler mpreisle@redhat.com because that account is disabled."> ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` See https://pagure.io/releng/pull-request/8480 - it now attempts not to add needinfo if it breaks. The script output is also useful in case we want to see what redhat bugzilla accounts are gone. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Closed as Fixed by churchyard.
churchyard added a new comment to an issue you are following: `` Thanks @smooge @mohanboddu ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` https://pagure.io/releng/issue/8522 is the request to retire FTBFS packages according to the policy ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Open by churchyard.
churchyard added a new comment to an issue you are following: `` I have a feeling that this stopped happening again. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` To clarify, the comments happened on 2019-07-02 and than stopped. No comments a week later, no comments 2 weeks later either. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
smooge added a new comment to an issue you are following: `` I made a spalling mistake in the cron job so it was not able to run. I have corrected the cron job and run the script by hand. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Closed as Fixed by smooge.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been updated to: Open by smooge.
smooge added a new comment to an issue you are following: `` Script failed to run: ``` [root@compose-x86-01 ~][PROD]# su -s /bin/sh ftbfs sh-5.0$ sh -x /usr/local/bin/ftbfs.sh ++ mktemp -d /tmp/ftbfs_reminder.XXXXXX + TMPDIR=/tmp/ftbfs_reminder.jutVAp + GITREPO=https://pagure.io/releng.git + SCRIPT=ftbfs_weekly_reminder.py + '[' 0 -eq 0 ']' + cd /tmp/ftbfs_reminder.jutVAp + git clone https://pagure.io/releng.git Cloning into 'releng'... remote: Counting objects: 7902, done. remote: Compressing objects: 100% (3401/3401), done. remote: Total 7902 (delta 4449), reused 7835 (delta 4405) Receiving objects: 100% (7902/7902), 1.36 MiB | 741.00 KiB/s, done. Resolving deltas: 100% (4449/4449), done. + '[' 0 -eq 0 ']' + cd releng/scripts + ./ftbfs_weekly_reminder.py Gathering bugz, this can take a while... There are 681 NEW bugz, will send a reminder Traceback (most recent call last): File "./ftbfs_weekly_reminder.py", line 126, in <module> if ignore: NameError: name 'ignore' is not defined + '[' 1 -ne 0 ']' + echo 'ftbfs_weekly_reminder.py had an error condition' ftbfs_weekly_reminder.py had an error condition + echo 'Look in /tmp/ftbfs_reminder.jutVAp for more info' Look in /tmp/ftbfs_reminder.jutVAp for more info + exit 1
``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
churchyard added a new comment to an issue you are following: `` https://pagure.io/releng/pull-request/8560 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7718
rel-eng@lists.fedoraproject.org
-
Kevin Fenzi -
Miro Hrončok
-
Mohan Boddu
-
Stephen J Smoogen
-
Suzanne Yeghiayan
-
Till Maas