On 03/18/2018 05:03 AM, Fabio Valentini wrote: > > I've got two updates sitting in F26 and F27 updates-testing: > > https://bodhi.fedoraproject.org/updates/FEDORA-2018-c455a245b0 > <https://bodhi.fedoraproject.org/updates/FEDORA-2018-c455a245b0> > https://bodhi.fedoraproject.org/updates/FEDORA-2018-4d0e13ff51 > > I can't push either of them to batched or stable (despite them being in > -testing for over 10 days now), because "no test results found". > > Which is really annoying and the error message is ... rather unhelpful, > since it just tells me "I can't do this _shrug_", but not how to fix it. > > I guess this is caused by a bug in one of the involved components? > Where can/should I report this issue? > How can I fix this (if I can do it myself), or does somebody with more > power have to step in to fix this? I've been told[1] to create waivers[2] but I have tried to do that today and I still cannot push my update[3].

I'd really prefer the test gates be disabled until the tooling is more complete. Right now the documentation is very spare and sometimes wrong as the tooling is in its early birthing stages and files are moving directories and new config options are added/removed.

On 03/18/2018 01:02 PM, Fabio Valentini wrote: > > I've looked at waiverdb-cli too, but since no tests seem to have run > at all, it looks like the wrong tool for the job: > I don't want to push an update despite a failed test, I want to push > my update despite no test data being available ... Randy said the tests refresh every 6 hours and/or every time the update is edited. Neither seemed to have occurred for you.

On Sun, 18 Mar 2018 20:25:31 +0100 Fabio Valentini <decathorpe(a)gmail.com&gt; wrote: > On Sun, Mar 18, 2018 at 7:13 PM, Michael Cronenworth > <mike(a)cchtml.com&gt; wrote: > > On 03/18/2018 01:02 PM, Fabio Valentini wrote: > >> > >> I've looked at waiverdb-cli too, but since no tests seem to have > >> run at all, it looks like the wrong tool for the job: > >> I don't want to push an update despite a failed test, I want to > >> push my update despite no test data being available ... > > > > > > Randy said the tests refresh every 6 hours and/or every time the > > update is edited. Neither seemed to have occurred for you. > > Exactly. The "no test results found" status in bodhi hasn't been > refreshed in over 10 days now. > > Bodhi also displays that all these tests were successful, bit still > blocks the update because "no test results found", which is > obviously just wrong. > > A manual lookup in resultdb shows me that the tests have in fact > been run and have all passed: https://bodhi.fedoraproject.org/updates/FEDORA-2018-200708ae05 is in the same situation, all tests are green, but "no test results found" is reported. It's not very user friendly ...

On Mon, Mar 19, 2018 at 03:41:15PM +0100, Dan Horák wrote: > On Mon, 19 Mar 2018 14:06:56 +0100 > Dan Horák <dan(a)danny.cz&gt; wrote: > >> On Sun, 18 Mar 2018 20:25:31 +0100 >> Fabio Valentini <decathorpe(a)gmail.com&gt; wrote: >> >>> On Sun, Mar 18, 2018 at 7:13 PM, Michael Cronenworth >>> <mike(a)cchtml.com&gt; wrote: >>>> On 03/18/2018 01:02 PM, Fabio Valentini wrote: >>>>> >>>>> I've looked at waiverdb-cli too, but since no tests seem to have >>>>> run at all, it looks like the wrong tool for the job: >>>>> I don't want to push an update despite a failed test, I want to >>>>> push my update despite no test data being available ... >>>> >>>> >>>> Randy said the tests refresh every 6 hours and/or every time the >>>> update is edited. Neither seemed to have occurred for you. >>> >>> Exactly. The "no test results found" status in bodhi hasn't been >>> refreshed in over 10 days now. >>> >>> Bodhi also displays that all these tests were successful, bit still >>> blocks the update because "no test results found", which is >>> obviously just wrong. >>> >>> A manual lookup in resultdb shows me that the tests have in fact >>> been run and have all passed: >> >> https://bodhi.fedoraproject.org/updates/FEDORA-2018-200708ae05 is in >> the same situation, all tests are green, but "no test results found" >> is reported. It's not very user friendly ... > > and https://bodhi.fedoraproject.org/updates/FEDORA-2018-71350d90a7 is > even more interesting with "The update can not be pushed: 1 of 2 > required tests not found", but the listed tests are again all green. No > idea what's missing from the output. All the tests can be green if the "important" ones are missing, they don't show :( The important ones are the ones defined in the policy that gates packages and are listed here: https://fedoraproject.org/wiki/CI/gating_updates waiverdb-cli should now support waiving missing results, I'm double-checking it and see if we can document it at: https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_fro... next to the other examples. Pierre

On Mon, 2018-03-19 at 15:57 +0100, Pierre-Yves Chibon wrote: > On Mon, Mar 19, 2018 at 03:41:15PM +0100, Dan Horák wrote: > > On Mon, 19 Mar 2018 14:06:56 +0100 > > Dan Horák <dan(a)danny.cz&gt; wrote: > > > > > On Sun, 18 Mar 2018 20:25:31 +0100 > > > Fabio Valentini <decathorpe(a)gmail.com&gt; wrote: > > > > > > > On Sun, Mar 18, 2018 at 7:13 PM, Michael Cronenworth > > > > <mike(a)cchtml.com&gt; wrote: > > > > > On 03/18/2018 01:02 PM, Fabio Valentini wrote: > > > > > > > > > > > > I've looked at waiverdb-cli too, but since no tests seem to have > > > > > > run at all, it looks like the wrong tool for the job: > > > > > > I don't want to push an update despite a failed test, I want to > > > > > > push my update despite no test data being available ... > > > > > > > > > > > > > > > Randy said the tests refresh every 6 hours and/or every time the > > > > > update is edited. Neither seemed to have occurred for you. > > > > > > > > Exactly. The "no test results found" status in bodhi hasn't been > > > > refreshed in over 10 days now. > > > > > > > > Bodhi also displays that all these tests were successful, bit still > > > > blocks the update because "no test results found", which is > > > > obviously just wrong. > > > > > > > > A manual lookup in resultdb shows me that the tests have in fact > > > > been run and have all passed: > > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2018-200708ae05 is in > > > the same situation, all tests are green, but "no test results found" > > > is reported. It's not very user friendly ... > > > > and https://bodhi.fedoraproject.org/updates/FEDORA-2018-71350d90a7 is > > even more interesting with "The update can not be pushed: 1 of 2 > > required tests not found", but the listed tests are again all green. No > > idea what's missing from the output. > > All the tests can be green if the "important" ones are missing, they don't show > :( > > The important ones are the ones defined in the policy that gates packages and > are listed here: https://fedoraproject.org/wiki/CI/gating_updates > > waiverdb-cli should now support waiving missing results, I'm double-checking it > and see if we can document it at: > https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_fro... > next to the other examples. It is also still the case that unpushing and re-pushing the update should re-trigger the tests in Taskotron, at the cost of re-setting the karma and 'wait time' clocks for the update (so you'll need to either get positive karma or wait 7 or 14 days before being able to push it, from the time of the re-push). One obvious easy win here would be to change the "No test results found" text, as it's clearly confusing. It could be something like "No results found for blocking tests", perhaps? We could even give Bodhi the ability to list the names of the 'expected' blocking tests, and have the text show that somehow, whether as a hyperlink or perhaps just as a mouseover or something?

On Mon, Mar 19, 2018 at 03:41:15PM +0100, Dan Horák wrote: > On Mon, 19 Mar 2018 14:06:56 +0100 > Dan Horák <dan(a)danny.cz&gt; wrote: > > > On Sun, 18 Mar 2018 20:25:31 +0100 > > Fabio Valentini <decathorpe(a)gmail.com&gt; wrote: > > > > > On Sun, Mar 18, 2018 at 7:13 PM, Michael Cronenworth > > > <mike(a)cchtml.com&gt; wrote: > > > > On 03/18/2018 01:02 PM, Fabio Valentini wrote: > > > >> > > > >> I've looked at waiverdb-cli too, but since no tests seem to > > > >> have run at all, it looks like the wrong tool for the job: > > > >> I don't want to push an update despite a failed test, I want > > > >> to push my update despite no test data being available ... > > > > > > > > > > > > Randy said the tests refresh every 6 hours and/or every time > > > > the update is edited. Neither seemed to have occurred for you. > > > > > > Exactly. The "no test results found" status in bodhi hasn't been > > > refreshed in over 10 days now. > > > > > > Bodhi also displays that all these tests were successful, bit > > > still blocks the update because "no test results found", which > > > is obviously just wrong. > > > > > > A manual lookup in resultdb shows me that the tests have in fact > > > been run and have all passed: > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2018-200708ae05 is > > in the same situation, all tests are green, but "no test results > > found" is reported. It's not very user friendly ... > > and https://bodhi.fedoraproject.org/updates/FEDORA-2018-71350d90a7 > is even more interesting with "The update can not be pushed: 1 of 2 > required tests not found", but the listed tests are again all > green. No idea what's missing from the output. All the tests can be green if the "important" ones are missing, they don't show :(

The important ones are the ones defined in the policy that gates packages and are listed here: https://fedoraproject.org/wiki/CI/gating_updates

waiverdb-cli should now support waiving missing results, I'm double-checking it and see if we can document it at: https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_fro... next to the other examples.

The important ones are the ones defined in the policy that gates packages and are listed here: https://fedoraproject.org/wiki/CI/gating_updates waiverdb-cli should now support waiving missing results, I'm double-checking it and see if we can document it at: https://fedoraproject.org/wiki/Package_update_HOWTO# Handling_feedback_from_automated_tests next to the other examples.

waiverdb-cli should now support waiving missing results, I'm double-checking it and see if we can document it at: https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_fro... next to the other examples.

On Wed, Mar 21, 2018 at 11:28 AM, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote: > On Mon, Mar 19, 2018 at 03:57:19PM +0100, Pierre-Yves Chibon wrote: >> waiverdb-cli should now support waiving missing results, I'm

>> and see if we can document it at: >>

>> next to the other examples. > > > The documentation on how to waive tests or missing tests have been

> url (above). > > Let us know if you need more or face issues with it! I've just "waived" the missing test results which blocked my 2 packages, and it worked, I could push them to stable now.

I don't know if there is a way to know this right now, other than waiting up to 6 hours. Bodhi does actually have a UI to submit waivers (to avoid the CLI and the confusing data hunting), but it needs an auth token configured to allow it to talk to waiverdb[1] (Bodhi hides the waiverdb UI when it doesn't have a token to talk to waiverdb). Fedora infrastructure is currently in a freeze, so we cannot add that token at this time. The freeze should lift on Wednesday.

Infra is still in a freeze (we will probably lift it sometime today). I can't personally take care of the ticket because I don't have access to create the necessary token. I also suspect the admins will be too busy to do this this week, because they are doing a mass update/reboot cycle, though that's just speculation on my part.

I apologize. I think it was a mistake to enable gating in Bodhi when things were this rough. Some of it we didn't know about when the enabling decision was made. Sometimes I consider raising the issue with FESCo to see if we should turn it off until some criteria are met. I'm not sure what the full list of criteria would be, but having this button could be a start. If you feel strongly about it, feel free to submit a FESCo ticket to see what they think. I believe the enablement might have been a Fedora Council objective, but I'm not sure.

I deployed bodhi-3.7.0[0] to production today. It has some improvements in the test gating area, though unfortunately not all the goals have been met. I wrote a bit more about it in this FESCo ticket: https://pagure.io/fesco/issue/1872#comment-511227 [0]https://bodhi.fedoraproject.org/docs/user/release_notes.html#v3-7-0