Delayed openQA test execution
by Adam Williamson
Hey folks! Wanted to send a note out for any maintainers who may be
waiting on openQA test results for critical path updates etc. Tests are
currently taking much longer than usual to be completed because there's
a large backlog. The Rawhide mass rebuild, plus the three critical bugs
(python-ptyprocess, systemd, and mesa) that I posted about yesterday,
caused 2-3 days' worth of Rawhide update tests to fail. After finally
sorting out those problems enough that the tests now pass, I
rescheduled that whole backlog, which turned out to be over 2000 tests.
The system is working its way through the backlog but it'll take a
while (I expect it should be clear by end of day today or so, it
depends a bit on how many updates are created in the mean time), and
until that gets done, it may take much longer for the tests for any
given update to be completed after the update is created.
I do apologize for this; with hindsight it might've been better to try
and hack up a way to reduce the priority on the backlogged Rawhide
update tests so tests for stable release updates ran first, but I
didn't think of that yesterday.
If there are any urgent security or critical bug fix updates for stable
releases which are waiting on testing and really need it run ASAP, let
me know and I can manually bump the priority of those tests so they run
sooner.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 3 months
Fedora CI outage - Testing Farm requests stuck in queued state
by Miroslav Vadkerti
Hi,
due to a deployment error Testing Farm was not processing correctly
requests from yesterday.
The issue should be resolved now, but all affected jobs might need a
restart:
* Fedora CI - restart affected Jenkins jobs
* Zuul Fedora CI - comment with "recheck" in merge requests
* Packit - comment with "/packit retest" in pull requests
Best regards,
/M
--
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE
IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252
Remote Czech Republic :: Red Hat Czech s.r.o
1 year, 3 months
Has anything changed recently wrt networking or DNS on the Testing
Farm?
by Miro Hrončok
Hello.
Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing
farm fail.
The same tests work on Rawhide and worked on Fedora 37/36 previously --
unfortunately this could have been a while ago, maybe even couple weeks, due to
the holidays.
See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
The test failures look like this:
======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind
======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind
======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind
======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind
----------------------------------------------------------------------
Ran 101 tests in 1.792s
All teh failed tests are related to either multiprocessing or asyncio.
All of them are either:
AssertionError: OSError not raised by bind
Or:
AssertionError: OSError not raised by Listener
The same tests pass during the Koji build when he package builds (unless they
are skipped due to lack of network, which I need to investigate).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 year, 3 months
Fedora CI / Zuul CI instability today
by Miroslav Vadkerti
Hi,
Unfortunately, we run into some troubles today and it has affected a ton of
jobs.
If you see your Fedora CI jobs failing, please restart them.
If you run into unexpected issues for Zuul based CI, please recheck.
We apologize for the inconvenience.
Testing Farm Team
--
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE
IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252
Remote Czech Republic :: Red Hat Czech s.r.o
1 year, 3 months
