Adam Litke has uploaded a new change for review.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
tests: Avoid ResourceManagerTests failures when running parallel tests
In the Jenkins test environment, tests are run in parallel. ResourceManagerTests.testStressTest spawns lots of threads and can easily exceed the system limit when multiple copies are active. To prevent failures in this case, allow the test to fallback to single threaded mode when more threads cannot be created. This does mean that the test may be less rigorous when run in parallel but I don't see a better way to work around this false positive issue. The test will still be full strength when run alone.
Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Signed-off-by: Adam Litke agl@us.ibm.com --- M tests/resourceManagerTests.py 1 file changed, 17 insertions(+), 13 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/34/12034/1
diff --git a/tests/resourceManagerTests.py b/tests/resourceManagerTests.py index c873394..58de49e 100644 --- a/tests/resourceManagerTests.py +++ b/tests/resourceManagerTests.py @@ -637,6 +637,15 @@ res.release() threadLimit.release()
+ def tryThread(fn, *args): + t = threading.Thread(target=fn, args=args) + try: + t.start() + except: + fn(*args) + else: + threads.append(t) + manager = self.manager rnd = Random()
@@ -645,9 +654,8 @@
threads = [] for i in range(nthreads): - threadLimit.acquire() - threads.append(threading.Thread(target=register)) - threads[-1].start() + threadLimit.acquire() + tryThread(register)
while len(threads) > 0: for t in threads[:]: @@ -657,18 +665,14 @@ while len(resources) > 0: while len(resources) > 1: threadLimit.acquire() - threads.append( - threading.Thread(target=releaseShared, - args=[requests.pop(), - resources.pop()])) - threads[-1].start() + req = requests.pop() + res = resources.pop() + tryThread(releaseShared, req, res)
threadLimit.acquire() - threads.append( - threading.Thread(target=releaseUnknown, - args=[requests.pop(), - resources.pop()])) - threads[-1].start() + req = requests.pop() + res = resources.pop() + tryThread(releaseUnknown, req, res)
def tearDown(self): manager = self.manager
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/378/ (1/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1232/ (2/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1267/ (3/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1:
Build Successful
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1232/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1267/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/378/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Adam Litke has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1: Verified
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Saggi Mizrahi has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File tests/resourceManagerTests.py Line 640: def tryThread(fn, *args): Line 641: t = threading.Thread(target=fn, args=args) Line 642: try: Line 643: t.start() Line 644: except: I'd rather skip it then run it synchronously Line 645: fn(*args) Line 646: else: Line 647: threads.append(t) Line 648:
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/379/ (2/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1233/ (3/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1268/ (1/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Adam Litke has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2: Verified
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2:
Build Successful
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1233/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1268/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/379/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Dan Kenigsberg has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File tests/resourceManagerTests.py Line 643: def tryThread(fn, *args): Line 644: t = threading.Thread(target=fn, args=args) Line 645: try: Line 646: t.start() Line 647: except: it's only a test, but would you catch only the specific exception that we are expecting here? I hate to falsly skip the test if there's something else happening. Line 648: stop.set() Line 649: while len(resources) > 0: Line 650: res = resources.pop() Line 651: res.release()
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Saggi Mizrahi has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2: Looks good to me, but someone else must approve
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1274/ (2/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/385/ (1/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1239/ (3/3)
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Adam Litke has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 2: (1 inline comment)
.................................................... File tests/resourceManagerTests.py Line 643: def tryThread(fn, *args): Line 644: t = threading.Thread(target=fn, args=args) Line 645: try: Line 646: t.start() Line 647: except: Done Line 648: stop.set() Line 649: while len(resources) > 0: Line 650: res = resources.pop() Line 651: res.release()
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Adam Litke has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3: Verified
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3:
Build Successful
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1239/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1274/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/385/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Adam Litke has abandoned this change.
Change subject: tests: Avoid ResourceManagerTests failures when running parallel tests ......................................................................
Patch Set 3: Abandoned
Fixed in Id74717584f0c4753cec58cb1b701e7095a735924 by just skipping this test in the Jenkins environment
-- To view, visit http://gerrit.ovirt.org/12034 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: abandon Gerrit-Change-Id: Ibd18d683ea3b8f72d569a6151066179ef21d3d26 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Adam Litke agl@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Vinzenz Feenstra vfeenstr@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
vdsm-patches@lists.fedorahosted.org