From: Don Zickus dzickus@redhat.com
Check for kernel config git-push failures
An automated nightly job is run to merge the upstream tree and check for new configs to be set. The new configs are automatically added to the redhat/configs/pending-ark directory and pushed. A second script then comes along and walks through those new configs and converts them into automatically generated config MRs for review.
Those MRs periodically fail to git-push due to infrastructure issues. The script reports success and the maintainers never realize an MR failed to be created until a pending config audit is performed before the GA release.
Let's update the script to check for those failures and report them (by failing the script). The script failure leads to an email to the maintainer who can then re-run the script by hand to regenerate the MRs or fix issues manually.
Tested by
* committing dummy change to Makefile * git push -o merge_request.create dzickus dummy * committing second dummy change to Makefile * git push -o merge_request.create dzickus dummy (git push succeeds with following warning message)
remote: remote: ================================================================ remote: remote: WARNINGS: Error encountered with push options remote: 'merge_request.create' 'merge_request.target=os-build' remote: 'merge_request.remove_source_branch': Validate branches Another remote: open merge request already exists for this source branch: !2164 remote: remote: =============================================================== remote:
Ran script changes by hand to verify shell logic. Untested in cronjob.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-update-configs.sh b/redhat/scripts/ci/ark-update-configs.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-update-configs.sh +++ b/redhat/scripts/ci/ark-update-configs.sh @@ -67,6 +67,8 @@ else fi
if test -n "$DIST_PUSH"; then + tmpfile=".push-warnings" + echo "Pushing branch $(git branch --show-current) to $(git remote get-url gitlab)" git push gitlab HEAD
@@ -76,8 +78,16 @@ if test -n "$DIST_PUSH"; then -o merge_request.create \ -o merge_request.target="$BRANCH" \ -o merge_request.remove_source_branch \ - gitlab "$branch" + gitlab "$branch" 2>&1 | tee -a $tmpfile done + + # GitLab server side warnings do not fail git-push but leave verbose + # WARNING messages. Grep for those and consider it a script + # failure. Make sure all branches are pushed first as follow up + # git-pushes may succeed. + grep -q "remote:[ ]* WARNINGS" $tmpfile && RC=1 || RC=0 + rm $tmpfile + test $RC && exit $RC else printf " To push all the release artifacts, run:
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165
From: Don Zickus on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165#note_1188537...
@jmflinuxtx - I believe this should catch the failed config MRs during the cronjob.
From: Justin M. Forbes on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165#note_1203309...
So it seems this will now fail if we have no warnings.
``` No new configuration values exposed from merging master into os-build 232Pushing branch os-build to git@gitlab.com:cki-project/kernel-ark.git 233Everything up-to-date 234Pushing config update branches 235grep: .push-warnings: No such file or directory 236rm: cannot remove '.push-warnings': No such file or directory 237make[1]: *** [Makefile:734: dist-merge-upstream] Error 1 238make[1]: Leaving directory '/builds/53/kernel-ark/redhat' 239make: *** [makefile:12: dist-merge-upstream-push] Error 2 241 Cleaning up project directory and file based variables 00:01 243ERROR: Job failed: exit code 1 ```
From: Patrick Talbert on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165#note_1203367...
Afaict we just need to touch the `.push-warnings` file?
From: Don Zickus on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165#note_1203471...
ugh how dumb of me. I can create an MR that 'touch'es or 'rm -f'. Any preference @jmflinuxtx @ptalbert ?
From: Don Zickus on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2165#note_1203488...
created !2183 with 'touch' fix
kernel@lists.fedoraproject.org