From: Herton R. Krzesinski <herton(a)redhat.com&gt; redhat: use tags from git notes for zstream to generate changelog This is forwarded ported from current rhel-9/centos9 tree. Due lots of recent changes in the ARK tree regarding Makefile and variables, some changes had to be done to this: dropped variable changes/additions to redhat/Makefile and redhat/genspec.sh, since those now are exported from the Makefile; for the same reason, the usage of ZSTREAM_FLAG variable was changed to the exported __ZSTREAM variable from the Makefile. From: Jan Stancek <jstancek(a)redhat.com&gt; Bugzilla: INTERNAL Upstream Status: RHEL only Conflicts: find_bz_in_line() is copied from rhel-8, because rhel-9 version didn't support BZ id without URL Forward-port of rhel-8 patch: commit cc89787cb88591e92c73a2b4ee432d50f36c5108 Author: Frantisek Hrbata <fhrbata(a)redhat.com&gt; Date: Tue Mar 2 13:56:21 2021 +0100 redhat: use tags from git notes for zstream to generate changelog Bugzilla: INTERNAL Upstream Status: RHEL only Tested: make rh-release for main and 8.3 branches and checked the generated changelog v2: - Use maj.min* notes ref for zstream. Previously only maj.min ref was used. This change allows kernel-rt to use their notes along with the zstream ones for rt specific patches. Previously for zstream we amended commit messages while applying patches from patchwork and we added tags: Bugzilla: ybzs, Z-Bugzilla: zbzs and CVE: cves if necessary. These were used by the genspec.sh to generate spec changelog for zstreams. Keeping the same approach with the gitlab workflow would mean to rebase commits in merge requests for zstream, which is something we probably don't want in the merge oriented workflow. Other approach would be to require these tags in the merge request commits. This would be fine for MRs created by maintainers, but could be confusing for others while creating zstream specific MRs. This would also require changes in bugzilla webhook. To avoid any additional requirements or changes in genlog.py, let's just use git notes to store zstream tags. The change is that zstream maintainer will not amend commit messages, but will be adding the required tags to git notes. With this change we will feed genlog.py with git notes instead of commit messages for zstreams. There should be no change for ystream. Each zstream is expected to have its own ref for notes based on the major and minor version. For example refs/notes/8.3 for 8.3.z. It's expected that maintainer makes sure the notes are presented and up-to-date. Fetching them automatically is IMHO not safe, because we cannot know the context. The format in git notes should be the same as expected by genlog.py. Bugzilla: ybzs Z-Bugzilla: zbzs CVE: cves Signed-off-by: Frantisek Hrbata <fhrbata(a)redhat.com&gt; Signed-off-by: Jan Stancek <jstancek(a)redhat.com&gt; Signed-off-by: Herton R. Krzesinski <herton(a)redhat.com&gt; diff --git a/redhat/genlog.py b/redhat/genlog.py index blahblah..blahblah 100755 --- a/redhat/genlog.py +++ b/redhat/genlog.py @@ -17,16 +17,18 @@ import sys def find_bz_in_line(line, prefix): """Return bug number from properly formated Bugzilla: line.""" - # BZs must begin with '{prefix}: ' and contain a complete BZ URL - line = line.rstrip() - pattern = prefix + r': http(s)?://bugzilla\.redhat\.com/(show_bug\.cgi\?id=)?(?P<bug>\d{4,8})$' - bznum_re = re.compile(pattern) - bzn = set() - match = bznum_re.match(line) - if match: - bzn.add(match.group('bug')) - return bzn - + # BZs must begin with '{prefix}: ' and contain a complete BZ URL or id + _bugs = set() + if not line.startswith(f"{prefix}: "): + return _bugs + bznum_re = re.compile(r'(?P<bug_ids> \d{4,8})|' + r'( http(s)?://bugzilla\.redhat\.com/(show_bug\.cgi\?id=)?(?P<url_bugs>\d{4,8}))') + for match in bznum_re.finditer(line[len(f"{prefix}:"):]): + for group in [ 'bug_ids', 'url_bugs' ]: + if match.group(group): + bid = match.group(group).strip() + _bugs.add(bid) + return _bugs def find_cve_in_line(line): """Return cve number from properly formated CVE: line.""" diff --git a/redhat/genspec.sh b/redhat/genspec.sh index blahblah..blahblah 100755 --- a/redhat/genspec.sh +++ b/redhat/genspec.sh @@ -87,6 +87,13 @@ done # test changes. [ -n "$RHSELFTESTDATA" ] && exit 0 +GIT_FORMAT="--format=- %s (%an)%n%b" +GIT_NOTES="" +if [ "$__ZSTREAM" != "no" ]; then + GIT_FORMAT="--format=- %s (%an)%n%N" + GIT_NOTES="--notes=refs/notes/${RHEL_MAJOR}.${RHEL_MINOR}*" +fi + echo > "$clogf" lasttag=$(git rev-list --first-parent --grep="^\[redhat\] kernel-${RPMKVERSION}.${RPMKPATCHLEVEL}" --max-count=1 HEAD) @@ -103,7 +110,7 @@ fi echo "Gathering new log entries since $lasttag" # master is expected to track mainline. -git log --topo-order --reverse --no-merges -z --format="- %s (%an)%n%b" \ +git log --topo-order --reverse --no-merges -z $GIT_NOTES "$GIT_FORMAT" \ ^"${UPSTREAM}" "$lasttag".. -- ':!/redhat/rhdocs' | "${0%/*}"/genlog.py >> "$clogf" grep -v "tagging $RPMVERSION" "$clogf" > "$clogf.stripped" -- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1764

From: Herton R. Krzesinski <herton(a)redhat.com&gt; redhat: generate distgit changelog in genspec.sh as well This is forwarded ported from current rhel-9/centos9 tree. Due lots of recent changes in the ARK tree regarding Makefile and variables, some changes had to be done to this: dropped variable changes/additions to redhat/Makefile, redhat/genspec.sh and redhat/scripts/rh-dist-git.sh, since those now are exported from the Makefile; for the same reason, the usage of some variables is changed from what was used as arguments before, to use the exported values from the Makefile. Another difference is also that the selftest data conditional made the changelog not use the stripped and updated changelog, this is adjusted/fixed here in this version. From: Jan Stancek <jstancek(a)redhat.com&gt; Bugzilla: INTERNAL Upstream Status: RHEL only Forward port of rhel-8 commit: commit 924856ea3952cc52781d252be70f45c1231ee8ba Author: Jan Stancek <jstancek(a)redhat.com&gt; Date: Wed Mar 9 11:15:16 2022 +0100 redhat: generate distgit changelog in genspec.sh as well Bugzilla: INTERNAL Upstream Status: RHEL only To break our dependence on ZSTREAM flag, make genlog.py generate also dist-git Resolves lines. Whether Resolves line uses ZBZs or YBZs, depends on whether genlog.py finds any ZBZs. After this change kernel.changelog* will start to contain also Resolves lines, which we need for dist-git changelog. For RPM changelog, we strip Resolves lines and use as before. Signed-off-by: Jan Stancek <jstancek(a)redhat.com&gt; Signed-off-by: Jan Stancek <jstancek(a)redhat.com&gt; Signed-off-by: Herton R. Krzesinski <herton(a)redhat.com&gt; diff --git a/redhat/genlog.py b/redhat/genlog.py index blahblah..blahblah 100755 --- a/redhat/genlog.py +++ b/redhat/genlog.py @@ -74,6 +74,8 @@ def parse_commit(commit): if __name__ == "__main__": + all_bzs = [] + all_zbzs = [] commits = sys.stdin.read().split('\0') for c in commits: if not c: @@ -84,11 +86,25 @@ if __name__ == "__main__": entry += " [" if zbugs: entry += " ".join(zbugs) + all_zbzs.extend(zbugs) if bugs and zbugs: entry += " " if bugs: entry += " ".join(bugs) + all_bzs.extend(bugs) entry += "]" if cves: entry += " {" + " ".join(cves) + "}" print(entry) + + resolved_bzs = [] + for bzid in (all_zbzs if all_zbzs else all_bzs): + if not bzid in resolved_bzs: + resolved_bzs.append(bzid) + print("Resolves: ", end="") + for i, bzid in enumerate(resolved_bzs): + if i: + print(", ", end="") + print(f"rhbz#{bzid}", end="") + print("\n") + diff --git a/redhat/genspec.sh b/redhat/genspec.sh index blahblah..blahblah 100755 --- a/redhat/genspec.sh +++ b/redhat/genspec.sh @@ -57,8 +57,6 @@ fi test -f "$SOURCES/$SPECFILE" && sed -i -e " - /%%CHANGELOG%%/r $SOURCES/$CHANGELOG - /%%CHANGELOG%%/d s/%%BUILDID%%/$BUILDID_DEFINE/ s/%%RPMKVERSION%%/$RPMKVERSION/ s/%%RPMKPATCHLEVEL%%/$RPMKPATCHLEVEL/ @@ -85,13 +83,17 @@ done # The self-test data doesn't currently have tests for the changelog or patch file, so the # rest of the script can be ignored. See redhat/Makefile setup-source target for related # test changes. -[ -n "$RHSELFTESTDATA" ] && exit 0 +if [ -n "$RHSELFTESTDATA" ]; then + test -f "$SOURCES/$SPECFILE" && + sed -i -e " + /%%CHANGELOG%%/r $SOURCES/$CHANGELOG + /%%CHANGELOG%%/d" "$SOURCES/$SPECFILE" + exit 0 +fi GIT_FORMAT="--format=- %s (%an)%n%N%n^^^NOTES-END^^^%n%b" GIT_NOTES="--notes=refs/notes/${RHEL_MAJOR}.${RHEL_MINOR}*" -echo > "$clogf" - lasttag=$(git rev-list --first-parent --grep="^\[redhat\] kernel-${RPMKVERSION}.${RPMKPATCHLEVEL}" --max-count=1 HEAD) # if we didn't find the proper tag, assume this is the first release if [[ -z $lasttag ]]; then @@ -106,11 +108,13 @@ fi echo "Gathering new log entries since $lasttag" # master is expected to track mainline. -git log --topo-order --reverse --no-merges -z $GIT_NOTES "$GIT_FORMAT" \ - ^"${UPSTREAM}" "$lasttag".. -- ':!/redhat/rhdocs' | "${0%/*}"/genlog.py >> "$clogf" +cname="$(git var GIT_COMMITTER_IDENT |sed 's/>.*/>/')" +cdate="$(LC_ALL=C date +"%a %b %d %Y")" +cversion="[$RPMVERSION]"; +echo "* $cdate $cname $cversion" > "$clogf" -grep -v "tagging $RPMVERSION" "$clogf" > "$clogf.stripped" -cp "$clogf.stripped" "$clogf" +git log --topo-order --no-merges -z $GIT_NOTES "$GIT_FORMAT" \ + ^"${UPSTREAM}" "$lasttag".. -- ':!/redhat/rhdocs' | "${0%/*}"/genlog.py >> "$clogf" if [ "$HIDE_REDHAT" = "1" ]; then grep -v -e "^- \[redhat\]" "$clogf" | @@ -144,25 +148,25 @@ if [ -n "$(git log --oneline --first-parent --grep="Merge ark patches" "$lasttag echo "- Merge ark-patches" >> "$clogf" fi -LENGTH=$(wc -l "$clogf" | awk '{print $1}') - -#the changelog was created in reverse order -#also remove the blank on top, if it exists -#left by the 'print version\n' logic above -cname="$(git var GIT_COMMITTER_IDENT |sed 's/>.*/>/')" -cdate="$(LC_ALL=C date +"%a %b %d %Y")" -cversion="[$RPMVERSION]"; -tac "$clogf" | sed "1{/^$/d; /^- /i\ -* $cdate $cname $cversion - }" > "$clogf.rev" - +# during rh-dist-git genspec runs again and generates empty changelog +# create empty file to avoid adding extra header to changelog +LENGTH=$(grep "^-" $clogf | wc -l | awk '{print $1}') if [ "$LENGTH" = 0 ]; then - rm -f "$clogf.rev"; touch "$clogf.rev" + rm -f $clogf + touch $clogf fi -cat "$clogf.rev" "$SOURCES/$CHANGELOG" > "$clogf.full" +cat "$clogf" "$SOURCES/$CHANGELOG" > "$clogf.full" mv -f "$clogf.full" "$SOURCES/$CHANGELOG" +# genlog.py generates Resolves lines as well, strip these from RPM changelog +cat "$SOURCES/$CHANGELOG" | grep -v -e "^Resolves: " > $clogf.stripped + +test -f "$SOURCES/$SPECFILE" && + sed -i -e " + /%%CHANGELOG%%/r $clogf.stripped + /%%CHANGELOG%%/d" "$SOURCES/$SPECFILE" + echo "MARKER is $MARKER" if [ "$SINGLE_TARBALL" = 0 ]; then @@ -179,4 +183,4 @@ else touch "${SOURCES}/patch-${RPMKVERSION}.${RPMKPATCHLEVEL}"-redhat.patch fi -rm -f "$clogf"{,.rev,.stripped}; +rm -f "$clogf"{,.stripped}; diff --git a/redhat/scripts/create_distgit_changelog.sh b/redhat/scripts/create_distgit_changelog.sh deleted file mode 100755 index blahblah..blahblah 0 --- a/redhat/scripts/create_distgit_changelog.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/bin/bash - -# Outputs a ready to use commit message for dist-git -# $1: spec file -# $2: zstream flag - -spec=$1; -zstream_flag=$2; -package_name=$3; -tmp=$(mktemp); - -function die -{ - echo "Error: $1" >&2; - rm -f "$tmp"; - exit 1; -} - -if [ ! -f "$spec" ]; then - die "$0 <spec file>"; -fi - -# this expression looks for the beginning of the changelog and extracts the -# first entry -sed -n -e '0,/%changelog/d; :abc; s/^$//; t end; p; d; N; b abc; :end; q;' "$spec" > "$tmp"; -if [ ! -s "$tmp" ]; then - die "Unable to extract the changelog"; -fi - -# getting the version from the first line -version=$(head -n 1 "$tmp" | sed -e 's/.*\[\(.*\)\]/\1/'); -if [ -z "$version" ]; then - die "Unable to get version/release"; -fi - -# extracting the BZs to create the "Resolves" line -if [ "$zstream_flag" == "no" ]; then - bzs=$(grep ^- "$tmp" | - sed -n -e "s/.*\[\([0-9 ]\+\)\].*/\1/p") -else - bzs=$(awk '/^-/ { - if(match($0, /\[([0-9]+ [0-9]+ )*[0-9]+ [0-9]+\]/)) { - n = split(substr($0, RSTART + 1, RLENGTH - 2), bzs) - for(i = 1; i <= n/2; i++) - print bzs[i] - } - }' "$tmp") -fi - -echo "$bzs" | - tr ' ' '\n' | - sort -u | - tr '\n' ',' | - sed -e "s/^/Resolves: rhbz#/; s/,\$//; s/,/, rhbz#/g;" >> "$tmp"; - -echo -e "${package_name}-${version}\n" -cat "$tmp"; -echo; -rm -f "$tmp"; diff --git a/redhat/scripts/rh-dist-git.sh b/redhat/scripts/rh-dist-git.sh index blahblah..blahblah 100755 --- a/redhat/scripts/rh-dist-git.sh +++ b/redhat/scripts/rh-dist-git.sh @@ -60,8 +60,10 @@ echo "Creating diff for review ($tmpdir/diff) and changelog" # differences were found diff -X "$REDHAT"/git/dontdiff -upr "$tmpdir/$PACKAGE_NAME" "$REDHAT"/rpm/SOURCES/ > "$tmpdir"/diff; # creating the changelog file -"$REDHAT"/scripts/create_distgit_changelog.sh "$REDHAT/rpm/SOURCES/$PACKAGE_NAME".spec \ - "$__ZSTREAM" "$PACKAGE_NAME" >"$tmpdir"/changelog + +# changelog has been created by genspec.sh, including Resolves line, just copy it here +echo -e "${PACKAGE_NAME}-${RPMVERSION}\n" > $tmpdir/changelog +awk '1;/^Resolves: /{exit};' $REDHAT/$CHANGELOG >> $tmpdir/changelog # all done echo "$tmpdir" -- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1764

From: Herton R. Krzesinski <herton(a)redhat.com&gt; redhat: add zstream switch for zstream release numbering Bugzilla: INTERNAL Upstream Status: RHEL only When we fork from CentOS into RHEL stable releases, we need to switch the release number to use zstream numbering, to not conflict with future centos release numbers. We use the ZSTREAM=yes value in that case, keep it set to no by default until we need it. Signed-off-by: Herton R. Krzesinski <herton(a)redhat.com&gt; diff --git a/Makefile.rhelver b/Makefile.rhelver index blahblah..blahblah 100644 --- a/Makefile.rhelver +++ b/Makefile.rhelver @@ -14,6 +14,28 @@ RHEL_MINOR = 99 # Do not trim this comment. RHEL_RELEASE = 34 +# +# ZSTREAM +# ------- +# +# This variable controls whether we use zstream numbering or not for the +# package release. The zstream release keeps the build number of the last +# build done for ystream for the Beta milestone, and increments a second +# number for each build. The third number is used for branched builds +# (eg.: for builds with security fixes or hot fixes done outside of the +# batch release process). +# +# For example, with ZSTREAM unset or set to "no", all builds will contain +# a release with only the build number, eg.: kernel-<kernel version>-X.el*, +# where X is the build number. With ZSTREAM set to "yes", we will have +# builds with kernel-<kernel version>-X.Y.Z.el*, where X is the last +# RHEL_RELEASE number before ZSTREAM flag was set to yes, Y will now be the +# build number and Z will always be 1 except if you're doing a branched build +# (when you give RHDISTGIT_BRANCH on the command line, in which case the Z +# number will be incremented instead of the Y). +# +ZSTREAM ?= no + # # Early y+1 numbering # -------------------- -- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1764