11:48 a.m.
From: Prarit Bhargava on gitlab.com
Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
The dist-configs target takes an annoying long time.
Add parallel processing of configs in dist-configs.
Testing Info
------------
make dist-configs time 1m22.909s
make -j20 dist-configs time 11.801s
Removed redhat/configs/common/generic/x86/x86_64/CONFIG_ACPI_HOTPLUG_MEMORY
to test unset error.
Found unset config items in x86_64-debug, please set them to an
appropriate
value
CONFIG_ACPI_HOTPLUG_MEMORY=n
Found unset config items in x86_64-kgcov, please set them to an appropriate
value
CONFIG_ACPI_HOTPLUG_MEMORY=n
Found unset config items in x86_64, please set them to an appropriate value
CONFIG_ACPI_HOTPLUG_MEMORY=n
Changed redhat/configs/common/generic/s390x/CONFIG_HID from 'is not set'
to '=n' to test mismatch error.
Error: Mismatches found in configuration files for s390x-debug
Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source
tree
Error: Mismatches found in configuration files for s390x-kgcov
Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source
tree
Error: Mismatches found in configuration files for s390x-zfcpdump
Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source
tree
Error: Mismatches found in configuration files for s390x
Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source
tree
make[1]: *** [Makefile:128: dist-configs] Error 1
Changed redhat/configs/common/generic/CONFIG_PREEMPT to '=y' to test
misconfigured error.
Found misconfigured config items in aarch64-debug, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64-debug.config.orig:4245:warning:
override: PREEMPT changes choice state
Found misconfigured config items in aarch64-kgcov, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64-kgcov.config.orig:4224:warning:
override: PREEMPT changes choice state
Found misconfigured config items in ppc64le-kgcov, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-
kgcov.config.orig:4028:warning: override: PREEMPT changes choice state
Found misconfigured config items in ppc64le, please set them to an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-ppc64le.config.orig:4026:warning: override:
PREEMPT changes choice state
Found misconfigured config items in x86_64-debug, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64-debug.config.orig:4212:warning:
override: PREEMPT changes choice state
Found misconfigured config items in x86_64-kgcov, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64-kgcov.config.orig:4192:warning:
override: PREEMPT changes choice state
Found misconfigured config items in x86_64, please set them to an
appropriate
value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64.config.orig:4190:warning: override:
PREEMPT changes choice state
Found misconfigured config items in aarch64, please set them to an
appropriate value
/home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64.config.orig:4222:warning: override:
PREEMPT changes choice state
Found misconfigured config items in ppc64le-debug, please set them to
an
appropriate value
/home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-
debug.config.orig:4045:warning: override: PREEMPT changes choice state
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
---
redhat/configs/build_configs.sh | 35 ++++++-
redhat/configs/process_configs.sh | 163 ++++++++++++++++++++++++-------------
redhat/Makefile | 22 +++-
3 files changed, 145 insertions(+), 75 deletions(-)
11:48 a.m.
New subject: [OS-BUILD PATCHv6 1/9] redhat/configs/process_configs.sh: Add
processing
config function
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/process_configs.sh: Add processing config function
Move the code to process a config into its own function. This will make
future changes easier.
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -218,68 +218,79 @@ function commit_new_configs()
git commit -m "[redhat] AUTOMATIC: New configs"
}
-function process_configs()
+function process_config()
{
- # assume we are in $source_tree/configs, need to get to top level
- pushd "$(switch_to_toplevel)" &>/dev/null
+ local cfg
+ local arch
+ local cfgtmp
+ local cfgorig
- for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
- do
- arch=$(head -1 "$cfg" | cut -b 3-)
+ cfg=$1
+ arch=$(head -1 "$cfg" | cut -b 3-)
- if [ "$arch" = "EMPTY" ]
- then
- # This arch is intentionally left blank
- continue
- fi
+ if [ "$arch" = "EMPTY" ]
+ then
+ # This arch is intentionally left blank
+ return
+ fi
- cfgtmp="${cfg}.tmp"
- cfgorig="${cfg}.orig"
- cat "$cfg" > "$cfgorig"
+ cfgtmp="${cfg}.tmp"
+ cfgorig="${cfg}.orig"
+ cat "$cfg" > "$cfgorig"
- echo -n "Processing $cfg ... "
+ echo -n "Processing $cfg ... "
- make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
- grep -E 'CONFIG_' .listnewconfig > .newoptions
- if test -n "$NEWOPTIONS" && test -s .newoptions
- then
- echo "Found unset config items, please set them to an appropriate value"
- cat .newoptions
- rm .newoptions
- RETURNCODE=1
- [ "$CONTINUEONERROR" ] || exit 1
- fi
+ make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
+ grep -E 'CONFIG_' .listnewconfig > .newoptions
+ if test -n "$NEWOPTIONS" && test -s .newoptions
+ then
+ echo "Found unset config items, please set them to an appropriate value"
+ cat .newoptions
rm .newoptions
+ RETURNCODE=1
+ [ "$CONTINUEONERROR" ] || exit 1
+ fi
+ rm .newoptions
- grep -E 'config.*warning' .listnewconfig > .warnings
- if test -n "$CHECKWARNINGS" && test -s .warnings
- then
- echo "Found misconfigured config items, please set them to an appropriate
value"
- cat .warnings
- rm .warnings
- RETURNCODE=1
- [ "$CONTINUEONERROR" ] || exit 1
- fi
+ grep -E 'config.*warning' .listnewconfig > .warnings
+ if test -n "$CHECKWARNINGS" && test -s .warnings
+ then
+ echo "Found misconfigured config items, please set them to an appropriate
value"
+ cat .warnings
rm .warnings
+ RETURNCODE=1
+ [ "$CONTINUEONERROR" ] || exit 1
+ fi
+ rm .warnings
- rm .listnewconfig
+ rm .listnewconfig
- make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
- echo "# $arch" > "$cfgtmp"
- cat "$cfgorig" >> "$cfgtmp"
- if test -n "$CHECKOPTIONS"
- then
- checkoptions "$cfg" "$cfgtmp"
- fi
- # if test run, don't overwrite original
- if test -n "$TESTRUN"
- then
- rm -f "$cfgtmp"
- else
- mv "$cfgtmp" "$cfg"
- fi
- rm -f "$cfgorig"
- echo "done"
+ make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
+ echo "# $arch" > "$cfgtmp"
+ cat "$cfgorig" >> "$cfgtmp"
+ if test -n "$CHECKOPTIONS"
+ then
+ checkoptions "$cfg" "$cfgtmp"
+ fi
+ # if test run, don't overwrite original
+ if test -n "$TESTRUN"
+ then
+ rm -f "$cfgtmp"
+ else
+ mv "$cfgtmp" "$cfg"
+ fi
+ rm -f "$cfgorig"
+ echo "done"
+}
+
+function process_configs()
+{
+ # assume we are in $source_tree/configs, need to get to top level
+ pushd "$(switch_to_toplevel)" &>/dev/null
+
+ for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
+ do
+ process_config "$cfg"
done
rm "$SCRIPT_DIR"/*.config*.old
popd > /dev/null
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 2/9] redhat/configs/process_configs.sh: Create
unique output
files
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/process_configs.sh: Create unique output files
Create unique output files from process_config().
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -60,6 +60,8 @@ switch_to_toplevel()
checkoptions()
{
+ count=$3
+
/usr/bin/awk '
/is not set/ {
@@ -82,10 +84,10 @@ checkoptions()
print "Found "a[1]"="a[2]" after generation, had "
a[1]"="configs[a[1]]" in Source tree";
}
}
- ' "$1" "$2" > .mismatches
+ ' "$1" "$2" > .mismatches${count}
checkoptions_error=false
- if test -s .mismatches
+ if test -s .mismatches${count}
then
while read -r LINE
do
@@ -97,12 +99,12 @@ checkoptions()
checkoptions_error=true
break
fi
- done < .mismatches
+ done < .mismatches${count}
! $checkoptions_error && return
echo "Error: Mismatches found in configuration files"
- cat .mismatches
+ cat .mismatches${count}
RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
fi
@@ -224,8 +226,11 @@ function process_config()
local arch
local cfgtmp
local cfgorig
+ local count
cfg=$1
+ count=$2
+
arch=$(head -1 "$cfg" | cut -b 3-)
if [ "$arch" = "EMPTY" ]
@@ -240,37 +245,37 @@ function process_config()
echo -n "Processing $cfg ... "
- make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
- grep -E 'CONFIG_' .listnewconfig > .newoptions
- if test -n "$NEWOPTIONS" && test -s .newoptions
+ make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig${count}
+ grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
+ if test -n "$NEWOPTIONS" && test -s .newoptions${count}
then
echo "Found unset config items, please set them to an appropriate value"
- cat .newoptions
- rm .newoptions
+ cat .newoptions${count}
+ rm .newoptions${count}
RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
fi
- rm .newoptions
+ rm .newoptions${count}
- grep -E 'config.*warning' .listnewconfig > .warnings
- if test -n "$CHECKWARNINGS" && test -s .warnings
+ grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
+ if test -n "$CHECKWARNINGS" && test -s .warnings${count}
then
echo "Found misconfigured config items, please set them to an appropriate
value"
- cat .warnings
- rm .warnings
+ cat .warnings${count}
+ rm .warnings${count}
RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
fi
- rm .warnings
+ rm .warnings${count}
- rm .listnewconfig
+ rm .listnewconfig${count}
make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
echo "# $arch" > "$cfgtmp"
cat "$cfgorig" >> "$cfgtmp"
if test -n "$CHECKOPTIONS"
then
- checkoptions "$cfg" "$cfgtmp"
+ checkoptions "$cfg" "$cfgtmp" "$count"
fi
# if test run, don't overwrite original
if test -n "$TESTRUN"
@@ -288,9 +293,11 @@ function process_configs()
# assume we are in $source_tree/configs, need to get to top level
pushd "$(switch_to_toplevel)" &>/dev/null
+ count=0
for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
do
- process_config "$cfg"
+ process_config "$cfg" "$count"
+ ((count++))
done
rm "$SCRIPT_DIR"/*.config*.old
popd > /dev/null
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 3/9] redhat/configs/process_configs.sh: Provide
better
messages
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/process_configs.sh: Provide better messages
Provide better error messages.
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -61,6 +61,7 @@ switch_to_toplevel()
checkoptions()
{
count=$3
+ variant=$4
/usr/bin/awk '
@@ -103,10 +104,12 @@ checkoptions()
! $checkoptions_error && return
- echo "Error: Mismatches found in configuration files"
+ sed -i "1s/^/Error: Mismatches found in configuration files for
${variant}\n/" .mismatches${count}
cat .mismatches${count}
RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
+ else
+ rm -f .mismatches${count}
fi
}
@@ -227,6 +230,7 @@ function process_config()
local cfgtmp
local cfgorig
local count
+ local variant
cfg=$1
count=$2
@@ -239,17 +243,19 @@ function process_config()
return
fi
+ variant=$(basename "$cfg" | cut -d"-" -f3- | cut -d"."
-f1)
+
cfgtmp="${cfg}.tmp"
cfgorig="${cfg}.orig"
cat "$cfg" > "$cfgorig"
- echo -n "Processing $cfg ... "
+ echo "Processing $cfg ... "
make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig${count}
grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
if test -n "$NEWOPTIONS" && test -s .newoptions${count}
then
- echo "Found unset config items, please set them to an appropriate value"
+ echo "Found unset config items in ${variant}, please set them to an appropriate
value"
cat .newoptions${count}
rm .newoptions${count}
RETURNCODE=1
@@ -260,7 +266,7 @@ function process_config()
grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
if test -n "$CHECKWARNINGS" && test -s .warnings${count}
then
- echo "Found misconfigured config items, please set them to an appropriate
value"
+ echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value"
cat .warnings${count}
rm .warnings${count}
RETURNCODE=1
@@ -275,7 +281,7 @@ function process_config()
cat "$cfgorig" >> "$cfgtmp"
if test -n "$CHECKOPTIONS"
then
- checkoptions "$cfg" "$cfgtmp" "$count"
+ checkoptions "$cfg" "$cfgtmp" "$count"
"$variant"
fi
# if test run, don't overwrite original
if test -n "$TESTRUN"
@@ -285,7 +291,7 @@ function process_config()
mv "$cfgtmp" "$cfg"
fi
rm -f "$cfgorig"
- echo "done"
+ echo "Processing $cfg complete"
}
function process_configs()
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 4/9] redhat/configs/process_configs.sh: Parallelize
execution
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/process_configs.sh: Parallelize execution
Parallelize the execution of each config processing.
The '+' adds job processing to 'non-make' commands [1].
[1] https://www.gnu.org/software/make/manual/html_node/Error-Messages.html
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -50,6 +50,14 @@ KABIDW := $(REDHAT)/kabi-dwarf
include Makefile.cross
+RHJOBS?=$(shell j=$$(echo $(MAKEFLAGS) | grep -Eo "\ -j[0-9]*" | xargs ); \
+ if [ -z "$${j}" ]; then \
+ echo "1"; \
+ else \
+ j=$$(echo "$${j}" | tr -d "\-j"); \
+ [ -z "$${j}" ] && nproc --all || echo $${j}; \
+ fi)
+
default: dist-help
dist-python-check:
@@ -121,12 +129,12 @@ dist-kabi-dw-check: dist-kabi
@rm -rf $(KABIDW)/base/$(CURARCH).tmp
dist-configs-commit: dist-configs-prep
- @cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
- ./process_configs.sh -z $(PACKAGE_NAME) $(KVERSION) "" $(FLAVOR)
+ +@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
+ ./process_configs.sh -z "$(PACKAGE_NAME)" "$(KVERSION)" ""
"$(FLAVOR)" "$(RHJOBS)"
dist-configs: dist-configs-prep
- @cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
- ./process_configs.sh $(PROCESS_CONFIGS_OPTS) $(PACKAGE_NAME) $(KVERSION)
+ +@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
+ ./process_configs.sh $(PROCESS_CONFIGS_OPTS) "$(PACKAGE_NAME)"
"$(KVERSION)" "" "" "$(RHJOBS)"
dist-fedora-configs: FLAVOR = fedora
dist-fedora-configs: dist-configs
@@ -137,7 +145,7 @@ dist-rhel-configs: dist-configs
rh-configs: dist-rhel-configs
dist-configs-check: dist-configs-prep
- cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
+ +cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
dist-configs-prep: dist-clean-configs dist-buildreq-check
cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip"
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -105,8 +105,6 @@ checkoptions()
! $checkoptions_error && return
sed -i "1s/^/Error: Mismatches found in configuration files for
${variant}\n/" .mismatches${count}
- cat .mismatches${count}
- RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
else
rm -f .mismatches${count}
@@ -255,8 +253,8 @@ function process_config()
grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
if test -n "$NEWOPTIONS" && test -s .newoptions${count}
then
- echo "Found unset config items in ${variant}, please set them to an appropriate
value"
- cat .newoptions${count}
+ echo "Found unset config items in ${variant}, please set them to an appropriate
value" >> .errors${count}
+ cat .newoptions${count} >> .errors${count}
rm .newoptions${count}
RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
@@ -266,10 +264,9 @@ function process_config()
grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
if test -n "$CHECKWARNINGS" && test -s .warnings${count}
then
- echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value"
- cat .warnings${count}
+ echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value" >> .errors${count}
+ cat .warnings${count} >> .errors${count}
rm .warnings${count}
- RETURNCODE=1
[ "$CONTINUEONERROR" ] || exit 1
fi
rm .warnings${count}
@@ -302,13 +299,31 @@ function process_configs()
count=0
for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
do
- process_config "$cfg" "$count"
+ process_config "$cfg" "$count" &
+ waitpids[${count}]=$!
((count++))
+ while [ "$(jobs | grep Running | wc -l)" -ge $RHJOBS ]; do :; done
+ done
+ for pid in ${waitpids[*]}; do
+ wait ${pid}
done
+
rm "$SCRIPT_DIR"/*.config*.old
+
+ if ls .errors* 1> /dev/null 2>&1; then
+ RETURNCODE=1
+ cat .errors*
+ rm .errors* -f
+ fi
+ if ls .mismatches* 1> /dev/null 2>&1; then
+ RETURNCODE=1
+ cat .mismatches*
+ rm .mismatches* -f
+ fi
+
popd > /dev/null
- echo "Processed config files are in $SCRIPT_DIR"
+ [ $RETURNCODE -eq 0 ] && echo "Processed config files are in
$SCRIPT_DIR"
}
CHECKOPTIONS=""
@@ -369,6 +384,7 @@ PACKAGE_NAME="${1:-kernel}" # defines the package name used
KVERREL="$(test -n "$2" && echo "-$2" || echo
"")"
SUBARCH="$(test -n "$3" && echo "-$3" || echo
"")"
FLAVOR="$(test -n "$4" && echo "-$4" || echo
"-common")"
+RHJOBS="$(test -n "$5" && echo "$5" || nproc
--all)"
SCRIPT=$(readlink -f "$0")
SCRIPT_DIR=$(dirname "$SCRIPT")
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 5/9] redhat/configs/build_configs.sh: Add local
variables
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/build_configs.sh: Add local variables
Add local variables to the merge_configs function. This will make future
changes for parallel processing easier.
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -53,6 +53,14 @@ function combine_config_layer()
function merge_configs()
{
+ local archvar
+ local arch
+ local configs
+ local order
+ local flavor
+ local name
+ local skip_if_missing
+
archvar=$1
arch=$(echo "$archvar" | cut -f1 -d"-")
configs=$2
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 6/9] redhat/configs/build_configs.sh: Create unique
output
files
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/build_configs.sh: Create unique output files
Create unique output files from merge_configs().
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -58,6 +58,7 @@ function merge_configs()
local configs
local order
local flavor
+ local count
local name
local skip_if_missing
@@ -66,10 +67,11 @@ function merge_configs()
configs=$2
order=$3
flavor=$4
+ count=$5
name=$OUTPUT_DIR/$PACKAGE_NAME-$archvar-$flavor.config
echo -n "Building $name ... "
- touch config-merging config-merged
+ touch config-merging.$count config-merged.$count
# apply based on order
skip_if_missing=""
@@ -81,10 +83,10 @@ function merge_configs()
test -n "$skip_if_missing" && test ! -e "$cfile"
&& continue
- if ! perl merge.pl "$cfile" config-merging > config-merged; then
+ if ! perl merge.pl "$cfile" config-merging.$count > config-merged.$count;
then
die "Failed to merge $cfile"
fi
- mv config-merged config-merging
+ mv config-merged.$count config-merging.$count
done
# first configs in $order is baseline, all files should be
@@ -104,8 +106,8 @@ function merge_configs()
else
echo "# $arch" > "$name"
fi
- sort config-merging >> "$name"
- rm -f config-merged config-merging
+ sort config-merging.$count >> "$name"
+ rm -f config-merged.$count config-merging.$count
echo "done"
}
@@ -149,7 +151,7 @@ function build_flavor()
esac
fi
- merge_configs "$arch" "$configs" "$order"
"$flavor"
+ merge_configs "$arch" "$configs" "$order"
"$flavor" "$count"
fi
done < "$control_file"
}
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 7/9] redhat/configs/build_configs.sh: Provide
better
messages
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/build_configs.sh: Provide better messages
Provide better error messages.
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -70,7 +70,7 @@ function merge_configs()
count=$5
name=$OUTPUT_DIR/$PACKAGE_NAME-$archvar-$flavor.config
- echo -n "Building $name ... "
+ echo "Building $name ... "
touch config-merging.$count config-merged.$count
# apply based on order
@@ -108,7 +108,7 @@ function merge_configs()
fi
sort config-merging.$count >> "$name"
rm -f config-merged.$count config-merging.$count
- echo "done"
+ echo "Building $name complete"
}
function build_flavor()
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 8/9] redhat/configs/build_configs.sh: Parallelize
execution
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/build_configs.sh: Parallelize execution
Parallelize the execution of each config building.
The '+' adds job processing to 'non-make' commands [1].
[1] https://www.gnu.org/software/make/manual/html_node/Error-Messages.html
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -148,8 +148,8 @@ dist-configs-check: dist-configs-prep
+cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
dist-configs-prep: dist-clean-configs dist-buildreq-check
- cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip"
- cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)"
"$(ARCH_MACH)" "$(FLAVOR)"
+ +cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip" "$(RHJOBS)"
+ +cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)"
"$(ARCH_MACH)" "$(FLAVOR)" "$(RHJOBS)"
dist-configs-arch: ARCH_MACH = $(MACH)
dist-configs-arch: dist-configs
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -16,6 +16,8 @@ else
echo "$3" > .flavors
fi
+RHJOBS="$(test -n "$4" && echo "$4" || nproc
--all)"
+
LANG=en_US.UTF-8
# to handle this script being a symlink
@@ -151,9 +153,16 @@ function build_flavor()
esac
fi
- merge_configs "$arch" "$configs" "$order"
"$flavor" "$count"
+ merge_configs "$arch" "$configs" "$order"
"$flavor" "$count" &
+ waitpids[$count]=$!
+ ((count++))
+ while [ "$(jobs | grep Running | wc -l)" -ge $RHJOBS ]; do :; done
fi
done < "$control_file"
+
+ for pid in ${waitpids[*]}; do
+ wait $pid
+ done
}
while read -r line
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
11:48 a.m.
New subject: [OS-BUILD PATCHv6 9/9] redhat/configs/process_configs.sh: Fix issue
with old
error files
From: Prarit Bhargava <prarit(a)redhat.com>
redhat/configs/process_configs.sh: Fix issue with old error files
Previous to this changeset, running process_configs.sh would result in an
.mismatches file being left after execution. If someone has done
make dist-configs
\# update to latest
make -j10 dist-configs
the '-j10 dist-configs' would fail because of the existence of the
.mismatches file.
Explicitly remove the .mismatches file if it exists.
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com>
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -296,6 +296,9 @@ function process_configs()
# assume we are in $source_tree/configs, need to get to top level
pushd "$(switch_to_toplevel)" &>/dev/null
+ # the old code left this file lying around
+ [ -f .mismatches ] && rm -f .mismatches
+
count=0
for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
do
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635
1:14 p.m.
New subject: [OS-BUILD PATCHv6 0/9] process_configs.sh: Add parallel
processing of
configs in dist-configs
From: Don Zickus on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8657...
Hmm. If you go 15 lines down it does the same thing but returns an error
code. Is that code redundant now or is there an error code to be tracked?
2:25 p.m.
New subject: [OS-BUILD PATCHv6 0/9] process_configs.sh: Add parallel
processing of
configs in dist-configs
From: Prarit Bhargava on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8658...
It's hard to explain. The *ONLY* reason for the last commit is to avoid the
issue @jmflinuxtx hit because he executed both the old code and the new code
in the same tree.
If you do
make dist-configs # old code
# this leaves a .mismatches file (you can do a ls -l .mismatches to find it)
# apply my code
make -j10 dist-configs # new code
# this code adds and removes .mismatches"${count}" files. The removal is done
(as you noticed) with a 'rm -f .mismatches*'. However, the code (without the
last commit I added) errors out by detecting '.mismatches*' -- which picks up
the OLD .mismatches file.
I'm not sure if that made any more sense or just confused you further.
Here's the kicker: If you were now to do a 'make -j10 dist-configs' the code
would return successfully because of the 'rm -f .mismatches*'.
2:29 p.m.
New subject: [OS-BUILD PATCHv6 0/9] process_configs.sh: Add parallel
processing of
configs in dist-configs
From: Don Zickus on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8658...
Ok. That wasn't obvious. So the correct way to fix it, is bring in your last
patch now as a standalone MR. Then this MR will remove/replace with new
removal code, right?
2:33 p.m.
New subject: [OS-BUILD PATCHv6 0/9] process_configs.sh: Add parallel
processing of
configs in dist-configs
From: Don Zickus on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8658...
Or at least add a fat blurb that those new lines are throwaway code to
transition to the parallel workflow. I just don't want that code lying around
and then a year later someone reads the code and goes why is that there.
Let's be clear the code can be removed in a month or so.
787
days inactive
787
days old
kernel@lists.fedoraproject.org
13 comments
2 participants
participants (2)
-
Don Zickus (via Email Bridge) -
Prarit Bhargava (via Email Bridge)