12:33 p.m.
Hi All,
We've been using merge.pl to merge Fedora's config file fragments for as
long as I can remember. However, in the interest of using what's
upstream, I took a look at the newly added merge_config.sh script and
what it would take to use that instead.
The good:
It actually worked fairly well. The resulting merged config files were
identical to the ones merge.pl creates.
It also provides a lot more output about the options present, what is
overriding what, and final config verification (if we used it to do the
oldconfig step). At some point we could use it to auto-disable new
config options too, but that's a big maybe.
The ok:
The changes to kernel.spec are pretty small. I basically used the same
approach we do today by creating Makefile.merge and calling the script
from within that. Doing similar creation steps in the spec itself was
pretty ugly. The largest change, aside from using a different merge
tool, is that the fragments are merged after all patches are applied. I
don't think there is a reason why they're merged before hand now, but it
is a small difference.
The ugly:
It's slow. All that checking and reporting and verification takes quite
a bit of processing. How much? Well, compare:
Current:
[jwboyer@vader kernel]$ time fedpkg -v prep --arch=noarch >
foo.log
real 0m46.898s
user 0m9.248s
sys 0m35.764s
[jwboyer@vader kernel]$
merge_config.sh (yes, for real):
[jwboyer@vader kernel]$ time fedpkg -v prep --arch=noarch >
foo.log
real 25m16.705s
user 0m58.587s
sys 19m57.125s
[jwboyer@vader kernel]$
merge_config.sh + patch:
[jwboyer@vader kernel]$ time fedpkg -v prep --arch=noarch >
foo.log
real 3m12.711s
user 0m15.655s
sys 2m47.259s
[jwboyer@vader kernel]$
So, ignoring the stock merge_config.sh case which is clearly not usable,
it would make the %prep section roughly 4x longer. The patch I added to
make it at least somewhat reasonable has been sent upstream.
I've attached a log file from a run with merge_config.sh for people to
glance over. Before I do anything else, do people think the output is
useful and this is perhaps beneficial?
josh
Spec diff:
diff --git a/kernel.spec b/kernel.spec
index 858c483..a21eb0a 100644
--- a/kernel.spec
+++ b/kernel.spec
@@ -574,6 +574,7 @@ Source15: merge.pl
Source16: mod-extra.list
Source19: Makefile.release
+Source18: Makefile.merge
Source20: Makefile.config
Source21: config-debug
Source22: config-nodebug
@@ -743,6 +744,8 @@ Patch21091: kmemleak.patch
Patch21092: udlfb-remove-sysfs-framebuffer-device-with-USB-disconnect.patch
+Patch21093: faster-merge.patch
+
# compat-wireless patches
Patch50000: compat-wireless-config-fixups.patch
Patch50001: compat-wireless-pr_fmt-warning-avoidance.patch
@@ -1265,28 +1268,6 @@ make -f %{SOURCE19} config-release
%endif
%endif
-# Dynamically generate kernel .config files from config-* files
-make -f %{SOURCE20} VERSION=%{version} configs
-
-%if %{with_backports}
-# Turn-off bits provided by compat-wireless
-for i in %{all_arch_configs}
-do
- mv $i $i.tmp
- ./merge.pl %{SOURCE200} $i.tmp > $i
- rm $i.tmp
-done
-%endif
-
-# Merge in any user-provided local config option changes
-%if %{?all_arch_configs:1}%{!?all_arch_configs:0}
-for i in %{all_arch_configs}
-do
- mv $i $i.tmp
- ./merge.pl %{SOURCE1000} $i.tmp > $i
- rm $i.tmp
-done
-%endif
ApplyPatch linux-2.6-makefile-after_link.patch
@@ -1439,6 +1420,8 @@ ApplyPatch
udlfb-remove-sysfs-framebuffer-device-with-USB-disconnect.patch
#rhbz 783211
ApplyPatch fs-Inval-cache-for-parent-block-device-if-fsync-called-on-part.patch
+ApplyPatch faster-merge.patch
+
# END OF PATCH APPLICATIONS
%endif
@@ -1466,6 +1449,45 @@ done
rm -f kernel-%{version}-*debug.config
%endif
+# Ok, now let's try to do this with merge_config.sh
+chmod +x scripts/kconfig/merge_config.sh
+
+# Dynamically generate kernel .config files from config-* files
+make -f %{SOURCE18} VERSION=%{version} configs
+
+%if %{with_backports}
+# Turn-off bits provided by compat-wireless
+for i in %{all_arch_configs}
+do
+ mv $i $i.tmp
+ scripts/kconfig/merge_config.sh -m -n $i.tmp %{SOURCE200}
+ mv .config $i
+ rm $i.tmp
+done
+%endif
+
+# Merge in any user-provided local config option changes
+%if %{?all_arch_configs:1}%{!?all_arch_configs:0}
+for i in %{all_arch_configs}
+do
+ mv $i $i.tmp
+ scripts/kconfig/merge_config.sh -m -n $i.tmp %{SOURCE1000}
+ mv .config $i
+ rm $i.tmp
+done
+%endif
+
+# Remove configs not for the buildarch
+for cfg in kernel-%{version}-*.config; do
+ if [ `echo %{all_arch_configs} | grep -c $cfg` -eq 0 ]; then
+ rm -f $cfg
+ fi
+done
+
+%if !%{debugbuildsenabled}
+rm -f kernel-%{version}-*debug.config
+%endif
+
# now run oldconfig over all the config files
for i in *.config
do
2:24 p.m.
On Wed, Feb 01, 2012 at 01:33:41PM -0500, Josh Boyer wrote:
Hi All,
We've been using merge.pl to merge Fedora's config file fragments for as
long as I can remember. However, in the interest of using what's
upstream, I took a look at the newly added merge_config.sh script and
what it would take to use that instead.
I remember looking at this last week and thinking the same thing. I
remember hacking on merge.pl years ago and optimizing to run really fast.
Even today there are RH folks who complain it still runs slow.
I thought the same as you, why not start using the upstream approach and
have one less thing to maintain/worry about in the fedora spec file.
But the major plus with using perl was you could cache the file in memory
and it only had to be read once. With the script, files are read multiple
times which add to the delay.
There is really no way around it unless upstream converts to perl or
python or even 'c'.
So I would be torn and lean away from it until the speed improves. I know
if this makes it to RHEL people would complain and it would get ripped out
in favor of the old way.
My two cents.
Cheers,
Don
2:32 p.m.
On Wed, Feb 01, 2012 at 03:24:23PM -0500, Don Zickus wrote:
On Wed, Feb 01, 2012 at 01:33:41PM -0500, Josh Boyer wrote:
> Hi All,
>
> We've been using merge.pl to merge Fedora's config file fragments for as
> long as I can remember. However, in the interest of using what's
> upstream, I took a look at the newly added merge_config.sh script and
> what it would take to use that instead.
I remember looking at this last week and thinking the same thing. I
remember hacking on merge.pl years ago and optimizing to run really fast.
Even today there are RH folks who complain it still runs slow.
Yes, merge.pl is rather fast.
I did happen to time _just_ the make -f Makefile.{config,merge} portion
in the %prep section. merge.pl takes 2 seconds. merge_config.sh+patch
takes 2m.
I thought the same as you, why not start using the upstream approach
and
have one less thing to maintain/worry about in the fedora spec file.
But the major plus with using perl was you could cache the file in memory
and it only had to be read once. With the script, files are read multiple
times which add to the delay.
There is really no way around it unless upstream converts to perl or
python or even 'c'.
So I would be torn and lean away from it until the speed improves. I know
if this makes it to RHEL people would complain and it would get ripped out
in favor of the old way.
If we're going on pure speed then sure. There's no reason to replace
what is working today with something that is much slower. What I'm more
interested in though, is either:
1) Does the additional verification from merge_config.sh prove useful
enough to warrant a slowdown? I can see it being useful to make sure we
don't screw something up on a rebase, etc.
2) Can merge.pl be adapted to produce similar output without making it
much slower?
If I have to go with 2, I'll be replacing merge.pl with merge.py because
I don't speak perl. ;)
Alternatively, we could make it an option to use the upstream stuff so
that if we're doing a rebase or just want to see what the hell is going
on, it's a trivial toggle of something to switch. I don't think the
code for that would be all that ugly at all.
josh
2:58 p.m.
On Wed, Feb 01, 2012 at 03:32:26PM -0500, Josh Boyer wrote:
>
> So I would be torn and lean away from it until the speed improves. I know
> if this makes it to RHEL people would complain and it would get ripped out
> in favor of the old way.
If we're going on pure speed then sure. There's no reason to replace
what is working today with something that is much slower. What I'm more
interested in though, is either:
1) Does the additional verification from merge_config.sh prove useful
enough to warrant a slowdown? I can see it being useful to make sure we
don't screw something up on a rebase, etc.
Sure. I'll agree with that.
2) Can merge.pl be adapted to produce similar output without making it
much slower?
If I have to go with 2, I'll be replacing merge.pl with merge.py because
I don't speak perl. ;)
It shouldn't be too hard if given to someone who knows python. That is
probably worth pursuing upstream.
Cheers,
Don
7:16 p.m.
On Wed, Feb 01, 2012 at 03:32:26PM -0500, Josh Boyer wrote:
> So I would be torn and lean away from it until the speed
improves. I know
> if this makes it to RHEL people would complain and it would get ripped out
> in favor of the old way.
If we're going on pure speed then sure. There's no reason to replace
what is working today with something that is much slower. What I'm more
interested in though, is either:
Given this is a common case (I couldn't guess how many 'make prep's I do
each day, but it's easily in double figures), I don't think that kind of
slowdown is going to be acceptable for us at all.
1) Does the additional verification from merge_config.sh prove
useful
enough to warrant a slowdown? I can see it being useful to make sure we
don't screw something up on a rebase, etc.
2) Can merge.pl be adapted to produce similar output without making it
much slower?
If I have to go with 2, I'll be replacing merge.pl with merge.py because
I don't speak perl. ;)
Alternatively, we could make it an option to use the upstream stuff so
that if we're doing a rebase or just want to see what the hell is going
on, it's a trivial toggle of something to switch. I don't think the
code for that would be all that ugly at all.
Or perhaps make an additional tool perhaps just to sanity check things ?
(Given it's only really something that we care about when we pull in
a new upstream snapshot for the most part)
My first thought when I read the output was 'yeah, we know this is over-riding
that'.
That's going to get tedious to read every make prep, and I think any real
problems would get lost in the noise. Perhaps we'd need a way to annotate
overrides in our template files so they could be ignored when we had reviewed
them and made sure they were ok.
Dave
8:18 a.m.
On Thu, Feb 02, 2012 at 08:16:40PM -0500, Dave Jones wrote:
On Wed, Feb 01, 2012 at 03:32:26PM -0500, Josh Boyer wrote:
> > So I would be torn and lean away from it until the speed improves. I know
> > if this makes it to RHEL people would complain and it would get ripped out
> > in favor of the old way.
>
> If we're going on pure speed then sure. There's no reason to replace
> what is working today with something that is much slower. What I'm more
> interested in though, is either:
Given this is a common case (I couldn't guess how many 'make prep's I do
each day, but it's easily in double figures), I don't think that kind of
slowdown is going to be acceptable for us at all.
Yeah. I got tired of waiting for it to complete while I was testing it,
so waiting for it daily might drive me nuts.
> 1) Does the additional verification from merge_config.sh prove
useful
> enough to warrant a slowdown? I can see it being useful to make sure we
> don't screw something up on a rebase, etc.
>
> 2) Can merge.pl be adapted to produce similar output without making it
> much slower?
>
> If I have to go with 2, I'll be replacing merge.pl with merge.py because
> I don't speak perl. ;)
>
> Alternatively, we could make it an option to use the upstream stuff so
> that if we're doing a rebase or just want to see what the hell is going
> on, it's a trivial toggle of something to switch. I don't think the
> code for that would be all that ugly at all.
Or perhaps make an additional tool perhaps just to sanity check things ?
(Given it's only really something that we care about when we pull in
a new upstream snapshot for the most part)
My first thought when I read the output was 'yeah, we know this is over-riding
that'.
That's going to get tedious to read every make prep, and I think any real
problems would get lost in the noise. Perhaps we'd need a way to annotate
overrides in our template files so they could be ignored when we had reviewed
them and made sure they were ok.
Right. The output I attached wasn't the best example, since the way I
have it coded to make it performant at all was to keep it from doing
'make oldconfig' for us. If you let it do that, it does some additional
verification to make sure the options you set are actually present in
the final config, and it tells you if they aren't (or are different).
That part is useful, but I agree it might be done better in a seperate
tool.
I'll look at that soon.
josh
4464
days inactive
4466
days old
kernel@lists.fedoraproject.org
5 comments
3 participants
participants (3)
-
Dave Jones -
Don Zickus -
Josh Boyer