Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
-sv
seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
+1
73 de Jeff
Jeff Johnson n3npq@nc.rr.com:
seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
+1
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
On Mon, 31 Jan 2005 17:37:29 -0500, Eric S. Raymond esr@thyrsus.com wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
they bloat the packages as well... which means they bloat the install media and bloat the download times over the network.
I'm hard pressed to think of a real-world situation where 2 year old package changelog notations in the packages are needed. Shouldn't access to the cvs server suffice for deep-diving into package changelogs?
-jef
Le lundi 31 janvier 2005 à 17:44 -0500, Jeff Spaleta a écrit :
On Mon, 31 Jan 2005 17:37:29 -0500, Eric S. Raymond esr@thyrsus.com wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
they bloat the packages as well... which means they bloat the install media and bloat the download times over the network.
I'm hard pressed to think of a real-world situation where 2 year old package changelog notations in the packages are needed. Shouldn't access to the cvs server suffice for deep-diving into package changelogs?
Aren't changelogs compressed yet ? I'm very surprised simple flat text data can make such a significant difference these days (the yum problem OTOH is real but can't yum truncat the changelogs it exports all by itself ?)
Regards,
Aren't changelogs compressed yet ? I'm very surprised simple flat text data can make such a significant difference these days (the yum problem OTOH is real but can't yum truncat the changelogs it exports all by itself ?)
1. we're not just talking about it for yum. 2. I could truncate them in createrepo, sure but the question is why are we keeping all this stuff on cds in 5 different places and why are we keeping it AT ALL on the cds.
-sv
Le lundi 31 janvier 2005 à 17:37 -0500, Eric S. Raymond a écrit :
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
changelogs are in repodata/other.xml (seems yum does not use this file). other.xml.gz = 5 Mo FC3 = 2.3 Go
On Mon, 2005-01-31 at 23:57 +0100, Féliciano Matias wrote:
Le lundi 31 janvier 2005 à 17:37 -0500, Eric S. Raymond a écrit :
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
changelogs are in repodata/other.xml (seems yum does not use this file). other.xml.gz = 5 Mo FC3 = 2.3 Go
yum can use this file:
yum generate-rss uses it.
-sv
On Mon, Jan 31, 2005 at 11:57:08PM +0100, Féliciano Matias wrote:
Le lundi 31 janvier 2005 à 17:37 -0500, Eric S. Raymond a écrit :
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
changelogs are in repodata/other.xml (seems yum does not use this file). other.xml.gz = 5 Mo FC3 = 2.3 Go
If yum is not affected then this discusion is about saving <= 0.5% (someone quoted ISO file size)? A bit silly IMO.
On Mon, Jan 31, 2005 at 05:37:29PM -0500, Eric S. Raymond wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
No, it's a *good* idea IMO. Problem is it does't go all the way. WTH are changelogs doing in .spec files?!? This is the job of the version control system, not of packaging specifications.
It probably comes from the (misguided) school of thought that includes $Log$ in source files...
On Mon, 31 Jan 2005 18:01:57 -0500, Dimitrie O. Paun wrote:
On Mon, Jan 31, 2005 at 05:37:29PM -0500, Eric S. Raymond wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
No, it's a *good* idea IMO. Problem is it does't go all the way. WTH are changelogs doing in .spec files?!? This is the job of the version control system, not of packaging specifications.
Uhm, they document package changes. Ever taken a look at
rpm --query --changelog kernel | less
?
It probably comes from the (misguided) school of thought that includes $Log$ in source files...
No.
On Tue, 2005-02-01 at 11:40 +0100, Michael Schwendt wrote:
On Mon, 31 Jan 2005 18:01:57 -0500, Dimitrie O. Paun wrote:
On Mon, Jan 31, 2005 at 05:37:29PM -0500, Eric S. Raymond wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
No, it's a *good* idea IMO. Problem is it does't go all the way. WTH are changelogs doing in .spec files?!? This is the job of the version control system, not of packaging specifications.
Uhm, they document package changes.
Do you document your changes inside of the source code? In my understanding, *specs are one part of rpm's sources.
It probably comes from the (misguided) school of thought that includes $Log$ in source files...
No.
I disagree. Adding %changelogs to specs is not any different from $Log$.
Having an entry in an rpm-header containing the last change might be useful for users being interested in the reason for a new rpm release, but I fail to understand why having a full %changelog-history inside of rpms or metadata files is useful.
Ralf
On Tue, 01 Feb 2005 12:13:27 +0100, Ralf Corsepius wrote:
On Tue, 2005-02-01 at 11:40 +0100, Michael Schwendt wrote:
On Mon, 31 Jan 2005 18:01:57 -0500, Dimitrie O. Paun wrote:
On Mon, Jan 31, 2005 at 05:37:29PM -0500, Eric S. Raymond wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
No, it's a *good* idea IMO. Problem is it does't go all the way. WTH are changelogs doing in .spec files?!? This is the job of the version control system, not of packaging specifications.
Uhm, they document package changes.
Do you document your changes inside of the source code? In my understanding, *specs are one part of rpm's sources.
With RPM there is no other place where you can document package changes and make the same document appear also in the binary rpms. It is a matter of convenience. You can download a package, take a look at the package changes, then extract the ChangeLog file, if included, and if you want to look at software changes, too. The spec changelog is even more important when already installed software doesn't work as expected and you need access to the package changelog quickly.
It probably comes from the (misguided) school of thought that includes $Log$ in source files...
No.
I disagree. Adding %changelogs to specs is not any different from $Log$.
Depends on what $Log$ expands to. You can put very useful comments in there which make a good reading. And everyone without immediate access to CVS would benefit from such comments, because they are included in source tarballs.
Having an entry in an rpm-header containing the last change might be useful for users being interested in the reason for a new rpm release, but I fail to understand why having a full %changelog-history inside of rpms or metadata files is useful.
Because not seldomly there are several package revisions between the "new rpm release" and the previous one. For instance, during steps from FC3 to FC4.
On Tue, Feb 01, 2005 at 01:00:35PM +0100, Michael Schwendt wrote:
I disagree. Adding %changelogs to specs is not any different from $Log$.
Depends on what $Log$ expands to. You can put very useful comments in there which make a good reading. And everyone without immediate access to CVS would benefit from such comments, because they are included in source tarballs.
See, same argument people give for $Log$. As I initially said, just like the $Log$ comments are misguided, so are complete changelogs in .spec files.
Useful comments that are akin to comments in a regular source files are commonly included in .spec files. Those are fine, nobody argues agaist them. Changelog entries are something else, and they just don't belong there.
If they are needed, they should just be in a separate file, but I suspect a link to a cvsweb/viewcvs for the CVS repository of the .spec file would be enough.
Le mardi 01 février 2005 à 08:37 -0500, Dimitrie O. Paun a écrit :
If they are needed, they should just be in a separate file, but I suspect a link to a cvsweb/viewcvs for the CVS repository of the .spec file would be enough.
Much good it will do you if the problem rpm killed your network link.
On Tue, 2005-02-01 at 13:00 +0100, Michael Schwendt wrote:
On Tue, 01 Feb 2005 12:13:27 +0100, Ralf Corsepius wrote:
On Tue, 2005-02-01 at 11:40 +0100, Michael Schwendt wrote:
On Mon, 31 Jan 2005 18:01:57 -0500, Dimitrie O. Paun wrote:
On Mon, Jan 31, 2005 at 05:37:29PM -0500, Eric S. Raymond wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
No, it's a *good* idea IMO. Problem is it does't go all the way. WTH are changelogs doing in .spec files?!? This is the job of the version control system, not of packaging specifications.
Uhm, they document package changes.
Do you document your changes inside of the source code? In my understanding, *specs are one part of rpm's sources.
With RPM there is no other place where you can document package changes and make the same document appear also in the binary rpms.
A matter of design and policy ... this is fedora-devel@ ... so to me your sentence is nothing but evidence for a deficiency either in rpm's implementation or RH's policy ...
It is a matter of convenience. You can download a package, take a look at the package changes, then extract the ChangeLog file, if included, and if you want to look at software changes, too. The spec changelog is even more important when already installed software doesn't work as expected and you need access to the package changelog quickly.
No, you are missing up a package's contents ChangeLog with the rpm's changelog.
The only reason to look into an rpm's changelog is to find answers questions related to "Why has this rpm been released?". Normally you won't find answers to "What has changed inside of the package's contents."
It probably comes from the (misguided) school of thought that includes $Log$ in source files...
No.
I disagree. Adding %changelogs to specs is not any different from $Log$.
Depends on what $Log$ expands to.
Here you say it: It's all about packaging policy.
Seth said he wants to feed rss from rpm changelogs, and wants to reduce the size of changelog entries in other.xml, you want to see the changelogs from the rpm, I don't see any use for changelog histories.
=> Implement a reasonable %changelog policy.
E.g. Add only those %changelogs to an rpm.spec that reach back to the preceding release and remove everything else.
Frankly speaking, the only situation I have found rpm.specs useful sofar, was to dig out the responsible maintainer of a package to get in direct contact to him.
Using a dedicated rpm-header for this purpose (e.g. packager) would seem much more reasonable to me, but I am not in a position to change RH nor to enforce this policy.
Having an entry in an rpm-header containing the last change might be useful for users being interested in the reason for a new rpm release, but I fail to understand why having a full %changelog-history inside of rpms or metadata files is useful.
Because not seldomly there are several package revisions between the "new rpm release" and the previous one. For instance, during steps from FC3 to FC4.
Again, this is just a matter of policy.
Ralf
On Tue, 01 Feb 2005 15:46:58 +0100, Ralf Corsepius wrote:
It is a matter of convenience. You can download a package, take a look at the package changes, then extract the ChangeLog file, if included, and if you want to look at software changes, too. The spec changelog is even more important when already installed software doesn't work as expected and you need access to the package changelog quickly.
No, you are missing up a package's contents ChangeLog with the rpm's changelog.
?
The only reason to look into an rpm's changelog is to find answers questions related to "Why has this rpm been released?".
As in whether any bugs were fixed, features added/removed, default configuration changed, important files relocated or dropped, ... there's a lot that makes sense to be mentioned in an rpm package %changelog.
Normally you won't find answers to "What has changed inside of the package's contents."
No, for that you see what %doc files are included and take a look at those. "rpm --query --docfiles ..."
Le mardi 01 février 2005 à 15:46 +0100, Ralf Corsepius a écrit :
=> Implement a reasonable %changelog policy.
E.g. Add only those %changelogs to an rpm.spec that reach back to the preceding release and remove everything else.
A lot of production systems are not upgraded continuously but when the admin has time to spend in front of them - ie version jump is *very* common (in some cases multi-year cross release rebuilds to accommodate an old workhorse nobody wants to reinstall from scratch)
On Tue, 2005-02-01 at 15:46 +0100, Ralf Corsepius wrote:
Frankly speaking, the only situation I have found rpm.specs useful sofar, was to dig out the responsible maintainer of a package to get in direct contact to him.
On the other hand, this is the very reason I have sometimes considered leaving out my email address entirely from the specfile changelogs. I cannot promise personal individual support for users of the packages I maintain. Bugzillas and mailing lists are much better for that.
Back to the original topic: I think it is ok to remove the changelogs from the packages altogether provided that a equally easily accessible and working replacement to "rpm -q --changelog" is provided. But I don't have ideas how to implement that right now, sorry.
On Tue, 2005-02-01 at 21:46 +0200, Ville Skyttä wrote:
On Tue, 2005-02-01 at 15:46 +0100, Ralf Corsepius wrote:
Frankly speaking, the only situation I have found rpm.specs useful sofar, was to dig out the responsible maintainer of a package to get in direct contact to him.
On the other hand, this is the very reason I have sometimes considered leaving out my email address entirely from the specfile changelogs. I cannot promise personal individual support for users of the packages I maintain. Bugzillas and mailing lists are much better for that.
Back to the original topic: I think it is ok to remove the changelogs from the packages altogether provided that a equally easily accessible and working replacement to "rpm -q --changelog" is provided. But I don't have ideas how to implement that right now, sorry.
May have a solution to that for you. Panu worked up a script using the xml metadata to be able to do simple queries of the metadata like you would with rpm -q
repoquery --changelog yum
worked for me.
-sv
On Tue, 2005-02-01 at 15:07 -0500, seth vidal wrote:
On Tue, 2005-02-01 at 21:46 +0200, Ville Skyttä wrote:
Back to the original topic: I think it is ok to remove the changelogs from the packages altogether provided that a equally easily accessible and working replacement to "rpm -q --changelog" is provided. But I don't have ideas how to implement that right now, sorry.
May have a solution to that for you. Panu worked up a script using the xml metadata to be able to do simple queries of the metadata like you would with rpm -q
Except that where does the changelog info get into the metadata? The packages, so if the info is removed from the packages... oops. :) Of course the changelog metada could be generated from cvs or such.
repoquery --changelog yum
worked for me.
Oh well, since you advertised it publically I guess I could just as well make the blasted thing available finally :) It haven't touched it in quite a while but what little is there seems to still work: http://laiskiainen.org/repoquery/repoquery.py
Note that you need to run "yum makecache" first for it to work (and without that you'd grow old waiting anyway)
- Panu -
Except that where does the changelog info get into the metadata? The packages, so if the info is removed from the packages... oops. :) Of course the changelog metada could be generated from cvs or such.
Details! :)
But to be clear, getting the changelog and other metadata from non-pkg sources is probably going to have to happen.
Oh well, since you advertised it publically I guess I could just as well make the blasted thing available finally :) It haven't touched it in quite a while but what little is there seems to still work: http://laiskiainen.org/repoquery/repoquery.py
Note that you need to run "yum makecache" first for it to work (and without that you'd grow old waiting anyway)
it's only a year and a day you'd have to wait.
sheesh, you act like that's a long time :)
-sv
Once upon a time, seth vidal skvidal@phy.duke.edu said:
May have a solution to that for you. Panu worked up a script using the xml metadata to be able to do simple queries of the metadata like you would with rpm -q
repoquery --changelog yum
worked for me.
That's nice if you are working on a network-connected computer. Also, where will the changelog data come from?
Le mardi 01 février 2005 à 12:13 +0100, Ralf Corsepius a écrit :
I disagree. Adding %changelogs to specs is not any different from $Log$.
Having an entry in an rpm-header containing the last change might be useful for users being interested in the reason for a new rpm release, but I fail to understand why having a full %changelog-history inside of rpms or metadata files is useful.
All that means is you never maintained a complex RH/FC system for a long period.
rpm changelogs are the first line of defense to understand what the heck happened to hose a long-working system. On Raw Hide they are as vital as the admin coffee.
All that means is you never maintained a complex RH/FC system for a long period.
rpm changelogs are the first line of defense to understand what the heck happened to hose a long-working system. On Raw Hide they are as vital as the admin coffee.
And keeping the last 6 months or a year of them is fine.
but cmon - do we really need 7 yrs or more worth of changelogs IN the rpm?
-sv
Le mardi 01 février 2005 à 14:07 -0500, seth vidal a écrit :
All that means is you never maintained a complex RH/FC system for a long period.
rpm changelogs are the first line of defense to understand what the heck happened to hose a long-working system. On Raw Hide they are as vital as the admin coffee.
And keeping the last 6 months or a year of them is fine.
but cmon - do we really need 7 yrs or more worth of changelogs IN the rpm?
Well, there are still people that to focused Raw Hide upgrades on systems partially upgraded from RH 7.1 to 7.3 (not kidding at all, really)
And keeping the last 6 months or a year of them is fine.
but cmon - do we really need 7 yrs or more worth of changelogs IN the rpm?
Well, there are still people that to focused Raw Hide upgrades on systems partially upgraded from RH 7.1 to 7.3 (not kidding at all, really)
And why exactly do we care about them for purposes of this dicussion?
-sv
Le mardi 01 février 2005 à 14:14 -0500, seth vidal a écrit :
And keeping the last 6 months or a year of them is fine.
but cmon - do we really need 7 yrs or more worth of changelogs IN the rpm?
Well, there are still people that to focused Raw Hide upgrades on systems partially upgraded from RH 7.1 to 7.3 (not kidding at all, really)
And why exactly do we care about them for purposes of this dicussion?
Because they are real users. A few megs of cd space savings are not sufficient to justify putting part of our users in deep s*t.
And why exactly do we care about them for purposes of this dicussion?
Because they are real users. A few megs of cd space savings are not sufficient to justify putting part of our users in deep s*t.
Which part of the users would this be? The people using fedora in 'mission critical environments'? I coulda sworn there was something about fedora core not be intended for those environments on the front page.
Maybe I'm mistaken - but why are we planning and making adjustments for folks that the distro is EXPLICITLY not intended for?
-sv
Le mardi 01 février 2005 à 14:36 -0500, seth vidal a écrit :
And why exactly do we care about them for purposes of this dicussion?
Because they are real users. A few megs of cd space savings are not sufficient to justify putting part of our users in deep s*t.
Which part of the users would this be? The people using fedora in 'mission critical environments'? I coulda sworn there was something about fedora core not be intended for those environments on the front page.
Well you should know that once you release a software in the wild some people will put it to interesting uses (I'm sure you have a found remembrance of when people started yuming Raw Hide;)
Regards,
On Tue, 2005-02-01 at 20:44 +0100, Nicolas Mailhot wrote:
Le mardi 01 février 2005 à 14:36 -0500, seth vidal a écrit :
[snip]
Well you should know that once you release a software in the wild some people will put it to interesting uses (I'm sure you have a found remembrance of when people started yuming Raw Hide;)
And The Fedora Project most definitely does not need to make adjustments for these 'interesting uses' if those uses go directly against the warnings from the very inception of the project.
Le mardi 01 février 2005 à 14:46 -0500, Paul Iadonisi a écrit :
On Tue, 2005-02-01 at 20:44 +0100, Nicolas Mailhot wrote:
Le mardi 01 février 2005 à 14:36 -0500, seth vidal a écrit :
[snip]
Well you should know that once you release a software in the wild some people will put it to interesting uses (I'm sure you have a found remembrance of when people started yuming Raw Hide;)
And The Fedora Project most definitely does not need to make adjustments for these 'interesting uses' if those uses go directly against the warnings from the very inception of the project.
There was no "inception of Fedora project" FC is a Red Hat Linux child, it has benefited a lot from this fact, writing off all this history like this is very bad form.
Some form of Red Hat Linux continuity was a very big unwritten Fedora objective. And till recently (as lwn.net pointed out) there was precious little of anything else going on (not that people weren't working hard behind the scenes, but a large number of FC1/FC2/FC3 users weren't there for the yet-to-happen extras repository)
On Tue, 2005-02-01 at 21:15 +0100, Nicolas Mailhot wrote:
There was no "inception of Fedora project" FC is a Red Hat Linux child, it has benefited a lot from this fact, writing off all this history like this is very bad form.
Interesting rewriting or re-interpretation of history, there. Are you implying that there is, was, or should be no difference between Fedora Core and it's predecessor, Red Hat Linux from the project/product perspective? Or that White Box Enterprise Linux had no inception because it is *also* a child of Red Hat Linux?
Puh-leez. There most *certainly* *was* an inception of the Fedora Project.
Or are you just playing semantic games? Honest question, there.
Some form of Red Hat Linux continuity was a very big unwritten Fedora objective.
Sez who? To paraphrase the bugzilla mantra (if it ain't in bugzilla, it doesn't exist): If it ain't written, it ain't an objective. It's a 'nice to have' but not a 'must have'. And as far as support goes for use in mission critical roles or any enterprise usage of it, it's basically this: you're on your own. If it breaks, you get to keep both pieces. That doesn't happen often (arguably less than RHEL updates have caused in the past year or so), and we're all here on this list, I presume, to help make sure it doesn't happen often, but set your expectations accordingly.
And till recently (as lwn.net pointed out) there was precious little of anything else going on (not that people weren't working hard behind the scenes, but a large number of FC1/FC2/FC3 users weren't there for the yet-to-happen extras repository)
Eh? What exactly are you talking about?
Well you should know that once you release a software in the wild some people will put it to interesting uses (I'm sure you have a found remembrance of when people started yuming Raw Hide;)
Sure -but I can't plan or count on whatever someone might do.
I have to plan on intended use.
-sv
seth vidal wrote:
Which part of the users would this be? The people using fedora in 'mission critical environments'? I coulda sworn there was something about fedora core not be intended for those environments on the front page.
Maybe I'm mistaken - but why are we planning and making adjustments for folks that the distro is EXPLICITLY not intended for?
-sv
I agree. Fedora should follow its purposes. RPM Changelogs are important and should be supported for supported Fedora, maybe even the most recent entry into Fedora Legacy. Anyone upgrading from an ancient release expects to do a bit more research than `rpm -qi --changelog package`.
Extremely historic Changelogs are a nuisance that just add noise to the important information-- what has changed since the last RPM.
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
-sv
On Mon, 2005-01-31 at 19:09 -0500, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
Switching the rpm payload to bzip2 would probably give you much more space
FWIW: AFAIK, SuSE uses bzip2 payloads.
Ralf
Ralf Corsepius wrote:
On Mon, 2005-01-31 at 19:09 -0500, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
Switching the rpm payload to bzip2 would probably give you much more space
FWIW: AFAIK, SuSE uses bzip2 payloads.
Yep, and so does PLD.
Very very very foolish imho, as rpm is rate limited by decompression, and bzip2 uncompress is known 5-7 times slower than gzip.
BUt feel free to change Fedora Core to "Be just like SuSE" if you want.
PLD (my favorite distro, they are quiet and sensible in Poland ;-), is considering reverting to gzip when I pointed out that bzip2 was 5-7 times slower.
Add --stats, any install, run your own benchmark.
Personally, I think LZO might be preferable to either gzip of bzip2.
And the real issue is that zlib/bzip2/LZO are known not to rsync very well.
Well, that is known for gzip fer sure, I believe also true for bzip2/LZO ...
73 de Jeff
On Mon, 2005-01-31 at 19:47 -0500, Jeff Johnson wrote:
Ralf Corsepius wrote:
On Mon, 2005-01-31 at 19:09 -0500, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
Switching the rpm payload to bzip2 would probably give you much more space
FWIW: AFAIK, SuSE uses bzip2 payloads.
Yep, and so does PLD.
Very very very foolish imho, as rpm is rate limited by decompression, and bzip2 uncompress is known 5-7 times slower than gzip.
Well, people are complaining about sizes, bandwidth and diskspace ...
... not about installation speed.
OK, people are used to using gzip-payloads and probably would start to complain when installing bzip'ed rpms (I recall me having complained about SuSE when installing bzip'ed rpms on my ancient i586 notebook).
BUt feel free to change Fedora Core to "Be just like SuSE" if you want.
No, that's not my intention. It's just that I can't deny nor ignore having used SuSE for ~8 years.
PLD (my favorite distro, they are quiet and sensible in Poland ;-), is considering reverting to gzip when I pointed out that bzip2 was 5-7 times slower.
Add --stats, any install, run your own benchmark.
I did some checks comparing gzip vs. bzip on metadata repositories sometime last year. IIRC, I posted the results to this list.
The result was not as eye-striking as one might expect.
Ralf
I did some checks comparing gzip vs. bzip on metadata repositories sometime last year. IIRC, I posted the results to this list.
The result was not as eye-striking as one might expect.
I remember those benchmarks - and on slower machines the difference is HUGE.
45s difference on a slower machine, iirc.
I may not be remembering correctly, though.
-sv
Ralf Corsepius wrote:
Add --stats, any install, run your own benchmark.
I did some checks comparing gzip vs. bzip on metadata repositories sometime last year. IIRC, I posted the results to this list.
The result was not as eye-striking as one might expect.
Benchmarks are in the eye of the beholder. My 5-7x slower was done in rpm-3.0.5, immediately after putting bzip2 into rpm. I have had no reason to go back and recheck the data.
But --stats is in rpm, specifically for the purpose of measuring payload unpacking speed in situ (and identifying other bottlenecks)
I will believe that benchmark and no other.
Have fun!
73 de Jeff
Jeff Johnson n3npq@nc.rr.com:
Very very very foolish imho, as rpm is rate limited by decompression, and bzip2 uncompress is known 5-7 times slower than gzip.
Is that a problem? Processors are speeding up faster than media are getting larger -- thus, trdsing clocks for space seems like the right thing.
I'm open to counterargument here -- I just want to be sure you've thought through the tradeoffs completely.
Eric S. Raymond wrote:
Jeff Johnson n3npq@nc.rr.com:
Very very very foolish imho, as rpm is rate limited by decompression, and bzip2 uncompress is known 5-7 times slower than gzip.
Is that a problem? Processors are speeding up faster than media are getting larger -- thus, trdsing clocks for space seems like the right thing.
I'm open to counterargument here -- I just want to be sure you've thought through the tradeoffs completely.
Is memory a bigger problem than cpu? How about broadband vs. dialup?
I have no clue where the balance points are. My rpm job is objective measurements through insturmentation, and as reliable and stable an implementation as possible.
Go make up your own answers.
73 de Jeff
Jeff Johnson n3npq@nc.rr.com:
Is memory a bigger problem than cpu? How about broadband vs. dialup?
I have no clue where the balance points are. My rpm job is objective measurements through insturmentation, and as reliable and stable an implementation as possible.
Go make up your own answers.
OK, here is an answer.
We don't know exactly what the "right" tradeoff between space efficiency and decompression time is. But we *do* know which direction that tradeoff is heading.
Processor clocks are getting cheaper faster than memory is. Memory is getting cheaper faster than disk is. Disk is getting cheaper a *lot* faster than bits-per-second of bandwidth.
The "right" tradeoff is to use lots of cheap resources in order to be able to use fewer expensive resources. Therefore, as these trends continue, we want to spend processor clocks to decrease use of bandwidth.
Thus, bzip2 wins over gzip.
On Tue, 1 Feb 2005, Eric S. Raymond wrote:
OK, here is an answer.
Here's another one ;)
Processor clocks are getting cheaper faster than memory is. Memory is getting cheaper faster than disk is. Disk is getting cheaper a *lot* faster than bits-per-second of bandwidth.
Memory is getting bigger a lot faster than it is getting faster. Disk is getting bigger a lot faster than it is getting faster. No matter at which part of a computer system you look, space increases more than speed does.
The "right" tradeoff is to use lots of cheap resources in order to be able to use fewer expensive resources. Therefore, as these trends continue,
... we no longer need to care as much about the size of files, since the real bottleneck is how fast we can get at the data.
I know my ADSL can download bzip2 files faster than I can decompress them...
There is no perfect answer, but gzip seems to provide a good compromise between fast and small.
On Mon, Jan 31, 2005 at 09:44:21PM -0500, Eric S. Raymond wrote:
Is that a problem? Processors are speeding up faster than media are getting larger -- thus, trdsing clocks for space seems like the right thing.
Processors are close to stopped on growth.
1990 386DX40 150Mb 2005 AMD64@~3GHz 400Gb
Its actually not clear that this is true, especially as bzip2 performance is heavily RAM not cache dependant and RAM hasnt scaled at all well (damned speed of light is just too low...)
Alan
On Mon, 2005-01-31 at 21:44 -0500, Eric S. Raymond wrote:
Is that a problem? Processors are speeding up faster than media are getting larger -- thus, trdsing clocks for space seems like the right thing.
Processors are not speeding up at all, to a first approximation. Haven't you noticed? Moore's law stopped operating about two years ago.
<b
Bryan O'Sullivan bos@serpentine.com:
On Mon, 2005-01-31 at 21:44 -0500, Eric S. Raymond wrote:
Is that a problem? Processors are speeding up faster than media are getting larger -- thus, trdsing clocks for space seems like the right thing.
Processors are not speeding up at all, to a first approximation. Haven't you noticed? Moore's law stopped operating about two years ago.
Pedantry...
Yes, speed increases in single processors have stalled. Which is why there are a lot more SMP and multicore systems deployed now. Clocks per dollar seems to be continuing to rise on a roughly 2^n curve, even though clocks per processor aren't.
On Tue, 2005-02-01 at 15:15 -0500, Eric S. Raymond wrote:
[snip]
Processors are not speeding up at all, to a first approximation. Haven't you noticed? Moore's law stopped operating about two years ago.
Pedantry...
Yes, speed increases in single processors have stalled. Which is why there are a lot more SMP and multicore systems deployed now. Clocks per dollar seems to be continuing to rise on a roughly 2^n curve, even though clocks per processor aren't.
But...
Unless rpm and yum are multi-threaded, this makes little difference. Are they? Seth? Jeff?
Paul Iadonisi wrote:
On Tue, 2005-02-01 at 15:15 -0500, Eric S. Raymond wrote:
[snip]
Processors are not speeding up at all, to a first approximation. Haven't you noticed? Moore's law stopped operating about two years ago.
Pedantry...
Yes, speed increases in single processors have stalled. Which is why there are a lot more SMP and multicore systems deployed now. Clocks per dollar seems to be continuing to rise on a roughly 2^n curve, even though clocks per processor aren't.
But...
Unless rpm and yum are multi-threaded, this makes little difference. Are they? Seth? Jeff?
Hi,
Just , FYI,
yum, being python based might have issues with multi threaded operation.
From what I have seen in my zope digging is that python has a thing
called the GIL (or similar?), that for most applications prevents efficient use of SMP. For a while, SMP could make things slower, but I believe now that threaded python apps behave OK on most SMP platforms. According to the info, a complete re-hash of the python interpreter would be required to give it the ability to scale well using threads on SMP architectures. Most folk seem to recommend using a multi process model for python to exploit SMP.
Regards,
Darren Steven
Ralf Corsepius rc040203@freenet.de:
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
Switching the rpm payload to bzip2 would probably give you much more space
Yes. That would be attacking the problem at the right level.
On Mon, 2005-01-31 at 16:09, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
how much space do they eat currently? -- Fernando
On Mon, 2005-01-31 at 17:04 -0800, Fernando Lopez-Lezcano wrote:
On Mon, 2005-01-31 at 16:09, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
how much space do they eat currently? -- Fernando
well they're stored in at LEAST 3 different places on the cds:
1. the packages 2. the srpms 3. the rpmdb-fedora 4. maybe in hdlist2? 5. the metadata
I think that's more than enough locations :)
-sv
seth vidal wrote:
On Mon, 2005-01-31 at 17:04 -0800, Fernando Lopez-Lezcano wrote:
On Mon, 2005-01-31 at 16:09, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
how much space do they eat currently? -- Fernando
well they're stored in at LEAST 3 different places on the cds:
- the packages
- the srpms
- the rpmdb-fedora
- maybe in hdlist2?
- the metadata
I think that's more than enough locations :)
Let's move changelogs to specspo so that developers can write in their native language!
73 de Jeff, still in denial about changelog encoding rules.
On Mon, 2005-01-31 at 20:10 -0500, seth vidal wrote:
On Mon, 2005-01-31 at 17:04 -0800, Fernando Lopez-Lezcano wrote:
On Mon, 2005-01-31 at 16:09, seth vidal wrote:
Oh, no. *Bad* idea. It's an attack on the symptom, not the problem.
If changelogs are bloating the headers yum has to download, then strip them out when generating the yum headers.
it's not bloating the headers - well not JUST that.
I'm also thinking of decreasing the useles space eaten up by them on the cd isos.
how much space do they eat currently? -- Fernando
well they're stored in at LEAST 3 different places on the cds:
- the packages
- the srpms
- the rpmdb-fedora
I've checked one of my "longer running" packages (gimp, changelog since 1999) and it accounts for about 10K uncompressed (ungzipped) and 3K gzipped in packages that are roughly 10MB in size, even for the kernel it would be only 28/10K vs. 30M. This tiny amount of space saved just doesn't warrant the effort IMO, sorry. Even if it's just a few K * (1 srpm + n * (binary rpms + rpmdb)).
- maybe in hdlist2?
- the metadata
I don't think that changelogs should be stored at all in the (repo) metadata or hdlist2.
Nils
Jeff Johnson wrote:
seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
+1
In fact, it's silly to carry changelogs in packages, since packaging changes are far more easily read from e-mail, or from a web-site, or just about any other way than rpm -q --changelog pkg
The majority of content (note that there are definite execptions) in package changelogs is rather useless.
Here's an example from a random package, util-linux:
* Mon Jan 26 2004 Elliot Lee sopwith@redhat.com 2.12pre-3 - Provides: mount losetup
* Mon Jan 26 2004 Dan Walsh dwalsh@redhat.com 2.12pre-2 - Add multiple to /etc/pam.d/login for SELinux
* Thu Jan 15 2004 Elliot Lee sopwith@redhat.com 2.12pre-1 - 2.12pre-1 - Merge mount/losetup packages into the main package (#112324) - Lose separate
* Mon Nov 03 2003 Dan Walsh dwalsh@redhat.com 2.11y-35.sel - remove selinux code from login and use pam_selinux
* Thu Oct 30 2003 Dan Walsh dwalsh@redhat.com 2.11y-34.sel - turn on selinux
* Fri Oct 24 2003 Elliot Lee sopwith@redhat.com 2.11y-34 - Add BuildRequires: texinfo (from a bug# I don't remember) - Fix #90588 with mountman patch142.
* Mon Oct 06 2003 Dan Walsh dwalsh@redhat.com 2.11y-33 - turn off selinux
* Thu Sep 25 2003 Dan Walsh dwalsh@redhat.com 2.11y-32.sel - turn on selinux - remove context selection
* Fri Sep 19 2003 Elliot Lee sopwith@redhat.com 2.11y-31 - Add patch140 (alldevs) to fix #101772. Printing the total size of all devices was deemed a lower priority than having all devices (e.g. /dev/ida/c0d9) displayed.
* Fri Sep 12 2003 Dan Walsh dwalsh@redhat.com 2.11y-31 - turn off selinux
* Fri Sep 12 2003 Dan Walsh dwalsh@redhat.com 2.11y-30.sel - turn on selinux
* Fri Sep 05 2003 Elliot Lee sopwith@redhat.com 2.11y-28 - Fix #103004, #103954
* Fri Sep 05 2003 Dan Walsh dwalsh@redhat.com 2.11y-27 - turn off selinux
* Thu Sep 04 2003 Dan Walsh dwalsh@redhat.com 2.11y-26.sel - build with selinux
* Mon Aug 11 2003 Elliot Lee sopwith@redhat.com 2.11y-25 - Use urandom instead for mkcramfs
* Tue Jul 29 2003 Dan Walsh dwalsh@redhat.com 2.11y-24 - add SELINUX 2.5 support
* Wed Jul 23 2003 Elliot Lee sopwith@redhat.com 2.11y-22 - #100433 patch
With no offense whatsoever to anyone, I humbly submit that the comments in the changelog are of rather limited use to any non-redhat developer, and are totally useless to any end-user.
So perhaps changelogs should be nuked entirely, and handled ouside of package content, instead.
73 de Jeff
On Mon, 31 Jan 2005 17:43:32 -0500, Jeff Johnson n3npq@nc.rr.com wrote:
With no offense whatsoever to anyone, I humbly submit that the comments in the changelog are of rather limited use to any non-redhat developer, and are totally useless to any end-user.
I don't know about totally useless. I've refered to changelogs in the past to track down issues when troubleshooting issues concerned when a certain bugfix or security fix have been applied. Changelog entries that refer to specific bug numbers or CAN numbers can be quite helpful in this regard. Limited, but not totally useless value. I will conceed that I could probably use a web lookup for all this information if it were made the primary means of access to changelogs. And I would be more at ease with losing the changelogs in the package completely if package update notification texts containing the information were more closely aligned with package update mechanisms.
-jef
Changelog entries that refer to specific bug numbers or CAN numbers can be quite helpful in this regard.
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
Just before each FC release the security team here go through a few years of security issues normalized to CVE names and make a list of how each FC package fixed it ("not vulnerable due to upstream version" or "contains backported fix"). It helps catch any missing fixes too ;)
(This is something I'm thinking we'll try to do after our FC4 audit).
Cheers, Mark
On Tue, 1 Feb 2005 09:28:45 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
I look forward to building pathological packages that have a requires on a CVE name provides.
-jef
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
On Tue, 1 Feb 2005 09:28:45 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system
could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
On Tue, 2005-02-01 at 16:02 +0100, Arjan van de Ven wrote:
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
On Tue, 1 Feb 2005 09:28:45 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system
could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
I agree with Jeremy. I think this is data that should be housed outside of the package. We're going to need to figure out how to do this anyway.
-sv
On Tue, 2005-02-01 at 16:02 +0100, Arjan van de Ven wrote:
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system
could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
This sort of requires a way to handle packages that you don't install - for example package flurble needs an empty package not-flurble (which conflicts with flurble) so that when CAN-9999-999 is issued for flurble, which then means fedora-secure-system now requires CAN-9999-999, a new empty not-flurble can also provide the CVE name.
The alternative is that following a CVE issue everyone's box gets a (hopefully fixed) version of the vulnerable package even if they were not running in previously.
This makes my head hurt.
Nigel.
The alternative is that following a CVE issue everyone's box gets a (hopefully fixed) version of the vulnerable package even if they were not running in previously.
The real point of using Provides is simply to give a definitive label that a package contains a backported fix for a particular named security issue - not that a package is or isn't vulnerable to an issue, and not to help keep a system up to date with security issues, or help enforce any security policies - Project like OVAL (http://oval.mitre.org) are designed to do that sort of thing. The Provides would go away once the backported patch was removed (due to moving to a newer upstream version etc)
Right now to determine if a particular issue is fixed you need to search the changelog, and if nothing is mentioned, unpack the SRPM, then look in each of the patches to see if the CVE name is mentioned, and if not if the patches included vaugely matches the patch for the issue. We do this in our pre-release audit - packages are horribly inconsistant.
Mark
On Tue, 1 Feb 2005 16:12:14 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
Right now to determine if a particular issue is fixed you need to search the changelog, and if nothing is mentioned, unpack the SRPM, then look in each of the patches to see if the CVE name is mentioned, and if not if the patches included vaugely matches the patch for the issue. We do this in our pre-release audit - packages are horribly inconsistant.
I'm not sure how turning this into a provides garuntees consistency. It still comes down whether or not a packager sticks the provides in manually.. not much different than sticking in a note in the changelog. Either one is easily forgotten by a packager. Bugzilla is riddled with bugs about missing provides and requires that are manually created by the packager. I don't see how moving it out of the changelog creates a consistency win.
But please consider that by encoding this information into a provides you are polluting the provides namespace that depsolvers are using with information that you have no intention of using like a dependancy. This will lead to unexpected problems.. when 'some' packagers get the bright idea to actually do a dependancy on this, that depsolves will have to negotiated. Maybe if the intent of the provides doesn't include using it as part of depsolving.. maybe this is the wrong place to encode this information.
-jef
Jeff Spaleta wrote:
On Tue, 1 Feb 2005 16:12:14 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
Right now to determine if a particular issue is fixed you need to search the changelog, and if nothing is mentioned, unpack the SRPM, then look in each of the patches to see if the CVE name is mentioned, and if not if the patches included vaugely matches the patch for the issue. We do this in our pre-release audit - packages are horribly inconsistant.
I'm not sure how turning this into a provides garuntees consistency. It still comes down whether or not a packager sticks the provides in manually.. not much different than sticking in a note in the changelog. Either one is easily forgotten by a packager. Bugzilla is riddled with bugs about missing provides and requires that are manually created by the packager. I don't see how moving it out of the changelog creates a consistency win.
A packager who choses not to add the Provides: changes nothing, the audit will be done.
Notes in changelogs are not easily automated, nor does rpm have keyword indices for changelogs.
Bugzilla is riddle with missing Requires:, not missing Provides:, as packagers only notice what is obvious, and an unmatched Requires: is invariably full stop.
That is not the mechanism being suggested for CAN/CVE entries.
But please consider that by encoding this information into a provides you are polluting the provides namespace that depsolvers are using with information that you have no intention of using like a dependancy. This will lead to unexpected problems.. when 'some' packagers get the bright idea to actually do a dependancy on this, that depsolves will have to negotiated. Maybe if the intent of the provides doesn't include using it as part of depsolving.. maybe this is the wrong place to encode this information.
The provides name space is there to be used, "polluting" is bottom trolling.
73 de Jeff
On Tue, 01 Feb 2005 11:29:28 -0500, Jeff Johnson n3npq@nc.rr.com wrote:
The provides name space is there to be used, "polluting" is bottom trolling.
I think the discussion already generated in this thread about fedora-secure-system is an indication of the ploblematic nature of casting this information into the provides of Core packages. Provides are there to be used... surely.. and easily misused. Using this provides for internal audit sure seems like a slick solution for internal needs... but at the same time exposing this information to external users in a way that makes it quite easy to misuse as mechanism in depsolving. As soon as Core drops these provides in, packagers will start playing with providing fedora-secure-system like metapackages that use these provides. If the original intent for creating the provides is solely for internal auditing needs, is it appropriate to expose to everyone in this way?
-jef
metapackages that use these provides. If the original intent for creating the provides is solely for internal auditing needs, is it appropriate to expose to everyone in this way?
Actually it's to assert that we're providing a backported patch for a security issue in a package. This is incredibly useful to end users, especially those who have to respond to auditors (we get many requests along these lines, where a customer wants to be able to show an auditor that the old version of, say, OpenSSH, contains a fix for some particular named issue).
Cheers, Mark
On Tue, 1 Feb 2005 17:15:32 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
Actually it's to assert that we're providing a backported patch for a security issue in a package. This is incredibly useful to end users, especially those who have to respond to auditors (we get many requests along these lines, where a customer wants to be able to show an auditor that the old version of, say, OpenSSH, contains a fix for some particular named issue).
I relent.
-jef
Nigel Metheringham wrote:
On Tue, 2005-02-01 at 16:02 +0100, Arjan van de Ven wrote:
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system
could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
This sort of requires a way to handle packages that you don't install - for example package flurble needs an empty package not-flurble (which conflicts with flurble) so that when CAN-9999-999 is issued for flurble, which then means fedora-secure-system now requires CAN-9999-999, a new empty not-flurble can also provide the CVE name.
...
This makes my head hurt.
How about fedora-secure-system have Conflicts: flurble <= <vulnerable version> # CAN-9999-999
If a package is known to be vulnerable, it conflicts with a secure system.
Wouldn't that accomplish what you want? It will install in the absence of flurble, but if a vulnerable version is installed, it will (should?) cause an upgrade.
Hmm.... And if there is no upgrade available, maybe a remove... In fact, I kind of like that idea. Update fedora-secure-system immediately upon disclosure of a problem, even in absense of a fix. Sysadmins can decide to uninstall or remain insecure based on whatever their constraints are.
Thoughts?
Eli --------------------. "If it ain't broke now, Eli Carter \ it will be soon." -- crypto-gram eli.carter(a)inet.com `-------------------------------------------------
------------------------------------------------------------------------ Confidentiality Notice: This e-mail transmission may contain confidential and/or privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or reliance upon the contents of this e-mail message is strictly prohibited. If you have received this e-mail transmission in error, please reply to the sender, so that proper delivery can be arranged, and please delete the message from your computer. Thank you. Tektronix Texas, LLC formerly Inet Technologies, Inc. ------------------------------------------------------------------------
Eli Carter wrote:
How about fedora-secure-system have Conflicts: flurble <= <vulnerable version> # CAN-9999-999
If a package is known to be vulnerable, it conflicts with a secure system.
Some other ideas: fedora-secure-system Requires: fedora-secure-remote-root Requires: fedora-secure-local-root Requires: fedora-secure-remote-user Requires: fedora-secure-other # ?
fedora-secure-remote-root would conflict with all packages vulnerable to remote root exploits fedora-secure-local-root would conflict with all packages vulnerable to local root exploits. ... etc.
That would allow a sysadmin to take the approach of: "I trust all my users, but I can't afford to have any remote exploits, and I need minimal change" => install fedora-secure-remote-root and fedora-secure-remote-user, but not fedora-secure-local-root.
Thoughts?
Eli --------------------. "If it ain't broke now, Eli Carter \ it will be soon." -- crypto-gram eli.carter(a)inet.com `-------------------------------------------------
------------------------------------------------------------------------ Confidentiality Notice: This e-mail transmission may contain confidential and/or privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or reliance upon the contents of this e-mail message is strictly prohibited. If you have received this e-mail transmission in error, please reply to the sender, so that proper delivery can be arranged, and please delete the message from your computer. Thank you. Tektronix Texas, LLC formerly Inet Technologies, Inc. ------------------------------------------------------------------------
Eli Carter wrote:
Nigel Metheringham wrote:
On Tue, 2005-02-01 at 16:02 +0100, Arjan van de Ven wrote:
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
This sort of requires a way to handle packages that you don't install - for example package flurble needs an empty package not-flurble (which conflicts with flurble) so that when CAN-9999-999 is issued for flurble, which then means fedora-secure-system now requires CAN-9999-999, a new empty not-flurble can also provide the CVE name.
...
This makes my head hurt.
How about fedora-secure-system have Conflicts: flurble <= <vulnerable version> # CAN-9999-999
If a package is known to be vulnerable, it conflicts with a secure system.
Wouldn't that accomplish what you want? It will install in the absence of flurble, but if a vulnerable version is installed, it will (should?) cause an upgrade.
Hmm.... And if there is no upgrade available, maybe a remove... In fact, I kind of like that idea. Update fedora-secure-system immediately upon disclosure of a problem, even in absense of a fix. Sysadmins can decide to uninstall or remain insecure based on whatever their constraints are.
Package metadata is not sufficiently reliable to be trusted to automagically insturment package choices based on security tags.
Who *exactly* determines the reliability of the tags that are in Conflict:?
On my systems, that answer is: Me and me alone.
73 de Jeff
On Tue, 2005-02-01 at 16:02 +0100, Arjan van de Ven wrote:
On Tue, 2005-02-01 at 09:50 -0500, Jeff Spaleta wrote:
On Tue, 1 Feb 2005 09:28:45 +0000 (GMT), Mark J Cox mjc@redhat.com wrote:
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
I look forward to building pathological packages that have a requires on a CVE name provides.
fedora-secure-system
could require all the CVE's that are ciritical to be fixed yum update fedora-secure-system would then only pull security updates down....
This scheme just doesn't cut it because:
- you might need more than one package to fix a certain CVE - you might think you have fixed a certain CVE with one package revision, but you didn't, you'll have to issue an update but now the old package still claims to fix this particular CVE
To get it right, we have to keep this separate from the individual packages IMO. We could think of a fedora-secure-system package that grabs CVEs and which packages are believed to fix them at build time, then just conflicts with every "%name < %{?epoch:%{epoch}:}%{version}%{release}" of the involved packages.
Nils
On Tue, 2005-02-01 at 09:28 +0000, Mark J Cox wrote:
Changelog entries that refer to specific bug numbers or CAN numbers can be quite helpful in this regard.
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
This really feels like the wrong place to put this information. Then, if we're not vulnerable for whatever reason, the provides isn't there and people think that it is. So, now we have to do an update to add a provides. And even if we say that newer versions don't need it, people will want it because doing a two-step process of "check version, check CAN" means they'll only do one step ;)
This just feels like metadata that doesn't belong in the package to me...
Jeremy
Jeremy Katz wrote:
On Tue, 2005-02-01 at 09:28 +0000, Mark J Cox wrote:
Changelog entries that refer to specific bug numbers or CAN numbers can be quite helpful in this regard.
What would be incredibly useful is to move (to being a Provides) the CVE names for issues that we're including a backported fix for. Where we've moved to an upstream version that contains fixes those CVE names are less important as they can be deduced by a simple NV check.
This really feels like the wrong place to put this information. Then, if we're not vulnerable for whatever reason, the provides isn't there and people think that it is. So, now we have to do an update to add a provides. And even if we say that newer versions don't need it, people will want it because doing a two-step process of "check version, check CAN" means they'll only do one step ;)
Unlike, say Provides: shared-library or Provides: kernel-abi = 2.6 or Provide: LSB a CAN or CVE marker usually has a patch to package sources.
Since a patch forces a rebuild, adding the Provides: is no additional work. Yes, retrofiiting the scheme will involve arbitrary additions to existing packages, but that can't be helped.
What still would be lacking is a hard, well defined, test that, indeed, that the Provides: is accurate.
In other words, there is the risk of false comfort if onl the Provides: is checked, any package might supply a false Provides:. So the package or header signature check is a necessary part of verifying a CAN or CVE Provides:
Distributing the exploit with the Provides: would be one way to insure that the CAN or CVE Provides: could be tested widely (but distributing exploits has other issues of course ;-)
This just feels like metadata that doesn't belong in the package to me...
I'd say that a CAN/CVE marker belongs in a package because it is indicating that the package includeds the appropriate and approved patch.
Any other scheme to attach meta-metadata with a package will be harder to vet and maintain.
73 de Jeff
On Mon, 31 Jan 2005, Jeff Spaleta wrote:
On Mon, 31 Jan 2005 17:43:32 -0500, Jeff Johnson n3npq@nc.rr.com wrote:
With no offense whatsoever to anyone, I humbly submit that the comments in the changelog are of rather limited use to any non-redhat developer, and are totally useless to any end-user.
I don't know about totally useless. I've refered to changelogs in the past to track down issues when troubleshooting issues concerned when a certain bugfix or security fix have been applied. Changelog entries that refer to specific bug numbers or CAN numbers can be quite helpful in this regard. Limited, but not totally useless value. I will conceed that I could probably use a web lookup for all this information if it were made the primary means of access to changelogs. And I would be more at ease with losing the changelogs in the package completely if package update notification texts containing the information were more closely aligned with package update mechanisms.
Package changelogs are anything but useless, I refer to them all the time. Dunno if others have noticed but I think the rpm changelogs have become much more useful and informative now that the rawhide changelog deltas are mailed to a public list :)
Sure, go ahead and nuke ancient entries from specs. The info is available in cvs and old packages for archeologists to dig if interested, a normal user is only interested in the few last changes, eg "this update broke my xxxx - what changed?".
I personally wouldn't mind if rpm -q --changelog just fetched the changelog info directly from cvs (or web, or whatever) either, just as long as the information is handily available.
- Panu -
On Mon, Jan 31, 2005 at 05:43:32PM -0500, Jeff Johnson wrote:
So perhaps changelogs should be nuked entirely, and handled ouside of package content, instead.
Why not just removing the changelog from the rpm header and having them only accessible via the spec file in the src.rpm. There it is useful for packagers (inside and outside RH) and doesn't bother "normal users".
On Mon, 31 Jan 2005 23:58:26 +0100, Jos Vos jos@xos.nl wrote:
On Mon, Jan 31, 2005 at 05:43:32PM -0500, Jeff Johnson wrote:
So perhaps changelogs should be nuked entirely, and handled ouside of package content, instead.
Why not just removing the changelog from the rpm header and having them only accessible via the spec file in the src.rpm. There it is useful for packagers (inside and outside RH) and doesn't bother "normal users".
and prevents duplication of the changelog across all the binary subpackages from the same srpm, which Féliciano Matias brings up in this thread. Hmmm.... i think i like this.
-jef
On Jan 31, 2005, Jeff Spaleta jspaleta@gmail.com wrote:
On Mon, 31 Jan 2005 23:58:26 +0100, Jos Vos jos@xos.nl wrote:
On Mon, Jan 31, 2005 at 05:43:32PM -0500, Jeff Johnson wrote:
So perhaps changelogs should be nuked entirely, and handled ouside of package content, instead.
Why not just removing the changelog from the rpm header and having them only accessible via the spec file in the src.rpm. There it is useful for packagers (inside and outside RH) and doesn't bother "normal users".
and prevents duplication of the changelog across all the binary subpackages from the same srpm, which Féliciano Matias brings up in this thread. Hmmm.... i think i like this.
And if the metadata generator could somehow easily extract the changelog from the srpm and provide it in a separately-downloadable file, we could still have a simple command-line tool to get to the same info.
And if the metadata generator could somehow easily extract the changelog from the srpm and provide it in a separately-downloadable file, we could still have a simple command-line tool to get to the same info.
Well it does.
other.xml.gz is all changelog information right now.
but it can't REMOVE it from the srpm/rpm afterward, though.
yum generate-rss uses that info - where do you think the info at:
http://fedoraproject.org/infofeed/ comes from?
-sv
On Feb 1, 2005, seth vidal skvidal@phy.duke.edu wrote:
And if the metadata generator could somehow easily extract the changelog from the srpm and provide it in a separately-downloadable file, we could still have a simple command-line tool to get to the same info.
Well it does.
other.xml.gz is all changelog information right now.
Right. Because this info is in the rpms and the srpms headers. If it was removed from them, as suggested in the e-mail upthread I responded to, even in the portion I quoted in my e-mail, createrepo would need changes to get the data from elsewhere.
/me hands seth a pair of glasses :-)
other.xml.gz is all changelog information right now.
Right. Because this info is in the rpms and the srpms headers. If it was removed from them, as suggested in the e-mail upthread I responded to, even in the portion I quoted in my e-mail, createrepo would need changes to get the data from elsewhere.
/me hands seth a pair of glasses :-)
I must have misread then. I thought you were saying the opposite. Sorry about that.
But we're going to have to start dealing with metadata that doesn't exist in the package header soon, anyway.
we need to store all sorts of stuff in the metadata, actually. For example: - security alert information - urls of CVE notices - maybe relationships of packages or sets, etc.
so createrepo is going to have to learn how to get information from other places.
-sv
On 01/31/2005 10:43 PM, Jeff Johnson wrote:
Jeff Johnson wrote:
seth vidal wrote:
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
+1
In fact, it's silly to carry changelogs in packages, since packaging changes are far more easily read from e-mail, or from a web-site, or just about any other way than rpm -q --changelog pkg
Hmmm... I must be different then ;-)
Seriously, recently I used this command to find out whhat's changed in NetworkManager when apt decided to grab bind with it.
[And that was also when I decided to 'rpm -e NetworkManager' and manage my net manually (it's simple enough: no wireless, no plugging/unplugging, just two ethernet cards and dhcp).]
With no offense whatsoever to anyone, I humbly submit that the comments in the changelog are of rather limited use to any non-redhat developer, and are totally useless to any end-user.
Well, I consider myself mostly end-user...
So perhaps changelogs should be nuked entirely, and handled ouside of package content, instead.
Perhaps. Just a note that it *may* sometimes be useful even to end-user.
Regards, Dariusz
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
-sv
-- fedora-devel-list mailing list fedora-devel-list@redhat.com http://www.redhat.com/mailman/listinfo/fedora-devel-list
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
Why first? What's the loss in doing it and moving along?
the changelogs would be in cvs FOR EVER and in old srpms and on old isos and... and... and...
it seems like it is an immediate, not-huge win.
heck it doesn't even have to happen immediately - just the next time someone edits the spec file - just chop out all the old entries beyond 2 yrs.
if we did that right before every release it's easy to keep track of.
hell, once a year would be enough.
-sv
Le lundi 31 janvier 2005 à 17:36 -0500, seth vidal a écrit :
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
Why first? What's the loss in doing it and moving along? the changelogs would be in cvs FOR EVER and in old srpms and on old isos and... and... and...
Meet you in the server room where outbound access is disabled for security reasons and cvs is not installed on fileservers anyway.
Stop thinking like a developer with full software support infrastructure and net access. Field people rely on simple console text editors for a reason - shit happens, and complex systems fail more often than simple ones. Just assume no convenience will be available and you're be not far from the reality you have today in countless enterprise or home premises.
%changelog is perfectly adapted to rpm usage. This is not one of the "features" like rpm groups no one ever found a serious use for.
Meet you in the server room where outbound access is disabled for security reasons and cvs is not installed on fileservers anyway.
Stop thinking like a developer with full software support infrastructure and net access. Field people rely on simple console text editors for a reason - shit happens, and complex systems fail more often than simple ones. Just assume no convenience will be available and you're be not far from the reality you have today in countless enterprise or home premises.
%changelog is perfectly adapted to rpm usage. This is not one of the "features" like rpm groups no one ever found a serious use for.
I'm not talking about removing ALL of them. Just the REALLY old ones.
tell me - when was the last time you needed, in your server room, to know what changes were made in 1998 to a package?
-sv
Le mardi 01 février 2005 à 14:33 -0500, seth vidal a écrit :
Meet you in the server room where outbound access is disabled for security reasons and cvs is not installed on fileservers anyway.
Stop thinking like a developer with full software support infrastructure and net access. Field people rely on simple console text editors for a reason - shit happens, and complex systems fail more often than simple ones. Just assume no convenience will be available and you're be not far from the reality you have today in countless enterprise or home premises.
%changelog is perfectly adapted to rpm usage. This is not one of the "features" like rpm groups no one ever found a serious use for.
I'm not talking about removing ALL of them. Just the REALLY old ones.
tell me - when was the last time you needed, in your server room, to know what changes were made in 1998 to a package?
2000/2001 ?
Really, this is very package dependent - some subsystems are very tied to a particular distribution release (because they depend on very specific features), others packages can happily be dropped in 5-years- old systems after just a rebuild.
As a matter of fact, since a RHEL lifetime is 5 years truncating anything older than that would probably be ok. But there *will* be people who install a first-gen RHEL2.1 (because that's the version their OS dpt validated) near RHEL2.1 end-of-life who'll then update it partially or completely to the latest updates. So in some cases, 5y is a reasonable minimum.
Regards,
On Tue, 01 Feb 2005 20:41:56 +0100, Nicolas Mailhot Nicolas.Mailhot@laposte.net wrote:
As a matter of fact, since a RHEL lifetime is 5 years truncating anything older than that would probably be ok. But there *will* be people who install a first-gen RHEL2.1 (because that's the version their OS dpt validated) near RHEL2.1 end-of-life who'll then update it partially or completely to the latest updates. So in some cases, 5y is a reasonable minimum.
RHEL users and the timescales invovled there might be a valid concern, if the fedora package histories are deeply mingled with the RHEL package histories. If Red Hat packagers end up stripping items from fedora packages and them just needing to put the items back for rhel packages, that seems a bit wasteful concerning the small amount of data that has been quantified so far in this list. Depends on what Red Hat does internally as to whether chopping the changelogs in fedora will affect rhel users later on.
-jef
On Tue, 2005-02-01 at 15:03, Jeff Spaleta wrote:
On Tue, 01 Feb 2005 20:41:56 +0100, Nicolas Mailhot Nicolas.Mailhot@laposte.net wrote:
As a matter of fact, since a RHEL lifetime is 5 years truncating anything older than that would probably be ok. But there *will* be people who install a first-gen RHEL2.1 (because that's the version their OS dpt validated) near RHEL2.1 end-of-life who'll then update it partially or completely to the latest updates. So in some cases, 5y is a reasonable minimum.
RHEL users and the timescales invovled there might be a valid concern, if the fedora package histories are deeply mingled with the RHEL package histories. If Red Hat packagers end up stripping items from fedora packages and them just needing to put the items back for rhel packages, that seems a bit wasteful concerning the small amount of data that has been quantified so far in this list. Depends on what Red Hat does internally as to whether chopping the changelogs in fedora will affect rhel users later on.
RHEL users also have something that many fedora developers don't: the ability to use support (in one of at least 15 languages) to help answer questions that may or may not be covered by a truncated, full, or non-existent changelog, and to expect an answer that meets their satisfaction in a defined amount of time. I think it's best (especially on this list) to focus on what best supports the fedora community--users, developers, and those who provide bandwidth. If Fedora works better on the whole by changing changelog policy, I'm all for it. And if we find ourselves creating other mechanisms to help users support their systems (with or without paid support) that doesn't mean we failed. It means we're evolving.
M
I'm not talking about removing ALL of them. Just the REALLY old ones.
tell me - when was the last time you needed, in your server room, to know what changes were made in 1998 to a package?
2000/2001 ?
Right! But look what year it is now.
I'm talking about deleting changelog entries OLDER than a certain point.
Really, this is very package dependent - some subsystems are very tied to a particular distribution release (because they depend on very specific features), others packages can happily be dropped in 5-years- old systems after just a rebuild.
then like owen suggested: how about the most recent year or the last 10, whichever is more?
As a matter of fact, since a RHEL lifetime is 5 years truncating anything older than that would probably be ok. But there *will* be people who install a first-gen RHEL2.1 (because that's the version their OS dpt validated) near RHEL2.1 end-of-life who'll then update it partially or completely to the latest updates. So in some cases, 5y is a reasonable minimum.
This is not a discussion of rhel.
this is fedora-devel-list.
-sv
Once upon a time, seth vidal skvidal@phy.duke.edu said:
This is not a discussion of rhel.
Fedora Core objective #13: Form the basis of Red Hat's commercially supported operating system products.
What goes into RHEL is relevant to FC.
I'm talking about deleting changelog entries OLDER than a certain point.
I think this is a reasonable request. Delete everything older than 2/3 years from being copied into binary rpm packages. Otherwise we add this as additional overhead to each package maintainer to remove old lines and move them to some "history file" within the src.rpm.
greetings,
Florian La Roche
Florian La Roche wrote:
I'm talking about deleting changelog entries OLDER than a certain point.
I think this is a reasonable request. Delete everything older than 2/3 years from being copied into binary rpm packages. Otherwise we add this as additional overhead to each package maintainer to remove old lines and move them to some "history file" within the src.rpm.
greetings,
Florian La Roche
I would say, that rpm can strip everything but the last, say ten, entries from the binary rpms.
On Mon, 31 Jan 2005, seth vidal wrote:
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
Why first? What's the loss in doing it and moving along?
the changelogs would be in cvs FOR EVER and in old srpms and on old isos and... and... and...
it seems like it is an immediate, not-huge win.
heck it doesn't even have to happen immediately - just the next time someone edits the spec file - just chop out all the old entries beyond 2 yrs.
if we did that right before every release it's easy to keep track of.
hell, once a year would be enough.
This is exactly one of the things I wanted to add to the pre-processing of the SPEC files before building. I hope we can add a modular plugin-system to pydar that is able to pre-process SPEC files before building binary packages and before building source packages (changing Release, Vendor, Packager, Distribution, SPEC filenames, trimming changelog, omitting epochs for cAos, remapping build-requirements/requirements, maybe using another pre-processing language...)
-- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [all I want is a warm bed and a kind word and unlimited power]
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
That's just a little over 11 megabytes for all change logs on a FC3 system. I've also got non-Core packages installed which would increase that size slightly. That's also (obviously) including changelog entries which would not be stripped. I'm just guessing here, but I suspect that the total savings would be significantly smaller. I don't think the effort or loss of history would really be worth it.
On Mon, 31 Jan 2005 14:55:45 -0800, Shahms King shahms@shahms.com wrote:
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
On Mon, 31 Jan 2005 17:04:36 -0500, seth vidal skvidal@phy.duke.edu wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
That's just a little over 11 megabytes for all change logs on a FC3 system. I've also got non-Core packages installed which would increase that size slightly. That's also (obviously) including changelog entries which would not be stripped. I'm just guessing here, but I suspect that the total savings would be significantly smaller. I don't think the effort or loss of history would really be worth it.
I'll backpedal and agree with you. 11 MB is very negligible when only concerned with the size of FC3.
-- Shahms E. King shahms@shahms.com Multnomah ESD
Public Key: http://shahms.mesd.k12.or.us/~sking/shahms.asc Fingerprint: 1612 054B CE92 8770 F1EA AB1B FEAB 3636 45B2 D75B
-- fedora-devel-list mailing list fedora-devel-list@redhat.com http://www.redhat.com/mailman/listinfo/fedora-devel-list
shahms@shahms.com (Shahms King) writes:
[... removal of %changelog from rpms ...] It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
As they are stored compressed, it is much less:
| $ rpm -qpC --nodigest --nosignature *.rpm |LANG=C wc | 511863 2364195 15619260
| $ rpm -qpC --nodigest --nosignature *.rpm | gzip -9 - | LANG=C wc | 12338 67545 3136942
| $ rpm -qpC --nodigest --nosignature *.rpm | bzip2 -9 - | LANG=C wc | 6339 37933 1813872
(FC3 binaries rpms).
All this discussion because of 2-3 MB additional space?
Enrico
Enrico Scholz wrote:
shahms@shahms.com (Shahms King) writes:
[... removal of %changelog from rpms ...] It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
As they are stored compressed, it is much less:
Headers are compressed only in yum-2.0.x archives, no place else, certainly not *.rpm files nor in /var/lib/rpm/Packages nor in any transaction set memory.
| $ rpm -qpC --nodigest --nosignature *.rpm |LANG=C wc | 511863 2364195 15619260
| $ rpm -qpC --nodigest --nosignature *.rpm | gzip -9 - | LANG=C wc | 12338 67545 3136942
| $ rpm -qpC --nodigest --nosignature *.rpm | bzip2 -9 - | LANG=C wc | 6339 37933 1813872
(FC3 binaries rpms).
All this discussion because of 2-3 MB additional space?
Yep, with the correction above.
73 de Jeff
Enrico Scholz wrote:
shahms@shahms.com (Shahms King) writes:
[... removal of %changelog from rpms ...] It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
As they are stored compressed, it is much less:
| $ rpm -qpC --nodigest --nosignature *.rpm |LANG=C wc | 511863 2364195 15619260
| $ rpm -qpC --nodigest --nosignature *.rpm | gzip -9 - | LANG=C wc | 12338 67545 3136942
| $ rpm -qpC --nodigest --nosignature *.rpm | bzip2 -9 - | LANG=C wc | 6339 37933 1813872
(FC3 binaries rpms).
All this discussion because of 2-3 MB additional space?
Um, you might try --changelog instead:
$ rpm --help | grep changelog --changelog list change logs for this package
73 de Jeff
n3npq@nc.rr.com (Jeff Johnson) writes:
| $ rpm -qpC --nodigest --nosignature *.rpm |LANG=C wc | 511863 2364195 15619260
... Um, you might try --changelog instead:
ouch... 'rpm --changelog' is one of the most used rpm commands, so I assumed that everybody would have
| # cat /etc/popt | rpm alias -C --changelog
on his system.
Enrico
On Mon, 2005-01-31 at 14:55 -0800, Shahms King wrote:
On Mon, 2005-01-31 at 17:34 -0500, Nick Bargnesi wrote:
It'd be interesting to see how much space something like this would save first.
$ rpm -qa --changelog | wc -c 11597562
That's just a little over 11 megabytes for all change logs on a FC3 system. I've also got non-Core packages installed which would increase that size slightly. That's also (obviously) including changelog entries which would not be stripped. I'm just guessing here, but I suspect that the total savings would be significantly smaller. I don't think the effort or loss of history would really be worth it.
For sake of completeness:
$ rpm -qa --changelog | LANG=C wc 246340 1183597 7835570 $ rpm -qa --changelog | sed -n '/^* .+ 200[45]/,/^* .+ 200[0-3]/p' | LANG=C wc 63836 321574 2182535
The thing that fascinates me though:
$ rpm -qa --changelog | grep -ic "To be, or not to be" 0
Le lundi 31 janvier 2005 à 17:04 -0500, seth vidal a écrit :
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
The "problem" I have about changelog is its duplications.
[fmatias@one i386]$ rpm -q --changelog xorg-x11 | wc 5369 35773 250956 <= big changelog [fmatias@one i386]$ rpm -q -a | grep xorg-x11 | xargs rpm -q --changelog | wc 75166 500822 3513384 <= very very big changelog (several times the same changelog).
On Tue, 2005-02-01 at 00:02 +0100, Féliciano Matias wrote:
The "problem" I have about changelog is its duplications.
[fmatias@one i386]$ rpm -q --changelog xorg-x11 | wc 5369 35773 250956 <= big changelog [fmatias@one i386]$ rpm -q -a | grep xorg-x11 | xargs rpm -q --changelog | wc 75166 500822 3513384 <= very very big changelog (several times the same changelog).
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
Nils
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
People might be able to install only one of the sub-packages.
greetings,
Florian La Roche
On Wed, 2005-02-02 at 15:13 +0100, Florian La Roche wrote:
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
People might be able to install only one of the sub-packages.
Um, I expressed myself not clearly. Of course the package files all contain the changelog, the normalization would only happen on the RPM DB level.
Nils
Nils Philippsen wrote:
On Wed, 2005-02-02 at 15:13 +0100, Florian La Roche wrote:
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
People might be able to install only one of the sub-packages.
Um, I expressed myself not clearly. Of course the package files all contain the changelog, the normalization would only happen on the RPM DB level.
Which cannot happen, because immutable header regions contain changelogs, and the blob is signature and/or digest verified.
73 de Jeff
On Wed, 2005-02-02 at 16:24 -0500, Jeff Johnson wrote:
Nils Philippsen wrote:
On Wed, 2005-02-02 at 15:13 +0100, Florian La Roche wrote:
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
People might be able to install only one of the sub-packages.
Um, I expressed myself not clearly. Of course the package files all contain the changelog, the normalization would only happen on the RPM DB level.
Which cannot happen, because immutable header regions contain changelogs, and the blob is signature and/or digest verified.
I'm fully aware of that this would need serious work and likely incompatible changes to the RPM DB. You can easily reconstruct that single blob as it is now from a split header + changelog and verify the resulting blob against the signature/digest.
Nils
Nils Philippsen wrote:
On Wed, 2005-02-02 at 16:24 -0500, Jeff Johnson wrote:
Nils Philippsen wrote:
On Wed, 2005-02-02 at 15:13 +0100, Florian La Roche wrote:
Then the question is: why is the same changelog, i.e. the changelog of sub packages with the same source package, stored for each package individually? It could be stored once and be referenced from each sub package. It gets deleted once all references on it vanish.
People might be able to install only one of the sub-packages.
Um, I expressed myself not clearly. Of course the package files all contain the changelog, the normalization would only happen on the RPM DB level.
Which cannot happen, because immutable header regions contain changelogs, and the blob is signature and/or digest verified.
I'm fully aware of that this would need serious work and likely incompatible changes to the RPM DB. You can easily reconstruct that single blob as it is now from a split header + changelog and verify the resulting blob against the signature/digest.
You cannot reconstruct from missing information at all.
Splitting information between package header and Something Else Instead is a rather awkward verify to develop trust in.
Adding changelogs outside of packages is what was suggested, and removing or at least truncating, changelogs too. Either is trivially done, and requires zero changes to existing rpm or rpmdb.
Or add changelogs to specspo, as that has been needed quite some time now, so that developers can describe changes in their native language.
Only your blind expectation that packages contain changelogs forevermore is is preventing you from seeing the simplicity of it all imho.
But, presumably, you are fully aware of that too. RFE in bugzilla to start getting your changes to RPMDB to normalize changelogs on installed client machines properly prioritized please.,
73 de Jeff
On Wed, 2005-02-02 at 17:07 -0500, Jeff Johnson wrote:
Nils Philippsen wrote:
[...]
I'm fully aware of that this would need serious work and likely incompatible changes to the RPM DB. You can easily reconstruct that single blob as it is now from a split header + changelog and verify the resulting blob against the signature/digest.
Apologies for not labeling my posts "theoretical musings about how changelogs could be made less redundant in the DB".
You cannot reconstruct from missing information at all.
If the header is now (everything else + changelog) and changelog gets split out I don't see how there is any missing information -- you just grab "everything else" and the changelog and have your blob again.
Splitting information between package header and Something Else Instead is a rather awkward verify to develop trust in.
If it were done (note the use of the subjunctive, theoretical discussion and all that) it would obviously have to be a two-way unique mapping from single blob to header plus split out redundant data (let's not talk about the changelog, as this only distracts) and vice versa.
Adding changelogs outside of packages is what was suggested, and removing or at least truncating, changelogs too. Either is trivially done, and requires zero changes to existing rpm or rpmdb.
Agreed, but I guess the opinions on whether this is desirable or not differ quite dramatically. On the one hand there's the "remove unneeded cruft" argument and on the other the "I might need just that bit of info when I'm cut off the net". I guess we won't bring these two together unless there is an easy way to take the changelogs with you when doing an "offline" job and to easily get the right changelog for the packages taken with you, i.e. verification that this changelog belongs to that package would also need to be thought about.
Or add changelogs to specspo, as that has been needed quite some time now, so that developers can describe changes in their native language.
Hmm. If you have one isolated developer, then yes. But if you have multiple people working on one package, it will (sensibly) lead to the common denominator being used (which is English) as it is done with string to be translated in applications.
Only your blind expectation that packages contain changelogs forevermore is is preventing you from seeing the simplicity of it all imho.
As outlined above, your simplicity for rpm and rpmdb is bought by complication elsewhere. Not that that'd necessarily be a bad thing ;-).
But, presumably, you are fully aware of that too. RFE in bugzilla to start getting your changes to RPMDB to normalize changelogs on installed client machines properly prioritized please.,
Hey, don't take it personal, it's only a theoretical discussion from my side, which I should have said before. Sorry.
Nils
Nils Philippsen wrote:
On Wed, 2005-02-02 at 17:07 -0500, Jeff Johnson wrote:
Hey, don't take it personal, it's only a theoretical discussion from my side, which I should have said before. Sorry.
NP. What needs to be done is to change rpmbuild, not rpm or thr rpmdb, to truncate changelogs at a certain point in time. That way you can have all changelogs in the spec file (so no mass *.spec churn needs to happen) and fewer changelogs in packages (which would be smaller).
Normalization is trivial possible if changelogs were saved as keys, rather than values, as normalization automagically happens with keys. That can be easily done through configuration, adding "Changelogs" one place and doing --rebuilddb.
Alas that normalizes, but saves no space, as headers still contain the original content. That is actually a feature, not a flaw, imho, because the indices can always be reconstructed by doing --rebuilddb at any time, the redundancy is more useful than whatever amount of space is saved.
Historically, rpm used to have a a parameter that truncated changelogs at N when installing packages.
That was a perfectly sensible optimization when disk space was less cheap, until header immutable regions came along.
Whether changelogs should be part of an immutable region or not is an open question too. It is (and was) certainly possible to define a header immutable region without including changelogs content, which would permit truncation or other forms of normalization, editing header content while installing.
I chose to put *all* tags into a header immutable region so that I would not have to have the discussion about which tags go where.
For example, the content in changelogs, if not hardened by digest and/or signature, might be part of a socially engineered exploit to disguise a maliciously modified package. It's very hard not believe what you read.
73 de Jeff
Jeff Johnson wrote:
For example, the content in changelogs, if not hardened by digest and/or signature, might be part of a socially engineered exploit to disguise a maliciously modified package. It's very hard not believe what you read.
/subliminal messages on
RPM IS GOOD FOR YOU, RPM IS GOOD FOR YOU, RPM IS GOOD FOR YOU...
/subliminal messages off
Sorry, couldn't resist. ;)
Read ya, Phil
On Thu, 2005-02-03 at 08:19 -0500, Jeff Johnson wrote:
Whether changelogs should be part of an immutable region or not is an open question too. It is (and was) certainly possible to define a header immutable region without including changelogs content, which would permit truncation or other forms of normalization, editing header content while installing.
I chose to put *all* tags into a header immutable region so that I would not have to have the discussion about which tags go where.
For example, the content in changelogs, if not hardened by digest and/or signature, might be part of a socially engineered exploit to disguise a maliciously modified package. It's very hard not believe what you read.
Well, I didn't propose anything of that sort (i.e. changelog outside of what is digested/signed) ;-). What I meant was that it is irrelevant whether you sign/digest an actually existing stream of bytes which contains the changelog or the result of a function which puts together this stream from changelog and the remainder of the header.
Nils
Nils Philippsen wrote:
On Thu, 2005-02-03 at 08:19 -0500, Jeff Johnson wrote:
Whether changelogs should be part of an immutable region or not is an open question too. It is (and was) certainly possible to define a header immutable region without including changelogs content, which would permit truncation or other forms of normalization, editing header content while installing.
I chose to put *all* tags into a header immutable region so that I would not have to have the discussion about which tags go where.
For example, the content in changelogs, if not hardened by digest and/or signature, might be part of a socially engineered exploit to disguise a maliciously modified package. It's very hard not believe what you read.
Well, I didn't propose anything of that sort (i.e. changelog outside of what is digested/signed) ;-). What I meant was that it is irrelevant whether you sign/digest an actually existing stream of bytes which contains the changelog or the result of a function which puts together this stream from changelog and the remainder of the header.
Yep, one can reassemble a header from components, and verify blobs.
That was the context of my previous comments: you cannot reassemble a blob unless the components are actually present, and there almost certainly will be some way for separate changelogs to go AWOL preventing reassembly.
Splitting changelogs out, but not changing how digest/signature are performed on headers, adds complexity and additional failure modes where there are none now, that are hard to "trust" for no perceivable gain to verifiability other than compatibility with the exsisting mechanism.
Move changelogs out of headers, or truncate to acceptable length, is better imho.
Or RFE an explicit mechanism for moving changelogs out of headers and normalizing content.
73 de Jeff
On Thu, 2005-02-03 at 12:39 -0500, Jeff Johnson wrote:
Nils Philippsen wrote:
On Thu, 2005-02-03 at 08:19 -0500, Jeff Johnson wrote:
Whether changelogs should be part of an immutable region or not is an open question too. It is (and was) certainly possible to define a header immutable region without including changelogs content, which would permit truncation or other forms of normalization, editing header content while installing.
I chose to put *all* tags into a header immutable region so that I would not have to have the discussion about which tags go where.
For example, the content in changelogs, if not hardened by digest and/or signature, might be part of a socially engineered exploit to disguise a maliciously modified package. It's very hard not believe what you read.
Well, I didn't propose anything of that sort (i.e. changelog outside of what is digested/signed) ;-). What I meant was that it is irrelevant whether you sign/digest an actually existing stream of bytes which contains the changelog or the result of a function which puts together this stream from changelog and the remainder of the header.
Yep, one can reassemble a header from components, and verify blobs.
That was the context of my previous comments: you cannot reassemble a blob unless the components are actually present, and there almost certainly will beAWOL some way for separate changelogs to go AWOL preventing reassembly.
Agreed.
Splitting changelogs out, but not changing how digest/signature are performed on headers, adds complexity and additional failure modes where there are none now, that are hard to "trust" for no perceivable gain to verifiability other than compatibility with the exsisting mechanism.
Move changelogs out of headers, or truncate to acceptable length, is better imho.
Or RFE an explicit mechanism for moving changelogs out of headers and normalizing content.
Just musing ;-): Individual signatures on each header component, along with a signed list of components that should be present. That way, if something goes corrupt, you can find out what is broken ("URL not ok") unless the list gets damaged and a list should be a smaller target to be hit by random disaster than a complete header blob. This of course doesn't bring any more security where malice is involved, but I can as easily corrupt a complete header blob as I can the list or other single components, so nothing lost here.
Nils
Nils Philippsen wrote:
On Thu, 2005-02-03 at 12:39 -0500, Jeff Johnson wrote:
Just musing ;-): Individual signatures on each header component, along with a signed list of components that should be present. That way, if
Smells too much like DNSSec to me.
Ever tried to babysit a DNSSEC config? PITA ...
something goes corrupt, you can find out what is broken ("URL not ok") unless the list gets damaged and a list should be a smaller target to be hit by random disaster than a complete header blob. This of course doesn't bring any more security where malice is involved, but I can as easily corrupt a complete header blob as I can the list or other single components, so nothing lost here.
Hint: encrypted/signed files and certificate management are far more interesting problems.
So is exploding header meatadata into LDAP or WebDAV attributes.
73 de Jeff
On Mon, 2005-01-31 at 17:04 -0500, seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
I like having the full changelogs in the spec files. Perhaps with the new CVS setup we could switch to having external changelogs and the same benefits.I wouldn't like to just go through all the packages and lop them off.
But I'd agree that shipping the whole changelog in the RPM header is silly. I wonder if it's possible to strip all but the last 10 entries / 6 months (whichever is more) as part of the build process?
Regards, Owen
I'm always a vaguely amused by the gtk2 changelog ... it has 630 lines of %changelog over 7 years, and finishes with:
* Thu Mar 12 1998 Marc Ewing marc@redhat.com - Reworked to integrate into gtk+ source tree - Truncated ChangeLog. Previous Authors: Trond Eivind Glomsrod teg@pvv.ntnu.no Michael K. Johnson johnsonm@redhat.com Otto Hammersmith otto@redhat.com
It can't have been more than 50 lines long at that point.
On 1 Feb 2005, at 00:45, Owen Taylor wrote:
On Mon, 2005-01-31 at 17:04 -0500, seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
I like having the full changelogs in the spec files. Perhaps with the new CVS setup we could switch to having external changelogs and the same benefits.I wouldn't like to just go through all the packages and lop them off.
Perhaps you could leave the full ChangeLog in the .spec file, but instruct rpmbuild to completely remove the ChangeLog except the last 10 entries.
On 01/31/2005 11:09:18 PM, Felipe Alfaro Solana wrote:
Perhaps you could leave the full ChangeLog in the .spec file, but instruct rpmbuild to completely remove the ChangeLog except the last 10 entries.
or last two years - which sometimes is less than 10 entries. I find them useful - to see how packagers dealt with specific changes in the distro layout/functionallity in the past. But over two years is not that useful.
On 01/31/2005 11:09:18 PM, Felipe Alfaro Solana wrote:
Perhaps you could leave the full ChangeLog in the .spec file, but instruct rpmbuild to completely remove the ChangeLog except the last 10 entries.
or last two years - which sometimes is less than 10 entries. I find them useful - to see how packagers dealt with specific changes in the distro layout/functionallity in the past. But over two years is not that useful.
For the PostgreSQL RPMs I only kept the changelog for the current major version. I put a pointer at the bottom to look at the previous versions. Otherwise it would be over a thousand lines of changelog by now.
This makes sense for PostgreSQL, which typically has had quite significant spec file revisions between major versions, things like entire subpackages being removed, added, etc. The build process has had to be completely started from scratch before due to improvements in the build (versions prior to 7.1 took significant steps to fully build all the various pieces; beginning with 7.1 the build was simplified somewhat. The 6.3.x builds can only be described as baroque.)
On Mon, 2005-01-31 at 17:04 -0500, seth vidal wrote:
Hi folks, this is a touch silly but possibly useful and it would definitely cut down on the old crap blocking up cdroms.
how about if we kill all rpm spec file changelog entries OLDER than 2 years.
they'll still live on in older srpms and rpms but it'd be a useful reduction and it would make the specfiles that much smaller, along with the rpm headers.
thoughts?
+1
I never understood their usefulness, but with *.specs in CVS they have become even more useless.
Ralf
Would commenting out the older changelogs just be enough to keep them out of the headers but still keep them in the spec file?
-
Alan Cox -
Alexandre Oliva -
Arjan van de Ven -
Axel Thimm -
Bryan O'Sullivan -
Chris Adams -
Dag Wieers -
Dariusz J. Garbowski -
Darren Steven -
Dimitrie O. Paun -
Eli Carter -
Enrico Scholz -
Eric S. Raymond -
Felipe Alfaro Solana -
Fernando Lopez-Lezcano -
Florian La Roche -
Féliciano Matias -
Harald Hoyer -
Jef Spaleta -
Jeff Johnson -
Jeremy Katz -
Jos Vos -
Lamar Owen -
Mark Cox -
Michael A. Peters -
Michael Schwendt -
Michael Tiemann -
Nick Bargnesi -
Nicolas Mailhot -
Nigel Metheringham -
Nils Philippsen -
Owen Taylor -
Panu Matilainen -
Paul Iadonisi -
Phil Knirsch -
Ralf Corsepius -
Rik van Riel -
Schlaegel -
seth vidal -
Shahms King -
Ville Skyttä -
Ziga Mahkovec