4:57 p.m.
Okay its come to my attention that the readahead configs have a
difficult time being kept in sync as package updates roll out. For fc6
right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package
control of the default configs? Is a readahead.d/ structure
appropriate here?
I want to point out that we lose significant value with read-ahead if
we can't ensure that the listings will be in-sync as package updates
are issued. And re-spinning readahead updates periodically to attempt
to sync a centralized config file is horrible inefficient and may
never be perfectly in-sync considering the churn of updates during a
release cycle.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=203329
-jef
5:09 p.m.
On Sun, 2006-11-12 at 13:57 -0900, Jeff Spaleta wrote:
Can we update readahead's implementation so we can get per
package
control of the default configs? Is a readahead.d/ structure
appropriate here?
I think this would be the Right Way(tm) to do it. Without native support
for it, we could still at least cat all of those files into the proper
readahead listing before running it.
--
Peter Gordon (codergeek42)
GnuPG Public Key ID: 0xFFC19479 / Fingerprint:
DD68 A414 56BD 6368 D957 9666 4268 CB7A FFC1 9479
My Blog: http://thecodergeek.com/blog/
5:15 p.m.
On 11/12/06, Jeff Spaleta <jspaleta(a)gmail.com> wrote:
Okay its come to my attention that the readahead configs have a
difficult time being kept in sync as package updates roll out. For fc6
right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package
control of the default configs? Is a readahead.d/ structure
appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was
added for fc6.... the problem is package maintainers aren't using it
yet. I guess what I need to do is parse the existing config file...
identify individual packages that could drop files in readahead.d/
and poke the appropriate maintainers in the eye about dropping a file
in there as part of package payloading.
Anyone want to help me construct the list of packages that could make
sure of readahead.d/ on a per package basis based on the default
readahead configs we have right now?
-jef"First up... the firefox maintainer... where's my eye poking
stick"spaleta
2:06 a.m.
On Sun, 2006-11-12 at 13:57 -0900, Jeff Spaleta wrote:
Okay its come to my attention that the readahead configs have a
difficult time being kept in sync as package updates roll out. For fc6
right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package
control of the default configs? Is a readahead.d/ structure
appropriate here?
maybe.
the downside is that you now add extra costs (seeks :) which.. well
readahead was trying to avoid. So if this is going to get used massively
it's less certain things will gain as much as before...
3:48 a.m.
On 11/12/06, Arjan van de Ven <arjan(a)fenrus.demon.nl> wrote:
the downside is that you now add extra costs (seeks :) which.. well
readahead was trying to avoid. So if this is going to get used massively
it's less certain things will gain as much as before...
I'm not saying use it for everything.. im just saying lets make sure
that the default items we have now in the readahead config actually
stay in sync with the libraries on disk so users can derive consistent
benefit from it over time. My current favorite example is firefox.
The libraries are listed in the default readahead config..but now that
firefox has seen an update, the listings are no longer in-sync with
the files on disk.. because the library locations are in a versioned
directory tree...so firefox won't get a readahead boost on an updated
system.
I understand the seek time issue associated with doing readahead
willy-nilly, so if you can think of a way to centralize a default
config and prevent out-of-sync conditions like we have with firefox at
the moment I'm all ears. Please go ahead and take a lot at the
listing in default.later and see if anything else besides firefox is
currently out of sync on an updated fc6 install.
-jef
3:48 a.m.
Hi.
On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
the downside is that you now add extra costs (seeks :) which.. well
readahead was trying to avoid. So if this is going to get used
massively it's less certain things will gain as much as before...
Maybe run a script on system shutdown which reads all the files under
/etc/readahead.d and creates a single config file for readahead to
read on the next boot?
4:07 a.m.
On Mon, 2006-11-13 at 10:48 +0100, Ralf Ertzinger wrote:
Hi.
On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
> the downside is that you now add extra costs (seeks :) which.. well
> readahead was trying to avoid. So if this is going to get used
> massively it's less certain things will gain as much as before...
Maybe run a script on system shutdown which reads all the files under
/etc/readahead.d and creates a single config file for readahead to
read on the next boot?
I really like this idea; it's a simple "cat" and it can be done at a
time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by
blocknumber" to be done at this point and taken out of the critical
latency part......
great idea!
10 a.m.
On Mon, Nov 13, 2006 at 11:07:41AM +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 10:48 +0100, Ralf Ertzinger wrote:
> Hi.
>
> On Mon, 13 Nov 2006 09:06:20 +0100, Arjan van de Ven wrote:
>
> > the downside is that you now add extra costs (seeks :) which.. well
> > readahead was trying to avoid. So if this is going to get used
> > massively it's less certain things will gain as much as before...
>
> Maybe run a script on system shutdown which reads all the files under
> /etc/readahead.d and creates a single config file for readahead to
> read on the next boot?
I really like this idea; it's a simple "cat" and it can be done at a
time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by
blocknumber" to be done at this point and taken out of the critical
latency part......
Good idea. Added to "TODO" list:
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156442
Karel
--
Karel Zak <kzak(a)redhat.com>
10:22 a.m.
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta <jspaleta(a)gmail.com> wrote:
>Okay its come to my attention that the readahead configs have a
>difficult time being kept in sync as package updates roll out. For fc6
>right now for example readahead is out of sync with firefox libraries.
>
>Can we update readahead's implementation so we can get per package
>control of the default configs? Is a readahead.d/ structure
>appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was
added for fc6.... the problem is package maintainers aren't using it
yet. I guess what I need to do is parse the existing config file...
identify individual packages that could drop files in readahead.d/
and poke the appropriate maintainers in the eye about dropping a file
in there as part of package payloading.
Anyone want to help me construct the list of packages that could make
sure of readahead.d/ on a per package basis based on the default
readahead configs we have right now?
I don't have machine with full FC6 installation, but you can create
the list very easily by shell command:
for f in `cat /etc/readahead.d/default.*`; do rpm -q --queryformat "%{NAME}\n"
-f $f 2> /dev/null; done | sort -u
-jef"First up... the firefox maintainer... where's my eye
poking
stick"spaleta
The problem is not firefox only. It's pretty generic problem, because
we have versions filenames for libraries:
$ grep '/usr/lib/lib.*[0-9].*' /etc/readahead.d/default.* | wc -l
136
It's means almost all things in /etc/readahead.d/default.* could be
obsolete after an update (except stable things like /bin/*).
Karel
--
Karel Zak <kzak(a)redhat.com>
11:39 a.m.
On 11/13/06, Karel Zak <kzak(a)redhat.com> wrote:
It's means almost all things in /etc/readahead.d/default.* could
be
obsolete after an update (except stable things like /bin/*).
I agree... which is why i think its important to have this discussion
in the list.. instead of just in the bugzilla report. The rpm query
is a neat trick. If you want to pump up a few throwaway readahead
packages to test new ideas to solve this issue, I'll be happy to offer
my system up for inadvertent destruction testing them.
-jef
9:23 p.m.
Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
I really like this idea; it's a simple "cat" and it can
be done at a
time where latency doesn't matter... (even in cron.daily)
Oh... this opens more options. This also allows the "sort by
blocknumber" to be done at this point and taken out of the critical
latency part......
great idea!
Of course, that then makes shutdown take twice as long.
What do you do if you have a box where readahead makes boot slower?
Bill
9:29 p.m.
On Mon, Nov 13, 2006 at 10:23:24PM -0500, Bill Nottingham wrote:
What do you do if you have a box where readahead makes boot slower?
rpm -e readahead?
:)
--
Matthew Miller mattdm(a)mattdm.org <http://mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
9:33 p.m.
Matthew Miller (mattdm(a)mattdm.org) said:
On Mon, Nov 13, 2006 at 10:23:24PM -0500, Bill Nottingham wrote:
> What do you do if you have a box where readahead makes boot slower?
rpm -e readahead?
You can, but it does seem odd to install something tha makes the
system less efficient. I should do more testing on laptops; I know
at least on some it slows things down.
Bill
9:43 p.m.
Of course, that then makes shutdown take twice as long.
In most cases (no changes to the files since last shutdown), then
a 'make' can shorten the latency to just a 'stat()' of each file.
The whole computation can run in parallel with all other
activities during shutdown, until just before the unmounts.
What do you do if you have a box where readahead makes boot slower?
Delete enough items from the list of filenames to be processed.
--
9:46 p.m.
On Mon, Nov 13, 2006 at 10:33:14PM -0500, Bill Nottingham wrote:
> > What do you do if you have a box where readahead makes boot
slower?
> rpm -e readahead?
You can, but it does seem odd to install something tha makes the
system less efficient. I should do more testing on laptops; I know
at least on some it slows things down.
Okay, I admit I actually take the more drastic solution of removing it from
comps.xml. I don't remember the numbers, but it was definitely making my FC4
laptop boot more slowly. I haven't given it another chance.... And on
non-laptop systems that don't reboot often, it's basically pointless anyway.
--
Matthew Miller mattdm(a)mattdm.org <http://mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
1:15 a.m.
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
> I really like this idea; it's a simple "cat" and it can be done at a
> time where latency doesn't matter... (even in cron.daily)
>
> Oh... this opens more options. This also allows the "sort by
> blocknumber" to be done at this point and taken out of the critical
> latency part......
>
> great idea!
Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :)
cron.daily/weekly is less so ;)
1:35 a.m.
On 11/14/2006 07:15 AM, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
> Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
>> I really like this idea; it's a simple "cat" and it can be done at
a
>> time where latency doesn't matter... (even in cron.daily)
>>
>> Oh... this opens more options. This also allows the "sort by
>> blocknumber" to be done at this point and taken out of the critical
>> latency part......
>>
>> great idea!
> Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :)
cron.daily/weekly is less so ;)
+1 for cron
-1 for shutdown -> what about boxens, which are shutdown only on kernel
updates? (like mine)
Regards,
Dariusz
Send instant messages to your online friends http://uk.messenger.yahoo.com
5:12 a.m.
On Tue, 2006-11-14 at 08:15 +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
> Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
> > I really like this idea; it's a simple "cat" and it can be done
at a
> > time where latency doesn't matter... (even in cron.daily)
> >
> > Oh... this opens more options. This also allows the "sort by
> > blocknumber" to be done at this point and taken out of the critical
> > latency part......
> >
> > great idea!
>
> Of course, that then makes shutdown take twice as long.
if you do it at shutdown, which is again a latency path :)
cron.daily/weekly is less so ;)
Or post-{install,remove} of the packages that put files in readahead.d.
There's no need to run the script if nothing has changed, and if you
upgrade affected packages and reboot before the next day, the libs and
config will be out of sync.
9:22 a.m.
Mark Rosenstand napsal(a):
On Tue, 2006-11-14 at 08:15 +0100, Arjan van de Ven wrote:
> On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
>> Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
>>> I really like this idea; it's a simple "cat" and it can be done
at a
>>> time where latency doesn't matter... (even in cron.daily)
>>>
>>> Oh... this opens more options. This also allows the "sort by
>>> blocknumber" to be done at this point and taken out of the critical
>>> latency part......
>>>
>>> great idea!
>> Of course, that then makes shutdown take twice as long.
> if you do it at shutdown, which is again a latency path :)
> cron.daily/weekly is less so ;)
Or post-{install,remove} of the packages that put files in readahead.d.
There's no need to run the script if nothing has changed,
Does prelink
overwrite the modified files in place, or does it create
new versions as separate files and rename()s them?
Mirek
6:52 a.m.
I've created a new project page for readahead (with jkeating's
help:-). It's fine if you edit the wiki page and contribute there.
http://hosted.fedoraproject.org/projects/readahead/
Note: GIT repo will be available next week.
Karel
On Sun, Nov 12, 2006 at 01:57:57PM -0900, Jeff Spaleta wrote:
Okay its come to my attention that the readahead configs have a
difficult time being kept in sync as package updates roll out. For fc6
right now for example readahead is out of sync with firefox libraries.
Can we update readahead's implementation so we can get per package
control of the default configs? Is a readahead.d/ structure
appropriate here?
I want to point out that we lose significant value with read-ahead if
we can't ensure that the listings will be in-sync as package updates
are issued. And re-spinning readahead updates periodically to attempt
to sync a centralized config file is horrible inefficient and may
never be perfectly in-sync considering the churn of updates during a
release cycle.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=203329
-jef
--
fedora-devel-list mailing list
fedora-devel-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
--
Karel Zak <kzak(a)redhat.com>
4:52 p.m.
On Tue, Nov 14, 2006 at 08:15:26AM +0100, Arjan van de Ven wrote:
On Mon, 2006-11-13 at 22:23 -0500, Bill Nottingham wrote:
> Arjan van de Ven (arjan(a)fenrus.demon.nl) said:
> > I really like this idea; it's a simple "cat" and it can be done
at a
> > time where latency doesn't matter... (even in cron.daily)
> >
> > Oh... this opens more options. This also allows the "sort by
> > blocknumber" to be done at this point and taken out of the critical
> > latency part......
> >
> > great idea!
>
> Of course, that then makes shutdown take twice as long.
Definitely no. We don't talk about full readahead.
I've implemented the idea and it seems that we can use it during
regular shutdown.
It's very fast, because almost all files from readahead lists are
already in the cache (and don't forget we don't call readahead(2) and
read full files. We need open(), stat() and read first block for each
file only. This is 10x faster than full readahead which we know from
system start up).
if you do it at shutdown, which is again a latency path :)
cron.daily/weekly is less so ;)
Well, some numbers:
./readahead --timing --verbose \
--output /etc/readahead.d/early.sorted \
--sort /etc/readahead.d/default.early
List: /etc/readahead.d/default.early
Time (read lists): 0 ms
Create list with 850 files
Time (read blocks): 103 ms <---------
Read blocks for 726/850 files (65778 KB)
List: /etc/readahead.d/default.later
Time (read lists): 2 ms
Create list with 3757 files
Time (read blocks): 281 ms <----------
Read blocks for 3056/3757 files (110159 KB)
The worst numbers which I've ever seen was 2500ms (sum from both lists)
and it was on an laptop with a lazy HDD.
Also, the idea that we can run it as parallel process is useful.
If you want to play with new readahead code, see:
http://people.redhat.com/kzak/readahead/readahead-11276006.tar.gz
Karel
--
Karel Zak <kzak(a)redhat.com>
3:41 a.m.
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
On 11/12/06, Jeff Spaleta <jspaleta(a)gmail.com> wrote:
>Okay its come to my attention that the readahead configs have a
>difficult time being kept in sync as package updates roll out. For fc6
>right now for example readahead is out of sync with firefox libraries.
>
>Can we update readahead's implementation so we can get per package
>control of the default configs? Is a readahead.d/ structure
>appropriate here?
Sorry... i didnt look hard enough... the readahead.d structure was
added for fc6.... the problem is package maintainers aren't using it
yet. I guess what I need to do is parse the existing config file...
identify individual packages that could drop files in readahead.d/
and poke the appropriate maintainers in the eye about dropping a file
in there as part of package payloading.
Anyone want to help me construct the list of packages that could make
sure of readahead.d/ on a per package basis based on the default
readahead configs we have right now?
Update: I've wrote a small and simple readahead-auditd that is able
to collect all filenames from boot process. It means everyone will be
able to generate unique list for his Fedora. And also I can maintain
default lists more effective now. I'm going to release an
experimental readahead package with this solution to FC7 next week.
Karel
--
Karel Zak <kzak(a)redhat.com>
8:09 a.m.
At 10:41 AM +0100 2/14/07, Karel Zak wrote:
On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
> On 11/12/06, Jeff Spaleta <jspaleta(a)gmail.com> wrote:
> >Okay its come to my attention that the readahead configs have a
> >difficult time being kept in sync as package updates roll out. For fc6
> >right now for example readahead is out of sync with firefox libraries.
> >
> >Can we update readahead's implementation so we can get per package
> >control of the default configs? Is a readahead.d/ structure
> >appropriate here?
>
> Sorry... i didnt look hard enough... the readahead.d structure was
> added for fc6.... the problem is package maintainers aren't using it
> yet. I guess what I need to do is parse the existing config file...
> identify individual packages that could drop files in readahead.d/
> and poke the appropriate maintainers in the eye about dropping a file
> in there as part of package payloading.
>
> Anyone want to help me construct the list of packages that could make
> sure of readahead.d/ on a per package basis based on the default
> readahead configs we have right now?
Update: I've wrote a small and simple readahead-auditd that is able
to collect all filenames from boot process. It means everyone will be
able to generate unique list for his Fedora. And also I can maintain
default lists more effective now. I'm going to release an
experimental readahead package with this solution to FC7 next week.
Would this work on FC6 as well? If the package can build cleanly on FC6
I'd try it out.
Does the daemon keep running after booting finishes? Will it have anything
more to do then?
--
____________________________________________________________________
TonyN.:' <mailto:tonynelson@georgeanelson.com>
' <http://www.georgeanelson.com/>
11:11 a.m.
On Wed, Feb 14, 2007 at 09:09:22AM -0500, Tony Nelson wrote:
At 10:41 AM +0100 2/14/07, Karel Zak wrote:
>On Sun, Nov 12, 2006 at 02:15:57PM -0900, Jeff Spaleta wrote:
>> On 11/12/06, Jeff Spaleta <jspaleta(a)gmail.com> wrote:
>> >Okay its come to my attention that the readahead configs have a
>> >difficult time being kept in sync as package updates roll out. For fc6
>> >right now for example readahead is out of sync with firefox libraries.
>> >
>> >Can we update readahead's implementation so we can get per package
>> >control of the default configs? Is a readahead.d/ structure
>> >appropriate here?
>>
>> Sorry... i didnt look hard enough... the readahead.d structure was
>> added for fc6.... the problem is package maintainers aren't using it
>> yet. I guess what I need to do is parse the existing config file...
>> identify individual packages that could drop files in readahead.d/
>> and poke the appropriate maintainers in the eye about dropping a file
>> in there as part of package payloading.
>>
>> Anyone want to help me construct the list of packages that could make
>> sure of readahead.d/ on a per package basis based on the default
>> readahead configs we have right now?
>
> Update: I've wrote a small and simple readahead-auditd that is able
> to collect all filenames from boot process. It means everyone will be
> able to generate unique list for his Fedora. And also I can maintain
> default lists more effective now. I'm going to release an
> experimental readahead package with this solution to FC7 next week.
Would this work on FC6 as well? If the package can build cleanly on FC6
I'd try it out.
No problem. I use FC6 too.
Does the daemon keep running after booting finishes? Will it have
anything
more to do then?
The default behaviour (depends on setting in a config file) will be
automated finish few minutes after boot. It should be a good way how
collect things that depend on user's habits (for example I start
gnome-terminal, firefox and mutt after computer startup).
I assume two steps:
1) collect filenames from boot process
2) post-processing -- things like sort by first block position
All this things should be on demand. Later we can thing about more
transparent and automated solution (or about completely different
solution like fcache ;-).
Karel
--
Karel Zak <kzak(a)redhat.com>
12:43 p.m.
At 6:11 PM +0100 2/14/07, Karel Zak wrote:
On Wed, Feb 14, 2007 at 09:09:22AM -0500, Tony Nelson wrote:
> At 10:41 AM +0100 2/14/07, Karel Zak wrote:
...
> > Update: I've wrote a small and simple readahead-auditd
that is able
> > to collect all filenames from boot process. It means everyone will be
> > able to generate unique list for his Fedora. And also I can maintain
> > default lists more effective now. I'm going to release an
> > experimental readahead package with this solution to FC7 next week.
>
> Would this work on FC6 as well? If the package can build cleanly on FC6
> I'd try it out.
No problem. I use FC6 too.
Good, thanks.
> Does the daemon keep running after booting finishes? Will it
have anything
> more to do then?
The default behaviour (depends on setting in a config file) will be
automated finish few minutes after boot. It should be a good way how
collect things that depend on user's habits (for example I start
gnome-terminal, firefox and mutt after computer startup).
I assume two steps:
1) collect filenames from boot process
2) post-processing -- things like sort by first block position
All this things should be on demand. Later we can thing about more
transparent and automated solution (or about completely different
solution like fcache ;-).
I think you're saying that the daemon will terminate after collecting data
during boot, but the readahead.whenever files will only be regenerated when
a separate command is run. Thus, after installing, say, a new Firefox, the
readahead.d/default.later file will have stale entries in it until the user
notices and runs that command. I expect it will become clear when I
examine the source.
--
____________________________________________________________________
TonyN.:' <mailto:tonynelson@georgeanelson.com>
' <http://www.georgeanelson.com/>
2:23 p.m.
On 2/14/07, Karel Zak <kzak(a)redhat.com> wrote:
Update: I've wrote a small and simple readahead-auditd that is
able
to collect all filenames from boot process. It means everyone will be
able to generate unique list for his Fedora. And also I can maintain
default lists more effective now. I'm going to release an
experimental readahead package with this solution to FC7 next week.
Are you interested in collecting those 'unique' lists for some
unscientific comparison of what
the common boot time items are out in the wild?
-jef
4:14 p.m.
On Wed, Feb 14, 2007 at 01:43:51PM -0500, Tony Nelson wrote:
I think you're saying that the daemon will terminate after
collecting data
during boot, but the readahead.whenever files will only be regenerated when
I'm not sure if we need to run the daemon for every boot. I don't
know how expensive is our audit system... we will see.
a separate command is run. Thus, after installing, say, a new
Firefox, the
readahead.d/default.later file will have stale entries in it until the user
Prepare a new version of the readahead.d/default.later file is
very simple (one qsort) and I think it will be possible to do it
automatically. So, without a separate command.
More attractive is to generate final version of the lists where
filenames are sorted by first data block -- see "fast mode" in
https://hosted.fedoraproject.org/projects/readahead/browser/README.
Now in FC6 readahead supports the "full mode" only. The readahead in
FC7 will support both modes, but regenerate lists for "fast mode" is
more expensive -- it seems like a good job for cron or some manually
started command.
notices and runs that command. I expect it will become clear when I
examine the source.
Yeah, that's always more clear than my mails. Sorry ;-)
Karel
--
Karel Zak <kzak(a)redhat.com>
6274
days inactive
6368
days old
28 comments
13 participants
participants (13)
-
Arjan van de Ven -
Bill Nottingham -
Dariusz J. Garbowski -
Jef Spaleta -
Jeffrey Ollie -
John Reiser -
Karel Zak -
Mark Rosenstand -
Matthew Miller -
Miloslav Trmač -
Peter Gordon -
Ralf Ertzinger -
Tony Nelson