3:01 p.m.
I've seen various recommendations for adding swap files after
system creation, and it occurs to me that the "standard" technique
may not be the most efficient. I realize that one rarely creates
swap files, but nonetheless on occasion one needs to "precreate"
some file or other, then do something to it, like mkfs etc.
Anyway, mostly trying to improve my general knowledge of
how best to use the abilities of the system, and my understanding
of the relative merits of doing things one way vs. another,
not trying to speed up rarely performed procedures.
The standard technique is to do something like
$ dd if=/dev/zero of=/path/to/new/file bs=1024 count=524288
$ mkswap /path/to/new/file
to create a 512MB file. The second command may be different,
depending upon the circumstances, but the technique remains
the same. In effect, the new file gets written twice.
It occurs to me that one could, instead, do
$ dd if=/dev/zero of=/path/to/new/file bs=1M seek=511 count=1
$ mkswap /path/to/new/file
and have the same results, requiring only writing the file
once.
$ time dd if=/dev/zero of=file1 bs=1024 count=524288
524288+0 records in
524288+0 records out
real 0m17.898s
user 0m0.528s
sys 0m4.895s
$ time /sbin/mkswap file1
Setting up swapspace version 1, size = 536866 kB
real 0m0.432s
user 0m0.000s
sys 0m0.024s
$ time dd if=/dev/zero of=file2 bs=1M seek=511 count=1
1+0 records in
1+0 records out
real 0m0.077s
user 0m0.000s
sys 0m0.007s
$ time /sbin/mkswap file2
Setting up swapspace version 1, size = 536866 kB
real 0m0.024s
user 0m0.000s
sys 0m0.004s
$ diff -s file1 file2
Files file1 and file2 are identical
My guess is that when the swap file with a "hole" first gets used, there
will be a long(ish?) pause while some part or parts of the sparse file
get filled in. This is not so good for a swap file, but when one
is actually going to rewrite most of the file anyway, and is only
using the file itself as more or less an indicator of the size, then
it might make sense.
Comments?
Mike
--
p="p=%c%s%c;main(){printf(p,34,p,34);}";main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
4:25 p.m.
Mike McCarty wrote, On 03/27/2009 04:01 PM:
I've seen various recommendations for adding swap files after
system creation, and it occurs to me that the "standard" technique
may not be the most efficient. I realize that one rarely creates
swap files, but nonetheless on occasion one needs to "precreate"
some file or other, then do something to it, like mkfs etc.
Anyway, mostly trying to improve my general knowledge of
how best to use the abilities of the system, and my understanding
of the relative merits of doing things one way vs. another,
not trying to speed up rarely performed procedures.
The standard technique is to do something like
$ dd if=/dev/zero of=/path/to/new/file bs=1024 count=524288
$ mkswap /path/to/new/file
to create a 512MB file. The second command may be different,
depending upon the circumstances, but the technique remains
the same. In effect, the new file gets written twice.
It occurs to me that one could, instead, do
$ dd if=/dev/zero of=/path/to/new/file bs=1M seek=511 count=1
$ mkswap /path/to/new/file
and have the same results, requiring only writing the file
once.
<SNIP>
My guess is that when the swap file with a "hole" first gets used, there
will be a long(ish?) pause while some part or parts of the sparse file
get filled in. This is not so good for a swap file, but when one
is actually going to rewrite most of the file anyway, and is only
using the file itself as more or less an indicator of the size, then
it might make sense.
Comments?
Mike
This brings up a question for me...
If using the first method, while in use or during the mkswap command, does the
bits written to the file end up at the same physical locations as the 0s
they are replacing?
if they do, then the first method on a disk that is not fragmented, would give
you a contiguous swap space.
if they don't, then like you say, what is the point of writing all the 0s to disk?
So then do we have to think about our file systems?
msdos/vfat would write to the same locations I think.
ext2 would write to the same locations I think.
ext3 without journaling, would write to the same locations I think.???
ext3 with journaling, pathology???
ext4 with/without journaling, I have not used or read about.???
riser???
others???
--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter
5:44 p.m.
Todd Denniston wrote:
This brings up a question for me...
If using the first method, while in use or during the mkswap command,
does the bits written to the file end up at the same physical locations
as the 0s they are replacing?
Yes - you are saving back to the same file. Not like
a work
processor that makes a new file, and then renames it to the name of
the original file.
if they do, then the first method on a disk that is not fragmented,
would give you a contiguous swap space.
if they don't, then like you say, what is the point of writing all the
0s to disk?
So then do we have to think about our file systems?
msdos/vfat would write to the same locations I think.
ext2 would write to the same locations I think.
ext3 without journaling, would write to the same locations I think.???
This is the
same as ext2. (ext3 is ext2 with journaling.)
ext3 with journaling, pathology???
ext4 with/without journaling, I have not used or read about.???
riser???
others???
Journaling should not make a difference. It is still reading/writing
the same file on the disk.
I am not sure how to explain this. You are accessing the file just
like you would access a hard drive. You are doing the equivalent of
accessing the track and sector of a hard drive. (Note: hard drives
are accessed as files under Linux/UNIX.) So if you read one block
from the file, modify it, and write it back to the same place, it is
also on the same place on the hard drive as the old block was.
Mikkel
--
Do not meddle in the affairs of dragons,
for thou art crunchy and taste good with Ketchup!
11:36 a.m.
Mikkel L. Ellertson wrote, On 03/27/2009 06:44 PM:
Todd Denniston wrote:
> This brings up a question for me...
> If using the first method, while in use or during the mkswap command,
> does the bits written to the file end up at the same physical locations
> as the 0s they are replacing?
Yes - you are saving back to the same file. Not like a work
processor that makes a new file, and then renames it to the name of
the original file.
> if they do, then the first method on a disk that is not fragmented,
> would give you a contiguous swap space.
> if they don't, then like you say, what is the point of writing all the
> 0s to disk?
>
> So then do we have to think about our file systems?
> msdos/vfat would write to the same locations I think.
> ext2 would write to the same locations I think.
> ext3 without journaling, would write to the same locations I think.???
This is the same as ext2. (ext3 is ext2 with journaling.)
> ext3 with journaling, pathology???
> ext4 with/without journaling, I have not used or read about.???
> riser???
> others???
>
Journaling should not make a difference. It is still reading/writing
the same file on the disk.
I am not sure how to explain this. You are accessing the file just
like you would access a hard drive. You are doing the equivalent of
accessing the track and sector of a hard drive. (Note: hard drives
are accessed as files under Linux/UNIX.) So if you read one block
from the file, modify it, and write it back to the same place, it is
also on the same place on the hard drive as the old block was.
Mikkel
My confusion sets in when I start thinking about the way the Journaling file
system works, which I probably do not understand fully.
I thought that when you did a write, the file system (ext[34] in journal mode)
would actually write to the journal area (which might be on different physical
media) and then do the real write to the final physical media and after that
was done it would mark the journal as done.... which, to me, might mean it was
not writing to the same PHYSICAL location in the media.
So yes everything is still a file to Unix, but are the gymnastics that are
going on in the Journaling file system driver making all the nice setup not
worth while?
Not enough time for me to investigate properly.
--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter
5:26 p.m.
On 27Mar2009 14:01, Mike McCarty <Mike.McCarty(a)sbcglobal.net> wrote:
I've seen various recommendations for adding swap files after
system creation, and it occurs to me that the "standard" technique
may not be the most efficient. I realize that one rarely creates
swap files, but nonetheless on occasion one needs to "precreate"
some file or other, then do something to it, like mkfs etc.
[...]
The standard technique is to do something like
$ dd if=/dev/zero of=/path/to/new/file bs=1024 count=524288
$ mkswap /path/to/new/file
to create a 512MB file. The second command may be different,
depending upon the circumstances, but the technique remains
the same. In effect, the new file gets written twice.
No. mkswap just marks up the control areas of th file for use, like mkfs
marks up the control areas of a filesystem partition for use. Neither
writes the _whole_ file, just a small section of it.
So you're only writing the file once (with dd), and marking up a small
area of it for use (with mkswap).
It occurs to me that one could, instead, do
$ dd if=/dev/zero of=/path/to/new/file bs=1M seek=511 count=1
$ mkswap /path/to/new/file
and have the same results, requiring only writing the file
once.
No. It only looks the same from the outside.
What you have done here is make a _sparse_ file. A UNIX file only
allocates data blocks for areas that have been written to. By writing a
single chunk of data at the end, you have not allocated all the data
blocks. A swap area _must_ have all the blocks allocated. Indeed,
"man mkswap" even says this:
To setup a swap file, it is necessary to create that file before
initializing it with mkswap , e.g. using a command like
# dd if=/dev/zero of=swapfile bs=1024 count=65536
Note that a swap file must not contain any holes (so, using cp(1)
to create the file is not acceptable).
A sparse file is a file with holes.
My guess is that when the swap file with a "hole" first
gets used, there
will be a long(ish?) pause while some part or parts of the sparse file
get filled in. This is not so good for a swap file, but when one
is actually going to rewrite most of the file anyway, and is only
using the file itself as more or less an indicator of the size, then
it might make sense.
Comments?
Leaving aside the possibilty that the swap system will simply refuse
to work on a file with holes, which it may, you also get much more
contiguous file if you preallocate. Data blocks allocated later must
come from where the free space is at that time. Might be anywhere.
Cheers,
--
Cameron Simpson <cs(a)zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/
Good judgement comes from experience. Experience comes from bad judgement.
6:11 a.m.
Mike McCarty wrote:
$ diff -s file1 file2
Files file1 and file2 are identical
Identical? Try this:
du file1 file2
and you will see that one of them has not allocated disk blocks,
it is sparse (has holes).
IIRC sparse files are unacceptable for swap (the recent "swap files
are as fast as swap partition" feature actually means that the kernel
identifies where the file is on the disk and uses those blocks as a
sort of "fragmented partition").
And even if it works, it's a bad idea: the swap file will allocate
blocks in a very fragmented way in the future
(and there would be a "out-of-space" risk at the filesystem level
during a swapout..., brrrr).
You are right in one thing: writing the zeroes should not be
needed; it happens to be that writing something is the
only way to allocate disk blocks in most filesystems
(persistent preallocation in ext4 will change this).
My guess is that when the swap file with a "hole" first
gets used, there
will be a long(ish?) pause while some part or parts of the sparse file
get filled in. This is not so good for a swap file, but when one
is actually going to rewrite most of the file anyway, and is only
using the file itself as more or less an indicator of the size, then
it might make sense.
In theory, but you would risk: fragmentation and possible disk-full.
Best regards.
--
Roberto Ragusa mail at robertoragusa.it
5531
days inactive
5534
days old
5 comments
5 participants
participants (5)
-
Cameron Simpson -
Mike McCarty -
Mikkel -
Roberto Ragusa -
Todd Denniston