8:58 a.m.
Hi Fedora developers,
I would like suggest you to take a look at bmaptool, which you can use
for flashing Fedora ISO images to USB sticks (or other block devices).
Let me set the scene. In order to flash a Fedora ISO image, typically
first download the image and then use dd to copy it to the USB stick.
Bmaptool does this, but
a) a lot faster (several times faster if you have the bmap file)
b) simpler, because it can flash directly from an URL (decompress
on-the fly too)
c) simpler, because it verifies data checksum automatically (only
if bmap file is present)
d) safer, since it won't destroy your mounted partition if you specify
wrong block device (say, for example, /dev/sda being the rootfs)
So what I am trying to "sell" here is a tool which flashes several times
faster than dd, which is safe, and can flash from http/ftp/https/ssh
URLs directly, and decompress gz/bz2/xz/gzip/tgz/tar.{gz/bz2/zx} files
on-the-fly.
We are using this in Tizen IVI, and I'd like to see other distros
using/supporting it. Or at least checking it.
The key element for bmaptool is the bmap file, which Fedora does not
produce for the ISO images. So bmaptool is not so useful, but the
community could find the tool useful and start providing the bmap
files.
Anyway, those who are interested in all the details may read the docs,
the links will be at the end. Let me give a couple of examples. I the
example I flash a Fedora image to my USB stick, which is /dev/sdd.
1. First of all, I can flash directly from the URL, I use the Finland
mirror here:
$ bmaptool copy --nobmap
ftp://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/release...
/dev/sdd
Note, I use --nobmap because there is no bmap file. With bmap file
flashing would be _a lot_ faster.
2. Let's check how fast is dd. I downloaded and decompressed the image
prior to running the tests.
$ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd
4194304+0 records in
4194304+0 records out
2147483648 bytes (2.1 GB) copied, 799.487 s, 2.7 MB/s
13.3 minutes.
3. With bmaptool, no bmap file.
$ bmaptool copy --nobmap Fedora-x86_64-19-20130627-sda.raw /dev/sdd
bmaptool: info: no bmap given, copy entire image to '/dev/sdd'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdd'
bmaptool: info: copying time: 7m 22.2s, copying speed 4.6 MiB/sec
Faster, but not too impressive. The speed is because bmaptool configures
the block device for optimal writing of large sequential files.
4. Now I've generated the bmap file myself. Note, the real, correct bmap
file can only be generated on the Fedora build server just after the
image is created, so this example is artificial, and not correct, do not
try to use it. And it will give better numbers than the real bmap file
would give, I can explain why if needed, or see the docs. Anyway, I've
made a sparse file out of the image, then generated the bmap file for
it, and flashed it:
# Make the image to be sparse
$ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw
Fedora-x86_64-19-20130627-sda.raw.sparse
# Generate the bmap file
$ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o
Fedora-x86_64-19-20130627-sda.raw.sparse.bmap
# Now flash with the bmap file
$ sudo ./bmaptool copy Fedora-x86_64-19-20130627-sda.raw.sparse /dev/sdd
bmaptool: info: discovered bmap file
'/home/dedekind/tmp/Fedora-x86_64-19-20130627-sda.raw.sparse.bmap'
bmaptool: info: block map format version 1.3
bmaptool: info: 524288 blocks of size 4096 (2.0 GiB), mapped 176288 blocks (688.6 MiB or
33.6%)
bmaptool: info: copying image 'Fedora-x86_64-19-20130627-sda.raw.sparse' to block
device '/dev/sdd' using bmap file
'Fedora-x86_64-19-20130627-sda.raw.sparse.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdd'
bmaptool: info: copying time: 2m 36.2s, copying speed 4.4 MiB/sec
Several times faster than dd.
And yes, bmaptool can flash the .raw.xz file as well, you do not have to
decompress.
Please, fine the bmaptool documentation here:
https://source.tizen.org/documentation/articles/bmaptool
It is a bit out-of-date, but not much, will be updated soon.
The tool is generic, not tizen-specific.
Please, see the information about the project here as well:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
Note, the .xz support is only available in v2.6, which I did not release
yet, so you need to take the sources, not the packages.
The project is a python project, and it provides APIs so that other
python projects could use the functionality.
Here is an example of a bmap file can be found here:
https://source.tizen.org/documentation/articles/bmaptool
And yes, I suggest that Fedora could also generate and publish bmap
files, if this is found to be useful. In Tizen IVI people are very happy
to use bmaptool versus pain dd.
Anyway, please, check the docs, give feed-back, ask questions.
Thanks, and sorry for a messy e-mail :-)
--
Best Regards,
Artem Bityutskiy
10:48 a.m.
On Tue, Aug 13, 2013 at 04:58:16PM +0300, Artem Bityutskiy wrote:
Hi Fedora developers,
$ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd
4194304+0 records in
4194304+0 records out
2147483648 bytes (2.1 GB) copied, 799.487 s, 2.7 MB/s
1.) Seems to be a slow USB flash drive.
2.) What time did you get, if you are specify a large bs value
on the dd command. For example
dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd bs=100M
Best Regards:
Jochen Schmitt
1:31 a.m.
On Tue, 2013-08-13 at 17:48 +0200, Jochen Schmitt wrote:
On Tue, Aug 13, 2013 at 04:58:16PM +0300, Artem Bityutskiy wrote:
> Hi Fedora developers,
> $ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd
> 4194304+0 records in
> 4194304+0 records out
> 2147483648 bytes (2.1 GB) copied, 799.487 s, 2.7 MB/s
1.) Seems to be a slow USB flash drive.
Sure, typical slow, but large size and cheap USB stick most people have.
2.) What time did you get, if you are specify a large bs value
on the dd command. For example
dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd bs=100M
It is faster: 2147483648 bytes (2.1 GB) copied, 529.886 s, 4.1 MB/s
bmaptool is still a bit faster than dd, because it sets the I/O
scheduler to 'noop' and tweaks the queue size. It also does not hog the
system, and it reacts on Ctrl-C almost immediately, unlike dd. I
carefully took care of this.
But this all is not the point, these are additional goodies. The main
point of bmaptool is in having the bmap file, and _then_ you get real
speed-up, based on the sparseness of the original image.
Other things like reading from remote sites, progress indicator,
protecting your mounted disks, uncompressing on-the-fly, checking sha1
of the data ond of the bmap file itself - are goodies, although
important ones.
This is not of interest for people who flash once a month. This is of
interest for people who flash images more often than that, e.g., for
testing, or in a production line (not Fedora case, though, I guess).
But even for people who flash occasionally it would be nice to run only
one single command:
bmaptool copy URL-to-.xz-file /dev/my-block-device
which would do everything, very quickly (if the connectivity is not a
bottleneck), and reliably.
Check the docs if you are interested, we have rpm/deb packages, publish
tarballs, etc.
But the base principle is to utilize the inherent sparseness most raw
images have a lot of, record this in the bmap file before it is lost,
then publish the image in any form (compressed or not), and use the bmap
file for fetching the sparseness information and writing/copying only
the real data, and leaving out the zeroes.
And the larger is the image, and the more often you have to copy/flash
it, the more useful bmap file is.
--
Best Regards,
Artem Bityutskiy
2 a.m.
On Wed, 2013-08-14 at 09:31 +0300, Artem Bityutskiy wrote:
But this all is not the point, these are additional goodies. The
main
point of bmaptool is in having the bmap file, and _then_ you get real
speed-up, based on the sparseness of the original image.
Just as a rough demonstration, as I described in the original e-mail,
I've "reconstructed" the original image sparseness using 'cp
--sparse=always', and then created the bmap file, which at the end of
this e-mail.
You can see that only 33.6% of the image contains non-zero blocks, so
with bmap file you'd copy only 688.6MiB instead of 2.1GiB. This is where
the speed would come from. This is why I got a seemingly unrealistically
fast flashing speed in the original e-mail.
But sure, 33.6% is incorrect number, the real number will probably be
slightly higher. The explanations why can be found here:
https://source.tizen.org/documentation/reference/bmaptool/introduction
in the "Reconstructing sparse files" section.
It is technically easy to add bmap file generation to the fedora image
build tools and generate the bmap file along with the image. I could try
to do this if people are interested.
Here is the bmap file I got:
<?xml version="1.0" ?>
<!-- This file contains the block map for an image file, which is basically
a list of useful (mapped) block numbers in the image file. In other words,
it lists only those blocks which contain data (boot sector, partition
table, file-system metadata, files, directories, extents, etc). These
blocks have to be copied to the target device. The other blocks do not
contain any useful data and do not have to be copied to the target
device.
The block map an optimization which allows to copy or flash the image to
the image quicker than copying of flashing the entire image. This is
because with bmap less data is copied: <MappedBlocksCount> blocks instead
of <BlocksCount> blocks.
Besides the machine-readable data, this file contains useful commentaries
which contain human-readable information like image size, percentage of
mapped data, etc.
The 'version' attribute is the block map file format version in the
'major.minor' format. The version major number is increased whenever an
incompatible block map format change is made. The minor number changes
in case of minor backward-compatible changes. -->
<bmap version="1.3">
<!-- Image size in bytes: 2.0 GiB -->
<ImageSize> 2147483648 </ImageSize>
<!-- Size of a block in bytes -->
<BlockSize> 4096 </BlockSize>
<!-- Count of blocks in the image file -->
<BlocksCount> 524288 </BlocksCount>
<!-- Count of mapped blocks: 688.6 MiB or 33.6% -->
<MappedBlocksCount> 176288 </MappedBlocksCount>
<!-- The checksum of this bmap file. When it is calculated, the value of
the SHA1 checksum has be zero (40 ASCII "0" symbols). -->
<BmapFileSHA1> 75b48dc596a5e92d7cc4935d8fcc3a91c2e48b0f </BmapFileSHA1>
<!-- The block map which consists of elements which may either be a
range of blocks or a single block. The 'sha1' attribute (if present)
is the SHA1 checksum of this blocks range. -->
<BlockMap>
<Range sha1="ce5cc4f31d623a34c01f791e6e5c1ee65456044b"> 0-15
</Range>
<Range sha1="4ca6ee9c01354785502ba201a7c03cdd4ffbaedb"> 240-1967
</Range>
<Range sha1="6f0869f38618931042495803f5b4c02f2d7d03e9"> 8224-9583
</Range>
<Range sha1="b7650fa6a6bb703e30438ae22ca7a53c107f72ad"> 9632-11599
</Range>
<Range sha1="4e04cad0efb9194e7e00d407caaf89c9bc19042d">
11632-11775 </Range>
<Range sha1="feebd3924ce0db12f1d7186577ca54985984ab69">
33008-33023 </Range>
<Range sha1="ce74e311d9c1316f03b3b5a64225b607196419b9">
33136-33567 </Range>
<Range sha1="1e6a1f19428243b09da296e5e5494d9443b90c64">
34032-34095 </Range>
<Range sha1="9b2b5218b79d1b51e00d67814a8eea41d5cc430f">
34544-34591 </Range>
<Range sha1="f36da167d811feba274928d5c293b569b9f8c0aa">
34656-34671 </Range>
<Range sha1="2ffd3f90346ba2d0a9841865ccca0adfa19b830e">
35056-35087 </Range>
<Range sha1="48a82266b1351a543a169df69274c7f82c743988">
35568-35615 </Range>
<Range sha1="71ffc02cadefd321af333fa4b8e54ca6a2a48fbe">
35888-35903 </Range>
<Range sha1="e3b51cd7304df8df536e9ebd6d84d6d0d516d0a0">
36080-36255 </Range>
<Range sha1="6939de8eed4c6c35691f6b6ccf805e7d7a141b70">
36304-36511 </Range>
<Range sha1="9d2b2e2c0cc771f2a2684317ffdaa8557d5962e1">
36592-37071 </Range>
<Range sha1="5ccf6bca3a0140f3f549b67cf114e8847d8fd82e">
37104-55551 </Range>
<Range sha1="a3259735e513eb38f27058e6c87bd555d060615a">
62656-62687 </Range>
<Range sha1="27c3ac0865d0b12180395e2d083f7bbdc0440bb3">
62992-63023 </Range>
<Range sha1="bfe6dbe0c325a0a95a6cf6d62226287f4dd0b99d">
64144-64175 </Range>
<Range sha1="ac97574508a8bd42f39bed6ccbe0ab1e57cbfd54">
65120-98559 </Range>
<Range sha1="2351cae4967fe643d5003b6f526c811cd311df45">
98672-100975 </Range>
<Range sha1="6a47e91dc5be78d69761ab762fc2e3ae1ea9bc8e">
101040-101615 </Range>
<Range sha1="4e9bbbb0f3b487efd37cc54808a371d27eac4c78">
101808-123759 </Range>
<Range sha1="05a6b82bcd9672a04252bc090b368e6deb3084b1">
123952-124015 </Range>
<Range sha1="531f3dcb1cefdc515233da007069761849058b13">
124064-141151 </Range>
<Range sha1="626d4a7ed0e01815230b1fc95e22e7095b3c2c5a">
141392-143327 </Range>
<Range sha1="04c93f6e63aec7320088037fab3b5d0edb994d23">
143360-146319 </Range>
<Range sha1="949d1861d7dde9c2ad916a0807ac1b2db6ee2002">
146624-155391 </Range>
<Range sha1="4e5395a152a382e5e487e24af9d2ed8930eb4531">
155888-164095 </Range>
<Range sha1="484548b0536d1276575f620139ff3596a455b017">
164208-165983 </Range>
<Range sha1="e44b975f3d627e4f8df3442d27ca0aeaa7329af5">
166016-166047 </Range>
<Range sha1="ce95c6eb68528b4d310d0dc087ba595b43473fef">
166080-167103 </Range>
<Range sha1="d42280d545afdc3c5fb6ae5a0e6e7e3430cb4d23">
167568-196143 </Range>
<Range sha1="2856f3fac20726e5c647d32fe4757f1b091309b0">
196848-213247 </Range>
<Range sha1="10614aa162d95797e64dff049707ae788a19d40f">
221424-223311 </Range>
<Range sha1="a4c0fef5c6c9a013fc641016face92e92329864b">
223344-223391 </Range>
<Range sha1="b7375c8568a14fb1217a1e31f52cfdffbdd9ce30">
223472-227375 </Range>
<Range sha1="855e049fb6ace103d4569465b0b06df47628b76f">
229616-229631 </Range>
<Range sha1="0c900792e5cc9613496ab53d1f30bb94b1c9ad9d">
295152-295167 </Range>
<Range sha1="2efc62831e5d49db8550c7ef401bc1c1c2a27a78">
295280-295295 </Range>
</BlockMap>
</bmap>
--
Best Regards,
Artem Bityutskiy
3:37 a.m.
On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
Other things like reading from remote sites, progress indicator,
protecting your mounted disks, uncompressing on-the-fly, checking sha1
of the data ond of the bmap file itself - are goodies, although
important ones.
Why sha1? If the check is there for security reasons, please use at
least sha256. You can also encode the checksum as base64, base85 or
base91 to reduce the size of the bmap file.
But the base principle is to utilize the inherent sparseness most
raw
images have a lot of, record this in the bmap file before it is lost,
then publish the image in any form (compressed or not), and use the bmap
file for fetching the sparseness information and writing/copying only
the real data, and leaving out the zeroes.
This does not sound safe, because it does not ensure that all data that
should be zero actually is a zero. It works well for unassigned file
systems blocks, but if there is a file containing zeroes in the file
system (that is not a sparse file) it might not contains zeroes
afterwards as far as I understand bmap. This does not sound like
something that is safe to do.
Regards
Till
4:21 a.m.
On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
> Other things like reading from remote sites, progress indicator,
> protecting your mounted disks, uncompressing on-the-fly, checking sha1
> of the data ond of the bmap file itself - are goodies, although
> important ones.
Why sha1? If the check is there for security reasons, please use at
least sha256.
Should not be difficult to implement if there is demand.
You can also encode the checksum as base64, base85 or
base91 to reduce the size of the bmap file.
May be, this was not a problem for sha1, but for sha256 it would
probably make sense, thanks for suggestions.
> But the base principle is to utilize the inherent sparseness most raw
> images have a lot of, record this in the bmap file before it is lost,
> then publish the image in any form (compressed or not), and use the bmap
> file for fetching the sparseness information and writing/copying only
> the real data, and leaving out the zeroes.
This does not sound safe, because it does not ensure that all data that
should be zero actually is a zero.
I think I covered this part in the documentation. But here is a short
description.
1. The bmap file should be created just after the image is generated.
2. The blocks where zeroes were explicitly written will be mapped to
real sectors which will contain zeroes.
3. The blocks which were not explicitely written to, will be unmapped.
4. Creation of the bmap file is done using the FIEMAP ioctl
5. Only unmapped blocks will be omited in the bmap files.
While on this, I should note that this works best on ext4 file-system. I
did not test ext2/3, but they should work as well as ext4. Btrfs was
also tested, but it is a little bit worse than ext4, I can explain why
if someone is interested.
It works well for unassigned file
systems blocks, but if there is a file containing zeroes in the file
system (that is not a sparse file) it might not contains zeroes
afterwards as far as I understand bmap.
It will, those blocks will be explicitly specified in the bmap file. And
the zeroes will be copied.
And this is exactly why I said that 'cp --sparse=always' does not
generate the correct bmap file, I used it only for demosntration
purposes.
In bmap-tools project we have the 'BmapCreate' python module, which
implements bmap file creation, you can use it in the Fedora image build
tool for generating the correct bmap file just after the raw image is
created.
This does not sound like
something that is safe to do.
It is safe, we are using it for a year already in Tizen IVI, so it is
somewhat tested as well. Of course, Tizen IVI community is a lot
smaller, but still, this is some indication of safeness already.
--
Best Regards,
Artem Bityutskiy
4:44 a.m.
On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote:
On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
> On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
>
> > Other things like reading from remote sites, progress indicator,
> > protecting your mounted disks, uncompressing on-the-fly, checking sha1
> > of the data ond of the bmap file itself - are goodies, although
> > important ones.
>
> Why sha1? If the check is there for security reasons, please use at
> least sha256.
Should not be difficult to implement if there is demand.
SHA-256 is used to create the signatures of other distributed files:
https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
Therefore if bmap is used it should also use at least SHA 256. It is
recommended against using SHA-1 for more than 7 years now:
http://csrc.nist.gov/groups/ST/hash/policy_2006.html
> It works well for unassigned file
> systems blocks, but if there is a file containing zeroes in the file
> system (that is not a sparse file) it might not contains zeroes
> afterwards as far as I understand bmap.
It will, those blocks will be explicitly specified in the bmap file. And
the zeroes will be copied.
And this is exactly why I said that 'cp --sparse=always' does not
generate the correct bmap file, I used it only for demosntration
purposes.
I see.
Regards
Till
4:52 a.m.
On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote:
On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote:
> On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
> > On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
> >
> > > Other things like reading from remote sites, progress indicator,
> > > protecting your mounted disks, uncompressing on-the-fly, checking sha1
> > > of the data ond of the bmap file itself - are goodies, although
> > > important ones.
> >
> > Why sha1? If the check is there for security reasons, please use at
> > least sha256.
>
> Should not be difficult to implement if there is demand.
SHA-256 is used to create the signatures of other distributed files:
https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
Therefore if bmap is used it should also use at least SHA 256. It is
recommended against using SHA-1 for more than 7 years now:
http://csrc.nist.gov/groups/ST/hash/policy_2006.html
Sure, good point, thank you, I'll implement sha-256 support.
--
Best Regards,
Artem Bityutskiy
5:24 a.m.
Artem Bityutskiy wrote:
On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote:
> On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote:
> > On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
> > > On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
> > >
> > > > Other things like reading from remote sites, progress
> > > > indicator, protecting your mounted disks, uncompressing
> > > > on-the-fly, checking sha1 of the data ond of the bmap file
> > > > itself - are goodies, although important ones.
> > >
> > > Why sha1? If the check is there for security reasons, please use
> > > at least sha256.
> >
> > Should not be difficult to implement if there is demand.
>
> SHA-256 is used to create the signatures of other distributed files:
> https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
>
> Therefore if bmap is used it should also use at least SHA 256. It is
> recommended against using SHA-1 for more than 7 years now:
> http://csrc.nist.gov/groups/ST/hash/policy_2006.html
Sure, good point, thank you, I'll implement sha-256 support.
Speaking of security, how is the integrity of the bmap file itself
verified? A checksum is of no use if you don't know who generated the
checksum. Fedora's checksum files are OpenPGP signed, as you can see in
the one that Till linked to. I don't see a cryptographic signature in
your example file. Are there detached signatures for the bmap files?
And does Bmaptool verify the signatures?
--
Björn Persson
Sent from my computer.
5:39 a.m.
On Wed, 2013-08-14 at 12:24 +0200, Björn Persson wrote:
Speaking of security, how is the integrity of the bmap file itself
verified?
This is not implemented, unfortunately. This is another thing which I
probably would need to do, and this is a very good point.
I will look at this, after I do the SHA256 thing.
A checksum is of no use if you don't know who generated the
checksum. Fedora's checksum files are OpenPGP signed, as you can see
in
the one that Till linked to.
Right, bmap file could also contain such a signature.
I don't see a cryptographic signature in
your example file. Are there detached signatures for the bmap files?
Well, of course detached signatures can be generated.
And does Bmaptool verify the signatures?
But no, bmaptool does not verify them. And again, if there is real
interest from Fedora community, I will try to implement this faster (or
accept someone's contribution :-))
Thanks for the feed-back!
--
Best Regards,
Artem Bityutskiy
8:38 a.m.
On Wed, 2013-08-14 at 12:24 +0200, Björn Persson wrote:
Artem Bityutskiy wrote:
>On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote:
>> On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote:
>> > On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
>> > > On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
>> > >
>> > > > Other things like reading from remote sites, progress
>> > > > indicator, protecting your mounted disks, uncompressing
>> > > > on-the-fly, checking sha1 of the data ond of the bmap file
>> > > > itself - are goodies, although important ones.
>> > >
>> > > Why sha1? If the check is there for security reasons, please use
>> > > at least sha256.
>> >
>> > Should not be difficult to implement if there is demand.
>>
>> SHA-256 is used to create the signatures of other distributed files:
>> https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
>>
>> Therefore if bmap is used it should also use at least SHA 256. It is
>> recommended against using SHA-1 for more than 7 years now:
>> http://csrc.nist.gov/groups/ST/hash/policy_2006.html
>
>Sure, good point, thank you, I'll implement sha-256 support.
Speaking of security, how is the integrity of the bmap file itself
verified? A checksum is of no use if you don't know who generated the
checksum. Fedora's checksum files are OpenPGP signed, as you can see in
the one that Till linked to. I don't see a cryptographic signature in
your example file. Are there detached signatures for the bmap files?
And does Bmaptool verify the signatures?
I've implemented gpg signature verification.
Now the bmap file can be gpg-signed and in this case bmaptool will
verify the signature. Both Fedora-like "clearsign" gpg signatures and
detached signatures are supported.
--
Best Regards,
Artem Bityutskiy
8:36 a.m.
On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote:
On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote:
> On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
> > On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
> >
> > > Other things like reading from remote sites, progress indicator,
> > > protecting your mounted disks, uncompressing on-the-fly, checking sha1
> > > of the data ond of the bmap file itself - are goodies, although
> > > important ones.
> >
> > Why sha1? If the check is there for security reasons, please use at
> > least sha256.
>
> Should not be difficult to implement if there is demand.
SHA-256 is used to create the signatures of other distributed files:
https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
FYI, I've implemented SHA-256 support and it will be the default for
bmaptool soon. I also made sure other hash functions can be used as well
(e.g., SHA-512).
--
Best Regards,
Artem Bityutskiy
11:16 p.m.
On 08/14/2013 02:21 AM, Artem Bityutskiy wrote:
I think I covered this part in the documentation. But here is a
short
description.
1. The bmap file should be created just after the image is generated.
2. The blocks where zeroes were explicitly written will be mapped to
real sectors which will contain zeroes.
3. The blocks which were not explicitely written to, will be unmapped.
4. Creation of the bmap file is done using the FIEMAP ioctl
5. Only unmapped blocks will be omited in the bmap files.
While on this, I should note that this works best on ext4 file-system. I
did not test ext2/3, but they should work as well as ext4. Btrfs was
also tested, but it is a little bit worse than ext4, I can explain why
if someone is interested.
Have you looked at partimage? It sounds like this except that it works
on many different filesystems and doesn't need the blocks to be unmapped
to compress it (i.e. it works on normal partitions as well as images).
12:13 a.m.
On Wed, 2013-08-14 at 21:16 -0700, Samuel Sieb wrote:
On 08/14/2013 02:21 AM, Artem Bityutskiy wrote:
> I think I covered this part in the documentation. But here is a short
> description.
>
> 1. The bmap file should be created just after the image is generated.
> 2. The blocks where zeroes were explicitly written will be mapped to
> real sectors which will contain zeroes.
> 3. The blocks which were not explicitely written to, will be unmapped.
> 4. Creation of the bmap file is done using the FIEMAP ioctl
> 5. Only unmapped blocks will be omited in the bmap files.
>
> While on this, I should note that this works best on ext4 file-system. I
> did not test ext2/3, but they should work as well as ext4. Btrfs was
> also tested, but it is a little bit worse than ext4, I can explain why
> if someone is interested.
Have you looked at partimage? It sounds like this except that it works
on many different filesystems and doesn't need the blocks to be unmapped
to compress it (i.e. it works on normal partitions as well as images).
No, never saw this project before. Yeah, it sounds like it uses similar
ideas to speed-up, but has different purposes and tries to know the
file-system internal format, and hence, does not support ext4/btrfs
simply because, as I guess, they are too complex and are developed too
quickly. It is just too difficult to maintain a parallel implementation
in user-space.
Bmaptool does not know anything about the internals of the file-system.
It does not care what is the FS underneath. bmaptool simply use the
FIEMAP ioctl and ask the FS about which blocks are mapped (used).
This would not work for partimage since it needs to know about all the
blocks (superblock, all the other meta-data blocks), not just blocks
belonging to a single file.
Now, why I said that ext4 is the best one to use on the server (most
probably ext2/3 are as good, but I did not verify). This is because ext4
is "perfect" in leaving the gaps. Even if you have one block gap, it
still will account it as unmapped. I have a test where I create random
mapped areas, and ext4 keeps all the gaps. But BTRFS sometimes maps
small 1-block gaps. This is related to its internal structure. So with
btrfs the bmap file becomes less ideal.
Anyway, thanks for letting me know about partimage.
--
Best Regards,
Artem Bityutskiy
3:35 a.m.
New subject: Suggestion: bmap files and bmaptool
在 2013-8-13 PM9:52，"Artem Bityutskiy" <dedekind1(a)gmail.xn--com>:-0s6m5392c
Hi Fedora developers,
I would like suggest you to take a look at bmaptool, which you can use
for flashing Fedora ISO images to USB sticks (or other block devices).
I've read an article about this tool in Chinese months ago.
I can help submit a package review tomorrow if you want, then let us test
it.
Cheers.
4:22 a.m.
New subject: Suggestion: bmap files and bmaptool
On Wed, 2013-08-14 at 16:35 +0800, Christopher Meng wrote:
在 2013-8-13 PM9:52，"Artem Bityutskiy"
<dedekind1(a)gmail.xn--com>:-0s6m5392c
>
> Hi Fedora developers,
>
> I would like suggest you to take a look at bmaptool, which you can
use
> for flashing Fedora ISO images to USB sticks (or other block
devices).
I've read an article about this tool in Chinese months ago.
I can help submit a package review tomorrow if you want, then let us
test it.
Sure, please. The packaging may not be entirely correct, I am open to
fix it, of course. Also, please, use the tip of the 'devel' branch in
your experiments so far.
--
Best Regards,
Artem Bityutskiy
12:34 p.m.
On 8/13/13 8:58 AM, Artem Bityutskiy wrote:
# Make the image to be sparse
$ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw
Fedora-x86_64-19-20130627-sda.raw.sparse
# Generate the bmap file
$ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o
Fedora-x86_64-19-20130627-sda.raw.sparse.bmap
So this is the part that interests me . . .
There seem to be two issues here; how do we efficiently (compress and) transport sparse
files while retaining sparseness, and how do we efficiently operate on files which are
already sparse.
For the latter, you're using your bmap tool to map what is hopefully a static file
(via fibmap or fiemap, I guess?).
I haven't looked at how you've done it, but you do need to be very careful that
the file is stable & quiesced on disk. Mapping it this way can be fraught with errors
if the file is changing, or has delalloc blocks, etc. And of course getting the mapping
wrong means data corruption. If the file is known to be sparse, then going forward, using
SEEK_HOLE / SEEK_DATA is probably the best approach.
But then there's the issue of transporting these sparse files around. We have had the
same problem in the past with large e2image metadata image files, which may be terabytes
in length, with only gigabytes or megabytes of real data. e2image _itself_ creates a
sparse file, but bzipping it or rsyncing it still processes terabytes of zeros, and loses
all notion of sparseness.
xfs_metadump worked around this by creating its own compact format describing a sparse
file's data & sparseness, which is "unpacked" into a normal sparse file
by xfs_mdrestore.
More recently e2image gained something slightly similar, but used the existing qcow format
to encode the sparseness. qemu-image convert to "raw" type turns it back into a
"normal" sparse file readable by e2fsprogs tools.
So I guess your solution requires 2 pieces of information; the existing file, and the
mapping file. Are there mechanisms to ensure that they are in sync?
Another approach which might (?) be more robust, is to somehow encode that sparseness in a
single file format that can be transported/compressed/copied w/o losing the sparseness
information, and another tool to operate efficiently on that format at the destination,
either by unpacking it to a normal sparse file or piping it to some other process.
Just some thoughts...
-Eric
3:46 a.m.
Hi Eric,
thanks for the question. Sorry my answers contain extra information, but
I assume that other people read this, and may benefit from the info.
After all, I am trying to get people interested, this is a good tool
IMO, and I would like to get more users and hopefully contributors. :-)
On Thu, 2013-08-15 at 12:34 -0500, Eric Sandeen wrote:
On 8/13/13 8:58 AM, Artem Bityutskiy wrote:
> # Make the image to be sparse
> $ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw
Fedora-x86_64-19-20130627-sda.raw.sparse
>
> # Generate the bmap file
> $ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o
Fedora-x86_64-19-20130627-sda.raw.sparse.bmap
So this is the part that interests me . . .
Before going further, I want to quckly note that the Tizen image
generation software uses the BmapCreate library API directly, instead of
running the bmaptool command-line utility. The library comes with the
bmap-tools project.
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
There seem to be two issues here; how do we efficiently (compress
and)
transport sparse files while retaining sparseness, and how do we
efficiently operate on files which are already sparse.
Yes, it is our assumption is that sparseness gets lost as soon as the
image is compressed or copied to the download server, or anywhere else.
The idea is that as soon as you generate the raw image on the build
server, you generate the bmap file right away, _before_ the sparseness
gets lost.
The sparseness information is then saved in the bmap file. Then you can
compress the image, copy it around, and lose the sparseness. The bmap
file preserves it.
And of course this means that you should not modify the image later on,
otherwise the bmap file becomes incorrect, and the checksums, which are
inside the bmap file, will probably mismatch.
For the latter, you're using your bmap tool to map what is
hopefully a
static file (via fibmap or fiemap, I guess?).
Yes, we use FIEMAP. Here is the python module which does the job:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
I haven't looked at how you've done it, but you do need to be
very
careful that the file is stable & quiesced on disk.
Right. We generate the bmap file on the _build server_, inside the tool
which generates the raw image. At this point we do know we fully control
the image, and no one touches it while we generate the bmap.
But yes, this is a good point, may be I need to put it to the man page.
Which, by the way, as I figure out now, needs to be somewhat updated:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
Mapping it this way can be fraught with errors if the file is
changing, or has delalloc blocks, etc.
Good point. Tizen image generator fsync()'s the file before creating
bmap, but I guess I have to do this in the BmapCreate library too, to be
safe.
Thanks!
And of course getting the mapping wrong means data corruption.
Right. But as I said, we are using bmaptool for a year now, and nothing
which looks like a corruption was reported so far.
But the importance of fsync() is a very good point, I'll improve the
library and make it explicitely fsync, and probably ignore the EROFS
error, in case the file is R/O.
If the file is known to be sparse, then going forward, using
SEEK_HOLE / SEEK_DATA is probably the best approach.
Why are they better than FIEMAP?
I did consider them, actually, but they are very new, and build servers
tend to use older kernels, so I chose FIEMAP. I actually first used
FIBMAP, but it is too slow, so I switched to FIEMAP.
But then there's the issue of transporting these sparse files around.
We have had the same problem in the past with large e2image metadata
image files, which may be terabytes in length, with only gigabytes or
megabytes of real data. e2image _itself_ creates a sparse file, but
bzipping it or rsyncing it still processes terabytes of zeros, and
loses all notion of sparseness.
Right, but the scenario I keep in mind is that the bmap file is created
at the _very_ beginning, and carried/published together with the image,
as a stand-alone file with the same basename and ".bmap" extension.
The zeroes in the image can be very well compressed with xz, so people
download/copy a lot less than Terabytes. And then people just run this
command to re-create the original sparse file:
$ bmaptool copy --bmap huge.img.bmap huge.img.xz a_sparse_copy.img
This will decompress huge.img.xz on-the-fly and put it to
a_sparse_copy.img. The a_sparse_copy.img file will be sparse.
Note, it bmaptool auto-discovers the bmap file if it has a common
basename with the image, and if it sits in the same directory, so this
command can instead be:
bmaptool copy huge.img.xz a_sparse_copy
(analogy to "cp from to").
And of course, "huge.img.xz" can be, say:
bmaptool copy http://my.server/x/y/huge.img.xz a_sparse_copy
When the target is a block device, bmaptool has some optimizations to
copy faster (e.g., switching to noop I/O scheduler), trash RAM less,
react on Ctrl-C, and some more.
xfs_metadump worked around this by creating its own compact format
describing a sparse file's data & sparseness, which is "unpacked" into
a normal sparse file by xfs_mdrestore.
Frankly, I know little about XFS so I do not really understand the
above.
More recently e2image gained something slightly similar, but used the
existing qcow format to encode the sparseness. qemu-image convert to
"raw" type turns it back into a "normal" sparse file readable by
e2fsprogs tools.
OK. So it could in theory use/generate a stand-alone bmapfile too.
So I guess your solution requires 2 pieces of information; the
existing file, and the mapping file.
Right.
Are there mechanisms to ensure that they are in sync?
There is built-in SHA1 for all mapped areas, you can look for example
here:
http://download.tizen.org/releases/milestone/tizen/ivi/latest/images/ivi-...
For unmapped areas we could check that they are all-zeros, but bmaptool
does not currently do this.
And this is another good idea, thanks!
So if sha1's for mapped areas match, and unmapped areas are all-zeros,
the integrity is fine.
Bmap file also carries sha1 of itself, which bmaptool verifies.
And yes, I already agreed that I should change sha1 with sha256.
And yes, I agreed that bmaptool should allow for a GPG signature in the
bmap file.
Another approach which might (?) be more robust, is to somehow encode
that sparseness in a single file format that can be
transported/compressed/copied w/o losing the sparseness information,
and another tool to operate efficiently on that format at the
destination, either by unpacking it to a normal sparse file or piping
it to some other process.
Err, not sure I fully understand, but it sounds like what bmap-tools
project actually does.
Piping is not implemented, because sparseness cannot be easily passed
though a pipe.
Just some thoughts...
Thanks a lot for the feed-back!
--
Best Regards,
Artem Bityutskiy
9:51 a.m.
On 8/16/13 3:46 AM, Artem Bityutskiy wrote:
> Another approach which might (?) be more robust, is to somehow
encode
> > that sparseness in a single file format that can be
> > transported/compressed/copied w/o losing the sparseness information,
> > and another tool to operate efficiently on that format at the
> > destination, either by unpacking it to a normal sparse file or piping
> > it to some other process.
Err, not sure I fully understand, but it sounds like what bmap-tools
project actually does.
by single file I meant a _single_ file, not the original file and a mapping
file. :)
I realize that you have a fully-fledged set of tools, and you're not looking
for new directions, but I was thinking about encoding mapping info & file data
into a single file, with a tool to extract it again. That way there's nothing
to get out of sync.
Actually, now that I think about it qemu-img can do that already:
(sorry, I'm getting a little off topic here, bear with me)
# truncate --size=1g fsfile
# mkfs.ext4 fsfile
# cp fsfile --sparse=never fsfile.copy
// fsfile is sparse; the copy is not.
# du -hc fsfile*
49M fsfile
1.0G fsfile.copy
# qemu-img convert -f raw -O qcow fsfile.copy fsfile.qcow
// the qcow image now contains only data+mapping info,
// no zero ranges:
# du -h fsfile.qcow
832K fsfile.qcow
// and can be re-extracted into a sparse file
# qemu-img convert -O raw fsfile.qcow fsfile.copy2
# du -hc fsfile.copy2
352K fsfile.copy2
Ok, sorry for that diversion, but that's cool - the tool I want
already exists, and I hadn't realized it. :)
So that's a decent option for encoding sparse files for efficient
transfer, too.
Piping is not implemented, because sparseness cannot be easily
passed
though a pipe.
Err, right ;)
-Eric
> > Just some thoughts...
> >
Thanks a lot for the feed-back!
6:29 a.m.
On Fri, 2013-08-16 at 09:51 -0500, Eric Sandeen wrote:
by single file I meant a _single_ file, not the original file and a
mapping
file. :)
Oh, sorry, OK.
I realize that you have a fully-fledged set of tools, and you're
not looking
for new directions, but I was thinking about encoding mapping info & file data
into a single file, with a tool to extract it again. That way there's nothing
to get out of sync.
Well, I do wish to know about alternatives, or the better ways to do it,
of course.
Actually, now that I think about it qemu-img can do that already:
(sorry, I'm getting a little off topic here, bear with me)
# truncate --size=1g fsfile
# mkfs.ext4 fsfile
# cp fsfile --sparse=never fsfile.copy
// fsfile is sparse; the copy is not.
# du -hc fsfile*
49M fsfile
1.0G fsfile.copy
# qemu-img convert -f raw -O qcow fsfile.copy fsfile.qcow
// the qcow image now contains only data+mapping info,
// no zero ranges:
# du -h fsfile.qcow
832K fsfile.qcow
// and can be re-extracted into a sparse file
# qemu-img convert -O raw fsfile.qcow fsfile.copy2
# du -hc fsfile.copy2
352K fsfile.copy2
So from 49M down to 352K, sparseness increased by 48M? Where it came
from? Must be that this command turned zero blocks into gaps. Like the
zeroed out inode tables, etc.
Ok, sorry for that diversion, but that's cool - the tool I want
already exists, and I hadn't realized it. :)
Exactly this usage is not good enough for flashing purposes, because
when flashing to a block device you have to flash the zeroes, you cannot
skip them as you do for holes.
But I am sure the qcow tools can save sparseness without turning zeroes
into gaps, or at least this should not be too difficult to implement
this.
But in case of bmaptool I chose to keep the sparseness information in a
separate file because I wanted to make sure the bmap is optional. Those
who use Windows/Mac or do not want to install any additional tools could
still use the old method of throwing the entire image to the target
block device.
Another goal I had is to make the additional bmap file to be a
human-readable text file.
But yes, having all in one file does have its own advantages.
--
Best Regards,
Artem Bityutskiy
10:43 a.m.
On 08/16/2013 09:46 AM, Artem Bityutskiy wrote:
Hi Eric,
thanks for the question. Sorry my answers contain extra information, but
I assume that other people read this, and may benefit from the info.
After all, I am trying to get people interested, this is a good tool
IMO, and I would like to get more users and hopefully contributors. :-)
On Thu, 2013-08-15 at 12:34 -0500, Eric Sandeen wrote:
> On 8/13/13 8:58 AM, Artem Bityutskiy wrote:
>> # Make the image to be sparse
>> $ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw
> Fedora-x86_64-19-20130627-sda.raw.sparse
>>
>> # Generate the bmap file
>> $ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o
> Fedora-x86_64-19-20130627-sda.raw.sparse.bmap
>
> So this is the part that interests me . . .
Before going further, I want to quckly note that the Tizen image
generation software uses the BmapCreate library API directly, instead of
running the bmaptool command-line utility. The library comes with the
bmap-tools project.
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
> There seem to be two issues here; how do we efficiently (compress and)
> transport sparse files while retaining sparseness, and how do we
> efficiently operate on files which are already sparse.
Yes, it is our assumption is that sparseness gets lost as soon as the
image is compressed or copied to the download server, or anywhere else.
The idea is that as soon as you generate the raw image on the build
server, you generate the bmap file right away, _before_ the sparseness
gets lost.
The sparseness information is then saved in the bmap file. Then you can
compress the image, copy it around, and lose the sparseness. The bmap
file preserves it.
And of course this means that you should not modify the image later on,
otherwise the bmap file becomes incorrect, and the checksums, which are
inside the bmap file, will probably mismatch.
> For the latter, you're using your bmap tool to map what is hopefully a
> static file (via fibmap or fiemap, I guess?).
Yes, we use FIEMAP. Here is the python module which does the job:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
> I haven't looked at how you've done it, but you do need to be very
> careful that the file is stable & quiesced on disk.
Right. We generate the bmap file on the _build server_, inside the tool
which generates the raw image. At this point we do know we fully control
the image, and no one touches it while we generate the bmap.
But yes, this is a good point, may be I need to put it to the man page.
Which, by the way, as I figure out now, needs to be somewhat updated:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/de...
> Mapping it this way can be fraught with errors if the file is
> changing, or has delalloc blocks, etc.
Good point. Tizen image generator fsync()'s the file before creating
bmap, but I guess I have to do this in the BmapCreate library too, to be
safe.
Thanks!
> And of course getting the mapping wrong means data corruption.
Right. But as I said, we are using bmaptool for a year now, and nothing
which looks like a corruption was reported so far.
But the importance of fsync() is a very good point, I'll improve the
library and make it explicitely fsync, and probably ignore the EROFS
error, in case the file is R/O.
> If the file is known to be sparse, then going forward, using
> SEEK_HOLE / SEEK_DATA is probably the best approach.
Why are they better than FIEMAP?
I did consider them, actually, but they are very new, and build servers
tend to use older kernels, so I chose FIEMAP. I actually first used
FIBMAP, but it is too slow, so I switched to FIEMAP.
>
> But then there's the issue of transporting these sparse files around.
> We have had the same problem in the past with large e2image metadata
> image files, which may be terabytes in length, with only gigabytes or
> megabytes of real data. e2image _itself_ creates a sparse file, but
> bzipping it or rsyncing it still processes terabytes of zeros, and
> loses all notion of sparseness.
Right, but the scenario I keep in mind is that the bmap file is created
at the _very_ beginning, and carried/published together with the image,
as a stand-alone file with the same basename and ".bmap" extension.
The zeroes in the image can be very well compressed with xz, so people
download/copy a lot less than Terabytes. And then people just run this
command to re-create the original sparse file:
$ bmaptool copy --bmap huge.img.bmap huge.img.xz a_sparse_copy.img
This will decompress huge.img.xz on-the-fly and put it to
a_sparse_copy.img. The a_sparse_copy.img file will be sparse.
Note, it bmaptool auto-discovers the bmap file if it has a common
basename with the image, and if it sits in the same directory, so this
command can instead be:
bmaptool copy huge.img.xz a_sparse_copy
(analogy to "cp from to").
And of course, "huge.img.xz" can be, say:
bmaptool copy http://my.server/x/y/huge.img.xz a_sparse_copy
When the target is a block device, bmaptool has some optimizations to
copy faster (e.g., switching to noop I/O scheduler), trash RAM less,
react on Ctrl-C, and some more.
> xfs_metadump worked around this by creating its own compact format
> describing a sparse file's data & sparseness, which is "unpacked"
into
> a normal sparse file by xfs_mdrestore.
Frankly, I know little about XFS so I do not really understand the
above.
>
> More recently e2image gained something slightly similar, but used the
> existing qcow format to encode the sparseness. qemu-image convert to
> "raw" type turns it back into a "normal" sparse file readable by
> e2fsprogs tools.
OK. So it could in theory use/generate a stand-alone bmapfile too.
> So I guess your solution requires 2 pieces of information; the
> existing file, and the mapping file.
Right.
> Are there mechanisms to ensure that they are in sync?
There is built-in SHA1 for all mapped areas, you can look for example
here:
http://download.tizen.org/releases/milestone/tizen/ivi/latest/images/ivi-...
For unmapped areas we could check that they are all-zeros, but bmaptool
does not currently do this.
And this is another good idea, thanks!
So if sha1's for mapped areas match, and unmapped areas are all-zeros,
the integrity is fine.
You definitely need the fsync before doing the fiemap.
We saw this on certain file systems including ext4 when adding
fiemap support (efficient reading of holes) to cp.
This is a bug in the fiemap interface IMHO in that it returns
fairly useless data unless FIEMAP_FLAG_SYNC is specified.
For a general utility like cp, we couldn't sync each file before copying
(even only large files), so we restrict fiemap usage to files that
have a different disk usage than apparent size and so probably contain holes.
cheers,
Pádraig.
6:46 a.m.
On Sun, 2013-08-18 at 16:43 +0100, Pádraig Brady wrote:
You definitely need the fsync before doing the fiemap.
We saw this on certain file systems including ext4 when adding
fiemap support (efficient reading of holes) to cp.
This is a bug in the fiemap interface IMHO in that it returns
fairly useless data unless FIEMAP_FLAG_SYNC is specified.
For a general utility like cp, we couldn't sync each file before copying
(even only large files), so we restrict fiemap usage to files that
have a different disk usage than apparent size and so probably contain holes.
I see, thanks a lot for suggestion.
Just to make it clear, this is more like working around file-system
bugs, which, as I guess, were present in early days of FIEMAP.
I'll look into adding FIEMAP_FLAG_SYNC to Fiemap.py in bmap-tools,
thanks!
--
Best Regards,
Artem Bityutskiy
8:39 a.m.
On Sun, 2013-08-18 at 16:43 +0100, Pádraig Brady wrote:
You definitely need the fsync before doing the fiemap.
We saw this on certain file systems including ext4 when adding
fiemap support (efficient reading of holes) to cp.
This is a bug in the fiemap interface IMHO in that it returns
fairly useless data unless FIEMAP_FLAG_SYNC is specified.
For a general utility like cp, we couldn't sync each file before copying
(even only large files), so we restrict fiemap usage to files that
have a different disk usage than apparent size and so probably contain holes.
FYI, I've just made sure I use the FIEMAP_FLAG_SYNC is used in the
project.
--
Best Regards,
Artem Bityutskiy
11:09 a.m.
On Thu, Aug 15, 2013 at 12:34:26PM -0500, Eric Sandeen wrote:
But then there's the issue of transporting these sparse files
around. We have had the same problem in the past with large e2image
metadata image files, which may be terabytes in length, with only
gigabytes or megabytes of real data. e2image _itself_ creates a
sparse file, but bzipping it or rsyncing it still processes
terabytes of zeros, and loses all notion of sparseness.
xz preserves sparseness. We use it for preserving and compressing
virt-sparsify'd images.
Another approach which might (?) be more robust, is to somehow
encode that sparseness in a single file format that can be
transported/compressed/copied w/o losing the sparseness information,
and another tool to operate efficiently on that format at the
destination, either by unpacking it to a normal sparse file or
piping it to some other process.
qcow2 :-)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
6:40 a.m.
On Sat, 2013-08-17 at 17:09 +0100, Richard W.M. Jones wrote:
On Thu, Aug 15, 2013 at 12:34:26PM -0500, Eric Sandeen wrote:
> But then there's the issue of transporting these sparse files
> around. We have had the same problem in the past with large e2image
> metadata image files, which may be terabytes in length, with only
> gigabytes or megabytes of real data. e2image _itself_ creates a
> sparse file, but bzipping it or rsyncing it still processes
> terabytes of zeros, and loses all notion of sparseness.
xz preserves sparseness. We use it for preserving and compressing
virt-sparsify'd images.
Right, this is a good solution for the problem area of just saving the
sparseness and later restoring it.
The problem area for bmaptool is a bit wider.
1. There are large images which are inherently sparse
2. They are distributed via ftp/http/etc servers
How do we enable the users flashing these images
a) very quickly
b) easily
c) without breaking the old way of flashing (dd)
For the first part, we exploit the sparseness information, which is
saved in the bmap file.
For the second part, we implement stream-reading directly from the
remote service, stream-decompressing on-the fly, and flashing in
parallel. This rules out the "xz saves sparseness" design.
For the third part, we keep the sparseness information in a separate
file which makes it entirely optional.
--
Best Regards,
Artem Bityutskiy
3873
days inactive
3908
days old
24 comments
9 participants
participants (9)
-
Artem Bityutskiy -
Björn Persson -
Christopher Meng -
Eric Sandeen -
Jochen Schmitt -
Pádraig Brady -
Richard W.M. Jones -
Samuel Sieb -
Till Maas