4:01 p.m.
Hello everyone,
any comments, suggestions are welcome..
Handling EOBs (end-of-blocks) for transparent
encryption, checking integrity and data authentication
DRAFT
This was designed for CloudFS, which uses 2-level protocol (high and
low) supported by xlators which reside on server and client sides
respectively.
Definition of EOB. Storage class
If file size isn't a multiple of cblock (cipher block) size, then we
also need to store special padding needed to decrypt its last block
with some cipher modes like CBC. This padding contains a part of
ciphertext and must be considered as a part of this file. We'll call
this padding end-of-file (EOF). If plain text has size a multiple of
cblock size, then encrypted file won't have (or will have empty) EOF.
Signatures (HMACs, etc) for checking integrity, data authentication,
etc. have the same nature as EOF. Every such signature is created
for some logical block in a file. This is not a padding though, as
in the case of EOF, but anyway such signatures are associated with
file's data, and we'll consider a class of object, which includes
EOFs, HMACs, etc, and call them EOBs (end-of-block).
We define storage class of EOBs as "data", i.e. this can be considered
as part of file's data: we can not read/write data block without
reading/writing its EOB.
Storing EOBs. Approaches and Issues
Approach 1: Storing EOBs as xattr values.
In this case we store a file in parts which are not adjacent
from the standpoint of Cloudfs. That said we need to split
read, and this makes this operation inatomic. This means
that read(2) will return data compound of parts of different
"versions".
Example:
Suppose we have a file F stored in 2 different parts F1 and F2.
Process A writes a file F (to be of version 1);
Process B reads a file F (part F1);
Process C writes a file F (to be of version 2);
Process B reads a file F (part F2);
As the result process B returns data compound of
parts of different versions 1 and 2.
This non-atomicity is different from the non-atomicity that takes
place in the kernel (local file systems): kernel guarantees
that all PAGE_SIZE reads with PAGE_SIZE-aligned offsets are
atomic (this is because reads and writes in kernel acquire
page locks). Whereas, in our case we'll have that F2 doesn't
necessarily have PAGE_SIZE-aligned offset.
That said it can happen that we'll get complaints from users,
who don't expect such non-atomicity. Moreover, in the case when
EOBs are HMACs for checking integrity, or authentication we'll
have false positives, as nobody guarantees that versions of HMAC
and respective data block will coincide.
Solution:
In this approach we need to serialize truncates, appending
writes and sequences RbRe (read block, read EOB).
Approach 2: Storing in file's body.
In this case EOBs are stored in file's body (via appending to
a file in the case of EOF, or interspacing a file with HMACs,
etc). So file with his EOBs is the whole from the standpoint
of Cloudfs, and there is no problems with atomicity specific
to Approach 1.
However, in this case all our files maintained by low-level
local fs will have increased sizes (added total size of all EOBs).
So that actual file size must be stored as additional attribute
(e.g. as xattr value).
->open() method of the high-level translator loads actual
file size to the cloudfs-specific part of inode via fetching
->getxattr(), so that it is persistent in the memory on server.
Any ->truncate() and appending ->write() of the high-level
xlator update in-core and on-disk actual sizes simultaneously
(via fetching ->setxattr() for the last one). This actual size
is what should be returned to user by ->fstat(), ->lookup(),
etc. as st_size.
4:24 p.m.
New subject: [Gluster-devel] Handling EOBs in CloudFS
How would you handle ->lseek()? Not so much of an issue for Approach 1, but
with interleaved HMACs...
On Thu, Jul 14, 2011 at 5:01 PM, Edward Shishkin <edward(a)redhat.com> wrote:
Hello everyone,
any comments, suggestions are welcome..
Handling EOBs (end-of-blocks) for transparent
encryption, checking integrity and data authentication
DRAFT
This was designed for CloudFS, which uses 2-level protocol (high and
low) supported by xlators which reside on server and client sides
respectively.
Definition of EOB. Storage class
If file size isn't a multiple of cblock (cipher block) size, then we
also need to store special padding needed to decrypt its last block
with some cipher modes like CBC. This padding contains a part of
ciphertext and must be considered as a part of this file. We'll call
this padding end-of-file (EOF). If plain text has size a multiple of
cblock size, then encrypted file won't have (or will have empty) EOF.
Signatures (HMACs, etc) for checking integrity, data authentication,
etc. have the same nature as EOF. Every such signature is created
for some logical block in a file. This is not a padding though, as
in the case of EOF, but anyway such signatures are associated with
file's data, and we'll consider a class of object, which includes
EOFs, HMACs, etc, and call them EOBs (end-of-block).
We define storage class of EOBs as "data", i.e. this can be considered
as part of file's data: we can not read/write data block without
reading/writing its EOB.
Storing EOBs. Approaches and Issues
Approach 1: Storing EOBs as xattr values.
In this case we store a file in parts which are not adjacent
from the standpoint of Cloudfs. That said we need to split
read, and this makes this operation inatomic. This means
that read(2) will return data compound of parts of different
"versions".
Example:
Suppose we have a file F stored in 2 different parts F1 and F2.
Process A writes a file F (to be of version 1);
Process B reads a file F (part F1);
Process C writes a file F (to be of version 2);
Process B reads a file F (part F2);
As the result process B returns data compound of
parts of different versions 1 and 2.
This non-atomicity is different from the non-atomicity that takes
place in the kernel (local file systems): kernel guarantees
that all PAGE_SIZE reads with PAGE_SIZE-aligned offsets are
atomic (this is because reads and writes in kernel acquire
page locks). Whereas, in our case we'll have that F2 doesn't
necessarily have PAGE_SIZE-aligned offset.
That said it can happen that we'll get complaints from users,
who don't expect such non-atomicity. Moreover, in the case when
EOBs are HMACs for checking integrity, or authentication we'll
have false positives, as nobody guarantees that versions of HMAC
and respective data block will coincide.
Solution:
In this approach we need to serialize truncates, appending
writes and sequences RbRe (read block, read EOB).
Approach 2: Storing in file's body.
In this case EOBs are stored in file's body (via appending to
a file in the case of EOF, or interspacing a file with HMACs,
etc). So file with his EOBs is the whole from the standpoint
of Cloudfs, and there is no problems with atomicity specific
to Approach 1.
However, in this case all our files maintained by low-level
local fs will have increased sizes (added total size of all EOBs).
So that actual file size must be stored as additional attribute
(e.g. as xattr value).
->open() method of the high-level translator loads actual
file size to the cloudfs-specific part of inode via fetching
->getxattr(), so that it is persistent in the memory on server.
Any ->truncate() and appending ->write() of the high-level
xlator update in-core and on-disk actual sizes simultaneously
(via fetching ->setxattr() for the last one). This actual size
is what should be returned to user by ->fstat(), ->lookup(),
etc. as st_size.
______________________________**_________________
Gluster-devel mailing list
Gluster-devel(a)nongnu.org
https://lists.nongnu.org/**mailman/listinfo/gluster-devel<https://list...
4:50 p.m.
New subject: [Gluster-devel] Handling EOBs in CloudFS
On 07/14/2011 11:24 PM, Devon Miller wrote:
How would you handle ->lseek()? Not so much of an issue for
Approach 1,
but with interleaved HMACs...
Hello.
Offset in the interspaced file is
new_off = off + (off >> block_bits << hmac_bits);
So, I think this is not a big deal?
Thanks!
Edward.
On Thu, Jul 14, 2011 at 5:01 PM, Edward Shishkin <edward(a)redhat.com
<mailto:edward@redhat.com>> wrote:
Hello everyone,
any comments, suggestions are welcome..
Handling EOBs (end-of-blocks) for transparent
encryption, checking integrity and data authentication
DRAFT
This was designed for CloudFS, which uses 2-level protocol (high and
low) supported by xlators which reside on server and client sides
respectively.
Definition of EOB. Storage class
If file size isn't a multiple of cblock (cipher block) size, then we
also need to store special padding needed to decrypt its last block
with some cipher modes like CBC. This padding contains a part of
ciphertext and must be considered as a part of this file. We'll call
this padding end-of-file (EOF). If plain text has size a multiple of
cblock size, then encrypted file won't have (or will have empty) EOF.
Signatures (HMACs, etc) for checking integrity, data authentication,
etc. have the same nature as EOF. Every such signature is created
for some logical block in a file. This is not a padding though, as
in the case of EOF, but anyway such signatures are associated with
file's data, and we'll consider a class of object, which includes
EOFs, HMACs, etc, and call them EOBs (end-of-block).
We define storage class of EOBs as "data", i.e. this can be considered
as part of file's data: we can not read/write data block without
reading/writing its EOB.
Storing EOBs. Approaches and Issues
Approach 1: Storing EOBs as xattr values.
In this case we store a file in parts which are not adjacent
from the standpoint of Cloudfs. That said we need to split
read, and this makes this operation inatomic. This means
that read(2) will return data compound of parts of different
"versions".
Example:
Suppose we have a file F stored in 2 different parts F1 and F2.
Process A writes a file F (to be of version 1);
Process B reads a file F (part F1);
Process C writes a file F (to be of version 2);
Process B reads a file F (part F2);
As the result process B returns data compound of
parts of different versions 1 and 2.
This non-atomicity is different from the non-atomicity that takes
place in the kernel (local file systems): kernel guarantees
that all PAGE_SIZE reads with PAGE_SIZE-aligned offsets are
atomic (this is because reads and writes in kernel acquire
page locks). Whereas, in our case we'll have that F2 doesn't
necessarily have PAGE_SIZE-aligned offset.
That said it can happen that we'll get complaints from users,
who don't expect such non-atomicity. Moreover, in the case when
EOBs are HMACs for checking integrity, or authentication we'll
have false positives, as nobody guarantees that versions of HMAC
and respective data block will coincide.
Solution:
In this approach we need to serialize truncates, appending
writes and sequences RbRe (read block, read EOB).
Approach 2: Storing in file's body.
In this case EOBs are stored in file's body (via appending to
a file in the case of EOF, or interspacing a file with HMACs,
etc). So file with his EOBs is the whole from the standpoint
of Cloudfs, and there is no problems with atomicity specific
to Approach 1.
However, in this case all our files maintained by low-level
local fs will have increased sizes (added total size of all EOBs).
So that actual file size must be stored as additional attribute
(e.g. as xattr value).
->open() method of the high-level translator loads actual
file size to the cloudfs-specific part of inode via fetching
->getxattr(), so that it is persistent in the memory on server.
Any ->truncate() and appending ->write() of the high-level
xlator update in-core and on-disk actual sizes simultaneously
(via fetching ->setxattr() for the last one). This actual size
is what should be returned to user by ->fstat(), ->lookup(),
etc. as st_size.
_________________________________________________
Gluster-devel mailing list
Gluster-devel(a)nongnu.org <mailto:Gluster-devel@nongnu.org>
https://lists.nongnu.org/__mailman/listinfo/gluster-devel
<https://lists.nongnu.org/mailman/listinfo/gluster-devel>
12:13 p.m.
New subject: [Gluster-devel] Handling EOBs in CloudFS
On Thu, Jul 14, 2011 at 5:50 PM, Edward Shishkin <edward(a)redhat.com> wrote:
On 07/14/2011 11:24 PM, Devon Miller wrote:
> How would you handle ->lseek()? Not so much of an issue for Approach 1,
> but with interleaved HMACs...
>
Hello.
Offset in the interspaced file is
new_off = off + (off >> block_bits << hmac_bits);
So, I think this is not a big deal?
Thanks!
Edward.
Ah, of course, that makes perfect sense.
Re: Approach 1
Is there a way for process B to use file locking (or some other mechanism)
such that it could be guaranteed a consistent view of F?
Devon
> On Thu, Jul 14, 2011 at 5:01 PM, Edward Shishkin <edward(a)redhat.com
> <mailto:edward@redhat.com>> wrote:
>
> Hello everyone,
> any comments, suggestions are welcome..
>
>
>
> Handling EOBs (end-of-blocks) for transparent
> encryption, checking integrity and data authentication
>
> DRAFT
>
>
>
> This was designed for CloudFS, which uses 2-level protocol (high and
> low) supported by xlators which reside on server and client sides
> respectively.
>
>
> Definition of EOB. Storage class
>
>
> If file size isn't a multiple of cblock (cipher block) size, then we
> also need to store special padding needed to decrypt its last block
> with some cipher modes like CBC. This padding contains a part of
> ciphertext and must be considered as a part of this file. We'll call
> this padding end-of-file (EOF). If plain text has size a multiple of
> cblock size, then encrypted file won't have (or will have empty) EOF.
>
> Signatures (HMACs, etc) for checking integrity, data authentication,
> etc. have the same nature as EOF. Every such signature is created
> for some logical block in a file. This is not a padding though, as
> in the case of EOF, but anyway such signatures are associated with
> file's data, and we'll consider a class of object, which includes
> EOFs, HMACs, etc, and call them EOBs (end-of-block).
>
> We define storage class of EOBs as "data", i.e. this can be considered
> as part of file's data: we can not read/write data block without
> reading/writing its EOB.
>
>
> Storing EOBs. Approaches and Issues
>
>
> Approach 1: Storing EOBs as xattr values.
>
>
> In this case we store a file in parts which are not adjacent
> from the standpoint of Cloudfs. That said we need to split
> read, and this makes this operation inatomic. This means
> that read(2) will return data compound of parts of different
> "versions".
>
> Example:
>
> Suppose we have a file F stored in 2 different parts F1 and F2.
>
> Process A writes a file F (to be of version 1);
> Process B reads a file F (part F1);
> Process C writes a file F (to be of version 2);
> Process B reads a file F (part F2);
>
> As the result process B returns data compound of
> parts of different versions 1 and 2.
>
> This non-atomicity is different from the non-atomicity that takes
> place in the kernel (local file systems): kernel guarantees
> that all PAGE_SIZE reads with PAGE_SIZE-aligned offsets are
> atomic (this is because reads and writes in kernel acquire
> page locks). Whereas, in our case we'll have that F2 doesn't
> necessarily have PAGE_SIZE-aligned offset.
>
> That said it can happen that we'll get complaints from users,
> who don't expect such non-atomicity. Moreover, in the case when
> EOBs are HMACs for checking integrity, or authentication we'll
> have false positives, as nobody guarantees that versions of HMAC
> and respective data block will coincide.
>
> Solution:
>
> In this approach we need to serialize truncates, appending
> writes and sequences RbRe (read block, read EOB).
>
>
> Approach 2: Storing in file's body.
>
>
> In this case EOBs are stored in file's body (via appending to
> a file in the case of EOF, or interspacing a file with HMACs,
> etc). So file with his EOBs is the whole from the standpoint
> of Cloudfs, and there is no problems with atomicity specific
> to Approach 1.
>
> However, in this case all our files maintained by low-level
> local fs will have increased sizes (added total size of all EOBs).
> So that actual file size must be stored as additional attribute
> (e.g. as xattr value).
>
> ->open() method of the high-level translator loads actual
> file size to the cloudfs-specific part of inode via fetching
> ->getxattr(), so that it is persistent in the memory on server.
>
> Any ->truncate() and appending ->write() of the high-level
> xlator update in-core and on-disk actual sizes simultaneously
> (via fetching ->setxattr() for the last one). This actual size
> is what should be returned to user by ->fstat(), ->lookup(),
> etc. as st_size.
>
> ______________________________**___________________
> Gluster-devel mailing list
> Gluster-devel(a)nongnu.org
<mailto:Gluster-devel@nongnu.**org<Gluster-devel@nongnu.org>
> >
>
>
https://lists.nongnu.org/__**mailman/listinfo/gluster-devel<https://li...
>
<https://lists.nongnu.org/**mailman/listinfo/gluster-devel<https://list...
> **>
>
>
>
1:30 p.m.
New subject: [Gluster-devel] Handling EOBs in CloudFS
On Fri, 15 Jul 2011 13:13:22 -0400
Devon Miller <devon.c.miller(a)gmail.com> wrote:
Re: Approach 1
Is there a way for process B to use file locking (or some other
mechanism) such that it could be guaranteed a consistent view of F?
It shouldn't need to. With either approach, the encryption translator
(actually a partner translator on the server side) is responsible for
providing any serialization necessary to prevent clients from seeing
intermediate states in the middle of a write. With approach 1, this
includes the state where the file has been extended but the pad bytes
have not yet been moved into an xattr. With approach 2, this includes
the state where an extending write has been issued but its result is
not yet known - so that it would be invalid to issue the setxattr to
persist the true EOF. Much of this is covered in
http://cloudfs.org/dist/design.md
That's the document which had previously been posted on cloudfs-devel,
and to which Edward was responding. I apologize for his attempt to
continue the discussion on another list without providing appropriate
context.
11:09 p.m.
On Jul 14, 2011, at 5:01 PM, Edward Shishkin wrote:
In this case we store a file in parts which are not adjacent
from the standpoint of Cloudfs. That said we need to split
read, and this makes this operation inatomic. This means
that read(2) will return data compound of parts of different
"versions".
Example:
Suppose we have a file F stored in 2 different parts F1 and F2.
Process A writes a file F (to be of version 1);
Process B reads a file F (part F1);
Process C writes a file F (to be of version 2);
Process B reads a file F (part F2);
As the result process B returns data compound of
parts of different versions 1 and 2.
This non-atomicity is different from the non-atomicity that takes
place in the kernel (local file systems): kernel guarantees
that all PAGE_SIZE reads with PAGE_SIZE-aligned offsets are
atomic (this is because reads and writes in kernel acquire
page locks). Whereas, in our case we'll have that F2 doesn't
necessarily have PAGE_SIZE-aligned offset.
That said it can happen that we'll get complaints from users,
who don't expect such non-atomicity.
The users should never see this non-atomicity, so those user
complaints are mythical. All accesses to a physical file on one brick
go through a single translator which can (and should) enforce any
necessary serialization.
Moreover, in the case when
EOBs are HMACs for checking integrity, or authentication we'll
have false positives, as nobody guarantees that versions of HMAC
and respective data block will coincide.
Solution:
In this approach we need to serialize truncates, appending
writes and sequences RbRe (read block, read EOB).
Approach 2: Storing in file's body.
In this case EOBs are stored in file's body (via appending to
a file in the case of EOF, or interspacing a file with HMACs,
etc). So file with his EOBs is the whole from the standpoint
of Cloudfs, and there is no problems with atomicity specific
to Approach 1.
However, in this case all our files maintained by low-level
local fs will have increased sizes (added total size of all EOBs).
So that actual file size must be stored as additional attribute
(e.g. as xattr value).
So we're back to being non-atomic, since the actual write and the
xattr operation to set the EOF marker are separate. Note also that
POSIX does allow partial writes, so we can't really issue the xattr op
until we have the *result* from the write in hand. It does no good to
say that the EOF can be stored in memory, because that would allow the
inconsistency to become persistent in case of a crash. Every time EOF
changes, the change must be recorded persistently. This will happen
for every write that extends the file, in contrast to approach 1 in
which an xattr update is only necessary if the old and/or new EOF is
unaligned.
With HMACs (out of scope for this version of CloudFS anyway) there's
another problem. We can read or write intermingled data and HMACs as
one contiguous extent easily enough, but converting between this form
and the one the user expects will require allocation of a second
buffer and piecemeal copying of pieces from one to the other. That's
not exactly free, even on fast processors, especially when GlusterFS
tends to become CPU-bound on 10GbE or better already. Avoiding seeks
is a noble goal, and this approach might do that for small requests on
a single spinning disk with no caching, but there are other scenarious
- large requests, SSDs, multiple disks, warm/non-volatile caches -
where it would be notably worse than separate data and HMAC areas. We
need to evaluate the effect for different configurations and access
patterns carefully, not just make guesses without evidence. I
strongly suspect that the separate-HMAC approach would actually serve
us better, but even as the project leader I wouldn't presume to treat
my guesses as fact.
->open() method of the high-level translator loads actual
file size to the cloudfs-specific part of inode via fetching
->getxattr(), so that it is persistent in the memory on server.
Any ->truncate() and appending ->write() of the high-level
xlator update in-core and on-disk actual sizes simultaneously
"Simultaneously" has no meaning here. The on-disk update is issued
"downward" (toward the disk) and is completed asynchronously some time
later. The only way for the in-memory update to be *effectively*
simultaneous is if we block all other accesses while the on-disk
update is in progress. This is exactly the kind of serialization and
careful sequencing of sub-operations that you identify as a necessity
for approach 1, but it turns out that it's necessary for approach 2 as
well. The need for serialization does not help us distinguish between
the two approaches, because they have that in common.
P.S. This doesn't really belong on gluster-devel, since it's purely a
CloudFS issue.
4691
days inactive
4692
days old
cloudfs-devel@lists.fedorahosted.org
5 comments
3 participants
participants (3)
-
Devon Miller -
Edward Shishkin -
Jeff Darcy