<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 24, 2013 at 1:18 PM, Edward Shishkin <span dir="ltr"><<a href="mailto:edward@redhat.com" target="_blank">edward@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi all,<br>
<br>
So, here is the "all-in-one-translator" version represented by the<br>
Patch Set #2 at <a href="http://review.gluster.org/4667" target="_blank">review.gluster.org/4667</a><br>
<br>
Everything has been addressed except encryption in NFS mounts (see<br>
next mail for details). That is:<br>
<br>
. New design of EOF (end-of-file) handling;<br>
. No oplock translator on the server side;<br>
. All locks are acquired/released by the crypt translator;<br>
. Now we can encrypt srtiped and(or) replicated volumes.<br>
<br>
Common comments.<br>
<br>
In the new design all files on the server are "padded", whereas the<br>
real file size is stored as xattr. So we introduce a special layer<br>
in the crypt translator, which performs file size translations: every<br>
time when any callback returns struct iatt, we update its ia_size<br>
with the real (non-padded) value.<br>
<br>
The most unpleasant thing in this new design is FOP->readdirp_cbk():<br>
in this case we need N translations, i.e. N calls to the server (N is<br>
number of directory entries).<br>
<br>
To perform translations we spawn N children. We need a valid list of<br>
dirents after returning from FOP->readdirp_cbk() of previous<br>
translator, but we don't want to create a copy of this list (which<br>
can be large enough). For this reason we introduce a reference counter<br>
in struct gf_dirent_t and allocate dynamic structures gf_dirent_t<br>
(instead of on-stack ones), see respective changes in<br>
<br>
./libglusterfs/src/gf-dirent.c<br>
./libglusterfs/src/gf-dirent.h<br>
./xlators/cluster/dht/src/dht-common.c<br>
./xlators/protocol/client/src/client-rpc-fops.c<br></blockquote><div><br></div><div><br></div><div>[pasting from internal email reply]</div><div><br></div>I had a look at the way you are handling readdirplus. I think it is
overly complex. FOP->readdirplus() already has a parameter @xdata in
which you can request per-entry xattr replies.
<br>
<br>So in crypt_readdirp() you need to: dict_set(xdata, FSIZE_XATTR_PREFIX, 0);
<br>
<br>Once you do that, in crypt_readdirp_cbk, you can expect each gf_dirent_t
to have its dirent->dict set with FSIZE_XATTR_PREFIX.
<br>
<br>So you just need to iterate over replies in crypt_readdirp_cbk, update
each dirent->d_stat.ia_size with value from
dict_get_uint64(dirent->xdata, FSIZE_XATTR_PREFIX)
<br>
<br>Please look at how posix-acl does something very similar (loading
per-entry ACLs into respective inodes via xattrs returned in readdirplus)
<br>
<br><div>Avati</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Thanks,<br>
Edward.<br>
<div class="im"><br>
<br>
On Mon, 14 Oct 2013 14:27:01 -0700<br>
Anand Avati <<a href="mailto:avati@redhat.com">avati@redhat.com</a>> wrote:<br>
<br>
</div><div class=""><div class="h5">> Edward,<br>
> It looks like this patch requires a higher version of openssl (I<br>
> recall you have mentioned before that that dependency was on version<br>
> 1.0.1c? I checked yum update on the build server and the latest<br>
> available version is 1.0.0-27. Is there a "clean" way to get the<br>
> right version of openssl to a RHEL/CENTOS-6.x server?<br>
><br>
> Also note that the previous submission of the patch was at<br>
> <a href="http://review.gluster.org/4667" target="_blank">http://review.gluster.org/4667</a>. The recent on<br>
> (<a href="http://review.gluster.org/6086" target="_blank">http://review.gluster.org/6086</a>) has a different Change-Id: in the<br>
> commit log. It will be good if you can re-submit the patch with the<br>
> old Change-Id (and abandon #6086) so that we can maintain the history<br>
> of resubmission and the old work on records.<br>
><br>
> Thanks!<br>
> Avati<br>
><br>
> On 10/14/2013 07:26 AM, Edward Shishkin (Code Review) wrote:<br>
> > Edward Shishkin has uploaded a new change for review.<br>
> ><br>
> > <a href="http://review.gluster.org/6086" target="_blank">http://review.gluster.org/6086</a><br>
> ><br>
> ><br>
> > Change subject: Transparent data encryption and metadata<br>
> > authentication in the systems with non-trusted server (take<br>
> > II) ......................................................................<br>
> ><br>
> > Transparent data encryption and metadata authentication<br>
> > in the systems with non-trusted server (take II)<br>
> ><br>
> > This new functionality can be useful in various cloud technologies.<br>
> > It is implemented via a special encryption/crypt translator, which<br>
> > works on the client side and performs encryption and authentication;<br>
> ><br>
> > 1. Class of supported algorithms<br>
> ><br>
> > The crypt translator can support any atomic symmetric block cipher<br>
> > algorithms (which require to pad plain/cipher text before performing<br>
> > encryption/decryption transform (see glossary in atom.c for<br>
> > definitions). In particular, it can support algorithms with the EOF<br>
> > issue (which require to pad the end of file by extra-data).<br>
> ><br>
> > Crypt translator performs translations<br>
> > user -> (offset, size) -> (aligned-offset, padded-size) ->server<br>
> > (and backward), and resolves individual FOPs (write(), truncate(),<br>
> > etc) to read-modify-write sequences.<br>
> ><br>
> > A volume can contain files encrypted by different algorithms of the<br>
> > mentioned class. To change some option value just reconfigure the<br>
> > volume.<br>
> ><br>
> > Currently only one algorithm is supported: AES_XTS.<br>
> ><br>
> > Example of algorithms, which can not be supported by the crypt<br>
> > translator:<br>
> ><br>
> > 1. Asymmetric block cipher algorithms, which inflate data, e.g. RSA;<br>
> > 2. Symmetric block cipher algorithms with inline MACs for data<br>
> > authentication.<br>
> ><br>
> > 2. Implementation notes.<br>
> ><br>
> > a) Atomic algorithms<br>
> ><br>
> > Since any process in a stackable file system manipulates with local<br>
> > data (which can be obsoleted by local data of another process), any<br>
> > atomic cipher algorithm without proper support can lead to non-POSIX<br>
> > behavior. To resolve the "collisions" we introduce locks: before<br>
> > performing FOP->read(), FOP->write(), etc. the process should first<br>
> > lock the file.<br>
> ><br>
> > b) Algorithms with EOF issue<br>
> ><br>
> > Such algorithms require to pad the end of file with some extra-data.<br>
> > Without proper support this will result in losing information about<br>
> > real file size. Keeping a track of real file size is a<br>
> > responsibility of the crypt translator. A special extended<br>
> > attribute with the name "trusted.glusterfs.crypt.att.size" is used<br>
> > for this purpose. All files contained in bricks of encrypted volume<br>
> > do have "padded" sizes.<br>
> ><br>
> > 3. Non-trusted servers and<br>
> > Metadata authentication<br>
> ><br>
> > We assume that server, where user's data is stored on is<br>
> > non-trusted. It means that the server can be subjected to various<br>
> > attacks directed to reveal user's encrypted personal data. We<br>
> > provide protection against such attacks.<br>
> ><br>
> > Every encrypted file has specific private attributes (cipher<br>
> > algorithm id, atom size, etc), which are packed to a string<br>
> > (so-called "format string") and stored as a special extended<br>
> > attribute with the name "trusted.glusterfs.crypt.att.cfmt". We<br>
> > protect the string from tampering. This protection is mandatory,<br>
> > hardcoded and is always on. Without such protection various attacks<br>
> > (based on extending the scope of per-file secret keys) are possible.<br>
> ><br>
> > Our authentication method has been developed in tight collaboration<br>
> > with Red Hat security team and is implemented as "metadata loader of<br>
> > version 1" (see file metadata.c). This method is NIST-compliant and<br>
> > is based on checking 8-byte per-hardlink MACs created(updated) by<br>
> > FOP->create(), FOP->link(), FOP->unlink(), FOP->rename() by the<br>
> > following unique entities:<br>
> ><br>
> > . file (hardlink) name;<br>
> > . verified file's object id (gfid).<br>
> ><br>
> > Every time, before manipulating with a file, we check it's MACs at<br>
> > FOP->open() time. Some FOPs don't require a file to be opened (e.g.<br>
> > FOP->truncate()). In such cases the crypt translator opens the file<br>
> > mandatory.<br>
> ><br>
> > 4. Generating keys<br>
> ><br>
> > Unique per-file keys are derived by NIST-compliant methods from the<br>
> ><br>
> > a) parent key;<br>
> > b) unique verified object-id of the file (gfid);<br>
> > Per-volume master key, provided by user at mount time is in the root<br>
> > of this "tree of keys".<br>
> ><br>
> > Those keys are used to:<br>
> ><br>
> > 1) encrypt/decrypt file data;<br>
> > 2) encrypt/decrypt file metadata;<br>
> > 3) create per-file and per-link MACs for metadata authentication.<br>
> ><br>
> > 5. Instructions<br>
> > Getting started with crypt translator<br>
> ><br>
> > Example:<br>
> ><br>
> > 1) Create a volume "myvol" by specifying the option "encrypt":<br>
> ><br>
> > # gluster volume create myvol encrypt pepelac:/vols/xvol<br>
> ><br>
> > 2) Set location (absolute pathname) of your master key:<br>
> ><br>
> > # gluster volume set myvol encryption.master-key /home/me/mykey<br>
> ><br>
> > 3) Set other options to override default options, if needed.<br>
> ><br>
> > 4) On the client side make sure that the file /home/me/mykey exists<br>
> > and contains proper per-volume master key (that is 256-bit AES<br>
> > key). This key has to be in hex form, i.e. should be represented<br>
> > by 64 symbols from the set {'0', ..., '9', 'a', ..., 'f'}.<br>
> > The key should start at the beginning of the file. All symbols<br>
> > at offsets >= 64 are ignored.<br>
> ><br>
> > 5) Mount the volume "myvol" on the client side:<br>
> ><br>
> > # glusterfs --volfile-server=pepelac --volfile-id=myvol /mnt<br>
> ><br>
> > After successful mount the file which contains master key may be<br>
> > removed. NOTE: Keeping the master key between mount sessions is<br>
> > in user's competence.<br>
> ><br>
> > **********************************************************************<br>
> ><br>
> > WARNING! Losing the master key will make content of all regular<br>
> > files inaccessible. Mount with improper master key allows to access<br>
> > content of directories: file names are not encrypted.<br>
> ><br>
> > **********************************************************************<br>
> ><br>
> > 6. Options of crypt translator<br>
> ><br>
> > 1) "master-key": specifies location (absolute pathname) of the file<br>
> > which contains per-volume master key. There is no default<br>
> > location for master key.<br>
> ><br>
> > 2) "data-key-size": specifies size of per-file key for data<br>
> > encryption Possible values:<br>
> > . "256" default value<br>
> > . "512"<br>
> ><br>
> > 3) "block-size": specifies atom size. Possible values:<br>
> > . "512"<br>
> > . "1024"<br>
> > . "2048"<br>
> > . "4096" default value;<br>
> ><br>
> > 7. Test cases<br>
> ><br>
> > Any workload, which involves the following file operations:<br>
> ><br>
> > ->create();<br>
> > ->open();<br>
> > ->readv();<br>
> > ->writev();<br>
> > ->truncate();<br>
> > ->ftruncate();<br>
> > ->link();<br>
> > ->unlink();<br>
> > ->rename();<br>
> > ->readdirp().<br>
> ><br>
> > 8. TODOs:<br>
> ><br>
> > 1) Currently size of IOs issued by crypt translator is restricted<br>
> > by block_size (4K by default). We can use larger IOs to improve<br>
> > performance.<br>
> ><br>
> > Change-Id: Idc3523a8752888d3dd10f4fe71aa38dfc53d9ded<br>
> > Signed-off-by: Edward Shishkin <<a href="mailto:edward@redhat.com">edward@redhat.com</a>><br>
> > ---<br>
> > M cli/src/cli-cmd-parser.c<br>
> > M <a href="http://configure.ac" target="_blank">configure.ac</a><br>
> > M doc/gluster.8<br>
> > M libglusterfs/src/gf-dirent.c<br>
> > M libglusterfs/src/gf-dirent.h<br>
> > M xlators/cluster/dht/src/dht-common.c<br>
> > M xlators/encryption/Makefile.am<br>
> > A xlators/encryption/crypt/Makefile.am<br>
> > A xlators/encryption/crypt/src/Makefile.am<br>
> > A xlators/encryption/crypt/src/atom.c<br>
> > A xlators/encryption/crypt/src/crypt-common.h<br>
> > A xlators/encryption/crypt/src/crypt-mem-types.h<br>
> > A xlators/encryption/crypt/src/crypt.c<br>
> > A xlators/encryption/crypt/src/crypt.h<br>
> > A xlators/encryption/crypt/src/data.c<br>
> > A xlators/encryption/crypt/src/keys.c<br>
> > A xlators/encryption/crypt/src/metadata.c<br>
> > A xlators/encryption/crypt/src/metadata.h<br>
> > M xlators/mgmt/glusterd/src/glusterd-store.c<br>
> > M xlators/mgmt/glusterd/src/glusterd-store.h<br>
> > M xlators/mgmt/glusterd/src/glusterd-volgen.c<br>
> > M xlators/mgmt/glusterd/src/glusterd-volume-ops.c<br>
> > M xlators/mgmt/glusterd/src/glusterd-volume-set.c<br>
> > M xlators/mgmt/glusterd/src/glusterd.h<br>
> > M xlators/protocol/client/src/client-rpc-fops.c<br>
> > 25 files changed, 8,733 insertions(+), 28 deletions(-)<br>
> ><br>
> ><br>
> > git pull ssh://<a href="http://git.gluster.org/glusterfs" target="_blank">git.gluster.org/glusterfs</a> refs/changes/86/6086/1<br>
> ><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
> <a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div></div>