<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 2:53 PM, Stephen Watt <span dir="ltr"><<a href="mailto:swatt@redhat.com" target="_blank">swatt@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Folks<br>
<br>
We are pleased to announce a major update to the glusterfs-hadoop project with the release of version 2.1. The glusterfs-hadoop project, available at The glusterfs-hadoop project team, provides an Apache licensed Hadoop FileSystem plugin which enables Apache Hadoop 1.x and 2.x to run directly on top of GlusterFS. This release includes a re-architected plugin which now extends existing functionality within Hadoop to run on local and POSIX File Systems.<br>
<br>
-- Overview --<br>
<br>
Apache Hadoop has a pluggable FileSystem Architecture. This means that if you have a filesystem or object store that you would like to use with Hadoop, you can create a Hadoop FileSystem plugin for it which will act as a mediator between the generic Hadoop FileSystem interface and your filesystem of choice. A popular example would be that over a million Hadoop clusters are spun up on Amazon every year, a lot of which use Amazon S3 as the Hadoop FileSystem.<br>
<br>
In order to configure the plugin, a specific deployment configuration is required. Firstly, it is required that the Hadoop JobTracker and TaskTrackers (or the Hadoop 2.x equivalents) are installed on servers within the gluster trusted storage pool for a given gluster volume. The JobTracker uses the plugin to query the extended attributes for job input files in gluster to ascertain file placement as well as the distribution of file replicas across the cluster. The TaskTrackers use the plugin to leverage a local fuse mount of the gluster volume in order to access the data required for the tasks. When the JobTracker receives a Hadoop job, it uses the locality information it ascertains via the plugin to send the tasks for the Hadoop Job to Hadoop TaskTrackers on servers that have the data required for the task within their local bricks. This ensures data is read from disk and not over the network. Please see the attached diagram which provides an overview of the entire solution for a Hadoop 1.x deployment.<br>
<br>
The community project, along with the documentation and available releases, is hosted within the Gluster Forge at <a href="http://forge.gluster.org/hadoop" target="_blank">http://forge.gluster.org/hadoop</a>. The glusterfs-hadoop project will also be available within the Fedora 20 release later this year, alongside fellow Fedora newcomer Apache Hadoop and the already available gluster project. The glusterfs-hadoop project team welcomes contributions and participation from the broader community.<br>
<br>
Stay tuned for upcoming posts around GlusterFS integration into the Apache Ambari and Fedora projects.<br>
<br>
Regards<br>
The glusterfs-hadoop project team<br>_______________________________________________<br>
Announce mailing list<br>
<a href="mailto:Announce@gluster.org">Announce@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/announce" target="_blank">http://supercolony.gluster.org/mailman/listinfo/announce</a><br>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div><div class="gmail_extra"><span style="font-family:arial,sans-serif;font-size:16px">Congratulations! This is great news!!</span><br>
</div><div class="gmail_extra"><span style="font-family:arial,sans-serif;font-size:16px"><br></span></div><div class="gmail_extra" style><span style="font-family:arial,sans-serif;font-size:16px">Avati</span></div></div>