<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1323778239;
mso-list-type:hybrid;
mso-list-template-ids:1464924026 -1365743196 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"\(%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><!--ppd1000033--><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Answers inline below as well </span><span style='font-size:11.0pt;font-family:Wingdings;color:#1F497D'>J</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hope this helps.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#003D79'>James Burnash, Unix Engineering<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Nyamul Hassan<br><b>Sent:</b> Wednesday, May 11, 2011 10:04 AM<br><b>To:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] [SPAM?] Storage Design Overview<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span class=apple-style-span><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thank you for the prompt and insightful answer, James. My remarks are inline.<o:p></o:p></span></span></p><div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><p class=MsoNormal><span style='color:#500050'>1.</span><span style='font-size:7.0pt;color:#500050'> </span><span style='color:#500050'>Can we mount a GlusterFS on a client and expect it to provide sustained throughput near wirespeed? </span><span style='color:#1F497D'><No></span><span style='color:#500050'><o:p></o:p></span></p></div></div></div></blockquote><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#500050'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>In your scenario, what were the maximum read speeds that you observed?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> Read (using dd) approximately 60MB/sec to 100MB/sec.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><p><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#500050'>3.</span><span style='font-size:7.0pt;color:#500050'> </span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#500050'>Does it put extra pressure on the client?</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'> <What do you mean by pressure? My clients (HP ProLiant DL360 G5 Quad Core with 32GB RAM) show up to 2GB of memory usage when the native Gluster client is used for mounts – but that is dependent on what you set the client cache max for – in my case, 2GB. CPU utilization is usually negligible in my systems, network bandwidth utilization and I/O throughput … depend on what the files sizes and access patterns look like></span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#500050'><o:p></o:p></span></p></div></div></div></blockquote></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thx for the insight. Can you describe your current deployment a bit more, like configs of the storage nodes, and the client nodes, and what type of application you are using it for? Don't want to be too intrusive, just to get an idea on what others are doing.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>All on Gluster 3.1.3<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Servers:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>4 CentOS 5.5 (ProLiant DL370 G6 servers, Intel Xeon 3200 MHz),<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Each with:<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.25in'><span style='color:#1F497D'>Single P812 Smart Array Controller, <o:p></o:p></span></p><p class=MsoNormal style='margin-left:.25in'><span style='color:#1F497D'>Single MDS600 with 70 2TB SATA drives configured as RAID 50<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.25in'><span style='color:#1F497D'>48 MB RAM<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Clients:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>185 CentOS 5.2 (mostly DL360 G6).<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2 is the mount point for a Duplicated-Replicate volume of 4 servers.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Volume Name: pfs-ro1<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Type: Distributed-Replicate<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Status: Started<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Number of Bricks: 20 x 2 = 40<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Transport-type: tcp<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Bricks:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick1: jc1letgfs17-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick2: jc1letgfs18-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick3: jc1letgfs17-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick4: jc1letgfs18-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick5: jc1letgfs17-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick6: jc1letgfs18-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick7: jc1letgfs17-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick8: jc1letgfs18-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick9: jc1letgfs17-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick10: jc1letgfs18-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick11: jc1letgfs17-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick12: jc1letgfs18-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick13: jc1letgfs17-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick14: jc1letgfs18-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick15: jc1letgfs17-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick16: jc1letgfs18-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick17: jc1letgfs17-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick18: jc1letgfs18-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick19: jc1letgfs17-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick20: jc1letgfs18-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick21: jc1letgfs14-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick22: jc1letgfs15-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick23: jc1letgfs14-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick24: jc1letgfs15-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick25: jc1letgfs14-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick26: jc1letgfs15-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick27: jc1letgfs14-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick28: jc1letgfs15-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick29: jc1letgfs14-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick30: jc1letgfs15-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick11: jc1letgfs17-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick12: jc1letgfs18-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick13: jc1letgfs17-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick14: jc1letgfs18-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick15: jc1letgfs17-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick16: jc1letgfs18-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick17: jc1letgfs17-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick18: jc1letgfs18-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick19: jc1letgfs17-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick20: jc1letgfs18-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick21: jc1letgfs14-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick22: jc1letgfs15-pfs1:/export/read-only/g01<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick23: jc1letgfs14-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick24: jc1letgfs15-pfs1:/export/read-only/g02<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick25: jc1letgfs14-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick26: jc1letgfs15-pfs1:/export/read-only/g03<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick27: jc1letgfs14-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick28: jc1letgfs15-pfs1:/export/read-only/g04<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick29: jc1letgfs14-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick30: jc1letgfs15-pfs1:/export/read-only/g05<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick31: jc1letgfs14-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick32: jc1letgfs15-pfs1:/export/read-only/g06<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick33: jc1letgfs14-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick34: jc1letgfs15-pfs1:/export/read-only/g07<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick35: jc1letgfs14-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick36: jc1letgfs15-pfs1:/export/read-only/g08<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick37: jc1letgfs14-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick38: jc1letgfs15-pfs1:/export/read-only/g09<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick39: jc1letgfs14-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Brick40: jc1letgfs15-pfs1:/export/read-only/g10<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Options Reconfigured:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>diagnostics.brick-log-level: ERROR<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>cluster.metadata-change-log: on<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>diagnostics.client-log-level: ERROR<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>performance.stat-prefetch: on<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>performance.cache-size: 2GB<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>network.ping-timeout: 10<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thank you once again for your remarks.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Cheers,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>HASSAN <o:p></o:p></span></p></div></div></div></div><br /><br />
<P><FONT color=navy><FONT size=1>DISCLAIMER:<BR>This e-mail, and any attachments
thereto, is intended only for use by the addressee(s)named herein and<BR>may
contain legally privileged and/or confidential information. If you are not the
intended recipient of this<BR>e-mail, you are hereby notified that any
dissemination, distribution or copying of this e-mail and any
attachments</FONT><FONT color=navy><BR></FONT><FONT size=1>thereto, is strictly
prohibited. If you have received this in error, please immediately notify me and
permanently<BR>delete the original and any printout thereof.</FONT></FONT><FONT
size=1> </FONT><FONT color=navy size=1>E-mail transmission cannot be guaranteed
to be secure or error-free.<BR>The sender therefore does not accept liability
for any errors or omissions in the contents of this message which<BR>arise as a
result of e-mail transmission.<BR>NOTICE REGARDING PRIVACY AND
CONFIDENTIALITY<BR>Knight Capital Group may, at its discretion, monitor and
review the content of all e-mail communications.</FONT></P>
<P><A href="http://www.knight.com/"><FONT color=navy
size=1>http://www.knight.com</FONT></A><BR></P>
</body></html>