<br><br><div class="gmail_quote">On Tue, May 22, 2012 at 3:35 AM, alfred de sollize <span dir="ltr"><<a href="mailto:desollize@gmail.com" target="_blank">desollize@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi All,<br>
We tried as Amar suggested the client io-threads but still I get following errors in the log files <br>
<br>
=================================================================================<br>
[2012-05-21 17:54:56.491110] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 17:56:30.328936] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 18:02:16.150848] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE RENAME() seeking deep resolution of
/tia/mom4p1/work/noriver/ascii/19610401.fms.out<br>
[2012-05-21 18:02:16.151193] D [server-resolve.c:201:resolve_deep_cbk]
0-home-export-server: /tia/mom4p1/work/noriver/ascii/19610401.fms.out:
failed to resolve (No such file or directory)<br>
[2012-05-21 18:02:16.151231] D
[server-resolve.c:170:resolve_deep_continue] 0-home-export-server:
return value of resolve_*_simple 1<br>
[2012-05-21 18:04:16.468] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=4<br>
[2012-05-21 18:04:16.533] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=3<br>
[2012-05-21 18:04:16.602] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=2<br>
[2012-05-21 18:04:31.348] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=1<br>
[2012-05-21 18:22:12.295387] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 18:24:03.353140] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 18:24:05.764889] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 18:26:27.378861] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 19:03:37.89056] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not
implemented)<br>
[2012-05-21 19:04:23.410529] D [io-threads.c:2017:__iot_workers_scale]
0-home-export-io-threads: scaled threads to 2 (queue_size=4/2)<br>
[2012-05-21 19:05:09.355671] D [io-threads.c:2017:__iot_workers_scale]
0-home-export-io-threads: scaled threads to 3 (queue_size=8/3)<br>
[2012-05-21 19:05:18.109547] D [io-threads.c:2017:__iot_workers_scale]
0-home-export-io-threads: scaled threads to 4 (queue_size=16/4)<br>
[2012-05-21 19:05:38.403356] D [io-threads.c:2017:__iot_workers_scale]
0-home-export-io-threads: scaled threads to 5 (queue_size=32/5)<br>
[2012-05-21 19:10:37.850018] D [posix.c:769:posix_do_chmod]
0-home-export-posix: /home9/gfskurt/kurt/CCSM/new/scripts/i128/.hf.swp
(Function not implemented)<br>
[2012-05-21 19:37:29.997392] D [posix.c:769:posix_do_chmod]
0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp
(Function not implemented)<br>
[2012-05-21 19:55:02.254952] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE OPEN() seeking deep resolution of
/tia/mom4p1/work/noriver/diag_table<br>
[2012-05-21 19:55:02.387242] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE RENAME() seeking deep resolution of
/tia/mom4p1/work/noriver/ascii/19610401.fms.out<br>
[2012-05-21 19:55:02.387592] D [server-resolve.c:201:resolve_deep_cbk]
0-home-export-server: /tia/mom4p1/work/noriver/ascii/19610401.fms.out:
failed to resolve (No such file or directory)<br>
[2012-05-21 19:55:02.387624] D
[server-resolve.c:170:resolve_deep_continue] 0-home-export-server:
return value of resolve_*_simple 1<br>
[2012-05-21 19:55:26.576525] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not
implemented)<br>
[2012-05-21 20:00:01.264940] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE UNLINK() seeking deep resolution of
/tia/mom4p1/work/noriver/history/<a href="http://19610401.ice_diag.nc" target="_blank">19610401.ice_diag.nc</a><br>
[2012-05-21 20:00:01.306673] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE UNLINK() seeking deep resolution of
/tia/mom4p1/work/noriver/history/<a href="http://19610401.ocean_diag.nc" target="_blank">19610401.ocean_diag.nc</a><br>
[2012-05-21 20:00:01.969034] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE UNLINK() seeking deep resolution of
/tia/mom4p1/work/noriver/history/<a href="http://19610401.ocean_temp.nc" target="_blank">19610401.ocean_temp.nc</a><br>
[2012-05-21 20:01:14.414520] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not
implemented)<br>
[2012-05-21 20:08:20.756821] D [server-resolve.c:256:resolve_path_deep]
0-/home9: RESOLVE RENAME() seeking deep resolution of
/tia/mom4p1/work/noriver/ascii/no_time_stamp.logfile.000000.out<br>
[2012-05-21 20:08:20.757176] D [server-resolve.c:201:resolve_deep_cbk]
0-home-export-server:
/tia/mom4p1/work/noriver/ascii/no_time_stamp.logfile.000000.out: failed
to resolve (No such file or directory)<br>
[2012-05-21 20:08:20.757214] D
[server-resolve.c:170:resolve_deep_continue] 0-home-export-server:
return value of resolve_*_simple 1<br>
[2012-05-21 20:17:50.736328] D [posix.c:769:posix_do_chmod]
0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp
(Function not implemented)<br>
[2012-05-21 20:26:34.648108] D [posix.c:769:posix_do_chmod]
0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp
(Function not implemented)<br>
[2012-05-21 20:33:56.293882] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/gfskurt/kurt/CCSM/new/scratch/i512/ocn/input/gx1v6_tavg_contents
(Function not implemented)<br>
[2012-05-21 20:34:14.798261] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not
implemented)<br>
[2012-05-21 20:34:14.840494] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not
implemented)<br>
[2012-05-21 20:34:14.933004] D [posix.c:769:posix_do_chmod]
0-home-export-posix:
/home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not
implemented)<br>
[2012-05-21 20:36:42.411] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=4<br>
[2012-05-21 20:36:42.533] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=3<br>
[2012-05-21 20:36:42.571] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=2<br>
[2012-05-21 20:39:29.233] D [io-threads.c:118:iot_worker] 0-home-export-io-threads: timeout, terminated. conf->curr_count=1<br>
[2012-05-21 22:24:07.309485] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-21 22:24:07.466879] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-22 02:48:07.317096] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
[2012-05-22 02:48:09.155900] D [posix.c:316:posix_lstat_with_gfid]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208)
[0x7f16165edda8]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf)
[0x7f16168025df]
(-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655)
[0x7f1616802505]))) 0-home-export-posix: failed to get gfid<br>
<br>
=============================================================================================================<br>
after which this brick would get disconnected and we get files missing and TENC errors<br>
Is there a limitation on simultaneous i-o that could be handled by glusterfs?<br>reg<br>Al<div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Tue, May 15, 2012 at 11:24 AM, Amar Tumballi <span dir="ltr"><<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 05/15/2012 09:02 AM, alfred de sollize wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Has anybody worked with ccsm , wrf on Glusterfs? These spawn huge number<br>
of threads for IO.<br>
What are xlators to disable and what oher tune-ups so it stops giving TENC<br>
Al<br>
<br>
<br>
</blockquote>
<br></div>
Considering you have lot of threads doing I/O, can you try 'gluster volume set <VOLNAME> client-io-threads enable' and see if it makes any difference?<br>
<br>
Regards,<br>
Amar<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<br>
<br>
<br>
On Sun, May 13, 2012 at 10:54 PM, alfred de sollize <<a href="mailto:desollize@gmail.com" target="_blank">desollize@gmail.com</a><br></div><div>
<mailto:<a href="mailto:desollize@gmail.com" target="_blank">desollize@gmail.com</a>>> wrote:<br>
<br>
Hi Amar,<br>
The version is in subject line - is Glusterfs-3.2.5<br>
<br>
TENC comes in clients for commands like - [cd /home] or [df -h /home]<br>
/home - Transport End point not connected<br>
it also appears in server logs,<br>
Attaching some part of client and server logs for one of the error<br>
bricks<br>
is some problem for application integrations also? We use Centos-6.1..<br>
Al<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Fri, May 11, 2012 at 11:32 AM, Amar Tumballi <<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a><br></div><div>
<mailto:<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>>> wrote:<br>
<br>
On 05/10/2012 09:39 PM, alfred de sollize wrote:<br>
<br>
We are setting up a 180 node cluster for weather modeling. 2<br>
Storage<br>
servers with 32GB Ram each. QDR INfiniband interconnect.<br>
When we run iozone with 1GB perthread (128Kb blocksize) from<br>
32 clients<br>
(2 iozone threads per client).<br>
The run succeeds however run fails for 64 clients and we<br>
start getting<br>
"Transport Endpoint not connected" errors.<br>
<br>
There are 10 bricks(5 from each server ) each of ~4.2TB<br>
making 42TB of<br>
export-volume that is fuse-mounted on the clients.<br>
<br>
There is no other error in the log files except for "TENC" .<br>
<br>
<br>
When you say log file, which files are you looking at. Also,<br>
which version of the glusterfs are you running ?<br>
<br>
-Amar<br>
<br>
<br>
<br>
<br>
<br></div>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/<u></u>mailman/listinfo/gluster-users</a><br>
</blockquote>
<br>
</blockquote></div><br>
</div></div></blockquote></div><br>