<div>Are all the clocks in sync on the servers?</div><div><br></div><div>You probably should configure memcache to be the banner cache (quick search for "OpenX banner cache" shows that is an option). You can't have 4 clients all opening/writing to the same file all at the same time.<br>
<br><div class="gmail_quote">On Mon, Mar 12, 2012 at 6:55 AM, Sean Fulton <span dir="ltr"><<a href="mailto:sean@gcnpublishing.com">sean@gcnpublishing.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have set up a replicated, four-node gluster config for a web farm. The idea is that each web node is its own server, and will have its own copy of the entire web root locally. It then serves the cluster to itself. We're running it over dual GigE NICs bonded.<br>
<br>
The problem I am having is when we switch live traffic to nodes in the cluster, they almost immediately get out of sync. The issue seems to be with cache files that are read/written a lot. Here is an excerpt pointing to issues with our OpenX banner cache:<br>
<br>
[2012-02-25 18:53:04.198326] E [afr-self-heal-common.c:2074:<u></u>afr_self_heal_completion_cbk] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal failed on /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php<br>
[2012-02-25 18:53:04.199191] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php: gfid differs on subvolume 0 (53fa373a-3830-4c5e-aa22-<u></u>6ed35c947d97, c12e0cdd-9b6c-4988-b793-<u></u>819db0472780)<br>
[2012-02-25 18:53:04.199210] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php: gfid differs on subvolume 0 (53fa373a-3830-4c5e-aa22-<u></u>6ed35c947d97, c12e0cdd-9b6c-4988-b793-<u></u>819db0472780)<br>
[2012-02-25 18:53:04.199219] W [afr-common.c:882:afr_detect_<u></u>self_heal_by_iatt] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php: gfid different on subvolume<br>
[2012-02-25 18:53:04.199236] I [afr-common.c:1038:afr_launch_<u></u>self_heal] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal triggered. path: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php<br>
[2012-02-25 18:53:04.200752] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php: gfid differs on subvolume 0 (53fa373a-3830-4c5e-aa22-<u></u>6ed35c947d97, c12e0cdd-9b6c-4988-b793-<u></u>819db0472780)<br>
[2012-02-25 18:53:04.200971] I [afr-self-heal-common.c:963:<u></u>afr_sh_missing_entries_done] 0-web-pub-replicate-0: split brain found, aborting selfheal of /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php<br>
[2012-02-25 18:53:04.200986] E [afr-self-heal-common.c:2074:<u></u>afr_self_heal_completion_cbk] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal failed on /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f8e7a8862cb80b4933c58acdf65aae<u></u>f5.php<br>
[2012-02-25 18:53:04.202159] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 1 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.202178] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 1 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.202188] W [afr-common.c:882:afr_detect_<u></u>self_heal_by_iatt] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid different on subvolume<br>
[2012-02-25 18:53:04.202204] I [afr-common.c:1038:afr_launch_<u></u>self_heal] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal triggered. path: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.203463] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.203678] I [afr-self-heal-common.c:963:<u></u>afr_sh_missing_entries_done] 0-web-pub-replicate-0: split brain found, aborting selfheal of /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.203693] E [afr-self-heal-common.c:2074:<u></u>afr_self_heal_completion_cbk] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal failed on /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.204759] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.204781] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.204800] W [afr-common.c:882:afr_detect_<u></u>self_heal_by_iatt] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid different on subvolume<br>
[2012-02-25 18:53:04.204818] I [afr-common.c:1038:afr_launch_<u></u>self_heal] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal triggered. path: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.206150] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.206384] I [afr-self-heal-common.c:963:<u></u>afr_sh_missing_entries_done] 0-web-pub-replicate-0: split brain found, aborting selfheal of /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.206400] E [afr-self-heal-common.c:2074:<u></u>afr_self_heal_completion_cbk] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal failed on /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.207725] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.207746] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
[2012-02-25 18:53:04.207756] W [afr-common.c:882:afr_detect_<u></u>self_heal_by_iatt] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid different on subvolume<br>
[2012-02-25 18:53:04.207772] I [afr-common.c:1038:afr_launch_<u></u>self_heal] 0-web-pub-replicate-0: background meta-data data missing-entry self-heal triggered. path: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php<br>
[2012-02-25 18:53:04.209217] W [afr-common.c:1121:afr_<u></u>conflicting_iattrs] 0-web-pub-replicate-0: /cust/site1/www/openx/var/<u></u>cache/deliverycache_<u></u>f901ff39b456df599289c590ed89b1<u></u>9d.php: gfid differs on subvolume 0 (375e1754-0420-4e26-9176-<u></u>bb2128c6596b, 3e9eca35-3351-450e-b8ab-<u></u>c62785968953)<br>
<br>
Nodes and network are fine. I have tried mounting the volumes using both the Gluster native client and with the Gluster NFS client but get the same results. It's killing performance.<br>
<br>
Here is the config:<br>
<br>
1: volume web-pub-client-0<br>
2: type protocol/client<br>
3: option remote-host web-web1<br>
4: option remote-subvolume /glusterfs/pub<br>
5: option transport-type tcp<br>
6: end-volume<br>
7:<br>
8: volume web-pub-client-1<br>
9: type protocol/client<br>
10: option remote-host web-web2<br>
11: option remote-subvolume /glusterfs/pub<br>
12: option transport-type tcp<br>
13: end-volume<br>
14:<br>
15: volume web-pub-client-2<br>
16: type protocol/client<br>
17: option remote-host web-web3<br>
18: option remote-subvolume /glusterfs/pub<br>
19: option transport-type tcp<br>
20: end-volume<br>
21:<br>
22: volume web-pub-client-3<br>
23: type protocol/client<br>
24: option remote-host web-web4<br>
25: option remote-subvolume /glusterfs/pub<br>
26: option transport-type tcp<br>
27: end-volume<br>
28:<br>
29: volume web-pub-replicate-0<br>
30: type cluster/replicate<br>
31: subvolumes web-pub-client-0 web-pub-client-1 web-pub-client-2 web-pub-client-3<br>
32: end-volume<br>
33:<br>
34: volume web-pub-write-behind<br>
35: type performance/write-behind<br>
36: subvolumes web-pub-replicate-0<br>
37: end-volume<br>
38:<br>
39: volume web-pub-read-ahead<br>
40: type performance/read-ahead<br>
41: subvolumes web-pub-write-behind<br>
42: end-volume<br>
43:<br>
44: volume web-pub-io-cache<br>
45: type performance/io-cache<br>
46: option cache-size 256MB<br>
47: subvolumes web-pub-read-ahead<br>
48: end-volume<br>
49:<br>
50: volume web-pub-quick-read<br>
51: type performance/quick-read<br>
52: option cache-size 256MB<br>
53: subvolumes web-pub-io-cache<br>
54: end-volume<br>
55:<br>
56: volume web-pub<br>
57: type debug/io-stats<br>
58: option latency-measurement off<br>
59: option count-fop-hits off<br>
60: subvolumes web-pub-quick-read<br>
61: end-volume<br>
62:<br>
63: volume nfs-server<br>
64: type nfs/server<br>
65: option nfs.dynamic-volumes on<br>
66: option rpc-auth.addr.web-pub.allow *<br>
67: option nfs3.web-pub.volume-id ac556d2e-e8a9-4857-bd17-<u></u>cab603820fcb<br>
68: subvolumes web-pub<br>
69: end-volume<br>
<br>
<br>
Any ideas or help would be greatly appreciated.<span class="HOEnZb"><font color="#888888"><br>
<br>
sean<br>
<br>
-- <br>
Sean Fulton<br>
GCN Publishing, Inc.<br>
Internet Design, Development and Consulting For Today's Media Companies<br>
<a href="http://www.gcnpublishing.com" target="_blank">http://www.gcnpublishing.com</a><br>
<a href="tel:%28203%29%20665-6211%2C%20x203" value="+12036656211" target="_blank">(203) 665-6211, x203</a><br>
<br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/<u></u>mailman/listinfo/gluster-users</a><br>
</font></span></blockquote></div><br></div>