<br><br><div class="gmail_quote">On Fri, Sep 4, 2009 at 7:31 PM, Brian Hirt <span dir="ltr"><<a href="mailto:bhirt@mobygames.com">bhirt@mobygames.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Raghavendra ,<br>
<br>
Thanks for looking into this, its great that you have identified the bug. Let's hope you can get a fix out soon since it's a pretty serious error that undermines data integrity and reliability of gluster. From your message, I can't quite tell if you are agreeing that there is a serious problem, or just explaining what's happening in gluster as an explanation that everything is working as planned.<br>
<br>
I understand what you are saying below, but from the viewpoint of a client application, there absolutely is data corruption. If a program is writing to a file, handling all errors reported to it by the operating system, and later reads from the same file and finds that the data it wrote to the file isn't in there that is fundamentally a data corruption issue with very serious implications for any program.<br>
<br>
Restarting servers is typical behavior for setups of all sizes. Machines require routine maintenance. It is guaranteed that servers will be brought offline for hardware changes and or operating system updates.<br>
<br>
The documentation is misleading, as it implies split brain as one of the new features in 2.0. I quote<br>
"Replicate correctness (includes cases of splitbrain, proper self-heal fixes etc) Replicate is one of GlusterFS's key features. It was questioned for its ability to function well in some split brain situations. There were some corner cases in AFR which needed to be re-worked. With 2.0.x releases, AFR has proper detection of split-brain situations and works smoothly in all cases."<br>
<br>
This documentation is also misleading and incorrect. The statement that there is no data corruption when reading a file is just sort of silly and appears to be an attempt to soften the actual truth that any opened, replicated file that is being written to during a restart will become corrupt. Then this statement "All file systems are vulnerable to such loses" is just another attempt to make it sound like gluster behaves like all other filesystems, when in fact that's not even close to the truth.<br>
</blockquote><div><br> The situation you're facing is because of self-heal not being done on open fds which in turn might lead to split brain scenarios. Self-heal not being done on open fds is a known issue and will be fixed. As far as the split brain scenarios, glusterfs can just identify that is a split-brain and manual intervention is needed to copy the correct file. And the documentation on split brain explains the reasons for it.<br>
<a href="http://www.gluster.org/docs/index.php/Understanding_AFR_Translator">http://www.gluster.org/docs/index.php/Understanding_AFR_Translator</a><br><br><br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<a href="http://gluster.org/docs/index.php/GlusterFS_Technical_FAQ#What_happens_in_case_of_hardware_or_GlusterFS_crash.3F" target="_blank">http://gluster.org/docs/index.php/GlusterFS_Technical_FAQ#What_happens_in_case_of_hardware_or_GlusterFS_crash.3F</a><br>
<br>
I would suggest you change it to something more accurate and specific:<br>
"WARNING: If gluster is used in environments where files are replicated and written to, you will experience data loss/corruption when you restart servers. The longer you leave a file open, the larger your exposure to this issue will be."<br>
<br>
If the documentation was a bit more honest about what gluster can and cannot do, instead of talking it up as the next best thing, it would go a long way towards helping the project out. Right now, after all the problems I've experienced, seen other people report on this list, along with the lack of responses or hand waving that these issues aren't serious, I have very little trust in this project. It's really a shame, because it seems like there is so much potential.<br>
<br>
Kind Regards,<br><font color="#888888">
<br>
Brian</font><div><div></div><div class="h5"><br>
<br>
<br>
On Sep 3, 2009, at 10:28 PM, Raghavendra G wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
with write behind being removed from configuration, success is not returned for<br>
writes which are failed. The above situation is caused because of replicate in<br>
the setup. If replicate is removed from the test setup, this issue is not<br>
<br>
observed. As an example consider replicate over 2 nodes and following sequence<br>
<br>
of operations.<br>
<br>
1. start writing on gluster mount, data is replicated to both the nodes.<br>
2. stop node1. Application still receives success on writes, but data is<br>
<br>
written only to node2.<br>
<br>
3. restart node1. Application still receives success on writes, but data is<br>
still not written to node1, since the file is no longer open on node1. Also<br>
note that self-heal will not be done from node2 to node1 since replicate does<br>
<br>
not support self-heal on open fds yet.<br>
<br>
4. stop node2. Application receives either ENOTCONN or EBADFD, based on the<br>
child from which replicate received the last reply for write. subvolume<br>
corresponding to node1 returns EBADFD and that of node2 returns ENOTCONN.<br>
<br>
5. Application sensing writes are failing, issues an open on the file. Note<br>
<br>
that node2 is still down. If it were to be up, data on node1 would be synced<br>
from node2.<br>
6. Now writes happen only on node1. Note that file on node1 was missing some<br>
<br>
writes happened on node2. Now, node2 will miss the future writes, leading to a<br>
<br>
split-brain situation.<br>
7. Bring up the node1. If open were to happen now, when both nodes are up,<br>
replicate identifies this as a split brain situation and a manual intervention<br>
<br>
is needed to identify the "more" correct version of file and remove the other.<br>
<br>
Then replicate copies the more correct version of file to other node.<br>
<br>
Hence the issue here is not writes being failed. Writes are happening but there<br>
<br>
is a split-brain situation because of the way servers have been restarted.<br>
<br>
<br>
On Wed, Sep 2, 2009 at 8:06 PM, Brian Hirt <<a href="mailto:bhirt@mobygames.com" target="_blank">bhirt@mobygames.com</a>> wrote:<br>
<br>
On Sep 2, 2009, at 7:12 AM, Vijay Bellur wrote:<br>
<br>
Brian Hirt wrote:<br>
<br>
The first part of this problem (open files not surviving gluster restarts) seems like a pretty major design flaw that needs to be fixed.<br>
Yes, we do know that this is a problem and we have our sights set on solving this.<br>
<br>
That is good to know. Do you know if is this planned on being back ported into 2.0 or is it going to be part of 2.1? Is there a bug report id so we can follow the progress?<br>
<br>
<br>
<br>
The second part (gluster not reporting the error to the writer when gluster chokes) is a critical problem that needs to be fixed.<br>
<br>
This is a bug in the write-behind translator and bug 242 has been tracked to address this.<br>
<br>
A discussion from the mailing list archives which could be of interest to you for the tail -f problem:<br>
<br>
<a href="http://gluster.org/pipermail/gluster-users/20090113/001362.html" target="_blank">http://gluster.org/pipermail/gluster-users/20090113/001362.html</a><br>
<br>
<br>
Is there any additional information I can provide in this bug report? I have disabled the following section from my test clients and can confirm that some errors that were not being reported are now being sent back to the writer program. It's certainly an improvement over no errors being reported.<br>
<br>
volume writebehind<br>
type performance/write-behind<br>
option window-size 1MB<br>
subvolumes distribute<br>
end-volume<br>
<br>
I've also discovered, that this problem is not isolated to the writebehind module. While some errors are being sent back to the writer, there is still data corruption in the gluster created file. Gluster is still reporting success to the writer when writes have failed. I have a simple program that writes 1, 2, 3, 4 ... N to a file at the rate of 100 lines per second. Whenever the writer gets an error returned from write() it waits a second, reopens the file and continues writing. While this writer is writing, I restart the gluster nodes one by one. Once this is done, I stop the writter and check it for corruption.<br>
<br>
One interesting observation I have made is that when restarting the gluster servers, sometimes errorno EBADFD is returned and sometimes it's ENOTCONN. When errno is ENOTCONN (107 in ubuntu 9.04) the file is not corrupted. When errno is EBADFD (77 in ubuntu 9.04) there is file corruption. These statements are based on a limited number of test runs, but were always true for me.<br>
<br>
Some sample output of some tests:<br>
<br>
bhirt@ubuntu:~/gluster-tests$ rm -f /unify/m/test1.2009-09-02 && ./write-numbers /unify/m/test1.2009-09-02<br>
problems writting to fd, reopening logfile (errno = 77) in one second<br>
^C<br>
bhirt@ubuntu:~/gluster-tests$ ./check-numbers /unify/m/test1.2009-09-02<br>
169 <> 480<br>
bhirt@ubuntu:~/gluster-tests$ rm -f /unify/m/test1.2009-09-02 && ./write-numbers /unify/m/test1.2009-09-02<br>
problems writting to fd, reopening logfile (errno = 107) in one second<br>
^C<br>
<br>
bhirt@ubuntu:~/gluster-tests$ ./check-numbers /unify/m/test1.2009-09-02<br>
OK<br>
<br>
The programs I use to test this are:<br>
<br>
bhirt@ubuntu:~/gluster-tests$ cat write-numbers.c check-numbers<br>
#include <stdio.h><br>
#include <stdlib.h><br>
#include <errno.h><br>
#include <fcntl.h><br>
#include <unistd.h><br>
#include <string.h><br>
<br>
#define BUFSIZE 65536<br>
<br>
/* write 100 entries per second */<br>
#define WRITE_DELAY 1000000 / 100<br>
<br>
int open_testfile(char *testfile)<br>
{<br>
int fd;<br>
<br>
fd = open(testfile, O_WRONLY | O_CREAT | O_APPEND, 0666);<br>
<br>
if (fd < 0) {<br>
perror("open");<br>
exit(2);<br>
}<br>
<br>
return(fd);<br>
}<br>
<br>
void usage(char *s)<br>
{<br>
fprintf(stderr, "\nusage: %s testfile\n\n",s);<br>
}<br>
<br>
int main (int argc, char **argv)<br>
{<br>
char buf[BUFSIZE];<br>
int logfd;<br>
int nread;<br>
int counter = 0;<br>
<br>
<br>
if (argc != 2) {<br>
usage(argv[0]);<br>
exit(1);<br>
}<br>
<br>
logfd = open_testfile(argv[1]);<br>
<br>
/* loop endlessly */<br>
for (;;) {<br>
<br>
snprintf(buf, sizeof(buf), "%d\n",counter);<br>
nread = strnlen(buf,sizeof(buf));<br>
<br>
/* write data */<br>
int nwrite = write(logfd, buf, nread);<br>
<br>
if (nwrite == nread) {<br>
counter++;<br>
usleep(WRITE_DELAY);<br>
} else {<br>
/* restarted gluster nodes give this error in 2.0.6 */<br>
if (errno == EBADFD || errno == ENOTCONN)<br>
{<br>
/* wait a second before re-opening the file */<br>
fprintf(stderr,"problems writting to fd, reopening logfile (errno = %d) in one second\n",errno);<br>
sleep(1);<br>
<br>
/* reopen log file, and set write again flag so the data tries to get written back */<br>
logfd = open_testfile(argv[1]);<br>
}<br>
else<br>
{<br>
perror("write");<br>
exit(2);<br>
}<br>
}<br>
}<br>
}<br>
<br>
#!/usr/bin/perl<br>
<br>
use strict;<br>
use warnings;<br>
<br>
my $i=0;<br>
<br>
while (<>) { die "$i <> $_" if $i++ != $_; }<br>
print STDERR "OK\n";<br>
<br>
The client log file during one of the tests I ran.<br>
<br>
[2009-09-02 09:59:23] E [saved-frames.c:165:saved_frames_unwind] remote1: forced unwinding frame type(1) op(FINODELK)<br>
[2009-09-02 09:59:23] N [client-protocol.c:6246:notify] remote1: disconnected<br>
[2009-09-02 09:59:23] E [socket.c:745:socket_connect_finish] remote1: connection to <a href="http://10.0.1.31:6996" target="_blank">10.0.1.31:6996</a> failed (Connection refused)<br>
[2009-09-02 09:59:26] N [client-protocol.c:5559:client_setvolume_cbk] remote1: Connected to <a href="http://10.0.1.31:6996" target="_blank">10.0.1.31:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:30] E [saved-frames.c:165:saved_frames_unwind] remote2: forced unwinding frame type(1) op(WRITE)<br>
[2009-09-02 09:59:30] W [fuse-bridge.c:1534:fuse_writev_cbk] glusterfs-fuse: 153358: WRITE => -1 (Transport endpoint is not connected)<br>
[2009-09-02 09:59:30] N [client-protocol.c:6246:notify] remote2: disconnected<br>
[2009-09-02 09:59:30] E [socket.c:745:socket_connect_finish] remote2: connection to <a href="http://10.0.1.32:6996" target="_blank">10.0.1.32:6996</a> failed (Connection refused)<br>
[2009-09-02 09:59:33] N [client-protocol.c:5559:client_setvolume_cbk] remote2: Connected to <a href="http://10.0.1.32:6996" target="_blank">10.0.1.32:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:34] N [client-protocol.c:5559:client_setvolume_cbk] remote1: Connected to <a href="http://10.0.1.31:6996" target="_blank">10.0.1.31:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:37] E [saved-frames.c:165:saved_frames_unwind] remote1: forced unwinding frame type(1) op(FINODELK)<br>
[2009-09-02 09:59:37] W [fuse-bridge.c:1534:fuse_writev_cbk] glusterfs-fuse: 153923: WRITE => -1 (File descriptor in bad state)<br>
[2009-09-02 09:59:37] N [client-protocol.c:6246:notify] remote1: disconnected<br>
[2009-09-02 09:59:40] N [client-protocol.c:5559:client_setvolume_cbk] remote1: Connected to <a href="http://10.0.1.31:6996" target="_blank">10.0.1.31:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:41] N [client-protocol.c:5559:client_setvolume_cbk] remote2: Connected to <a href="http://10.0.1.32:6996" target="_blank">10.0.1.32:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:44] N [client-protocol.c:5559:client_setvolume_cbk] remote1: Connected to <a href="http://10.0.1.31:6996" target="_blank">10.0.1.31:6996</a>, attached to remote volume 'brick'.<br>
[2009-09-02 09:59:51] W [fuse-bridge.c:882:fuse_err_cbk] glusterfs-fuse: 155106: FLUSH() ERR => -1 (File descriptor in bad state)<br>
[2009-09-02 09:59:51] W [fuse-bridge.c:882:fuse_err_cbk] glusterfs-fuse: 155108: FLUSH() ERR => -1 (File descriptor in bad state)<br>
<br>
<br>
<br>
Regards,<br>
Vijay<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
<br>
-- <br>
Raghavendra G<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>