<div dir="ltr">Hi all. I have been running FS sanity on daily builds(glusterfs mounts only at this point) for a few days for a few days and I have been hitting a couple of problems:<div><br></div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">
================ final pass/fail report =================
Test Date: Sat Jul 5 01:53:00 EDT 2014
Total : [44]
Passed: [41]
Failed: [3]
Abort : [0]
Crash : [0]
---------------------------------------------------------
[ PASS ] FS Sanity Setup
[ PASS ] Running tests.
[ PASS ] FS SANITY TEST - arequal
[ PASS ] FS SANITY LOG SCAN - arequal
[ PASS ] FS SANITY LOG SCAN - bonnie
[ PASS ] FS SANITY TEST - glusterfs_build
[ PASS ] FS SANITY LOG SCAN - glusterfs_build
[ PASS ] FS SANITY TEST - compile_kernel
[ PASS ] FS SANITY LOG SCAN - compile_kernel
[ PASS ] FS SANITY TEST - dbench
[ PASS ] FS SANITY LOG SCAN - dbench
[ PASS ] FS SANITY TEST - dd
[ PASS ] FS SANITY LOG SCAN - dd
[ PASS ] FS SANITY TEST - ffsb
[ PASS ] FS SANITY LOG SCAN - ffsb
[ PASS ] FS SANITY TEST - fileop
[ PASS ] FS SANITY LOG SCAN - fileop
[ PASS ] FS SANITY TEST - fsx
[ PASS ] FS SANITY LOG SCAN - fsx
[ PASS ] FS SANITY LOG SCAN - fs_mark
[ PASS ] FS SANITY TEST - iozone
[ PASS ] FS SANITY LOG SCAN - iozone
[ PASS ] FS SANITY TEST - locks
[ PASS ] FS SANITY LOG SCAN - locks
[ PASS ] FS SANITY TEST - ltp
[ PASS ] FS SANITY LOG SCAN - ltp
[ PASS ] FS SANITY TEST - multiple_files
[ PASS ] FS SANITY LOG SCAN - multiple_files
[ PASS ] FS SANITY TEST - posix_compliance
[ PASS ] FS SANITY LOG SCAN - posix_compliance
[ PASS ] FS SANITY TEST - postmark
[ PASS ] FS SANITY LOG SCAN - postmark
[ PASS ] FS SANITY TEST - read_large
[ PASS ] FS SANITY LOG SCAN - read_large
[ PASS ] FS SANITY TEST - rpc
[ PASS ] FS SANITY LOG SCAN - rpc
[ PASS ] FS SANITY TEST - syscallbench
[ PASS ] FS SANITY LOG SCAN - syscallbench
[ PASS ] FS SANITY TEST - tiobench
[ PASS ] FS SANITY LOG SCAN - tiobench
[ PASS ] FS Sanity Cleanup
[ FAIL ] FS SANITY TEST - bonnie
[ FAIL ] FS SANITY TEST - fs_mark
[ FAIL ] /rhs-tests/beaker/rhs/auto-tests/components/sanity/fs-sanity-tests-v2</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">
Bonnie++ is just very slow(running for 10+ hours on 1 16 GB file) and FS mark has been failing. The bonnie slowness is in re read, here is the best explanation I can find on it:</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">
<span style="font-family:arial"><a href="https://blogs.oracle.com/roch/entry/decoding_bonnie">https://blogs.oracle.com/roch/entry/decoding_bonnie</a></span><br></pre><pre style="word-wrap:break-word"><b style="color:rgb(85,85,85);white-space:normal;font-family:Arial,Verdana,sans-serif;font-size:12px;line-height:18px">Rewriting...done</b><span style="color:rgb(85,85,85);white-space:normal;font-family:Arial,Verdana,sans-serif;font-size:12px;line-height:18px"> </span><br style="color:rgb(85,85,85);font-family:Arial,Verdana,sans-serif;font-size:12px;line-height:18px;white-space:normal">
<br style="color:rgb(85,85,85);font-family:Arial,Verdana,sans-serif;font-size:12px;line-height:18px;white-space:normal"><font color="#555555" face="Arial, Verdana, sans-serif"><span style="line-height:18px;white-space:normal">This gets a little interesting. It actually reads 8K, lseek back to the start of the block, overwrites the 8K with new data and loops. (see article for more.).</span></font><font color="#000000"><span style="white-space:pre-wrap"><br>
</span></font></pre><pre style="word-wrap:break-word">On FS mark I am seeing:</pre><pre style="word-wrap:break-word"><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"># fs_mark -d . -D 4 -t 4 -S 5
# Version 3.3, 4 thread(s) starting at Sat Jul 5 00:54:00 2014
# Sync method: POST: Reopen and fsync() each file in order after main write loop.
# Directories: Time based hash between directories across 4 subdirectories with 180 seconds per subdirectory.
# File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
# Files info: size 51200 bytes, written with an IO size of 16384 bytes per write
# App overhead is time in microseconds spent in the test not doing file writing related system calls.
FSUse% Count Size Files/sec App Overhead
Error in unlink of ./00/53b784e8~~~~~~~~SKZ0QS9BO7O2EG1DIFQLRDYY : No such file or directory
fopen failed to open: fs_log.txt.26676
fs-mark pass # 5 failed</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">I am working on reporting so look for a daily status report email from my jenkins server soon. How do we want to handle failures like this moving forward? Should I just open a BZ after I triage? Do you guys do a new BZ for every failure in the normal regressions tests?</pre>
<pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">-b</pre></pre></div></div>