[2011-01-14 18:25:07.913674] W [io-stats.c:1644:init] vms01: dangling volume. check volfile [2011-01-14 18:25:07.943851] W [dict.c:1205:data_to_str] dict: @data=(nil) [2011-01-14 18:25:07.943889] W [dict.c:1205:data_to_str] dict: @data=(nil) Given volfile: +------------------------------------------------------------------------------+ 1: volume vms01-client-0 2: type protocol/client 3: option remote-host 192.168.56.40 4: option remote-subvolume /mnt/vms01 5: option transport-type tcp 6: end-volume 7: 8: volume vms01-client-1 9: type protocol/client 10: option remote-host 192.168.56.50 11: option remote-subvolume /mnt/vms01 12: option transport-type tcp 13: end-volume 14: 15: volume vms01-replicate-0 16: type cluster/replicate 17: subvolumes vms01-client-0 vms01-client-1 18: end-volume 19: 20: volume vms01-write-behind 21: type performance/write-behind 22: subvolumes vms01-replicate-0 23: end-volume 24: 25: volume vms01-read-ahead 26: type performance/read-ahead 27: subvolumes vms01-write-behind 28: end-volume 29: 30: volume vms01-io-cache 31: type performance/io-cache 32: subvolumes vms01-read-ahead 33: end-volume 34: 35: volume vms01-quick-read 36: type performance/quick-read 37: subvolumes vms01-io-cache 38: end-volume 39: 40: volume vms01-stat-prefetch 41: type performance/stat-prefetch 42: subvolumes vms01-quick-read 43: end-volume 44: 45: volume vms01 46: type debug/io-stats 47: subvolumes vms01-stat-prefetch 48: end-volume +------------------------------------------------------------------------------+ [2011-01-14 18:25:11.911087] I [client-handshake.c:1005:select_server_supported_programs] vms01-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-01-14 18:25:11.911601] I [client-handshake.c:841:client_setvolume_cbk] vms01-client-1: Connected to 192.168.56.50:24009, attached to remote volume '/mnt/vms01'. [2011-01-14 18:25:11.911639] I [afr-common.c:2572:afr_notify] vms01-replicate-0: Subvolume 'vms01-client-1' came back up; going online. [2011-01-14 18:25:11.921497] I [fuse-bridge.c:2821:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 [2011-01-14 18:25:11.922208] I [client-handshake.c:1005:select_server_supported_programs] vms01-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-01-14 18:25:11.922496] I [afr-common.c:819:afr_fresh_lookup_cbk] vms01-replicate-0: added root inode [2011-01-14 18:25:11.923095] I [client-handshake.c:841:client_setvolume_cbk] vms01-client-0: Connected to 192.168.56.40:24009, attached to remote volume '/mnt/vms01'. [2011-01-14 18:25:43.114724] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] vms01-replicate-0: entry self-heal triggered. path: /, reason: checksums of directory differ, forced merge option set [2011-01-14 18:25:43.144426] E [afr-common.c:110:afr_set_split_brain] vms01-replicate-0: invalid argument: inode [2011-01-14 18:25:43.144486] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] vms01-replicate-0: background entry self-heal completed on / [2011-01-14 18:25:52.922594] I [afr-common.c:613:afr_lookup_self_heal_check] vms01-replicate-0: size differs for /gluster-perfs [2011-01-14 18:25:52.922683] I [afr-common.c:716:afr_lookup_done] vms01-replicate-0: background meta-data data self-heal triggered. path: /gluster-perfs [2011-01-14 18:25:52.932465] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] vms01-replicate-0: background meta-data data self-heal completed on /gluster-perfs [2011-01-14 18:50:54.191102] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.875171 [2011-01-14 18:50:54.192364] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.875411 [2011-01-14 18:50:54.193005] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.875710 [2011-01-14 18:50:54.193701] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.876128 [2011-01-14 18:50:54.194342] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.876379 [2011-01-14 18:50:54.194931] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.876638 [2011-01-14 18:50:54.195670] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.876896 [2011-01-14 18:50:54.196219] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:50:53.877140 [2011-01-14 18:50:54.198472] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.877404 [2011-01-14 18:50:54.198578] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.885338 [2011-01-14 18:50:54.198655] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.894455 [2011-01-14 18:50:54.198729] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.906049 [2011-01-14 18:50:54.198801] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.917132 [2011-01-14 18:50:54.198873] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.928372 [2011-01-14 18:50:54.198945] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.939443 [2011-01-14 18:50:54.199017] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.950630 [2011-01-14 18:50:54.199125] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:50:53.961647 [2011-01-14 18:50:54.199191] I [client.c:1590:client_rpc_notify] vms01-client-0: disconnected [2011-01-14 18:50:54.199235] E [socket.c:1661:socket_connect_finish] vms01-client-0: connection to 192.168.56.40:24009 failed (Connection refused) [2011-01-14 18:55:01.926118] W [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 861560: GETXATTR() /vz/private/132/usr/lib/python2.5/bsddb/dbobj.py.dpkg-new => -1 (Success) [2011-01-14 18:55:42.344386] W [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 968665: GETXATTR() /vz/private/132/usr/lib/python2.5/encodings/mac_centeuro.pyc => -1 (Transport endpoint is not connected) [2011-01-14 18:55:46.780125] W [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 975233: GETXATTR() /vz/private/132/usr/lib/python2.5/distutils/archive_util.pyc => -1 (Transport endpoint is not connected) [2011-01-14 18:55:46.823135] W [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 975309: GETXATTR() /vz/private/132/usr/lib/python2.5/email/mime/multipart.pyc => -1 (Transport endpoint is not connected) [2011-01-14 18:55:48.927673] W [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 977830: GETXATTR() /vz/private/132/usr/lib/python2.5/idlelib/tabpage.pyc => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.188402] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174404 [2011-01-14 18:56:11.188794] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993620: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.191089] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174431 [2011-01-14 18:56:11.191447] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993653: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.193104] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174457 [2011-01-14 18:56:11.193463] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993686: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.195205] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174483 [2011-01-14 18:56:11.195518] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993719: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.197237] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174508 [2011-01-14 18:56:11.197396] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993752: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.197905] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.174536 [2011-01-14 18:56:11.198039] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993761: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.198560] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175127 [2011-01-14 18:56:11.198697] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993770: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.199184] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175156 [2011-01-14 18:56:11.199317] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993779: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.199794] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175181 [2011-01-14 18:56:11.199929] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993788: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.201596] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175207 [2011-01-14 18:56:11.201752] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993797: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.202296] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175232 [2011-01-14 18:56:11.202439] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993806: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.202928] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2011-01-14 18:56:11.175255 [2011-01-14 18:56:11.203059] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993815: WRITE => -1 (Transport endpoint is not connected) [2011-01-14 18:56:11.203515] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993817: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.205305] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993850: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.207139] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993883: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.208947] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993916: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.210696] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993949: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.211463] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993958: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.213288] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 993991: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.215100] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994024: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.216882] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994057: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.218585] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994090: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.219136] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994099: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.219643] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994108: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.220151] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994117: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.220677] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994126: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.221204] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994135: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.221723] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994144: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.222209] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994153: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.222615] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994155: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.224367] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994188: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.226137] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994221: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.227890] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994254: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.229718] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994287: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.230557] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994296: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.232425] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994329: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.234312] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994362: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.236139] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994395: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.237785] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994428: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.238337] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994437: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.238897] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994446: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.239417] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994455: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.239954] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994464: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.240467] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994473: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.240980] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994482: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.241451] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994491: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.241819] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994493: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.243768] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994526: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.245646] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994559: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.247490] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994592: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.249134] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994625: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.249897] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994634: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.251750] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994667: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.253510] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994700: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.255366] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994733: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.256985] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994766: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.257531] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994775: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.258079] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994784: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.258630] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994793: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.259211] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994802: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.259821] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994811: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.260411] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994820: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.260947] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994829: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.261368] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994831: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.263257] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994864: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.265079] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994897: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.266948] W [fuse-bridge.c:1761:fuse_writev_cbk] glusterfs-fuse: 994930: WRITE => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.269379] W [fuse-bridge.c:831:fuse_err_cbk] glusterfs-fuse: 994935: FSYNC() ERR => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.273137] W [fuse-bridge.c:353:fuse_truncate_cbk] glusterfs-fuse: 994938: FTRUNCATE() ERR => -1 (Resource temporarily unavailable) [2011-01-14 18:56:11.273544] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2011-01-14 18:56:11.178308 [2011-01-14 18:56:11.273886] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2011-01-14 18:56:11.178948 [2011-01-14 18:56:11.273969] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2011-01-14 18:56:11.180775 [2011-01-14 18:56:11.274061] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2011-01-14 18:56:11.184497 [2011-01-14 18:56:11.274158] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_notify+0x77) [0x7f4756bb9d27] (-->/usr/local//lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7f4756bb94be] (-->/usr/local//lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f4756bb941e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2011-01-14 18:56:11.188186 [2011-01-14 18:56:11.275429] I [client.c:1590:client_rpc_notify] vms01-client-1: disconnected [2011-01-14 18:56:11.275491] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:11.275840] I [client-handshake.c:1005:select_server_supported_programs] vms01-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-01-14 18:56:11.276144] I [client-handshake.c:841:client_setvolume_cbk] vms01-client-1: Connected to 192.168.56.50:24009, attached to remote volume '/mnt/vms01'. [2011-01-14 18:56:11.276194] I [client-handshake.c:710:client_post_handshake] vms01-client-1: 162 fds open - Delaying child_up until they are re-opened [2011-01-14 18:56:11.290335] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/init.log succeeded (remote-fd = 0) [2011-01-14 18:56:11.290443] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/sbin/init-logger succeeded (remote-fd = 1) [2011-01-14 18:56:11.290487] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/sbin/portmap succeeded (remote-fd = 2) [2011-01-14 18:56:11.290558] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libwrap.so.0.7.6 succeeded (remote-fd = 3) [2011-01-14 18:56:11.290606] I [client-handshake.c:403:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/run/portmap.pid failed (File exists) [2011-01-14 18:56:11.290651] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/run/portmap_mapping succeeded (remote-fd = 4) [2011-01-14 18:56:11.290695] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/sbin/rsyslogd succeeded (remote-fd = 5) [2011-01-14 18:56:11.290737] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libz.so.1.2.3.3 succeeded (remote-fd = 6) [2011-01-14 18:56:11.290816] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/rsyslog/lmnet.so succeeded (remote-fd = 7) [2011-01-14 18:56:11.290867] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/rsyslog/lmtcpclt.so succeeded (remote-fd = 8) [2011-01-14 18:56:11.290907] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/rsyslog/imuxsock.so succeeded (remote-fd = 9) [2011-01-14 18:56:11.290946] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/rsyslog/imklog.so succeeded (remote-fd = 10) [2011-01-14 18:56:11.290985] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/syslog succeeded (remote-fd = 11) [2011-01-14 18:56:11.291023] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/daemon.log succeeded (remote-fd = 12) [2011-01-14 18:56:11.291061] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/kern.log succeeded (remote-fd = 13) [2011-01-14 18:56:11.291100] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/lpr.log succeeded (remote-fd = 14) [2011-01-14 18:56:11.291138] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/mail.log succeeded (remote-fd = 15) [2011-01-14 18:56:11.291177] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/user.log succeeded (remote-fd = 16) [2011-01-14 18:56:11.291215] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/mail.info succeeded (remote-fd = 17) [2011-01-14 18:56:11.291253] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/mail.warn succeeded (remote-fd = 18) [2011-01-14 18:56:11.291291] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/mail.err succeeded (remote-fd = 19) [2011-01-14 18:56:11.291330] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/news/news.crit succeeded (remote-fd = 20) [2011-01-14 18:56:11.291372] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/news/news.err succeeded (remote-fd = 21) [2011-01-14 18:56:11.291411] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/news/news.notice succeeded (remote-fd = 22) [2011-01-14 18:56:11.291449] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/debug succeeded (remote-fd = 23) [2011-01-14 18:56:11.291488] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/messages succeeded (remote-fd = 24) [2011-01-14 18:56:11.291527] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/sbin/sshd succeeded (remote-fd = 25) [2011-01-14 18:56:11.291565] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libwrap.so.0.7.6 succeeded (remote-fd = 26) [2011-01-14 18:56:11.291603] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libpam.so.0.81.12 succeeded (remote-fd = 27) [2011-01-14 18:56:11.291643] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libselinux.so.1 succeeded (remote-fd = 28) [2011-01-14 18:56:11.291681] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libz.so.1.2.3.3 succeeded (remote-fd = 29) [2011-01-14 18:56:11.291719] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libcom_err.so.2.1 succeeded (remote-fd = 30) [2011-01-14 18:56:11.291757] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libkeyutils-1.2.so succeeded (remote-fd = 31) [2011-01-14 18:56:11.291822] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/postfix/master succeeded (remote-fd = 32) [2011-01-14 18:56:11.291862] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-global.so.1.0.1 succeeded (remote-fd = 33) [2011-01-14 18:56:11.291901] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-util.so.1.0.1 succeeded (remote-fd = 34) [2011-01-14 18:56:11.291942] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libsasl2.so.2.0.22 succeeded (remote-fd = 35) [2011-01-14 18:56:11.291981] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libdb-4.6.so succeeded (remote-fd = 36) [2011-01-14 18:56:11.292019] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libz.so.1.2.3.3 succeeded (remote-fd = 37) [2011-01-14 18:56:11.292057] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/sbin/atd succeeded (remote-fd = 38) [2011-01-14 18:56:11.292095] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libpam.so.0.81.12 succeeded (remote-fd = 39) [2011-01-14 18:56:11.292137] I [client-handshake.c:403:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/spool/postfix/pid/master.pid failed (File exists) [2011-01-14 18:56:11.292177] I [client-handshake.c:403:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/lib/postfix/master.lock failed (File exists) [2011-01-14 18:56:11.292217] I [client-handshake.c:403:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/run/atd.pid failed (File exists) [2011-01-14 18:56:11.292254] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/sbin/cron succeeded (remote-fd = 40) [2011-01-14 18:56:11.292293] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libpam.so.0.81.12 succeeded (remote-fd = 41) [2011-01-14 18:56:11.292333] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libselinux.so.1 succeeded (remote-fd = 42) [2011-01-14 18:56:11.292372] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/run/crond.pid succeeded (remote-fd = 43) [2011-01-14 18:56:11.292410] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/postfix/pickup succeeded (remote-fd = 44) [2011-01-14 18:56:11.292448] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/postfix/qmgr succeeded (remote-fd = 45) [2011-01-14 18:56:11.292486] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-master.so.1.0.1 succeeded (remote-fd = 46) [2011-01-14 18:56:11.292525] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-master.so.1.0.1 succeeded (remote-fd = 47) [2011-01-14 18:56:11.292565] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-global.so.1.0.1 succeeded (remote-fd = 48) [2011-01-14 18:56:11.292604] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-global.so.1.0.1 succeeded (remote-fd = 49) [2011-01-14 18:56:11.292643] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-util.so.1.0.1 succeeded (remote-fd = 50) [2011-01-14 18:56:11.292682] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libpostfix-util.so.1.0.1 succeeded (remote-fd = 51) [2011-01-14 18:56:11.292742] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libsasl2.so.2.0.22 succeeded (remote-fd = 52) [2011-01-14 18:56:11.292782] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libsasl2.so.2.0.22 succeeded (remote-fd = 53) [2011-01-14 18:56:11.292821] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libdb-4.6.so succeeded (remote-fd = 54) [2011-01-14 18:56:11.292860] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libdb-4.6.so succeeded (remote-fd = 55) [2011-01-14 18:56:11.292897] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libz.so.1.2.3.3 succeeded (remote-fd = 56) [2011-01-14 18:56:11.292937] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libz.so.1.2.3.3 succeeded (remote-fd = 57) [2011-01-14 18:56:11.292975] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/bin/bash succeeded (remote-fd = 58) [2011-01-14 18:56:11.293013] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libncurses.so.5.7 succeeded (remote-fd = 59) [2011-01-14 18:56:11.293051] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libgcc_s.so.1 succeeded (remote-fd = 60) [2011-01-14 18:56:11.293088] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/ld-2.7.so succeeded (remote-fd = 61) [2011-01-14 18:56:11.293125] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/sbin/init succeeded (remote-fd = 62) [2011-01-14 18:56:11.293162] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libsepol.so.1 succeeded (remote-fd = 63) [2011-01-14 18:56:11.293199] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libselinux.so.1 succeeded (remote-fd = 64) [2011-01-14 18:56:11.293237] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libc-2.7.so succeeded (remote-fd = 65) [2011-01-14 18:56:11.293277] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libdl-2.7.so succeeded (remote-fd = 66) [2011-01-14 18:56:11.293315] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/bin/apt-get succeeded (remote-fd = 67) [2011-01-14 18:56:11.293420] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/ld-2.7.so succeeded (remote-fd = 68) [2011-01-14 18:56:11.293555] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libapt-pkg-libc6.7-6.so.4.6.0 succeeded (remote-fd = 69) [2011-01-14 18:56:11.293654] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libutil-2.7.so succeeded (remote-fd = 70) [2011-01-14 18:56:11.293748] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/usr/lib/libstdc++.so.6.0.10 succeeded (remote-fd = 71) [2011-01-14 18:56:11.293845] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libm-2.7.so succeeded (remote-fd = 72) [2011-01-14 18:56:11.293941] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libgcc_s.so.1 succeeded (remote-fd = 73) [2011-01-14 18:56:11.294077] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/lib/libc-2.7.so succeeded (remote-fd = 74) [2011-01-14 18:56:11.294175] I [client-handshake.c:407:client3_1_reopen_cbk] vms01-client-1: reopen on /vz/private/132/var/log/auth.log succeeded (remote-fd = 75) [2011-01-14 18:56:13.514602] W [fuse-bridge.c:409:fuse_attr_cbk] glusterfs-fuse: 994951: STAT() /vz/private/132 => -1 (Transport endpoint is not connected) [2011-01-14 18:56:14.89430] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:14.89522] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:17.90173] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:17.90286] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:19.477074] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 994987: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:56:19.477212] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 994988: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:56:19.477326] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 994990: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:56:19.477456] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 994992: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:56:20.90861] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:20.90977] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:23.91362] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:23.91448] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:26.92061] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:26.92175] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:29.92638] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:29.92725] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:32.93314] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:32.93407] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:35.93888] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:35.94000] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:38.94576] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:38.94689] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:41.95304] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:41.95411] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:43.340613] W [fuse-bridge.c:2213:fuse_statfs_cbk] glusterfs-fuse: 995041: ERR => -1 (Transport endpoint is not connected) [2011-01-14 18:56:44.95915] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:44.96074] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:47.96574] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:47.96678] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:50.97241] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:50.97345] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:53.97875] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:53.97977] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:56.98516] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:56.98627] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:59.99215] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:56:59.99321] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:01.269486] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 995046: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:57:01.269635] W [fuse-bridge.c:1687:fuse_readv_cbk] glusterfs-fuse: 995047: READ => -1 (Transport endpoint is not connected) [2011-01-14 18:57:02.99932] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:02.100044] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:05.100544] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:05.100651] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:08.101176] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:08.101284] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:11.101977] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:11.102101] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:14.102547] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:14.102655] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:17.103213] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:17.103319] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:20.103844] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:20.103985] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:23.104450] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:23.104542] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:26.105168] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:26.105266] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:29.105664] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:29.105747] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:32.106326] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:32.106418] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:35.106909] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:35.107007] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:38.107510] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:38.107604] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:41.108148] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:41.108231] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:44.108811] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:44.108921] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:47.109484] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:47.109587] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:50.110172] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:50.110276] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:53.110717] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:53.110802] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:56.111297] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:56.111386] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:59.112000] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:57:59.112129] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:02.112610] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:02.112705] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:05.113108] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:05.113192] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:08.113681] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:08.113776] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:11.114361] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:11.114450] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:14.114983] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:14.115091] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:17.115631] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:17.115742] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:20.116338] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:20.116434] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:23.116861] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:23.116946] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:26.117432] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:26.117519] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:29.118005] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:29.118111] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:32.118641] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:32.118728] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:35.119303] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:35.119411] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:38.119912] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:38.120024] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:41.120591] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:41.120697] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:44.121271] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:44.121377] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:47.121998] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:47.122245] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:50.122792] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:50.123083] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:53.123613] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:53.123729] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:56.124299] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-01-14 18:58:56.124414] E [afr-common.c:2602:afr_notify] vms01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.