[2017-02-09 09:22:30.977203] I [MSGID: 100030] [glusterfsd.c:2412:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.4 (args: /usr/sbin/glusterfs --volfile-server=10.70.37.196 --volfile-id=/shifu /mnt/repvol) [2017-02-09 09:22:30.988124] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-02-09 09:22:30.997354] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2017-02-09 09:22:30.998376] W [fuse-bridge.c:5283:notify] 0-fuse: got event 12 on graph 0 [2017-02-09 09:22:30.998435] I [MSGID: 114020] [client.c:2356:notify] 0-shifu-client-0: parent translators are ready, attempting connect on transport [2017-02-09 09:22:31.003199] I [MSGID: 114020] [client.c:2356:notify] 0-shifu-client-1: parent translators are ready, attempting connect on transport [2017-02-09 09:22:31.005222] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-shifu-client-0: changing port to 49153 (from 0) [2017-02-09 09:22:31.006832] I [MSGID: 114020] [client.c:2356:notify] 0-shifu-client-2: parent translators are ready, attempting connect on transport [2017-02-09 09:22:31.011192] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-shifu-client-1: changing port to 49153 (from 0) [2017-02-09 09:22:31.011262] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-shifu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) Final graph: +------------------------------------------------------------------------------+ 1: volume shifu-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /shifu 6: option client-version 3.8.4 7: option process-uuid rhs-client15.lab.eng.blr.redhat.com-15139-2017/02/09-09:22:30:970365-shifu-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 30 10: option remote-host dhcp37-169.lab.eng.blr.redhat.com 11: option remote-subvolume /gluster/brick1/b1 12: option transport-type socket 13: option transport.address-family inet 14: option filter-O_DIRECT off 15: option send-gids true 16: end-volume 17: 18: volume shifu-client-1 19: type protocol/client 20: option ping-timeout 30 21: option remote-host dhcp37-188.lab.eng.blr.redhat.com 22: option remote-subvolume /gluster/brick1/b1 23: option transport-type socket 24: option transport.address-family inet 25: option filter-O_DIRECT off 26: option send-gids true 27: end-volume 28: 29: volume shifu-client-2 30: type protocol/client 31: option ping-timeout 30 32: option remote-host dhcp37-196.lab.eng.blr.redhat.com 33: option remote-subvolume /gluster/brick1/b1 34: option transport-type socket 35: option transport.address-family inet 36: option filter-O_DIRECT off 37: option send-gids true 38: end-volume 39: 40: volume shifu-replicate-0 41: type cluster/replicate 42: option data-self-heal-algorithm full 43: option eager-lock enable 44: option quorum-type auto 45: option shd-max-threads 8 46: option shd-wait-qlength 10000 47: option locking-scheme granular 48: option use-compound-fops on 49: subvolumes shifu-client-0 shifu-client-1 shifu-client-2 50: end-volume 51: 52: volume shifu-dht 53: type cluster/distribute 54: option lock-migration off 55: subvolumes shifu-replicate-0 56: end-volume 57: 58: volume shifu-shard 59: type features/shard 60: option shard-block-size 4MB 61: subvolumes shifu-dht 62: end-volume 63: 64: volume shifu-write-behind 65: type performance/write-behind 66: option strict-O_DIRECT on 67: subvolumes shifu-shard 68: end-volume 69: 70: volume shifu-readdir-ahead 71: type performance/readdir-ahead 72: subvolumes shifu-write-behind 73: end-volume 74: 75: volume shifu-open-behind 76: type performance/open-behind 77: subvolumes shifu-readdir-ahead 78: end-volume 79: 80: volume shifu-io-threads 81: type performance/io-threads 82: option low-prio-threads 32 83: subvolumes shifu-open-behind 84: end-volume 85: 86: volume shifu 87: type debug/io-stats 88: option log-level INFO 89: option latency-measurement off 90: option count-fop-hits off 91: subvolumes shifu-io-threads 92: end-volume 93: 94: volume meta-autoload 95: type meta 96: subvolumes shifu 97: end-volume 98: +------------------------------------------------------------------------------+ [2017-02-09 09:22:31.013844] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-shifu-client-0: Connected to shifu-client-0, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:22:31.013868] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-shifu-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:22:31.013932] I [MSGID: 108005] [afr-common.c:4656:afr_notify] 0-shifu-replicate-0: Subvolume 'shifu-client-0' came back up; going online. [2017-02-09 09:22:31.014558] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-shifu-client-0: Server lk version = 1 [2017-02-09 09:22:31.014707] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-shifu-client-2: changing port to 49153 (from 0) [2017-02-09 09:22:31.016932] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-shifu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:22:31.019221] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-shifu-client-1: Connected to shifu-client-1, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:22:31.019246] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-shifu-client-1: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:22:31.019349] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-shifu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:22:31.019885] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-shifu-client-1: Server lk version = 1 [2017-02-09 09:22:31.021503] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-shifu-client-2: Connected to shifu-client-2, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:22:31.021533] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-shifu-client-2: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:22:31.021661] W [fuse-bridge.c:5283:notify] 0-fuse: got event 5 on graph 0 [2017-02-09 09:22:31.025947] W [fuse-bridge.c:5302:notify] 0-fuse: set the graph 0 [2017-02-09 09:22:31.025972] W [fuse-bridge.c:5315:notify] 0-fuse: received event (5)(CHILD_UP) id=0, setting event_recvd to true [2017-02-09 09:22:31.026054] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-shifu-client-2: Server lk version = 1 [2017-02-09 09:22:31.026613] I [fuse-bridge.c:4154:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2017-02-09 09:22:31.030887] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-shifu-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 [2017-02-09 09:22:31.036986] I [fuse-bridge.c:4847:fuse_graph_sync] 0-meta-autoload: switched to new-graph: 0 old-graph: -1. new_subvol=0x7fcd04015830 old_subvol=(nil)old_subvol->winds = 18446744073709551615 unique = 2 [2017-02-09 09:22:55.091063] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-shifu-dht: Found anomalies in /.trashcan (gfid = 00000000-0000-0000-0000-000000000005). Holes=1 overlaps=0 [2017-02-09 09:24:56.094790] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 0-mgmt: Volume file changed [2017-02-09 09:24:56.182176] W [fuse-bridge.c:5283:notify] 0-fuse: got event 12 on graph 2 [2017-02-09 09:24:56.182225] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-0: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.188116] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-1: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.191915] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-2: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.196127] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-3: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.198538] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-2: changing port to 49153 (from 0) [2017-02-09 09:24:56.199945] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-4: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.204618] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-3: changing port to 49152 (from 0) [2017-02-09 09:24:56.206364] I [MSGID: 114020] [client.c:2356:notify] 2-shifu-client-5: parent translators are ready, attempting connect on transport [2017-02-09 09:24:56.209389] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:24:56.210260] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) Final graph: +------------------------------------------------------------------------------+ 1: volume shifu-client-0 2: type protocol/client 3: option ping-timeout 30 4: option remote-host dhcp37-169.lab.eng.blr.redhat.com 5: option remote-subvolume /gluster/brick1/b1 6: option transport-type socket 7: option transport.address-family inet 8: option filter-O_DIRECT off 9: option send-gids true 10: end-volume 11: 12: volume shifu-client-1 13: type protocol/client 14: option ping-timeout 30 15: option remote-host dhcp37-188.lab.eng.blr.redhat.com 16: option remote-subvolume /gluster/brick1/b1 17: option transport-type socket 18: option transport.address-family inet 19: option filter-O_DIRECT off 20: option send-gids true 21: end-volume 22: 23: volume shifu-client-2 24: type protocol/client 25: option clnt-lk-version 1 26: option volfile-checksum 0 27: option volfile-key /shifu 28: option client-version 3.8.4 29: option process-uuid rhs-client15.lab.eng.blr.redhat.com-15139-2017/02/09-09:22:30:970365-shifu-client-2-2-0 30: option fops-version 1298437 31: option ping-timeout 30 32: option remote-host dhcp37-196.lab.eng.blr.redhat.com 33: option remote-subvolume /gluster/brick1/b1 34: option transport-type socket 35: option transport.address-family inet 36: option filter-O_DIRECT off 37: option send-gids true 38: end-volume 39: 40: volume shifu-replicate-0 41: type cluster/replicate 42: option data-self-heal-algorithm full 43: option eager-lock enable 44: option quorum-type auto 45: option shd-max-threads 8 46: option shd-wait-qlength 10000 47: option locking-scheme granular 48: option use-compound-fops on 49: subvolumes shifu-client-0 shifu-client-1 shifu-client-2 50: end-volume 51: 52: volume shifu-client-3 53: type protocol/client 54: option clnt-lk-version 1 55: option volfile-checksum 0 56: option volfile-key /shifu 57: option client-version 3.8.4 58: option process-uuid rhs-client15.lab.eng.blr.redhat.com-15139-2017/02/09-09:22:30:970365-shifu-client-3-2-0 59: option fops-version 1298437 60: option ping-timeout 30 61: option remote-host dhcp37-196.lab.eng.blr.redhat.com 62: option remote-subvolume /gluster/brick2/newb 63: option transport-type socket 64: option transport.address-family inet 65: option filter-O_DIRECT off 66: option send-gids true 67: end-volume 68: 69: volume shifu-client-4 70: type protocol/client 71: option ping-timeout 30 72: option remote-host dhcp37-188.lab.eng.blr.redhat.com 73: option remote-subvolume /gluster/brick2/newb 74: option transport-type socket 75: option transport.address-family inet 76: option filter-O_DIRECT off 77: option send-gids true 78: end-volume 79: 80: volume shifu-client-5 81: type protocol/client 82: option ping-timeout 30 83: option remote-host dhcp37-169.lab.eng.blr.redhat.com 84: option remote-subvolume /gluster/brick2/newb 85: option transport-type socket 86: option transport.address-family inet 87: option filter-O_DIRECT off 88: option send-gids true 89: end-volume 90: 91: volume shifu-replicate-1 92: type cluster/replicate 93: option data-self-heal-algorithm full 94: option eager-lock enable 95: option quorum-type auto 96: option shd-max-threads 8 97: option shd-wait-qlength 10000 98: option locking-scheme granular 99: option use-compound-fops on 100: subvolumes shifu-client-3 shifu-client-4 shifu-client-5 101: end-volume 102: 103: volume shifu-dht 104: type cluster/distribute 105: option lock-migration off 106: subvolumes shifu-replicate-0 shifu-replicate-1 107: end-volume 108: 109: volume shifu-shard 110: type features/shard 111: option shard-block-size 4MB 112: subvolumes shifu-dht 113: end-volume 114: 115: volume shifu-write-behind 116: type performance/write-behind 117: option strict-O_DIRECT on 118: subvolumes shifu-shard 119: end-volume 120: 121: volume shifu-readdir-ahead 122: type performance/readdir-ahead 123: subvolumes shifu-write-behind 124: end-volume 125: 126: volume shifu-open-behind 127: type performance/open-behind 128: subvolumes shifu-readdir-ahead 129: end-volume 130: 131: volume shifu-io-threads 132: type performance/io-threads 133: option low-prio-threads 32 134: subvolumes shifu-open-behind 135: end-volume 136: 137: volume shifu 138: type debug/io-stats 139: option log-level INFO 140: option latency-measurement off 141: option count-fop-hits off 142: subvolumes shifu-io-threads 143: end-volume 144: 145: volume meta-autoload 146: type meta 147: subvolumes shifu 148: end-volume 149: +------------------------------------------------------------------------------+ [2017-02-09 09:24:56.213062] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-3: Connected to shifu-client-3, attached to remote volume '/gluster/brick2/newb'. [2017-02-09 09:24:56.213081] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-3: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.213131] I [MSGID: 108005] [afr-common.c:4656:afr_notify] 2-shifu-replicate-1: Subvolume 'shifu-client-3' came back up; going online. [2017-02-09 09:24:56.214059] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-3: Server lk version = 1 [2017-02-09 09:24:56.217711] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-2: Connected to shifu-client-2, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:24:56.217733] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-2: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.217789] I [MSGID: 108005] [afr-common.c:4656:afr_notify] 2-shifu-replicate-0: Subvolume 'shifu-client-2' came back up; going online. [2017-02-09 09:24:56.218083] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-2: Server lk version = 1 [2017-02-09 09:24:56.262388] W [MSGID: 114010] [client-callback.c:28:client_cbk_fetchspec] 2-shifu-client-1: this function should not be called [2017-02-09 09:24:56.315598] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-1: changing port to 49153 (from 0) [2017-02-09 09:24:56.316747] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-4: changing port to 49152 (from 0) [2017-02-09 09:24:56.320456] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:24:56.322262] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-1: Connected to shifu-client-1, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:24:56.322283] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-1: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.322704] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-1: Server lk version = 1 [2017-02-09 09:24:56.324118] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:24:56.325516] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-4: Connected to shifu-client-4, attached to remote volume '/gluster/brick2/newb'. [2017-02-09 09:24:56.325533] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-4: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.326017] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-4: Server lk version = 1 [2017-02-09 09:24:56.346573] W [MSGID: 114010] [client-callback.c:28:client_cbk_fetchspec] 2-shifu-client-0: this function should not be called [2017-02-09 09:24:56.404136] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-0: changing port to 49153 (from 0) [2017-02-09 09:24:56.406872] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 2-shifu-client-5: changing port to 49152 (from 0) [2017-02-09 09:24:56.409808] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:24:56.412648] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 2-shifu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-02-09 09:24:56.412985] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-0: Connected to shifu-client-0, attached to remote volume '/gluster/brick1/b1'. [2017-02-09 09:24:56.413008] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.414160] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-0: Server lk version = 1 [2017-02-09 09:24:56.415327] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 2-shifu-client-5: Connected to shifu-client-5, attached to remote volume '/gluster/brick2/newb'. [2017-02-09 09:24:56.415345] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 2-shifu-client-5: Server and Client lk-version numbers are not same, reopening the fds [2017-02-09 09:24:56.415453] W [fuse-bridge.c:5283:notify] 0-fuse: got event 5 on graph 2 [2017-02-09 09:24:56.419907] W [fuse-bridge.c:5302:notify] 0-fuse: set the graph 2 [2017-02-09 09:24:56.419941] W [fuse-bridge.c:5315:notify] 0-fuse: received event (5)(CHILD_UP) id=2, setting event_recvd to true [2017-02-09 09:24:56.420039] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-shifu-client-5: Server lk version = 1 [2017-02-09 09:24:56.425688] I [fuse-bridge.c:4847:fuse_graph_sync] 0-meta-autoload: switched to new-graph: 2 old-graph: 0. new_subvol=0x7fccf40715d0 old_subvol=0x7fcd04015830old_subvol->winds = 2 unique = 53585 [2017-02-09 09:24:56.490751] I [fuse-bridge.c:4447:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x7fcd00ec206c) to newfd (0x7fccf28a106c) (inode-gfid:52b6db23-22f2-4c03-bb7d-2d91856dcb64)(old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2) [2017-02-09 09:24:56.511642] I [fuse-bridge.c:4447:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x7fcd00ec20f8) to newfd (0x7fccf28a10f8) (inode-gfid:d31b7d12-befa-404d-b51b-bfeb35060a27)(old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2) [2017-02-09 09:24:56.514891] I [fuse-bridge.c:4447:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x7fcd00ec2184) to newfd (0x7fccf28a1184) (inode-gfid:80152ddb-11a2-41e7-925a-7a0bf5097e1a)(old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2) [2017-02-09 09:24:56.518223] I [fuse-bridge.c:4447:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x7fcd00ec229c) to newfd (0x7fccf28a1210) (inode-gfid:c2d30771-8e17-4a13-ac50-f6b3cad00832)(old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2) [2017-02-09 09:24:56.521521] I [fuse-bridge.c:4447:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x7fcd00ec2210) to newfd (0x7fccf28a129c) (inode-gfid:69b86752-ce71-44ca-83c9-abe11d3cd72d)(old-subvolume:meta-autoload-0 new-subvolume:meta-autoload-2) [2017-02-09 09:24:56.521640] I [fuse-bridge.c:4864:fuse_graph_sync] 0-fuse: sending PARENT_DOWN on graph:0 subvol: 0x7fcd04015830 unique = 53585 [2017-02-09 09:24:56.521790] I [MSGID: 114021] [client.c:2365:notify] 0-shifu-client-0: current graph is no longer active, destroying rpc_client [2017-02-09 09:24:56.521857] I [MSGID: 114021] [client.c:2365:notify] 0-shifu-client-1: current graph is no longer active, destroying rpc_client [2017-02-09 09:24:56.521900] I [MSGID: 114021] [client.c:2365:notify] 0-shifu-client-2: current graph is no longer active, destroying rpc_client [2017-02-09 09:24:56.521905] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-shifu-client-0: disconnected from shifu-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2017-02-09 09:24:56.521962] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-shifu-client-1: disconnected from shifu-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2017-02-09 09:24:56.521975] W [MSGID: 108001] [afr-common.c:4770:afr_notify] 0-shifu-replicate-0: Client-quorum is not met [2017-02-09 09:24:56.521995] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-shifu-client-2: disconnected from shifu-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2017-02-09 09:24:56.522210] E [MSGID: 108006] [afr-common.c:4681:afr_notify] 0-shifu-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2017-02-09 09:24:56.522288] W [fuse-bridge.c:5283:notify] 0-fuse: got event 6 on graph 0 [2017-02-09 09:24:56.522300] W [fuse-bridge.c:5302:notify] 0-fuse: set the graph 0 [2017-02-09 09:24:56.522305] W [fuse-bridge.c:5315:notify] 0-fuse: received event (6)(CHILD_DOWN) id=0, setting event_recvd to true [2017-02-09 09:24:56.593681] W [fuse-bridge.c:2229:fuse_readv_cbk] 0-glusterfs-fuse: 53616: READ => -1 gfid=52b6db23-22f2-4c03-bb7d-2d91856dcb64 fd=0x7fccf28a106c (Input/output error) [2017-02-09 09:24:56.593761] W [fuse-bridge.c:2313:fuse_writev_cbk] 0-glusterfs-fuse: 53620: WRITE => -1 gfid=52b6db23-22f2-4c03-bb7d-2d91856dcb64 fd=0x7fccf28a106c (Input/output error) subvol=0x7fccf40715d0 [2017-02-09 09:24:56.594019] W [fuse-bridge.c:2229:fuse_readv_cbk] 0-glusterfs-fuse: 53622: READ => -1 gfid=52b6db23-22f2-4c03-bb7d-2d91856dcb64 fd=0x7fccf28a106c (Input/output error) [2017-02-09 09:24:56.594310] W [fuse-bridge.c:2313:fuse_writev_cbk] 0-glusterfs-fuse: 53624: WRITE => -1 gfid=52b6db23-22f2-4c03-bb7d-2d91856dcb64 fd=0x7fccf28a106c (Input/output error) subvol=0x7fccf40715d0 [2017-02-09 09:24:56.594870] W [fuse-bridge.c:2313:fuse_writev_cbk] 0-glusterfs-fuse: 53626: WRITE => -1 gfid=52b6db23-22f2-4c03-bb7d-2d91856dcb64 fd=0x7fccf28a106c (Input/output error) subvol=0x7fccf40715d0 [2017-02-09 09:24:57.049925] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.049989] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050008] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050025] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050065] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050152] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050205] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050224] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050240] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050274] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050291] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050306] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050355] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050373] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050402] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050416] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050429] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050442] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050477] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050495] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050511] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050525] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050538] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050550] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050589] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050606] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050622] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050637] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050653] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050668] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050733] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050753] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050769] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050783] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050796] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050812] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050851] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050870] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050886] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.050900] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050915] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050930] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.050965] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.050983] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.050998] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051013] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051027] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051042] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051078] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.051095] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.051111] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051125] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051140] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051155] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051190] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.051207] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.051230] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051245] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051261] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051275] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051311] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.051329] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.051344] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051358] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051374] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051389] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051425] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.051442] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.051458] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051472] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051487] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051501] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051536] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:57.051553] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:57.051569] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:57.051583] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051597] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:57.051612] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.051754] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.051822] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.051841] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.051856] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.051896] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.051940] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052014] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052039] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052056] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052073] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052088] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052101] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052140] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052159] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052174] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052187] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052202] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052218] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052254] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052271] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052286] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052299] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052315] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052330] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052367] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052384] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052400] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052415] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052432] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052446] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052494] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052512] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052527] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052541] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052556] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052571] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052632] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052659] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052687] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052710] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052726] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052741] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052781] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052798] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052814] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052828] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052843] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052858] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052897] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.052914] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.052930] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.052944] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052958] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.052973] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053008] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.053025] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.053041] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.053063] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053078] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053093] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053129] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-0: remote operation failed [2017-02-09 09:24:58.053146] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-1: remote operation failed [2017-02-09 09:24:58.053162] W [MSGID: 114031] [client-rpc-fops.c:1830:client3_3_fxattrop_cbk] 0-shifu-client-2: remote operation failed [2017-02-09 09:24:58.053176] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-0: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053191] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-1: remote operation failed [Transport endpoint is not connected] [2017-02-09 09:24:58.053205] E [MSGID: 114031] [client-rpc-fops.c:1601:client3_3_finodelk_cbk] 0-shifu-client-2: remote operation failed [Transport endpoint is not connected]