[2019-02-14 05:24:30.938112] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-glusterfs: Started running glusterfs version 4.1.5 (args: glusterfs -s --volfile-id patchy /mnt/glusterfs/0 --entry-timeout=0 --attribute-timeout=0) [2019-02-14 05:24:30.945943] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-02-14 05:24:30.949398] D [MSGID: 0] [io-threads.c:845:__iot_workers_scale] 0-patchy-io-threads: scaled threads to 1 (queue_size=0/1) [2019-02-14 05:24:30.949510] D [MSGID: 0] [quick-read.c:923:check_cache_size_ok] 0-patchy-quick-read: Max cache size is 8257273856 [2019-02-14 05:24:30.949547] D [MSGID: 0] [io-cache.c:1638:check_cache_size_ok] 0-patchy-io-cache: Max cache size is 8257273856 [2019-02-14 05:24:30.949571] D [MSGID: 0] [options.c:1259:xlator_option_init_size_uint64] 0-patchy-readdir-ahead: option rda-request-size using set value 131072 [2019-02-14 05:24:30.949586] D [MSGID: 0] [options.c:1259:xlator_option_init_size_uint64] 0-patchy-readdir-ahead: option rda-cache-limit using set value 10MB [2019-02-14 05:24:30.949598] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-readdir-ahead: option parallel-readdir using set value off [2019-02-14 05:24:30.949719] D [MSGID: 0] [options.c:1256:xlator_option_init_uint32] 0-patchy-dht: option directory-layout-spread using set value 3 [2019-02-14 05:24:30.949745] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-dht: option lock-migration using set value off [2019-02-14 05:24:30.949762] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-dht: option force-migration using set value off [2019-02-14 05:24:30.949807] D [MSGID: 0] [dht-shared.c:381:dht_init_regex] 0-patchy-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2019-02-14 05:24:30.949912] D [MSGID: 0] [options.c:1257:xlator_option_init_int32] 0-patchy-client-3: option ping-timeout using set value 42 [2019-02-14 05:24:30.949923] D [MSGID: 0] [options.c:1264:xlator_option_init_path] 0-patchy-client-3: option remote-subvolume using set value /d/backends/patchy3 [2019-02-14 05:24:30.949937] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-client-3: option send-gids using set value true [2019-02-14 05:24:30.949948] D [rpc-clnt.c:1062:rpc_clnt_connection_init] 0-patchy-client-3: defaulting frame-timeout to 30mins [2019-02-14 05:24:30.949956] D [rpc-clnt.c:1071:rpc_clnt_connection_init] 0-patchy-client-3: setting ping-timeout to 42 [2019-02-14 05:24:30.949977] D [rpc-transport.c:280:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/local/lib/glusterfs/4.1.5/rpc-transport/socket.so [2019-02-14 05:24:30.950154] D [socket.c:4375:socket_init] 0-patchy-client-3: Configued transport.tcp-user-timeout=0 [2019-02-14 05:24:30.950163] D [socket.c:4393:socket_init] 0-patchy-client-3: Reconfigued transport.keepalivecnt=9 [2019-02-14 05:24:30.950169] D [socket.c:4477:socket_init] 0-patchy-client-3: SSL support on the I/O path is NOT enabled [2019-02-14 05:24:30.950174] D [socket.c:4480:socket_init] 0-patchy-client-3: SSL support for glusterd is NOT enabled [2019-02-14 05:24:30.950179] D [socket.c:4497:socket_init] 0-patchy-client-3: using system polling thread [2019-02-14 05:24:30.950186] D [rpc-clnt.c:1645:rpcclnt_cbk_program_register] 0-patchy-client-3: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-02-14 05:24:30.950192] D [MSGID: 0] [client.c:2526:client_init_rpc] 0-patchy-client-3: client init successful [2019-02-14 05:24:30.950225] D [MSGID: 0] [options.c:1257:xlator_option_init_int32] 0-patchy-client-2: option ping-timeout using set value 42 [2019-02-14 05:24:30.950235] D [MSGID: 0] [options.c:1264:xlator_option_init_path] 0-patchy-client-2: option remote-subvolume using set value /d/backends/patchy2 [2019-02-14 05:24:30.950249] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-client-2: option send-gids using set value true [2019-02-14 05:24:30.950260] D [rpc-clnt.c:1062:rpc_clnt_connection_init] 0-patchy-client-2: defaulting frame-timeout to 30mins [2019-02-14 05:24:30.950265] D [rpc-clnt.c:1071:rpc_clnt_connection_init] 0-patchy-client-2: setting ping-timeout to 42 [2019-02-14 05:24:30.950280] D [rpc-transport.c:280:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/local/lib/glusterfs/4.1.5/rpc-transport/socket.so [2019-02-14 05:24:30.950415] D [socket.c:4375:socket_init] 0-patchy-client-2: Configued transport.tcp-user-timeout=0 [2019-02-14 05:24:30.950421] D [socket.c:4393:socket_init] 0-patchy-client-2: Reconfigued transport.keepalivecnt=9 [2019-02-14 05:24:30.950428] D [socket.c:4477:socket_init] 0-patchy-client-2: SSL support on the I/O path is NOT enabled [2019-02-14 05:24:30.950432] D [socket.c:4480:socket_init] 0-patchy-client-2: SSL support for glusterd is NOT enabled [2019-02-14 05:24:30.950437] D [socket.c:4497:socket_init] 0-patchy-client-2: using system polling thread [2019-02-14 05:24:30.950443] D [rpc-clnt.c:1645:rpcclnt_cbk_program_register] 0-patchy-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-02-14 05:24:30.950451] D [MSGID: 0] [client.c:2526:client_init_rpc] 0-patchy-client-2: client init successful [2019-02-14 05:24:30.950484] D [MSGID: 0] [options.c:1257:xlator_option_init_int32] 0-patchy-client-1: option ping-timeout using set value 42 [2019-02-14 05:24:30.950493] D [MSGID: 0] [options.c:1264:xlator_option_init_path] 0-patchy-client-1: option remote-subvolume using set value /d/backends/patchy1 [2019-02-14 05:24:30.950506] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-client-1: option send-gids using set value true [2019-02-14 05:24:30.950513] D [rpc-clnt.c:1062:rpc_clnt_connection_init] 0-patchy-client-1: defaulting frame-timeout to 30mins [2019-02-14 05:24:30.950519] D [rpc-clnt.c:1071:rpc_clnt_connection_init] 0-patchy-client-1: setting ping-timeout to 42 [2019-02-14 05:24:30.950529] D [rpc-transport.c:280:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/local/lib/glusterfs/4.1.5/rpc-transport/socket.so [2019-02-14 05:24:30.950682] D [socket.c:4375:socket_init] 0-patchy-client-1: Configued transport.tcp-user-timeout=0 [2019-02-14 05:24:30.950692] D [socket.c:4393:socket_init] 0-patchy-client-1: Reconfigued transport.keepalivecnt=9 [2019-02-14 05:24:30.950700] D [socket.c:4477:socket_init] 0-patchy-client-1: SSL support on the I/O path is NOT enabled [2019-02-14 05:24:30.950706] D [socket.c:4480:socket_init] 0-patchy-client-1: SSL support for glusterd is NOT enabled [2019-02-14 05:24:30.950713] D [socket.c:4497:socket_init] 0-patchy-client-1: using system polling thread [2019-02-14 05:24:30.950721] D [rpc-clnt.c:1645:rpcclnt_cbk_program_register] 0-patchy-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-02-14 05:24:30.950730] D [MSGID: 0] [client.c:2526:client_init_rpc] 0-patchy-client-1: client init successful [2019-02-14 05:24:30.950764] D [MSGID: 0] [options.c:1257:xlator_option_init_int32] 0-patchy-client-0: option ping-timeout using set value 42 [2019-02-14 05:24:30.950775] D [MSGID: 0] [options.c:1264:xlator_option_init_path] 0-patchy-client-0: option remote-subvolume using set value /d/backends/patchy0 [2019-02-14 05:24:30.950792] D [MSGID: 0] [options.c:1262:xlator_option_init_bool] 0-patchy-client-0: option send-gids using set value true [2019-02-14 05:24:30.950800] D [rpc-clnt.c:1062:rpc_clnt_connection_init] 0-patchy-client-0: defaulting frame-timeout to 30mins [2019-02-14 05:24:30.950807] D [rpc-clnt.c:1071:rpc_clnt_connection_init] 0-patchy-client-0: setting ping-timeout to 42 [2019-02-14 05:24:30.950818] D [rpc-transport.c:280:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/local/lib/glusterfs/4.1.5/rpc-transport/socket.so [2019-02-14 05:24:30.950996] D [socket.c:4375:socket_init] 0-patchy-client-0: Configued transport.tcp-user-timeout=0 [2019-02-14 05:24:30.951003] D [socket.c:4393:socket_init] 0-patchy-client-0: Reconfigued transport.keepalivecnt=9 [2019-02-14 05:24:30.951011] D [socket.c:4477:socket_init] 0-patchy-client-0: SSL support on the I/O path is NOT enabled [2019-02-14 05:24:30.951017] D [socket.c:4480:socket_init] 0-patchy-client-0: SSL support for glusterd is NOT enabled [2019-02-14 05:24:30.951030] D [socket.c:4497:socket_init] 0-patchy-client-0: using system polling thread [2019-02-14 05:24:30.951037] D [rpc-clnt.c:1645:rpcclnt_cbk_program_register] 0-patchy-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-02-14 05:24:30.951044] D [MSGID: 0] [client.c:2526:client_init_rpc] 0-patchy-client-0: client init successful [2019-02-14 05:24:30.951113] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-3: option 'transport.address-family' is not recognized [2019-02-14 05:24:30.951131] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-3: option 'transport.tcp-user-timeout' is not recognized [2019-02-14 05:24:30.951145] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-3: option 'transport.socket.keepalive-time' is not recognized [2019-02-14 05:24:30.951158] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-3: option 'transport.socket.keepalive-interval' is not recognized [2019-02-14 05:24:30.951172] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-3: option 'transport.socket.keepalive-count' is not recognized [2019-02-14 05:24:30.951197] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-2: option 'transport.address-family' is not recognized [2019-02-14 05:24:30.951214] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-2: option 'transport.tcp-user-timeout' is not recognized [2019-02-14 05:24:30.951228] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-2: option 'transport.socket.keepalive-time' is not recognized [2019-02-14 05:24:30.951248] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-2: option 'transport.socket.keepalive-interval' is not recognized [2019-02-14 05:24:30.951261] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-2: option 'transport.socket.keepalive-count' is not recognized [2019-02-14 05:24:30.951286] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-1: option 'transport.address-family' is not recognized [2019-02-14 05:24:30.951300] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-1: option 'transport.tcp-user-timeout' is not recognized [2019-02-14 05:24:30.951321] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-1: option 'transport.socket.keepalive-time' is not recognized [2019-02-14 05:24:30.951335] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-1: option 'transport.socket.keepalive-interval' is not recognized [2019-02-14 05:24:30.951348] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-1: option 'transport.socket.keepalive-count' is not recognized [2019-02-14 05:24:30.951373] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-0: option 'transport.address-family' is not recognized [2019-02-14 05:24:30.951387] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-0: option 'transport.tcp-user-timeout' is not recognized [2019-02-14 05:24:30.951400] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-0: option 'transport.socket.keepalive-time' is not recognized [2019-02-14 05:24:30.951411] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-0: option 'transport.socket.keepalive-interval' is not recognized [2019-02-14 05:24:30.951422] D [MSGID: 101174] [graph.c:403:_log_if_unknown_option] 0-patchy-client-0: option 'transport.socket.keepalive-count' is not recognized [2019-02-14 05:24:30.951442] D [fuse-bridge.c:5386:notify] 0-fuse: got event 12 on graph 0 [2019-02-14 05:24:30.951469] I [MSGID: 114020] [client.c:2328:notify] 0-patchy-client-0: parent translators are ready, attempting connect on transport [2019-02-14 05:24:30.951794] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip- (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.951810] D [MSGID: 0] [common-utils.c:360:gf_resolve_ip6] 0-resolver: next DNS query will return: ip-10.20.81.111 port-24007 [2019-02-14 05:24:30.951828] D [socket.c:3208:socket_fix_ssl_opts] 0-patchy-client-0: disabling SSL for portmapper connection [2019-02-14 05:24:30.951911] I [MSGID: 114020] [client.c:2328:notify] 0-patchy-client-1: parent translators are ready, attempting connect on transport [2019-02-14 05:24:30.951979] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2019-02-14 05:24:30.952005] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-0: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.952042] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.952230] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip- (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.952247] D [MSGID: 0] [common-utils.c:360:gf_resolve_ip6] 0-resolver: next DNS query will return: ip-10.20.81.111 port-24007 [2019-02-14 05:24:30.952257] D [socket.c:3208:socket_fix_ssl_opts] 0-patchy-client-1: disabling SSL for portmapper connection [2019-02-14 05:24:30.952335] I [MSGID: 114020] [client.c:2328:notify] 0-patchy-client-2: parent translators are ready, attempting connect on transport [2019-02-14 05:24:30.952444] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.952457] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.952471] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-1: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.952481] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-1: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.952605] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip- (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.952617] D [MSGID: 0] [common-utils.c:360:gf_resolve_ip6] 0-resolver: next DNS query will return: ip-10.20.81.111 port-24007 [2019-02-14 05:24:30.952625] D [socket.c:3208:socket_fix_ssl_opts] 0-patchy-client-2: disabling SSL for portmapper connection [2019-02-14 05:24:30.952622] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.952646] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.952669] D [MSGID: 0] [client-handshake.c:1470:server_has_portmap] 0-patchy-client-0: detected portmapper on server [2019-02-14 05:24:30.952682] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.952700] I [MSGID: 114020] [client.c:2328:notify] 0-patchy-client-3: parent translators are ready, attempting connect on transport [2019-02-14 05:24:30.952801] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.952817] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.952836] D [MSGID: 0] [client-handshake.c:1470:server_has_portmap] 0-patchy-client-1: detected portmapper on server [2019-02-14 05:24:30.952847] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-1: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.952941] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.952952] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.952961] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip- (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.952967] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.952981] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.952984] D [MSGID: 0] [common-utils.c:360:gf_resolve_ip6] 0-resolver: next DNS query will return: ip-10.20.81.111 port-24007 [2019-02-14 05:24:30.952992] D [socket.c:3208:socket_fix_ssl_opts] 0-patchy-client-3: disabling SSL for portmapper connection Final graph: +------------------------------------------------------------------------------+ 1: volume patchy-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 5: option remote-subvolume /d/backends/patchy0 6: option transport-type socket 7: option transport.address-family inet [2019-02-14 05:24:30.953090] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed 8: option username 81e6517d-ac80-4d0f-9180-ca2b5a63ab8a 9: option password 3eef276b-b00d-4246-985e-00d0b685a69d 10: option transport.tcp-user-timeout 0 11: option transport.socket.keepalive-time 20 12: option transport.socket.keepalive-interval 2 13: option transport.socket.keepalive-count 9 14: option send-gids true 15: end-volume 16: 17: volume patchy-client-1 18: type protocol/client 19: option ping-timeout 42 20: option remote-host 21: option remote-subvolume /d/backends/patchy1 22: option transport-type socket 23: option transport.address-family inet 24: option username 81e6517d-ac80-4d0f-9180-ca2b5a63ab8a 25: option password 3eef276b-b00d-4246-985e-00d0b685a69d 26: option transport.tcp-user-timeout 0 27: option transport.socket.keepalive-time 20 28: option transport.socket.keepalive-interval 2 29: option transport.socket.keepalive-count 9 30: option send-gids true 31: end-volume 32: 33: volume patchy-client-2 34: type protocol/client 35: option ping-timeout 42 36: option remote-host 37: option remote-subvolume /d/backends/patchy2 [2019-02-14 05:24:30.953103] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) 38: option transport-type socket 39: option transport.address-family inet [2019-02-14 05:24:30.953245] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-3: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.953270] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-3: error returned while attempting to connect to host:(null), port:0 40: option username 81e6517d-ac80-4d0f-9180-ca2b5a63ab8a 41: option password 3eef276b-b00d-4246-985e-00d0b685a69d 42: option transport.tcp-user-timeout 0 43: option transport.socket.keepalive-time 20 44: option transport.socket.keepalive-interval 2 45: option transport.socket.keepalive-count 9 46: option send-gids true 47: end-volume 48: 49: volume patchy-client-3 50: type protocol/client 51: option ping-timeout 42 52: option remote-host 53: option remote-subvolume /d/backends/patchy3 54: option transport-type socket 55: option transport.address-family inet 56: option username 81e6517d-ac80-4d0f-9180-ca2b5a63ab8a 57: option password 3eef276b-b00d-4246-985e-00d0b685a69d 58: option transport.tcp-user-timeout 0 59: option transport.socket.keepalive-time 20 60: option transport.socket.keepalive-interval 2 61: option transport.socket.keepalive-count 9 62: option send-gids true 63: end-volume 64: 65: volume patchy-dht 66: type cluster/distribute 67: option directory-layout-spread 3 68: option lock-migration off 69: option force-migration off [2019-02-14 05:24:30.953445] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed 70: subvolumes patchy-client-0 patchy-client-1 patchy-client-2 patchy-client-3 71: end-volume 72: 73: volume patchy-write-behind 74: type performance/write-behind 75: subvolumes patchy-dht 76: end-volume 77: 78: volume patchy-read-ahead 79: type performance/read-ahead 80: subvolumes patchy-write-behind 81: end-volume 82: 83: volume patchy-readdir-ahead 84: type performance/readdir-ahead 85: option parallel-readdir off 86: option rda-request-size 131072 87: option rda-cache-limit 10MB 88: subvolumes patchy-read-ahead 89: end-volume [2019-02-14 05:24:30.953515] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.953628] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-patchy-client-0: changing port to 49152 (from 0) [2019-02-14 05:24:30.953665] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-patchy-client-1: changing port to 49153 (from 0) 90: 91: volume patchy-io-cache 92: type performance/io-cache 93: subvolumes patchy-readdir-ahead 94: end-volume 95: 96: volume patchy-quick-read 97: type performance/quick-read 98: subvolumes patchy-io-cache 99: end-volume 100: 101: volume patchy-open-behind 102: type performance/open-behind 103: subvolumes patchy-quick-read 104: end-volume 105: 106: volume patchy-md-cache 107: type performance/md-cache 108: subvolumes patchy-open-behind 109: end-volume 110: 111: volume patchy-io-threads 112: type performance/io-threads 113: subvolumes patchy-md-cache 114: end-volume 115: 116: volume patchy 117: type debug/io-stats 118: option log-level DEBUG 119: option latency-measurement off 120: option count-fop-hits off 121: subvolumes patchy-io-threads 122: end-volume 123: 124: volume meta-autoload 125: type meta [2019-02-14 05:24:30.953693] D [MSGID: 0] [client-handshake.c:1470:server_has_portmap] 0-patchy-client-2: detected portmapper on server 126: subvolumes patchy 127: end-volume 128: +------------------------------------------------------------------------------+ [2019-02-14 05:24:30.953891] D [glusterfsd-mgmt.c:2761:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given [2019-02-14 05:24:30.953841] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.954126] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.954138] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.954159] D [MSGID: 0] [client-handshake.c:1470:server_has_portmap] 0-patchy-client-3: detected portmapper on server [2019-02-14 05:24:30.954174] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.954226] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:30.954238] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-0: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:30.954247] D [MSGID: 0] [client.c:2283:client_rpc_notify] 0-patchy-client-0: disconnected (skipped notify) [2019-02-14 05:24:30.954295] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.954307] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.954317] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:30.954325] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-1: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:30.954334] D [MSGID: 0] [client.c:2283:client_rpc_notify] 0-patchy-client-1: disconnected (skipped notify) [2019-02-14 05:24:30.954356] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x68)[0x3ffabc8f520] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x24)[0x3ffabc8fa74] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x16e3e)[0x3ffa7316e3e] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.954372] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip-10.20.81.111 (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.954427] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x68)[0x3ffabc8f520] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x24)[0x3ffabc8fa74] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x16e3e)[0x3ffa7316e3e] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.954488] D [logging.c:1983:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2019-02-14 05:24:30.954447] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip-10.20.81.111 (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.954487] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-0: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.954514] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.954549] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-1: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.954563] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-1: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.954783] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49152: ping timer event already removed [2019-02-14 05:24:30.954798] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.956050] W [dict.c:923:str_to_data] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x419f0) [0x3ffa73419f0] -->/usr/local/lib/libglusterfs.so.0(dict_set_str+0x1e) [0x3ffabd25776] -->/usr/local/lib/libglusterfs.so.0(str_to_data+0xa4) [0x3ffabd223bc] ) 0-dict: value is NULL [Invalid argument] [2019-02-14 05:24:30.956074] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-patchy-client-0: failed to set process-name in handshake msg [2019-02-14 05:24:30.956158] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.956281] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49153: ping timer event already removed [2019-02-14 05:24:30.956294] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.956313] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-patchy-client-2: changing port to 49156 (from 0) [2019-02-14 05:24:30.956341] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-patchy-client-3: changing port to 49157 (from 0) [2019-02-14 05:24:30.956418] W [dict.c:923:str_to_data] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x419f0) [0x3ffa73419f0] -->/usr/local/lib/libglusterfs.so.0(dict_set_str+0x1e) [0x3ffabd25776] -->/usr/local/lib/libglusterfs.so.0(str_to_data+0xa4) [0x3ffabd223bc] ) 0-dict: value is NULL [Invalid argument] [2019-02-14 05:24:30.956431] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-patchy-client-1: failed to set process-name in handshake msg [2019-02-14 05:24:30.956453] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-1: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.956645] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49153: ping timer event already removed [2019-02-14 05:24:30.956689] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.956702] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:30.956712] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:30.956722] D [MSGID: 0] [client.c:2283:client_rpc_notify] 0-patchy-client-2: disconnected (skipped notify) [2019-02-14 05:24:30.956826] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x68)[0x3ffabc8f520] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x24)[0x3ffabc8fa74] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x16e3e)[0x3ffa7316e3e] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.956842] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip-10.20.81.111 (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.956954] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:30.956965] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-3: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:30.956975] D [MSGID: 0] [client.c:2283:client_rpc_notify] 0-patchy-client-3: disconnected (skipped notify) [2019-02-14 05:24:30.957104] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x68)[0x3ffabc8f520] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x24)[0x3ffabc8fa74] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x16e3e)[0x3ffa7316e3e] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:30.957125] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip-10.20.81.111 (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:30.957253] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.957270] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.957407] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49156: ping timer event already removed [2019-02-14 05:24:30.957422] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.957440] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-3: got RPC_CLNT_CONNECT [2019-02-14 05:24:30.957454] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.957589] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49157: ping timer event already removed [2019-02-14 05:24:30.957609] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.958897] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49152: ping timer event already removed [2019-02-14 05:24:30.958925] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.959067] W [dict.c:923:str_to_data] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x419f0) [0x3ffa73419f0] -->/usr/local/lib/libglusterfs.so.0(dict_set_str+0x1e) [0x3ffabd25776] -->/usr/local/lib/libglusterfs.so.0(str_to_data+0xa4) [0x3ffabd223bc] ) 0-dict: value is NULL [Invalid argument] [2019-02-14 05:24:30.959086] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-patchy-client-2: failed to set process-name in handshake msg [2019-02-14 05:24:30.959121] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.960318] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49156: ping timer event already removed [2019-02-14 05:24:30.960336] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.960386] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-patchy-client-0: Connected to patchy-client-0, attached to remote volume '/d/backends/patchy0'. [2019-02-14 05:24:30.960401] D [MSGID: 0] [client-handshake.c:1012:client_post_handshake] 0-patchy-client-0: No fds to open - notifying all parents child up [2019-02-14 05:24:30.960578] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49152: ping timer event already removed [2019-02-14 05:24:30.960638] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-patchy-client-1: Connected to patchy-client-1, attached to remote volume '/d/backends/patchy1'. [2019-02-14 05:24:30.960655] D [MSGID: 0] [client-handshake.c:1012:client_post_handshake] 0-patchy-client-1: No fds to open - notifying all parents child up [2019-02-14 05:24:30.961049] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49153: ping timer event already removed [2019-02-14 05:24:30.961130] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-patchy-client-2: Connected to patchy-client-2, attached to remote volume '/d/backends/patchy2'. [2019-02-14 05:24:30.961150] D [MSGID: 0] [client-handshake.c:1012:client_post_handshake] 0-patchy-client-2: No fds to open - notifying all parents child up [2019-02-14 05:24:30.961138] W [dict.c:923:str_to_data] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x419f0) [0x3ffa73419f0] -->/usr/local/lib/libglusterfs.so.0(dict_set_str+0x1e) [0x3ffabd25776] -->/usr/local/lib/libglusterfs.so.0(str_to_data+0xa4) [0x3ffabd223bc] ) 0-dict: value is NULL [Invalid argument] [2019-02-14 05:24:30.961243] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-patchy-client-3: failed to set process-name in handshake msg [2019-02-14 05:24:30.961268] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:30.961413] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49157: ping timer event already removed [2019-02-14 05:24:30.961425] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-3: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:30.961464] D [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-patchy-client-0: Ping latency is 0ms [2019-02-14 05:24:30.961559] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-1': avail_percent is: 69.00 and avail_space is: 4917207040 and avail_inodes is: 88.00 [2019-02-14 05:24:30.961625] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-0': avail_percent is: 69.00 and avail_space is: 4917207040 and avail_inodes is: 88.00 [2019-02-14 05:24:30.961649] D [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-patchy-client-1: Ping latency is 0ms [2019-02-14 05:24:30.963106] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49156: ping timer event already removed [2019-02-14 05:24:30.963203] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-patchy-client-3: Connected to patchy-client-3, attached to remote volume '/d/backends/patchy3'. [2019-02-14 05:24:30.963213] D [MSGID: 0] [client-handshake.c:1012:client_post_handshake] 0-patchy-client-3: No fds to open - notifying all parents child up [2019-02-14 05:24:30.963232] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-2': avail_percent is: 69.00 and avail_space is: 4917202944 and avail_inodes is: 88.00 [2019-02-14 05:24:30.963256] D [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-patchy-client-2: Ping latency is 0ms [2019-02-14 05:24:30.963368] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: 10.20.81.111:49157: ping timer event already removed [2019-02-14 05:24:30.963438] D [fuse-bridge.c:5386:notify] 0-fuse: got event 5 on graph 0 [2019-02-14 05:24:30.963616] D [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-patchy-client-3: Ping latency is 0ms [2019-02-14 05:24:30.963691] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-3': avail_percent is: 69.00 and avail_space is: 4917202944 and avail_inodes is: 88.00 [2019-02-14 05:24:30.964953] D [fuse-bridge.c:4969:fuse_get_mount_status] 0-fuse: mount status is 0 [2019-02-14 05:24:30.965031] D [fuse-bridge.c:4236:fuse_init] 0-glusterfs-fuse: Detected support for FUSE_AUTO_INVAL_DATA. Enabling fopen_keep_cache automatically. [2019-02-14 05:24:30.965052] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26 [2019-02-14 05:24:30.965061] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-02-14 05:24:30.965121] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: LOOKUP scheduled as fast fop [2019-02-14 05:24:30.965522] D [MSGID: 0] [dht-common.c:3468:dht_lookup] 0-patchy-dht: Calling fresh lookup for / on patchy-client-0 [2019-02-14 05:24:30.966212] D [MSGID: 0] [dht-common.c:3020:dht_lookup_cbk] 0-patchy-dht: fresh_lookup returned for / with op_ret 0 [2019-02-14 05:24:30.966818] D [MSGID: 0] [dht-common.c:1415:dht_lookup_dir_cbk] 0-patchy-dht: Internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-02-14 05:24:30.967225] D [logging.c:1983:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [dht-common.c:1415:dht_lookup_dir_cbk] 0-patchy-dht: Internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 " repeated 3 times between [2019-02-14 05:24:30.966818] and [2019-02-14 05:24:30.967221] [2019-02-14 05:24:30.967225] I [MSGID: 109063] [dht-layout.c:710:dht_layout_normalize] 0-patchy-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 [2019-02-14 05:24:30.967256] D [MSGID: 0] [dht-common.c:1479:dht_lookup_dir_cbk] 0-patchy-dht: fixing assignment on / [2019-02-14 05:24:30.967273] D [MSGID: 0] [dht-selfheal.c:2089:dht_selfheal_layout_new_directory] 0-patchy-dht: chunk size = 0xffffffff / 20325 = 211314.504059 [2019-02-14 05:24:30.967287] D [MSGID: 0] [dht-selfheal.c:2130:dht_selfheal_layout_new_directory] 0-patchy-dht: assigning range size 0x55555555 to patchy-client-0 [2019-02-14 05:24:30.967296] D [MSGID: 0] [dht-selfheal.c:2130:dht_selfheal_layout_new_directory] 0-patchy-dht: assigning range size 0x55555555 to patchy-client-1 [2019-02-14 05:24:30.967305] D [MSGID: 0] [dht-selfheal.c:2130:dht_selfheal_layout_new_directory] 0-patchy-dht: assigning range size 0x55555555 to patchy-client-2 [2019-02-14 05:24:30.975814] D [MSGID: 109036] [dht-common.c:10935:dht_log_new_layout_for_dir_selfheal] 0-patchy-dht: Setting layout of / with [Subvol_name: patchy-client-0, Err: -1 , Start: 0 , Stop: 1431655764 , Hash: 1 ], [Subvol_name: patchy-client-1, Err: -1 , Start: 1431655765 , Stop: 2863311529 , Hash: 1 ], [Subvol_name: patchy-client-2, Err: -1 , Start: 2863311530 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: patchy-client-3, Err: -1 , Start: 0 , Stop: 0 , Hash: 0 ], [2019-02-14 05:24:30.975848] D [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.dht' is would not be sent on wire in future [Invalid argument] [2019-02-14 05:24:30.976576] D [dict.c:2530:dict_get_bin] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x703ca) [0x3ffa73703ca] -->/usr/local/lib/glusterfs/4.1.5/xlator/cluster/distribute.so(+0x20338) [0x3ffa7220338] -->/usr/local/lib/libglusterfs.so.0(dict_get_bin+0x15a) [0x3ffabd25bba] ) 0-dict: key dht-get-iatt-in-xattr, pointer type asked, has iatt type [Invalid argument] [2019-02-14 05:24:30.976705] D [dict.c:2530:dict_get_bin] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x703ca) [0x3ffa73703ca] -->/usr/local/lib/glusterfs/4.1.5/xlator/cluster/distribute.so(+0x20338) [0x3ffa7220338] -->/usr/local/lib/libglusterfs.so.0(dict_get_bin+0x15a) [0x3ffabd25bba] ) 0-dict: key dht-get-iatt-in-xattr, pointer type asked, has iatt type [Invalid argument] [2019-02-14 05:24:30.976929] D [dict.c:2530:dict_get_bin] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x703ca) [0x3ffa73703ca] -->/usr/local/lib/glusterfs/4.1.5/xlator/cluster/distribute.so(+0x20338) [0x3ffa7220338] -->/usr/local/lib/libglusterfs.so.0(dict_get_bin+0x15a) [0x3ffabd25bba] ) 0-dict: key dht-get-iatt-in-xattr, pointer type asked, has iatt type [Invalid argument] [2019-02-14 05:24:30.977026] D [dict.c:2530:dict_get_bin] (-->/usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x703ca) [0x3ffa73703ca] -->/usr/local/lib/glusterfs/4.1.5/xlator/cluster/distribute.so(+0x20338) [0x3ffa7220338] -->/usr/local/lib/libglusterfs.so.0(dict_get_bin+0x15a) [0x3ffabd25bba] ) 0-dict: key dht-get-iatt-in-xattr, pointer type asked, has iatt type [Invalid argument] [2019-02-14 05:24:31.044108]:++++++++++ G_LOG:./tests/bugs/glusterfs/bug-902610.t: TEST: 58 ls -l /mnt/glusterfs/0 ++++++++++ [2019-02-14 05:24:31.050192] D [logging.c:1983:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.dht' is would not be sent on wire in future [Invalid argument]" repeated 3 times between [2019-02-14 05:24:30.975848] and [2019-02-14 05:24:30.975974] [2019-02-14 05:24:31.050190] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: LOOKUP scheduled as fast fop [2019-02-14 05:24:31.051871] D [MSGID: 0] [dht-common.c:1593:dht_revalidate_cbk] 0-patchy-dht: revalidate lookup of / returned with op_ret 0 [2019-02-14 05:24:31.051890] D [MSGID: 0] [dht-common.c:1690:dht_revalidate_cbk] 0-patchy-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-02-14 05:24:31.051919] D [MSGID: 0] [dht-common.c:1593:dht_revalidate_cbk] 0-patchy-dht: revalidate lookup of / returned with op_ret 0 [2019-02-14 05:24:31.051930] D [MSGID: 0] [dht-common.c:1690:dht_revalidate_cbk] 0-patchy-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-02-14 05:24:31.051961] D [MSGID: 0] [dht-common.c:1593:dht_revalidate_cbk] 0-patchy-dht: revalidate lookup of / returned with op_ret 0 [2019-02-14 05:24:31.051971] D [MSGID: 0] [dht-common.c:1690:dht_revalidate_cbk] 0-patchy-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-02-14 05:24:31.051999] D [MSGID: 0] [dht-common.c:1593:dht_revalidate_cbk] 0-patchy-dht: revalidate lookup of / returned with op_ret 0 [2019-02-14 05:24:31.052010] D [MSGID: 0] [dht-common.c:1690:dht_revalidate_cbk] 0-patchy-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-02-14 05:24:31.052068] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: STAT scheduled as fast fop [2019-02-14 05:24:31.052455] D [logging.c:1983:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2019-02-14 05:24:31.052369] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: STAT scheduled as fast fop [2019-02-14 05:24:31.052455] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: OPENDIR scheduled as fast fop [2019-02-14 05:24:31.068849] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: STAT scheduled as fast fop [2019-02-14 05:24:31.069006] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: READDIRP scheduled as fast fop [2019-02-14 05:24:31.069249] D [MSGID: 0] [dht-common.c:6468:dht_readdirp_cbk] 0-patchy-dht: Processing entries from patchy-client-0 [2019-02-14 05:24:31.069263] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-0: entry = ., type = 4 [2019-02-14 05:24:31.069282] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.069306] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.069316] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.069328] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.069342] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.069350] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(.) and subvolume(patchy-client-2) has hash=3553706931 [2019-02-14 05:24:31.069359] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-0: entry = .., type = 4 [2019-02-14 05:24:31.069380] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.069388] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.069398] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.069407] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.069415] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.069422] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(..) and subvolume(patchy-client-2) has hash=3781782680 [2019-02-14 05:24:31.070034] D [MSGID: 0] [dht-common.c:6468:dht_readdirp_cbk] 0-patchy-dht: Processing entries from patchy-client-1 [2019-02-14 05:24:31.070047] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-1: entry = ., type = 4 [2019-02-14 05:24:31.070064] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.070075] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.070088] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.070097] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070109] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070117] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(.) and subvolume(patchy-client-2) has hash=3553706931 [2019-02-14 05:24:31.070135] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-1: entry = .., type = 4 [2019-02-14 05:24:31.070145] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.070153] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.070165] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.070174] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070181] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070192] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(..) and subvolume(patchy-client-2) has hash=3781782680 [2019-02-14 05:24:31.070755] D [MSGID: 0] [dht-common.c:6468:dht_readdirp_cbk] 0-patchy-dht: Processing entries from patchy-client-2 [2019-02-14 05:24:31.070772] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-2: entry = ., type = 4 [2019-02-14 05:24:31.070794] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.070802] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.070810] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.070819] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070826] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070837] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(.) and subvolume(patchy-client-2) has hash=3553706931 [2019-02-14 05:24:31.070851] D [MSGID: 0] [dht-common.c:6610:dht_readdirp_cbk] 0-patchy-dht: patchy-client-2: Adding entry = . [2019-02-14 05:24:31.070864] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-2: entry = .., type = 4 [2019-02-14 05:24:31.070874] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.070885] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.070892] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.070899] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070912] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.070921] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(..) and subvolume(patchy-client-2) has hash=3781782680 [2019-02-14 05:24:31.070931] D [MSGID: 0] [dht-common.c:6610:dht_readdirp_cbk] 0-patchy-dht: patchy-client-2: Adding entry = .. [2019-02-14 05:24:31.071440] D [MSGID: 0] [dht-common.c:6468:dht_readdirp_cbk] 0-patchy-dht: Processing entries from patchy-client-3 [2019-02-14 05:24:31.071457] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-3: entry = ., type = 4 [2019-02-14 05:24:31.071474] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.071482] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.071491] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.071499] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.071513] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.071521] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(.) and subvolume(patchy-client-2) has hash=3553706931 [2019-02-14 05:24:31.071532] D [MSGID: 0] [dht-common.c:6475:dht_readdirp_cbk] 0-patchy-dht: patchy-client-3: entry = .., type = 4 [2019-02-14 05:24:31.071544] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:31.071552] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:31.071563] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:31.071571] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.071579] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:31.071597] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(..) and subvolume(patchy-client-2) has hash=3781782680 [2019-02-14 05:24:31.071886] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: READDIRP scheduled as fast fop [2019-02-14 05:24:32.152698] D [socket.c:568:__socket_rwv] 0-patchy-client-2: EOF on socket [2019-02-14 05:24:32.152736] W [socket.c:599:__socket_rwv] 0-patchy-client-2: readv on 10.20.81.111:49156 failed (No data available) [2019-02-14 05:24:32.152747] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:32.152763] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:32.153019] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-patchy-client-2: disconnected from patchy-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2019-02-14 05:24:34.945792] D [MSGID: 0] [common-utils.c:339:gf_resolve_ip6] 0-resolver: returning ip- (port-24007) for hostname: and port: 24007 [2019-02-14 05:24:34.945880] D [MSGID: 0] [common-utils.c:360:gf_resolve_ip6] 0-resolver: next DNS query will return: ip-10.20.81.111 port-24007 [2019-02-14 05:24:34.945895] D [socket.c:3208:socket_fix_ssl_opts] 0-patchy-client-2: disabling SSL for portmapper connection [2019-02-14 05:24:34.946194] D [MSGID: 0] [client.c:2231:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_CONNECT [2019-02-14 05:24:34.946263] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:34.946664] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:34.946677] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:34.946710] D [MSGID: 0] [client-handshake.c:1470:server_has_portmap] 0-patchy-client-2: detected portmapper on server [2019-02-14 05:24:34.946725] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-patchy-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-14 05:24:34.946835] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(+0x13b72)[0x3ffabc93b72] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x2d4)[0x3ffabc904f4] (--> /usr/local/lib/glusterfs/4.1.5/xlator/protocol/client.so(+0x163c4)[0x3ffa73163c4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:34.946844] D [rpc-clnt-ping.c:335:rpc_clnt_start_ping] 0-patchy-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) [2019-02-14 05:24:34.946977] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-patchy-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2019-02-14 05:24:34.947006] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:34.947020] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-2: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:34.947028] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-patchy-client-2: disconnected from patchy-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2019-02-14 05:24:34.947125] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x132)[0x3ffabd2e1fa] (--> /usr/local/lib/libgfrpc.so.0(+0x13342)[0x3ffabc93342] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x68)[0x3ffabc8f520] (--> /usr/local/lib/libgfrpc.so.0(+0x1019c)[0x3ffabc9019c] (--> /usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x2c)[0x3ffabc8c2f4] ))))) 0-: :24007: ping timer event already removed [2019-02-14 05:24:35.242837] D [socket.c:568:__socket_rwv] 0-patchy-client-3: EOF on socket [2019-02-14 05:24:35.242881] W [socket.c:599:__socket_rwv] 0-patchy-client-3: readv on 10.20.81.111:49157 failed (No data available) [2019-02-14 05:24:35.242891] D [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-02-14 05:24:35.242911] D [MSGID: 0] [client.c:2241:client_rpc_notify] 0-patchy-client-3: got RPC_CLNT_DISCONNECT [2019-02-14 05:24:35.242940] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-patchy-client-3: disconnected from patchy-client-3. Client process will keep trying to connect to glusterd until brick's port is available [2019-02-14 05:24:37.250965] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: STAT scheduled as fast fop [2019-02-14 05:24:37.251228] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.251307] W [MSGID: 114031] [client-rpc-fops_v2.c:328:client4_0_stat_cbk] 0-patchy-client-2: remote operation failed [Transport endpoint is not connected] [2019-02-14 05:24:37.251330] D [MSGID: 0] [client-rpc-fops_v2.c:333:client4_0_stat_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.251350] D [MSGID: 0] [dht-inode-read.c:297:dht_attr_cbk] 0-patchy-dht: subvolume patchy-client-2 returned -1 [Transport endpoint is not connected] [2019-02-14 05:24:37.251364] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-3: connection in disconnected state [2019-02-14 05:24:37.251374] W [MSGID: 114031] [client-rpc-fops_v2.c:328:client4_0_stat_cbk] 0-patchy-client-3: remote operation failed [Transport endpoint is not connected] [2019-02-14 05:24:37.251403] D [MSGID: 0] [client-rpc-fops_v2.c:333:client4_0_stat_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-3 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.251413] D [MSGID: 0] [dht-inode-read.c:297:dht_attr_cbk] 0-patchy-dht: subvolume patchy-client-3 returned -1 [Transport endpoint is not connected] [2019-02-14 05:24:37.252026] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: LOOKUP scheduled as fast fop [2019-02-14 05:24:37.252222] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:37.252241] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:37.252251] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:37.252261] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.252271] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.252280] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(dir1) and subvolume(patchy-client-2) has hash=3567524015 [2019-02-14 05:24:37.252294] D [MSGID: 0] [dht-common.c:3468:dht_lookup] 0-patchy-dht: Calling fresh lookup for /dir1 on patchy-client-2 [2019-02-14 05:24:37.252318] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.252333] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.252362] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.252372] D [MSGID: 0] [dht-common.c:3020:dht_lookup_cbk] 0-patchy-dht: fresh_lookup returned for /dir1 with op_ret -1 [Transport endpoint is not connected] [2019-02-14 05:24:37.252450] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.252466] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.252482] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.252495] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-2 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.252517] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-3: connection in disconnected state [2019-02-14 05:24:37.252528] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-3: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.252544] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-3 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.252555] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-3 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.253048] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-1 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253069] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-1 returned error [No such file or directory] [2019-02-14 05:24:37.253087] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-0 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253113] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-0 returned error [No such file or directory] [2019-02-14 05:24:37.253126] E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-patchy-dht: dict is null [2019-02-14 05:24:37.253135] D [MSGID: 0] [dht-common.c:1505:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-dht returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253147] D [MSGID: 0] [write-behind.c:2396:wb_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-write-behind returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253161] D [MSGID: 0] [io-cache.c:268:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-io-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253173] D [MSGID: 0] [quick-read.c:473:qr_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-quick-read returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253185] D [MSGID: 0] [md-cache.c:1130:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-md-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253195] D [MSGID: 0] [defaults.c:1328:default_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-io-threads returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253209] D [MSGID: 0] [io-stats.c:2278:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253223] D [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/dir1: failed to resolve (No such file or directory) [2019-02-14 05:24:37.253262] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: LOOKUP scheduled as fast fop [2019-02-14 05:24:37.253353] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:37.253368] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:37.253374] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:37.253380] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.253386] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.253392] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(dir1) and subvolume(patchy-client-2) has hash=3567524015 [2019-02-14 05:24:37.253401] D [MSGID: 0] [dht-common.c:3468:dht_lookup] 0-patchy-dht: Calling fresh lookup for /dir1 on patchy-client-2 [2019-02-14 05:24:37.253417] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.253426] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.253435] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.253442] D [MSGID: 0] [dht-common.c:3020:dht_lookup_cbk] 0-patchy-dht: fresh_lookup returned for /dir1 with op_ret -1 [Transport endpoint is not connected] [2019-02-14 05:24:37.253504] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.253513] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.253521] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.253530] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-2 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.253541] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-3: connection in disconnected state [2019-02-14 05:24:37.253552] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-3: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.253560] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-client-3 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.253568] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-3 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.253854] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-client-1 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253872] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-1 returned error [No such file or directory] [2019-02-14 05:24:37.253890] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-client-0 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253909] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-0 returned error [No such file or directory] [2019-02-14 05:24:37.253920] E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-patchy-dht: dict is null [2019-02-14 05:24:37.253928] D [MSGID: 0] [dht-common.c:1505:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-dht returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253938] D [MSGID: 0] [write-behind.c:2396:wb_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-write-behind returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253948] D [MSGID: 0] [io-cache.c:268:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-io-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253958] D [MSGID: 0] [quick-read.c:473:qr_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-quick-read returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253967] D [MSGID: 0] [md-cache.c:1130:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-md-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253976] D [MSGID: 0] [defaults.c:1328:default_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-io-threads returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.253985] D [MSGID: 0] [io-stats.c:2278:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254044] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: STAT scheduled as fast fop [2019-02-14 05:24:37.254154] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: LOOKUP scheduled as fast fop [2019-02-14 05:24:37.254200] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:37.254211] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:37.254218] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:37.254224] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.254230] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.254237] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(dir1) and subvolume(patchy-client-2) has hash=3567524015 [2019-02-14 05:24:37.254248] D [MSGID: 0] [dht-common.c:3468:dht_lookup] 0-patchy-dht: Calling fresh lookup for /dir1 on patchy-client-2 [2019-02-14 05:24:37.254263] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.254273] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.254283] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.254297] D [MSGID: 0] [dht-common.c:3020:dht_lookup_cbk] 0-patchy-dht: fresh_lookup returned for /dir1 with op_ret -1 [Transport endpoint is not connected] [2019-02-14 05:24:37.254351] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.254373] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-2: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.254384] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.254393] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-2 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.254404] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-3: connection in disconnected state [2019-02-14 05:24:37.254412] W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-patchy-client-3: remote operation failed. Path: /dir1 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2019-02-14 05:24:37.254419] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-3 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.254428] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-3 returned error [Transport endpoint is not connected] [2019-02-14 05:24:37.254769] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-1 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254785] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-1 returned error [No such file or directory] [2019-02-14 05:24:37.254817] D [MSGID: 0] [client-rpc-fops_v2.c:2548:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-client-0 returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254839] D [MSGID: 0] [dht-common.c:1378:dht_lookup_dir_cbk] 0-patchy-dht: lookup of /dir1 on patchy-client-0 returned error [No such file or directory] [2019-02-14 05:24:37.254854] E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-patchy-dht: dict is null [2019-02-14 05:24:37.254868] D [MSGID: 0] [dht-common.c:1505:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-dht returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254883] D [MSGID: 0] [write-behind.c:2396:wb_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-write-behind returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254899] D [MSGID: 0] [io-cache.c:268:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-io-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254912] D [MSGID: 0] [quick-read.c:473:qr_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-quick-read returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254927] D [MSGID: 0] [md-cache.c:1130:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-md-cache returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254940] D [MSGID: 0] [defaults.c:1328:default_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy-io-threads returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254953] D [MSGID: 0] [io-stats.c:2278:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x3ff90007778, patchy returned -1 error: No such file or directory [No such file or directory] [2019-02-14 05:24:37.254977] D [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/dir1: failed to resolve (No such file or directory) [2019-02-14 05:24:37.254996] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-patchy-io-threads: MKDIR scheduled as normal fop [2019-02-14 05:24:37.255160] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.255197] W [MSGID: 114031] [client-rpc-fops_v2.c:605:client4_0_statfs_cbk] 0-patchy-client-2: remote operation failed [Transport endpoint is not connected] [2019-02-14 05:24:37.255209] D [MSGID: 0] [client-rpc-fops_v2.c:608:client4_0_statfs_cbk] 0-stack-trace: stack-address: 0x3ff9c0051d8, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255220] W [MSGID: 109075] [dht-diskusage.c:45:dht_du_info_cbk] 0-patchy-dht: failed to get disk info from patchy-client-2 [Transport endpoint is not connected] [2019-02-14 05:24:37.255231] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-3: connection in disconnected state [2019-02-14 05:24:37.255239] W [MSGID: 114031] [client-rpc-fops_v2.c:605:client4_0_statfs_cbk] 0-patchy-client-3: remote operation failed [Transport endpoint is not connected] [2019-02-14 05:24:37.255246] D [MSGID: 0] [client-rpc-fops_v2.c:608:client4_0_statfs_cbk] 0-stack-trace: stack-address: 0x3ff9c0051d8, patchy-client-3 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255256] W [MSGID: 109075] [dht-diskusage.c:45:dht_du_info_cbk] 0-patchy-dht: failed to get disk info from patchy-client-3 [Transport endpoint is not connected] [2019-02-14 05:24:37.255281] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:37.255287] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:37.255294] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:37.255300] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.255306] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.255312] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(dir1) and subvolume(patchy-client-2) has hash=3567524015 [2019-02-14 05:24:37.255327] W [dht-layout.c:165:dht_layout_search] 0-patchy-dht: layout->count=4 [2019-02-14 05:24:37.255334] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[0].start=0 and layout->list[0].stop=1431655764 [2019-02-14 05:24:37.255341] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[1].start=1431655765 and layout->list[1].stop=-1431655767 [2019-02-14 05:24:37.255350] W [dht-layout.c:167:dht_layout_search] 0-patchy-dht: Inside FOR=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.255357] W [dht-layout.c:171:dht_layout_search] 0-patchy-dht: Insied IF=>layout->list[2].start=-1431655766 and layout->list[2].stop=-1 [2019-02-14 05:24:37.255362] W [dht-layout.c:183:dht_layout_search] 0-patchy-dht: Name(dir1) and subvolume(patchy-client-2) has hash=3567524015 [2019-02-14 05:24:37.255363] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-0': avail_percent is: 69.00 and avail_space is: 4917166080 and avail_inodes is: 88.00 [2019-02-14 05:24:37.255393] D [MSGID: 0] [client.c:163:client_submit_request] 0-patchy-client-2: connection in disconnected state [2019-02-14 05:24:37.255406] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-patchy-client-2: remote operation failed [Transport endpoint is not connected] [2019-02-14 05:24:37.255432] D [MSGID: 0] [client-rpc-fops_v2.c:1309:client4_0_inodelk_cbk] 0-stack-trace: stack-address: 0x3ff9c008398, patchy-client-2 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255443] E [MSGID: 109119] [dht-lock.c:1052:dht_blocking_inodelk_cbk] 0-patchy-dht: inodelk failed on subvol patchy-client-2, gfid:00000000-0000-0000-0000-000000000001 [Transport endpoint is not connected] [2019-02-14 05:24:37.255485] W [MSGID: 109114] [dht-common.c:9052:dht_mkdir_guard_parent_layout_cbk] 0-patchy-dht: mkdir (00000000-0000-0000-0000-000000000001/dir1) (path: /dir1): Acquiring lock on parent to guard against layout-change failed. [Transport endpoint is not connected] [2019-02-14 05:24:37.255495] D [MSGID: 0] [dht-common.c:9075:dht_mkdir_guard_parent_layout_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-dht returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255504] D [MSGID: 0] [md-cache.c:1513:mdc_mkdir_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-md-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255517] D [MSGID: 0] [defaults.c:1282:default_mkdir_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy-io-threads returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255529] D [MSGID: 0] [io-stats.c:2333:io_stats_mkdir_cbk] 0-stack-trace: stack-address: 0x3ffa0070b08, patchy returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected] [2019-02-14 05:24:37.255539] W [fuse-bridge.c:565:fuse_entry_cbk] 0-glusterfs-fuse: 14: MKDIR() /dir1 => -1 (Transport endpoint is not connected) [2019-02-14 05:24:37.255833] D [MSGID: 0] [dht-diskusage.c:97:dht_du_info_cbk] 0-patchy-dht: subvolume 'patchy-client-1': avail_percent is: 69.00 and avail_space is: 4917166080 and avail_inodes is: 88.00 [2019-02-14 05:24:37.363285]:++++++++++ G_LOG:./tests/bugs/glusterfs/bug-902610.t: TEST: 67 0 echo 1 ++++++++++ [2019-02-14 05:24:37.396042] D [fuse-bridge.c:5087:fuse_thread_proc] 0-glusterfs-fuse: terminating upon getting ENODEV when reading /dev/fuse [2019-02-14 05:24:37.396079] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /mnt/glusterfs/0 [2019-02-14 05:24:37.396157] D [logging.c:1795:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages [2019-02-14 05:24:37.396179] D [logging.c:1798:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages [2019-02-14 05:24:37.396248] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x79c0) [0x3ffab8079c0] -->glusterfs(glusterfs_sigwaiter+0xce) [0x1008abe] -->glusterfs(cleanup_and_exit+0x74) [0x10088bc] ) 0-: received signum (15), shutting down [2019-02-14 05:24:37.396270] D [mgmt-pmap.c:79:rpc_clnt_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given [2019-02-14 05:24:37.396279] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/mnt/glusterfs/0'. [2019-02-14 05:24:37.396290] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/mnt/glusterfs/0'.