[2016-10-18 16:45:57.765794] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 16:45:57.780168] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-18 16:45:57.787067] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 16:45:57.787103] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 16:45:57.787508] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 16:45:57.787501] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-18 16:45:57.787528] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 16:45:57.791645] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-18 16:45:57.791747] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-18 16:45:57.796044] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-18 16:45:57.796083] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-18 16:45:57.796100] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-18 16:45:57.796117] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-18 16:45:57.796131] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-18 16:45:57.811774] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-18 16:45:57.812684] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-18 16:45:57.812725] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-18 16:45:57.812756] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-18 16:45:57.812813] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-18 16:45:57.812885] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option rpc-auth-allow-insecure on 136: option transport-type tcp 137: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 138: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 139: option auth.addr./rhgs/engine/engine.allow * 140: option transport.tcp-user-timeout 30 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-18 16:45:57.907341] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:45:57.907372] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-15890-2016/10/18-16:45:57:861640-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:45:58.912617] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:45:58.912649] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-20776-2016/10/18-16:45:56:32684-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:45:58.975066] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:45:58.975091] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:45:58.975621] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:45:58.975637] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:45:58.978359] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:45:58.978380] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:45:58.978622] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:45:58.978636] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:45:59.158015] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:45:59.158047] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-5867-2016/10/18-16:45:58:991917-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:46:01.885809] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:01.885845] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:01.886280] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:01.886300] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:01.887916] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:01.887949] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:01.888204] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:01.888223] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:02.891641] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:02.891672] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:02.891886] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:02.891901] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:02.893327] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:02.893346] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:02.893582] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:46:02.893597] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 16:46:49.478507] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:46:49.478552] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-16134-2016/10/18-16:46:49:402916-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:46:49.550508] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-16134-2016/10/18-16:46:49:402916-engine-client-0-0-0 [2016-10-18 16:46:49.550567] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-16134-2016/10/18-16:46:49:402916-engine-client-0-0-0 [2016-10-18 16:46:49.851886] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:46:49.851917] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-16240-2016/10/18-16:46:49:805566-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:46:50.252547] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-16240-2016/10/18-16:46:49:805566-engine-client-0-0-0 [2016-10-18 16:46:50.252670] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-16240-2016/10/18-16:46:49:805566-engine-client-0-0-0 [2016-10-18 16:46:50.713461] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 16:46:50.713493] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-16348-2016/10/18-16:46:50:667679-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 16:46:50.776735] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-16348-2016/10/18-16:46:50:667679-engine-client-0-0-0 [2016-10-18 16:46:50.776781] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-16348-2016/10/18-16:46:50:667679-engine-client-0-0-0 [2016-10-18 16:56:03.001801] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 16:56:03.001848] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 16:56:03.001801] and [2016-10-18 16:56:03.004068] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 16:56:03.001848] and [2016-10-18 16:56:03.004070] [2016-10-18 17:01:59.623405] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:01:59.623439] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-17551-2016/10/18-17:01:59:577001-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:02:00.421938] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 17:02:00.422002] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 316: WRITEV 0 (7edacaa1-40d6-4dff-9471-8de9ef1b17fb) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:02:23.320118] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=7edacaa1-40d6-4dff-9471-8de9ef1b17fb, fd=0x7f0c280d67bc, offset=127488 size=512, buf=0x7f0bf0005000 [Invalid argument] [2016-10-18 17:02:23.320175] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 351: READV -2 (7edacaa1-40d6-4dff-9471-8de9ef1b17fb) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:06:04.001903] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:06:04.001966] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 17:06:04.001903] and [2016-10-18 17:06:04.004271] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 17:06:04.001966] and [2016-10-18 17:06:04.004272] [2016-10-18 17:08:28.502876] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:08:28.502910] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-17970-2016/10/18-17:08:28:457176-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:08:28.566645] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-17970-2016/10/18-17:08:28:457176-engine-client-0-0-0 [2016-10-18 17:08:28.566697] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-17970-2016/10/18-17:08:28:457176-engine-client-0-0-0 [2016-10-18 17:09:51.265196] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:09:51.265254] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-18136-2016/10/18-17:09:51:219698-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:09:51.326972] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-18136-2016/10/18-17:09:51:219698-engine-client-0-0-0 [2016-10-18 17:09:51.327052] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-18136-2016/10/18-17:09:51:219698-engine-client-0-0-0 [2016-10-18 17:09:51.700322] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-17551-2016/10/18-17:01:59:577001-engine-client-0-0-0 [2016-10-18 17:09:51.700385] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-17551-2016/10/18-17:01:59:577001-engine-client-0-0-0 [2016-10-18 17:09:52.201661] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:09:52.201691] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-18241-2016/10/18-17:09:52:158060-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:09:52.266586] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-18241-2016/10/18-17:09:52:158060-engine-client-0-0-0 [2016-10-18 17:09:52.266666] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-18241-2016/10/18-17:09:52:158060-engine-client-0-0-0 [2016-10-18 17:14:15.861917] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:14:15.861966] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-18755-2016/10/18-17:14:15:814349-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:14:16.588655] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 17:14:16.588719] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 331: WRITEV 0 (ed0167b3-6b33-4320-8ce5-930b8d5f2edc) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:14:39.552994] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=ed0167b3-6b33-4320-8ce5-930b8d5f2edc, fd=0x7f0c280d67bc, offset=127488 size=512, buf=0x7f0c1c005000 [Invalid argument] [2016-10-18 17:14:39.553054] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 366: READV -2 (ed0167b3-6b33-4320-8ce5-930b8d5f2edc) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:16:05.001755] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:16:05.001812] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 17:16:05.001755] and [2016-10-18 17:16:05.004179] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 17:16:05.001812] and [2016-10-18 17:16:05.004182] [2016-10-18 17:26:06.001864] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:26:06.001910] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 17:26:06.001864] and [2016-10-18 17:26:06.004248] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 17:26:06.001910] and [2016-10-18 17:26:06.004249] [2016-10-18 17:31:47.966565] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-22127-2016/10/18-17:31:45:35395-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:31:48.025120] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-22127-2016/10/18-17:31:45:35395-engine-client-0-0-0 [2016-10-18 17:31:48.025180] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-22127-2016/10/18-17:31:45:35395-engine-client-0-0-0 [2016-10-18 17:31:48.267517] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-22232-2016/10/18-17:31:45:363152-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:34:19.503652] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-22441-2016/10/18-17:34:16:599397-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:34:19.558219] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-22441-2016/10/18-17:34:16:599397-engine-client-0-0-0 [2016-10-18 17:34:19.558268] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-22441-2016/10/18-17:34:16:599397-engine-client-0-0-0 [2016-10-18 17:34:19.813357] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-22232-2016/10/18-17:31:45:363152-engine-client-0-0-0 [2016-10-18 17:34:19.813387] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-22232-2016/10/18-17:31:45:363152-engine-client-0-0-0 [2016-10-18 17:34:20.220493] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-22538-2016/10/18-17:34:17:316499-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:34:20.276598] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-22538-2016/10/18-17:34:17:316499-engine-client-0-0-0 [2016-10-18 17:34:20.276635] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-22538-2016/10/18-17:34:17:316499-engine-client-0-0-0 [2016-10-18 17:36:07.001674] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:36:07.001720] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 17:36:07.001674] and [2016-10-18 17:36:07.003801] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 17:36:07.001720] and [2016-10-18 17:36:07.003803] [2016-10-18 17:43:32.559663] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-23612-2016/10/18-17:43:29:644416-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:43:33.224407] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 17:43:33.224464] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 289: WRITEV 0 (579e4bf2-eb96-4a4a-81a9-0ed22670d011) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:43:56.173067] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=579e4bf2-eb96-4a4a-81a9-0ed22670d011, fd=0x7f0c280d6900, offset=127488 size=512, buf=0x7f0bf000e000 [Invalid argument] [2016-10-18 17:43:56.173143] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 319: READV -2 (579e4bf2-eb96-4a4a-81a9-0ed22670d011) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:45:56.362152] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-20776-2016/10/18-16:45:56:32684-engine-client-0-0-0 [2016-10-18 17:45:56.362218] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-20776-2016/10/18-16:45:56:32684-engine-client-0-0-0 [2016-10-18 17:45:57.362454] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x7f0c3bc8a262] -->/usr/sbin/glusterfsd(glusterfs_handle_terminate+0x15) [0x7f0c3c1214d5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f0c3c11e78b] ) 0-: received signum (15), shutting down [2016-10-18 17:46:02.394526] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 17:46:02.408597] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-18 17:46:02.415274] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 17:46:02.415310] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 17:46:02.415706] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 17:46:02.415729] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 17:46:02.415740] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-18 17:46:02.416989] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-18 17:46:02.417092] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-18 17:46:02.419504] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-18 17:46:02.419538] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-18 17:46:02.419573] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-18 17:46:02.419590] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-18 17:46:02.419604] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-18 17:46:02.419885] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-18 17:46:02.420647] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-18 17:46:02.420686] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-18 17:46:02.420717] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-18 17:46:02.420772] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-18 17:46:02.420840] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized [2016-10-18 17:46:02.427808] W [MSGID: 113036] [posix.c:2351:posix_rename] 0-engine-posix: found directory at /rhgs/engine/engine/.trashcan/ while expecting ENOENT [File exists] Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option rpc-auth-allow-insecure on 136: option transport-type tcp 137: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 138: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 139: option auth.addr./rhgs/engine/engine.allow * 140: option transport.tcp-user-timeout 30 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-18 17:46:02.515211] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:46:02.515239] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-19639-2016/10/18-17:46:02:473148-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:46:02.518914] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:46:02.518961] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-6932-2016/10/18-17:46:02:372958-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:46:02.532965] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:46:02.532997] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 17:46:02.533490] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:46:02.533505] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 17:46:02.536411] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:46:02.536438] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 17:46:02.536675] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:46:02.536689] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 17:46:06.393706] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:46:06.393747] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-23810-2016/10/18-17:45:59:503149-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:46:07.555055] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-23612-2016/10/18-17:43:29:644416-engine-client-0-0-1 (version: 3.7.9) [2016-10-18 17:46:07.956365] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 17:46:07.956398] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-18755-2016/10/18-17:14:15:814349-engine-client-0-0-1 (version: 3.7.9) [2016-10-18 17:54:29.004335] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-24273-2016/10/18-17:54:26:89072-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:54:29.058125] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-24273-2016/10/18-17:54:26:89072-engine-client-0-0-0 [2016-10-18 17:54:29.058180] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-24273-2016/10/18-17:54:26:89072-engine-client-0-0-0 [2016-10-18 17:54:29.338755] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-23612-2016/10/18-17:43:29:644416-engine-client-0-0-1 [2016-10-18 17:54:29.338828] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-23612-2016/10/18-17:43:29:644416-engine-client-0-0-1 [2016-10-18 17:54:29.753467] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-24382-2016/10/18-17:54:26:838401-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:54:29.810203] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-24382-2016/10/18-17:54:26:838401-engine-client-0-0-0 [2016-10-18 17:54:29.810258] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-24382-2016/10/18-17:54:26:838401-engine-client-0-0-0 [2016-10-18 17:56:03.001730] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 17:56:03.001770] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 17:56:03.001730] and [2016-10-18 17:56:03.005809] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 17:56:03.001770] and [2016-10-18 17:56:03.005811] [2016-10-18 17:58:55.912748] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-24884-2016/10/18-17:58:52:995037-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 17:58:56.451602] W [MSGID: 113026] [posix.c:1415:posix_mkdir] 0-engine-posix: mkdir (/.shard): gfid (be318638-e8a0-4c6d-977d-7a937aa84806) isalready associated with directory (/rhgs/engine/engine/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/.shard). Hence,both directories will share same gfid and thiscan lead to inconsistencies. [2016-10-18 17:58:56.451649] E [MSGID: 113027] [posix.c:1522:posix_mkdir] 0-engine-posix: mkdir of /rhgs/engine/engine/.shard failed [File exists] [2016-10-18 17:58:56.602689] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 17:58:56.602758] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 289: WRITEV 0 (c59ddfd5-23ea-484a-9667-cee270f00f3d) ==> (Invalid argument) [Invalid argument] [2016-10-18 17:59:19.536318] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=c59ddfd5-23ea-484a-9667-cee270f00f3d, fd=0x7fb5f00cdf7c, offset=127488 size=512, buf=0x7fb5e4007000 [Invalid argument] [2016-10-18 17:59:19.536401] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 319: READV -2 (c59ddfd5-23ea-484a-9667-cee270f00f3d) ==> (Invalid argument) [Invalid argument] [2016-10-18 18:06:04.001718] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 18:06:04.001756] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 18:06:04.001718] and [2016-10-18 18:06:04.003815] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 18:06:04.001756] and [2016-10-18 18:06:04.003816] [2016-10-18 18:16:05.002014] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 18:16:05.002057] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 18:16:05.002014] and [2016-10-18 18:16:05.004157] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 18:16:05.002057] and [2016-10-18 18:16:05.004158] [2016-10-18 18:22:09.073100] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-24884-2016/10/18-17:58:52:995037-engine-client-0-0-0 [2016-10-18 18:22:09.073158] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-24884-2016/10/18-17:58:52:995037-engine-client-0-0-0 [2016-10-18 18:22:19.552046] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fb601c51dc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x7fb6032cb915] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7fb6032cb78b] ) 0-: received signum (15), shutting down [2016-10-18 19:59:49.154594] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 19:59:49.170313] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-18 19:59:49.331383] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 19:59:49.331430] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 19:59:49.331804] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-18 19:59:49.331818] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 19:59:49.331857] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 19:59:49.335759] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-18 19:59:49.335871] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-18 19:59:49.338395] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-18 19:59:49.338430] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-18 19:59:49.338449] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-18 19:59:49.338465] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-18 19:59:49.338480] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-18 19:59:49.352141] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-18 19:59:49.354070] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-18 19:59:49.354116] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-18 19:59:49.354148] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-18 19:59:49.354207] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-18 19:59:49.354287] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized [2016-10-18 19:59:49.361607] W [MSGID: 113036] [posix.c:2351:posix_rename] 0-engine-posix: found directory at /rhgs/engine/engine/.trashcan/ while expecting ENOENT [File exists] Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option rpc-auth-allow-insecure on 136: option transport-type tcp 137: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 138: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 139: option auth.addr./rhgs/engine/engine.allow * 140: option transport.tcp-user-timeout 30 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-18 19:59:49.920331] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 19:59:49.920381] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-3654-2016/10/18-19:52:20:258775-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 19:59:53.165468] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 19:59:53.165504] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-16268-2016/10/18-19:59:46:336904-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 19:59:53.177268] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 19:59:53.177294] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-2405-2016/10/18-19:59:49:155763-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 19:59:53.180208] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 19:59:53.180241] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 19:59:53.180934] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 19:59:53.180954] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 19:59:53.183896] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 19:59:53.183930] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 19:59:53.184289] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 19:59:53.184305] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 20:09:47.567816] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:09:47.618193] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:09:47.620374] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 20:09:47.620403] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 20:09:47.620566] I [MSGID: 121037] [changetimerecorder.c:1960:reconfigure] 0-engine-changetimerecorder: set [2016-10-18 20:09:47.620801] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 20:09:47.621985] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 20:09:47.622197] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 20:09:47.622214] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 20:09:47.622332] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.622355] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.622376] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.622505] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:09:47.753437] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.753465] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-4046-2016/10/18-20:09:47:613611-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:09:47.783335] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.783370] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-16516-2016/10/18-20:09:44:852873-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:09:47.810455] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:09:47.810487] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-16051-2016/10/18-20:09:47:210950-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:09:54.001648] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:09:54.001694] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 20:10:41.040357] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.074820] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.079490] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.082178] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:41.082699] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:41.083136] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:41.333715] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.368999] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.372510] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 20:10:41.372540] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 20:10:41.372701] I [MSGID: 121037] [changetimerecorder.c:1960:reconfigure] 0-engine-changetimerecorder: set [2016-10-18 20:10:41.372815] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 20:10:41.372849] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 20:10:41.373060] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 20:10:41.373080] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 20:10:41.373196] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373219] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373240] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373260] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373281] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373301] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.373431] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:41.615881] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.649874] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.653458] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 20:10:41.653488] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 20:10:41.653704] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 20:10:41.653723] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 20:10:41.653930] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 20:10:41.654050] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654074] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654094] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654114] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654134] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654154] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:41.654294] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:41.900988] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:41.653950] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 20:10:41.940494] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-4046-2016/10/18-20:09:47:613611-engine-client-0-0-0 [2016-10-18 20:10:41.940564] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-4046-2016/10/18-20:09:47:613611-engine-client-0-0-0 [2016-10-18 20:10:42.952953] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:42.955702] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:42.956276] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:43.000574] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-16516-2016/10/18-20:09:44:852873-engine-client-0-0-0 [2016-10-18 20:10:43.000646] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-16516-2016/10/18-20:09:44:852873-engine-client-0-0-0 [2016-10-18 20:10:43.008256] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt06.gamull.com-16051-2016/10/18-20:09:47:210950-engine-client-0-0-0 [2016-10-18 20:10:43.008287] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt06.gamull.com-16051-2016/10/18-20:09:47:210950-engine-client-0-0-0 [2016-10-18 20:10:43.053945] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:43.053969] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-4186-2016/10/18-20:10:42:947899-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:10:44.098249] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.098281] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-16576-2016/10/18-20:10:41:192378-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:10:44.144649] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.144683] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-16111-2016/10/18-20:10:43:552769-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 20:10:44.225389] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.264979] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.269839] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.271986] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.272475] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.272901] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.531048] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.565438] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.568707] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 20:10:44.568737] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' The message "I [MSGID: 121037] [changetimerecorder.c:1960:reconfigure] 0-engine-changetimerecorder: set" repeated 2 times between [2016-10-18 20:10:41.372701] and [2016-10-18 20:10:44.568904] [2016-10-18 20:10:44.569002] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 20:10:44.569058] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 20:10:44.569263] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 20:10:44.569265] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 20:10:44.569392] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569416] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569437] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569458] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569479] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569498] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:44.569630] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.819673] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.857887] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.863212] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:44.865379] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.865874] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:44.866304] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:45.128735] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:45.165997] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-4186-2016/10/18-20:10:42:947899-engine-client-0-0-0 [2016-10-18 20:10:45.166058] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-4186-2016/10/18-20:10:42:947899-engine-client-0-0-0 [2016-10-18 20:10:46.169909] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:46.172207] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:46.172765] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:46.217740] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-16576-2016/10/18-20:10:41:192378-engine-client-0-0-0 [2016-10-18 20:10:46.217779] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-16576-2016/10/18-20:10:41:192378-engine-client-0-0-0 [2016-10-18 20:10:46.221642] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt06.gamull.com-16111-2016/10/18-20:10:43:552769-engine-client-0-0-0 [2016-10-18 20:10:46.221702] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt06.gamull.com-16111-2016/10/18-20:10:43:552769-engine-client-0-0-0 [2016-10-18 20:10:47.423623] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:47.457811] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:47.460345] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:47.460664] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:47.704212] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:47.737213] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:47.739668] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:47.740001] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:47.990877] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.024379] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.026610] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.026989] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.280623] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.314672] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.317083] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.317458] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.571099] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.603681] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.606136] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.606498] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:48.859010] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.892195] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:48.895865] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 20:10:48.895905] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 20:10:48.896059] I [MSGID: 121037] [changetimerecorder.c:1960:reconfigure] 0-engine-changetimerecorder: set [2016-10-18 20:10:48.896178] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 20:10:48.896215] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 20:10:48.896403] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 20:10:48.896429] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 20:10:48.896544] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:48.896567] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:48.896588] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 20:10:48.896725] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:50.795165] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:50.829648] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 20:10:50.832132] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 20:10:50.832502] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:09:54.001648] and [2016-10-18 20:09:54.003755] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:09:54.001694] and [2016-10-18 20:09:54.003756] [2016-10-18 20:19:55.001837] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:19:55.001917] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:19:55.001837] and [2016-10-18 20:19:55.004165] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:19:55.001917] and [2016-10-18 20:19:55.004167] [2016-10-18 20:29:56.001748] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:29:56.001802] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:29:56.001748] and [2016-10-18 20:29:56.004308] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:29:56.001802] and [2016-10-18 20:29:56.004310] [2016-10-18 20:39:57.001670] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:39:57.001722] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:39:57.001670] and [2016-10-18 20:39:57.003750] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:39:57.001722] and [2016-10-18 20:39:57.003752] [2016-10-18 20:49:58.001686] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:49:58.001742] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:49:58.001686] and [2016-10-18 20:49:58.003781] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:49:58.001742] and [2016-10-18 20:49:58.003782] [2016-10-18 20:59:59.001675] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 20:59:59.001729] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 20:59:59.001675] and [2016-10-18 20:59:59.003777] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 20:59:59.001729] and [2016-10-18 20:59:59.003778] [2016-10-18 21:10:00.001692] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:10:00.001746] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 21:10:00.001692] and [2016-10-18 21:10:00.003792] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 21:10:00.001746] and [2016-10-18 21:10:00.003793] [2016-10-18 21:20:01.001711] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:20:01.001764] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 21:20:01.001711] and [2016-10-18 21:20:01.003804] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 21:20:01.001764] and [2016-10-18 21:20:01.003805] [2016-10-18 21:30:02.001841] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:30:02.001902] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 21:30:02.001841] and [2016-10-18 21:30:02.004149] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 21:30:02.001902] and [2016-10-18 21:30:02.004151] [2016-10-18 21:40:03.001816] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:40:03.001864] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 21:40:03.001816] and [2016-10-18 21:40:03.004100] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 21:40:03.001864] and [2016-10-18 21:40:03.004101] [2016-10-18 21:50:04.001837] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:50:04.001883] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 21:50:04.001837] and [2016-10-18 21:50:04.003934] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 21:50:04.001883] and [2016-10-18 21:50:04.003936] [2016-10-18 21:52:20.163215] W [socket.c:701:__socket_rwv] 0-glusterfs: readv on 10.1.200.84:24007 failed (No data available) [2016-10-18 21:52:31.563055] E [socket.c:2395:socket_connect_finish] 0-glusterfs: connection to 10.1.200.84:24007 failed (Connection refused) [2016-10-18 21:52:32.654340] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f2c00194dc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x7f2c0180e915] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f2c0180e78b] ) 0-: received signum (15), shutting down [2016-10-18 21:52:32.664012] E [MSGID: 101075] [common-utils.c:306:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2016-10-18 21:52:32.664043] E [name.c:261:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host node1 [2016-10-18 21:52:32.664214] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x26 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2016-10-18 21:54:31.838095] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 21:54:31.855684] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-18 21:54:31.913321] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 21:54:31.913366] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 21:54:31.913734] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-18 21:54:31.913747] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 21:54:31.913786] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 21:54:31.915039] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-18 21:54:31.915141] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-18 21:54:31.918487] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-18 21:54:31.918535] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-18 21:54:31.918553] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-18 21:54:31.918569] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-18 21:54:31.918583] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-18 21:54:31.940423] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-18 21:54:31.941586] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-18 21:54:31.941628] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-18 21:54:31.941660] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-18 21:54:31.941719] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-18 21:54:31.941804] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized [2016-10-18 21:54:31.949781] W [MSGID: 113036] [posix.c:2351:posix_rename] 0-engine-posix: found directory at /rhgs/engine/engine/.trashcan/ while expecting ENOENT [File exists] Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option rpc-auth-allow-insecure on 136: option transport-type tcp 137: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 138: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 139: option auth.addr./rhgs/engine/engine.allow * 140: option transport.tcp-user-timeout 30 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-18 21:54:33.660850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 21:54:33.660900] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-3654-2016/10/18-19:52:20:258775-engine-client-0-0-1 (version: 3.7.9) [2016-10-18 21:54:33.959916] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 21:54:33.959962] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-16268-2016/10/18-19:59:46:336904-engine-client-0-0-1 (version: 3.7.9) [2016-10-18 21:54:35.863069] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 21:54:35.863098] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-2410-2016/10/18-21:54:31:842535-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 21:54:35.865585] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:54:35.865621] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 21:54:35.866267] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:54:35.866286] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 21:54:35.869075] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:54:35.869102] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 21:54:35.869359] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 21:54:35.869374] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 22:04:36.001900] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:04:36.001953] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 22:04:56.681450] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 22:04:56.716034] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 22:04:56.720110] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 22:04:56.720509] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-18 22:05:11.726510] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 22:05:11.762290] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2016-10-18 22:05:11.767533] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 22:05:11.767564] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 22:05:11.768541] I [MSGID: 121037] [changetimerecorder.c:1960:reconfigure] 0-engine-changetimerecorder: set [2016-10-18 22:05:11.768712] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 25000 [2016-10-18 22:05:11.769852] I [MSGID: 0] [gfdb_sqlite3.c:1356:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 12500 [2016-10-18 22:05:11.770083] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 22:05:11.770101] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 22:05:11.770250] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 22:05:11.770274] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 22:05:11.770294] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 22:05:11.770547] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:04:36.001900] and [2016-10-18 22:04:36.004234] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:04:36.001953] and [2016-10-18 22:04:36.004235] [2016-10-18 22:14:37.001903] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:14:37.001957] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:14:37.001903] and [2016-10-18 22:14:37.004219] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:14:37.001957] and [2016-10-18 22:14:37.004221] [2016-10-18 22:24:38.001706] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:24:38.001762] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:24:38.001706] and [2016-10-18 22:24:38.003877] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:24:38.001762] and [2016-10-18 22:24:38.003878] [2016-10-18 22:34:39.001743] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:34:39.001803] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:34:39.001743] and [2016-10-18 22:34:39.003861] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:34:39.001803] and [2016-10-18 22:34:39.003862] [2016-10-18 22:44:40.001702] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:44:40.001757] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:44:40.001702] and [2016-10-18 22:44:40.003845] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:44:40.001757] and [2016-10-18 22:44:40.003846] [2016-10-18 22:54:41.001726] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 22:54:41.001783] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 22:54:41.001726] and [2016-10-18 22:54:41.003872] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 22:54:41.001783] and [2016-10-18 22:54:41.003873] [2016-10-18 23:04:42.001865] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:04:42.001943] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:05:31.035568] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:05:31.035613] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5369-2016/10/18-23:05:30:958583-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:05:31.103914] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5369-2016/10/18-23:05:30:958583-engine-client-0-0-0 [2016-10-18 23:05:31.104006] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5369-2016/10/18-23:05:30:958583-engine-client-0-0-0 [2016-10-18 23:05:31.413493] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:05:31.413525] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5475-2016/10/18-23:05:31:365628-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:05:31.805836] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5475-2016/10/18-23:05:31:365628-engine-client-0-0-0 [2016-10-18 23:05:31.805898] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5475-2016/10/18-23:05:31:365628-engine-client-0-0-0 [2016-10-18 23:05:32.265448] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:05:32.265480] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5584-2016/10/18-23:05:32:218810-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:05:32.332209] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5584-2016/10/18-23:05:32:218810-engine-client-0-0-0 [2016-10-18 23:05:32.332296] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5584-2016/10/18-23:05:32:218810-engine-client-0-0-0 The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 23:04:42.001865] and [2016-10-18 23:04:42.004191] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 23:04:42.001943] and [2016-10-18 23:04:42.004192] [2016-10-18 23:09:49.830291] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:09:49.830322] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-6132-2016/10/18-23:09:49:784115-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:09:50.610228] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 23:09:50.610299] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 310: WRITEV 0 (526684a1-b37b-4de0-8669-8ddc14d8b268) ==> (Invalid argument) [Invalid argument] [2016-10-18 23:10:13.322161] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=526684a1-b37b-4de0-8669-8ddc14d8b268, fd=0x7f34340ccf9c, offset=127488 size=512, buf=0x7f33fc003000 [Invalid argument] [2016-10-18 23:10:13.322224] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 345: READV -2 (526684a1-b37b-4de0-8669-8ddc14d8b268) ==> (Invalid argument) [Invalid argument] [2016-10-18 23:11:21.665854] W [trash.c:1925:trash_rmdir] 0-engine-trash: rmdir issued on /.trashcan/, which is not permitted [2016-10-18 23:11:21.665913] I [MSGID: 115056] [server-rpc-fops.c:461:server_rmdir_cbk] 0-engine-server: 413: RMDIR /.trashcan/internal_op (00000000-0000-0000-0000-000000000005/internal_op) ==> (Operation not permitted) [Operation not permitted] [2016-10-18 23:13:17.818001] W [trash.c:1925:trash_rmdir] 0-engine-trash: rmdir issued on /.trashcan/, which is not permitted [2016-10-18 23:13:17.818066] I [MSGID: 115056] [server-rpc-fops.c:461:server_rmdir_cbk] 0-engine-server: 449: RMDIR /.trashcan/internal_op (00000000-0000-0000-0000-000000000005/internal_op) ==> (Operation not permitted) [Operation not permitted] [2016-10-18 23:13:45.889422] W [trash.c:1925:trash_rmdir] 0-engine-trash: rmdir issued on /.trashcan/, which is not permitted [2016-10-18 23:13:45.889490] I [MSGID: 115056] [server-rpc-fops.c:461:server_rmdir_cbk] 0-engine-server: 484: RMDIR /.trashcan/internal_op (00000000-0000-0000-0000-000000000005/internal_op) ==> (Operation not permitted) [Operation not permitted] [2016-10-18 23:14:06.229114] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-6132-2016/10/18-23:09:49:784115-engine-client-0-0-0 [2016-10-18 23:14:06.229226] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-6132-2016/10/18-23:09:49:784115-engine-client-0-0-0 [2016-10-18 23:14:07.293465] W [socket.c:701:__socket_rwv] 0-glusterfs: readv on 10.1.200.84:24007 failed (No data available) [2016-10-18 23:14:19.646864] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3445d1fdc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x7f3447399915] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f344739978b] ) 0-: received signum (15), shutting down [2016-10-18 23:14:19.646965] I [socket.c:3434:socket_submit_request] 0-glusterfs: not connected (priv->connected = 0) [2016-10-18 23:14:19.646979] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x7 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2016-10-18 23:16:18.184424] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 23:16:18.202336] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-18 23:16:18.202946] E [MSGID: 100018] [glusterfsd.c:1973:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid lock failed [Resource temporarily unavailable] [2016-10-18 23:16:18.203935] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-18 23:16:18.269171] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-18 23:16:18.269218] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-18 23:16:18.269676] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-18 23:16:18.269669] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-18 23:16:18.269717] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-18 23:16:18.270998] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-18 23:16:18.271091] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-18 23:16:18.274568] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-18 23:16:18.274612] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-18 23:16:18.274630] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-18 23:16:18.274647] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-18 23:16:18.274661] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-18 23:16:18.296508] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-18 23:16:18.297706] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-18 23:16:18.297749] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-18 23:16:18.297790] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-18 23:16:18.297854] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-18 23:16:18.297936] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized [2016-10-18 23:16:18.305489] W [MSGID: 113036] [posix.c:2351:posix_rename] 0-engine-posix: found directory at /rhgs/engine/engine/.trashcan/ while expecting ENOENT [File exists] Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option transport-type tcp 136: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 137: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 138: option auth.addr./rhgs/engine/engine.allow * 139: option transport.tcp-user-timeout 30 140: option rpc-auth-allow-insecure on 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-18 23:16:20.426871] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:16:20.426924] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-3654-2016/10/18-19:52:20:258775-engine-client-0-0-2 (version: 3.7.9) [2016-10-18 23:16:20.765363] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:16:20.765417] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-16268-2016/10/18-19:59:46:336904-engine-client-0-0-2 (version: 3.7.9) [2016-10-18 23:16:22.206122] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:16:22.206150] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-2392-2016/10/18-23:16:18:184887-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:16:22.209748] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:16:22.209812] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:16:22.210363] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:16:22.210379] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:16:22.213340] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:16:22.213363] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:16:22.213684] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:16:22.213700] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:16:26.292058] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:16:26.292109] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-19851-2016/10/18-23:16:22:702848-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:16:26.332000] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-19851-2016/10/18-23:16:22:702848-engine-client-0-0-0 [2016-10-18 23:16:26.332072] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-19851-2016/10/18-23:16:22:702848-engine-client-0-0-0 [2016-10-18 23:16:30.044460] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:16:30.044491] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-19870-2016/10/18-23:16:26:454495-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:16:30.081370] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-19870-2016/10/18-23:16:26:454495-engine-client-0-0-0 [2016-10-18 23:16:30.081405] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-19870-2016/10/18-23:16:26:454495-engine-client-0-0-0 [2016-10-18 23:18:36.822566] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:18:36.822623] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-19922-2016/10/18-23:18:33:229891-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:18:36.859851] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-19922-2016/10/18-23:18:33:229891-engine-client-0-0-0 [2016-10-18 23:18:36.859915] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-19922-2016/10/18-23:18:33:229891-engine-client-0-0-0 [2016-10-18 23:23:12.982064] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:23:12.982106] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-20134-2016/10/18-23:23:09:355146-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:23:13.037696] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-20134-2016/10/18-23:23:09:355146-engine-client-0-0-0 [2016-10-18 23:23:13.037766] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-20134-2016/10/18-23:23:09:355146-engine-client-0-0-0 [2016-10-18 23:23:13.278080] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:23:13.278152] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-20239-2016/10/18-23:23:09:682527-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:23:13.586237] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-20239-2016/10/18-23:23:09:682527-engine-client-0-0-0 [2016-10-18 23:23:13.586280] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-20239-2016/10/18-23:23:09:682527-engine-client-0-0-0 [2016-10-18 23:23:14.002843] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:23:14.002872] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-20348-2016/10/18-23:23:10:405418-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:23:14.057056] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-20348-2016/10/18-23:23:10:405418-engine-client-0-0-0 [2016-10-18 23:23:14.057093] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-20348-2016/10/18-23:23:10:405418-engine-client-0-0-0 [2016-10-18 23:26:23.001739] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:26:23.001803] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 23:26:23.001739] and [2016-10-18 23:26:23.004152] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 23:26:23.001803] and [2016-10-18 23:26:23.004153] [2016-10-18 23:36:24.001712] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:36:24.001769] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 23:36:24.001712] and [2016-10-18 23:36:24.003988] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 23:36:24.001769] and [2016-10-18 23:36:24.003989] [2016-10-18 23:39:36.198631] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:39:36.198687] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-21035-2016/10/18-23:39:32:591813-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:39:36.736326] W [MSGID: 113026] [posix.c:1415:posix_mkdir] 0-engine-posix: mkdir (/.shard): gfid (be318638-e8a0-4c6d-977d-7a937aa84806) isalready associated with directory (/rhgs/engine/engine/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/.shard). Hence,both directories will share same gfid and thiscan lead to inconsistencies. [2016-10-18 23:39:36.736370] E [MSGID: 113027] [posix.c:1522:posix_mkdir] 0-engine-posix: mkdir of /rhgs/engine/engine/.shard failed [File exists] [2016-10-18 23:39:36.855607] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 23:39:36.855661] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 291: WRITEV 0 (922432f3-787d-4b1d-83a3-cb658e677497) ==> (Invalid argument) [Invalid argument] [2016-10-18 23:39:59.717228] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=922432f3-787d-4b1d-83a3-cb658e677497, fd=0x7f8a380cd02c, offset=127488 size=512, buf=0x7f8a2c007000 [Invalid argument] [2016-10-18 23:39:59.717303] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 321: READV -2 (922432f3-787d-4b1d-83a3-cb658e677497) ==> (Invalid argument) [Invalid argument] [2016-10-18 23:42:46.233394] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:42:46.233448] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-21298-2016/10/18-23:42:42:626221-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:42:46.285629] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-21298-2016/10/18-23:42:42:626221-engine-client-0-0-0 [2016-10-18 23:42:46.285698] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-21298-2016/10/18-23:42:42:626221-engine-client-0-0-0 [2016-10-18 23:43:11.361315] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:43:11.361359] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-21368-2016/10/18-23:43:07:754707-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:43:11.398063] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-21368-2016/10/18-23:43:07:754707-engine-client-0-0-0 [2016-10-18 23:43:11.398125] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-21368-2016/10/18-23:43:07:754707-engine-client-0-0-0 [2016-10-18 23:43:17.762023] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:43:17.762071] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-21386-2016/10/18-23:43:14:158428-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:43:17.801105] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-21386-2016/10/18-23:43:14:158428-engine-client-0-0-0 [2016-10-18 23:43:17.801141] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-21386-2016/10/18-23:43:14:158428-engine-client-0-0-0 [2016-10-18 23:44:17.412249] W [trash.c:1925:trash_rmdir] 0-engine-trash: rmdir issued on /.trashcan/, which is not permitted [2016-10-18 23:44:17.412326] I [MSGID: 115056] [server-rpc-fops.c:461:server_rmdir_cbk] 0-engine-server: 383: RMDIR /.trashcan/internal_op (00000000-0000-0000-0000-000000000005/internal_op) ==> (Operation not permitted) [Operation not permitted] [2016-10-18 23:44:34.777582] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-21035-2016/10/18-23:39:32:591813-engine-client-0-0-0 [2016-10-18 23:44:34.777653] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-21035-2016/10/18-23:39:32:591813-engine-client-0-0-0 [2016-10-18 23:44:53.489841] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-16268-2016/10/18-19:59:46:336904-engine-client-0-0-2 [2016-10-18 23:44:53.489929] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-16268-2016/10/18-19:59:46:336904-engine-client-0-0-2 [2016-10-18 23:46:25.001903] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:46:25.001959] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-18 23:46:39.387034] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:46:39.387064] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-2278-2016/10/18-23:46:35:499560-engine-client-0-0-0 (version: 3.7.9) The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 7 times between [2016-10-18 23:46:25.001903] and [2016-10-18 23:46:40.442978] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 7 times between [2016-10-18 23:46:25.001959] and [2016-10-18 23:46:40.442979] [2016-10-18 23:52:40.700505] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:52:40.700554] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-3990-2016/10/18-23:52:36:847849-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:52:40.755869] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-3990-2016/10/18-23:52:36:847849-engine-client-0-0-0 [2016-10-18 23:52:40.755933] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-3990-2016/10/18-23:52:36:847849-engine-client-0-0-0 [2016-10-18 23:52:41.013737] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:52:41.013790] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-4100-2016/10/18-23:52:37:194125-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:52:41.354225] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-4100-2016/10/18-23:52:37:194125-engine-client-0-0-0 [2016-10-18 23:52:41.354267] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-4100-2016/10/18-23:52:37:194125-engine-client-0-0-0 [2016-10-18 23:52:41.770004] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:52:41.770035] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-4209-2016/10/18-23:52:37:949539-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:52:41.822672] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-4209-2016/10/18-23:52:37:949539-engine-client-0-0-0 [2016-10-18 23:52:41.822709] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-4209-2016/10/18-23:52:37:949539-engine-client-0-0-0 [2016-10-18 23:56:39.625264] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:56:39.625309] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-4718-2016/10/18-23:56:35:799935-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:56:40.297395] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-18 23:56:40.297451] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 289: WRITEV 0 (bc8db1ca-7e4a-406f-97c9-6e35a2a3012b) ==> (Invalid argument) [Invalid argument] [2016-10-18 23:56:41.001693] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-18 23:56:41.001737] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-18 23:56:41.001737] and [2016-10-18 23:56:41.003977] [2016-10-18 23:57:03.181200] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=bc8db1ca-7e4a-406f-97c9-6e35a2a3012b, fd=0x7f8a380cd098, offset=127488 size=512, buf=0x7f8a2c00b000 [Invalid argument] [2016-10-18 23:57:03.181293] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 319: READV -2 (bc8db1ca-7e4a-406f-97c9-6e35a2a3012b) ==> (Invalid argument) [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-18 23:56:41.001693] and [2016-10-18 23:56:41.003976] [2016-10-18 23:58:57.823819] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5327-2016/10/18-23:58:53:999919-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:58:57.877829] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5327-2016/10/18-23:58:53:999919-engine-client-0-0-0 [2016-10-18 23:58:57.877862] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5327-2016/10/18-23:58:53:999919-engine-client-0-0-0 [2016-10-18 23:58:58.123909] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5430-2016/10/18-23:58:54:299143-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:58:58.772066] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5430-2016/10/18-23:58:54:299143-engine-client-0-0-0 [2016-10-18 23:58:58.772112] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5430-2016/10/18-23:58:54:299143-engine-client-0-0-0 [2016-10-18 23:58:59.239246] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5587-2016/10/18-23:58:55:417715-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:58:59.294460] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5587-2016/10/18-23:58:55:417715-engine-client-0-0-0 [2016-10-18 23:58:59.294489] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5587-2016/10/18-23:58:55:417715-engine-client-0-0-0 [2016-10-18 23:59:25.879570] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-18 23:59:25.879606] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5706-2016/10/18-23:59:22:55190-engine-client-0-0-0 (version: 3.7.9) [2016-10-18 23:59:25.930224] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5706-2016/10/18-23:59:22:55190-engine-client-0-0-0 [2016-10-18 23:59:25.930255] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5706-2016/10/18-23:59:22:55190-engine-client-0-0-0 [2016-10-19 00:00:03.909572] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5840-2016/10/19-00:00:00:87603-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:00:03.962982] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5840-2016/10/19-00:00:00:87603-engine-client-0-0-0 [2016-10-19 00:00:03.963022] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5840-2016/10/19-00:00:00:87603-engine-client-0-0-0 [2016-10-19 00:00:04.214200] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-5943-2016/10/19-00:00:00:390315-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:00:04.760470] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-5943-2016/10/19-00:00:00:390315-engine-client-0-0-0 [2016-10-19 00:00:04.760503] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-5943-2016/10/19-00:00:00:390315-engine-client-0-0-0 [2016-10-19 00:00:05.222986] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-6078-2016/10/19-00:00:01:400586-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:00:05.279361] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-6078-2016/10/19-00:00:01:400586-engine-client-0-0-0 [2016-10-19 00:00:05.279394] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-6078-2016/10/19-00:00:01:400586-engine-client-0-0-0 [2016-10-19 00:03:36.525753] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-6602-2016/10/19-00:03:32:700704-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:04:38.373234] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-6811-2016/10/19-00:04:34:545908-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:04:38.426081] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-6811-2016/10/19-00:04:34:545908-engine-client-0-0-0 [2016-10-19 00:04:38.426112] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-6811-2016/10/19-00:04:34:545908-engine-client-0-0-0 [2016-10-19 00:04:39.072649] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-6602-2016/10/19-00:03:32:700704-engine-client-0-0-0 [2016-10-19 00:04:39.072683] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-6602-2016/10/19-00:03:32:700704-engine-client-0-0-0 [2016-10-19 00:04:39.535797] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-6971-2016/10/19-00:04:35:710999-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:04:39.591549] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-6971-2016/10/19-00:04:35:710999-engine-client-0-0-0 [2016-10-19 00:04:39.591579] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-6971-2016/10/19-00:04:35:710999-engine-client-0-0-0 [2016-10-19 00:06:42.001768] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:06:42.001824] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-19 00:07:15.107080] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-7468-2016/10/19-00:07:11:277049-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:07:32.516540] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-4718-2016/10/18-23:56:35:799935-engine-client-0-0-0 [2016-10-19 00:07:32.516618] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-4718-2016/10/18-23:56:35:799935-engine-client-0-0-0 [2016-10-19 00:07:32.523203] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-7468-2016/10/19-00:07:11:277049-engine-client-0-0-0 [2016-10-19 00:07:32.523231] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-7468-2016/10/19-00:07:11:277049-engine-client-0-0-0 [2016-10-19 00:07:55.313835] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt05.gamull.com-2278-2016/10/18-23:46:35:499560-engine-client-0-0-0 [2016-10-19 00:07:55.313907] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt05.gamull.com-2278-2016/10/18-23:46:35:499560-engine-client-0-0-0 The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:06:42.001768] and [2016-10-19 00:06:42.004190] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:06:42.001824] and [2016-10-19 00:06:42.004191] [2016-10-19 00:09:33.110650] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:09:33.110680] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-2405-2016/10/19-00:09:28:738338-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:09:35.602940] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:09:35.602969] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:09:35.602940] and [2016-10-19 00:09:35.605343] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:09:35.602969] and [2016-10-19 00:09:35.605345] [2016-10-19 00:19:36.001901] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:19:36.001944] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:19:36.001901] and [2016-10-19 00:19:36.004298] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:19:36.001944] and [2016-10-19 00:19:36.004299] [2016-10-19 00:23:25.806929] W [socket.c:701:__socket_rwv] 0-glusterfs: readv on 10.1.200.84:24007 failed (No data available) [2016-10-19 00:23:36.431104] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-10-19 00:23:36.454915] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-2392-2016/10/18-23:16:18:184887-engine-client-0-0-0 [2016-10-19 00:23:36.454976] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-2392-2016/10/18-23:16:18:184887-engine-client-0-0-0 [2016-10-19 00:23:37.527929] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:23:37.527969] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5379-2016/10/19-00:23:37:458816-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:23:37.531222] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:23:37.531258] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:23:37.531222] and [2016-10-19 00:23:37.533549] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:23:37.531258] and [2016-10-19 00:23:37.533550] [2016-10-19 00:30:21.837913] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:30:21.837945] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5772-2016/10/19-00:30:21:760082-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:30:21.906864] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5772-2016/10/19-00:30:21:760082-engine-client-0-0-0 [2016-10-19 00:30:21.906928] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5772-2016/10/19-00:30:21:760082-engine-client-0-0-0 [2016-10-19 00:30:22.218602] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:30:22.218657] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5877-2016/10/19-00:30:22:171440-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:30:22.608590] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5877-2016/10/19-00:30:22:171440-engine-client-0-0-0 [2016-10-19 00:30:22.608669] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5877-2016/10/19-00:30:22:171440-engine-client-0-0-0 [2016-10-19 00:30:23.066269] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:30:23.066299] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5986-2016/10/19-00:30:23:21194-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:30:23.130853] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-5986-2016/10/19-00:30:23:21194-engine-client-0-0-0 [2016-10-19 00:30:23.130891] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-5986-2016/10/19-00:30:23:21194-engine-client-0-0-0 [2016-10-19 00:33:38.001950] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:33:38.002005] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:33:38.001950] and [2016-10-19 00:33:38.004382] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:33:38.002005] and [2016-10-19 00:33:38.004383] [2016-10-19 00:34:44.032149] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:34:44.032198] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-6987-2016/10/19-00:34:43:987563-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:34:44.096803] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-6987-2016/10/19-00:34:43:987563-engine-client-0-0-0 [2016-10-19 00:34:44.096874] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-6987-2016/10/19-00:34:43:987563-engine-client-0-0-0 [2016-10-19 00:35:53.476113] E [MSGID: 101075] [common-utils.c:306:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2016-10-19 00:35:53.476165] E [name.c:261:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host node1 [2016-10-19 00:35:53.710692] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f8a49c41dc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x7f8a4b2bb915] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f8a4b2bb78b] ) 0-: received signum (15), shutting down [2016-10-19 00:35:53.714866] E [MSGID: 101075] [common-utils.c:306:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2016-10-19 00:35:53.714888] E [name.c:261:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host node1 [2016-10-19 00:35:53.714944] I [socket.c:3434:socket_submit_request] 0-glusterfs: not connected (priv->connected = -1) [2016-10-19 00:35:53.714956] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x5 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2016-10-19 00:37:51.327027] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-19 00:37:51.343186] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.9 (args: /usr/sbin/glusterfsd -s node1 --volfile-id engine.node1.rhgs-engine-engine -p /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid -S /var/run/gluster/57d6e4671a8795467a8a8bcfc1d409c5.socket --brick-name /rhgs/engine/engine -l /var/log/glusterfs/bricks/rhgs-engine-engine.log --xlator-option *-posix.glusterd-uuid=06a9f810-8e15-474a-8384-5fca1bd78507 --brick-port 49152 --xlator-option engine-server.listen-port=49152) [2016-10-19 00:37:51.343909] E [MSGID: 100018] [glusterfsd.c:1973:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/lib/glusterd/vols/engine/run/node1-rhgs-engine-engine.pid lock failed [Resource temporarily unavailable] [2016-10-19 00:37:51.346153] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-19 00:37:51.413494] I [graph.c:269:gf_add_cmdline_options] 0-engine-server: adding option 'listen-port' for volume 'engine-server' with value '49152' [2016-10-19 00:37:51.413551] I [graph.c:269:gf_add_cmdline_options] 0-engine-posix: adding option 'glusterd-uuid' for volume 'engine-posix' with value '06a9f810-8e15-474a-8384-5fca1bd78507' [2016-10-19 00:37:51.413954] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-10-19 00:37:51.413992] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login./rhgs/engine/engine.allow [2016-10-19 00:37:51.414014] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/rhgs/engine/engine: skip format check for non-addr auth option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password [2016-10-19 00:37:51.415400] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2016-10-19 00:37:51.415495] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-engine-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2016-10-19 00:37:51.419041] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2016-10-19 00:37:51.419097] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2016-10-19 00:37:51.419116] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2016-10-19 00:37:51.419134] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off [2016-10-19 00:37:51.419149] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-engine-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2016-10-19 00:37:51.433276] I [trash.c:2369:init] 0-engine-trash: no option specified for 'eliminate', using NULL [2016-10-19 00:37:51.434555] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-10-19 00:37:51.434597] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-unix' is not recognized [2016-10-19 00:37:51.434629] W [graph.c:357:_log_if_unknown_option] 0-engine-server: option 'rpc-auth.auth-null' is not recognized [2016-10-19 00:37:51.434687] W [graph.c:357:_log_if_unknown_option] 0-engine-quota: option 'timeout' is not recognized [2016-10-19 00:37:51.434789] W [graph.c:357:_log_if_unknown_option] 0-engine-trash: option 'brick-path' is not recognized [2016-10-19 00:37:51.442147] W [MSGID: 113036] [posix.c:2351:posix_rename] 0-engine-posix: found directory at /rhgs/engine/engine/.trashcan/ while expecting ENOENT [File exists] Final graph: +------------------------------------------------------------------------------+ 1: volume engine-posix 2: type storage/posix 3: option glusterd-uuid 06a9f810-8e15-474a-8384-5fca1bd78507 4: option directory /rhgs/engine/engine 5: option volume-id 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 6: option brick-uid 36 7: option brick-gid 36 8: end-volume 9: 10: volume engine-trash 11: type features/trash 12: option trash-dir .trashcan 13: option brick-path /rhgs/engine/engine 14: option trash-internal-op off 15: subvolumes engine-posix 16: end-volume 17: 18: volume engine-changetimerecorder 19: type features/changetimerecorder 20: option db-type sqlite3 21: option hot-brick off 22: option db-name engine.db 23: option db-path /rhgs/engine/engine/.glusterfs/ 24: option record-exit off 25: option ctr_link_consistency off 26: option ctr_lookupheal_link_timeout 300 27: option ctr_lookupheal_inode_timeout 300 28: option record-entry on 29: option ctr-enabled off 30: option record-counters off 31: option ctr-record-metadata-heat off 32: option sql-db-cachesize 12500 33: option sql-db-wal-autocheckpoint 25000 34: subvolumes engine-trash 35: end-volume 36: 37: volume engine-changelog 38: type features/changelog 39: option changelog-brick /rhgs/engine/engine 40: option changelog-dir /rhgs/engine/engine/.glusterfs/changelogs 41: option changelog-barrier-timeout 120 42: subvolumes engine-changetimerecorder 43: end-volume 44: 45: volume engine-bitrot-stub 46: type features/bitrot-stub 47: option export /rhgs/engine/engine 48: subvolumes engine-changelog 49: end-volume 50: 51: volume engine-access-control 52: type features/access-control 53: subvolumes engine-bitrot-stub 54: end-volume 55: 56: volume engine-locks 57: type features/locks 58: subvolumes engine-access-control 59: end-volume 60: 61: volume engine-upcall 62: type features/upcall 63: option cache-invalidation off 64: subvolumes engine-locks 65: end-volume 66: 67: volume engine-io-threads 68: type performance/io-threads 69: option low-prio-threads 32 70: subvolumes engine-upcall 71: end-volume 72: 73: volume engine-marker 74: type features/marker 75: option volume-uuid 12c2ac8c-112a-450e-af1f-e8ea3fc477ec 76: option timestamp-file /var/lib/glusterd/vols/engine/marker.tstamp 77: option quota-version 0 78: option xtime off 79: option gsync-force-xtime off 80: option quota off 81: option inode-quota off 82: subvolumes engine-io-threads 83: end-volume 84: 85: volume engine-barrier 86: type features/barrier 87: option barrier disable 88: option barrier-timeout 120 89: subvolumes engine-marker 90: end-volume 91: 92: volume engine-index 93: type features/index 94: option index-base /rhgs/engine/engine/.glusterfs/indices 95: option xattrop-dirty-watchlist trusted.afr.dirty 96: option xattrop-pending-watchlist trusted.afr.engine- 97: subvolumes engine-barrier 98: end-volume 99: 100: volume engine-quota 101: type features/quota 102: option volume-uuid engine 103: option server-quota off 104: option timeout 0 105: option deem-statfs off 106: subvolumes engine-index 107: end-volume 108: 109: volume engine-worm 110: type features/worm 111: option worm off 112: subvolumes engine-quota 113: end-volume 114: 115: volume engine-read-only 116: type features/read-only 117: option read-only off 118: subvolumes engine-worm 119: end-volume 120: 121: volume /rhgs/engine/engine 122: type debug/io-stats 123: option log-level INFO 124: option latency-measurement off 125: option count-fop-hits off 126: subvolumes engine-read-only 127: end-volume 128: 129: volume engine-server 130: type protocol/server 131: option transport.socket.listen-port 49152 132: option rpc-auth.auth-glusterfs on 133: option rpc-auth.auth-unix on 134: option rpc-auth.auth-null on 135: option transport-type tcp 136: option auth.login./rhgs/engine/engine.allow e4b1452d-958b-4224-80d1-3624127e29b6 137: option auth.login.e4b1452d-958b-4224-80d1-3624127e29b6.password 3697b3a5-2740-4d21-8e1d-6d906fd3fac8 138: option auth.addr./rhgs/engine/engine.allow * 139: option transport.tcp-user-timeout 30 140: option rpc-auth-allow-insecure on 141: subvolumes /rhgs/engine/engine 142: end-volume 143: +------------------------------------------------------------------------------+ [2016-10-19 00:37:52.382323] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:37:52.382362] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt05.gamull.com-2405-2016/10/19-00:09:28:738338-engine-client-0-0-1 (version: 3.7.9) [2016-10-19 00:37:54.051389] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:37:54.051434] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt06.gamull.com-3654-2016/10/18-19:52:20:258775-engine-client-0-0-3 (version: 3.7.9) [2016-10-19 00:37:55.352107] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:37:55.352136] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-2394-2016/10/19-00:37:51:327673-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:37:55.355211] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:37:55.355238] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-19 00:37:55.355918] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:37:55.355938] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-19 00:37:55.358827] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:37:55.358852] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-19 00:37:55.359096] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:37:55.359111] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] [2016-10-19 00:43:08.748948] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:43:08.748980] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-4394-2016/10/19-00:43:08:671408-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:43:08.820182] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-4394-2016/10/19-00:43:08:671408-engine-client-0-0-0 [2016-10-19 00:43:08.820251] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-4394-2016/10/19-00:43:08:671408-engine-client-0-0-0 [2016-10-19 00:43:09.130290] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:43:09.130319] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-4499-2016/10/19-00:43:09:82533-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:43:09.620126] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-4499-2016/10/19-00:43:09:82533-engine-client-0-0-0 [2016-10-19 00:43:09.620214] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-4499-2016/10/19-00:43:09:82533-engine-client-0-0-0 [2016-10-19 00:43:10.133921] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:43:10.133969] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-4618-2016/10/19-00:43:10:87399-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:43:10.199392] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-engine-server: disconnecting connection from virt04.gamull.com-4618-2016/10/19-00:43:10:87399-engine-client-0-0-0 [2016-10-19 00:43:10.199473] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-engine-server: Shutting down connection virt04.gamull.com-4618-2016/10/19-00:43:10:87399-engine-client-0-0-0 [2016-10-19 00:47:31.649782] I [login.c:81:gf_auth] 0-auth/login: allowed user names: e4b1452d-958b-4224-80d1-3624127e29b6 [2016-10-19 00:47:31.649817] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-engine-server: accepted client from virt04.gamull.com-5132-2016/10/19-00:47:31:603846-engine-client-0-0-0 (version: 3.7.9) [2016-10-19 00:47:32.485087] E [MSGID: 113072] [posix.c:3317:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2016-10-19 00:47:32.485149] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-engine-server: 310: WRITEV 0 (e8e871ba-17cf-4047-bdec-8cedaec8196b) ==> (Invalid argument) [Invalid argument] [2016-10-19 00:47:55.277539] E [MSGID: 113040] [posix.c:3043:posix_readv] 0-engine-posix: read failed on gfid=e8e871ba-17cf-4047-bdec-8cedaec8196b, fd=0x7fa5840cd02c, offset=127488 size=512, buf=0x7fa578006000 [Invalid argument] [2016-10-19 00:47:55.277598] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-engine-server: 345: READV -2 (e8e871ba-17cf-4047-bdec-8cedaec8196b) ==> (Invalid argument) [Invalid argument] [2016-10-19 00:47:56.001731] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:47:56.001777] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:47:56.001731] and [2016-10-19 00:47:56.004151] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:47:56.001777] and [2016-10-19 00:47:56.004152] [2016-10-19 00:57:57.001974] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null) [2016-10-19 00:57:57.002018] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument] The message "E [MSGID: 113091] [posix.c:178:posix_lookup] 0-engine-posix: null gfid for path (null)" repeated 3 times between [2016-10-19 00:57:57.001974] and [2016-10-19 00:57:57.004234] The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-engine-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-10-19 00:57:57.002018] and [2016-10-19 00:57:57.004236]