1
|
- Robin T. Miller
- This Software is Provided
- By
- Robin’s Nest Software Inc.
- 752 Pinnacle Court
- Mesquite, NV 89027
- (702) 345-2236
- WARNING!!!
- This software MUST BE CONSIDERED DESTRUCTIVE to any data stored
- on the device under test. NO safety precautions are taken
- to ensure existing data is
preserved during testing!!!
- Please Note
- DT is Open Source. You are free
to copy and distribute it!
- Please see copyright notice/disclaimer for details.
|
2
|
- The data test program (dt), has become very popular over the last 20
years! (showing my age J)
- This overview describes dt’s history, device support, options, and
various examples to help get you started with developing tests.
|
3
|
- Originally developed while working at Compugraphic initially for testing
tape devices on SUN/OS.
- Written to replace Unix ‘dd’ utility.
- Command format similar to ‘dd’.
- Later disk, serial lines, and other device type support was added.
- Goal was to be generic yet flexible enough to customize your tests.
|
4
|
- List of OS’s supported:
- AIX
- HP-UX
- Linux
- Solaris
- FreeBSD
- MACOS
- SCO Unixware
- Tru64 Unix
- QNX
- Windows(*)
- most OS’s supporting POSIX API’s
- Features supported vary by OS (e.g., AIO, etc)
- (*) Windows requires Cygwin toolkit today.
|
5
|
- Device drivers (disks, tapes, adapters, etc)
- File systems (UFS/VxFS/JFS/NFS, etc)
- Network interfaces (rsh pipes/NFS traffic).
- Virtual Memory via memory mapped files.
- Serial or parallel lines.
- Pipes (regular or named).
- Most any device capable of supporting standard open/read/write/close()
API’s.
- Exclude non-essential features by removing:
- compilation flags: -DMMAP -DFIFO -DTTY
|
6
|
- To write then read/verify data:
- # dt of=filename bs=value limit=value …
- The read/verify previously written data:
- # dt if=filename bs=value limit=value …
- To write without doing read/verify pass:
- # dt of=filename bs=value disable=verify …
- To read without doing data comparision:
- # dt if=filename bs=value disable=compare …
- To specify the device type and device size:
- # dt of=filename dtype=type dsize=value …
- Note: filename is a valid path for your OS.
|
7
|
- For sequential do I/O until EOF.
- For random, do I/O until data limit.
- Options:
- limit=value – data limit bytes.
- rlimit=value – random I/O bytes.
- records=value – record count.
- capacity=value – user set capacity.
- volumes=value – number of volumes.
|
8
|
- bs=value - sets the block size.
- min=value - sets the minimum size.
- max=value - sets the maximum size.
- incr=value - sets the increment size.
- incr=var enables variable increments.
- dsize=value - sets the device size.
- default is 512 bytes for disks.
- set via IOCTL on some OS’s.
|
9
|
- Normally each pass is controlled by the capacity or one of these
options:
- limit=value – Specifies the data limit.
- records=value – Specifies record count.
- for sequential I/O, stop at EOF/EOM.
- for random I/O, loop until limit reached.
- runtime=time – Runtime in seconds.
- Careful: Short runtimes may prevent read/verify pass depending on
limits.
- passes=value – Specifies pass limit.
|
10
|
- Data buffers page aligned by default.
- align=value to misalign (e.g. align=rotate)
- Pad bytes allocated for data buffers.
- On writes, initialized with inverted pattern.
- On reads, verified after normal data.
- Capacity set via seek/reads or IOCTL.
- capacity=value sets user capacity.
- Data verification enabled by default.
|
11
|
- w = words (4 bytes)
- b = blocks (512 bytes)
- m = megabytes
- g = gigabytes
- t = terabytes
- inf or INF = Infinite
- q = quadwords (8 bytes)
- k = kilobytes (1024 bytes)
- p = page size (varies)
- Simple arithmetic allowed.
- Bitwise operations allowed.
- ‘expr’ for complex math.
|
12
|
- Time Input:
- d = days (86400 seconds), h = hours (3600 seconds)
- m = minutes (60 seconds), s = seconds (the default)
- Arithmetic characters are
permitted, and implicit addition is performed on strings of the form
'1d5h10m30s'.
- Examples:
- # dt ... alarm=10s noprogt=1m runtime=24h
|
13
|
- pattern=string format controls supported:
- Pattern String Input:
- \\ = Backslash \a = Alert (bell) \b = Backspace
- \f = Formfeed \n = Newline \r = Carriage Return
- \t = Tab \v = Vertical Tab \e or \E = Escape
- \ddd = Octal Value \xdd or \Xdd = Hexadecimal Value
- Example:
- # dt of=- pattern="Robin’s test message\n\a" count=1
stats=none | head -2
- Robin’s test message
- Robin’s test message
- #
|
14
|
- Assortment of data patterns (pattern= or pf= options).
- Control of I/O request sizes (min=, max=, incr=, bs= opts).
- Checks for buffer overwrites (pad bytes w/inverted pattern).
- Allows misaligned data buffers (align= option vs. page aligned).
- Allows block number encoding (enable=lbdata or pattern=iot).
- LBA’s are 32-bits
- lbdata big-endian LBA’s
- iot encodes little-endian LBA’s
- Reports performance statistics (total stats after test complete).
- Variable I/O request sizes (incr=variable option).
- Various flags to control program behavior (debug, verify, etc).
- Intuitive numeric and runtime string input formats.
- Flags to control test operation (flags=direct,excl,sync etc).
- No I/O progress options to detect and report slow or no I/O.
|
15
|
- Limit I/O by data limit (limit= option).
- Limit I/O by record limit (count= or records= options).
- Limit I/O by volume limit (volumes= and vrecords= options).
- Limit I/O by runtime limit (runtime=timeString option).
- Control of error limit (errors= option, default is 1).
- Control of pass limit (passes= option, default is 1).
- Control of multiple processes (procs= option).
- Control of process behavior (oncerror=abort or continue).
- Control of device types, size, etc. (dtype= and dsize= options).
- POSIX Asynchronous I/O (enable=aio or aios= options).
|
16
|
- 32 bit hex data pattern (pattern=value).
- 13 internal data patterns cycled through for each write/read pass.
- IOT test pattern (pattern=iot).
- Incrementing data pattern:
- 0, 1, 2, … 255 (pattern=incr)
- Pattern string (pattern=“string”)
- Pattern file (pf=filename)
- whole file used as pattern data.
|
17
|
- Data patterns cycled through for each pass:
- 0x39c39c39U
- 0x00ff00ffU
- 0x0f0f0f0fU
- 0xc6dec6deU
- 0x6db6db6dU
- 0x55555555U
- 0xaaaaaaaaU Complement of previous data pattern.
- 0x33333333U Continuous worst case pattern (media defects)
- 0x26673333U Frequency burst worst case pattern #1.
- 0x66673326U Frequency burst worst case pattern #2.
- 0x71c7c71cU Dibit worst case data pattern.
- 0x00000000U
- 0xffffffffU
- Multiple slices also cycle through these data patterns.
|
18
|
- There are several predefined data pattern files included in the source
kit, containing the following data patterns:
- pattern_0 - psuedo-random pattern.
- pattern_1 - psuedo-random pattern alternating with 0's.
- pattern_2 - all 0s.
- pattern_3 - all 1s (FFH).
- pattern_4 - 16 bit shifting 1 in a field of 0s.
- pattern_5 - 16 bit shifting 0 in a field of 1s.
- pattern_6 - alternating 01 pattern (AAH).
- pattern_7 - byte incrementing (OOH - FFH).
- pattern_8 - encrypted data pattern.
- pattern_9 - psuedo-random pattern1 alternating with 0's.
- pattern_all – concatenation of all of the above patterns.
- These files can be specified with the pf=file option.
|
19
|
- The Logical Block Address (LBA) is encoded in the first 4 bytes of each
data block (little-endian byte ordering).
- The constant value 0x01010101 is added to the LBA as a seed for
subsequent words to generate uniqueness throughout the entire data
block.
- Example:
- # dt of=- count=1 pattern=iot stats=none | od -X
- 0000000 00000000 01010101
02020202 03030303
- 0000020 04040404 05050505
06060606 07070707
- 0000740 78787878 79797979
7a7a7a7a 7b7b7b7b
- 0000760 7c7c7c7c 7d7d7d7d
7e7e7e7e 7f7f7f7f
- 0001000
- #
|
20
|
- Common Open Flags:
- excl (O_EXCL) Exclusive
open. (don't share)
- ndelay (O_NDELAY) Non-delay
open. (don't block)
- nonblock (O_NONBLOCK) Non-blocking open/read/write.
- direct (O_DIRECT) Direct disk
access. (don't cache data).
- rsync (O_RSYNC) Synchronize
read operations.
- sync (O_SYNC) Sync
updates for data/file attributes.
- large (O_LARGEFILE) Enable
large (64-bit) file system support.
- Output Open Flags:
- append (O_APPEND) Append data
to end of existing file.
- defer (O_DEFER) Defer
updates to file during writes.
- dsync (O_DSYNC) Sync data
to disk during write operations.
- trunc (O_TRUNC) Truncate an
existing file before writing.
- Example:
- # dt of=dt.data flags=direct oflags=trunc ...
|
21
|
- Random & Sequential I/O (iotype=sequential or random).
- Supports raw & block devices (gleamed from stat() or IOCTL).
- Allows multiple file system files (unique files using child PID).
- Supports copy/verify of disks (iomode=copy, test, or verify).
- copy/verify is for image mode copy operations.
- Supports reverse direction (iodir=forward or reverse).
- reverse I/O is a variant of random I/O.
- Supports multiple disk slices (slices= options).
- shared LUN testing via slices=hosts slice=host#
- Supports skipping blocks (step=value option).
- creates holes (empty space) in FS files.
- Control FS output file disposition (dispose=delete or keep).
|
22
|
|
23
|
- Supports multiple tapes files (files= option):
- writes file marks between files.
- rewind to beginning of tape (BOT).
- Forward Space File operation used between tape files.
- Supports read-after-write (enable=raw, Backspace Record).
- Supports tape repositioning (enable=eei,resets):
- Extended Error Information (EEI) unique to Tru64 Unix.
- Supports multi-volume tapes (enable=multi or volumes= opts).
- Verifies media changed is ready via Rewind operation.
- Sorry, no media changer support for loaders or libraries:
- scu is commonly used to move media in test scripts.
|
24
|
- Supports setting of flow control (flow=none,cts_rts, xon_xoff).
- Supports setting of parity (parity=even,odd, or none).
- Supports setting of speed (speed= option, none gives list).
- Handles line desciplines on Tru64 Unix.
- Single line loopback tests (enable=loopback)
- Dual line loopback tests (if=/dev/tty00 of=/dev/tty01)
- Dual line between different hosts and different OS’s too!
- Limited Tru64 modem support (enable=modem, on Tru64).
- Parallel loopback support too (if=/dev/pig or /dev/pog, Agfa).
|
25
|
- Distributed Lock Manager (DLM) (munsa=cr, cw, pr, pw, or ex)
- Tape Extended Error Information (EEI) (enable=eei, is default). On tape errors, full tape stats and
EEI data is displayed.
- Tape bus/device reset support (enable=resets, repositions tape).
- Device info via DEC IOCTL() (DEVIOCGET or DEVGETINFO).
- Optionally logs errors to system binary error logger *if* root!
(enable=diag) Diagnostic message events are 350 (ELMSGT_DIAG)
- Tru64 Unix disk label caveats (first 8K of disk protected, must either
skip label or use ‘disklabel –z’ to zero the disk label).
- Use “sysconfig -q rt” to query the Async I/O (AIO) limits.
|
26
|
- enable=flag options:
- debug – 1st level debug (open, etc).
- Debug – full debug output (verbose).
- edebug – end of file/media debug.
- rdebug – random I/O debug.
- timestamps – timestamp blocks.
- trigger=type – specifies trigger type.
- dispose=keep – don’t delete FS files.
|
27
|
- Option: trigger=type
- Trigger Types:
- br = Execute a bus reset.
- bdr = Execute a bus device reset.
- seek = Issue a seek to the failing lba.
- cmd:string = Execute string w/these args:
- string dname op dsize offset position lba errno
- The first three options require Scu in your PATH.
- ( may replace this with SCSI library in future )
|
28
|
- string dname op dsize offset position lba errno
- string = executable program or script.
- dname = The device or file name.
- op = The operation in error.
- dsize = The device size.
- offset = The 64-bit file offset.
- position = Position in block of DC.
- lba = The logical block address (real or relative)
- errno = The error number.
|
29
|
- Trigger script exit value action (only pertains to no-progress I/O
feature):
- 0 = continue
- 1 = terminate
- 2 = sleep
- 3 = abort
- Example:
- % cat trigger
- echo $*
- exit 2
- % dt of=/var/tmp/dt.data alarm=3s trigger="cmd:trigger"
disable=stats flags=direct bs=32k noprogt=1s limit=1m procs=100
- dt (16308): No progress made for 2 seconds!
- dt (16308): Executing: trigger /var/tmp/dt.data-16308 noprog 512 131072
0 0 0
- /var/tmp/dt.data-16308 noprog 512 131072 0 0 0
- dt (16308): Trigger exited with status 2!
- dt (16308): Sleeping forever...
- ...
|
30
|
- When executing a trigger script, the op argument is set to one of the
following:
- open – device open failed
- close – close operation failed
- read – read operation failed
- write – write operation failed
- noprog – no progress time exceeded
- miscompare – data compare error
- <various tape operations>, i.e. rewind, etc
|
31
|
- Prefix strings copied to start of each block.
- Used to create uniqueness between hosts and/or devices/files.
- Prefix Format Control Strings:
- %d = The device name.
%D = The real device name.
- %h = The host name.
%H = The full host name.
- %p = The process ID.
%P = The parent PID.
- %u = The user name.
- Example: prefix="%u@%h
(pid %p)"
|
32
|
- Originally added for HP’s Hazard to avoid PTO’s.
- Emits user defined message during long test runs.
- Allows you to roll your own pass/total messages.
- alarm=time controls frequency of keepalive msgs.
- keepalive=string - defines the runtime message.
- pkeepalive=string - defines per pass message.
- tkeepalive=string - defines the totals message.
- Used when brief statistics enabled (stats=brief):
- Per Pass Default:
- %d Stats: mode %i, blocks %l, %m Mbytes, pass %p/%P, elapsed %t
- Totals Default:
- %d Totals: %L blocks, %M Mbytes, errors %e/%E, passes %p/%P, elapsed
%T
|
33
|
- %b = The bytes read or
written. %B = Total bytes read
and written.
- %c = Record count for
this pass. %C = Total records for this test.
- %d = The device
name. %D = The real
device name.
- %e = The number of
errors. %E = The error
limit.
- %f = The files read or
written. %F = Total files read
and written.
- %h = The host
name. %H = The full
host name.
- %k = The kilobytes
this pass. %K = Total kilobytes
for this test.
- %l = Blocks read or
written. %L = Total blocks
read and written.
- %m = The megabytes
this pass. %M = Total megabytes
for this test.
- %p = The pass
count. %P = The pass
limit.
- %r = Records read this
pass. %R = Total records read
this test.
- %s = The seconds this
pass. %S = The total seconds
this test.
- %t = The pass elapsed
time. %T = The total elapsed
time.
- %i = The I/O mode
(read/write) %u = The user
(login) name.
- %w = Records written
this pass. %W = Total records
written this test.
- Performance Keywords:
- %bps = The bytes per second. %lbps = Logical blocks per second.
- %kbps = Kilobytes per
second. %mbps = The megabytes
per second.
- %iops = The I/O's per
second. %spio = The seconds per
I/O.
- Lowercase means per pass
stats, while uppercase means total stats.
|
34
|
- Redirect output to a log file via log= option.
- logs dt’s command line at top of log.
- Log files are opened in append mode.
- Beware: No mounted file system checks!
- Tru64 Unix & Solaris, skip first 8K to avoid overwriting the disk
label (position=8k).
- Success exit status is zero (0).
- General failure exit status is 255.
- End of file/media exit status is 254.
- Signals cause non-zero exit status too.
|
35
|
- For file system I/O, there are two preferred command lines to consider:
- # dt of=dt.data min=1 max=256k incr=var limit=100m enable=lbdata
procs=10
- OR
- # dt of=dt.data min=b max=256k incr=var limit=100m pattern=iot procs=10
- Alternate between buffered I/O and direct I/O (flags=direct).
- For raw I/O disk tests:
- # dt of=/dev/hdisk7 min=b max=256k incr=var pattern=iot slices=10
aios=4 align=rotate
- For performance disk tests:
- # dt of=dt.data bs=64k limit=1g disable=compare
- OR
- # dt of=/dev/hdisk7 bs=64k aios=32 disable=compare
- It’s advisable to alternate between sequential and random I/O, and when
doing sequential, alternate between forward and reverse direction.
|
36
|
- TEST DESCRIPTION: This test shows
a method to verify large TB sized disks without doing I/O to the full
capacity. A number of options are
used, here are the key ones:
- slices=16 to divide the capacity into 16 sections, each with their own
process
- aios=4 to queue 16 requests per process for improved performance.
- step=4g to seek after every record, thereby avoid writing every block.
- iodir=reverse to start at the end of sections then work forward.
- other options are left as an exercise to the reader! J
- shaix11# dt \
- of=/dev/rhdisk10 \
- slices=16 \
- step=4g \
- disable=pstats \
- aios=4 \
- oncerr=abort \
- min=b \
- max=256k \
- incr=var \
- align=rotate \
- pattern=iot \
- iodir=reverse \
- prefix='%d@%h' \
- alarm=5s \
- noprogt=10s \
- runtime=1h
|
37
|
- alarm=value and noprogt=value options useful to detect slow or no I/O
times, esp. during failovers.
- alarm() generates SIGALRM signal which can only be delivered during
interruptible kernel sleeps, thus it’s not always sent.
- Example:
- shaix11# ./dt of=/dev/rhdisk7 bs=64k aios=16 pattern=iot \
- alarm=3s noprogt=15s
runtime=5m stats=brief
- [ Initiated controller takeover to force path failover. ]
- dt: No progress made for 18 seconds!
- dt: No progress made for 21 seconds!
- ...
- dt: No progress made for 39 seconds!
|
38
|
- dt’s iomode= option controls {copy, test, or verify} modes.
- The copy is an image mode copy automatically verified.
- Useful for copying CD/DVD ISO’s to disk (preferred over dd).
- Example:
- shaix11# dt of=/dev/hdisk0 bs=64k pattern=iot aios=16 stats=none (prime
hdisk0 data)
- shaix11# dt if=/dev/hdisk0 of=/dev/rhdisk1 bs=64k aios=16 iomode=copy
- Copy Statistics:
- Data operation performed:
Copied '/dev/hdisk0' to '/dev/rhdisk1'.
- Total records processed:
16384 @ 65536 bytes/record (64.000 Kbytes)
- Total bytes transferred:
1073741824 (1048576.000 Kbytes, 1024.000 Mbytes)
- Average transfer rates:
35078139 bytes/sec, 34255.995 Kbytes/sec
- Number I/O's per second:
535.250
- Total passes completed: 0/1
- Total errors detected: 0/1
- Total elapsed time:
00m30.61s
- Total system time:
00m00.79s
- Total user time:
00m00.06s
- Starting time: Wed
May 16 09:53:02 2007
- Ending time: Wed
May 16 09:53:33 2007
- ...
- shaix11# dt if=/dev/rhdisk1 bs=64k aios=16 pattern=iot stats=brief (verify
hdisk1 data)
- /dev/rhdisk1 Totals: 2097152 blocks, 1024.000 Mbytes, errors 0/1,
passes 1/1, elapsed 01m13.10s
- shaix11#
|
39
|
- Data corruption example (forced):
- # dt of=dt.data count=3 dispose=keep disable=stats bs=64k pattern=iot
- # dt if=dt.data count=3 disable=stats bs=64k enable=lbdata
- dt: Error number 1 occurred on Fri May
4 16:03:12 2007
- dt: Elapsed time since beginning of pass: 00m00.00s
- dt: Elapsed time since beginning of test: 00m00.00s
- dt: Data compare error at byte 4 in record number 1
- dt: Relative block number where the error occurred is 0, position 4
(offset 4)
- dt: Data expected = 0x39, data found = 0x1, byte count = 65536
- dt: The correct data starts at address 0x20005c78 (marked by asterisk
'*')
- dt: Dumping Pattern Buffer (base = 0x20005c78, offset = 0, limit = 4
bytes):
- 0x20005c78 *39 9c c3 39
- dt: The incorrect data starts at address 0x20007004 (marked by asterisk
'*')
- dt: Dumping Data Buffer (base = 0x20007000, offset = 4, limit = 512
bytes):
- 0x20007000 00 00 00 00*01 01 01
01 02 02 02 02 03 03 03 03
- 0x20007010 04 04 04 04 05 05 05
05 06 06 06 06 07 07 07 07
- ...
- 0x200071e0 78 78 78 78 79 79 79
79 7a 7a 7a 7a 7b 7b 7b 7b
- 0x200071f0 7c 7c 7c 7c 7d 7d 7d
7d 7e 7e 7e 7e 7f 7f 7f 7f
- #
|
40
|
- Premature end of file on writes can lead to false data compare errors
with random I/O, since more data may get read than written. In general, dt expects to read as much
as is written. Note: Correct for
raw I/O, need to fix for file system I/O! L
- shaix11# dt of=/tmp/dt.data bs=32k
iotype=random capacity=250m limit=1g \
- stats=brief enable=edebug dlimit=32
- dt: WARNING: Record #4223, attempted to write 32768 bytes, wrote only
28160 bytes.
- dt: End of media detected, count = -1, errno = 28 [file #1, record
#4224]
- End of Write pass 0/1, 270263 blocks, 131.964 Mbytes, 4223 records,
errors 0/1, elapsed 00m05.05s
- dt: Error number 1 occurred on Wed May 16 16:12:36 2007
- dt: Elapsed time since beginning of pass: 00m04.27s
- dt: Elapsed time since beginning of test: 00m09.32s
- dt: Data compare error at byte 28160 in record number 4223
- dt: Relative block number where the error occurred is 154232, position
78966784
- dt: Data expected = 0x39, data found = 0, byte count = 32768
- dt: The correct data starts at address 0x20005b18 (marked by asterisk
'*')
- dt: Dumping Pattern Buffer (base = 0x20005b18, offset = 0, limit = 4
bytes):
- 0x20005b18 *39 9c c3 39
- dt: The incorrect data starts at address 0x2000de00 (marked by asterisk
'*')
- dt: Dumping Data Buffer (base = 0x20007000, offset = 28160, limit = 32
bytes):
- 0x2000ddf0 39 9c c3 39 39 9c c3
39 39 9c c3 39 39 9c c3 39
- 0x2000de00 *00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
- End of Read pass 1/1, 270272 blocks, 131.969 Mbytes, 4223 records,
errors 0/1, elapsed 00m04.27s
- /tmp/dt.data Totals: 540535 blocks, 263.933 Mbytes, errors 0/1, passes
1/1, elapsed 00m09.32s
- shaix11#
|
41
|
- Beware: False data corruptions will
occur doing random I/O with unique data.
- Unique data means different pattern data in each block.
- Why? Random I/O oftentimes
overwrites previously written block data.
- Resolving this requires maintaining a bitmap of blocks written to avoid
overwrites.
- Workaround: Don’t use pattern files larger than the device block size
(512, 1K, etc).
- Example:
- shaix11# dt of=/var/tmp/dt.data min=b max=256k pf=pattern_all limit=100m
iotype=random dlimit=32 stats=brief
- End of Write pass 0/1, 204800 blocks, 100.000 Mbytes, 895 records,
errors 0/1, elapsed 00m04.05s
- dt: Error number 1 occurred on Thu May 17 10:10:03 2007
- dt: Elapsed time since beginning of pass: 00m00.00s
- dt: Elapsed time since beginning of test: 00m04.05s
- dt: Data compare error at byte 0 in record number 4
- dt: Relative block number where the error occurred is 142297, position
72856064
- dt: Data expected = 0xa1, data found = 0x9d, byte count = 2048
- dt: The correct data starts at address 0x20006c00 (marked by asterisk
'*')
- dt: Dumping Pattern Buffer (base = 0x20006000, offset = 3072, limit = 32
bytes):
- 0x20006bf0 f0 f1 f2 f3 f4 f5 f6
f7 f8 f9 fa fb fc fd fe ff
- 0x20006c00 *a1 38 61 a2 1e 0f a4 bd 0a 3d 6d f0 ac c3 ac b4
- dt: The incorrect data starts at address 0x2000c000 (marked by asterisk
'*')
- dt: Dumping Data Buffer (base = 0x2000c000, offset = 0, limit = 32
bytes):
- 0x2000c000 *9d 9d 86 d2 aa 82 67 b7 d3 a6 13 cd 41 6c 7d f4
- 0x2000c010 36 c7 ba 04 73 9d 86
16 a5 e4 57 a6 02 ed da 71
- End of Read pass 1/1, 10 blocks, 0.005 Mbytes, 4 records, errors 1/1,
elapsed 00m00.00s
- /var/tmp/dt.data Totals: 204810 blocks, 100.005 Mbytes, errors 1/1,
passes 1/1, elapsed 00m04.05s
- shaix11#
|
42
|
- One device per process.
- AIO must be used in lieu of threads.
- AIO read-after-write (enable=raw) option does synchronous read(), thus
slowing down AIO’s.
- Encoded LBA’s wrap at 2TB with 512 byte block size.
- Missing file system tests:
- file locking
- append/truncate (oflags= can be used)
- holes (read back as zeros, step= can be used)
- automatic open() file flags (randomly select)
- directory operations
- etc.
- Remember: dt wasn’t designed for file systems!
|
43
|
- Example of building Linux dt with debug symbols:
- linux% pwd
- /u/rtmiller/Tools/dt.d-WIP/linux2.6-x86
- linux% make -f ../Makefile.linux clean
- linux% make -f ../Makefile.linux PORG=-g VPATH=..
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dt.o ../dt.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtaio.o ../dtaio.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtfifo.o ../dtfifo.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtgen.o ../dtgen.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtinfo.o ../dtinfo.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtmmap.o ../dtmmap.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtprocs.o ../dtprocs.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtread.o ../dtread.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtwrite.o ../dtwrite.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtstats.o ../dtstats.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dttape.o ../dttape.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dttty.o ../dttty.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtutil.o ../dtutil.c
- cc -g -DAIO -DFIFO -DMMAP -DTTY -D__linux__ -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o
dtusage.o ../dtusage.c
- loading dt ...done
- linux% ./dt version
- --> Date: May 13th, 2007,
Version: 15.37, Author: Robin T. Miller <--
- linux%
|
44
|
- Hung dt processes are oftentimes encountered during perturbation
(disruptive) type tests (failovers, panics, etc).
- Usually this is the result of an I/O stack (driver), adapter firmware,
switch, and/or storage device issue.
- Normally dt will be hung in some system call, e.g. open(), read(),
write(), fsync(), etc.
- Depending on the OS, either the kernel debugger or user level debugger
can be used to report dt’s stack trace.
- In lieu of a stack trace, forcing a system crash to send the OS vendor
for analysis is usually necessary and required.
- Gather OS error logs, switch logs, storage logs, etc.
- Use dt’s no I/O progress feature to trigger an analyzer.
- FWIW: Oftentimes folks wish to run another I/O tool, to reproduce the
original hang. Personally, I
believe this is a waste of time, since dt is not the reason for the
hang!
- Whether hangs or DC’s, one must learn to trust the tool. J
|
45
|
- With most OS’s, gdb can be used to attach to dt then request a user
level stack trace. Although this
lets you know dt’s last system call, this method will not provide the
kernel stack trace which may reveal where dt is really hung.
- Linux Example:
- linux# ps -ef | fgrep dt | grep -v fgrep
- root 20580 20256 49 10:45
pts/1 00:00:33 /var/tmp/dt
of=/var/tmp/dt.data limit=100m disable=stats runtime=1h
- linux# gdb dt 20580 <- PID
- GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
- …
- Attaching to program: /u/rtmiller/Tools/dt.d-WIP/linux2.6-x86/dt,
process 20580
- (gdb) bt
- #0 0x00ab17a2 in
_dl_sysinfo_int80 () from /lib/ld-linux.so.2
- #1 0x00d343f3 in
__write_nocancel () from /lib/tls/libpthread.so.0
- #2 0x080556c4 in write_record
(dip=0x87e8028,
- buffer=0x87eb000 ...,
bsize=512, dsize=512,
- status=0xbfe758dc) at
../dtwrite.c:475
- #3 0x080552c4 in write_data
(dip=0x87e8028) at ../dtwrite.c:311
- #4 0x08051410 in write_file
(dip=0x87e8028) at ../dtgen.c:853
- #5 0x0804ad1a in main (argc=5,
argv=0xbfe75a34) at ../dt.c:2006
- (gdb) quit
- The program is running. Quit
anyway (and detach it)? (y or n) y
- Detaching from program: /u/rtmiller/Tools/dt.d-WIP/linux2.6-x86/dt,
process 20580
- linux#
|
46
|
- The AIX kernel debugger (kdb) can be used on both crash dumps and a live
system to map a process and obtain a stack trace.
- Example:
- # kdb
- (0)> p * | fgrep dt
- pvproc+00B400 45 dt ACTIVE 002D028 0000001
00000000704D8400 0 0001 ...
- (0)> tpid 002D028
- SLOT NAME STATE TID PRI RQ CPUID CL
WCHAN
- pvthread+00CE00 206 dt SLEEP 0CE0F5 03C 0 0
- (0)> sw 206
- Switch to thread:
<pvthread+00CE00>
- (0)> f
- pvthread+00CE00 STACK:
- [000535A4]et_wait+0002B0 (0000000010007558, 000000000000D0B2,
- 0000000000000000 [??])
- [001B78D4]uphyswait+000158 (??, ??, ??, ??)
- [00457B4C]uphysio+000474 (??, ??, ??, ??, ??, ??, ??)
- [03FC882C]scsidisk_rdwr+000174 (??, ??, ??, ??, ??)
- [03FC8B60]scsidisk_read+000080 (??, ??, ??, ??)
- ...
- (0)>
|
47
|
- RedHat Linux ships a crash utility which can map a process and provide a
kernel stack trace (usually more useful than gdb):
- linux# crash
- crash 4.0-2.30
- ...
- crash> ps | fgrep dt
21490 20426 0 c21021b0
UN 0.1 2596 812
dt
crash> set pid 21490
set: invalid task or pid value: pid
PID: 21490
COMMAND: "dt"
TASK: c21021b0 [THREAD_INFO: d5229000]
CPU: 1
STATE: TASK_UNINTERRUPTIBLE
crash> bt
PID: 21490 TASK: c21021b0 CPU: 1
COMMAND: "dt"
#0 [d5229e50] schedule at c02d25c9
#1 [d5229eb4] log_wait_commit at f88da65c
#2 [d5229f0c] journal_stop at f88d66c8
#3 [d5229f2c] journal_force_commit at f88d6712
#4 [d5229f30] ext3_force_commit at f89b0843
#5 [d5229f34] write_inode at c0177263
#6 [d5229f40] __sync_single_inode at c01772d8
#7 [d5229f60] sync_inode at c0177a69
#8 [d5229f6c] ext3_sync_file at f89a6e91
#9 [d5229fa8] sys_fsync at c015c468
#10 [d5229fc0] system_call at c02d46fc
EAX: 00000076 EBX: 00000003 ECX:
00000000 EDX: f9c00000
DS: 007b
ESI: ffffffff ES: 007b EDI:
06400000
SS: 007b
ESP: bff257f8 EBP: bff25828
CS: 0073
EIP: 00ab17a2 ERR: 00000076 EFLAGS: 00000246
- crash>
- Note: kernel debug symbols must be loaded or crash will not start.
|
48
|
- Some of the features requested include (not prioritized):
- Develop corruption analysis logic.
What is this? Folks
familiar with HP’s Hazard know how valuable this is: data re-read
logic, I/O history, metadata prowlers, and detailed analysis of
expected and received data. A
lot of work is involved here, especially with file system prowlers,
which are responsible for converting file system data structures to
physical underlying LBA’s, to help identify bad data in analyzer
traces. See http://web.rtp.netapp.com/~rtmiller/prowlers.html
- Improved file system testing.
Although not developed as a file system exerciser, many folks
use it this way. Multiple
processes creating unique data files generates a data load, but many
file system specific features, such as truncating files, file locking,
creating lots of metadata (via subdirectories), and many more are not
tested. Major effort here!
- Supporting multiple devices in one dt invocation (perhaps a comma
separated list). Although multiple processes or threads could
accomplish this, it does add complexity requiring locking and switching
to reentrant library API’s, and the savings in shared code is minimal
(I think) since most of the address space is data buffers.
- Multiple threads for I/O is likely to be implemented one day. This hasn’t been implemented (to
date) since POSIX AIO provided for my need, and most modern day OS’s
now support POSIX AIO.
Interestingly enough, the Linux AIO library is implemented via
POSIX threads! Threads are
lightweight processes so improve
system resource utilization (reduce process slots, less
paging/swap space, etc).
- Support 64-bit LBA encoding for large disk capacity testing.
- Implement block tagging. Encodes
more information than prefix strings.
|
49
|
- Add OS specific method, like syslog(), to log dt’s error messages.
- Incorporate SCSI library to implement bus/target/lun reset triggers,
etc.
- faster triggering of analyzers (system() API slow with heavy loads)
- Interactive interface to keep the device open, like scu does, to allow
more creative tests, especially for tapes and tape libraries (although
most folks use dt for disk testing).
- Add output formats to allow statistics to be imported to tools such as
MS Excel, etc. (*keepalive= options may provide these formats)
- GUI front-end? Might be nice,
but not necessary for test automation.
- desirable for dynamic features (stats, stop/start device, etc)
- Port to VMS? There’s a need, so
given the time, this may happen.
- Native Windows? Mostly there, thanks to the HP Hazard India team. The code needs a few tweaks for file
system testing and multiple processes (originally ported for raw I/O
testing only).
- My personal goal is to create a modular design with support for testing
multiple concurrent devices, POSIX threads, and an interactive mode,
fully customizable, dynamic statistics, and on-the-fly device
interaction.
- Please send me your wish list!!!
|
50
|
- “dt help” gives option summary.
- DT Overview (dt-Overview.ppt)
- DT User’s Guide (dt-UsersGuide.doc)
- DT Source Overview (DtSourceOverview.doc)
- Above docs contained in DT source kit.
- Also available off my DT web page.
|
51
|
- Within NetApp:
- http://web.rtp.netapp.com/~rtmiller/dt.html
- External to NetApp:
- http://home.comcast.net/~SCSIguy/SCSI_FAQ/RMiller_Tools/dt.html
- Latest Executables:
- /u/rtmiller/Tools/dt.d-WIP/{OSdir}
- Perforce Source:
- //depot/tools/main/performance/
- subdirectory dt.d-v15.27/ (and later)
- Contact author via e-mail to:
- Questions and comments welcome!
|