pipechar [-P < maximum Probes> ] [-S < sleep time> [ -hop #] [-l] destination This form has been explained in the generic page . pipechar [-irtc] [other options] destination "-irtc" option will ignore the routing change. Use this option when you see pipechar fails due to frequently routing changes. pipechar [-Q [#]] destination Router queue size test. The # is in KB, e.g., 30 means 30 KB. The default value is 128 (128KB). For example: [112] test.lbl.gov: pipechar xxx.yyy.zz.hhh -Q 512 0: localhost [8 hops : 8] testing queue size 512 KB car size 16 KB 1: ir100gw-r2.lbl.gov (131.243.2.1) 0 car dropped 2: er100gw.lbl.gov (131.243.128.5) 0 car dropped 3: lbl2-gig-e.es.net (198.129.224.2) 0 car dropped 4: snv-oc3-lbl2.es.net (134.55.208.214) 1 car dropped 5: chi-s-snv.es.net (134.55.205.102) 2 cars dropped 6: anl-chi.es.net (134.55.208.42) 0 car dropped 7: anl-esanl2.es.net (198.124.254.166) 0 car dropped 8: destination (xxx.yyy.zz.hhh) 0 car dropped [113] test.lbl.gov: pipechar xxx.yyy.zz.hhh -Q 768 0: localhost [8 hops : 8] testing queue size 768 KB car size 24 KB 1: ir100gw-r2.lbl.gov (131.243.2.1) 0 car dropped 2: er100gw.lbl.gov (131.243.128.5) 0 car dropped 3: lbl2-gig-e.es.net (198.129.224.2) 0 car dropped 4: snv-oc3-lbl2.es.net (134.55.208.214) 3 cars dropped 5: chi-s-snv.es.net (134.55.205.102) 17 cars dropped 6: anl-chi.es.net (134.55.208.42) 0 car dropped 7: anl-esanl2.es.net (198.124.254.166) 0 car dropped 8: destination (xxx.yyy.zz.hhh) 0 car droppedHere we see that node 5 seems to have a smallest queue (about 480KB = 512 - 16KB * 2) amount these routers. Re-test it with "-Q 480" should see no car dropped at node 5 :-)
1.1b: use ncs
Start ncsd on one of your local machine with command "ncsd [-l] [-irtc]". The "-l" option is for bypassing non responsive router and passing through firewall. The "-irtc" is to Ignore RouTing Change. After the ncsd is running on your local network, you can use "ncsC" -- ncs client -- to inquiry the path information. For example,The option "-ah" queries all hops, and the option "-TCPWin" queries tuned TCP buffer size.
1.2: netest-1 (netest version 1)
In this section, the netest-1 is used to analysis on a problematic router that has been detected by pipechar, and detect TCP congestion window. As we know the netest-1 is a SRP tool. That is, it may have more options than a SO tool. To reduce the pressure on how to use it, we simplify the usage by saying: "to use UDP only for network analysis, and do not use other netest-1 options." netest-1 is also a network traffic simulator (wave generator) and analyzer, but, we use it solely for network analysis here. Usage: receiver% netest-1 -u -P [-p port] [-N] sender% netest-1 -u -P [-p port] [-N] -t receiver
[-u] option tells netest-1 to send/receive UDP traffic instead of TCP [the default].
netest-1 was designed to do clock synchronization in UDP mode "-u".
netest-1 synchronizes the system clocks on both sender and receiver
sides, and does two way measurement. Examples is given in
section 6 and section 7 .
This works well when network speed below OC-12 (622 Mb/s) or the network
do not lose a lot of packets. [-N] option is for disable this feature.
In case the network loses packets, [-N] option should be used, and
netest-1 will do one way throughput measurement only.
[-P] is for UDP receiver to show packet loss pattern (packet monitor).
See traffic monitor for details.
2. plot chart of pipe characteristics for a path:
The script plot-pipechar is for plotting pipechar/"pipechar -PxCHAR"/pchar
output, and the script plot-ncs is for plotting ncsC output.
Examples: pipechar -P 6 www.yahoo.com | plot-pipechar [-gp] or ncsC -s ncsd_host_name -ah www.yahoo.com | plot-ncs [-gp]The default plot tool is xgraph. "-gp" option is to use GNU plot. So, one of these plotting tools needs to be installed on your system. If you do not have either xgraph or GNU plot on your system, and you use ncs, you may use our web server to plot the result by sending an email to ncs@lbl.gov with body information
add NCS server: SYMBOL=server-nameSYMBOL is any string (white space is OK) that will show on the submit button; see Web Inquiry Interface for example. server-name is the host name where your ncsd is running. Then you can use the Web Inquiry Interface to query paths from your local network to other places. To remove this service from our web server, send mail to the same email address with body information
remove NCS server: SYMBOL=server-name [user@mail-host]The user@mail-host is required only if you send the remove request from a different host rather than the original host where the "add NCS server" message was sent. The service can be removed only when both the SYMBOL=server-name and the mail address are matched.
3. get tuned TCP buffer size:
ncsC -TCPWin destinationtells what is an optimized TCP receiving/sending buffer size from the testing host to the destination. If you use pipechar, you may compute the buffer size by multiple the lowest bandwidth and the longest RTT between the testing host and the destination.
4. find achievable TCP bandwidth:
If you have access to the destination host, you can run netest-1 or iperf or ttcp to get the achievable TCP bandwidth. For example,
start receiver first:
receiver% netest-1 [-p port]
then sender
sender% netest-1 [-p port] -t receiver
There is no other way to achieve this goal.
5. determine TCP congestion window:
This also requires access to the remote destination host.
start receiver first:
receiver% netest-1 [-p port] -u -P -N
then sender
sender% netest-1 [-p port] -u -P -N -l 4096 -t receiver
If you see many "x" in pattern report, stop sender, then restart the sender
sender% netest-1 [-p port] -u -P -N -l 4096 -t receiver -n 48You may keep reduce # for "-n" option and restart the testing until the "x" disappears.
If you see no "x" in pattern report, restart the sender with larger # for "-n" option:
sender% netest-1 [-p port] -u -P -N -l 4096 -t receiver -n 72and keep increasing the # for "-n" option until seeing the "x" in pattern report.
Multiple 4096 with # used for "-n" option, the product is the TCP congestion window. How to set this window in your TCP stack is a question at this moment.
The netest had a second version for automatic TCP congestion window detection. It will automatically measure UDP/TCP performance, available bandwidth, the maximum burst size, parallel streams needed to improve performance, etc. This significantly simplifies users' job.
If you do not have access to the remote host, you may use pipechar to guess it.
pipechar -hop 0 -Q 144 destination 0: localhost [1 hops : 14] testing queue size 144 KB car size 4 KB 14: test.lbl.gov (131.243.2.211) 21 cars dropped The congestion window can be estimated as: C_WIN = 144KB - 21 x 4KB = 60KB
6. analyze and locate problematic routers:
When end-to-end TCP performance is significantly lower than the available bandwidth measured by pipechar/ncs, it indicates that there is a routing problem. Some common problems are:- incorrect auto-negotiation in 10/100BT local network
- incorrect level-4 filter deployed
- queue configured too small
- od asymmetric path
6.1. detect routing problem
Usually, the mis-configured router is shown on the pipechar/ncs chart
via Fluid Spray Effect (FSE). The node 5, in the picture at the right side,
presents the FSE that indicates problems at that router. Under this
circumstance, obviously there is mis-configuration problem at the router.
This type of problems is easy to pin down if a remote end system is accessible.
The approach is to use netest-1 to find packet drop pattern, and figure out
what is the problem.
Case 1:
An real example of using netest-1 to analyze a router, and its pattern report is in netest-1 receiver output . By default, netest-1 transfers 64 packets in a burst. The packet size is automatically determined by the operating system. Due to the clock synchronizing failure, we add "-N" option for the further testing. In this example, we see all packet (size=32739) are lost in the first three transmission. Since we saw the transmission changed from "2 000001 buf_seq#:" to "2 000003 buf_seq#:", we know that only large packet getting lost; otherwise, the receiver should not report anything and be sat there very quiet if the small packets (timing packet -- pseudo train) are also lost. Therefore, we restarted the sender with smaller packet (the sending seqeunce is shown in the output):and all packets are travel through. Then increasing the packet with 28 more bytes (actually 1 byte is enough),
and we saw six (6) packets arrived with interleaved (mis-ordered) synch packets (8-byte long), then another packet, and some corrupt packet that makes IRIX coredump {only IRIX send bad packet up to the user space, the other OS will toss the packet, and program will report lost packet by saying "x x 6 x x 9 ..." rather than causing program coredump, that is why we used a IRIX as the receiver for this example}. The next example under this one uses a Sun workstation as a receiver that reports many lost packet and mis-ordered packet (strange packet without sync header 1478 x x x 51).
At this point, we know that router randomly drops the fragmented packet.
Why? later we found out there was a filter trying to filter some packet
based on transport layer information. However, the fragmented packets
have no such information in there, so they are the sacrificer.
These testing takes only a few minutes. The problematic router can be found
in real-time. As long as we are capable to access a remote host, we can
quickly understand what is the pattern of the router dropping the packet.
Contact the router administrator and have them to fix the problem.
Case 2:
Another real case:
sending sequence --
1 netest-1 -p 1910 -u -P -N -t receiver
almost all packet lost
2 netest-1 -p 1910 -u -P -N -t receiver -l 4096 -n 24
seeing packet lost at end of the train
3 netest-1 -p 1910 -u -P -N -t receiver -l 4096 -n 32
seeing packet lost at the similar position as test 2
4 netest-1 -p 1910 -u -P -N -t receiver -l 8192 -n 32
packet lost at the similar size (about 80KB) as test 2 and 3
Conclusion:
If TCP knows how to set congestion window to 80KB, application
will be able to gain the maximum throughput as the available bandwidth.
receiver% netest-1 -p 1910 -u -P -N
socket with Window = 32739 real BUFFERSIZE: 32739
1 netest-r: npkt=64, pktlen=32739, port=1910, burst period=5000, pkt_delay=0
connected BUFFERSIZE: 32739
Receiving SYNC hdr [8] from Host: 128.117.13.119 (port 45241)
recv data pkt: 28752
0 1
lost packets = 62, recvfrom err=0
2 000001 buf_seq#: handle
3 23.0894 sec. Thu Aug 30 16:09:54 2001 [from SYNC | start receiving bst]
4 23.1537 sec. Thu Aug 30 16:09:54 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 7.156448 Mbits (receiveing rate)
Receiving SYNC hdr [8] from Host: 128.117.13.119 (port 45246)
--------- 2 -------
recv data pkt: 4096
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
16 17 18 19 x x 22
lost packets = 3, recvfrom err=0
2 000001 buf_seq#: handle
3 52.4507 sec. Thu Aug 30 16:10:23 2001 [from SYNC | start receiving bst]
4 52.4620 sec. Thu Aug 30 16:10:23 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 61.145954 Mbits (receiveing rate)
recv data pkt: 4096
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
16 17 18 19 20 x 22
lost packets = 2, recvfrom err=0
2 000002 buf_seq#: handle
3 57.4513 sec. Thu Aug 30 16:10:28 2001 [from SYNC | start receiving bst]
4 57.4599 sec. Thu Aug 30 16:10:28 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 83.776955 Mbits (receiveing rate)
Receiving SYNC hdr [8] from Host: 128.117.13.119 (port 45247)
--------- 3 -------
recv data pkt: 4096
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
16 17 18 19 20 21 22 x x 25 x 27
lost packets = 7, recvfrom err=0
2 000001 buf_seq#: handle
3 73.3718 sec. Thu Aug 30 16:10:44 2001 [from SYNC | start receiving bst]
4 73.3820 sec. Thu Aug 30 16:10:44 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 80.878014 Mbits (receiveing rate)
recv data pkt: 4096
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
16 17 18 19 20 21 x x x 25 x 27 x 29
lost packets = 7, recvfrom err=0
2 000002 buf_seq#: handle
3 78.3726 sec. Thu Aug 30 16:10:49 2001 [from SYNC | start receiving bst]
4 78.3833 sec. Thu Aug 30 16:10:49 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 76.506020 Mbits (receiveing rate)
Receiving SYNC hdr [8] from Host: 128.117.13.119 (port 45248)
--------- 4 -------
recv data pkt: 8192
0 1 2 3 4 5 6 7 8 9 10 x x 13
lost packets = 20, recvfrom err=0
2 000001 buf_seq#: handle
3 99.9236 sec. Thu Aug 30 16:11:11 2001 [from SYNC | start receiving bst]
4 99.9379 sec. Thu Aug 30 16:11:11 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 55.078625 Mbits (receiveing rate)
recv data pkt: 8192
0 1 2 3 4 5 6 7 8 9 10 11
lost packets = 20, recvfrom err=0
2 000002 buf_seq#: handle
3 104.9235 sec. Thu Aug 30 16:11:16 2001 [from SYNC | start receiving bst]
4 104.9392 sec. Thu Aug 30 16:11:16 2001 [from SYNC | end of receiving bst]
6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W32739]
1 49.938931 Mbits (receiveing rate)
7. use netest-1 for traffic generator and monitor
-
7.1: Use -l and -tl options to detect router drp packet issue
-l UDP packet length (payload) [default is the maximum (O.S. specific) 32KB - some overhead]
-n number of UDP packets per burst [default is 64]
-tl test loops (number of tests) [default is infinite]start a UDP monitor on a receiver host: traffic-monitor% netest-1 -N -u -P six commands to test how router handles the different size of packets. each send one packet train "-tl 1" with 64 UDP packets (default) and different packet length "-l #" traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor -l 16272 traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor -l 8872 traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor -l 4432 traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor -l 2952 traffic-generator% netest-1 -N -u -tl 1 -t traffic-monitor -l 1472 or use the following script for differnt MTU: ------------------- begin of script ----------------- #!/bin/sh # traffic generator command CMD="netest-1 -N -u -p 1910 -tl 1" MTU=1500 intv=1 # pause 1 second between each train while [ $# -gt 0 ]; do case $1 in -i) intv=$2; shift ;; -mtu) MTU=$2; shift ;; -*) echo "$0 [-mtu MTU] destination" exit ;; *) DST=$1 ;; esac shift done [ -z "$DST" ] && { echo no destination exit 1 } #send a default train $CMD -t $DST || { echo destination does not exist exit 2 } sleep 1 # must pause after a fast large train. for np in 16 11 6 3 2 1 ; do pkt_len=`echo $MTU $np | awk '{printf("%d", $1 * $2 - $2 * 20 - 8);}'` $CMD -t $DST -l $pkt_len sleep $intv done ------------------- end of script ----------------- results: Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1111) recv data pkt: 32536 0 x 2 3 x 5 x 7 x 9 10 x 12 13 x 15 x 17 x 19 x 21 x 23 x 25 x 27 x 29 x 31 x 33 x 35 x 37 x 39 x 41 x 43 x 45 x 47 x 49 x 51 x 53 x 55 x 57 x 59 x 61 x 63 lost packets = 30, recvfrom err=0 2 000001 buf_seq#: handle 3 526.0117 sec. Sun May 18 13:38:47 2003 [from SYNC | start receiving bst] 4 526.0452 sec. Sun May 18 13:38:47 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 263.611816 Mbits (receiveing rate) 1106280 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1113) recv data pkt: 23672 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 x x x 21 22 23 24 25 26 27 28 29 30 31 32 33 x x 36 37 38 39 40 41 42 43 44 45 46 47 48 x 50 x x 53 54 55 56 57 58 59 60 61 62 63 lost packets = 8, recvfrom err=0 2 000001 buf_seq#: handle 3 527.0720 sec. Sun May 18 13:38:48 2003 [from SYNC | start receiving bst] 4 527.0963 sec. Sun May 18 13:38:48 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 436.025177 Mbits (receiveing rate) 1325680 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1115) recv data pkt: 16272 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 x 20 x 22 23 24 x 26 27 x 29 30 31 32 x 34 35 x x x 39 x 41 x 43 44 45 x 47 48 49 50 x x x 54 x 56 57 x 59 60 61 62 63 lost packets = 16, recvfrom err=0 2 000001 buf_seq#: handle 3 528.1222 sec. Sun May 18 13:38:49 2003 [from SYNC | start receiving bst] 4 528.1392 sec. Sun May 18 13:38:49 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 368.662659 Mbits (receiveing rate) 781104 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1117) recv data pkt: 8872 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 x 22 x x x x x 28 x 30 31 32 33 34 35 36 37 x x x x 42 43 44 45 x 47 48 49 50 x 52 53 54 x x x x x 60 x 62 63 lost packets = 19, recvfrom err=0 2 000001 buf_seq#: handle 3 529.1725 sec. Sun May 18 13:38:50 2003 [from SYNC | start receiving bst] 4 529.1820 sec. Sun May 18 13:38:50 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 335.395203 Mbits (receiveing rate) 399288 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1119) recv data pkt: 4432 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 x x 62 63 lost packets = 2, recvfrom err=0 2 000001 buf_seq#: handle 3 530.2126 sec. Sun May 18 13:38:51 2003 [from SYNC | start receiving bst] 4 530.2177 sec. Sun May 18 13:38:51 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 435.377441 Mbits (receiveing rate) 274832 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1121) recv data pkt: 2952 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 lost packets = 0, recvfrom err=0 2 000001 buf_seq#: handle 3 531.2530 sec. Sun May 18 13:38:52 2003 [from SYNC | start receiving bst] 4 531.2568 sec. Sun May 18 13:38:52 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 394.093842 Mbits (receiveing rate) 188968 Receiving SYNC hdr [8] from Host: 131.243.sss.hhh (port 1123) recv data pkt: 1472 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 lost packets = 0, recvfrom err=0 2 000001 buf_seq#: handle 3 532.2931 sec. Sun May 18 13:38:53 2003 [from SYNC | start receiving bst] 4 532.2952 sec. Sun May 18 13:38:53 2003 [from SYNC | end of receiving bst] 6 0.000000 sec. clock diff from recv rle to send ('-' is recv faster) [W65072] 1 353.847046 Mbits (receiveing rate) 94256 -
7.2: rate control
-S burst interval and -br burst rateExample in 7.1 send packet train at the maximum system speed. User may want to send measurement burst in specific rate. -br and -S are the options for this purpose.
-br buadrate sets specific burst rate for test
Notice that the maximum controllable rate is different in different O.S.
baudrate can be in #[kK], #[mM], #[gG], or # b/s
-S sleep time specifies the interval between each packet trains (bursts)For example, netest -N -u -br 456m -S 1000 -tl 5 This command sets burst rate to 456 Mb/s for each train, and 1 second [the default is 5 seconds] interval between 5 trains.
This section is still under
Please send any suggestion on how to improve this page to author
| Back to the parent page | [an error occurred while processing this directive] |
| Updated Wednesday, 31-Oct-2001 11:00:00 PST |