Chapter 8. NetSnoop

NetSnoop captures and displays every packet of information that travels through the network. Analyzer is a visual interface to NetSnoop. You may find that Analyzer is easier to use than NetSnoop, because Analyzer displays a decoded packet in a way that is easy to understand.

If Analyzer is easier to use, then why use NetSnoop? With NetSnoop you can:

In addition, you can use NetSnoop with IRIX utilities and the shell to:

This chapter explains how to:

In addition, a variety of NetSnoop examples are provided. Using NetSnoop to get protocol information that can be used to create filters is explained in “Using NetSnoop to Find Filter Operands” in Chapter 10. For complete information on NetSnoop command line options and resources, see the netsnoop(1M) manual page in Appendix F, “NetVisualyzer Manual Pages.”


Note: You must be superuser to use NetSnoop while direct snooping. To use NetSnoop with RPC snooping, you must be authorized in the /usr/etc/rpc.snoopd.auth file. See “Authorizing NetVisualyzer Users for Snooping” in Chapter 1 and Appendix B, “Authorization Reference,” for details.


Starting and Stopping NetSnoop

You can start NetSnoop from the netvis directory view, the command line, or a shell script. For example, to start NetSnoop from the command line give this command as root:

netsnoop 

When you first start NetSnoop, it looks for the .netsnooprc file in your home directory. You can specify NetSnoop options in the .netsnooprc file. If the ~/.netsnooprc file does not exist and no interface is specified with the –i command line option, NetSnoop starts snooping on the default interface on your workstation and captures packets until you stop it. See “NetSnoop Configuration File” in Appendix D for more information about the .netsnooprc file.

The command line options to NetSnoop allow you to snoop on an interface you specify, snoop for a specified time period, and collect a specified number of packets. You can also list information about a specific protocol or field of a protocol. The netsnoop(1M) manual page in Appendix F describes all of the NetSnoop options.

You can specify NetSnoop options in these ways:

  • on the command line, for example:

    netsnoop –c 5 
    

  • in the .netsnooprc configuration file, for example:

    option –c 5 
    

Options on the command line override options in the .netsnooprc configuration file.

To stop snooping, generate a keyboard interrupt (for example, <Ctrl-C>).

Specifying an Interface to NetSnoop

A useful command line option is –i interface, which you can use to snoop on a remote Data Station or on a specific interface on your workstation. You do not need to be superuser when you use the –i option. The format is:

netsnoop -i station:ifname

where station is the name or IP address of a Data Station on which you have snooping authorization, and ifname is the name of the interface you want to use. Both station and ifname are optional. Table 8-1 shows the different forms of station:ifname, the type of snooping used, and the permission requirements.

Table 8-1. netsnoop –i station : ifname Forms

station : ifname Form

Type of Snooping

Authorization Required

station : ifname

RPC snooping is done on station using the interface ifname.

User must be authorized in /usr/etc/rpc.snoopd.auth on station.

station

If station is the name or address of the local host (from the Name or Address columns of netstat -i output or an IP address), direct snooping is used. If station is a remote host, RPC snooping on the default interface of station is used.

User must be superuser for direct snooping; user must be authorized in /usr/etc/rpc.snoopd.auth on station for RPC snooping.

station :

RPC snooping is done on the default interface of station.

User must be authorized in /usr/etc/rpc.snoopd.auth on station.

ifname

Direct snooping (on the local host) using the interface ifname.

User must be superuser.

For example:

netsnoop -i reddog:fxp0 

snoops on the additional EFast Ethernet board installed on the Data Station named reddog.

If you want to snoop on the default interface of a Data Station you do not need to specify ifname. Two examples are:

netsnoop -i yeti: 
netsnoop -i yeti 

In the first example, RPC snooping is used regardless of whether yeti is the local host or not. In the second example, direct snooping is used if yeti is the local host and RCP snooping is used if yeti is a remote host. In all cases, the default interface is used.

When only an interface is given, direct snooping on the local host is used. For example:

netsnoop -i fxp0 

Interpreting NetSnoop Output

If you specify a filter on the NetSnoop command line, NetSnoop prints an internal, “corrected” version of the filter enclosed in brackets ([]) before it starts capturing so you can verify the filter.

An example of NetSnoop output looks like this:

0198:   len  166   time 15:49:27.446
    ether:  src 8:0:69:2:29:38/SGI        dst ff:ff:ff:ff:ff:ff        
    ip:     src paganini.wpd.sgi.com      dst 192.26.79.255            
    udp:    sport 520 (rip)               dport 520 (rip)             
    rip:    cmd RESPONSE

0199:   len   86   time 15:49:27.456
    ether:  src 8:0:69:6:8e:f8/SGI        dst 8:0:69:6:17:88/SGI       
    ip:     src whizkid.wpd.sgi.com       dst outland.wpd.sgi.com      
    tcp:    sport 6000 (x11)              dport 4984                  

0200:   len  118   time 15:49:27.476
    ether:  src 8:0:69:6:8e:f8/SGI        dst 8:0:69:6:17:88/SGI       
    ip:     src whizkid.wpd.sgi.com       dst outland.wpd.sgi.com      
    tcp:    sport 6000 (x11)              dport 4984                  

0201:   len  150   time 15:49:27.476
    ether:  src 2:cf:1f:11:46:32/CMC      dst 2:cf:1f:11:9:13/CMC      
    ip:     src bigsur.wpd.sgi.com        dst magrathea.wpd.sgi.com    
    udp:    sport 1023                    dport 2049 (sunrpc)         
    sunrpc: xid 151111          direction CALL      credtype AUTH_UNIX 
    nfs:    proc LOOKUP    

0202:   len  170   time 15:49:27.476
    ether:  src 2:cf:1f:11:9:13/CMC       dst 2:cf:1f:11:46:32/CMC     
    ip:     src magrathea.wpd.sgi.com     dst bigsur.wpd.sgi.com       
    udp:    sport 2049 (sunrpc)           dport 1023                  
    sunrpc: xid 151111          direction REPLY     stat MSG_ACCEPTED
    nfs:    proc LOOKUP         status NFS_OK            

For each packet captured, NetSnoop shows the sequence number of the packet, the length of the packet in bytes, and the reception time.

Next, NetSnoop decodes protocol data. Data is grouped into frames labeled by protocol (in the example, frames are labeled as ether, ip, udp, rip, tcp, sunrpc, and nfs).

Each frame is decoded as a sequence of fields. In the previous example, src, dst, sport, and dport are a few of the fields in packet 0198. src shows the source address, dst shows the destination address, sport shows the UDP source port, and dport shows the UDP destination port. The source's physical address is 8:0:69:2:29:38.

Getting Statistics on Dropped Packets from NetSnoop

To get statistics on dropped and captured packets from NetSnoop, use the –s option. It lists the number of packets received by the network interface and records how many were subsequently dropped due to kernel resource limits. Keep the following points in mind when interpreting the results:

  • When snooping on a specified interface, the statistics you get with the –s option often show more packets seen at the interface than are returned. This result stems from the lag between NetSnoop getting all the packets it wants and signaling snoopd that it then wants statistics. The statistics may show that packets were dropped, but this is not a problem because the packets were dropped after NetSnoop received all the packets of interest.

  • When NetSnoop is snooping locally on the default interface, dropped packets are indicated by gaps in the sequence numbers of the packets captured. The sum of the values for packets dropped at network interface and packets dropped at socket buffer equals the number of missing sequence numbers.

  • When RPC snooping is used, there are no gaps in the packet sequence numbers because sequence numbers are re-numbered on the receiving side, even though packets may have been dropped. When direct snooping is used, packets are not re-numbered.

Configuring NetSnoop for Best Performance

This section gives recommendations designed to minimize the possibility that NetSnoop will drop packets and to maximize NetSnoop performance. They are conservative recommendations for worst-case network traffic scenarios. They should be used as starting points for adjusting the values to your needs.

NetSnoop command lines with the following options give the best results:

netsnoop -o output -b buffer_size -l packet_length simple_filter

The –o option causes the raw packets to be put into the file output. There is no decoding or formatting of the data.

buffer_size should be less than 200 for networks with traffic that is mostly MTU-size and less than 50 for network traffic made up of small packets. With large memory configurations, this argument can produce dramatic improvements. Leaving it out altogether or using the –c option with these values is also satisfactory.

packet_length is the number of bytes of a captured packet saved by NetSnoop. It does not include the MTU header (14 bytes for Ethernet and 16 bytes for FDDI). A reasonable length for many cases is about 200 bytes. This number allows NetSnoop to capture all the protocol headers currently supported plus some user data.

Using NetSnoop with direct snooping rather than RPC snooping is the optimal configuration for packet capture because there is no interprocess communication overhead. There is no communication latency due to stopping and getting statistics. RPC snooping on a remote interface gives lower performance than RPC snooping on a local interface.

There are two types of simple_filters: a filter that can be translated into the kernel's filter string so that the kernel can do filtering instead of snoopd, or an Ethernet address that enables the Ethernet hardware to do the filtering instead of snoopd. Examples of simple_filter expressions that can be filtered by the kernel are:

src == 8:0:69:4:0:16
ip
host(8:0:69:1f:0:4)

Other NetVisualyzer tools should not be run while running NetSnoop because they will dramatically affect NetSnoop performance. Performance is improved if the Data Station is dedicated to NetVisualyzer, and no applications or services beyond the minimum are running.

NetSnoop Examples

The following examples show how to use NetSnoop to track an overloaded Ethernet gateway and to monitor remote use of resources.

Using NetSnoop to Track an Overloaded Ethernet Gateway

This example explains how to troubleshoot a router or gateway that displays performance problems due to an unusually large amount of traffic. To begin, use NetSnoop to track down the node from which the packets originate:

netsnoop -c 20 -t 2 -i ec0 -y dst=BROADCAST or \ dst=8:0:69:2:26:11 

where:

dst=BROADCAST or dst=8:0:69:2:26:11  


filters all broadcast packets plus packets specifically destined for one of the Ethernet controllers in the gateway (physical address 8:0:69:2:26:11).

–c 5  

captures 20 packets that match the filter. All 20 packets are buffered before they are decoded.

–t 2  

limits captures to at most 2 seconds; capture may stop sooner if the 20 packets specified by the –c option are received.

–i ec0  

specifies snooping on the interface ec0.

–y  

causes NetSnoop to consult NIS in order to translate IP addresses to node names. The output from this command may contain node names that are not translated. This may be because they are new nodes and therefore unknown to NIS.

A portion of the output looks like this:

0002:   len  310   time 16:26:45.437
    ether:  src mountain.wpd.sgi.com      dst 8:0:69:2:26:11/SGI
    ip:     src littlesnoop.wpd.sgi.com   dst yeti.wpd.sgi.com
    tcp:    sport 1036                    dport 639

0003:   len   60   time 16:26:45.635
    ether:  src 8:0:69:6:2c:83/SGI        dst ff:ff:ff:ff:ff:ff
    arp:    op REQUEST   
    arpip:  sha 8:0:69:6:2c:83/SGI        spa whizzer.wpd.sgi.com
            tha 0:0:0:0:0:0               tpa ram.wpd.sgi.com

0004:   len   60   time 16:26:46.093
    ether:  src 8:0:69:6:1c:99/SGI        dst 8:0:69:2:26:11/SGI
    ip:     src patton.wpd.sgi.com        dst babylon.wpd.sgi.com
    tcp:    sport 1021                    dport 513 (rlogin)

0005:   len   60   time 16:26:46.301
    ether:  src 8:0:69:6:1c:99/SGI        dst 8:0:69:2:26:11/SGI
    ip:     src patton.wpd.sgi.com        dst babylon.wpd.sgi.com
    tcp:    sport 1021                    dport 513 (rlogin)

The ether: src lines show the nodes that generate the traffic. In this example the traffic originates from several sources.

You can use a variation of this command to find only the traffic forwarded by the router:

netsnoop -c 20 -t 2 -i ec0 -y '(dst=BROADCAST or \ dst=8:0:69:2:26:11) and ip.dst != gateway' >> /tmp/snoop.log 

The ignored destination, gateway, is the node name of the IP router. This filter captures traffic received by the router but not destined for it at the IP layer (and therefore probably destined for a node on the other side). The Ethernet destination (dst) is the router's Ethernet address.

Using NetSnoop to Track Remote Use of Resources

A common source of poor system performance is the use of system resources from a remote system. In many cases, this use of resources is either unnecessary or completely unknown to the user. For example, you can accidentally impact the resources of a remote system by running the find(1) command from the root directory while a remote file system is mounted.

The following example uses NetSnoop to compute a histogram of the traffic from an NFS server's clients. The most active client is displayed first. Note the use of awk(1) and sort(1) in ordering and managing NetSnoop's output.

netsnoop -c 100 ip.dst=server and nfs | awk \ 
'$1 == "ip:" { hist[$3]++ } \ 
END { for (client in hist) \ 
printf "%-25.25s %d\n", client, hist[client] }' \ 
| sort +1 -n -r 

In this command, server is the server's name. The awk script looks at NetSnoop output lines that begin with ip:. They have this form:

ip: src client dst server

The awk command interprets the variable $1 as ip:, $2 as src, and $3 as client. Given traffic from three clients, 192.82.172.25, squaw, and stingray, the awk code builds an associative array, hist, indexed by client node name:

hist["192.82.172.25"]
hist["squaw"]
hist["stingray"]

The awk code prints the histogram at the end of input, sorted numerically based on the second field, and displays the output in reverse. The most active client appears first in the sorted output. An example of the output is:

squaw.eng.sgi.com         70
stingray.eng.sgi.com      24
192.82.172.25             6

Using NetSnoop for Error Snooping

You can start the following NetSnoop command in a window and use it to collect data for a period of days. A large sample, such as the sample created after days of monitoring for errors, can be useful in tracking down intermittent network problems.

netsnoop -vv -x -e any > /usr/tmp/error.trace & 

This command produces a hexadecimal dump (–x) of any packet with an error (–e any). An example of the output looks like this:

0000:!  len  100   time 15:25:03.840 CHECKSUM
    ether:  src 2:cf:1f:10:41:73/CMC      dst 8:0:69:2:19:58/SGI       
            type ip    
    ip:     v 4       hl 5      tos 0     len 84    id 21896  off 0
            ttl 60    p udp     sum 0x2e06
            src gate-orange.wpd.sgi.com   dst work.wpd.sgi.com         
    udp:    sport 691                     dport 887                   
            len 64    sum 0xa293
    00042:                                2a d2 0d ce 00 00            *.....
    00048:  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    00064:  00 00 00 00 00 01 00 00 00 0c 74 63 70 6d 75 78  ..........tcpmux
    00080:  20 31 2f 74 63 70 00 00 00 05 31 2f 52 55 b5 aa   1/tcp....1/RU..
    00096:  52 d5 aa 55                                      R..U