Monitoring network with NetFlow and cflowd

Monitoring network bandwidth utilization with NetFlow and cflowd

While mrtg (the "Multi-Router Traffic Grapher") is useful for detecting long-term trends in network utilization and capacity management, it is not capable of tracing network activity in a router by port or source IP number. This is essential in determining the cause of network congestion. Packet sniffing, a frequently suggested alternative, does not always work in a switched environment. Modern-day networks often use switches instead of hubs and ACLs instead of traditional firewalls. This means that unless the offending computer is sending packets through the server or a computer-hosted firewall, packets often go directly from the user's broken computer to the router and are invisible to the server. This streamlining of network traffic is, of course, the main advantage of using switches; but it makes network accounting and intrusion detection more difficult.

Therefore, it is advantageous to monitor the traffic on the router itself. One way to do this is with Cisco's NetFlow, which is a part of Cisco's router IOS, to collect network statistics. Cisco doesn't make it easy to figure out how to do this. Their Documentation CD did little more than print garbage characters on my screen, and their Website often seems like a hopeless maze. Nonetheless, Netflow works quite well, and we use NetFlow to look for anomalies in network traffic patterns.

NetFlow periodically sends UDP packets to a specified server. The server captures the packets and writes the information to raw data files. Other software can then read the raw data files and formats it for human consumption. Cisco sells software for this purpose. However, it is quite expensive and only runs on Solaris and HP-UX. We use cflowd, which is freely available by http or ftp, to collect information from the router. This information can be parsed and analyzed to find out why, for example, your bandwidth utilization suddenly jumped to 100% and ping times shot up from 24 msec to 200 msec. Another advantage of cflow is that cflow actually comes with some documentation; and according to Google, there actually seems to be some people using it. However, it is difficult to configure.

INSTALLING AND CONFIGURING CFLOW

Obtain arts++ and cflowd from www.caida.org/tools/measurement/cflowd
Compile and install arts++ and cflowd. Note that cflowd will not compile until arts++ is installed. On a SuSE Linux 7.3 system, both packages compiled and installed without any problems.
Add /usr/local/arts/bin to PATH (in /etc/profile) and re-login.
Modify etc/cflowd.conf.example in the cflow source tree and copy it to /etc/cflowd.conf
Modify etc/cfdcollect.conf.example and copy it to /etc/cfdcollect.conf
Create directory /usr/local/arts/data/cflowd/flows. This is the directory cflowd will fill with raw files, as specified in its /etc/cflowd.conf file. If this directory is not present, cflowd will silently fail.
Create directory /usr/arts/data/cflowd/flows. This is the directory cfdcollect claims (at startup) it will fill with ARTS files. However, it doesn't actually use this directory. It also ignores the path specified in its /etc/cfdcollect.conf file. Cfdcollect actually writes ARTS files into a directory under '/' named 63.127.146.193 (assuming this is the IP number of your router).

On Cisco router, activate netflow output on each input interface. Then, configure Loopback0 to be the export source to ensure a consistent source address. Here are the commands we used to activate NetFlow on a Cisco 2620 router:

     router(config)#ip flow-export version 5 
     router(config)#ip flow-export destination 63.127.146.196 9991
     router(config)#ip flow-export source Loopback 0
     router(config)#interface Loopback 0
     router(config-if)#ip route-cache flow

The final router configuration will look something like this:

    interface Loopback0
      no ip address
    ip route-cache flow
    !
    interface FastEthernet0/0
      description To Office FastEthernet
      ip address 65.198.102.65 255.255.255.192 secondary
      ip address 63.127.146.193 255.255.255.192
      ip route-cache policy
      ip route-cache flow
      duplex auto
      speed auto
    ! 
    ip flow-export source Loopback0
    ip flow-export version 5
    ip flow-export destination 63.127.146.196 9991
    ip classless
    ip route 0.0.0.0 0.0.0.0 Serial0/0.1
    no ip http server

We picked 9991 as the port number. The number 63.127.146.196 is the IP address of the server on which cflowd is running. The "version 5" is essential for cflowd to work properly. Note that the version 5 option is only available on IOS 11.3 or later. Although it seems counter-intuitive, it is necessary to use the router's loopback interface as the source for the data, and not the Serial or FastEthernet interfaces. Also, note that since the data are sent using UDP, the router doesn't care whether the server receives it. This means less overhead for the router.

Activate (in /cflow/cflowd-2-1-b1/apps) the following three programs in the order shown:
cflowdmux/cflowdmux /etc/cflowd.conf cflowd/cflowd /etc/cflowd.conf cfdcollect/cfdcollect /etc/cfdcollect.conf
Note: it may take a couple of restarts before cfdcollect actually creates any ARTS files.

Cisco's NetFlow configuration guide may also be helpful in configuring the router.

OTHER UTILITIES

flowscan ( http://net.doit.wisc.edu/~plonka/flowscan/) = searches for patterns in data and creates graphs. Configuration of flowscan is described below.
artsdump = view contents of ARTS file
artstoc = print table of contents of ARTS file (a brief summary)
artsases = prints summary of each AS matrix in an ARTS file
artsnets = artsnets [-p][-s srcnet/mask][-d destnet/mask] infile(s)
prints a summary of each net matrix object in ARTS file.
Example: artsnets arts.19980817
artsportms = print a summary of each port matrix in ARTS file
Syntax: artsportms [-p] [-s srcport] [-d dstport] infile(s)
artsports = print summary of each port table object
artsprotos = print summary of protocol table objects
artsintfms = print summary of each interface matrix object
artsnexthops = print summary of each nexthop object
artsasagg = print summary of each AS object, and aggregates them over time
Syntax: artsasagg [-h hours][-o][-q] outfile infile
artsnetagg ... etc ...
artsportmagg
artsintfmagg
artsnexthopagg
cdflows = dumps raw flow data file into human-readable format
flowdump = similar to tcpdump. Uses C-like grammar to select particular flows from a raw flow file and dump to stdout.
cfdports = view current traffic port data
cfdprotos = view current traffic protocol data
cfdnets = view current traffic net matrix data
cfdases = view current traffic AS data

BEHAVIOR OF CFLOWD

The behavior of cflow is monitored by the messages it sends to syslog. For example, when cflowdmux is started, it should say something like:
cflowdmux: [I] cflowdmux (version cflowd-2-1-b1) started. cflowdmux: [I] created 2101248 byte packet queue shmem segment {CflowdPacketQueue.cc:247} cflowdmux: [I] attached to 2101248 byte packet queue at 0x401b9000 cflowdmux: [I] created semaphore: id 2195459 cflowdmux: [I] set UDP recv queue to 261040 bytes for fd 4 (port 9991)

When cflowd is started it should say something like:
cflowd: [I] cflowd (version cflowd-2-1-b1) started. cflowd: [I] got semaphore: id 2195459 cflowd: [I] attached to 2101248 byte packet queue at 0x403a3000
The specified number of files (ten in our case) are created in /usr/local/arts/data/cflowd/flows/ and filled with zeros.

When cfdcollect is started it should say something like:
cfdcollect: [I] cfdcollect (version cflowd-2-1-b1) started with 1 cflowd instances. cfdcollect: [I] connected to entropy.sh-hole.org:9991 cflowd: [I] sent data to 63.127.146.196:37235 cfdcollect: [I] entropy.sh-hole.org has data for 1 router. cfdcollect: [I] got data for router 63.127.146.193 from entropy.sh-hole.org cfdcollect: Outdirname /63.127.146.193 cfdcollect: Opening file /63.127.146.193/arts.20020323 cfdcollect: [I] wrote data for router 63.127.146.193 cfdcollect: [I] sleeping for 295 seconds.
It creates a zero-length file in /63.127.146.193 named arts.20020323. We also modified our copy of CflowdServer.cc to log the location of its data files.

Every five minutes, it wakes up and updates the data files:
cfdcollect: [I] awakened by alarm. cfdcollect: [I] connected to entropy.sh-hole.org:9991 cflowd: [I] sent data to 63.127.146.196:37238 cfdcollect: [I] entropy.sh-hole.org has data for 1 router. cfdcollect: [I] got data for router 63.127.146.193 from entropy.sh-hole.org cfdcollect: Outdirname /63.127.146.193 cfdcollect: Opening file /63.127.146.193/arts.20020323 cfdcollect: [I] wrote data for router 63.127.146.193 cfdcollect: [I] sleeping for 295 seconds.

If all goes well, the files in /usr/local/arts/data/cflowd/flows will be filled up with binary data.

TROUBLESHOOTING

Problem: 'flows' files not created.
Solution: Make sure path specified in /etc/cflowd.conf, e.g. '/usr/local/arts/data/cflowd/flows', exists and is writable.
Problem: No ARTS files
Solution: This means either the 'flows' files contain all zeros, cfdcollect is not running, or the router is not sending any information. Check IP numbers and ports in the configuration files. Make sure all three programs (cflowdmux, cflowd, and cfdcollect) are still running and that the correct config files are specified on the command line. The port number should be the same in all config files and on the router. Also, make sure the dummy directory '/usr/arts/data/cflowd/flows' is present.
Problem: 'flows' files contain all zeros.
Solution: This means no data is coming from router. Check router IOS version. Make sure 'ip route-cache flow' is applied to the correct interface. Do not activate AS on the router.
Problem: Can't find ARTS or other output files.
Solution: Edit CflowdServer.cc and add the lines:
syslog(LOG_ERR,"Outdirname %s", outdirname.str()); syslog(LOG_ERR,"Opening file %s", outfilename.str());
at line 580 and recompile. This will put the filename and path in the syslog.

CONFIGURATION FILE CONTENTS

/etc/cflowd.conf
OPTIONS { LOGFACILITY: entropy.sh-hole.org TCPCOLLECTPORT: 9991 PKTBUFSIZE: 2097152 TABLESOCKFILE: /usr/local/arts/etc/cflowdtable.socket FLOWDIR: /usr/local/arts/data/cflowd/flows FLOWFILELEN: 1000000 NUMFLOWFILES: 10 MINLOGMISSED: 1000 } COLLECTOR { HOST: 63.127.146.196 # IP address of central collector ADDRESSES: { 63.127.146.196 } } CISCOEXPORTER { HOST: 63.127.146.193 # IP address of Cisco sending data. ADDRESSES: { 63.127.146.193, # Addresses of interfaces on Cisco 65.198.102.65 } # sending data. CFDATAPORT: 9991 # Port on which to listen for data. SNMPCOMM: 'snmp_password_here' # SNMP community name. COLLECT: { protocol, portmatrix, ifmatrix, nexthop, netmatrix, asmatrix, tos, flows } }

/etc/cfdcollect.conf
system { logFacility: entropy.sh-hole.org # Syslog to local6 facility. dataDirectory: /usr/local/arts/data/cflowd filePrefix: arts pidFile: /usr/local/arts/etc/cfdcollect.pid } cflowd { host: entropy.sh-hole.org tcpCollectPort: 9991 minPollInterval: 300 }

TESTING - ON ROUTER

The flow cache can be printed by typing the command:
router#show ip cache verbose flow
or
router#show ip cache flow
This prints a table similar to this:
IP packet size distribution (12812 total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .510 .239 .047 .007 .006 .008 .014 .018 .000 .002 .001 .001 .000 .000 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .001 .001 .003 .007 .126 .000 .000 .000 .000 .000 .000 IP Flow Switching Cache, 278544 bytes 21 active, 4075 inactive, 1932 added 33835 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) -------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-Telnet 54 0.0 43 95 0.0 22.0 13.4 TCP-FTP 2 0.0 10 140 0.0 6.1 15.8 TCP-WWW 161 0.0 14 968 0.0 0.9 2.5 TCP-SMTP 5 0.0 156 42 0.0 3.7 1.4 TCP-NNTP 114 0.0 41 58 0.0 37.5 13.0 TCP-other 424 0.0 2 312 0.0 0.4 13.1 UDP-DNS 99 0.0 2 73 0.0 1.9 15.4 UDP-NTP 11 0.0 1 76 0.0 0.0 15.5 UDP-other 553 0.0 2 149 0.0 0.2 15.4 ICMP 28 0.0 1 63 0.0 0.0 15.4 Total: 1451 0.0 8 261 0.0 4.2 13.0 SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/0 63.127.146.195 Null 65.198.102.94 06 0C36 0071 1 Fa0/0 63.127.146.201 Null 63.127.146.255 11 0089 0089 1 Fa0/0 63.127.146.201 Null 63.127.146.255 11 008A 008A 1 Fa0/0 63.127.146.198 Null 63.127.146.255 11 008A 008A 2 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 0453 0050 1 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 0452 0050 14 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 0451 0050 13 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 0450 0050 8 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 0454 0050 1 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 044F 0050 8 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 044E 0050 5 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 044D 0050 5 Fa0/0 65.198.102.94 Se0/0.1 64.4.20.253 06 044C 0050 5 Fa0/0 63.127.146.195 Se0/0.1 198.6.0.25 06 0077 8EB3 18 Fa0/0 63.127.146.196 Local 63.127.146.193 06 9193 0017 59 Fa0/0 63.127.146.195 Null 65.198.102.87 06 008B 0405 1 Fa0/0 63.127.146.198 Null 65.198.102.127 11 008A 008A 2 Fa0/0 63.127.146.195 Se0/0.1 198.6.0.26 06 0077 F65C 35 Fa0/0 65.198.102.94 Se0/0.1 64.4.56.7 06 044A 0050 6 Fa0/0 63.127.146.196 Se0/0.1 63.215.153.207 06 0017 8024 2 Fa0/0 63.127.146.196 Se0/0.1 63.215.153.207 06 0017 8025 32
If you don't mind logging onto the router all the time, you could theoretically use the 'show ip cache flow' command to monitor your network, instead of using cflow.

TESTING - ON SERVER

'netstat -an | grep 9991' should show open ports on 9991:
tcp 0 0 0.0.0.0:9991 0.0.0.0:* LISTEN udp 0 0 0.0.0.0:9991 0.0.0.0:*
'ipcs -a' should show a shared memory segment and a semaphore owned by the owner of cflowdmux (which is last entry as shown here).
------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 5537792 root 600 1056768 4 dest 0x00000000 5603329 wwwrun 600 24 4 dest 0x00000000 5636098 wwwrun 600 368644 3 dest 0x00280267 327683 root 644 1048576 1 0x0001fe1b 5668868 root 744 2101248 2 ------ Semaphore Arrays -------- key semid owner perms nsems status 0x00000000 2162688 wwwrun 600 1 0x00280269 131073 root 666 14 0x00035c38 2031618 root 777 2 0x0001fe1b 2195459 root 777 2 ------ Message Queues -------- key msqid owner perms used-bytes messages
Use 'ps -aux | grep cflowd' to get the PID of cflowd (for example, 5257). Then use 'lsof -p 5257' to see what files it is reading from and writing to.
COMMAND PID NAME cflowd 5257 /home/tjnelson/cflow/cflowd-2-1-b1/apps cflowd 5257 / cflowd 5257 /home/tjnelson/cflow/cflowd-2-1-b1/apps/cflowd/cflowd cflowd 5257 /lib/ld-2.2.so cflowd 5257 /lib/libnsl.so.1 cflowd 5257 /usr/lib/libstdc++-3-libc6.2-2-2.10.0.so cflowd 5257 /lib/libm.so.6 cflowd 5257 /lib/libc.so.6 cflowd 5257 /usr/local/arts/data/cflowd/flows/63.127.146.193.flows.0 cflowd 5257 /dev/ttyp0 cflowd 5257 /dev/ttyp0 cflowd 5257 /dev/ttyp0 cflowd 5257 socket cflowd 5257 /usr/local/arts/etc/cflowdtable.socket cflowd 5257 *:9991 (LISTEN)
Finally, check to see that the data files are being created.
entropy:/usr/local/arts/data/cflowd/flows# ls -lart total 1961 drwxr-xr-x 3 root root 56 Mar 22 20:49 .. -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.9 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.8 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.7 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.6 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.5 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.4 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.3 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.2 -rw-r--r-- 1 root root 1000000 Mar 22 20:50 63.127.146.193.flows.1 -rw-r--r-- 1 root root 1000000 Mar 23 12:20 63.127.146.193.flows.0 drwxr-xr-x 2 root root 436 Mar 23 13:56 . ls -l /63.127.146.193 total 136 -rw-r--r-- 1 root root 131731 Mar 23 18:22 arts.20020323
Even though the file dates don't change, the raw binary 'flows' files are overwritten by cfdcollect every five minutes, starting with flows.0.

ANALYZING THE DATA

Use the arts++ programs to read the binary files created by cflowd. For example, artsportms, after a little reprogramming to fix the print format error, gives this highly informative output (only part of the output is shown):
bash$ artsportms arts.20020323 router: 63.127.146.193 ifIndex: 2 (FastEthernet0/0 65.198.102.65) period: 03/25/2002 10:57:45 - 03/25/2002 11:02:47 EST srcPort dstPort Pkts Pkts/sec Bytes Bits/sec ------- ------- ------------- ------------- ------------- ------------- 1184 6699 6856 22.702 2844016 75338.2 1186 6699 5693 18.851 2815678 74587.5 6699 2992 2321 7.68543 1301511 34477.1 1676 6699 2744 9.08609 1079606 28598.8 6257 6257 388 1.28477 96820 2564.77 110 2667 47 0.155629 51106 1353.8 2339 80 40 0.13245 161854 28.742 2338 80 39 0.129139 16135 427.417 3282 80 45 0.149007 14716 389.828 3283 80 38 0.125828 14314 379.179 138 138 41 0.135762 10064 266.596 119 51482 97 0.321192 6652 176.212 80 3093 6 0.0198675 606 0160.53 80 22430 12 0.0397351 534 1141.483 80 25225 12 0.0397351 530 5140.53 2355 80 17 0.0562914 513 2135.947 2354 80 17 0.0562914 510 9135.338 4556 25 14 0.0463576 500 2132.503 119 51579 62 0.205298 4191 111.02 119 58343 54 0.178808 4167 110.384 2364 80 74 0.245033 4112 108.927 2363 80 71 0.235099 3992 105.748 2358 80 13 0.0430464 396 9105.139 2357 80 12 0.0397351 391 7103.762 1297 80 8 0.0264901 331 087.6821 119 58516 46 0.152318 3242 85.8808 1306 80 10 0.0331126 323 585.6954 1606 80 28 0.0927152 312 282.702 1700 6699 74 0.245033 3063 81.1391 1607 80 27 0.089404 3057 80.9801 1604 80 26 0.0860927 302 880.2119 2337 80 38 0.125828 2807 74.3576 2370 80 34 0.112583 2633 69.7483 1556 80 43 0.142384 2163 57.298 1601 80 28 0.0927152 216 157.245 2356 80 21 0.0695364 214 356.7682 2359 80 21 0.0695364 214 356.7682 2346 80 23 0.0761589 211 255.947 1550 80 26 0.0860927 208 955.3377 [...]

This shows immediately that someone on our network was running Napster (port 6699). With this knowledge, you can take appropriate action if necessary.

Another troublesome port is 3163, used by Kazaa. Outgoing traffic from a single PC on these two ports can easily saturate a T1.

To fix the format error, change lines 207-216 in the file artsportms.cc to the following:
if (PortEntryMatches(portEntry)) { cout << setfill(' '); cout << portEntry->Src() << '\t' << portEntry->Dst() << '\t' << portEntry->Pkts() << '\t' <<" " << (double)portEntry->Pkts() / timeInterval << '\t' << '\t' << portEntry->Bytes() << '\t' <<((double)portEntry->Bytes() * 8.0) / timeInterval << endl; entriesShown++;}

You can get reams of data from your router using this software. If you convert it to pie charts and make PowerPoint presentations to management to show them all the packets your network is getting, it should be no problem to convince them that you need more stuff.

FLOWSCAN

FlowScan ( http://www.caida.org/tools/utilities/flowscan) is software that converts the raw flow files to graphs. It converts your raw flow files to "rrd" files that can be converted into pie charts and other types of graphs. Here is the procedure for compiling FlowScan:

Make sure cflow is installed and is higher than 1.024 and is working

Obtain patch for cflowd at

http://net.doit.wisc.edu/~plonka/cflowd/

or, get flow-tools and use flow-capture instead, as described here:

 http://net.doit.wisc.edu/~plonka/list/flowscan/archive/1117.html

Install patch

cd cflowd-2-1-b1
patch -p0 < ../cflowd-2-1-b1-djp.patch

Rebuild and reinstall the new cflowd. Start the patched cflowd with the command

cflowd -s 300 -O 0 -m /path/to/cflowd.conf 
i.e.,
/usr/local/arts/sbin/cflowd -s 300 -O 0 -m /etc/cflowd.conf
or 
/usr/local/arts/sbin/cflowd /etc/cflowd.conf

and wait a couple hours to make sure the files are converted. It should write the following to the syslog:

cflowd[23823]: [I] cflowd (version cflowd-2-1-b1) started.
cflowd[23823]: [I] got semaphore: id 65537
cflowd[23823]: [I] attached to 2101248 byte packet queue at 0x403a3000

Check to make sure the patched cflowd is running by typing:
strings /usr/local/arts/sbin/cflowd | grep usage
It should say:
usage cflowd [ -s logtimeout_seconds [ -O skip_output_if [-m] ] ]
If it doesn't give a usage string, it is not the patched version.
install ksh
www.research.att.com/sw/download

Install rrdtool. RRDtool is described at www.caida.org

    make site-perl-install
    cd /usr/local
    ln -s rrdtool-1.0.35 rrdtool
    export PATH=$PATH:/usr/local/rrdtool:/usr/local/rrdtool/bin:\
       /usr/local/arts/bin/

install Boulder::Stream (perl module)

   http://stein.cshl.org/software/boulder/boulder.tar.gz
   perl Makefile.PL
   make 
   make install

install Net::Patricia (perl module)

   http://www.cpan.org/modules/by-module/Net/Net-Patricia-1.010.tar.gz 
   perl Makefile.PL
   make
   make install

install ConfigReader

   http://cpan.valueclick.com/modules/by-module/\
   ConfigReader/ConfigReader-0.5.tar.gz
   cd /usr/lib/perl5/site_perl/5.6.0/
   mkdir ConfigReader
   cp *.pm /usr/lib/perl5/site_perl/5.6.0/ConfigReader

Install cflow perl module (totally distinct from cflowd)

   http://www.caida.org/dynamic/archives/cflowd/0243.html
   http://net.doit.wisc.edu/~plonka/Cflow/
   perl Makefile.PL
   make
   make install

Create graphs directory

   cd FlowScan-1.006
   make
   cp graphs.mf $PREFIX/graphs/Makefile   (substituting whatever $PREFIX is)
   cd $PREFIX/graphs
   make
   cd -
   make install

edit cf/flowscan.cf to make sure the path to the raw flow files created by cflow is correct.
FlowFileGlob /usr/local/arts/data/cflowd/flows/flows.*:*[0-9]
Move flowscan.cf to the directory where flowscan is located.
Edit CampusIO.cf or SubNetIO.cf as desired. One of these files must be listed in flowscan.cf. Move these files to the same directory where flowscan is located.

Edit our_subnets:

     SUBNET=63.127.146.192/26
     DESCRIPTION=This is our first subnet, don't you go calling it.
     SUBNET=65.198.102.64/26
     DESCRIPTION=Our second subnet

Substitute your own subnet for the above numbers.

Edit local_nets

    SUBNET=63.127.146.192/26
    DESCRIPTION=our network

Move the above files to flowscan directory
chmod a+x flowscan
Wait at least 10 minutes to ensure the flow files created by the modified cflowd replace the old versions.

Check to make sure the flows.current file is being created:

    ls -l /usr/local/arts/data/cflowd/flows/flow*
    -rw-r--r-- 1 root  root  36245 May  9 16:50 flows.20020509_16:50:36-0400
    -rw-r--r-- 1 root  root   3795 May  9 16:51 flows.current

Start flowscan with no arguments: ./flowscan
It should say:

      NextHops and OutputIfIndexes are undefined.
      Identifying outbound flows based solely on destination address ...
      Loading "local_nets" ...
      Loading "our_subnets" ...
      2002/05/09 16:54:11 working on file /usr/local/arts/data/cflowd\
         /flows/flows.20020509_16:50:36-0400...
      2002/05/09 16:54:11 flowscan-1.020 SubNetIO: Cflow::find took  \
         0 wallclock secs ( 0.13 usr +  0.00 sys =  0.13 CPU) for 36245 \
         flow file bytes, flow hit ratio: 502/659
      ERROR updating /usr/local/arts/data/cflowd/flows/graphs/\
         63.127.146.192_26.rrd: illegal attempt to update using\
         time 1020977436 when last update time is 1020977436 \
         (minimum one second step)
      2002/05/09 16:54:12 flowscan-1.020 SubNetIO: report took\
         1 wallclock secs ( 0.12 usr +  0.04 sys =  0.16 CPU)
      sleep 300...

Every five minutes thereafter, it will say hypnotically:

      sleep 300...
      sleep 300...
      sleep 300...
      sleep 300...

Flowscan should create "rrd" files in /usr/local/arts/data/cflowd/flows/graphs or wherever specified in the files SubNetIO.cf or CampusIO.cf. For instance, if SubNetIO.cf contains the line:

   OutputDir graphs

Typing:

   ls -l  /usr/local/arts/data/cflowd/flows/graphs

should give a long list of .rrd files:

     -rw-r--r-- 1 root  root    95660 May  9 16:58 0:0.rrd
     -rw-r--r-- 1 root  root   253780 May  9 16:58 63.127.146.192_26.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 MCAST.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 Quake.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 RealAudio.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 arcp_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 arcp_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftp-data_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftp-data_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftpPASV_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftpPASV_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftp_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 ftp_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 http_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 http_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 icmp.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 nntp_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 nntp_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 rtsp_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 rtsp_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 smtp_dst.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 smtp_src.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 tcp.rrd
     -rw-r--r-- 1 root  root   253780 May  9 16:58 total.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 udp.rrd
     -rw-r--r-- 1 root  root   190532 May  9 16:58 unknown.rrd

If these graphs are not created, it usually means flowscan did not find the flow file, probably because either an unpatched version of cflowd is running or the FileGlob is not specified correctly in flowscan.cf.

Create graphs using RRDtool. For example:

     rrdtool graph a.gif --start 0 --end 1 \
     DEF:graph=/usr/local/arts/data/cflowd/flows/graphs/0:0.rrd:\
     bytes:AVERAGE \
     LINE2:mygraph#FF000

This will create a graph named a.gif using the specified rrd file.

LIMITATIONS

The visualization software (FlowScan) is quite tedious to install.
Cflowd does not have sufficient granularity to show source IPs. This means you still need to use iptraf, flowdump, or tcpdump to find out which computer is broken or infected.
For example:
flowdump /usr/local/arts/data/cflowd/flows/63.127.146.193.flows.0 > stuff
to see everything or
flowdump -e srcport=6699 \ /usr/local/arts/data/cflowd/flows/63.127.146.193.flows.0 > stuff
to see all the Napster traffic. Unfortunately, the data produced by flowdump are not entirely correct -- the date and time are wrong, for example, which makes it difficult to correlate the results with actual traffic.
To visualize router traffic using iptraf, tap into the router's ethernet port by connecting an old-fashioned 8-port hub between it and your network and then connecting a Linux box running iptraf in promiscuous mode to one of the six unused ports on the hub. Alternatively, managed switches like the H/P Procurve allow you to select one port to monitor traffic. Once you have the IP number, it is easy to track down the offending PC by looking at the dhcpd.leases file.

UPDATE - Mar 12, 2010

Cflowd no longer compiles in Suse 11.0. To get it running again after upgrading the system, we did the following:

Install arts++-1.1.a13. This version compiled and installed in Suse 11.

Rename the C++ include files to fix the "iostream.h not found" errors when compiling cflowe:

 cd /usr/include/c++/4.3
cp iostream iostream.h
cd /usr/include/c++/4.3/backward/
cp strstream strstream.h
cd /usr/include/c++/4.3
cp vector vector.h

Even with these changes, cflowd-2-1-b1 no longer compiles:

make[3]: Entering directory `/home/tjnelson/cflow/cflowd-2-1-b1/snmp++/classes/src'
g++ -D__unix__ -I../include/snmp++ -I./ -g -O2 -c oid.cpp -o oid.o
In file included from oid.cpp:59:
../include/snmp++/oid.h:181: error: 'vector' has not been declared
../include/snmp++/oid.h:181: error: expected ',' or '...' before '<' token
oid.cpp:503: error: variable or field 'get_oid' declared void
oid.cpp:503: error: 'vector' was not declared in this scope
oid.cpp:503: error: expected primary-expression before '>' token
oid.cpp:503: error: 'outVect' was not declared in this scope
make[3]: *** [oid.lo] Error 1
make[3]: Leaving directory `/home/tjnelson/cflow/cflowd-2-1-b1/snmp++/classes/src'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/tjnelson/cflow/cflowd-2-1-b1/snmp++/classes'
make[1]: *** [classes/lib/libsnmp++.a] Error 2
make[1]: Leaving directory `/home/tjnelson/cflow/cflowd-2-1-b1/snmp++'
make: *** [all] Error 2

So we restored the old versions of cflowd, cflowdmux, and cfdcollect from backup.

Unfortunately, running ldd cflowd showed that it also needed an old library:
ldd cflowd libstdc++-libc6.2-2.so.3 => not found
So we restored this one from a backup to /usr/lib and ran ldconfig.

Re-create the directories

cd /usr
mkdir 111.111.111.111   (change to IP of router)
ln -s /usr/111.111.111.111  /111.111.111.111
mkdir /usr/local/arts/data/cflowd/flows
mkdir /usr/local/arts/etc

Change the startup script to

cd /home/tjnelson/cflow/
cflowdmux /etc/cflowd.conf
cflowd /etc/cflowd.conf 
cfdcollect /etc/cfdcollect.conf
cd -

Restore cfdcollect.conf and cflowd.conf to /etc
Start it up ... she lives!

Back