Page 1 of 1

How to troubleshoot NetFlow Exports are being received

Posted: Mon Aug 24, 2009 11:49 am
by digitoll
Check the license key is valid and is for the correct version of CySight:
  • Click 'Configuration' -> 'Administrator' -> 'License' to load License Details page.
    The License Key type will be displayed in this page.


Image
  • If collection is running normally, the License Key type will indicate if software version is Standard Version or Enterprise Version.
  • If there is a mismatch between the License Key type and the software version, CySight will be unable to collect NetFlow.
How to check the availability of a NetFlow stream:
  • FrontEnd:
    • In the FrontEnd, after logging in, the Home page will be displayed.

      Check the Device screen and confirm that Devices have been automatically created.

      After 1 minute of flows, check the Home page or the Overview page to see flows are being received.

      The Overview page can be accessed by clicking the 'Overview' button on top of any other page. This page will show current active devices. This page will refresh itself every minute to ensure data is always up to date.

      Image

      Note: the traffic data stopping proceeding in Overview page does not necessarily means no NetFlow stream is being received. Some mis-configuration can also lead to the stopping. Troubleshooting is needed in this case.
    Linux:
    • tcpdump

      The tcpdump command will enable you to check if the incoming UDP stream is reaching the collector

      The tcpdump command to check the incoming UDP stream is:
      • tcpdump udp
      or, to a particular port:
      • tcpdump udp port [port number]
      Note: Please make sure the port number is correct.

      If there are incoming UDP packets, tcpdump will keep displaying lines like the following:

      17:51:11.659985 IP 192.168.0.100.9912 > 192.168.0.88.2055: UDP, length 1416
      17:51:13.290553 IP 192.168.0.241.51890 > 192.168.0.88.2055: UDP, length 1420
      17:51:15.118116 IP 192.168.0.100.9912 > 192.168.0.88.2055: UDP, length 1416
      17:51:17.016780 IP 192.168.0.100.9912 > 192.168.0.88.2055: UDP, length 1416
      17:51:19.290189 IP 192.168.0.241.51890 > 192.168.0.88.2055: UDP, length 1420
      17:51:20.321611 IP 192.168.0.100.9912 > 192.168.0.88.2055: UDP, length 1416
      ...
    Windows / Linux:

    CySight Command line test :
    • In CySight Standard Version, run the following command to enter the interactive command line session:
      • telnet localhost 30000
      Type "devices" after the CySight "DigiToll>" prompt to list all the plugins in backend:

      DigiToll> devices
      --------------------------------------------------

      Device 1001
      Device Label : ScheduleController1001
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1002
      Device Label : ScheduleController1002
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1071
      Device Label : NetFlow2055
      Device Enabled : True
      Device Plugin : CiscoNetFlow

      Device 1072
      Device Label : NetFlow9995
      Device Enabled : True
      Device Plugin : CiscoNetFlow

      Device 1101
      Device Label : DNLookup_1101
      Device Enabled : True
      Device Plugin : DNLookupBot

      --------------------------------------------------
      Use deviceinfo <DeviceID> for more information

      Type "deviceinfo" command to see the running status of any plugin. The 'CiscoNetFlow' plugins are those responsible for NetFlow collection, and "deviceinfo" command will display details like listening port, active or not, router IP, etc.

      DigiToll> deviceinfo 1071
      --------------------------------------------------
      DeviceID 1071
      Device Label : NetFlow2055
      Device Enabled : True
      Device Plugin : CiscoNetFlow
      Input count : 8505
      Thread Alive : True
      NetFlow Port : 2055

      Router : 192.168.0.241
      Live Map Size : 121
      Last Live Dump : Mon Aug 24 18:15:00 EST 2009
      Hour Map Size : 6
      Last Hour Dump :

      Router : 192.168.0.100
      Live Map Size : 399
      Last Live Dump : Mon Aug 24 18:15:00 EST 2009
      Hour Map Size : 22
      Last Hour Dump :
      --------------------------------------------------

      If a plugin has not received any NetFlow export, the following message will be displayed:
      DigiToll> deviceinfo 1072

      --------------------------------------------------
      DeviceID 1072
      Device Label : NetFlow9995
      Device Enabled : True
      Device Plugin : CiscoNetFlow
      Input count : 0
      Thread Alive : True
      NetFlow Port : 9995
      --------------------------------------------------
    Enterprise Collection Process Checking:
    • First check if C collector for Enterprise version is running:
      • ps -ef | grep dt_nf | grep netflow
      The following output indicates that the Enterprise Version collectors are running. An empty output means no collector is running and troubleshooting is needed. The parameter after '-p' in output (in bold) is the listening port number of each collector.

      root 9816 1 0 Aug21 ? 00:16:28 /usr/local/digitoll/bin/netflow_v9 -i 192 0 0 100 -p 2055 -v 5 -k /digitoll/keyblocks/digitoll/ -d /digitoll/packets/dt_nf/3001/pkt3001 -w /digitoll/packets/dt_nf/3001/pkt3001.tmp -hs 1299827 -md 60 -ad 60 -uid 500 -gid 500 -basec /usr/local/digitoll/conf/dt_nf_ALL.conf
      root 10454 1 4 Aug21 ? 02:37:11 /usr/local/digitoll/bin/netflow_v9 -i 192 0 0 241 -p 9995 -v 5 -k /digitoll/keyblocks/digitoll/ -d /digitoll/packets/dt_nf/3002/pkt3002 -w /digitoll/packets/dt_nf/3002/pkt3002.tmp -hs 1299827 -md 60 -ad 60 -uid 500 -gid 500 -basec /usr/local/digitoll/conf/dt_nf_ALL.conf

      If Collector is running, change directory to /digitoll/packets/dt_nf/:
      • cd /digitoll/packets/dt_nf
      Run the command to monitor the contents of all sub-directory continuously:
      • watch -n 1 "ls -l *"
      New files can be seen to pop up and then disappear periodically like the following.

      3001:
      total 268
      -rw-r--r-- 1 digitoll digitoll 268430 Aug 24 04:47 pkt3001_4A925386

      3002:
      total 2220
      -rw-r--r-- 1 digitoll digitoll 2347986 Aug 24 04:47 pkt3002_4A925386

      If the file size (marked in red) is larger than 50, then there is NetFlow stream being received. Otherwise no NetFlow Export hits the collector.
      Please note each collector listens on only one port. Make sure port number is correct. And Enterprise Version does not have a command to tell you which collector is handling which device. End user have to correlate the listening port number of collector with the tcpdump output.
    Steps to detect Standard Version or Enterprise Version:

    1. Check if the Enterprise Version is running:
    • ps -ef | grep dt_nf | grep netflow
    Enterprise Version runs a separate high speed collector for better performance and fault-tolerance. In normal running scenario, The Enterprise collector must be running.

    2. Check plugin type:

    First enter the command line session:
    • telnet localhost 30000
    List all plugins:
    • DigiToll> devices
    If there are any 'CiscoNetFlow' plugins, then it is a Standard Version; if you see any 'NetflowLive' and 'TrendBridge' plugins, it is the Enterprise version.

    Typical plugins for Standard Version:
    • DigiToll> devices

      --------------------------------------------------

      Device 1001
      Device Label : ScheduleController1001
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1002
      Device Label : ScheduleController1002
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1071
      Device Label : NetFlow2055
      Device Enabled : True
      Device Plugin : CiscoNetFlow

      Device 1072
      Device Label : NetFlow9995
      Device Enabled : True
      Device Plugin : CiscoNetFlow

      Device 1101
      Device Label : DNLookup_1101
      Device Enabled : True
      Device Plugin : DNLookupBot

      --------------------------------------------------
      Use deviceinfo <DeviceID> for more information
    Typical plugins for Enterprise Version:
    • --------------------------------------------------

      Device 1001
      Device Label : ScheduleController1001
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1002
      Device Label : ScheduleController1002
      Device Enabled : True
      Device Plugin : ScheduleController

      Device 1101
      Device Label : DNLookup_1101
      Device Enabled : True
      Device Plugin : DNLookupBot

      Device 3001
      Device Label : Netflow2055
      Device Enabled : True
      Device Plugin : NetflowLive

      Device 3002
      Device Label : Netflow2056
      Device Enabled : True
      Device Plugin : NetflowLive

      Device 6001
      Device Label : Netflow2055 TrendBridge
      Device Enabled : True
      Device Plugin : TrendBridge

      Device 6002
      Device Label : Netflow2056 TrendBridge
      Device Enabled : True
      Device Plugin : TrendBridge

      --------------------------------------------------