Thursday, 4 April 2013

Fixing application problems with Wireshark


After reading Jeremy Stretch’s recent blog post at Packetlife regarding a problem he managed to help resolve by analyzing the output of a traffic capture, I too thought I would share my own recent experience.

Recently the desktop support team encountered a group of clerical users who were reluctant to give up their Windows XP desktops and move to Windows 7.  The reason cited was unacceptably slow performance when using a particular business critical application on Windows 7 workstations.  This had been previously reported by a member of the clerical team but at the time that workstation was replaced back with XP which is obviously not a long-term answer.

This particular application was browser based and the ‘slowness’ only occurred when users clicked links within the application, e.g. to open customer records, or browse to a different page.  This led to speculation the problem could be related to browser incompatibility.  Some basic testing soon eliminated this possibility.  The testing also found this did not appear to be a problem on workstations belonging to network administrators, regardless of who was logged on to the workstation or who was logged into the application.

Since the problem seemed to be occurring on nearly all other standard workstations, regardless of location or subnet, I thought it might be useful to take a packet capture to help shed some light on the situation.

I installed and loaded up Wireshark and logged into the application on a suspect workstation.  I started the packet capture and entered the filter ‘ip.addr == 10.0.2.8’, the IP address of the application server.  I then attempted to replicate the issue by clicking any link within the application.  The application paused for around 4 seconds and then the page refreshed.  Jumping back to the packet capture:


If you click the screenshot above, you can see that the application server is making a NetBIOS name query directly to the client workstation 4 times, after which the page appears to load. Then I compared this to an administrators workstation which did not have the slowness problem and found that the server was not making any such requests.

I took at look at the DNS server used by the server and found that the client workstations did not have a pointer record in the reverse lookup zone.  Old XP workstations did, and lo-and-behold, machines used by administrators with static IP addresses did too.

A quick Google search tells us that “The behavior of Client not registering PTR record by default is modified prior to Windows7 (mostly during Vista time) and is the intended behavior.  The Dhcp Server is responsible for performing PTR record registration on behalf of client.” - http://social.technet.microsoft.com/Forums/en-US/w7itpronetworking/thread/3a1c9334-54ba-4845-b7c0-ef8ce5454276.  This also helps to explain why workstations with static IP addresses were unaffected.  I put this to the test and manually created some PTR records for a couple of client workstations who were having the application problem and it was resolved straight away.

I’m guessing that a packet capture from the server would have shown a DNS lookup and then a NetBIOS name query, which in hindsight would have helped for a slightly quicker resolution.

Sunday, 7 October 2012

Monitoring throughput with Iperf and PsExec

Often you might need to check the amount of throughput your network can handle to help diagnose (or prevent) performance problems.  I've done this across WAN links, from WLAN clients and on lots of other occasions.

But how best can you measure it?  I use a small tool called Iperf.  It allows you to measure the throughput between 2 computers; at one end you set it up so it's listening, and at the other end you run it and direct traffic to the other machine.  It works really well providing you have access to both computers to get it setup, but what happens when you don't?  This is where PsExec comes in handy since it can be used to run programs on remote computers.


To help with this I've written a small script to automate this process.  I keep it handy on a USB stick, on my laptop, or on network storage so I can get to it whenever I need it.


Save the batch file below to the same folder as a copy of Iperf, PsExec and PsKill and you're good to go.  If you save the script as 'networktest.bat', run it from the command line by typing 'networktest 192.168.1.2' - just substitute the IP address for that of the other computer at the remote end.


Couple of things to bear in mind:  You will need permission to connect to the remote machine in order for PsExec and PsKill to run.  I try mapping a drive to the remote end first to make sure I can.  Finally, anti-virus software might prevent Iperf, PsExec or PsKill from running either on the local machine or at the remote end. 


@echo off
echo.

echo. 
pskill \\%1 iperf.exe
rem kills any existing instance of iperf already running on the remote computer.
echo.
echo.
psexec \\%1 -scd iperf.exe -s
rem sets iperf in listening mode on the remote computer.
echo.
echo.
iperf -c %1 -r -t 30 -P 10 -w 64KB -l 128KB -i 10
rem generates some traffic and outputs to the screen. check out the iperf help for different switches and parameters.
echo.
echo.
pskill \\%1 iperf.exe
rem kills the instance of iperf running on the remote computer.
echo.
echo.
pause


When I run the script between my laptop and my home PC, both wireless on my standard ISP-issued router, the output looks as below.  The key value to look for is the SUM to give you an idea of the network speed.  It looks like between my laptop and my home PC I'm getting around 9 to 10 Mbps.

C:\iperf_script>networktest RICK-HOMEPC



Starting iperf on RICK-HOMEPC

Connecting with pskill service on RICK-HOMEPC...
Killing process 0 on RICK-HOMEPC...
Error killing process(es) named iperf.exe on RICK-HOMEPC:
Process iperf.exe does not exist on RICK-HOMEPC.





------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to RICK-HOMEPC, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[440] local 192.168.0.6 port 8399 connected with 192.168.0.2 port 5001
[424] local 192.168.0.6 port 8397 connected with 192.168.0.2 port 5001
[408] local 192.168.0.6 port 8395 connected with 192.168.0.2 port 5001
[392] local 192.168.0.6 port 8393 connected with 192.168.0.2 port 5001
[376] local 192.168.0.6 port 8391 connected with 192.168.0.2 port 5001
[432] local 192.168.0.6 port 8398 connected with 192.168.0.2 port 5001
[416] local 192.168.0.6 port 8396 connected with 192.168.0.2 port 5001
[400] local 192.168.0.6 port 8394 connected with 192.168.0.2 port 5001
[384] local 192.168.0.6 port 8392 connected with 192.168.0.2 port 5001
[368] local 192.168.0.6 port 8390 connected with 192.168.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[416]  0.0-10.0 sec  1.00 MBytes   839 Kbits/sec
[384]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[440]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[424]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[408]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[392]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[376]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[432]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[400]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[368]  0.0-10.0 sec  1.13 MBytes   944 Kbits/sec
[SUM]  0.0-10.0 sec  11.1 MBytes  9.33 Mbits/sec
[400] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[384] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[368] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[440] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[424] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[416] 10.0-20.0 sec  1.00 MBytes   839 Kbits/sec
[392] 10.0-20.0 sec   896 KBytes   734 Kbits/sec
[408] 10.0-20.0 sec  1.00 MBytes   839 Kbits/sec
[376] 10.0-20.0 sec  1.00 MBytes   839 Kbits/sec
[ ID] Interval       Transfer     Bandwidth
[432] 10.0-20.0 sec  1.00 MBytes   839 Kbits/sec
[SUM] 10.0-20.0 sec  9.25 MBytes  7.76 Mbits/sec
[408] 20.0-30.0 sec  1.00 MBytes   839 Kbits/sec
[440] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[376] 20.0-30.0 sec  1.00 MBytes   839 Kbits/sec
[432] 20.0-30.0 sec  1.00 MBytes   839 Kbits/sec
[416] 20.0-30.0 sec  1.00 MBytes   839 Kbits/sec
[400] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[384] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[368] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[392] 20.0-30.0 sec  1.00 MBytes   839 Kbits/sec
[424] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[SUM] 20.0-30.0 sec  10.6 MBytes  8.91 Mbits/sec
[408]  0.0-30.6 sec  3.25 MBytes   890 Kbits/sec
[432]  0.0-30.6 sec  3.25 MBytes   890 Kbits/sec
[440]  0.0-30.6 sec  3.25 MBytes   890 Kbits/sec
[376]  0.0-30.6 sec  3.25 MBytes   890 Kbits/sec
[416]  0.0-30.7 sec  3.13 MBytes   855 Kbits/sec
[400]  0.0-30.7 sec  3.25 MBytes   889 Kbits/sec
[384]  0.0-30.7 sec  3.25 MBytes   889 Kbits/sec
[ ID] Interval       Transfer     Bandwidth
[368]  0.0-30.7 sec  3.25 MBytes   889 Kbits/sec
[392]  0.0-30.7 sec  3.13 MBytes   854 Kbits/sec
[424]  0.0-30.7 sec  3.25 MBytes   888 Kbits/sec
[SUM]  0.0-30.7 sec  32.3 MBytes  8.81 Mbits/sec
[156] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15041
[172] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15042
[128] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15043
[392] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15044
[356] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15045
[384] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15046
[400] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15047
[416] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15048
[372] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15049
[436] local 192.168.0.6 port 5001 connected with 192.168.0.2 port 15050
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[172]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[392]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[356]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[384]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[400]  0.0-10.0 sec  1.50 MBytes  1.26 Mbits/sec
[416]  0.0-10.0 sec  1.00 MBytes   840 Kbits/sec
[372]  0.0-10.0 sec  1.00 MBytes   840 Kbits/sec
[436]  0.0-10.0 sec  1.00 MBytes   841 Kbits/sec
[128]  0.0-10.0 sec   897 KBytes   735 Kbits/sec
[SUM]  0.0-10.0 sec  12.9 MBytes  10.8 Mbits/sec
[372] 10.0-20.0 sec  1.50 MBytes  1.26 Mbits/sec
[172] 10.0-20.0 sec  1.37 MBytes  1.15 Mbits/sec
[128] 10.0-20.0 sec  1024 KBytes   839 Kbits/sec
[436] 10.0-20.0 sec   894 KBytes   732 Kbits/sec
[356] 10.0-20.0 sec  1.50 MBytes  1.26 Mbits/sec
[156] 10.0-20.0 sec   895 KBytes   734 Kbits/sec
[416] 10.0-20.0 sec   895 KBytes   734 Kbits/sec
[392] 10.0-20.0 sec   895 KBytes   734 Kbits/sec
[400] 10.0-20.0 sec  1.13 MBytes   944 Kbits/sec
[ ID] Interval       Transfer     Bandwidth
[384] 10.0-20.0 sec   895 KBytes   734 Kbits/sec
[SUM] 10.0-20.0 sec  10.9 MBytes  9.12 Mbits/sec
[384] 20.0-30.0 sec   897 KBytes   735 Kbits/sec
[372] 20.0-30.0 sec  1.50 MBytes  1.26 Mbits/sec
[392] 20.0-30.0 sec   897 KBytes   735 Kbits/sec
[400] 20.0-30.0 sec  1.13 MBytes   944 Kbits/sec
[128] 20.0-30.0 sec  1.75 MBytes  1.47 Mbits/sec
[436] 20.0-30.0 sec  1.25 MBytes  1.05 Mbits/sec
[156] 20.0-30.0 sec  1024 KBytes   839 Kbits/sec
[172] 20.0-30.0 sec  1.75 MBytes  1.47 Mbits/sec
[416] 20.0-30.0 sec   897 KBytes   735 Kbits/sec
[356] 20.0-30.0 sec  1.63 MBytes  1.36 Mbits/sec
[SUM] 20.0-30.0 sec  12.6 MBytes  10.6 Mbits/sec
[384]  0.0-31.1 sec  3.50 MBytes   943 Kbits/sec
[128]  0.0-31.2 sec  3.88 MBytes  1.04 Mbits/sec
[372]  0.0-31.3 sec  4.25 MBytes  1.14 Mbits/sec
[392]  0.0-31.4 sec  3.50 MBytes   936 Kbits/sec
[436]  0.0-31.3 sec  3.38 MBytes   904 Kbits/sec
[400]  0.0-31.4 sec  4.00 MBytes  1.07 Mbits/sec
[172]  0.0-31.4 sec  4.88 MBytes  1.30 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-31.5 sec  3.63 MBytes   967 Kbits/sec
[356]  0.0-31.5 sec  4.88 MBytes  1.30 Mbits/sec
[416]  0.0-31.5 sec  3.00 MBytes   798 Kbits/sec
[SUM]  0.0-31.6 sec  38.9 MBytes  10.3 Mbits/sec



Connecting with pskill service on RICK-HOMEPC...
Killing process 0 on RICK-HOMEPC...
Process iperf.exe killed on RICK-HOMEPC.



Press any key to continue . . .