Local services not responding when adding load

robconnolly

Hi,

I'm experiencing a rather weird problem which has reared it's head during my setup of some IP cameras on my network. My setup is this:

I have several VLANs on one which the camera is located, the security server which the camera streams to is on another VLAN. The camera is streaming a 640x480 standard def MJPEG stream.

Whilst the camera is streaming I see the load average on the firewall increasing from ~0.15 to between 0.4 and 0.6. CPU usage is around 14% on average, so the machine doesn't seem to be highly loaded. Periodically everything on my network seems to crash, including outgoing internet access. I've traced this back to the DNS server in pfsense not responding to queries. During this time I can access services on the local network via IP address and can ping out to 8.8.8.8. I can even access the camera stream. However, nothing on the pfsense box responds. The web UI connects, in that I get the HTTPS warning, but the page never loads. Similar behaviour for SSH and OpenVPN. The machine itself is pingable.

This happened a couple of times yesterday and resolved itself after around 20 mins. This morning it happened repeatedly, with the final time being over an hour in duration and only coming back when I killed the camera stream.

Looking into the logs I don't see anything untoward, however there are gaps in my RRD graphs during this time. In the times where everything is working there is about 8.5Mb/s on the interfaces in question.

It is almost as if the whole of userspace is locked up during this time, with only kernel functions working.

I am using the latest version of pfsense (2.2.5-RELEASE-i386).

Any help appreciated.

Thanks in advance.

Derelict

Weird. Doesn't sounds like a very heavy load. Less than the typical HD netflix stream.

I am using the latest version of pfsense (2.2.5-RELEASE-i386).

Hardware not 64-bit?

Harvy66

Do you have Snort installed?
Do you have Squid installed?
What packages, if any, did you install?
What shows up as using CPU under System Activity?

robconnolly

@Derelict:

Weird. Doesn't sounds like a very heavy load. Less than the typical HD netflix stream.

I am using the latest version of pfsense (2.2.5-RELEASE-i386).

Hardware not 64-bit?

Unfortunately not. It's a fairly old machine.

@Harvy66:

Do you have Snort installed?

Do you have Squid installed?

What packages, if any, did you install?

What shows up as using CPU under System Activity?

Nope
Nope
The only package installed is: OpenVPN Client Export Utility
With the stream running:

last pid: 60234;  load averages:  0.56,  0.82,  0.49  up 0+08:59:19    17:24:45
123 processes: 3 running, 100 sleeping, 20 waiting

Mem: 10M Active, 67M Inact, 60M Wired, 32K Cache, 41M Buf, 829M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root     155 ki31     0K    16K RUN     1 513:18  93.65% [idle{idle: cpu1}]
   11 root     155 ki31     0K    16K CPU0    0 399:01  75.29% [idle{idle: cpu0}]
    0 root     -92    0     0K    96K -       0 105:40  20.26% [kernel{ale0 taskq}]
35721 root      34    0 82992K 25720K piperd  1   0:01   7.28% php-fpm: pool lighty (php-fpm)
93371 root      21    0 11384K  2560K select  0   0:01   1.07% top
15463 root      20    0 10356K  1940K bpf     0   4:22   0.10% /usr/local/sbin/filterlog -i pflog0 -p /va
   12 root     -60    -     0K   160K WAIT    0   4:14   0.00% [intr{swi4: clock}]
    5 root     -16    -     0K     8K pftm    0   4:02   0.00% [pf purge]
   16 root     -72    -     0K   168K -       1   4:00   0.00% [usb{usbus4}]
   15 root     -16    -     0K     8K -       1   3:13   0.00% [rand_harvestq]
43128 root      20    0 17048K 10192K select  0   2:30   0.00% /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p
62544 root      20    0 10292K  1976K select  0   1:57   0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/v
19733 root      20    0 10124K  1788K select  0   1:56   0.00% /usr/local/sbin/apinger -c /var/etc/apinge
   17 root     -16    -     0K     8K tzpoll  1   1:14   0.00% [acpi_thermal]
33181 nobody    20    0 11400K  3688K select  0   1:14   0.00% [dnsmasq]
48251 root      20    0 10168K  1700K select  1   1:12   0.00% /usr/sbin/powerd -b adp -a adp -n hadp
   12 root     -88    -     0K   160K WAIT    0   1:05   0.00% [intr{irq23: uhci0 ehc}]
   21 root      16    -     0K     8K syncer  0   0:57   0.00% [syncer]

The first time I tried this the system immediately locked up and I wasn't able to get any data. Stopping the stream revived it. The second time it didn't lock up and I was able to get the above.

Derelict

Unfortunately not. It's a fairly old machine.

Might be getting a little tired.

cmb

@Derelict:

Unfortunately not. It's a fairly old machine.

Might be getting a little tired.

Possibly. Also not sure how well ale(4) NIC driver works or how good that hardware is in general. If it's quirky, maybe specifically to that combination of hardware, it could mess up the entire machine.

robconnolly

@cmb:

@Derelict:

Unfortunately not. It's a fairly old machine.

Might be getting a little tired.

Possibly. Also not sure how well ale(4) NIC driver works or how good that hardware is in general. If it's quirky, maybe specifically to that combination of hardware, it could mess up the entire machine.

The hardware specs are:

Intel Atom N270 1.6GHz
1GB RAM
Atheros Giga Ethernet Card (can't remember the chipset, but it does use the ale driver)
WAN interface uses a generic USB/ethernet adapter (this works fine as the upstream connection isn't fast enough to notice)

Not actually sure how to find out the wireless chipset on pfsense - lspci doesn't appear to be installed.

Is there anything I can do to tweak the ale which might solve the problem?