Local services not responding when adding load
-
Hi,
I'm experiencing a rather weird problem which has reared it's head during my setup of some IP cameras on my network. My setup is this:
I have several VLANs on one which the camera is located, the security server which the camera streams to is on another VLAN. The camera is streaming a 640x480 standard def MJPEG stream.
Whilst the camera is streaming I see the load average on the firewall increasing from ~0.15 to between 0.4 and 0.6. CPU usage is around 14% on average, so the machine doesn't seem to be highly loaded. Periodically everything on my network seems to crash, including outgoing internet access. I've traced this back to the DNS server in pfsense not responding to queries. During this time I can access services on the local network via IP address and can ping out to 8.8.8.8. I can even access the camera stream. However, nothing on the pfsense box responds. The web UI connects, in that I get the HTTPS warning, but the page never loads. Similar behaviour for SSH and OpenVPN. The machine itself is pingable.
This happened a couple of times yesterday and resolved itself after around 20 mins. This morning it happened repeatedly, with the final time being over an hour in duration and only coming back when I killed the camera stream.
Looking into the logs I don't see anything untoward, however there are gaps in my RRD graphs during this time. In the times where everything is working there is about 8.5Mb/s on the interfaces in question.
It is almost as if the whole of userspace is locked up during this time, with only kernel functions working.
I am using the latest version of pfsense (2.2.5-RELEASE-i386).
Any help appreciated.
Thanks in advance.
-
Weird. Doesn't sounds like a very heavy load. Less than the typical HD netflix stream.
I am using the latest version of pfsense (2.2.5-RELEASE-i386).
Hardware not 64-bit?
-
- Do you have Snort installed?
- Do you have Squid installed?
- What packages, if any, did you install?
- What shows up as using CPU under System Activity?
-
Weird. Doesn't sounds like a very heavy load. Less than the typical HD netflix stream.
I am using the latest version of pfsense (2.2.5-RELEASE-i386).
Hardware not 64-bit?
Unfortunately not. It's a fairly old machine.
- Do you have Snort installed?
- Do you have Squid installed?
- What packages, if any, did you install?
- What shows up as using CPU under System Activity?
- Nope
- Nope
- The only package installed is: OpenVPN Client Export Utility
- With the stream running:
last pid: 60234; load averages: 0.56, 0.82, 0.49 up 0+08:59:19 17:24:45 123 processes: 3 running, 100 sleeping, 20 waiting Mem: 10M Active, 67M Inact, 60M Wired, 32K Cache, 41M Buf, 829M Free Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 16K RUN 1 513:18 93.65% [idle{idle: cpu1}] 11 root 155 ki31 0K 16K CPU0 0 399:01 75.29% [idle{idle: cpu0}] 0 root -92 0 0K 96K - 0 105:40 20.26% [kernel{ale0 taskq}] 35721 root 34 0 82992K 25720K piperd 1 0:01 7.28% php-fpm: pool lighty (php-fpm) 93371 root 21 0 11384K 2560K select 0 0:01 1.07% top 15463 root 20 0 10356K 1940K bpf 0 4:22 0.10% /usr/local/sbin/filterlog -i pflog0 -p /va 12 root -60 - 0K 160K WAIT 0 4:14 0.00% [intr{swi4: clock}] 5 root -16 - 0K 8K pftm 0 4:02 0.00% [pf purge] 16 root -72 - 0K 168K - 1 4:00 0.00% [usb{usbus4}] 15 root -16 - 0K 8K - 1 3:13 0.00% [rand_harvestq] 43128 root 20 0 17048K 10192K select 0 2:30 0.00% /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p 62544 root 20 0 10292K 1976K select 0 1:57 0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/v 19733 root 20 0 10124K 1788K select 0 1:56 0.00% /usr/local/sbin/apinger -c /var/etc/apinge 17 root -16 - 0K 8K tzpoll 1 1:14 0.00% [acpi_thermal] 33181 nobody 20 0 11400K 3688K select 0 1:14 0.00% [dnsmasq] 48251 root 20 0 10168K 1700K select 1 1:12 0.00% /usr/sbin/powerd -b adp -a adp -n hadp 12 root -88 - 0K 160K WAIT 0 1:05 0.00% [intr{irq23: uhci0 ehc}] 21 root 16 - 0K 8K syncer 0 0:57 0.00% [syncer]The first time I tried this the system immediately locked up and I wasn't able to get any data. Stopping the stream revived it. The second time it didn't lock up and I was able to get the above.
-
Unfortunately not. It's a fairly old machine.
Might be getting a little tired.
-
Unfortunately not. It's a fairly old machine.
Might be getting a little tired.
Possibly. Also not sure how well ale(4) NIC driver works or how good that hardware is in general. If it's quirky, maybe specifically to that combination of hardware, it could mess up the entire machine.
-
@cmb:
Unfortunately not. It's a fairly old machine.
Might be getting a little tired.
Possibly. Also not sure how well ale(4) NIC driver works or how good that hardware is in general. If it's quirky, maybe specifically to that combination of hardware, it could mess up the entire machine.
The hardware specs are:
-
Intel Atom N270 1.6GHz
-
1GB RAM
-
Atheros Giga Ethernet Card (can't remember the chipset, but it does use the ale driver)
-
WAN interface uses a generic USB/ethernet adapter (this works fine as the upstream connection isn't fast enough to notice)
Not actually sure how to find out the wireless chipset on pfsense - lspci doesn't appear to be installed.
Is there anything I can do to tweak the ale which might solve the problem?
-
