Pfsense stops. Please help.

emammadov

Hello,

I am using pfsense 2.4.4. Until today pfsense was working okay. Today I saw all network stopped, went to server room and saw there is a red light flashing in front of server (HP ML150G6), but server was on. I looked at monitor, pfsense was on, but frozen, keyboard not answering. I turned server and started again, it began to work. But after a few hours the problem occurred again. I looked at logs, but didn't understand much more. I am attaching the photo and logs here for your review. Please help me to solve this issue.

Logs

0_1540833256801_logs.txt

viragomann

Check the log in HPs iLO for errors.

emammadov

You think, it is server problem, right? Did you have a look at logs?

stephenw10

Mmm, a red light on that little ECG logo sure looks like some hardware failure...

Steve

emammadov

Thanks. Did you have a look at logs? Did you see any strange things there? I don't know much more about logs.

For example:
1.
Oct 29 19:38:05 php-cgi rc.bootup: The command '/usr/sbin/arp -s '192.168.4.245' '00:50:56:8a:94:a4'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'
Oct 29 19:38:04 php-cgi rc.bootup: The command '/usr/sbin/arp -s '192.168.4.244' 'f0:76:1c:37:62:35'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'

Oct 29 19:38:04 kernel ppc0: cannot reserve I/O port range
Oct 29 19:38:04 kernel atkbd0: [GIANT-LOCKED]

Oct 29 19:38:05 sshd 93222 Server listening on 0.0.0.0 port 33426.
Oct 29 19:38:05 sshd 93222 Server listening on :: port 33426.

Oct 29 19:38:04 kernel pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
Oct 29 19:38:04 kernel pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
Oct 29 19:38:04 kernel pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)

Oct 29 19:38:04 kernel uhub3: 2 ports with 2 removable, self powered
Oct 29 19:38:04 kernel uhub1: 2 ports with 2 removable, self powered

stephenw10

Only that first log is a real error. It could be a number of things but potentially it could be bad RAM.

You need to check the server logs if you can reach them.

Steve

viragomann

The flashing LED is the system health indicator.
Blinking red means: Critical system failure detected (processor, memory, regulator, thermal event, fan, NMI)

In iLO you may find detailed information to the failure.

emammadov

@stephenw10 This error can be because of bad RAM?

Oct 29 19:38:05 php-cgi rc.bootup: The command '/usr/sbin/arp -s '192.168.4.245' '00:50:56:8a:94:a4'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'

This server is old and I don't know how to setup its management port and see server logs.

viragomann

You will also find system failure events in the BIOS.

emammadov

I will have a look at that events at work tomorrow. Could you please let me know the meanings of logs with numbers above?

stephenw10

pfSense tried to create an ARP entry for that IP/MAC and failed because it couldn't write to the routing socket due to a memory allocations failure. Hard to say more than that. I imagine those are fixed DHCP leases you have set static ARP on.

Really I wouldn't even look at that until your hardware issue is addressed. Which resolve it anyway.

Steve

emammadov

This memory allocations failure is due to hardware problem, right? After fixing that, these error will not occur again? Until today, I have never seen these errors before.

stephenw10

Then they are probably related.

Steve

emammadov

I changed all rams with new ones, but still get this errors in system logs.

Oct 31 09:06:05 php-fpm 336 /rc.linkup: The command '/usr/sbin/arp -s '192.168.2.240' '00:1a:81:00:1a:f4'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'
Oct 31 09:06:05 php-fpm 336 /rc.linkup: The command '/usr/sbin/arp -s '192.168.2.235' '00:0c:29:a8:72:2b'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'
Oct 31 09:06:05 php-fpm 336 /rc.linkup: The command '/usr/sbin/arp -s '192.168.2.234' '00:0c:29:23:82:78'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'
Oct 31 09:06:05 php-fpm 336 /rc.linkup: The command '/usr/sbin/arp -s '192.168.2.230' '00:0c:29:d0:17:c5'' returned exit code '1', the output was 'arp: writing to routing socket: Cannot allocate memory'

stephenw10

Is that a static DHCP lease defined on the firewall? If not what is that device, where is it defined?

Did you see those errors logged prior to the hardware event?

Steve

emammadov

Clients are getting ip address via dhcp with static mappings. I began to see these errors after restarting pfsense. Actually I have noticed pfsense after restart in the past and haven't see these error logs.

0_1541007532153_Screenshot from 2018-10-31 21-29-22.png