PfSense 2.3: Slow GUI and no SSH access

iam

Hello,

I'm running pfSense on two different systems:
1. ALIX with CF-Card
2. APU1 with SD-Card

On both I have the same problems after the upgrade to 2.3:
1. Access over SSH is not possible. I'm getting this


connection closed by $IP

2. The GUI reacts slow. Especially after choosing the "Update" menu point I'm getting this:


504 Gateway Time-out

iam

Some logs from the APU:


Apr 13 14:08:40 	php-fpm 	88097 	/widgets/widgets/system_information.widget.php: The command '/sbin/mount -u -w -o sync,noatime /' returned exit code '1', the output was 'mount: /dev/ufs/pfsense0: Device busy'
Apr 13 14:08:40 	php-fpm 	88097 	/widgets/widgets/system_information.widget.php: / File system is dirty.
Apr 13 14:08:40 	php-fpm 	88097 	/widgets/widgets/system_information.widget.php: The command '/sbin/mount -u -w -o sync,noatime /' returned exit code '1', the output was 'mount: /dev/ufs/pfsense0: Device busy'
Apr 13 14:09:14 	php-fpm 	8590 	/pkg_mgr_install.php: The command '/sbin/mount -u -w -o sync,noatime /cf' returned exit code '1', the output was 'mount: /dev/ufs/cf: Device busy'
Apr 13 14:09:14 	php-fpm 	8590 	/pkg_mgr_install.php: The command '/sbin/mount -u -w -o sync,noatime /cf' returned exit code '1', the output was 'mount: /dev/ufs/cf: Device busy'
Apr 13 14:09:46 	php-fpm 	8590 	/pkg_mgr_install.php: The command '/sbin/mount -u -w -o sync,noatime /' returned exit code '1', the output was 'mount: /dev/ufs/pfsense0: Device busy'
Apr 13 14:09:46 	php-fpm 	8590 	/pkg_mgr_install.php: / File system is dirty.
Apr 13 14:11:03 	apu-fqdn 		nginx: 2016/04/13 14:11:03 [error] 26527#0: *69 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.100.2, server: , request: "GET /pkg_mgr_install.php?id=firmware HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "apu-fqdn", referrer: "https://apu-fqdn/"

jimp

Diag > NanoBSD, set it to be permanently RW.

iam

Hello, thanks that works on the APU :)

Unluckily the ALIX ist REALLY slow. Do you have an idea how to fix it there? AFAIK is the menu on the serial console also disabled.

iam

With enough tries it worked at the end also on the ALIX :)

Starko

@jimp:

Diag > NanoBSD, set it to be permanently RW.

We also had this problem when we switched from 2.2.6 to "2.3.1-DEVELOPMENT (amd64) built on Tue Apr 19 07:18:40 CDT 2016" on NanoBSD-4G. It was really a pain to use the WegGUI.
Is this a bug? Do you need a bug report for this? Or are you aware of it and it will be fixed in 2.3.1?
Thanks for clarification! We have dozens of NanoBSD-4G installatins on won't switch until we know that we can reach the GUI afterwards.

wm408

Hi Chris,

I have 5 Soekris Net 5501 in the field. It's an AMD Geode, similar performance to the ALIX and same behaviors described by others in this thread.

I've observed the same behavior that you describe in: https://redmine.pfsense.org/issues/6177, but not specifically related to the Internet dropping out. This seems to happen more due to a less performant system like the Net 5501 in comparison to this Netgate https://www.netgate.com/products/rcc-ve-2440.html which I also have, running 2.3 which does not have these issues.

It seems like each time that I go to the router's dashboard/main page, that update check is possibly compounding or creating a hang effect with each load of the main page.

For example, If I waited say 20 seconds on the main page observing the little spinning gear checking each package "Checking update status"… then I navigated away to say... system log, and then clicked back to the main page.. my thought is that possibly now in effect there's two "Checking update status" tasks running, (I did watch the command line with ps aux |grep pkg and observed each individual package being checked).

At this point I get the "connection closed" when I try to SSH in.

The only remedy at that point is to power cycle the unit, not considering logging into the serial console as I don't have the cable with me. Note: I did come back and try to login around 10 minutes later to the main page and it actually loaded. I could then SSH in.

I also found that option 16 on the SSH login menu appeared to instantly remedy this problem also.

The problem is repeatable and should be easy for testing if you have similar hardware to review on.

Routing appears to operate normally... fortunately. So the places that I have this deployed aren't slamming to a halt. I'll limp along until next release.

I also note this thread as possibly being related as well because I got the same 'gateway timeout' in certain situations. https://forum.pfsense.org/index.php?topic=110121.0

You also mentioned earlier to set the NanoBSD to RW, it appears to already be set to that as the option is to change to RO, and that it's a temporary setting.

Thanks.

Guest

@wm408

like the Net 5501 in comparison to this Netgate https://www.netgate.com/products/rcc-ve-2440.html which I also have, running 2.3 which does not have these issues.

The Soekris net5501 is more a comparable to the Alix boards, with the same CPU but with 256 MB more of RAM.
That are i386 (32Bit) hardware and not likes the RCC-VE-2440 fully 64Bit capable hardware, that is pretty new
and right sorted with AES-NI, ECC RAM support, Intel QAT and/or a multi-core CPU/SoC.

It seems like each time that I go to the router's dashboard/main page, that update check is possibly compounding or creating a hang effect with each load of the main page.

Each system is reaching at one time a level where perhaps new hardware should be in usage or in the field,
and in my eyes the older boxes were running between 6 - 8 years and should be upgraded now, since the
32Bit (i386 pfSense image) will be die once in the future.

Would it be perhaps solving out your problem to install pfSense without the dashboard and widgets?

Phereal

Hi,

solved this issue by killing the PHP-FPM process by selecting point "16) Restart PHP-FPM" in the ssh menu. Afterwards navigate directly to the address -> https://<ip_address:port>/system_update_settings.php and disable the automatic dashboard auto-update check. Save settings and return to the dashboard.

Now again kill the PHP-FPM process via ssh and reload the page. Afterwards remove the "traffic-graph" widget from the dashboard. Reboot the system to verify your changes.

Now everything should working fine again. Seems like an PHP-module causing trouble.

EDIT: other posts mention problems with some widgets [IP-SEC], if your problems remain, the solution might be to remove all your widgets and then perform a restart.
Afterwards it might be possible to add the widgets again.</ip_address:port>

wm408

@BlueKobold:

@wm408

like the Net 5501 in comparison to this Netgate https://www.netgate.com/products/rcc-ve-2440.html which I also have, running 2.3 which does not have these issues.

The Soekris net5501 is more a comparable to the Alix boards, with the same CPU but with 256 MB more of RAM.
That are i386 (32Bit) hardware and not likes the RCC-VE-2440 fully 64Bit capable hardware, that is pretty new
and right sorted with AES-NI, ECC RAM support, Intel QAT and/or a multi-core CPU/SoC.

That's what I meant, they're not really comparable.

It seems like each time that I go to the router's dashboard/main page, that update check is possibly compounding or creating a hang effect with each load of the main page.

Each system is reaching at one time a level where perhaps new hardware should be in usage or in the field,
and in my eyes the older boxes were running between 6 - 8 years and should be upgraded now, since the
32Bit (i386 pfSense image) will be die once in the future.

I agree - and I think that the 5501 are still capable (but depending on the purpose)

Would it be perhaps solving out your problem to install pfSense without the dashboard and widgets?

As Phereal responds in the previous thread… it would appear so at this point in time. Hopefully some considerations can be made to optimize this behavior for slower systems on the next release.

kazino

@Phereal:

Hi,

solved this issue by killing the PHP-FPM process by selecting point "16) Restart PHP-FPM" in the ssh menu. Afterwards navigate directly to the address -> https://<ip_address:port>/system_update_settings.php and disable the automatic dashboard auto-update check. Save settings and return to the dashboard.

Now again kill the PHP-FPM process via ssh and reload the page. Afterwards remove the "traffic-graph" widget from the dashboard. Reboot the system to verify your changes.

Now everything should working fine again. Seems like an PHP-module causing trouble.

EDIT: other posts mention problems with some widgets [IP-SEC], if your problems remain, the solution might be to remove all your widgets and then perform a restart.
Afterwards it might be possible to add the widgets again.</ip_address:port>

Had the same issues and followed the solution proposed by Phereal and it worked. Just wanted to say that if at some point you feel your pfsense is stuck just use 16 on console (obviously not when you will be saving your settings).

Thx,