Dell R430 bge (Built-in Port Hangs ) & Captive Portal Stop working

Gertjan

@kiokoman said in Dell R430 bge (Built-in Port Hangs ) & Captive Portal Stop working:

so we can confirm that the socket is used by squid dpinger dhcpd etc etc

Correct.
All these programs use or call scripts that are PHP based.

wazim4u

@kiokoman said in Dell R430 bge (Built-in Port Hangs ) & Captive Portal Stop working:

sockstat -l | grep /var/run/php-fpm.socket

got this output

root     sleep      99245 12 stream /var/run/php-fpm.socket
root     sleep      99245 13 stream /var/run/php-fpm.socket
root     sleep      97728 12 stream /var/run/php-fpm.socket
root     sleep      97728 13 stream /var/run/php-fpm.socket
unbound  unbound    20075 12 stream /var/run/php-fpm.socket
unbound  unbound    20075 13 stream /var/run/php-fpm.socket
root     php-fpm    83461 13 stream /var/run/php-fpm.socket
root     php-fpm    83358 13 stream /var/run/php-fpm.socket
root     php-fpm    83301 13 stream /var/run/php-fpm.socket
root     php-fpm    83152 15 stream /var/run/php-fpm.socket
root     nginx      55388 12 stream /var/run/php-fpm.socket
root     nginx      55388 13 stream /var/run/php-fpm.socket
root     nginx      55234 12 stream /var/run/php-fpm.socket
root     nginx      55234 13 stream /var/run/php-fpm.socket
root     nginx      54950 12 stream /var/run/php-fpm.socket
root     nginx      54950 13 stream /var/run/php-fpm.socket
root     nginx      54735 12 stream /var/run/php-fpm.socket
root     nginx      54735 13 stream /var/run/php-fpm.socket
root     nginx      54498 12 stream /var/run/php-fpm.socket
root     nginx      54498 13 stream /var/run/php-fpm.socket
root     nginx      54344 12 stream /var/run/php-fpm.socket
root     nginx      54344 13 stream /var/run/php-fpm.socket
root     nginx      54164 12 stream /var/run/php-fpm.socket
root     nginx      54164 13 stream /var/run/php-fpm.socket
root     nginx      53913 12 stream /var/run/php-fpm.socket
root     nginx      53913 13 stream /var/run/php-fpm.socket
root     nginx      53902 12 stream /var/run/php-fpm.socket
root     nginx      53902 13 stream /var/run/php-fpm.socket
root     nginx      53685 12 stream /var/run/php-fpm.socket
root     nginx      53685 13 stream /var/run/php-fpm.socket
root     nginx      53620 12 stream /var/run/php-fpm.socket
root     nginx      53620 13 stream /var/run/php-fpm.socket
root     sh         94097 12 stream /var/run/php-fpm.socket
root     sh         94097 13 stream /var/run/php-fpm.socket
root     sh         93897 12 stream /var/run/php-fpm.socket
root     sh         93897 13 stream /var/run/php-fpm.socket
root     dpinger    83648 12 stream /var/run/php-fpm.socket
root     dpinger    83648 13 stream /var/run/php-fpm.socket
root     dpinger    83295 12 stream /var/run/php-fpm.socket
root     dpinger    83295 13 stream /var/run/php-fpm.socket

kiokoman

ok nginx is the webserver for captive portal, check that if the problem still occurs

wazim4u

@kiokoman System was working without any problem for 28 days in heavy load 2000 plus users. suddenly started getting errors in ( portal interface ) and three times system goes down " just portal interface stop working for in/out traffic. disable it and enable it start working for sometimes one day or sometimes two days then again it happens.

i am not changing any setting and if one service is working other goes down. now captive portal is stuck on boot screen and not let system to load menu startup. I haven't change any setting in Captive portal.

kiokoman

so now we have random services that go down and captive portal stuck? I will Try fsck on the filesystem to see if it helps and a memtest

wazim4u

@Gertjan is there a way to reinstall captive portal package only. captive portal serivce show running when started from Dashboard. but if i try to save configuration of captive portal it keep waiting and finally gives error 504 Gateway time out. users are connected directly like on LAN and portal seems to be not working anymore.

Gertjan

The captive portal isn't a package.
It's using :
Another instance of nginx running on 127.0.0.1 port 80 (and 443 if uou use https).
This nginx instance uses a landing (php) page.
A helper script /etc/inc/captiveportal.inc

Activating the portal also activates the firewall program calle "ipfw" that can handle MAC addresses on the captive portal interface..

Actually, a captive portal as implemented by pfSense is pretty simple. No special processes.

You run the captive portal on a dedicated interface called OPTx, right ?
Swap LAN and this OPTx interface and see if the issue persists.

kiokoman

you can also check Status / System Logs / Captive Portal Auth to see for any errors

wazim4u

@Gertjan yes its running on OPT1 interface. i have restarted system 3 times didn't fix the issue so only one thing i have changed in system tuneable mentioned by @kiokoman kern.ipc.soacceptqueue to 1024 . I deleted it and rebooted system and captive portal is back. Not sure if its related to captive portal.

Gertjan

@wazim4u said in Dell R430 bge (Built-in Port Hangs ) & Captive Portal Stop working:

Listen queue overflow: 193 already in queue awaiting acceptance (232 occurrences)

Check these.

wazim4u

I would like to update changes i have made in case someone else is facing same issue now or in future to fix similar problems. given below is status after 2 Days System Running.

Captive Portal Status ( 2000+ Users )

Network Status ( so far only 1 Error )

Changes I've Made
1- In /boot/loader.conf.local - Add the following (created new file): for Broadcom NIC

kern.ipc.nmbclusters="1000000"
hw.bge.tso_enable=0
hw.pci.enable_msix=0

2- In Interface/Portal ( Portal Interface bge2 ) Speed & Duplex select

1000baseT full-duplex

( Auto setting was giving some errors on Zyxel switch logs )

3- In /usr/local/etc/php-fpm.d/www.conf

listen.backlog = 511 ( default )

to

listen.backlog = -1

php-fpm related errors gone after changing listen.backlog ( didn't get any till now in two days )
php-fpm error

kernel: sonewconn: pcb 0xfffff80130a2f0f0: Listen queue overflow: 193 already in queue awaiting acceptance (155 occurrences)

now only one issue i am sorting out related to nginx given below.

nginx: 2019/07/20 10:25:19 [alert] 95352#100567: send() failed (40: Message too long)