SG-3100 rebooting

axxxxe

/var/log/nginx.log

The only entry in that log from today is:

Nov 3 02:33:09 gateway nginx: 2022/11/03 02:33:09 [emerg] 56330#100178: bind() to 0.0.0.0:443 failed (48: Address already in use)

... and the unit has rebooted itself several times today.

Edit: My mistake, there are actually several lines like the above for today. All messages from today are failures to bind.

stephenw10

Anything in /var/log/nginx/error.log ?

And do you see nginx running?

[22.05-RELEASE][root@1100-2.stevew.lan]/root: ps -aux | grep nginx
root      433   0.0  4.3 137168 43232  -  I    15:04     0:34.49 php-fpm: pool nginx (php-fpm)
root      434   0.0  4.3 137276 43012  -  I    15:04     0:35.13 php-fpm: pool nginx (php-fpm)
root      643   0.0  4.3 136776 43240  -  I    15:04     0:57.10 php-fpm: pool nginx (php-fpm)
root      653   0.0  4.3 133960 42920  -  I    15:04     0:58.32 php-fpm: pool nginx (php-fpm)
root    22487   0.0  0.8  27488  7632  -  Is   15:14     0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root    22544   0.0  0.8  28400  8480  -  I    15:14     0:04.04 nginx: worker process (nginx)
root    22799   0.0  0.9  28480  8552  -  I    15:14     0:02.79 nginx: worker process (nginx)
root    15452   0.0  0.2  11000  2504 u0  S+   16:24     0:00.01 grep nginx

axxxxe

@stephenw10
[22.05-RELEASE][root@gateway.home.localdomain]/var/log: ps -aux | grep nginx
root 385 0.0 1.5 93592 30708 - I 05:00 0:14.08 php-fpm: pool nginx (php-fpm)
root 386 0.0 1.5 93620 30672 - I 05:00 0:14.14 php-fpm: pool nginx (php-fpm)
root 682 0.0 1.5 93584 31152 - I 05:00 0:14.33 php-fpm: pool nginx (php-fpm)
root 12681 0.0 1.5 93644 30752 - I 05:06 0:13.39 php-fpm: pool nginx (php-fpm)
root 19185 0.0 1.5 93592 30700 - I 05:06 0:14.04 php-fpm: pool nginx (php-fpm)
root 82767 0.0 1.5 93584 30564 - I 05:06 0:12.51 php-fpm: pool nginx (php-fpm)
root 94004 0.0 1.5 93604 30268 - I 05:06 0:12.32 php-fpm: pool nginx (php-fpm)
root 1237 0.0 0.1 4756 2292 u0 S+ 17:25 0:00.01 grep nginx

/var/log/nginx/error.log has no entries since April of 2021.

stephenw10

Hmm, so not running.

Well I would look at the OpenVPN client initially. If it's an assigned interface but never gets created that could be causing it. Does it show in ifconfig?

axxxxe

@stephenw10
ovpnc4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::208:a2ff:fe0d:eb7e%ovpnc4 prefixlen 64 scopeid 0x28
inet 10.14.110.78 --> 10.14.110.1 netmask 0xffffff00
groups: tun openvpn
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 95554

Without the WebGUI I am lost as far as manipulating the config...

stephenw10

Hmm, that looks fine. So are any interfaces shown as 0.0.0.0?

axxxxe

@stephenw10
No.

[22.05-RELEASE][root@gateway.home.localdomain]/var/log: ifconfig | grep 'inet '
inet 10.123.123.1 netmask 0xffffff00 broadcast 10.123.123.255
inet 84.73.89.100 netmask 0xfffff800 broadcast 255.255.255.255
inet 127.0.0.1 netmask 0xff000000
inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
inet 192.168.101.1 netmask 0xffffff00 broadcast 192.168.101.255
inet 192.168.102.1 netmask 0xffffff00 broadcast 192.168.102.255
inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255
inet 192.168.104.1 netmask 0xffffff00 broadcast 192.168.104.255
inet 192.168.105.1 netmask 0xffffff00 broadcast 192.168.105.255
inet 192.168.106.1 netmask 0xffffff00 broadcast 192.168.106.255
inet 192.168.107.1 netmask 0xffffff00 broadcast 192.168.107.255
inet 172.27.0.1 netmask 0xffffff00 broadcast 172.27.0.255
inet 192.168.108.1 netmask 0xffffff00 broadcast 192.168.108.255
inet 192.168.109.1 netmask 0xffffff00 broadcast 192.168.109.255
inet 192.168.110.1 netmask 0xffffff00 broadcast 192.168.110.255
inet 192.168.111.1 netmask 0xffffff00 broadcast 192.168.111.255
inet 192.168.112.1 netmask 0xffffff00 broadcast 192.168.112.255
inet 192.168.113.1 netmask 0xffffff00 broadcast 192.168.113.255
inet 192.168.114.1 netmask 0xffffff00 broadcast 192.168.114.255
inet 192.168.115.1 netmask 0xffffff00 broadcast 192.168.115.255
inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
inet 192.168.12.1 netmask 0xffffff00 broadcast 192.168.12.255
inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255
inet 10.200.2.1 --> 10.200.2.2 netmask 0xffffff00
inet 10.200.3.1 --> 10.200.3.2 netmask 0xffffff00
inet 10.200.1.1 --> 10.200.1.2 netmask 0xffffff00
inet 10.14.110.78 --> 10.14.110.1 netmask 0xffffff00

stephenw10

0.0.0.0 can also be a placeholder for 'all IPs'. If it can't bind to it is there something else already listening on port 443? HAProxy perhaps? One of the OpenVPN servers?

What does sockstat -l show?

axxxxe

@stephenw10 said in SG-3100 rebooting:

sockstat -l

[22.05-RELEASE][root@gateway.home.localdomain]/var/log: sockstat -l | grep 443
root dpinger 87220 6 stream /var/run/dpinger_OVPN443_USER_VPNV4~10.200.1.1~10.200.1.1.sock
root openvpn 90098 6 tcp4 84.73.89.100:443 :

Yes, there is an OpenVPN server listening on 443. 84.73.89.100 is the WAN interface. Is it bad for me to have set it up this way?

Gertjan

@axxxxe said in SG-3100 rebooting:

Is it bad

Yes.
A single server process can listen on a port. Ones started, another server can't use that port.

You could use UDP as a protocol for you OpenVPN on port 443.
As nginx, the pfSense web server, uses port 443 protocol TCP, this will be ok.

Or move the pfSense web server GUI to another port, like 8080.

Or just leave the OpenVPN on the default 1194 port.

axxxxe

@gertjan
Ok, thanks for the helpful info. I find it bizarre that this config worked for years and only in the last 2 weeks suddenly broke the WebGUI.

Two questions:

How do I fix this from the console, at least so I can get in over the web interface and adjust the OVPN server settings?

Does this explain the random rebooting of the unit?

Gertjan

@axxxxe said in SG-3100 rebooting:

How do I fix this from the console, at least so I can get in over the web interface and adjust the OVPN server settings?

You have the console access working ?
Then easy : edit** the pfSense master config file /cf/conf/confiig.xml
Locate the openvpn server settings, you'll see the port number "443", change that for "1234".
Save.
Reboot.
All is well.

There is a command that can help you :

viconfig

(if you know what 'vi' is )

@axxxxe said in SG-3100 rebooting:

Does this explain the random rebooting of the unit?

The GUI web server is also use to 'execute' many PHP scripts that are needed for the pfSense household tasks.
With the GUI not running, the system will become .... undefined to me.

stephenw10

I would say it doesn't explain the rebooting.

I also find it odd that it prevented the webgui process starting at all. I would have expected it to be unavailable on the WAN address only. However that is a conflict so I would start out by removing it.

You can use the Easy Editor ee instead of vi if you're not a masochist! It's built in.

Or you can use the 'Set interface(s) IP address' function from the menu. If you set one of the interfaces to the same address it already has it will ask if you wish to revert the webgui to http on port 80. You can then connect there and make other changes in the gui.

Steve

axxxxe

@stephenw10 said in SG-3100 rebooting:

Or you can use the 'Set interface(s) IP address' function from the menu. If you set one of the interfaces to the same address it already has it will ask if you wish to revert the webgui to http on port 80. You can then connect there and make other changes in the gui.

I followed this route. Once in over the WebGUI I just deleted the OVPN server since I didn't actually need it anymore. After a reboot the system came up with far less complaining in the log. Fingers crossed that the issues is resolved.

I have to say that it seems like a bug to me that one can break the system by listening on 443 on the WAN port. This didn't used to be the case. I've had that OVPN server configured to listen on 443 since at least January of 2018 and until recently there was no issue. Note that I only upgraded from 2.4.5 to 22.05 two months ago. Maybe this issue was introduced in 22.05?

Anyway, thank you Steve and Gertjan for the help!

stephenw10

Indeed I don't expect to see the gui stop functioning entirely with that sort of conflict.
I'll try to replicate it.

Glad you were able to recover access though.

Steve

stephenw10

I couldn't replicate this in 22.05 (amd64). Setting a OpenVPN server to TCP port 443 does exactly what I expect it to; it prevent accessing the webgui on the interface it's running on but not on others.
Screenshot from 2022-11-06 17-12-11.png

Did you have it con figured in any special way? UDP+TCP maybe? Multihome?

axxxxe

@stephenw10
Mine was TCP only, no multihome.
But mine was on the WAN port. Maybe the failure only occurs when it's on the WAN port?

stephenw10

Possible, though I can't see why. Yet.

Gertjan

@axxxxe said in SG-3100 rebooting:

've had that OVPN server configured to listen on 443 since at least January of 2018 and until recently there was no issue.

If you've set up OpenVPN using UDP, it could co exist on port 443, as the nginx GUI web server uses TCP.

This : Sharing a Port with OpenVPN and a Web Server tells me that it is possible to use TCP for both a web server and OpenVPN to use port 443/TCP.