Cable modem reboot kills Webconfigurator



  • Hello,

    This afternoon, I was working with Comcast on an issue and they remotely rebooted my modem. After the modem restarted, I could not connect to the internet. I tried to connect to my pfsense sg 2440 appliance and could not connect to the webconfigurator either. I power cycled the 2440 and modem twice without any change. I went to the 2440 and connected via usb to serial and restarted. I noticed that the comments when services were loading, the webconfigurator failed.

    I seem to remember this happening to me a while back and again today. Wondering if this is a known issue. I did some research and found a 2011 post where this happened and associated it with how the cable modem assigns a local ip address followed by the public one once the connection is established. My 2440 was at the latest version of pfsense when this happened.

    Thanks,
    Jerold



  • Hi,

    @jpvonhemel said in Cable modem reboot kills Webconfigurator:

    I did some research and found a 2011 post where this happened and associated it with how the cable modem assigns a local ip address followed by the public one once the connection is established.

    That has solved ages ago ^^
    Of course, "Comcast" shouldn't hand out a 'local' IP if its running in bridge mode. Brides ought to be transparent .... (imho).
    So, a work around was implemented :
    Go your WAN interface setup.
    Go to the part that states "DHCP Client Configuration" and check "Advanced Configuration".
    Now you can fill in "Reject leases from" ....

    The other solution was :
    Reboot your modem.
    Wait a minute or two - or more, I don't have such a modem.
    When it is finally started up, start pfSense.


  • Netgate Administrator

    Yup that^. Though that should not ever prevent the webconfigurator starting.

    Steve



  • @stephenw10 said in Cable modem reboot kills Webconfigurator:

    Though that should not ever prevent the webconfigurator starting.

    When the WAN is signaled as UP, even though dpinger screams at 100 % loss and local DNS is sending requests into a black hole, the GUI will become 'less responsive'.
    I'm using myself an upstream ISP router, and believe me, when my 'VDSL' connection goes down, the GUI does behave differently.

    Probably related to https://redmine.pfsense.org/issues/8987 with the difference that in my case the WAN stays up - the upstream connection is lost.


  • Netgate Administrator

    Yeah it slows things down dramatically! But it would not prevent it starting at boot. At least I've never seen that.

    Steve



  • @Gertjan said in Cable modem reboot kills Webconfigurator:

    "Reject leases from" ....

    I must do this.

    I experienced this exact thing two days ago. I rebooted my Comcast modem and then could not get into webConfig.
    Waiting about 30 minutes while no internet and no webConfig access. Finally was able to SSH into my SG2440 and restarted PHP-FPM, followed by web configurator.



  • So, as soon as I added Reject leases from....192.168.100.1 and saved, I lost connection (cable modem was not restarted during this time).
    No internet, webConfig not accessible, SSH very unresponsive. Took about 20 minutes to finally get into SSH and restart PHP and webConfig. What else could I do/check?



  • @bartkowski said in Cable modem reboot kills Webconfigurator:

    I experienced this exact thing two days ago. I rebooted my Comcast modem and then could not get into webConfig.

    What were LAN settings and WAN IP settings at that moment ?
    Example : if your LAN is 192.168.1.0/24 and the Comcast, when waking up, gives an IP to pfSense (using DHCP client on WAN) in the 192.168.1.0/24 network, it's understandable the webconfigurator goes belly up.

    On the other hand, when Comcast gives an 10.0.0.0/24 IP on WAN, it's understanble that you have no Internet connection. The webconfigurator should still work albeit a bit slow because DNS is out of business.

    So, what did you saw ?



  • @Gertjan My LAN and VLAN's are in 192.168.2.x 192.168.10.x 192.168.20.x.
    Comcast modem GUI is accessible on 192.168.100.1 (it is a Netgear CM600 - owned by me).
    The GUI is very limited (Password, Status, and couple other useless menus), there are no settings to control or see IP.

    Modem is in bridge mode and at the time, my WAN IP was 73.X.X.190.
    I have "Disable Gateway Monitoring Action" checked on WAN gateway. "Disable Gateway Monitoring" is NOT checked.

    When I tried accessing webConfig, I saw the NGINX 502 error.



  • @bartkowski said in Cable modem reboot kills Webconfigurator:

    I saw the NGINX 502 error.

    Which means : the PHP subsystem became unreachable.

    Connect to pfSense using SSH or console, use option 8.
    Type

    top
    

    and enter.

    Now, power down your modem.

    In the GUI you should see that you lost your WAN 'Internet' IP.

    Wait several seconds.
    What does "top" show you ?
    PHP process is eating away all CPU ( colom WCPU ) resources ?

    ef64cf17-70b4-426e-9640-b29a9683ed47-image.png ede5a277-de04-4cec-8138-8af722132c19-image.png

    Visit in the GUI Status -> Interfaces
    WAN should be marked as down - and no 'local' IP from Comcast in the range 192.168.100.x should be accepted.

    Power the modem up.

    Eventually, Status -> Interfaces should indicate that the WAN is up, using some "73.X.X.190" IP which it obtained by DHCP, received from your ISP.

    Btw : not using ComCast myself - I'm living in 'Europe' - but : there are many people using ComCast and pfSense, some workable setup is be possible.



  • @Gertjan Thanks for this info.

    I'll add that it seems to me that this happens anytime I make a change to impact the WAN gateway/interface from PFSense GUI.
    For example, after unchecking "Block private networks and loopback addresses" on WAN interface, I lost connection. Exactly the same behavior as when the modem restarts.

    It would be nice if pfSense could gracefully recover from such an event without manual intervention. I made the mistake of doing the above change while VPN'd from work. Could not get back in until I got back home.



  • @bartkowski said in Cable modem reboot kills Webconfigurator:

    For example, after unchecking "Block private networks and loopback addresses" on WAN interface, I lost connection. Exactly the same behavior as when the modem restarts

    Well.
    That seems very normal to me.
    When you change teh setting of "Block private networks and loopback addresses", all firewall rules are wiped. Then, one by one added again representing the new "rules". "Block private networks and loopback addresses" is just a macro for a firewall rule on that (WAN) interface.
    Typically, this take a second or two for me.

    But hey, you do this only ones ...

    When you modem restart it's also reasonable understandable that you loose connection to the net. The GUI access should still work, though. That why I asked above : can you see with 'top' that PHP gets overloaded (the scripts that make up the GUI) so that NGINX - the GUI web server - spews out an "502".
    If so, we'll spend some time looking what it is doing and /or why it is doing so.

    "Block private networks and loopback addresses"

    @bartkowski said in Cable modem reboot kills Webconfigurator:

    of doing the above change while VPN'd from work.

    Worse : when you redo your WAN network settings, you should restart the VPN server anyway ... which is impossible because you just "hupped" the WAN ...You're locked out.
    The OpenVPN access for remote admin is fine, but there is an exception : do not edit interface (WAN) stuff.

    Think about using an UPS ....



  • Hello,

    I have finally had some time to sit down and work in this issue. My SG-2440 is in my office where my desktop pc is and currently does not have any wan connected. I am currently connected to the internet via my comcast gateway with bridge mode off. This is the case until I can get my 2440 back up and running, then the gateway goes back to bridge.

    I can get in to pfsense8
    via console. When the 2440 boots, I see Starting Webconfigurator... failed. When I choose Restart WebConfigurator, I get this:

    Enter an option: 11

    Restarting webConfigurator...
    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    Message from syslogd@pfSense at Sep 24 18:38:20 ...
    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: still could not bind()
    done.

    Appreciate any ideas to help me.

    Thanks



  • @jpvonhemel said in Cable modem reboot kills Webconfigurator:

    currently does not have any wan connected.

    This is a rather normal situation.
    The first time you powered up your SG-2440, right after assigning interfaces, there was no Internet connection. Console access wasn'teven needed at that moment, you could access right away http://192.168.1.1:80 and login using admin & password..

    But now :

    @jpvonhemel said in Cable modem reboot kills Webconfigurator:

    When the 2440 boots, I see Starting Webconfigurator... failed.

    I presume you see this on the console logs.
    So nginx fails already during booting.

    When you restart the webConfigurator, you see multiple

    @jpvonhemel said in Cable modem reboot kills Webconfigurator:

    pfSense nginx: 2019/09/24 18:38:20 [emerg] 22897#100160: bind() to 0.0.0.0:443 failed (48: Address already in use)

    this means that the initial nginx process that started during boot can't be killed. It's like frozen in the system, and keeps port 443 (the https port) locked. In such a case, a new nginx instance can't be launched, because it can bind to the same port 443 port.

    That's not some WAN interface- or DNS bug. probably not even a Comcast thing. nginx should not fail at all like that.

    I advise you to save your config file - can can find it here : /conf/config.xml and save it.

    Now, reset (option 4) to factory default. Or even better re install pfSense from scratch.
    Such an operation will never hurt pfSense and guarantees good files and initial settings.
    Do the same tests.
    Does nginx launches correctly now ? With WAN hooked up, or not ?

    After importing your config, is everything still ok, or the problem comes back ? (and if so, throw your config file out of the window).

    During all this, the system logs could mention important info.

    Look here :

    a59a648a-41d9-4d5e-b824-335e090ba2c7-image.png

    ( in Status > System Logs > Settings )

    What do you think ? It's time to see some (nginx boot) errors. Set this option, and reboot your device. Show any suspicious boot - system log - messages.


Log in to reply