Problems with flaky internet and pfSense



  • I have internet over cable and the connection is flaky, on some days more and on others less.
    When my ISP-Modem is in router-mode it will cope with that, as soon as the problem is over, the connection is up again. But when this box is in bridge-mode, so that there is no double-NAT on my side, the connection often stays down, for hours.
    When I come home I can go to the status-page of the WAN interface and press the release button of DHCP twice and the connections comes back.
    I already changed DHCP Client Configurations but it didn't helped much if any.

    Capture.JPG

    Is there something I can do in pfSense to get the connection back automatically? My ISP doesn't fix the bad connection and I can't change either.

    Thanks
    Bob



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    When my ISP-Modem is in router-mode it will cope with that, as soon as the problem is over, the connection is up again. But when this box is in bridge-mode, so that there is now double-NAT on my side

    That's backwards. In bridge mode, your modem is just that - a bridge. No double-NAT. When your modem is acting as a router, THEN you're in a double-NAT situation.

    Strange that the DHCP timeout fix doesn't work for you. Perhaps do a packet capture on WAN when it's down to see what's going on.



  • @KOM Thanks, there was a mistype on my side, corrected it. I am no network professional so packet capturing is not an option for me.



  • Capture is easy with pfSense (Diagnostics - Packet Capture), and we can help you read & decode the capture.



  • What I just noticed is that an incoming ICMP echo request (by the external Broadband Quality Monitor) continuously got answered by pfSense in that time where all the other machines had no internet. So how is this possible or related to the fact, that the connectivity to all the other machines only comes back when the DHCP of the internet connection is manually renewed. Any ideas?


  • LAYER 8 Global Moderator

    How is what possible for an IP to answer a ping? Do you enabled pfsense to answer pings on its wan? Out of the box it does not allow that btw.

    So lets say your dhcp lease didn't renew and you had IP address 1.2.3.4 (some public IP because your isp gateway is in bridge mode).

    Its quite possible someone else now has IP 1.2.3.4 and is answering the external monitors pings.. Doesn't mean you will work..

    We need some actual info to work with if you want help..

    When your internet stops working - what is the IP on pfsense wan.. Is pfsense showing the gateway up, ie it can pings its gateway.. say 1.2.3.3 ?

    When you renew/release/renew your wan dhcp lease - do you get a different IP?

    If your gateway is showing up, and can not get to internet from your clients.. Using pfsense diag, ping some IP - say 8.8.8.8 does that respond? Can pfsense resolve stuff.. On its diag page, trying looking up some website you have not been too, say www.cnn.com or something - does that resolve?

    Need something to work with here if you want help.



  • @johnpoz So IP stays the same and sure I enabled the ICMP echo request. It had not occurred(?) to me that pfsense by itself could have internet... So I will test the ping thing. And yes the gateway on the WAN-Interface would shown up.
    Also my ISP is doing some sort of special CGC-NAT and pfSense is running in hyper-v...


  • LAYER 8 Global Moderator

    So if your isp is doing CGnat - how is it you can ping your IP.. What IP are you pinging and from where? Your IP is public on pfsense or its rfc1918, if its public then its not CGnat, if its rfc1918 (10.x.x.x, 192.168.x.x, 172.16-31.x.x) then I find it unlikely that they are passing the pings on to you, etc.

    When a carrier does CGnat - they are having multiple clients share a public IP.. They can not pass ping to all clients sharing that IP, etc. etc. While it is possible for them to port forward some ports to you.. This would have to be setup with the ISP, etc.



  • @johnpoz Like I said before it is special. It looks like a NAT-IP, but at the same time it seems exposed, so I can open ports on my side etc. Don't ask me why they do it like that and I have some servers running like WP, Nextcloud etc.. ☺


  • LAYER 8 Global Moderator

    Not saying your your lying or anything - but I am doubtful... What is more likely is your behind your isp devices nat.. And your pfsense is maybe set as dmz host?

    What is the ping time to your "gateway" in pfsense?



  • @johnpoz No, but I am no network professional.



  • What is the ping time to your "gateway" in pfsense?

    I am not sure what you mean by that but I will try to find out.


  • LAYER 8 Global Moderator

    What does pfsense show for its response time to your gateway on the dashboard widget?

    pingtimes.jpg

    Who is the ISP exactly?



  • @johnpoz Just added it:

     WAN_DHCP
    100.65.191.254
    	8.9ms 	2.0ms 	0.0%
    

    Telecolumbus in germany


  • LAYER 8 Global Moderator

    They are using 100.64/10 space - yes that is CGnat space..

    I still find odd that they would be doing 1:1 nat to some public IP - and passing all ports to you..

    But ok - when your clients are having a problem - does pfsense still have access to internet?

    But now we are getting some info, so we can understand your setup, and figure out what is going on.



  • @johnpoz I will test this next time it happens.



  • Depending on the specific model of cable modem you have, you may need to have pfSense "reject" DHCP lease offers from the internal IP of the cable modem.

    For example, I had a Motorola cable modem where 192.168.100.1 was the internal LAN-side IP of the cable modem. I used the modem in bridge mode so my external public IP for the modem was passed to my pfSense firewall. When the external cable signal went down and the modem went into a carrier search and retrain mode, it would seemingly switch out of bridge mode and offer my firewall WAN an IP address from the modem's internal 192.168.100.0/24 net block. My firewall's WAN interface would happily accept the new IP. However, once the external cable signal came back online and the modem switched to bridge mode again, my WAN would frequently be left holding onto that 192.168.100.x IP address and thus I had no Internet connectivity. I would have to manually "release the lease" and renew to pick back up the bridge mode public IP.

    You can prevent this by putting your cable modem's private LAN IP address in the Reject leases from box on the WAN interface settings page --

    WAN_DHCP_RejectIP.png



  • @bmeeks I already Block private networks and loopback addresses but if this is something different, I will happily try this. Thank you! 🖖



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    @bmeeks I already Block private networks and loopback addresses but if this is something different, I will happily try this. Thank you! 🖖

    Yes, this setting is different from that.



  • @bmeeks It will not hurt anyway I guess. ☺



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    @bmeeks It will not hurt anyway. 👏

    Just be sure to use the actual internal IP (or LAN gateway address) of your cable modem. Your model very well could use a different default internal IP from mine. You can find out by doing some Google research using the brand of your modem.



  • @bmeeks It is the same here. Although I am wondering how pfsense could still answer the echorequests. But again, it will not hurt to try that.



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    @johnpoz Like I said before it is special. It looks like a NAT-IP, but at the same time it seems exposed, so I can open ports on my side etc. Don't ask me why they do it like that and I have some servers running like WP, Nextcloud etc.. ☺

    What is your WAN IP address?



  • What is your WAN IP address?

    It is different.



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    What is your WAN IP address?

    It is different.

    Well, that tells me a lot. What do you mean by different?



  • @JKnott It is just a normal IPv4-address. I will not post it here unless "reasons". To be more precise I meant my actual WAN-IP. pfSense has a CG-NAT Address at WAN.



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    To be more precise I meant my actual WAN-IP. pfSense has a CG-NAT Address at WAN

    If it actually is a CG-NAT address, then you can announce it far and wide, as it's impossible for anyone to reach it from elsewhere.



  • @JKnott True but also pointless. I mean it is running now.



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    @JKnott True but also pointless. I mean it is running now.

    <sigh>

    Go to www.grc.com and click on Services > ShieldsUp!. This will show you your "real" address, as seen by the rest of the world. You can then do a port scan to see what ports are open. Try opening some ports and see if they show up in the scan. If you don't see them, then the real address is not mapped to your CG-NAT address. In that case, ping will not reach your network from elsewhere.



  • @JKnott You are missing the point. I already told here that I run servers at home and can open ports etc.
    I know this special NAT my ISP is doing is very interesting for you guys, I have to explain this all the time when I mention it here. 😅

    @Bob-Dig said in Problems with flaky internet and pfSense:

    @johnpoz Like I said before it is special. It looks like a NAT-IP, but at the same time it seems exposed, so I can open ports on my side etc. Don't ask me why they do it like that and I have some servers running like WP, Nextcloud etc.. ☺

    But please let us stay on topic, thank you.



  • So today, sadly, I just experienced it again. My connection came up several times after going down and in the end of this flakyness I had no internet on the clients, in between, the clients had internet... PfSense shows "online", I even had configured an external monitoring IP this time.
    Renewing the dhcp-lease manually on WAN solved it instantly for the clients. ☹



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    So today, sadly, I just experienced it again. My connection came up several times after going down and in the end of this flakyness I had no internet on the clients, in between, the clients had internet... PfSense shows "online", I even had configured an external monitoring IP this time.
    Renewing the dhcp-lease manually on WAN solved it instantly for the clients. ☹

    Sounds like something weird going on between your ISP's DHCP server and the DHCP client inside pfSense for the WAN.

    So before you did the manual lease renew, was your WAN showing the correct public IP address? And did that IP address change after you did the manual renew?



  • @bmeeks In my case it is an CG-NAT-Address, so I haven't watched it closely. My (external) WAN-IP-Address didn't changed and I think that pfSense had a connection... but didn't "shared" it.
    Next time i will do some ping-tests within pfSense and watch those IPs more closely.

    Btw your "trick" helped me anyways I think, I had peace for 20 days but this connection here is just... ☹



  • @Bob-Dig said in Problems with flaky internet and pfSense:

    @bmeeks In my case it is an CG-NAT-Address, so I haven't watched it closely. My (external) WAN-IP-Address didn't changed and I think that pfSense had a connection... but didn't "shared" it.
    Next time i will do some ping-tests within pfSense and watch those IPs more closely.

    Btw your "trick" helped me anyways I think, I had peace for 20 days but this connection here is just... ☹

    When I said "public IP" what I really mean is whatever the "normal and working" IP should be. Whether it is CG NAT or a true public IP would not matter. You would just be looking to see what it is when it is not working, and then compare that to what it is when the connection is working. That info might help with troubleshooting.



  • And here we go again...
    abc.PNG
    I see no difference. Ping from within pfSense to a Website also failed. After renewing instant internet.



  • I see now that you appear to be running pfSense on a Hyper-V host (the hn0 NIC driver is a virtualized NIC for Hyper-V).

    Some quick Google searching found a few posts about issues with that NIC driver and FreeBSD 11. That might be the root of your problem.

    Maybe you mentioned it earlier in the thread and I missed it, but knowing that you have pfSense virtualized and on which platform is very valuable information. Virtualized hardware is NOT the same as physical hardware of course, and the drivers used are different.



  • Here are some of the links I found with a quick Google search:

    1. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229990
    2. https://forum.netgate.com/topic/128384/pfsense-network-interface-sometimes-hangs-on-hyper-v
    3. http://freebsd.1045724.x6.nabble.com/11-1-running-on-HyperV-hn-interface-hangs-td6207926.html

    Not sure all of these apply, but they illustrate there can be potential issues with Hyper-V and FreeBSD guests (pfSense is essentially a FreeBSD guest).

    If you want to virtualize pfSense, I strongly recommend using ESXi. Or else just buy physical hardware. The Netgate appliances cost less than a Windows Server license.


  • Netgate Administrator

    I would check for a missing or bad default route when this happens. Diag > Routes

    If there is no default route client traffic will not be able to get out. pfSense itself would not be able to ping out to arbitrary sites.

    However the gateway monitoring will show onlint because that has a static route via the WAN gateway.

    Do you have more than one gateway in System > Routing > Gateways?

    If the default IPv4 gateway is set to automatic setting it to the WAN dhcp gateway instead should get you back a default route if that is what you're hitting.

    Steve


Log in to reply