PfSense install blocks Internet, but computer into modem has Internet?



  • Hi,
        I was working on fixing issue https://forum.pfsense.org/index.php?topic=75516.msg412742#msg412742 which seems to be a faulty cable modem not accepting passwords 100% of the time.

    However, I happened to factory restore pfSense and now the network has no Internet.
    The network can ping other network devices, can ping pfSense, can ping the modem, but can't ping the Internet.

    When I unplug the modem from pfSense and plug the modem directly into a computer, the computer resets the DHCP and can browse the Internet.

    pfSense does work with the modem like before, but maybe I need to upload my topology map for others to understand?



  • Cable modems tend to bond to the MAC address of the device plugged into them when they are booted.

    You did reboot the modem right?  If not- try that.



  • That sounds like the same issue discussed in the other thread in having overlapping or conflicting subnets on LAN and WAN.



  • Thanks for the suggestions.
    I have rebooted the cable modem twice.

    The pfSense WAN is 192.168.0.2/24.
    The pfSense LAN is 192.168.1.155/24.
    So this is not an overlapping or conflicting of subnets.


  • Netgate Administrator

    Gateway on LAN perhaps?

    Can you ping internet hosts from the pfSense diagnostic page? Is it able to check for updates?

    Steve



  • pfSense diagnostics fails to ping the Internet.

    I tried a few other cable connections on the network.

    Tried a few more configurations:
    pfSense factory restored.
    pfSense WAN: 192.168.0.2/24.
    pfSense LAN: 192.168.1.155/24.

    Modem settings:
    LAN IP: 192.168.0.50.
    Firewall > Port Forwarding > ports 1 - 65535 to Local IP Address: 192.168.0.2.
    WAN setup > DMZ Address: 192.168.0.0.

    Modem connected to pfSense:
    Network pings other network devices.
    Network pings pfSense LAN 192.168.1.155.
    Network pings pfSense's WAN 192.168.0.2.
    Network pings modem's LAN 192.168.0.50.
    Network can’t ping Internet.

    Modem disconnected from pfSense and connected to Mac.
    Mac pings Internet.
    MAC DHCP IP 192.168.0.200.
    Mac Subnet Mask: 255.255.255.0.

    Modem disconnected from pfSense and connected to Linux Mint.
    Linux can’t ping Internet.
    Linux pings modem 192.168.0.50.
    Linux DHCP IP 192.168.0.201.
    Linux Bcast: 192.168.0.255.
    Linux Subnet Mask: 255.255.255.0.
    I changed Modem > Firewall > Port Forwarding > Ports 1 - 65535 to Local IP Address: 192.168.0.201.



  • Well, I turned off the cable modem for 30 minutes.
    I then plugged the modem into pfSense and turned on the cable modem.
    My Mac computer had Internet for 5 seconds.
    My Linux computer had no Internet.

    I am guessing the modem may have locked Internet access to a MAC address on the Mac computer, rather than on the pfSense router.

    So, I think now that the modem is allowing Internet via the pfSense router's MAC address, that pfSense is now blocking Internet as pfSense has been factory restored and I need to find out the settings to allow Internet.

    Tests:
    Linux unable to ping 192.168.0.2 (pfSense WAN).
    Linux unable to ping 192.168.0.50 (modem LAN).
    Linux able to ping Mac on 192.168.1.40.
    Linux unable to ping Internet.
    Mac unable to ping 192.168.0.2 (pfSense WAN).
    Mac unable to ping 192.168.0.50 (pfSense LAN).
    Mac able to ping Linux on 192.168.1.120.
    Mac unable to ping Internet.
    pfSense unable to ping 192.168.0.2 (pfSense WAN).
    pfSense unable to ping 192.168.0.50 (modem LAN).
    pfSense able to ping computer Mac on 192.168.1.40.
    pfSense able to ping computer Linux on 192.168.1.120.

    I am now disconnecting the modem from pfSense and connecting the modem to the Mac.
    I now have Internet to post this and to look for pfSense settings to allow Internet.



  • So I think I have narrowed down the problem to pfSense > Diagnostics > can't ping pfSense WAN 192.168.0.2.

    Any suggestions please?



  • I turned off the modem for 30 minutes (the first turn off was for 10 minutes), connected the modem to pfSense, started up the modem.
    Mac now has Internet.
    Linux has no Internet.
    Linux pings Mac.
    Linux pings pfSense LAN 192.168.1.155.
    Linux pings pfSense WAN 192.168.0.2.
    Linux pings modem LAN 192.168.0.50.

    Any suggestions on fixing this?



  • Well, Mac has limited Internet.
    No YouTube and some sites load, then don't load, then load?


  • Netgate Administrator

    Please confirm that you have only one gateway setup in pfSense that it's on the WAN interface and is set as default.
    You can do so in System: Routing: Gateways: or check the gateways in Status: Gateways:
    If you could post a screenshot of  System: Routing: Gateways: that would be great.

    Not being able to ping itself implies it's sending traffic the wrong way which can happen if you have an incorrect gateway setup. Its a very common setup error.

    Steve



  • Yes, there is only one gateway setup in pfSense, on the WAN interface, set as default.

    ![pfSense System Routing Gateways.png](/public/imported_attachments/1/pfSense System Routing Gateways.png)
    ![pfSense System Routing Gateways.png_thumb](/public/imported_attachments/1/pfSense System Routing Gateways.png_thumb)



  • Your WAN is within a private network so uncheck Block private networks on WAN interface settings.



  • Thank you. Yes, I have done that and Internet seems to work 100% on the Mac now, however the Linux computer still has no Internet?

    Linux has a DHCP:
    inet addr:192.168.1.14
    Bcast:192.168.1.255
    Mask:255.255.255.0

    I rebooted the Linux computer after pfSense's change, but same issue?


  • Netgate Administrator

    Ok.
    The pfSense box still can't ping its own WAN address?

    Lets look at your routing table. Diagnostics: Routes:

    Do you use IPv6 at all? You might consider disabling it completely if you don't.

    What is the Linux box using for its gateway?

    Steve



  • So, pfSense can now ping from the pfSense LAN 192.168.1.155 to the pfSense WAN 192.168.0.2.

    The Mac computer is now working on the Internet 100% it seems.
    The Linux computer has no Internet.
    Linux computer has a DHCP IP 192.168.1.120.
    Default gateway: 192.168.1.155.
    Subnet: 255.255.255.0.
    Bcast:192.168.1.255.


  • Netgate Administrator

    Hmm. Nothing in the firewall logs I assume.

    The pfSense box can now ping external addresses too?

    The only other reason that one device might not be getting routed to the internet is that the NAT rules are not capturing traffic from it correctly. Have you switched to manual outbound NAT rules?

    Possibly this is some IPv6 issue such as Linux is using IPv6 as a preference but your router does not support it.

    Steve



  • Can you ping pfSense and things on the Internet (Google DNS, 8.8.8.8?) from your Linux machine? If yes, does DNS work on the Linux machine?



  • Thanks for the suggestions and yes, everything worked before I had to factory restore pfSense.

    Odd behaviour:
    Linux > ping > 8.8.8.8.
    Linux > ping > 192.168.0.2 (pfSense WAN).
    Linux > ping > 192.168.0.50 (modem LAN).
    Linux > ping > 192.168.1.40 (computer Mac).
    Linux > ping > 192.168.1.155 (pfSense LAN).
    Linux > ping > www.google.com fails.
    Linux flushed the DNS with command $ sudo /etc/init.d/dns-clean start > rebooted > same issue with no Internet.

    Mac pings Internet.
    Mac browsing is better. Browses 90% instead of 50% of Internet since I navigated to pfSense > Interfaces > WAN > Private networks > Block private networks: unticked > Block bogon networks: unticked.

    pfSense > Diagnostics > Ping > www.google.com fails.
    pfSense > Diagnostics > Ping > 8.8.8.8.
    pfSense > Diagnostics > Ping > 192.168.0.2 (pfSense WAN).
    pfSense > Diagnostics > Ping > 192.168.0.50 (modem LAN).
    pfSense > Diagnostics > Ping > 192.168.1.40 (computer Mac).
    pfSense > Diagnostics > Ping > 192.168.1.120 (computer Linux).
    pfSense > Diagnostics > Ping > 192.168.1.155 (pfSense LAN).

    Wi-Fi: no Internet.



  • In the pfSense gui, click system then general setup.  Make sure you have atleast 1 DNS setup there and tell it your gateway (modem IP)



  • Yes, I just added in a pfSense primary and secondary DNS which seems to have helped the Mac have 100% Internet.

    Linux still no Internet.

    Started rebuilding pfSense and installing Snort, then Mac stopped having Internet.
    Had to disconnect modem from pfSense and connect to Mac to have Internet.

    Tried pfSense restore to a previous version several hours earlier when modem in pfSense allow Mac to have Internet.
    Still no Internet.
    Factory restored modem and rebuilt and still no Internet when modem plugged into pfSense.

    Either pfSense is faulty or the modem is faulty, however the modem works when plugged directly into a computer?

    I think the modem model needs to be named and shamed at this point.

    Model: BigPond NETGEAR Wireless Cable Modem Gateway CG814WG.
    Cable MAC Address: 00:26:f2:36:1d:41.
    CM certificate: Installed.
    Device MAC Address: 00:26:f2:36:1d:43.
    Hardware Version: 1.03.
    Software Version: V3.9.26R15.
    Standard Specification Compliant: DOCSIS 2.0.

    Any suggestions on Internet through pfSense?
    Last time I think I simply turned off the modem for 30 minutes, then plugged into pfSense and turned the modem on, and the Internet worked on the MAC (but not on the Linux).



  • It shouldn't be nearly this hard, and I think your problem is the combined gateway/modem/router/wireless_access_point device you just mentioned.

    All cable modems that I've seen have supplied an IP address to pfSense WAN, from the ISP.  Your device seems to be operating as its own firewall, dhcp server, etc.  Can you put the device into bridge mode?  Perhaps there's a later, less draconian, firmware available to flash.

    No good can come from pfSense competing with, and downstream from, a SOHO router device.



  • Sound advice charliem.

    After resetting pfSense to factory defaults and rebuilding again, same issues:
    pfSense > Diagnostics > Ping > 192.168.0.2 (pfSense WAN).
    pfSense > Diagnostics > ping > 192.168.0.50 (modem LAN) fails.
    Rest of computer networks can ping each other via pfSense, but nothing to 192.168.0.50.

    Unfortunately, the cable modem has no bridge mode.
    The ISP has booked a technician to bring a new cable modem tomorrow (I won't hold my breath). I requested a 'business' modem or a modem with bridge mode, so I'm sure I'll end up with another crappy router with all the requests lost in ISP cyberspace.

    Is there some hardware I can buy that pfSense forumers know about?
    I would like some recommendations please.



  • I have repeated the same tests several times and the modem works with Internet when directly in computers.
    When pfSense is connected modem, pfSense will not ping 192.168.0.50 (Modem LAN).
    Why?


  • Netgate Administrator

    I have no idea.  :-\

    The problem here is that there is now much conflicting information here. You have posted several times already that the both the pfSense box and th clients behind it are able to ping the modem LAN interface on 192.168.0.50. What has changed since then?

    We also need to clarify something. If your Linux client behind the pfSense box is able to ping 8.8.8.8 then IMHO that box has 'internet'. It is able to send packets to a remote box and receive replies. If it's unable to ping google.com then it has a route to the internet but does not have DNS. I think this may have confused me.
    Additionally when you try to ping something and it fails there is always an error given and that error message is usually helpful in diagnosing why it failed, 'no route to host' 'unknown host' etc.

    Putting the pfSense box behind another NATing router is far from ideal but it should work just fine. Even if you leave the 'block private networks' box checked that will NOT prevent client behind the pfSense box having general internet access.

    Like Charlie said it should not be nearly this difficult!  ;)

    When diagnosing a problem like this the way to do it is always one step at a time. If you are starting from scratch and reinstalling pfSense, as it appears you have, start with a plain vanilla install and test what works at each stage. Do not install Snort until everything else is working! That's still my top suspect in any situation with bizarre behavior.

    Steve



  • Thanks for the reply.

    Yes, I think there is conflicting information because although the fault is the same (pfSense WAN 192.168.0.2 won't ping the modem's LAN 192.168.0.50), I have once been able to make the pfSense WAN 192.168.0.2 ping the modem's LAN 192.168.0.50).

    I believe the conflict happened because of a 'lucky' fix, when I turned the modem off for about 30 minutes, then plugged the modem into pfSense and turned on the modem. However, having tried to repeat this step several times, I am unable to fix the issue.

    Since this lucky fix, I have factory reset pfSense to test and repeat this step/fix, which didn't work. Also, the computers had Internet, but the computers only had 80% Internet, which I later learned was for not having the DNS 8.8.8.8, 4.2.2.2 and the pfSense > Interfaces > WAN > Private networks > Block private networks: untick > Block bogon networks: untick.

    I think the Linux pinging 8.8.8.8 was when the Internet was working, however this issue was fixed by the step above using the pfSense Setup Wizard configuration.

    Yes, this is the only setup I can do, I don't understand how another setup can be done. I have an ISP cable modem with no bridge mode, so this seems to be the best setup. How do others do a different setup? Is there some special business hardware I need to buy?

    Yes, I have walked through step by step and it seems the problem repeats with pfSense static WAN IP 192.168.0.2 not pinging the modem LAN 192.168.0.50.
    When the modem is directly into a computer, the Internet works.

    So, the question I really have is, why or how can I make pfSense WAN 192.168.0.2 (subnet 255.255.255.0) ping the modem's LAN 192.168.0.50 (subnet 255.255.255.0)? I might draw up a topology?


  • Netgate Administrator

    There should be absolutely no reason that ping wouldn't work. It should work will all the settings at their install defaults (including 'block private networks').
    So why is the ping not working. Some possibilities:
    The pfSense box is sending the ping request out the wrong interface. That should never happen, particularly if you're pinging via the Diagnostic in the webgui where you have to specify which interface to use. I assume you're specifying the correct interface.  ;)
    The cables are run incorrectly (NICs connected wrongly). That situation is normally pretty obvious because nothing works, however if you have disabled the firewall or have sufficiently open rules you might get some traffic.
    The modem is not receiving the ping requests. Hard to see how that might happen.
    The modem is not replying. That's distinct possiblity but why is it not replying? Perhaps it only replies to devices in it's DHCP table, have you tried DHCP on the pfSense WAN?

    The best way to get to the bottom of this is to run a packet capture on the WAN while you try to ping the modem. Then you can see if the packets are leaving and replys coming back.

    Steve



  • Ok, so ISP cable person provided a cable modem which has bridge mode.

    I changed pfSense from static WAN IP to DHCP.
    Connected new modem (in bridge mode) into pfSense and no Internet?

    I connect bridged modem or non-bridged modem into computer and Internet.

    I'm researching now if I'm missing a pfSense setting, as pfSense is a fresh factory reset with default settings for now.



  • Tested pfSense > Diagnostics > Packet Capture > 8.8.8.8 > Start > Stop > no results.
    pfSense > WAN > DHCP > green arrow up > IP 0.0.0.0.

    Rebooted modem several times.
    Rebooted pfSense several times.

    Shouldn't pfSense be receiving an IP from the modem in bridge mode (like the public IP the modem passes through)?
    Could there be a MAC address block or something from the ISP. I don't understand the MAC address allocation of IPs yet, so perhaps someone could help me with that angle of troubleshooting?


  • Moderator

    Doing a packet capture is meaningless if there isn't any traffic going to that ip at that moment.
    As you don't even get an IP on the DHCP set WAN interface I'd check the cable modem/pfS connection/cabling.


  • Netgate Administrator

    Indeed if you do a packet capture you want to capture everything, don't filter just traffic to/from 8.8.8.8.
    At the very least you should see the DHCP requests, and seemingly no replies.
    Yes the pfSense box should be getting an IP from the modem. Using modem only device and receiving a public IP is the situation where the modem may only talk to one MAC address though. If you plugged in your client directly you may need to power cycle the modem to get it to talk to the pfSense box.

    I think we have to get basic here!  ;)
    Since you have the green up arrow the NIC is clearly seeing the connection, the cable can't be completely broken.
    Speculation: the cable has a broken conductor such that not all pairs are working and it's trying to talk at 1Gbps and can't. What does Status: Interfaces: show for the WAN connection? Errors?
    What NICs are these? Are they both the same? Is it obvious which one is WAN? We have seen people struggle for a long time with NIC connected incorrectly.

    Steve



  • First, let me sign in so I can give stephenw10 applause.
    This is the reality of troubleshooting…patience, persistence and politeness.

    So, I should mention I have also tried:
    pfSense > Interfaces > WAN > MAC address > WAN NIC MAC address > Save > Apply.
    pfSense > Interfaces > WAN > MAC address > LAN NIC MAC address > Save > Apply.
    pfSense > Interfaces > WAN > MAC address > Modem Device MAC address > Save > Apply.
    pfSense > Status > Dashboard > WAN > IP 0.0.0.0.

    Cable should be ok, as the modem direct to computer works fine with Internet.
    I might trying swapping the cable tomorrow if pfSense still won't allow Internet.

    What NICs are these? Not sure if you mean the brand? The NICs are 1000baseT full-duplex.
    Are they both the same? Same brand?
    Is it obvious which one is WAN. Yes, I am clear on the WAN and LAN (yes, it is often confused). This is confirmed by connecting the modem directly to the computer. Computers can still connect to pfSense via the LAN port 192.168.1.155.


  • Netgate Administrator

    You mean you have tried spoofing the WAN NIC MAC address to those things? You shouldn't have to spoof the MAC. Also I seem to recall that some NICs don't respect the commands behind that tool. Check it with ifconfig.

    I meant that if your NICs are, for example, both Intel then they will appear as em0 and em1. If they are both cards then there is no easy way to know which is em0 and which is em1. One good test is to unplug the cable from the WAN NIC and check that the green arrow changes to red or look at the media status in ifconfig.
    The only reason I suggested it really is that initially your WAN subnet mask included the LAN such that a machine connecting to the WAN NIC with a LAN address might still be able to access the webgui.

    Do you know if the modem is gigabit?
    Something I've seen before a few times is that a cable with one bad wire can support a 100Mbps no problems but won't run at 1Gbps. Worse though a gigabit connection only uses 2 pairs to negotiate the speed but requires all 4 to actually send traffic so you end up with nothing at all. Hence a machine with a 10/100 card work perfectly and a gigabit card doesn't work at all. I agree it doesn't look like this is your problem but it's an easy test to just swap out the cable.

    What NICs are they? Did I already ask that?  Too many threads! ::)

    Steve



  • Ok, so I removed the pfSense > Interfaces > WAN > MAC address, from the WAN NIC.

    WAN NIC is clear which it is, yes, due to pulling out and seeing WAN down and red, rather than up and green when I connect the modem back into the pfSense WAN port.

    I didn't have an em0 and em1 on pfSense factory reset.
    I had re0 and re1.

    The WAN NIC is a MOBO NIC (motherboard Network Interface Card). Motherboard MSi Z87-G43.
    The LAN NIC is a TP-LINK TG-3269 10/100/1000Mbps PCI Adapter.

    I tried another Ethernet cable and same issue.

    I can't really change the WAN NIC without changing the MOBO. I thought this was a good MOBO so I am inclined not to focus on the MOBO.
    Perhaps I need to buy an ALIX or something?
    Is there a recommended pfSense hardware that uses little power?


  • Moderator

    You haven't accidentally set the MAC addresses on WAN or LAN to addresses, that the modem or Clients use in any case? As Steve already mentioned you shouldn't have to override the MAC address on your NICs. Only case I had to do that was when a cable company insisted on binding your static public IPv4 address to a MAC address and we switched the box pfSense was running at. So I had to override the original MAC with the old one so we get our static IP back. But otherwise you shouldn't have to set a value there.

    For devices with little power: that depends on the bandwith, packages you wanna run on it and states it needs to handle. For small environments, an ALIX or the bigger APU should be a good match. I also have a few Soekris devices running smoothly or some bigger devices from Lanner Inc (Atom D510 based boxed with 4-6 NICs).

    Greets


  • Netgate Administrator

    Ok, so you have Realtek NICs, re0 and re1.

    Sometimes, with usually no explanation, a particular piece of hardware just won't work with some other piece of hardware. I have run into this only a few times, most recently a Realtek NIC that couldn't talk to my SMC switch no matter what OS/driver I used. Since you're seeing a link negotiated it's probably not that but even so.
    Try swapping the NIC assignments such that WAN is using the TP-Link NIC (which is known to be good).

    I have to say that establishing a link correctly and seeing that correctly in software but not actually being to send any traffic down it is an odd one.  :-\

    If you run a packet capture on WAN without any filtering, you don't see any traffic at all?

    Steve



  • Thanks for the suggestion. I swapped the NIC assignments.

    pfSense > Interfaces > (assign) > WAN: re0 to re1.
    pfSense > Interfaces > (assign) > WAN: re1 to re0.
    Connected modem to pfSense WAN TP-LINK NIC.
    Connected router to pfSense LAN MOBO NIC.

    No Internet.
    Unable to ping pfSense LAN 192.168.1.155.
    Can't access pfSense any longer on computer to change NIC assignment back.
    I guess I'll need to stick a keyboard and monitor into pfSense and hopefully be able to fix swap the NICs back.

    This seems to indicate the MOBO NIC is faulty. The NIC had one green LED and one orange LED.
    The TP-LINK NIC (the usual pfSense LAN) always has 2 green LEDs.

    Seems I might need a new NIC so I bought:
    http://www.mini-box.com/APU-1C-AMD-G-Series-T40E-APU?sc=8&category=1361
    http://www.mini-box.com/Enclosure-1d2-Alix-2-red

    I couldn't find anything with wireless, because at the moment I have an 8 port Switch connected to a 4 port Wi-Fi router connecting to the pfSense LAN.
    Perhaps an ALIX with Wi-Fi might be better?


  • Netgate Administrator

    If it is just a bad NIC then you can always add a second network card. Plenty of spare slots on that mobo.
    If you can you should hook up a console and check that the pfSense box is working correctly via the TP-Link NIC.

    You already bought that APU board?
    You can add wifi to the APU board, it has miniPCIe slots and the cases usually have antenna cut-outs. I'm not familiar with the minibox case though if it's different to others.

    Steve



  • Okay,
    so my new ALIX board arrived.
    http://www.mini-box.com/APU-1C-AMD-G-Series-T40E-APU?sc=8&category=1361
    http://www.mini-box.com/Enclosure-1d2-Alix-2-red

    I think I need a power supply and SD card or mini SD card? What size SD or mini SD to fit pfSense with the Snort package?
    How do I do the Wi-Fi thing stephenw10 mentioned below? This would mean I can remove my old Wi-Fi router currently providing Wi-Fi.
    This would remove the 4 ports aswell that I use.

    Is there a good switch with 12 - 20 ports that anyone recommends with the ALIX board?

    This is my first time to install pfSense on the ALIX board. Do I connect a mini SD or SD in, connect an Ethernet to the ALIX, then SSH in to install somehow?


  • Moderator

    @eiger: I'd recommend open another thread as your last post has nothing to do with blocked internet anymore, does it?

    As the installation of an APU is a whole different can of worms, I'd say to open a new thread to not confuse people away from helping you with that. BTW: You don't own an ALIX Board. You bought an APU. This is NOT the same board at all. Don't mix and match them, as that will confuse the heck out of people ;) The APU is an amd dualcore chip as the ALIX was a x86 compatible thingy with far less cpu power and RAM. So even if PCengines used the Alix casing (as the boards are the same size and similar layout), you don't have an ALIX :)

    I'd recommend trying it with a USB stick and then either buying an SD card with 4GB or 8GB or going with the mSATA module and install it there if you plan to use a few packages.

    Greets