Double NAT: cannot ping Comcast LAN interface from Pfsense WAN interface

yossarian3217 · Oct 16, 2021, 9:04 PM

First time using Pfsense and first time ever messing with anything other than Packet Tracer- so apologies.

I'm trying to setup a homelab and I know in general double NAT is bad, but I'm just trying to setup a temporary network that I'll ultimately be taking down and replacing the Comcast modemrouter with my Pfsense.

My network layout is:
Internet ----- Comcast modemrouter LAN IP 10.1.10.1 ------ Pfsense WAN 10.0.0.X/24 ----- Pfsense LAN 172.15.1.X/26

I'm using the latest version of Pfsense. I cannot ping from Pfsense WAN interface to my Comcast modemrouter LAN interface.

Things I have done/tried:

Created a firewall rule to allow ICMP any source any destination on Pfsense WAN interface
Unchecked "block private networks" and "block bogon networks" on Pfsense WAN interface
Tried different IPs assigned to WAN interface like IP on comcast internal network, IPs not on it
Made sure System>routing>gateways is set to WAN int to go to Comcast router LAN IP
Disabling packet filtering (disabling firewall and NAT). Curently NAT is default again
Checking System>Firewall logs and Packet Capture(not seeing any ICMP traffic besides RA advertisements when Im pinging to test
Checking ARP table of Comcast modemrouter and Pfsense. Comcast modemrouter has ARP entry for PFsense but PFsense doesnt have Comcast modemrouter

I'm really at a loss. Have been scratching my head for days and not sure what to do besides maybe reinstall Pfsense. Have googled a ton, but not seeing anything to fix in terms of this Double NAT setup. Help is super appreciated! I can provide whatever info upon request and not afraid to tear this down and redo if need be.

johnpoz · Oct 16, 2021, 9:23 PM

@yossarian3217

Pointless. Rules are inbound only, unsolicited traffic.. Has nothing to do with you pinging something outbound - is your router trying to ping pfsense?
Also pointless, as this is again unsolicited inbound traffic - is your router trying to ping pfsense?
Huh.. If your router is handing out dhcp to pfsense wan - why would you think changing it would have anything to do with the problem?
Huh? Pfsense would use the interface connected to a network to ping an IP on said network.. Gateway being set has nothing to do with talking to an IP on a network pfsense directly connected to.
Again why - there are rules (hidden) that allow pfsense to do really anything it wants outbound.
This was good test - do you see ARP.. Can not ping an IP if can not arp for it.
PFsense doesnt have Comcast modemrouter - ding ding ding.. And there you go - this would prevent pinging, this would prevent from sending traffic to it to get to the internet, etc. etc..

You need to figure out why you can not arp.. In step 6 when you sniffed, do you see arp being asked for.. Clear your arp table, prob has an imcomplete list for the

BTW - is this a typo?

modemrouter LAN IP 10.1.10.1 ------ Pfsense WAN 10.0.0.X/24

That would never work.. Even if the modem/router had a mask of /8.. Your saying pfsense gets a 10.0.0.x/24 IP when its set to be dhcp?? If your going to set pfsense to be static on your modem/routers lan network - then you need to validate that its on the same network, with the same mask, etc. For test, you should set pfsense wan IP to dhcp.. Does it get an IP?

yossarian3217 · Oct 17, 2021, 1:10 AM

@johnpoz Thanks for the reply. I apologize as I am very new to everything, so I am shooting in the dark in my troubleshooting. It is helpful to know that I wont be getting any success pinging without the ARP table of BOTH devices populated. Here are some things I tried/looked into based off what you said:

Went to WAN interface of pfsense:

changed from static to DHCP
Gave interface a hostname of “mypfsense”
Provided alias IPv4 address of 10.1.10.50/24 (so same network as Comcast router)

Went on Comcast router, logged into web gui:

Verified Pfsense is in Comcast router ARP table. I can see that now the ARP table is using the pfsense hostname I set instead of the MAC address it had before

Checked Pfsense ARP table:

At first saw Comcast router listed. Cleared ARP table and now it is gone
Also, noticed my pc connected to pfsense to use pfsense web gui is no longer listed. It never came back, but my web gui access is fine

Went to Pfsense gui>status>interfaces

Released and renewed WAN. Am seeing an IP of 0.0.0.0, so DHCP doesnt seem to be working
Noticed that status of WAN interface says “no carrier” and DHCP shows down. But after changing the speed+duplex setting of pfsense WAN int and waiting the status changed to up and DHCP up. Still no IP address showed and cant ping comast router. Still no ARP entry for CC router in pfsense

Took Windows 10 laptop:

Verified Ethernet adapter is set to DHCP. Plugged it into router and it got an IP address. So something on Pfsense must be up (obviously)

Googled some on the “no carrier” status. Then:

Changed WAN interface to speed+duplex 100baseTX full-duplex (after seeing this as a possible solution). Didn’t help
Checked ports of comcast router and see that port 1 (where Pfsense is connected) shows a solid orange link light and on the web gui shows connection speed of 100Mbps (after I set the above on my pfsense)

Pretty confused as to why the Comcast router can have the Pfsense ARP in its table, but the Pfsense cant reciprocate. The comcast router has even picked up the hostname for the pfsense, but DHCP isn’t working. The pfsense isn’t getting an IP, so cant even ping it with it set to DHCP since it isn’t getting an address. I have to get that ARP table populated, but seems layer 2 doesnt work, let alone layer 3 with DHCP now set

Anything else you think I ought to do? I didnt run another packet capture, but I can. I didnt see anything like ARP requests or ICMP traffic before, so I dont know how it'll play out

johnpoz · Oct 17, 2021, 1:20 AM

Here is the thing if dhcp should work, and its not.. And other devices get address via dhcp, then setting static isn't going to magic fix that. Something wrong..

You have something wrong with the port on the comcast or pfsense, or the cable if your seeing stuff like no carrier.. Does this isp device have other ports?

Try a different port, try a different cable that are known to work.

yossarian3217 · Oct 17, 2021, 3:17 AM

Hey again, @johnpoz . Really appreciate you helping me with this.

At first in the troubleshooting from my last note it said “no carrier” and “DHCP down” for WAN interface, but then it did switch back to up for both up again. This of course didn’t actually mean DHCP was working.

Testing physical connectivity:

Tried known good ethernet cable on a pc and it worked for DHCP from the CC router. Ran ipconfig /release and renew to ensure pc would get a fresh DHCP address. Worked fine. Using that cable with Pfsense now (it is the same cable I have been using with the pfsense, but just verified it works on pc for DHCP)
Plugged ethernet cable into different port of CC modemrouter (it has like eight ports on it). Was still seeing orange link light, rather than the green blinking link light I see for my desktop pc on different port. Looked it up and the link light indicates what speed that port is running- so I switched the speed+duplex for Pfsense WAN to 1000baseT full-duplex and got the link light to blinking green like desktop. This didn’t end up doing anything functionality wise though
I have a few interface ports on my pfsense device. One port is labeled WAN, another LAN, and the rest OPT1-4. I took Opt1 and added it to the interfaces. I set it to DHCP and it did get an IP address almost instantly 10.1.10.130 (which would be on comcast network). Checked my ARP table on pfsense and see two new entries: one is the IP mentioned above for OPT1 and the other is the IP of the CC router labeled under OPT1 as well. CC router sees Opt1 interface as well in its ARP table.Tried pinging CC router from OPT1 and worked!

Trying to configure OPT1 as the new Pfsense WAN port:

System>routing>gateways- changed interface here from WAN to OPT1
Checked Firewall>Nat>Outbound and looks to have auto selected OPT1
Noticeably OPT1 could successfully ping CC router, but the firewall rule I set earlier was for WAN port to be able to get ping replies back(not OPT1). I guess since OPT1 initiated the ping I just did, the ping replies went through since the FW saw this as legit. Just for kicks, tried pinging OPT1 from CC router and it failed; changed the FW rule to allow all ICMP on OPT1 port but pings from CC router still failed..?

So my questions now are:

Why can the CC router not ping OPT1?
Is OPT1 recognized by the Pfsense as WAN interface now because I set it as the gateway interface in system>routing>gateways? I just want to make sure an implicit deny all is going on with this interface now. Not sure if there are additional configurations to make an interface the “WAN”.
Do you think the WAN port of my device is defective? The ARP table just wasnt having it and CC router’s DHCP either.

stephenw10 · Oct 18, 2021, 9:58 PM

Can we see a screenshot of the firewall rule you added on OPT1?

If that is the only gateway defined on the system it will be the default gateway. As long as it's defined on the interface itself it pfSense will have added outbound NAT rules for it in automatic mode.

Potentially it could be a bad port causing that. The fact you had to set it to 1G fixed rather than autoselect implies some low level incompatibility. Are WAN and OPT the same NIC type?

Steve

yossarian3217 · Oct 18, 2021, 9:58 PM

@stephenw10 Hello and thanks for the reply!

OPT1 looks to be gateway/WAN port:
Since OPT1 is the only gateway out of the system and I have defined it as the WAN port in System>routing>gateways – youre saying it will act as a typical WAN port out of the box? Meaning it will have the implicit deny all applied? Just want to make sure. NAT is set to automatic and I see automatic rules set for OPT1 as the NAT address, so looks like OPT1 is recognized as the gateway.

Potential bad port:
I’m using the Protectli FW6D (6 port) NUC. It has six intel NICs on it. This is the link to it on the site: https://protectli.com/product/fw6d/ . The WAN and OPT1 ports as far as I know are exactly the same.

I had set the actual WAN port(not OPT1) to the 1G to see why the link light on the CC router was orange rather than green and overall to see if that would affect the connectivity issue. Turns out the link light just indicates the speed, not actual connectivity. Also, seems changing the speed+duplex of the port didn’t help at all. I had read online that some people had issues with speed+duplex, but that doesn’t seem to be the case for me.

I guess I should contact Protectli, but I just want to ensure it’s a hardware issue and not somehow a misconfigured thing with Pfsense. My installation is pretty fresh, so I just don’t see how it could be a Pfsense misconfiguration.

CC router can’t ping OPT1 port:
Here is the only rule on OPT1. Apologies I took this pic with my phone. If there is a good log/diagnostic to check, please let me know. I’ve checked Status>System Logs> Firewall, but I don’t see any ICMP protocol traffic when I run the pings

johnpoz · Oct 18, 2021, 11:01 PM

@yossarian3217 said in Double NAT: cannot ping Comcast LAN interface from Pfsense WAN interface:

CC router can’t ping OPT1 port:

And what are the actual rules on this opt1 port - show all of them.. If your blocking rfc1918 and your CC router is using rfc1918.. Then no you wouldn't be able to ping, because that rule would be above your allow rule.

Also your destination should be the opt1 address, not any.

stephenw10 · Oct 18, 2021, 11:05 PM

Yep, that^. We need to see the actual rules list from the OPT1 interface not just one rule.

Steve

yossarian3217 · Oct 19, 2021, 11:26 PM

@stephenw10 @johnpoz

So, that rule I sent was the only rule I had set up on the OPT1 interface. I also failed to mention that I modeled the OPT1 interface after what I had the WAN interface configured to- which was to NOT block private or bogon networks.

But I just found out with more testing that my comcast router cannot actually ping any of my devices...So, not worried about that. My devices (including Pfsense) can ping the CC router and that's fine.

My only worry now is why the WAN interface didnt work with all the same settings configured as OPT1. Everything is the same between the two, but I'll take that up with Protectli if my own troubleshooting doesnt do anything.

Thank you both for the help! I'm hoping to become more proficient with Pfsense and incorporate it into my career, so it's been great to have good support just starting out. Appreciate ya'll