pfSense behind ISP modem (Double NAT) trouble

clarkx86 · Jan 7, 2025, 10:26 AM

Hello dear netgear Community,

I used pfSesnse for a long time without problems back before I moved recently using a fiber cable modem and logging in on the WAN interface with PPPOE with my carrier. Now after I moved, I'm forced to use my landlords ISP cable modem (Vodafone Germany). This modem has the ability to be put in bridge mode, but I can't enable it or put the pfSense box in a DMZ (the router is dumb and doesn't have this feature), so I have to fall back to double-nat.
I've got my new pfSense box set up real quick, with the WAN interface grabbing an IP from my modem via DHCP (192.168.0.100) and created a LAN interface on VLAN 10 (192.168.10.0/24) for my main network I want to use. I configured my managed switch, and set some ports to PVID 10 to join the VLAN and the device successfully grabbed a DHCP address from the LAN interface (192.168.10.102).
I've created some rules to allow access to the WAN interface (like the default anti lock-out rule on the LAN interface), so I still can access the pfSense from my modems network, as well as rule to any on both interfaces. I also set custom nameservers for the DHCP server on the LAN interface.
When I'm connected the VLAN 10, my host can ping other hosts in VLAN 10, except the firewall itself (192.168.10.1), even though the rule should allow it. I also don't have any internet access (though the nameservers on the host are the ones I set in pfSense). Weirdly enough, when I use the Web UI's ping tool, I can ping the internet from both WAN and LAN interfaces.

I've disabled the bogon network boxes on the WAN interface, created a gateway for the LAN interface, switched to Outbound NAT Hybrid mode and created a rule to translate 192.168.10.0/24 LAN to WAN, and tried to set NAT Reflection mode to Pure NAT, but I still can't seem to ping the firewall on pfSense and don't get any internet. I'm guessing I'm missing some routes or other critical configuration I'm missing.

I browsed a lot of threads here on this forum as well as on reddit, but I'm at the point where I thought I'd consult this forum :).

If you need any detailed screenshots/ rules, let me know.

Thanks in advance!

Gblenn · Jan 7, 2025, 11:02 AM

@clarkx86 To get started with a VLAN you only need one rule, the allow VLAN to any rule. Then you may want to add blocking rules to limit access to other VLAN's and default LAN.
Perhaps you can paste a picture of your rules? Just paste a snippet directly into the post.

No changes in NAT Outbound needed to get access to pfsense or internet, keep it at Auto unless you specifically need something like static port for some reason.

What do you mean "created a gateway for LAN"? The default gateway is WAN and it should work without changes...

You can also keep your block private and bogon networks on WAN, they only matter for traffic other than that coming from the modem.

BTW, you say it can be set in bridge mode? Did you mean it can't, since you say it's dumb?

clarkx86 · Jan 7, 2025, 11:46 AM

@Gblenn Thanks for the quick reply!

Here are some screenshots of my configuration:
Interfaces > Interface Assignments

Firewall > Rules > WAN

Firewall > Rules > LAN

Firewall > NAT> Outbound

System > Routing > Gateways

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

BTW, you say it can be set in bridge mode? Did you mean it can't, since you say it's dumb?

The bridge mode has to be activated over the ISP website and not on the modem itself, and I don't have access to this account. We're trying to get his credentials to the ISPs website to login, but it's a lengthy process, so I just thought I just go with double NAT.

EDIT: Maybe it's good to mention that I run this on a single NIC with VLANs, but only VLAN 1 and VLAN 10 are used at the moment. Though I wouldn't think that makes any difference since I can ping other hosts on the VLAN, just not the firewall itself, and no internet.
Also, I'm just caring about IPv4 right now before I try anything with IPv6 :D.

Gertjan · Jan 7, 2025, 12:09 PM

@clarkx86

pfSense on a device using just one physical NIC ?
Never saw one myself, but I saw the movie.
From what I recall, you do need a L2/L3 capable switch.

edit :
Stupid me, forgot the source : pfsense with one NIC.

stephenw10 · Jan 7, 2025, 1:59 PM

Router-on-a-stick is a valid method, though I wouldn't use it unless you really can't have two NICs.

Anyway that should work as long as your switch is configured correctly. And it appears to be at least mostly if LAN clients are able to pull a DHCP lease. Is that lease actually coming from pfSense?

Gblenn · Jan 7, 2025, 2:24 PM

@clarkx86 Hmmm, your "anti lockout rule" looks overly permissive... I'd say you have opened up to the world actually... remove all of them...

You already have an anti-lockout rule on LAN, where it should be... If you need to access your pfsense UI from the outside, it's generally adviced to use a VPN, so you access it from the inside instead.
Otherwise the LAN rules look fine even though the default rule normally has the LAN subnet as the source, I think, unless I changed it myself...

Outside of those two things, it all looks good which leads me to think it's you switch setup that isn't entirely correct. So you have to make sure that the port attached to pfsense is TAGGED for VLAN 10, and the ports used for the PC's and other devices are UNTAGGED VLAN 10 (and have PVID 10 set as well).
What switch are you using?

clarkx86 · Jan 7, 2025, 2:52 PM

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

@clarkx86 Hmmm, your "anti lockout rule" looks overly permissive... I'd say you have opened up to the world actually... remove all of them...

Normally I wouldn't do that, but I wanted to access the pfSense from the modems network before setting all VLANs and PVIDs and I thought because it's actually behind my modem with it's own firewall (I have no ports open at all), it should be as "safe" as all my other hosts on the modems main network. Correct me if I'm wrong however!

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

the ports used for the PC's and other devices are UNTAGGED VLAN 10 (and have PVID 10 set as well).

I made sure and it does work, a device attached to the VLAN 10 port even gets a IP from the pfSense LAN DHCP (192.168.10.102).

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

What switch are you using?

I use a Zyxel GS1920-8HPv2.

The weird thing is that I can ping other hosts on the VLAN 10 192.168.10.0/24 network, except the pfSense itself 192.168.10.1, though I think the rules should allow that?

Maybe it's a problem with the single NIC and the WAN on VLAN 1, but I'm not sure how, since I do get an IP address on both interfaces.

elvisimprsntr · Jan 7, 2025, 3:31 PM

@clarkx86

Pick up a used Qotom for ~$50 and save your sanity trying to use router-on-a-stick VLANs

eBay

stephenw10 · Jan 7, 2025, 5:08 PM

Try running a packet capture in pfSense whilst trying to ping it from something in the LAN. See what it actually arriving there.

Gblenn · Jan 7, 2025, 5:56 PM

@clarkx86 said in pfSense behind ISP modem (Double NAT) trouble:

Normally I wouldn't do that, but I wanted to access the pfSense from the modems network before setting all VLANs and PVIDs and I thought because it's actually behind my modem with it's own firewall (I have no ports open at all),it should be as "safe" as all my other hosts on the modems main network. Correct me if I'm wrong however!

Yes it's ok in this case, since you do have a firewall in front of pfsense.

@clarkx86 said in pfSense behind ISP modem (Double NAT) trouble:

I use a Zyxel GS1920-8HPv2

I took a peek at the manual for that switch but I'm afraid I couldn't really make sense of the VLAN settings in it...

Is there a status screen where you see your switch ports and their VLAN settings somehow?? If so perhaps you can paste it here?
Also the VLANs tab in pfsense (under interfaces), I suppose it has the LAN interface there doesn't it??

clarkx86 · Jan 7, 2025, 7:04 PM

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

Is there a status screen where you see your switch ports and their VLAN settings somehow??

Yes, here is my VLAN overview. Mind you this sitting on my desk as a test setup. Port 1 is the pfSense with tag 1 & 2 and Port 2 on the switch is my test device attached to VLAN 10. Ignore Port 9, it's for trunking to another switch.

The modem is just connected in the default VLAN 1 network.

@elvisimprsntr said in pfSense behind ISP modem (Double NAT) trouble:

Pick up a used Qotom for ~$50 and save your sanity trying to use router-on-a-stick VLANs

This is a good idea, but unfortunately I'm stuck with hardware (it is a Realtek NIC I believe).

Maybe I should set the modem to a different VLAN ID than the default 1 and also use VLAN for the WAN interface...

@stephenw10 said in pfSense behind ISP modem (Double NAT) trouble:

Is that lease actually coming from pfSense?

Yes, I can confirm because the host configures itself with the correct nameservers I also specified in pfSense and has a correct IP in the range I assigned in the DHCP server settings on the LAN interface.

stephenw10 · Jan 7, 2025, 7:10 PM

I would remove port 2 as an untagged port on VLAN1 in the switch. Though that shouldn't actually prevent it working.

Then I'd pcap in pfSense and see what's happening.

Gblenn · Jan 7, 2025, 7:48 PM

@clarkx86 said in pfSense behind ISP modem (Double NAT) trouble:

Yes, here is my VLAN overview. Mind you this sitting on my desk as a test setup. Port 1 is the pfSense with tag 1 & 2 and Port 2 on the switch is my test device attached to VLAN 10. Ignore Port 9, it's for trunking to another switch.

I kind of looks ok, although it's confusing to see that ID 10 is listed as untagged for ports 1 - 10, which includes port 2. Perhaps it's a limitation of the UI, and I would have expected it to read 1, 3-10. Sicne you don't want any VID 1 traffic ending up on port 2... Are you sure you are actually seeing the devices picking up DHCP from pfsense or is it from the modem?

Port 1 being Tagged for VLAN 10 looks good though.

clarkx86 · Jan 7, 2025, 7:50 PM

@Gblenn said in pfSense behind ISP modem (Double NAT) trouble:

I kind of looks ok, although it's confusing to see that VID is listed as untagged for ports 1 - 10, which includes port 2. Perhaps it's a limitation of the UI, and I would have expected it to read 1, 3-10. Sicne you don't want any VID 1 traffic ending up on port 2... Are you sure you are actually seeing the devices picking up DHCP from pfsense or is it from the modem?

I set port 2 to PVID 10 so the traffic from this port always falls into VLAN 10, I will try to disable this port for ID 1 however.

Also I will do a pcap and report my results later.