How configure VLAN WAN/UI access rules?

[Irk 0]

Thanks @viragomann, I've removed the useless WANDSL rule.

Thanks @stephenw10, I'll bear the reject rules point in mind (and revist the docs).

Having briefly "had" things working (pfSense's DNS has just stopped working, but that's a separate topic), here's my understanding of what was happening and what was needed to get things working:

I think the following rule is why VLAN 150 clients on 192.168.150.0/24 could access the pfSense GUI on 192.168.100.254 (i.e. TESTPOC Destination=* allows TESTPOC to go anywhere, including the MANAGEMENT LAN 192.168.100.254/24):

Well, that rule plus the fact the GUI is served on all LAN and even WAN interfaces.

For VLAN 150 internet access to work I had to make these changes to Firewall->NAT->Outbound (outbound NAT being mentioned in Troubleshooting Network Connectivity):

I guess without this, VLAN 150 packets went out to the internet but could never get back.

I now have the following rules which seem to work:

For those new to pfSense, the greyed-out first row above is a disabled rule.

Curiously, the TESTPOC "Allow traffic to internet" rule doesn't seem to need the WANDSL gateway set, even though I've set System->Routing->Gateways->Default gateway to None for IPv4 and IPv6 (I'm wondering if I may need to add WANDSL gateway back when I get round to bringing the other 2 WAN links online though).

This is probably a topic for another post, but I'm now having issues with pfSense's DNS Resolver, which I had working, but now isn't.

I had to add this entry to Services->DNS Resolver->Access Lists before VLAN 150 DNS client requests worked though (thanks @bingo600 for this post):

Before adding the above the following was reported in the logs (after increasing Services->DNS Resolver->Advanced Settings->Log Level):

I've got DNS working again from the pfSense diagnostic GUI (after adding 8.8.8.8 as DNS in System->General Setup), but VLAN clients still get "status: SERVFAIL" running e.g. dig @192.168.150.254 github.com (dig @8.8.8.8 github.com works fine).

I may persevere a little to see if I can work out what the DNS Resolver issue is, but as I plan to run a separate DNS server anyway won't spend too long on that.

[stephenw10]

Hmm, a few interesting things there.

You should not have had to add an outbound NAT rule for that, the automatic rules should contain all the internal subnets the firewall knows about, locally attached or routed.

Unbound should have ACLs to allow access from all internal subnets.

Both those suggest pfSense either doesn't see 192.168.150.0/24 or it doesn't see it as an internal subnet. The latter seems more likely and could only be caused by having a gateway set on the interface itself which would cause it to be seen as a WAN.

With the original policy routing firewall rule in place I would not expect to have been able to reach 192.168.100.254 because, although destination 'any' includes it, the presence of the gateway should force the traffic out of the WAN.

Your new rule to allow traffic to !This_Firewall will prevent hosts on vlan 150 reaching the webgui on any IP but it will allow traffic to reach any other host in any other connected subnet. Which is probably not what you want.

Steve

[Irk 0]

Thanks @stephenw10, that's really helpful .

I added another VPN (151 / 192.168.151.0/24) to port igb3 and can see that this was added to the automatic outbound NAT rules:

Perhaps I was tinkering with the gateway options for VLAN 150 before and that somehow flagged it as a WAN interface.

I couldn't see anywhere in pfSense that told me it thinks VLAN 150 was a WAN link though (as far as I could tell it didn't have a gateway assigned). After deleting VLAN 150 and re-adding it, it too was included in the outbound NAT rules, which are now just:

The fact I've lost 2 rows in automatic rules for VLAN 150 backs up your "pfSense likely thinks VLAN 150 is a WAN" observation (and makes me realise I should have included the automatic rules in my earlier post... sorry about that).

Once I get all the WAN and VLAN connectivity working I'll endeavour to adopt a better/default-deny strategy to address the !This_Firewall issue you pointed out.

I have a feeling my tempermental DNS may be related to this issue so I'm giving up on using pfSense's version of unbound for now and will endeavour to set up a seperate DNS server instead.

[stephenw10]

Yup, that fact it was adding outbound rules for TestPoc as a WAN confirms that. It must have had a gateway assigned at least at some point. I've no idea how it would still be using that if it wasn't shown on the interface config though.

I use Unbound everywhere and have never seen that issue. I guess it's possible.
If it was seeing TestPoc as a WAN though Unbound would be blocking queries from it by default.

Steve

[Irk 0]

Things make a bit more sense after some restarts "seemed" to break things even more.

The DNS Resolver seems to be working again now I've added a default IPv4 gateway back in System->Routing->Gateways.

I'd set both the IPv4 and IPv6 default gateways to None thinking it would be best to explicity define the gateway to use in this multi-wan setup. This mean't I had to specify the WANDSL gateway for VLAN 150, i.e. I had to use this rule (instead of the old disabled rule that I think worked courtesy of having a default gateway set):

But having no default route also meant the GUI's Diagnostics->Ping utility couldn't even ping 1.1.1.1 or 8.8.8.8; so there was no way Diagnostics->DNS Lookup was going to work either (or unbound).

Which leaves me wondering if it's possible to get unbound to work without having a default gateway set. That may be a question for another post...

[stephenw10]

You can set the outbound interface(s) for Unbound to use as source IP. In that case it should hit the 'route-to' for that interface and be sent to the gateway on it.

Steve

[Irk 0]

You can set the outbound interface(s) for Unbound to use as source IP

Thanks @stephenw10, I did try that but couldn't get it to work when no default gateway is set.

I set default gateways to None and the following DNS Resolver options:

Then rebooted pfSense and client dig requests just get status: SERVFAIL responses.

In this state pfSense's IPv4 routing table is:

Once the default IPv4 gateway is set to Automatic, client dig requests start working again; the routing table in this case being:

I've since found this post which claims:

"The outgoing interfaces option in Unbound is just an interface by interface on/off switch whether to use the interface at all for outgoing traffic, it doesn't allow any redirection of DNS queries, they will still follow the system routing table"

It seems unbound is only able to work when a default gateway is set. Which currently seems odd to me; I guess I have some more reading to do to understand how this could work in my (concurrent, rather than failover) multi WAN setup.

[stephenw10]

Yes, the outbound interface only sets the source IP that is used. But if that is a WAN the outbound firewall rule will look something like:

pass out  route-to ( em0 172.21.16.1 ) from 172.21.16.22 to !172.21.16.0/24 ridentifier 1000002761 keep state allow-opts label "let out anything from firewall host itself"

So traffic using it should routed via the gateway.

However PPPoE can be a special case here because the gateway in use is often not in the WAN subnet. It doesn't have to be in a point-to-point connection. It's still in your routing table on the link though....

Steve

[Irk 0]

OK, thanks @stephenw10, I see one can view those rules with pfctl so will look for similar entries on my box.

Should anyone else attempting to lock a VLAN to a single WAN interface wind up reading this, it turns out setting the default gateways to none wasn't the only way to do this (setting no gateways having the downside of stopping pfSense services such as DNS from working).

It turns out there's a "Skip Rules When Gateway is Down" option that I had to set under System->Advanced->Miscellaneous:

Until I set the above option the gateway I explicity set for the VLAN was being removed if it was down, meaning pfSense's default gateway was used instead (which being a gateway group with all WAN interfaces, meant traffic found a path out).

That seems an odd default thing to do to me; I'd expected the default to be no connectivity. If one wants failover to another gateway it may be clearer to explicitly say so by assigning a gateway group that includes the specific failover interface(s) required.

[stephenw10]

Yes, when I referenced that initially I suspected that might be the case. It's one reason why you should not rely on policy routing to block traffic. Use block or deny rules above it to prevent access via any gateway. But setting that will remove the rule entirely as you found.
It defaults to that because in most cases policy routing is used for failover or load-balancing and it's preferable to have connections succeed.

You can also see the ruleset pf loads in /tmp/rules.debug.

Steve

[Irk 0]

Eek. You did indeed mention the skip rules thing before! Sorry. I guess the significance of that statement didn't register with me at the time. Doh .

At one point in my journey I did try using reject rules to block the other gateways, but it seemed to me that the "default gateway" was treated as a special case and couldn't be blocked. It's entirely possible that I had things half-baked at the time though. Perhaps I will find time to revisit that approach at some point.

Is setting "skip rules when gateway is down" considered a bad practice, or something that should be avoided?

If there's a good reason to not use it I can revert to the default setting, but I'd prefer to take the approach of "block everything" and "only allow what one wants" that the "skip rules..." option seems to give.

For example, to allow VLAN 151 to access VLAN 150, but not VLAN 150 to access VLAN 151, with both having internet and DNS access, it appears I can just do the following (without having to explicitly block anything, which might be a pain should more VLANs be added later):

[stephenw10]

Yes, that should work. However generally considered best practice would be something like this:

If the 'pass-all' rule were to change to something without a gateway it would not then allow local traffic.

Steve

[Irk 0]

Thanks @stephenw10, the rule screenshot's really helpful.

I was able to get that rule set working (as long as the "skip rules..." option was selected).