Inter-VLAN Communication and WAN Connectivity Issues After Switching from PPPoE to DHCP/Static
-
@fxpro said in Inter-VLAN Communication and WAN Connectivity Issues After Switching from PPPoE to DHCP/Static:
WAN Interface >> vlanaddress >> * Source >> * Source Port >> * Destination >> * Port >> VPN_INTERFACE NAT Address >> * Port
Um, that doesn't make any sense. Can you give a screenshot of the rules on one of the VLANs.
Basically if the only passrule you have defined on there has a gateway set then all traffic will be forced via that gateway unless the gateway is down. Hence you would normally have bypass rules for locally routed traffic.
@fxpro said in Inter-VLAN Communication and WAN Connectivity Issues After Switching from PPPoE to DHCP/Static:
Please note that previous to this, whilst on my previous ISP, I NEVER had this problem at all, it was working solid as a rock with no dns leaks or any connectivity problems.
I assume you didn't have the additional gateway added with the previous ISP though? The pppoe session, like dhcp, should dynamically add a gateway when it comes up.
The only real issue I have seen here is there you have no default route in some situations. So anything not specifically policy routed will fail. With just the dhcp WAN configured it should set the dynamic gateway to be the default route when it comes up. Are you still not seeing that? Are there any errors shown in the dhcp log when it connects?
-
@stephenw10 thanks for the reply.
I think I've found out the reason for the issue, and so this 'could' be either the way I've configured the router or there must be some kind of bug in the pfSense firmware due to the routing. I shall explain here:
-
On the previous ISP I was using a PPPoE connection/configuration on the WAN. This worked perfectly and I never had any issues whatsoever. However there is one single configuration that is different on this connection compared to the DHCP connection/configuration.
Note: The config was as follows:
SYSTEM > ROUTING > GATEWAYS
Gateways:
PPPoE (Default IPv4) Interface WAN Gateway ###########
*** Default Gateway > Default gateway IPv4 was set to 'NONE' -
On the current ISP that requires the DHCP connection/configuration on the WAN, this is where the problems started and after 2 weeks of trying every single combination I could imagine, even using GPT to assist, I finally found a half solution, but it only solves 1 of 2 problems.
The configuration that now allows me to correctly connect to the public internet, and even after reboot it will return to normal public internet connectivity is as follows:
SYSTEM > ROUTING > GATEWAYS
Gateways:
WAN_DHCP (Default IPv4) Interface WAN Gateway ###########
*** Default Gateway > Default gateway IPv4 was set to 'WAN_DHCP'-
As you can see the difference, on the PPPoE connection the default gateway was selected to NONE and you might say that was already incorrect, I disagree because my Public Internet Access was flawless as well as the inter-VLAN connectivity through the PPPoE connection. Yet now when the default gateway is set to WAN_DHCP I get the public internet access but I lose inter-VLAN connectivity. It has nothing to do with firewall rules either, as you will see below.
-
This problem strongly suggests that the pfSense’s routing (and possibly NAT) behaviour is “pulling” my LOCALLY sourced packets toward the WAN rather than routing them internally. In other words, when the WAN interface is active and its default gateway is set, pfSense ends up sending (or “reply‐to’ing”) even local traffic out via the WAN instead of handling it directly between its local interfaces.
-
In order to test this, I disabled the WAN_DHCP from SYSTEM > ROUTING > GATEWAYS and the VPN was still connected at this point in time, it didn't drop, which is weird. I could ping google.com. I could not ping 10.3.3.2 in VLAN2. I then rebooted the pfSense.
-
On boot up, I could ping 10.3.3.2 in VLAN2 but I had no access to the public Internet, I could not ping google.com, obviously because I had disabled the WAN_DHCP. I then enabled the WAN_DHCP, and that didn't result in connecting to the public internet, but I could still ping 10.3.3.2. I then rebooted the pfSense.
-
On boot up, I could no longer ping 10.3.3.2 in VLAN2, but I had public internet access !
You can try this yourself, I've repeated this process more than 6 times and every single time it did the same thing happened, the same exact results occurred.
Therefore either there is some crazy mis-configuration on my part, or there is a major bug in the pfSense firmware that has something to do with DHCP connectivity and NAT routing when the DHCP is enabled, as described above.
As far as answering your question about the additional gateway not being added with my previous ISP... yeah I never added it, it was already given by the PPPoE connection. The strange thing is, the DHCP connection obtains the same information, its not like there is something missing. This is what makes it so strange and makes me think its a bug in the firmware.
By the way, I'm running the latest firmware 2.7.2-RELEASE (amd64) built on Tue Mar 5 6:53:00 AEDT 2024 FreeBSD 14.0-CURRENTI'd love to be wrong, I hope I've been a dill and configured something incorrectly, but as far as I can see, this is a bug with the pfSense regarding the routing. Somehow it appears that the presence of the default route installed by the WAN’s DHCP lease is “stealing” or overriding the local (directly connected) routing for inter‑VLAN traffic. In other words, when WAN_DHCP is enabled, even though the local routes are correctly present, pfSense’s routing (or its state reply‑to logic) is forcing return traffic out via the WAN interface rather than through the directly connected interfaces.
Any ideas on how to resolve this problem?
At this point, I'm completely out of ideas which means I'll be tossing the pfSense if I can't resolve this problem. I'm unable to do any work on this network at all. It shouldn't be this hard.
Please help.Heres the image you wanted... although it doesn't prove much because I haven' proven it works when the WAN_DHCP isn't connectetd, you can see I've opened it right up and yet still the packets aren't routed where they should be:
-
-
There is no rule there allowing a host on VLAN3 to ping a host on VLAN2. Or rather the only rule that would allow it is disabled.
How exactly are you testing? What IP are you trying to ping from?
-
@stephenw10 how is there no rule to ping to VLAN3?
look here:
How is this not a rule?And if it was incorrect, how come it works when I remove the WAN_DHCP? Doesn't make sense.
In relation to your question about the rules and where I'm pinging from...
I'm pinging from (VLAN2) 172.16.2.150 to (VLAN3) 10.3.3.2
Thats exactly what the rule above does.
I can even make it specific using the actual IP Addresses, it makes no difference, the same result occurs.Additionally, I have made a rule also where ALLOW ALL/ANY and I still get the same result, I cannot ping whilst WAN_DHCP is enabled. The moment I disable the WAN_DHCP and reboot the pfSense, when its booted up again, I can ping between the VLANs as correctly specified in the image.
The rule you see disabled is on purpose, but even with it on doesn't change anything. That rule is for VLAN3 to ping VLAN2 which is not what I'm doing as mentioned above. Either way, with it enabled, changes nothing because I ping from VLAN2 172.16.2.150 to VLAN3 10.3.3.2.
I don't know what else to do, I'm completely at a stand still. Please help.
Any idea why its doing this?
do you want to see more config?
I can also show you logs if you need? -
If you are pinging from VLAN2 to VLAN3 then the rules governing that are on VLAN2. Your screenshot above shows the rules on VLAN3 so none of those apply.
What rules do you have on VLAN2?
-
@stephenw10 here are the VLAN2 rules:
As you can see, VLAN2 allows traffic to VLAN3
Thus:
VLAN2 ---> VLAN3 is here:
and on VLAN3, allows traffic from VLAN2 to VLAN3 here:
Again, this has nothing to do with the problem.
As I've shown previously, this was working when I was on the PPPoE connection.
In fact, I can also get this working if I Disable WAN_DHCP !!!But the moment I Enable WAN_DHCP, I lose inter-VLAN connectivity !
This has absolutely NOTHING to do with firewall rules and from what I can see, everything to do with Routing to 'a' gateway. The problem is WHY do the packets route to the WAN gateway for internal inter-VLAN traffic when it should be routing to the local network gateway/s, when I have enabled WAN_DHCP ?When WAN_DHCP is disabled, I will get the correct routing to the local gateway for inter-VLAN traffic. Why is this occurring? Maybe there is a section in the GUI where I can force this?
If I can find the reason why this is happening, I can solve the problem, because to me it is clear this is a Routing to gateway problem and I can prove it repeatedly, if I disable WAN_DHCP because I will regain inter-VLAN connectivity.
- Surely this is a bug?
- Is there a work around?
- Can I make a bug report to Netgate? if so where?
Please help, getting desperate now.
-
Ok so that top rule will catch all traffic from anything is VLAN2 and force it via the VPN gateway. The only time that doesn't happen is if the VPN is down.
You need a rule above that to pass traffic from VLAN2 to VLAN3. And if you want that to allow pings it must be either 'any' protocol or ICMP.
Firewall rules (on any tab except floating) are applied to traffic coming into that interface.
So the rule you have with source VLAN3 subnets there can never match any traffic. All traffic on the VLAN2 tab will have source VLAN2 subnets. The same is true for some rules you have on the VLAN3 tab.So it looks like this explains what you are seeing with regards to the gateway. When the system has a default gateway the VPN connects that then the top rule forces all traffic via that preventing pings between VLAN2 and VLAN3. Without a default gateway set the VPN cannot connect so the VPN gateway is invalid. The default behaviour is for the rule to be applied without a gateway set so that then allows pings. That can be changed.
But adding a rule to by-poass policy routing for local traffic on VLAN2 will correct this.
-
@stephenw10 Thanks mate, you've just proven I'm a complete dill
I modified the rules to allow ANY and then move the rules to the top and voila' I get the Inter-VLAN connectivity.
I don't know how to say thank you, You've really made my day.
Thank you so very much.Problem solved. I can now ping/access all VLANs (after adjusting the order of the rule accordingly to the top). It seems that for there not to be problems, the inter-VLAN configs need to be above any WAN/VPN connections. I hope this helps anyone else that is a dill like me
-
@fxpro It certainly helps to know, as @stephenw10 said, that rules work from the top to bottom. But the key here is the moment a rule catches some form of traffic, no further rules below will apply to that traffic.
Also, check the numbers in the states column, where 0/0 indicates the rule has not been used at all.
Also good to know, when you created your VLAN, you already have one invisible rule in place, the default deny all. So nothing goes out anywhere until you allow it to.
Typically the first rule to add, is that "default" allow VLAN-X to any rule. This will give VLAN IP's internet access, as well as access to any other subnets in your setup.
However, one key reason for VLANs is to isolate different types of traffic or users from each other. So most commonly you would add deny rules to block inter VLAN access, above that all to any rule. Something like what I have for my GuestVLAN used on wifi, allowing internet access but nothing else.
If you then want to add some policy that allows some IPs to access another subnet, you start adding those rules above the block rules. Perhaps you have a PiHole on LAN that you want to use also for VLAN2 and VLAN3, so then you create allow rules from VLAN2 and VLAN3 respectively with Destination PiHoleIP, and so on.
-
Glad you were able to get it sorted.
