Multiple VLAN routing issue - works on F5 but cant emulate on PFS

[alexnyc]

Hi Everyone.
We have a bit of an interesting issue. Our production setup is currently on F5 7000S as main router and we are attempting to emulate a specific functionality on PFS.

Long story short, the basic issue is this:

Multiple VLANs defined on the device. (Public IP (/24) VLANs and NAT VLANS (RFC1918). The issue exists for all combinations of VLAN connection.

Issue example:
Consider a device, dual homed attached to 2 separate VLANs. Lets assume all subnets are /24 for my example.
For example NIC1 is attached to VLAN1 and has IP and GW from that VLAN. IP 192.168.0.50 Gateway 192.168.0.1
NIC2 is attached to VLAN2 and has ONLY the IP from that VLAN (no default GW). IP 192.168.1.50 NO GW
Now, we have a second device on VLAN1. IP 192.168.0.60 GW 192.168.0.1
When you attempt to ping (or connect) from 192.168.0.60 to the NIC2 IP of the Dual Homed machine above (192.168.1.50) you get a failure.
Now, i believe that the behavior is actually correct, as the second NIC does NOT have a default GW so that packet gets lost.
However, and this is the kicker - this behavior works with F5 being the main router, and we are being pressured to "make it work" just like F5 does with PFS.
Adding a default GW on NIC2 solves the issue (obviously), but it is probably not a good idea to have a machine with 2 default gateways on separate interfaces/separate VLANs.
Are there any advanced settings on PFS that would enable such behavior?

[viragomann]

@alexnyc said in Multiple VLAN routing issue - works on F5 but cant emulate on PFS:

When you attempt to ping (or connect) from 192.168.0.60 to the NIC2 IP of the Dual Homed machine above (192.168.1.50) you get a failure.

I would expect, that the request packet is routed through the router, since the destination IP is in the other VLAN. But normally the device should respond on NIC1, since the destination is in VLAN1 then.
I would expect, that this works for ICMP, but not for TCP.

However, and this is the kicker - this behavior works with F5 being the main router, and we are being pressured to "make it work" just like F5 does with PFS.

Can you sniff the traffic to see, how it behaves?
Maybe the F5 is masquerading the traffic, so the device would respond on NIC2. This can also be done on pfSense with an outbound NAT rule.

[johnpoz]

@alexnyc said in Multiple VLAN routing issue - works on F5 but cant emulate on PFS:

Consider a device, dual homed attached to 2 separate VLANs

Well yeah that is problematic for sure.. That is almost always a horrible idea, unless one of the vlans is only used for backup or storage without any gateways set on it, etc..

Can almost promise you your running into asymmetric traffic flow..

"router" is way different than a router/firewall.. You can for sure make it work via what @viragomann mentions in natting traffic, ie outbound so traffic is sure to route back the same way through your "firewall".. I personally would rethink what your accomplishing with multihomed devices..

[viragomann]

@johnpoz said in Multiple VLAN routing issue - works on F5 but cant emulate on PFS:

I personally would rethink what your accomplishing with multihomed devices..

At least, what's the sense of accessing a device which has an IP within the some VLAN by using the IP of another.

[alexnyc]

I agree with both of you on the reason.
Multihomed boxes are NGNX+ load balancers that the devs insist on having, they also have a good "defence" "it works on F5" so lets make it work on PFS.
I have explained that it is a bad idea but i have been overruled :)

Also, F5 is absolutely configured with asymmetric profile. The reason for this is that we are actually multiwan, aggregating 2 separate /30's from the ISP. The profile applies to the entire routing domain on F5, so that probably explains why F5 can handle it.

I tried playing with asymmetric routing options on PFS (advanced FW rules and advanced NAT option disable reply-to), but it did not help.

[viragomann]

@alexnyc
No, that has nothing to do with reply-to. If it is due to asymmetric routing, you can try to add a pass rule on both interface and allow sloppy states in the advanced options.

[alexnyc]

@viragomann
That is what i tried. Sloppy state on advanced rule.
Did not help.
Firewall already configured to allow any.
Is there anything specific needed on outbound NAT?
For public VLANs we bypass NAT entirely (For obvious reasons).

[johnpoz]

@alexnyc if you want help running in a asymmetrical flow a drawing would be every helpful.. It impossible to say where you need what without some clue to the traffic pattern.

[alexnyc]

Here is the diagram with the issue.

I verified this to be the same issue in simpler setup (isolated PFS with 2 VLANS and dual homed box attached to both, no fancy multiwan or public vlans).
The issue is only with dual homed boxes.
Setting default GW(10.10.10.1) on nic2 fixes the issue but is suboptimal for known reasons.

[johnpoz]

@alexnyc On vlan3 set an outbound nat so that when 77.4.5 pings 10.10.10.4 it looks like it comes from pfsense 10.10.10.1 address.

But why would you even want that to be used, why would 77.4.5 not just access 777.4.4?

I would put a outbound nat on that vlan3 so any source comming from 71.77.4/24 going to 10.10.10/24 would look like it came from 10.10.10.1