Bridging two interfaces WAN(vRouter) & LAN(LAN Router) w/ OPT1(MGMT)

[juesor]

Version: 2.5.1

WAN (ToVrtr)
LAN (ToLANrrt)
OPT1 (MGMT) 192.168.10.10/26 GW 192.168.10.1
OPT2 (Bridge)

System / Advanced / System Tunables
net.link.bridge.pfil_bridge 1
net.link.bridge.pfil_member 0

Interfaces / Bridges
BRIDGE0 WAN,LAN

Interfaces / Interface Assignments
WAN vmx0
LAN vmx1
OPT1 vmx2
OPT2 BRIDGE0

Firewall / NAT / Outbound
Disabled Outbound NAT

Ok so my LAN router has a VLAN interface 100 with the IP of 172.22.0.49
And my vRouter on the other side of pfsense is on VLAN 110 with the IP of 172.22.0.50

WAN: vl110
LAN: vl100
OPT1: vl1022

So I want traffic coming from VLAN 100 to go through pfsense and reach the vrouter on VLAN 110. Transparent bridging. I have followed the following guides and I cannot ping through pf for the life of me.

https://support.adamnet.works/t/running-on-a-transparent-pfsense-bridge/79
https://static.spiceworks.com/attachments/post/0016/4768/Transparent_Firewall.pdf
https://lawrencesystems.com/how-to-setup-a-transparent-bridge-firewall-with-pfsense-and-suricata/
https://freekang.tistory.com/attachment/cfile22.uf@264B543654D067143175CD.pdf

Any help, pointers, and/or direction to see what's changed in the last 10 years for why I cannot do this. Would be greatly appreciated.

[NollipfSense]

@juesor It seems that you followed instructions; however, It would be easier to see your setup if you post screen shots. You must have missed a step somewhere. Most likely the issue is a firewall rule to allow traffic form LAN (V100) to OPT1.net ...

[juesor]

@nollipfsense

Pictures.

I rebuilt the pf box this morning with a fresh install.

[NollipfSense]

@juesor Got it working the way you want? Just to be clear, you understand what the links share that the bridging is not recommended.

[juesor]

@nollipfsense

No, I still cannot access the vyos device through the bridge.

I will turn the filtering back on and lockdown rules but for the setup state, I disabled it and it's still not working.

Does anything look wrong is my screenshots?

[juesor]

So right now both ends of the bridge cannot ping each other.

But packet capture using the opt2 (bridge) shows ARP requests when I ping from either side.

who-has 172.26.0.49 tell 172.26.0.50
who has 172.26.0.50 tell 172.26.0.49

But PF is not passing the bridge traffic.

[NollipfSense]

@juesor Show your firewall rules for LAN ... that's where your issues are, I believe.

[juesor]

@nollipfsense

I added any any allow rules in pf under
To_VYOS - WAN
To_3850 - LAN
OPT2 - BRIDGE0

Status / System Logs / Firewall /Normal View

Shows no blocks since May 7 12:45:54

[NollipfSense]

@juesor Do you really want any to connect to Opt2 or just source LAN.net to connect to Opt2.net?

[juesor]

@nollipfsense

Right now I want WAN and LAN to connect to each other over OPT2(Bridge).

Anything that lives in WAN and anything that lives in LAN should be able to bridge through pf and get to the other side.

[juesor]

Here is a quick drawing.

MGMT connects between pfsense and 3850 on a different vl810

[juesor]

Part of me is thinking that due to the mgmt interface and its gateway pf isn't bridging the interfaces.

But I cannot find any reason for this to not be working.

[NollipfSense]

@juesor Have you looked at this: https://docs.netgate.com/pfsense/en/latest/bridges/index.html

From the above reference: "Also, in order for these functions to work, the IP address on the bridge must be the address used by clients as their gateway. These issues are discussed more in-depth in Bridging interoperability."

You seems close, keep working at it.

[juesor]

@nollipfsense

From that documentation, my setup is going to be a transparent firewall and it says i don't need to IP anything which is where we started.

Internal/external bridges connect a LAN to a WAN resulting in what is commonly called a “transparent firewall”.

Internal/External Bridges
An Internal/External type bridge, also known as a “transparent firewall”, is used to insert a firewall between two segments without altering the other devices. Most commonly this is used to bridge a WAN to an internal network so that the WAN subnet may be used “inside” the firewall, or internally between local segments as an in-line filter. Another common use is for devices behind the firewall to obtain IP addresses via DHCP from an upstream server on the WAN.

In a transparent firewall configuration, the firewall does not receive the traffic directly or act as a gateway, it merely inspects the traffic as it passes through the firewall.

Note

Devices on the internal side of this bridge must continue to use the upstream gateway as their own gateway. Do not set any IP address on the firewall as a gateway for devices on a transparent bridge.

[NollipfSense]

@juesor In my research, I came across this image which claim to need three NIC as yours, except your is virtual.

[juesor]

@nollipfsense

Isn't that exactly what my setup shows?

Did that website explain anything more.

[NollipfSense]

@juesor Yes, here is the PDF: http://users.ox.ac.uk/~clas0415/assets/Setting-up-pfSense-as-a-Stateful-Bridging-Firewall-with-commodity-hardware.pdf

[juesor]

@nollipfsense

Still no go.

Followed that pdf guide to the T and I also changed the bridge from WAN & LAN to WAN & OPT.

I'm wondering if anyone else has ever seen it when the bridge would basically not work.

When I packet capture on the interface facing the 3850.

13:58:30.484256 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:30.484445 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:35.490849 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:35.491015 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46

Great pings from the VLAN interface are looking for who has 172.26.0.50

Let's try that in reverse from VYOS trying to ping the 3850.

14:49:43.800853 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 1, length 64
14:49:44.830652 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 2, length 64
14:49:45.854536 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 3, length 64
14:49:46.878667 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 4, length 64
14:49:47.902641 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 5, length 64
14:49:48.894546 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:48.927016 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 6, length 64
14:49:49.918516 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:49.950569 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 7, length 64
14:49:50.942582 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:50.974568 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 8, length 64

Ok so the ICMP from VYOS is being seen.

Let's do the same but pinging from VYOS and capture on the 3850 interface.

14:51:08.286935 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:08.286959 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:09.310991 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:09.311011 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:10.335411 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:10.335433 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:11.358975 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:11.358984 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:12.383035 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:12.383045 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46

I don't see the ICMP but for some reason, that looks like it is working?

Ok getting into VYOS I can run a tcpdump

vyos@vyos:~$ monitor traffic interface eth2
tcpdump: verbose output suppressed, use -v or --v for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes

ok sweet the tcpdump is running let's see if we see anything when pinging 172.26.0.50 from the 3850.

#ping 172.26.0.50
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.0.50, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Crap.

Nothing in VYOS showing packets are received and failing

Now let's keep running the packet capture on VYOS and capture on the pfsense. starting at the to_3850 and then to_vyos.

CRAP!!!

The to_3850 shows NOTHING

The to_VYOS shows NOTHING

What is going on here?¿?¿

[A Former User]

@juesor

First of all, you must examine your Layer2 Topology and ensure you won't create any Layer2 Loop with this Setup. If all Uplinks of the ESXi are connected to the same Layer2 Segment proceed with cautions. Spanning-Tree won't save you here, cause STP-BPDUs are filtered on the ESXi per default.

Spoiler

You need to allow Forged transmits and Promiscuous Mode on the used Port Groups for the bridged interfaces. This is a necessary configuration on the Virtualization Platform to make this setup work. After Changing these settings you have to either reboot the firewall or disconnect and reconnect the interfaces to allocated a new virtual port with the updated settings.

Better to configure MAC-Learning for these Port-Groups. This Option is available on Distributed Switch since version 6.6.0 through API Calls.

[juesor]

@artes

Thanks. I set promiscuous mode on the vlan's within ESXi.

It still isn't working.

So i stood up a new lab.

3 vlans in esxi

910
911
912

4 vm's

v1 and v2 are windows 10 machines to simulate traffic over the bridge
w1 is the admin lan connection
And pf is in the middle

For 1 brief second, I get a good ping during a reboot of the pfsense server.

Now it makes me think that there is something filtering this traffic in pfsense but without anything in the logs and only arp requests found in a packet capture.

What could i look for to figure this out?