Bridging two interfaces WAN(vRouter) & LAN(LAN Router) w/ OPT1(MGMT)
-
@juesor Have you looked at this: https://docs.netgate.com/pfsense/en/latest/bridges/index.html
From the above reference: "Also, in order for these functions to work, the IP address on the bridge must be the address used by clients as their gateway. These issues are discussed more in-depth in Bridging interoperability."
You seems close, keep working at it.
-
From that documentation, my setup is going to be a transparent firewall and it says i don't need to IP anything which is where we started.
Internal/external bridges connect a LAN to a WAN resulting in what is commonly called a “transparent firewall”.
Internal/External Bridges
An Internal/External type bridge, also known as a “transparent firewall”, is used to insert a firewall between two segments without altering the other devices. Most commonly this is used to bridge a WAN to an internal network so that the WAN subnet may be used “inside” the firewall, or internally between local segments as an in-line filter. Another common use is for devices behind the firewall to obtain IP addresses via DHCP from an upstream server on the WAN.In a transparent firewall configuration, the firewall does not receive the traffic directly or act as a gateway, it merely inspects the traffic as it passes through the firewall.
Note
Devices on the internal side of this bridge must continue to use the upstream gateway as their own gateway. Do not set any IP address on the firewall as a gateway for devices on a transparent bridge.
-
@juesor In my research, I came across this image which claim to need three NIC as yours, except your is virtual.
-
-
@juesor Yes, here is the PDF: http://users.ox.ac.uk/~clas0415/assets/Setting-up-pfSense-as-a-Stateful-Bridging-Firewall-with-commodity-hardware.pdf
-
Still no go.
Followed that pdf guide to the T and I also changed the bridge from WAN & LAN to WAN & OPT.
I'm wondering if anyone else has ever seen it when the bridge would basically not work.
When I packet capture on the interface facing the 3850.
13:58:30.484256 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:30.484445 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:35.490849 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46
13:58:35.491015 ARP, Request who-has 172.26.0.50 tell 172.26.0.49, length 46Great pings from the VLAN interface are looking for who has 172.26.0.50
Let's try that in reverse from VYOS trying to ping the 3850.
14:49:43.800853 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 1, length 64
14:49:44.830652 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 2, length 64
14:49:45.854536 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 3, length 64
14:49:46.878667 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 4, length 64
14:49:47.902641 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 5, length 64
14:49:48.894546 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:48.927016 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 6, length 64
14:49:49.918516 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:49.950569 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 7, length 64
14:49:50.942582 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:49:50.974568 IP 172.26.0.50 > 172.26.0.49: ICMP echo request, id 26267, seq 8, length 64Ok so the ICMP from VYOS is being seen.
Let's do the same but pinging from VYOS and capture on the 3850 interface.
14:51:08.286935 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:08.286959 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:09.310991 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:09.311011 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:10.335411 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:10.335433 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:11.358975 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:11.358984 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:12.383035 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46
14:51:12.383045 ARP, Request who-has 172.26.0.49 tell 172.26.0.50, length 46I don't see the ICMP but for some reason, that looks like it is working?
Ok getting into VYOS I can run a tcpdump
vyos@vyos:~$ monitor traffic interface eth2
tcpdump: verbose output suppressed, use -v or --v for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytesok sweet the tcpdump is running let's see if we see anything when pinging 172.26.0.50 from the 3850.
#ping 172.26.0.50
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.0.50, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)Crap.
Nothing in VYOS showing packets are received and failing
Now let's keep running the packet capture on VYOS and capture on the pfsense. starting at the to_3850 and then to_vyos.
CRAP!!!
The to_3850 shows NOTHING
The to_VYOS shows NOTHING
What is going on here?¿?¿
-
First of all, you must examine your Layer2 Topology and ensure you won't create any Layer2 Loop with this Setup. If all Uplinks of the ESXi are connected to the same Layer2 Segment proceed with cautions. Spanning-Tree won't save you here, cause STP-BPDUs are filtered on the ESXi per default.
You need to allow Forged transmits and Promiscuous Mode on the used Port Groups for the bridged interfaces. This is a necessary configuration on the Virtualization Platform to make this setup work. After Changing these settings you have to either reboot the firewall or disconnect and reconnect the interfaces to allocated a new virtual port with the updated settings.
Better to configure MAC-Learning for these Port-Groups. This Option is available on Distributed Switch since version 6.6.0 through API Calls.
-
@artes
Thanks. I set promiscuous mode on the vlan's within ESXi.
It still isn't working.
So i stood up a new lab.
3 vlans in esxi
910
911
9124 vm's
v1 and v2 are windows 10 machines to simulate traffic over the bridge
w1 is the admin lan connection
And pf is in the middleFor 1 brief second, I get a good ping during a reboot of the pfsense server.
Now it makes me think that there is something filtering this traffic in pfsense but without anything in the logs and only arp requests found in a packet capture.
What could i look for to figure this out?
-
@juesor Found this very recent link from Broadcom I want to share, maybe it can help: https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/proxysg/6-7/Overview_ISG_SGW_VA/ISG_SWG_VA_before_you_begin/ISG_SWG_VA_create_a_virtual_switch.html
-
That's just outlining vSwitchs in ESXi.
I'm curious about the 1 good ping during the reboot. It's almost as if the firewall was down and came up and then started blocking the flow.
-
@juesor I shared because of this statement:
However, if you have one good ping, you'll need to find what's blocking ... note I said a firewall rule in my first response.
-
That doesn't explain that when I do "pfctl -d" pings still fail.
I wouldn't expect bridging the interfaces had anything to do with the firewall service.
And the fact that the FW log doesn't show any block's.
And the fact that packet capture only shows ARP from both sides but not ICMP received.
-
@juesor said in Bridging two interfaces WAN(vRouter) & LAN(LAN Router) w/ OPT1(MGMT):
And the fact that packet capture only shows ARP from both sides but not ICMP received.
I believe by default the firewall doesn't accept ICMP on WAN. So, one would need to add a rule Action : Pass , Interface : WAN , Protocol : ICMP , Source Type : Any, and Destination : WAN address. Although you're only using pfSense as a bridge, you still need to pass traffic from WAN to the bridge. So, that's why you'll have one good ping at the NIC but as soon as it reach the firewall, it gets shutdown.
-
-
@juesor said in Bridging two interfaces WAN(vRouter) & LAN(LAN Router) w/ OPT1(MGMT):
But there is no address on the wan interface so WAN address throws an !
Did all resources you have looked at as guide had WAN with an IP address? I'll visit the spiceworks and Lawrence sources again to confirm. Okay, from the spicework source note WAN has an IP.
Also notice from the Lawrence video, WAN has IP ... it's just the bridge (transparent) that doesn't have an IP.
-
Yea thats for bridging and using WAN as the gateway for traffic.
In my case i'm just using pf to filter traffic between two locations.
The whole transparent bridge i shouldn't need to IP anything in pfsense outside of my mgmt interface.
I moved the two VM's to a separate vlan 913 and they can ping each other.
Which shows connectivity. Now i'll move them back to the separate vlans 910 & 911 which are part of the bridge 910 = OPT1(vmx2) and 911 = WAN(vmx0)
-
Ok so when i moved it back you can see that 2 pings worked 2 failed.
PF shouldn't be this willy nilly about things.
-
Ok this is the kick in the pants.
If every vm is on the same host it works. i can ping across the bridge.
vmotion kept moving stuff around.
Now I don't know how this is going to work by moving back to vyos, pfsense, and my 3850 as the 3850 is physical but we will see once i wrap my head around this.
I was running on the cli of pfsense "tcpdump -i bridge0 host 192.168.1.1" and it started working like a Christmas miracle.
-
@juesor said in Bridging two interfaces WAN(vRouter) & LAN(LAN Router) w/ OPT1(MGMT):
I was running on the cli of pfsense "tcpdump -i bridge0 host 192.168.1.1" and it started working like a Christmas miracle.
Congratulations, your persistence paid off!
-
One down side is I downgraded to 2.4.4 and it worked i put this on my production setup and it's not working.
I confirmed all my settings were exactly the same as the 2.4.4.
There is no traffic on the bridge interface but the ICMP is found on the vmx0 and vmx2 interfaces.
