Pfsense with 3 NICS

Jamerson

There is NO bridge out of the box that is for SURE!! You must of created it - delete that!!

And your saying your clients are getting this via dhcp

192.168.6.11
255.255.255.0
192.168.6.1

Well so 6.1 is your gateway? But pfsense doesn't have an IP on 6.1 via your ifconfig – it has .0 as its address.. Which is the WIRE and not a valid host IP for your mask of /24

em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:8c:a6:3b
inet 192.168.6.0 netmask 0xffffff00 broadcast 192.168.6.255
inet6 fe80::250:56ff:fe8c:a63b%em2 prefixlen 64 scopeid 0x3
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast>

the 192.168.6.0 i fixed it,
the LAN 2 NIC was assigned with the wrong IP 192.168.6.0 but now it's 192.168.6.1
can you direct me how to create the bridg between the two LAN ?

em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:a8:1a:4a
        inet 192.168.2.3 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::250:56ff:fea8:1a4a%em0 prefixlen 64 scopeid 0x1
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:8c:7e:c8
        inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
        inet6 fe80::250:56ff:fe8c:7ec8%em1 prefixlen 64 scopeid 0x2
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:8c:a6:3b
        inet6 fe80::250:56ff:fe8c:a63b%em2 prefixlen 64 scopeid 0x3
        inet 192.168.6.1 netmask 0xffffff00 broadcast 192.168.6.255
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
pflog0: flags=100 <promisc>metric 0 mtu 33144
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
        syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 02:1b:ed:00:ce:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: em1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000
ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fea8:1a4a%ovpns1 prefixlen 64 scopeid 0xa
        inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 10306</performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

doktornotor

@Jamerson:

can you direct me how to create the bridg between the two LAN ?

Huh?! You already have one even though you should NOT have any as there's no reason for that!


bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 02:1b:ed:00:ce:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: em1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast>

Are you actually listening to what people are writing here? WHY do you want to create yet another bridge? :o

Jamerson

@doktornotor:

@Jamerson:

can you direct me how to create the bridg between the two LAN ?

Huh?! You already have one even though you should NOT have any as there's no reason for that!


bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 02:1b:ed:00:ce:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: em1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast>

Are you actually listening to what people are writing here? WHY do you want to create yet another bridge? :o

i already answer this will be for testing, thank you .

LAN1 is production and LAN2 is for testing

doktornotor

Well, dude… this goes nowhere. WHAT are you testing with a setup that is totally FUBARed and you refuse any advise to get it fixed? You already bridged 2 out of 3 LANs, so what else do you want to bridge and WHY? . Like, some real reason what you expect from that. No, "it's for testing" is NOT any reason that makes sense. You have already tested that your setup does not work.

johnpoz

And you would not "bridge" between "production" and "testing".. You might bridge between physical segment that you need to extend into different locations - using 1 address network block. You would not bridge 2 different networks segments. That is utter nonsense..

And as already pointed out you already have one - which you say you didn't create.. WTF dude?? delete it - or better yet. Just wipe this pfsense vm completely and start over with a download iso you boot your new vm to and install. You should have nice clean setup with what you want to do it like 5 minutes top.. Out of the box you have to do like 3 things - set your wan static, change your lan to your ip range you want to use vs the default 192.168.1.0/24 and then create your 2nd segment and put in the any any firewall rule on this lan2 segment. There your done!!

Then you can start testing - what I have no idea..

Jamerson

@johnpoz:

And you would not "bridge" between "production" and "testing".. You might bridge between physical segment that you need to extend into different locations - using 1 address network block. You would not bridge 2 different networks segments. That is utter nonsense..

And as already pointed out you already have one - which you say you didn't create.. WTF dude?? delete it - or better yet. Just wipe this pfsense vm completely and start over with a download iso you boot your new vm to and install. You should have nice clean setup with what you want to do it like 5 minutes top.. Out of the box you have to do like 3 things - set your wan static, change your lan to your ip range you want to use vs the default 192.168.1.0/24 and then create your 2nd segment and put in the any any firewall rule on this lan2 segment. There your done!!

Then you can start testing - what I have no idea..

John thank you so much for your continu support.
i will use a Physicall NIC to LAN 2.
really appreciate all your support guys.
and apologies if i caused any furstration !

johnpoz

Physical nic?? What?? Dude are you on medication or something?

stephenw10

Try to remain calm. ;D

We are somewhat going around in circles here.

Jamerson, what is your final goal here? Could you describe, perhaps with a diagram, what you are hoping to achieve.

Steve

johnpoz

Dude this guy is just loopy.. He is either off his medication or on some, or english is not his native language is only thing that explains this nonsense.

As to what he is trying to achieve, isn't it clear ;) "i am using all my 3 NICS to use Teaming so all traffic will go thought the WAN."

So either stoned or not a native speaker is my guess ;) I really want to help the guy - but its getting a bit ridiculous, maybe he is just trolling and having a good laugh?

podilarius

I am sure he doesn't know what bridging and teaming are. So it would seem that others don't either. It looks like he is wanting a routed solution. teaming in VMWare, Windows, and linux is akin to LAGG in pfSense. Bridging is completely different.
On LAN and LAN2, please unset the Gateways. Turn off blocking of private and bogons on LAN and LAN2. Not on WAN unless WAN network is private. Then if you changed from autoNAT, turn it back on unless you plan on some expert configuration.

Create an two aliases. One that has the LAN subnet in there 10.0.10.0/24 and another Alias that has LAN2 subnet in it (192.168.2.0/24). Then create an rule in LAN (or modify the existing one) that states Any proto, from LAN subnet, any port ::: to ::: !LAN2 Subnet (the alias you created) on any port ::: Allow.
Create a similar rule on LAN2 that allows any traffic NOT going to LAN to pass.
This will block LAN and LAN2 from communicating.

Hope that helps.

Jamerson

John,
yes English is not my First Language !
what i want is explained on the attached Diagrams
Vswitch 2 on the ESXI doesn't have a Physique NIC
my Win7 can't reach the internet when the Vswitch 2 doesn't have a Physical NIC, when i connect a Physical NIC to the switch the internet start working,

i used to use Vyatta for the laste 6 years, and it Always works fine like this way ( wihtout Physical NIC on the Vswitch ).
with Vyatta i've bridged the LAN 1 and LAN 2 to each others to go over the WAN and reach the internet wihtout needing a Physical NIC on the Vswitch.
i hope it clear to you what i want.

Vswitch.jpg_thumb

podilarius

Its clear to me. You just don't need to bridge unless you are going to run LAN and LAN2 in the same subnet. Routing will work just fine. You don't need as physical NIC in the vswitch. Everyone else is just saying that you have to have something in there that is part of pfsense and another machine. I have used this setup in a lab and it works just fine.

Any more clarification would be good.

doktornotor

@johnpoz:

Physical nic?? What??

Hopefully he's not going to try to physically hammer the physical NIC to the virtual machine. :o

@OP: Remove the bridges nonsense. We won't move anywhere while it's still there in place. Plus, bridging on FBSD does not exactly work like you'd think it does when it comes to firewall behaviour, at least not unless you've flipped a couple of system tunables and assigned the bridge interface itself instead of its members, e.g. like here:


bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 02:5f:58:aa:bb:00
        inet 10.20.31.254 netmask 0xffffff00 broadcast 10.20.31.255
        inet6 2001:470:xx:xx::254 prefixlen 64
        nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vr0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 1 priority 128 path cost 55
        member: vr2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 55
        member: vr1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast>

(the above being an Alix box which serves pretty much as a dumb WiFi AP plus hotspot with captive portal), other than that, no firewall, routing only, all the physical RJ45 ports being bridged on a WAN - which is attached to another pfSense box via a LAN interface - and basically acting just as a dumb switch.)

stephenw10

I think you screenshot from ESXi was cropped incorrectly, I can't see much of it. ;)

I agree with what others have said. You do not need a physical NIC to get connectivity. You do not need to bridge the two LANs to get connectivity, and in fact briding the LANs largely negates the point of having two separate LANs.

Steve

Jamerson

@doktornotor:

@johnpoz:

Physical nic?? What??

Hopefully he's not going to try to physically hammer the physical NIC to the virtual machine. :o

haha you are really funny guy !! :o

@stephenw10:

I think you screenshot from ESXi was cropped incorrectly, I can't see much of it. ;)

I agree with what others have said. You do not need a physical NIC to get connectivity. You do not need to bridge the two LANs to get connectivity, and in fact briding the LANs largely negates the point of having two separate LANs.

Steve

Steve,
I have 3 Virtual NIC on the PFSENSE,
one is on the group of Vswitch 1 which is the WAN of the ESXI
second one is the VSWITCH 2 which is LAN1
and 3rd one is on the Vswitch 3 which is LAN 2

I understand I don't need a Physical NIC to get the connectivity,
Why when I remove the Physical NIC from the VSwitch 2 /3 the connectivity drops down ?
attached is a screenshot of the Network Diagram on the ESXI.
that why I've attached a Physical NIC to LAN1 otherwise it won't work

on the Physique side , i will have a physical computers that will be a member of the domain controller that is running on the ESXI and need to have the access to the LAN 1 subnet over the WAN
Like Physique computer on the room will need to have access to 192.168.4.0/24 and need to use the PFSENSE as it gateway.

Thank you

LAN.jpg_thumb

johnpoz

You DO NOT need a bridge!! You DO NOT need a physical nic connected to your vswitch.

Remove the BRIDGE! And you will be good.

I run this exact setup. Multiple physical lans, and one that is only vms (dmz) I have no bridge in pfsense (you would not normally bridge 2 different address space segments).. My box in dmz can use the internet and is blocked from talking to lan or wlan because that is how I setup the firewall rules. But lan or wlan can talk to dmz (this is normally how a dmz is setup)

You have seen my esxi setup in previous - and you can see it in the background in the attached.

So my workstation that is on lan (via physical nic connection in esxi to that lan vswitch) on 192.168.1.100 can ping the dmz win7 box. See it attached on vswitch where pfsense also has interface (dmz 192.168.3.253) and w7 box is on 192.168.3.100/24 can ping google, but it can not ping the 192.168.1.100 machine because my firewall rules for dmz says you can go anywhere you want other than locals. That is what the ! in front of it means "not" locals. Locals is an alias in pfsense that includes 192.168.1.0/24 and 192.168.2.0/24 and my openvpn networks. So per my firewalls rules on my lan I can go to the dmz.. But dmz can only "create" connections to networks that are NOT my local networks. This common setup for a "dmz" segment.

edit: Seems I went over the attachment size for my dmz rules - see next post

confignophynic-vswitch.png_thumb

johnpoz

Here is my dmz firewall rules showing ! locals as dest

dmzfirewallrules.png_thumb

stephenw10

To be fair you do need a physical NIC on vSwitch1 if you have physical machines that need to connect to that subnet.

If however you are only getting general internet connectivity with that in place then it sounds like the VMs on vSwitch1 are using some external route and not the pfSense VM for there gateway.

Steve

johnpoz

From what I have been able to make out of this thread.. He only wants vms on this esxi host to connect through pfsense (vm) to the real network via pfsense wan connection. If that is the case then the only vswitch that needs connectivity to the physical world is vswitch that pfsense wan is connected too.

Think of the physical nic you connect to a vswitch as just normal uplink you use in real switches. If you have machines on switch1 and machines on switch2 how do you connect them.. You run a wire between the switches.

This is really all that connecting a physical nic in esxi to a vswitch does - it connects that vswitch to the real world switch the wire from that nic runs too.

Your vmkern portgroup would need a physical connection or you would not be able to manage the esxi host box. From your other drawings this is the same vswitch you have pfsense wan connected too. Your other lan and dmz segment vswitches only need physical connectivity if you as stated by stephen you have real world machines on those segments.

doktornotor

@Jamerson:

Why when I remove the Physical NIC from the VSwitch 2 /3 the connectivity drops down ?

P.S. Removed the BS bridge yet, or still feel like wasting more of our time with that nonsense?