Pfsense with 3 NICS

johnpoz

Why would you have a block rfc1918 rule on your LAN interface? You do understand your lan2 would be private, and pfsense own IPs on that lan would be rfc1918 as well

LAN 1 is 192.168.4.0/24
LAN2 is 192.168.6.0/24

And how is anything suppose to get to lan2 when there is nothing in your vswitch for lan 2 - what interface in pfsense is connected to lan2?

stephenw10

Yep that's your problem.
Go to Interfaces: LAN1: and uncheck 'block private networks' and 'block bogon networks'.
Bogons may not be necessary but there's no need to block them on LAN.

Steve

bryan.paradis

@johnpoz:

Why would you have a block rfc1918 rule on your LAN interface? You do understand your lan2 would be private, and pfsense own IPs on that lan would be rfc1918 as well

LAN 1 is 192.168.4.0/24
LAN2 is 192.168.6.0/24

And how is anything suppose to get to lan2 when there is nothing in your vswitch for lan 2 - what interface in pfsense is connected to lan2?

@stephenw10:

Yep that's your problem.
Go to Interfaces: LAN1: and uncheck 'block private networks' and 'block bogon networks'.
Bogons may not be necessary but there's no need to block them on LAN.

Steve

What is the default on those when configuring a new interface ?

johnpoz

OFF, it only defaults to be on WAN interface.

bryan.paradis

@johnpoz:

OFF, it only defaults to be on WAN interface.

This is what I had thought so I assume he must have changed options or maybe reversed his iunterfaces at some point and had to reassign?

Jamerson

Why would you have a block rfc1918 rule on your LAN interface? You do understand your lan2 would be private, and pfsense own IPs on that lan would be rfc1918 as well

LAN 1 is 192.168.4.0/24
LAN2 is 192.168.6.0/24

And how is anything suppose to get to lan2 when there is nothing in your vswitch for lan 2 - what interface in pfsense is connected to lan2?

hi Bryan,
i am using all my 3 NICS to use Teaming so all traffic will go thought the WAN.

 Yep that's your problem.
Go to Interfaces: LAN1: and uncheck 'block private networks' and 'block bogon networks'.
Bogons may not be necessary but there's no need to block them on LAN.

Steve

those rules were added automatically when i created the rules.
you mean i need to uncheck all private Network between the LANS ?

is it possible to have all traffic go out and in thought the WAN wihtout assign a physique NIC on the Vswitch ?

creating a Vlan with one Physique NIC could fix the issue ?

stephenw10

Uncheck the two boxes on the interface setup page.
It doesn't matter that all the traffic is going to the WAN. The source is the LAN subnet which is a private network. If you look at your firewall rules for that interface you'll see that the 'block private networks' rules is at the top of the list so it matches the incoming packets first and the 'allow all rules' are never reached. The 'block private networks' check box is only there for WAN interfaces where you would not expect traffic arriving with an RFC1918 source address unless it's been spoofed.

Steve

johnpoz

"those rules were added automatically when i created the rules."

No I highly doubt that, nnless you created a WAN interface, by putting a gateway on it?

"i am using all my 3 NICS to use Teaming so all traffic will go thought the WAN."

What – no you are not.. I suggest you look up the term "teaming" as it has to do with network interfaces - you are not doing that ;)

"is it possible to have all traffic go out and in thought the WAN wihtout assign a physique NIC on the Vswitch ?"

Sure - if your Other vm's have an interface in that vswitch - see example below on my dmz vswitch. There is an interface connected to my pfsense VM, and then the other vms that need to use this interface on pfsense are tied to that same vswitch. It does not need to be tied to the physical network. But it can access the physical network through pfsense that has its other lan and wlan in my case tied to physical networks. See attachment 2

nophysical.png_thumb

Jamerson

@johnpoz:

"those rules were added automatically when i created the rules."

No I highly doubt that, nnless you created a WAN interface, by putting a gateway on it?
"i am using all my 3 NICS to use Teaming so all traffic will go thought the WAN."

What -- no you are not.. I suggest you look up the term "teaming" as it has to do with network interfaces - you are not doing that ;)
this where i want to go to, use all my NIC for teaming

"is it possible to have all traffic go out and in thought the WAN wihtout assign a physique NIC on the Vswitch ?"

Sure - if your Other vm's have an interface in that vswitch - see example below on my dmz vswitch. There is an interface connected to my pfsense VM, and then the other vms that need to use this interface on pfsense are tied to that same vswitch. It does not need to be tied to the physical network. But it can access the physical network through pfsense that has its other lan and wlan in my case tied to physical networks. See attachment 2

the machine on your DMZ can reach the internet even there is no Physique interface?
this exactly what i am looking for.

johnpoz

Yes why would they not.. They are connected to a switch that is in connected to pfsense interface (be it all virtual), which has physical connection to the real network.

That is not TEAMING.. where did you come up that you think you want to do teaming?

Teaming in reference to network, call it link aggregation, port trunking, nic teaming, bonding has nothing to do with this.

But notice in your drawing - there is NOTHING connected to your switch, no vm's no interface in pfsense that your calling lan2, but where is that connected to a vswitch?

stephenw10

Are you saying that you would like to able to this without any physical NICs (which you can) because you want to use the NICs you have in an aggregated connection? You have setup you NICs as they are currently because you didn't realise you could do it all virtually without any NICs?

Steve

bryan.paradis

i am using all my 3 NICS to use Teaming so all traffic will go thought the WAN.

Guess it could be a lot of things. Jamerson you need to really clarify and perhaps draw a diagram of what you want.

johnpoz

Well clearly his NOT doing that, since he has a nic in his lan network and nic in his wan network - so how could he be "teaming" them. He clearly doesn't understand the term.

For that matter he doesn't even have 3 nics, be it physical or VM.. Look at his esxi host networking he posted

He doesn't even have a lan2 interface listed in his VM networking for pfsense, which I take is the firewall vm.. Yet he says he created firewall rules in pfsense for lan? So what did he connect to lan2? Mystery to be sure.

Jamerson

sorry guys for my late reply ( been sick ).
about the NIC teaming haven't configured it yet on the LAB and I know what is NIC teaming by the way, that why you don't see it on the Diagram.
what I need is when VM on the same LAN2 on ESXI that they need to reach the internet.
My PFSENSE has 3 NICS,
WAN and LAN1 and LAN2
LAN 1 and LAN 2 are not attached to Physical NIC,
attached is screenshot of the LAB
I didn't add the NIC yet of the PFSENSE to each group LAN 1 and LAN 2.
on LAN 1 will have PFSENSE NIC 2
on LAN 2 will have PFSENSE NIC 3
so when I move a VM to the LAN 1 0r LAN 2 it doesn't reach the internet. this is my problem.
when I attached a physical NIC to LAN 1 or LAN 2 on the ESXI it works

LAN1.jpg_thumb

johnpoz

Dude how is anything going to use pfsense if pfsense has no VIRTUAL nic in the switch??

How is any VM going to connect to pfsense lan1 or lan2 if there is NOTHING connected to the vswitch? Does not matter if the vswitch is not connected to the physical world or not..

See bottom of my virtual network from esxi

See how pfsense has a virtual interface connected into the vswitch, so that other virtual machines with a virtual interface connected to this vswitch can connect to pfsense and then access the real world (internet or lan or wlan) via the physical nics connected to those vswitches.

You have nothing connected to your lan1 or lan2 from your drawing - no pfsense interfaces, no vms – so NO how would anything use it.. What does pfsense show for its interfaces, since from your drawing you don't show any lan1 or lan2 interfaces for pfsense at all.

So for example


[2.1.1-PRERELEASE][admin@pfsense.local.lan]/root(1): ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:02
        inet 192.168.1.253 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::250:56ff:fe00:2%em0 prefixlen 64 scopeid 0x1
        inet6 2001:470:1f11:9c4::1 prefixlen 64
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:01
        inet 24.13.xx.xx netmask 0xfffff800 broadcast 255.255.255.255
        inet6 fe80::250:56ff:fe00:1%em1 prefixlen 64 scopeid 0x2
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:1e:18:90
        inet 192.168.2.253 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::20c:29ff:fe1e:1890%em2 prefixlen 64 scopeid 0x3
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:1e:18:9a
        inet 192.168.3.253 netmask 0xffffff00 broadcast 192.168.3.255
        inet6 fe80::20c:29ff:fe1e:189a%em3 prefixlen 64 scopeid 0x4
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active</full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

Those are the interfaces in my virtual pfsense.. Se how pfsense has 4 interfaces. One in each segment (wan, lan, wlan, dmz) notice how in my esxi network pfsense is listed as having a connection in each vswitch that is connected to each one of pfsense interfaces. If I connect a virtual machine to that vswitch, via a virtual interface. Like the W7 machine on the bottom above pfsense on the dmz vswitch.. It can talk to pfsense over the vswitch on its dmz virtual interface and then pfsense can route it to any of its other interfaces over virtual - which may or may not have access into the physical world via a physical adapter connected to that vswitch.

pfsensemissing.png_thumb

pfsenseconnected.png_thumb

Jamerson

@johnpoz:

Dude how is anything going to use pfsense if pfsense has no VIRTUAL nic in the switch??

How is any VM going to connect to pfsense lan1 or lan2 if there is NOTHING connected to the vswitch? Does not matter if the vswitch is not connected to the physical world or not..

See bottom of my virtual network from esxi

See how pfsense has a virtual interface connected into the vswitch, so that other virtual machines with a virtual interface connected to this vswitch can connect to pfsense and then access the real world (internet or lan or wlan) via the physical nics connected to those vswitches.

You have nothing connected to your lan1 or lan2 from your drawing - no pfsense interfaces, no vms – so NO how would anything use it.. What does pfsense show for its interfaces, since from your drawing you don't show any lan1 or lan2 interfaces for pfsense at all.

So for example
[2.1.1-PRERELEASE][admin@pfsense.local.lan]/root(1): ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:02
        inet 192.168.1.253 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::250:56ff:fe00:2%em0 prefixlen 64 scopeid 0x1
        inet6 2001:470:1f11:9c4::1 prefixlen 64
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:01
        inet 24.13.xx.xx netmask 0xfffff800 broadcast 255.255.255.255
        inet6 fe80::250:56ff:fe00:1%em1 prefixlen 64 scopeid 0x2
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:1e:18:90
        inet 192.168.2.253 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::20c:29ff:fe1e:1890%em2 prefixlen 64 scopeid 0x3
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=98 <vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:1e:18:9a
        inet 192.168.3.253 netmask 0xffffff00 broadcast 192.168.3.255
        inet6 fe80::20c:29ff:fe1e:189a%em3 prefixlen 64 scopeid 0x4
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active</full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast> 
Those are the interfaces in my virtual pfsense.. Se how pfsense has 4 interfaces. One in each segment (wan, lan, wlan, dmz) notice how in my esxi network pfsense is listed as having a connection in each vswitch that is connected to each one of pfsense interfaces. If I connect a virtual machine to that vswitch, via a virtual interface. Like the W7 machine on the bottom above pfsense on the dmz vswitch.. It can talk to pfsense over the vswitch on its dmz virtual interface and then pfsense can route it to any of its other interfaces over virtual - which may or may not have access into the physical world via a physical adapter connected to that vswitch.

hi John,
Let me thank you for your continue help really appreciate it.
apologies i've uploaded the wrong screenshot !
the attached is the correct screenshot

when a machine using LAN1 NIC as Gateway is not accessible the internet !

em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:a8:1a:4a
        inet 192.168.2.3 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::250:56ff:fea8:1a4a%em0 prefixlen 64 scopeid 0x1
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:8c:7e:c8
        inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
        inet6 fe80::250:56ff:fe8c:7ec8%em1 prefixlen 64 scopeid 0x2
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:8c:a6:3b
        inet 192.168.6.0 netmask 0xffffff00 broadcast 192.168.6.255
        inet6 fe80::250:56ff:fe8c:a63b%em2 prefixlen 64 scopeid 0x3
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
pflog0: flags=100 <promisc>metric 0 mtu 33144
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
        syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        ether 02:1b:ed:00:ce:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: em1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000
ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fea8:1a4a%ovpns1 prefixlen 64 scopeid 0xa
        inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 10306</performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

LAN1.jpg_thumb

johnpoz

Well for one this address is wrong

inet 192.168.6.0 that you have on em2

That is a network (wire address) with that mask netmask 0xffffff00 (/24 or 255.255.255.0) not a host or interface address. I would think it should be 192.168.6.1 - so fix that for starters.

Now on lan2, connect some Vms. Did you post the wrong drawing again - there are no other vms connected to vswitch2 or lan 2 in your drawing - so how is anything expected to connect to pfsense lan2 interface. Same with lan1 - there are NO other vms there either. Did you not understand my last post?

Once you have a vm actually connected to lan2 vswitch, and give pfsense an actual valid host IP that you want to use for the network you put it on.. Can the clients connected to that vswitch on the same network as pfsense interface ping it? What are the firewall rules for that interface. While the first lan you create when pfsense is installed will have a any any rule on it and allow outbound traffic to anywhere. When you add additional interfaces there will be NO rules, and you have to allow the traffic you want out on this new network segment. So what are you rules on lan2?

stephenw10

You have bridge0 containing both em1 and em2. They both have static IP addresses in different subnets. Are they both running dhcp? (Is that possible?). You have bridged the two subnets so that they both exist across both network segments. I'm fairly sure you don't want to do that.

Steve

doktornotor

I cannot see any pfSense anywhere on the networking diagrams you posted. Really dunno what you are trying to do there. When nothing's connected, nothing will work. Other than that - flush the mess you have created down the drain, and produce a barebones basic setup with 3 NICs. No "teaming", "bridging" and similar nonsense.

johnpoz

Good catch.. Didn't even get that far in his listing

bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 02:1b:ed:00:ce:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 20000
member: em1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000

I just don't get it.. You really just click click and pfsense works out of the box – its like these users go out of their way to have issues.. Why would he have created a bridge?? When clearly he mentions multiple segments. And he seems to have a hard time posting actual info that is what his network is.. How do you not notice that what you posts shows NOTHING connected to your vswitches? ;)</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast>