Creating Static Routes for different subnets on the same physical interface
-
The gateway of the core switch is not your pfSense Lan but it's the default lan IP of the Juniper switch?
Either your diagram is wrong or someone configured a massive mess. Your call. -
Sorry…The core switch gateway is 192.168.1.1.
-
Would you mind hitting the "Modify" button on top at the right hand side of your post and correct your "diagram" then? Thank you.
…and what are the current netmasks and GWs of the Juniper switch VLANs? Might be we're lucky and that's not top secret so you don't have to shoot us after revealing. Maybe?
-
Updated my diagram. For the routed vlan interfaces (RVI) on the Juniper switch, you cannot specify a gateway.
-
There has to be an overall default gateway in the juniper. I assume that's 192.168.1.1?
And you didn't correct the gateway on the Quanta. That's also 192.168.1.1?
Create a gateway on pfSense under System > Routing. Put it on interface LAN. Call it JUNIPERL3 with an IP address of 192.168.1.2
Create static routes for 192.168.10.0/24 and 10.0.0.0/24 with JUNIPERL3 as the gateway.
Check Firewall > NAT, Outbound to be sure 192.168.10.0/24 and 10.0.0.0/24 are listed as sources (if automatic outbound NAT they should be). If not, add hybrid outbound NAT rules for them.
-
Sorry.. I modified the diagram again….
-
There has to be an overall default gateway in the juniper. I assume that's 192.168.1.1?
And you didn't correct the gateway on the Quanta. That's also 192.168.1.1?
Create a gateway on pfSense under System > Routing. Put it on interface LAN. Call it JUNIPERL3 with an IP address of 192.168.1.2
Create static routes for 192.168.10.0/24 and 10.0.0.0/24 with JUNIPERL3 as the gateway.
Check Firewall > NAT, Outbound to be sure 192.168.10.0/24 and 10.0.0.0/24 are listed as sources (if automatic outbound NAT they should be). If not, add hybrid outbound NAT rules for them.
In pfsense, you want me to create a LAN gateway with the IP address of 192.168.1.2. This is the same IP address of the default vlan on the Juniper switch (see below).
[Juniper Switch - EX3300]
Default vlan = 192.168.1.2, SubNetMask = 255.255.255.0, No GW
Vlan 10 = 192.168.10.1, SubNetMask = 255.255.255.0, No GW
Vlan 20 = 10.0.0.1, SubNetMask = 255.255.255.0, No GWAdditionally, on the pfsense setup, do I need to create firewall rules as well for 192.168.10.0/24 and 10.0.0.0/24?
-
yes if you have downstream router and you are going to allow traffic from these downstream networks into pfsense via a transit network then yes the rules on the transit interface need to allow those downstream networks ip ranges, or setup an alias with all of them or use a mask that allows for all the downstream networks, for example 192.168/16 would for sure cover it but you could use say 192.168.0/21 if you had 192.168.0 - 7 /24's downstream, etc.
-
yes if you have downstream router and you are going to allow traffic from these downstream networks into pfsense via a transit network then yes the rules on the transit interface need to allow those downstream networks ip ranges, or setup an alias with all of them or use a mask that allows for all the downstream networks, for example 192.168/16 would for sure cover it but you could use say 192.168.0/21 if you had 192.168.0 - 7 /24's downstream, etc.
Great.. I have created the rules…
-
Also I feel I have not configured an overall IP address and gateway for the Juniper switch. It looks like we are using the default vlan IP address (192.168.1.2) on the Juniper switch as the LAN gateway in pfsense.
Maybe that is my problem as well. I found the article below.
http://www.juniper.net/documentation/en_US/junos15.1/topics/example/switch-name-domain-name-ip-address-system-id-configuring.html
Also I am somewhat confused on the gateway to assign to the core switch (LB6M) and the host PCs. Should it be 192.168.1.1 or 192.168.1.2 (see diagram below)?
Internet
|
|
|
[pfsense]
WAN = 192.168.50.x
LAN = 192.168.1.1
|
|
|
[Core Switch - Quanta LB6M]
IP = 192.168.1.5
GW = 192.168.1.2 [Should the Gateway be 192.168.1.1?]
SubNetMask = 255.255.255.0
|
|
|
[Juniper Switch - EX3300]
Default vlan = 192.168.1.2, SubNetMask = 255.255.255.0, No GW
Vlan 10 = 192.168.10.1, SubNetMask = 255.255.255.0, No GW
Vlan 20 = 10.0.0.1, SubNetMask = 255.255.255.0, No GW
|
|
|
[Host PC A]
IP = 192.168.1.207
GW = 192.168.1.2 [Should the Gateway be 192.168.1.1?]
SubNetMask = 255.255.255.0 -
When you say core switch.. Do you have more vlans off this switch?? Or devices on this 192.168.1/24 network - if so then that is not a transit network..
you clearly have hosts on this network… 192.168.1/24 that is NOT a transit network... Your going to have all kinds of problems..
A downstream router needs to be on transit network or your going to have asymmetrical routing issues.
So you can have multiple networks/vlans hanging off pfsense.. But when you connect to downstream it should be via a transit network, ie no other hosts on this network.. You can have more routers all connected together via a transit, etc. But you do not put hosts on a transit network. Or talking to these hosts is going to have issues with asymmetrical routing unless you create routes on all these hosts on which gateway/router to use to get to specific networks.
See attached. The transit in this drawing it the 172.16.0/30 lets say pfsense is .1 and router is .2 on that transit. The routers default gateway would be 172.16.0.1 ie pfsense. The other interfaces on the router would not have gw set. Devices on those vlans would point to the routers IP in those vlans ie 10.0.1.1, 10.0.2.1 and 10.0.3.1 as their gw.
Devices on the 192.168.0/24 would point to pfsense IP in that interface as their gw 192.168.0.1 lets call it. Again there are NO hosts on the 172.16.0/30 it is a transit network!!!
So see attachment 2 with multiple downstream routers. So now we needed to expand our transit to allow for more IP so lets make it a /29 now pfsense would be .1 router would be .2 and router2 will be .3 in this transit network.
So router 1 default route would be to pfsense at .1 on the transit. But he would need route to 10.0.4 and 10.0.5/24 pointing to router 2 at 172.168.0.3
Router 2 would need default pointing to pfsense and routes for 10.0.1, .2 and .3 pointing to router 1 at 172.16.0.2
Pfsense would need routes saying hey to get to 10.0.1,2 and .3 talk to router 1 at 172.168.0.2 and routes for 10.0.4 and .5 talk to router 2 at 172.168.0.3
-
When you say core switch.. Do you have more vlans off this switch?? Or devices on this 192.168.1/24 network - if so then that is not a transit network..
you clearly have hosts on this network… 192.168.1/24 that is NOT a transit network... Your going to have all kinds of problems..
A downstream router needs to be on transit network or your going to have asymmetrical routing issues.
So you can have multiple networks/vlans hanging off pfsense.. But when you connect to downstream it should be via a transit network, ie no other hosts on this network.. You can have more routers all connected together via a transit, etc. But you do not put hosts on a transit network. Or talking to these hosts is going to have issues with asymmetrical routing unless you create routes on all these hosts on which gateway/router to use to get to specific networks.
See attached. The transit in this drawing it the 172.16.0/30 lets say pfsense is .1 and router is .2 on that transit. The routers default gateway would be 172.16.0.1 ie pfsense. The other interfaces on the router would not have gw set. Devices on those vlans would point to the routers IP in those vlans ie 10.0.1.1, 10.0.2.1 and 10.0.3.1 as their gw.
Devices on the 192.168.0/24 would point to pfsense IP in that interface as their gw 192.168.0.1 lets call it. Again there are NO hosts on the 172.16.0/30 it is a transit network!!!
So see attachment 2 with multiple downstream routers. So now we needed to expand our transit to allow for more IP so lets make it a /29 now pfsense would be .1 router would be .2 and router2 will be .3 in this transit network.
So router 1 default route would be to pfsense at .1 on the transit. But he would need route to 10.0.4 and 10.0.5/24 pointing to router 2 at 172.168.0.3
Router 2 would need default pointing to pfsense and routes for 10.0.1, .2 and .3 pointing to router 1 at 172.16.0.2
Pfsense would need routes saying hey to get to 10.0.1,2 and .3 talk to router 1 at 172.168.0.2 and routes for 10.0.4 and .5 talk to router 2 at 172.168.0.3On the core switch, I have vlan 1 (default), vlan 10, and vlan 20. That is all the vlans that will be configured on the core switch. The default vlan on the core switch will not be used. For vlan 10 on the core switch, I have connected my hypervisor server and other physical servers. For vlan 20 on the core switch, I have connected my NAS/SAN Storage server.
-
And now you have a problem don't you..
So you have attached.. Is this how you have it setup? So gateway of say your server on vlan 10 is your svi for vlan 10 on your downstream router (juniper)?
So PC wants to talk to your server he talks to pfsense.. Who then sends back to juniper who routes it to vlan 10, then sends it back across the same line again through your core switch. Then where does server send traffic back.. To the juniper which is the gateway for vlan 10, who says oh you want to talk to vlan 1 sure thats right here and now you have asymmetrical problem.. Let alone lots of hairpins, etc.
So lets say you make the pc gateway for this vlan 1 the juniper IP in this vlan 192.168.1.2 so that gets rid of your mess talking to other vlans
Now you have this mess when pc wants to talk to internet. So he bounces off juniper since that is his gateway, juniper says oh my default gw is pfsense at .1 on it goes. Return traffic pfsense says oh you want to talk to IP in 192.168.1 yeah I am directly connected and sends direct to the pc machine in 192.168.1.207
So again lets be clear if your going to have downstream routers you need to use a transit network.
-
This is exactly my setup. The downstream router is my Juniper EX3300 switch that provides all inter-vlan routing.
I will do some research on creating a transit network.
Thank you do much on all the expert advice.
-
not a problem. As your network grows/expands beyond 1 flat network lots of things start coming into play that need to be taken into account.
For starters as you start adding downstream or daisy chained switches you need to worry about bottlenecks and or hairpins, if you add downstream routers asymmetrical routing comes into play as well. As you start to grow more away from just a core switch/router all at 1 spot do you need a distribution layer for your switches or just closet/access layer.
As it grows and you start to do failover or load balancing for your uplinks between your switches spanning tree and or loops become a possible issue, etc. etc. etc..
Having what your calling your core switch between your edge and an internal router and placements of devices on different vlans location and where most of your traffic flows needs to be taken into account when you do your layout so you don't have bottlenecks or multiple hairpins and asymmetrical routing..
For example might be better even with a transit network to put what your calling your core below your downstream.. Why do you have your nas on different vlan than your servers? Do your servers not access the storage and only users? Maybe it would be better to put your servers and nas all on same vlan so your not routing between them? And best to maybe be on the same switch so your not having to go through an uplink?
Would need to know your physical location of your servers/infrastructure type devices and where your users sit and where any closet switches are, etc. And what the major data flows are to best layout the network and vlans, etc. And then what security you would want/need between your segments. Is that juniper layer 4+? Can it do ACL's to filter/block traffic you don't want between your vlans? Or does it just route? I would assume you can do acl's there and filter traffic as you need too, etc.
As the network grows is when it all gets fun! ;)
-
not a problem. As your network grows/expands beyond 1 flat network lots of things start coming into play that need to be taken into account.
For starters as you start adding downstream or daisy chained switches you need to worry about bottlenecks and or hairpins, if you add downstream routers asymmetrical routing comes into play as well. As you start to grow more away from just a core switch/router all at 1 spot do you need a distribution layer for your switches or just closet/access layer.
As it grows and you start to do failover or load balancing for your uplinks between your switches spanning tree and or loops become a possible issue, etc. etc. etc..
Having what your calling your core switch between your edge and an internal router and placements of devices on different vlans location and where most of your traffic flows needs to be taken into account when you do your layout so you don't have bottlenecks or multiple hairpins and asymmetrical routing..
For example might be better even with a transit network to put what your calling your core below your downstream.. Why do you have your nas on different vlan than your servers? Do your servers not access the storage and only users? Maybe it would be better to put your servers and nas all on same vlan so your not routing between them? And best to maybe be on the same switch so your not having to go through an uplink?
Would need to know your physical location of your servers/infrastructure type devices and where your users sit and where any closet switches are, etc. And what the major data flows are to best layout the network and vlans, etc. And then what security you would want/need between your segments. Is that juniper layer 4+? Can it do ACL's to filter/block traffic you don't want between your vlans? Or does it just route? I would assume you can do acl's there and filter traffic as you need too, etc.
As the network grows is when it all gets fun! ;)
Hopefully the attached drawing provide more details on my current setup and the proposed setup using a transit vlan. Please don't laugh at my drawing. I still need to figure out the details on implementing the proposed setup using a transit vlan.
One change I will make in the drawing is to use a /29 subnet mask just in case in the future I have more downstream routers.
Thanks for all the help…
-
Here is a new version of the proposed setup using a transit vlan 2000.
-
Why are trunking the connection to pfsense? It would only ever see the transit vlan, and that doesn't have to be tagged even, etc.
So your physical connection you have a hairpin for when devices on your core want to talk to the internet. So they go down the trunk to get to the gw on the l3, then they have to come back the same trunk port go through their switch again and then to pfsense.
If you can directly connect your L3 then you don't have this problem.. No device on either switch when talking to the internet need to hairpin. While you do have to hairpin if talking to different vlan on same downstream switch. That is hard to get rid of which is why you try and not put devices on a downstream switch on different vlans if they need to talk to each other, etc. ;)
So your running 10Ge isnt the LB6M a generic 10ge sfp switch.. Doesn't it do layer 3 as well? I have to assume your uplink between for sure is 10ge. If so you just make your quanta the L3 and turn your juniper into just L2 and you don't even have to move any wires.
-
Why are trunking the connection to pfsense? It would only ever see the transit vlan, and that doesn't have to be tagged even, etc.
So your physical connection you have a hairpin for when devices on your core want to talk to the internet. So they go down the trunk to get to the gw on the l3, then they have to come back the same trunk port go through their switch again and then to pfsense.
If you can directly connect your L3 then you don't have this problem.. No device on either switch when talking to the internet need to hairpin. While you do have to hairpin if talking to different vlan on same downstream switch. That is hard to get rid of which is why you try and not put devices on a downstream switch on different vlans if they need to talk to each other, etc. ;)
So your running 10Ge isnt the LB6M a generic 10ge sfp switch.. Doesn't it do layer 3 as well? I have to assume your uplink between for sure is 10ge. If so you just make your quanta the L3 and turn your juniper into just L2 and you don't even have to move any wires.
The connection from the core switch (LB6M) to pfSense is a LAGG/LACP connection using port 25 & 26 for failover and load balancing.
The LB6M has twenty four 10ge SFP+ ports. The Layer 3 capability on the switch is very flaky. Not reliable. The uplink to the Juniper switch is a 10ge LAGG/LACP connection.
Here is a more detailed view of the current setup I was trying to implement .. I have not implemented everything in the diagram yet.
-
well that is much more detailed drawing for sure ;) hehehe
If your running 10ge uplinks is not going to really matter for sure.. Your internet not going to be anywhere close to that so I wouldn't worry about it. But you do have a hairpin that could be avoided. Currently when any device on the quanta which is only in layer 2 mode and not routing wants to go to the internet it has to transverse the uplink to the juniper doing the routing get routed and then back through the same uplink to get to the quanta again and then on to the pfsense to go to the internet. Now maybe these boxes rarely talk to the internet, or maybe they pull down 100's and 100's of GB I don't know.. Its just best to avoid such hairpins no matter if your working 10mb or 40Ge etc.. as your pipe..
So this is your home network?? You bastard!!! ;) heheeheh Let me guess no wife that complains that you spend to much on your "toys" hehehehe
