Creating Static Routes for different subnets on the same physical interface

pglover19 · Aug 24, 2016, 11:19 AM

Hello All,

I have been working on a Juniper switch issue with the Juniper technical support team and they have advised me that I need to create some static routes in pfsense in order to solve a routing problem I am having.

The recommendation to solve my problem is to create the following static routes in pfsense:

10.0.0.0/24 next-hop 192.168.1.2

192.168.10.0/24 next-hop 192.168.1.2

Can someone please provide me detail instructions on how to setup the static routes. If you need more information, I can provide the details.

Here of some examples of what I am trying to accomplish.

http://serverfault.com/questions/635253/pfsense-routing-a-packet-out-the-same-interface-it-arrived-on

https://forum.pfsense.org/index.php?topic=11973.0

johnpoz · Aug 24, 2016, 11:59 AM

So you have a downstream router? Or layer3 switch doing routing which is a router ;)

Please draw your network.. More than happy show you how to add a route to pfsense, my concern is even adding the routes will just cause an asymmetrical routing issue. You can not have hosts on a transit network between 2 routers. If you do connectivity to those hosts will be asymmetrical unless you do routing on the hosts that sit on the transit.

https://doc.pfsense.org/index.php/Static_Routes

Again please draw your network you do not want asymmetrical routing

pglover19 · Aug 24, 2016, 2:21 PM

Here is a description of my network:

Juniper EX3300 switch -> Core switch ->pfsense PC -> cable modem

The juniper switch is a layer 3 switch performing inter-vlan routing.

I currently have 3 vlans with the following subnets:

Default vlan gateway = 192.168.1.2
Vlan 10 gateway = 192.168.10.1
Vlan 20 gateway = 10.0.0.1

Please let me know if you need more information.

jahonix · Aug 24, 2016, 5:50 PM

Throwing in 3 IPs (netmask ??) isn't what I'd call "drawing a network".
Honestly, if you want help then provide infos.

pglover19 · Aug 24, 2016, 6:12 PM

Here is a simple diagram:

Internet
|
|
|
[pfsense]
WAN = 192.168.50.x
LAN = 192.168.1.1
|
|
|
[Core Switch - Quanta LB6M]
IP = 192.168.1.5
GW = 192.168.1.1
SubNetMask = 255.255.255.0
|
|
|
[Juniper Switch - EX3300]
Default vlan = 192.168.1.2, SubNetMask = 255.255.255.0, No GW
Vlan 10 = 192.168.10.1, SubNetMask = 255.255.255.0, No GW
Vlan 20 = 10.0.0.1, SubNetMask = 255.255.255.0, No GW

jahonix · Aug 24, 2016, 8:06 PM

The gateway of the core switch is not your pfSense Lan but it's the default lan IP of the Juniper switch?
Either your diagram is wrong or someone configured a massive mess. Your call.

pglover19 · Aug 24, 2016, 8:19 PM

Sorry…The core switch gateway is 192.168.1.1.

jahonix · Aug 24, 2016, 8:32 PM

Would you mind hitting the "Modify" button on top at the right hand side of your post and correct your "diagram" then? Thank you.

…and what are the current netmasks and GWs of the Juniper switch VLANs? Might be we're lucky and that's not top secret so you don't have to shoot us after revealing. Maybe?

pglover19 · Aug 24, 2016, 11:10 PM

Updated my diagram. For the routed vlan interfaces (RVI) on the Juniper switch, you cannot specify a gateway.

Derelict · Aug 25, 2016, 3:49 AM

There has to be an overall default gateway in the juniper. I assume that's 192.168.1.1?

And you didn't correct the gateway on the Quanta. That's also 192.168.1.1?

Create a gateway on pfSense under System > Routing. Put it on interface LAN. Call it JUNIPERL3 with an IP address of 192.168.1.2

Create static routes for 192.168.10.0/24 and 10.0.0.0/24 with JUNIPERL3 as the gateway.

Check Firewall > NAT, Outbound to be sure 192.168.10.0/24 and 10.0.0.0/24 are listed as sources (if automatic outbound NAT they should be). If not, add hybrid outbound NAT rules for them.

pglover19 · Aug 25, 2016, 10:04 AM

Sorry.. I modified the diagram again….

pglover19 · Aug 25, 2016, 10:41 AM

@Derelict:

There has to be an overall default gateway in the juniper. I assume that's 192.168.1.1?

And you didn't correct the gateway on the Quanta. That's also 192.168.1.1?

Create a gateway on pfSense under System > Routing. Put it on interface LAN. Call it JUNIPERL3 with an IP address of 192.168.1.2

Create static routes for 192.168.10.0/24 and 10.0.0.0/24 with JUNIPERL3 as the gateway.

Check Firewall > NAT, Outbound to be sure 192.168.10.0/24 and 10.0.0.0/24 are listed as sources (if automatic outbound NAT they should be). If not, add hybrid outbound NAT rules for them.

In pfsense, you want me to create a LAN gateway with the IP address of 192.168.1.2. This is the same IP address of the default vlan on the Juniper switch (see below).

[Juniper Switch - EX3300]
Default vlan = 192.168.1.2, SubNetMask = 255.255.255.0, No GW
Vlan 10 = 192.168.10.1, SubNetMask = 255.255.255.0, No GW
Vlan 20 = 10.0.0.1, SubNetMask = 255.255.255.0, No GW

Additionally, on the pfsense setup, do I need to create firewall rules as well for 192.168.10.0/24 and 10.0.0.0/24?

johnpoz · Aug 25, 2016, 10:54 AM

yes if you have downstream router and you are going to allow traffic from these downstream networks into pfsense via a transit network then yes the rules on the transit interface need to allow those downstream networks ip ranges, or setup an alias with all of them or use a mask that allows for all the downstream networks, for example 192.168/16 would for sure cover it but you could use say 192.168.0/21 if you had 192.168.0 - 7 /24's downstream, etc.

pglover19 · Aug 25, 2016, 11:08 AM

@johnpoz:

yes if you have downstream router and you are going to allow traffic from these downstream networks into pfsense via a transit network then yes the rules on the transit interface need to allow those downstream networks ip ranges, or setup an alias with all of them or use a mask that allows for all the downstream networks, for example 192.168/16 would for sure cover it but you could use say 192.168.0/21 if you had 192.168.0 - 7 /24's downstream, etc.

Great.. I have created the rules…

pglover19 · Aug 25, 2016, 11:14 AM

Also I feel I have not configured an overall IP address and gateway for the Juniper switch. It looks like we are using the default vlan IP address (192.168.1.2) on the Juniper switch as the LAN gateway in pfsense.

Maybe that is my problem as well. I found the article below.

http://www.juniper.net/documentation/en_US/junos15.1/topics/example/switch-name-domain-name-ip-address-system-id-configuring.html

Also I am somewhat confused on the gateway to assign to the core switch (LB6M) and the host PCs. Should it be 192.168.1.1 or 192.168.1.2 (see diagram below)?

Internet
|
|
|
[pfsense]
WAN = 192.168.50.x
LAN = 192.168.1.1
|
|
|
[Core Switch - Quanta LB6M]
IP = 192.168.1.5
GW = 192.168.1.2 [Should the Gateway be 192.168.1.1?]
SubNetMask = 255.255.255.0
|
|
|
[Juniper Switch - EX3300]
Default vlan = 192.168.1.2, SubNetMask = 255.255.255.0, No GW
Vlan 10 = 192.168.10.1, SubNetMask = 255.255.255.0, No GW
Vlan 20 = 10.0.0.1, SubNetMask = 255.255.255.0, No GW
|
|
|
[Host PC A]
IP = 192.168.1.207
GW = 192.168.1.2 [Should the Gateway be 192.168.1.1?]
SubNetMask = 255.255.255.0

johnpoz · Aug 25, 2016, 11:34 AM

When you say core switch.. Do you have more vlans off this switch?? Or devices on this 192.168.1/24 network - if so then that is not a transit network..

you clearly have hosts on this network… 192.168.1/24 that is NOT a transit network... Your going to have all kinds of problems..

A downstream router needs to be on transit network or your going to have asymmetrical routing issues.

So you can have multiple networks/vlans hanging off pfsense.. But when you connect to downstream it should be via a transit network, ie no other hosts on this network.. You can have more routers all connected together via a transit, etc. But you do not put hosts on a transit network. Or talking to these hosts is going to have issues with asymmetrical routing unless you create routes on all these hosts on which gateway/router to use to get to specific networks.

See attached. The transit in this drawing it the 172.16.0/30 lets say pfsense is .1 and router is .2 on that transit. The routers default gateway would be 172.16.0.1 ie pfsense. The other interfaces on the router would not have gw set. Devices on those vlans would point to the routers IP in those vlans ie 10.0.1.1, 10.0.2.1 and 10.0.3.1 as their gw.

Devices on the 192.168.0/24 would point to pfsense IP in that interface as their gw 192.168.0.1 lets call it. Again there are NO hosts on the 172.16.0/30 it is a transit network!!!

So see attachment 2 with multiple downstream routers. So now we needed to expand our transit to allow for more IP so lets make it a /29 now pfsense would be .1 router would be .2 and router2 will be .3 in this transit network.

So router 1 default route would be to pfsense at .1 on the transit. But he would need route to 10.0.4 and 10.0.5/24 pointing to router 2 at 172.168.0.3
Router 2 would need default pointing to pfsense and routes for 10.0.1, .2 and .3 pointing to router 1 at 172.16.0.2
Pfsense would need routes saying hey to get to 10.0.1,2 and .3 talk to router 1 at 172.168.0.2 and routes for 10.0.4 and .5 talk to router 2 at 172.168.0.3

pglover19 · Aug 25, 2016, 12:02 PM

@johnpoz:

When you say core switch.. Do you have more vlans off this switch?? Or devices on this 192.168.1/24 network - if so then that is not a transit network..

you clearly have hosts on this network… 192.168.1/24 that is NOT a transit network... Your going to have all kinds of problems..

A downstream router needs to be on transit network or your going to have asymmetrical routing issues.

So you can have multiple networks/vlans hanging off pfsense.. But when you connect to downstream it should be via a transit network, ie no other hosts on this network.. You can have more routers all connected together via a transit, etc. But you do not put hosts on a transit network. Or talking to these hosts is going to have issues with asymmetrical routing unless you create routes on all these hosts on which gateway/router to use to get to specific networks.

See attached. The transit in this drawing it the 172.16.0/30 lets say pfsense is .1 and router is .2 on that transit. The routers default gateway would be 172.16.0.1 ie pfsense. The other interfaces on the router would not have gw set. Devices on those vlans would point to the routers IP in those vlans ie 10.0.1.1, 10.0.2.1 and 10.0.3.1 as their gw.

Devices on the 192.168.0/24 would point to pfsense IP in that interface as their gw 192.168.0.1 lets call it. Again there are NO hosts on the 172.16.0/30 it is a transit network!!!

So see attachment 2 with multiple downstream routers. So now we needed to expand our transit to allow for more IP so lets make it a /29 now pfsense would be .1 router would be .2 and router2 will be .3 in this transit network.

So router 1 default route would be to pfsense at .1 on the transit. But he would need route to 10.0.4 and 10.0.5/24 pointing to router 2 at 172.168.0.3
Router 2 would need default pointing to pfsense and routes for 10.0.1, .2 and .3 pointing to router 1 at 172.16.0.2
Pfsense would need routes saying hey to get to 10.0.1,2 and .3 talk to router 1 at 172.168.0.2 and routes for 10.0.4 and .5 talk to router 2 at 172.168.0.3

On the core switch, I have vlan 1 (default), vlan 10, and vlan 20. That is all the vlans that will be configured on the core switch. The default vlan on the core switch will not be used. For vlan 10 on the core switch, I have connected my hypervisor server and other physical servers. For vlan 20 on the core switch, I have connected my NAS/SAN Storage server.

johnpoz · Aug 25, 2016, 12:46 PM

And now you have a problem don't you..

So you have attached.. Is this how you have it setup? So gateway of say your server on vlan 10 is your svi for vlan 10 on your downstream router (juniper)?

So PC wants to talk to your server he talks to pfsense.. Who then sends back to juniper who routes it to vlan 10, then sends it back across the same line again through your core switch. Then where does server send traffic back.. To the juniper which is the gateway for vlan 10, who says oh you want to talk to vlan 1 sure thats right here and now you have asymmetrical problem.. Let alone lots of hairpins, etc.

So lets say you make the pc gateway for this vlan 1 the juniper IP in this vlan 192.168.1.2 so that gets rid of your mess talking to other vlans

Now you have this mess when pc wants to talk to internet. So he bounces off juniper since that is his gateway, juniper says oh my default gw is pfsense at .1 on it goes. Return traffic pfsense says oh you want to talk to IP in 192.168.1 yeah I am directly connected and sends direct to the pc machine in 192.168.1.207

So again lets be clear if your going to have downstream routers you need to use a transit network.

pglover19 · Aug 25, 2016, 2:40 PM

This is exactly my setup. The downstream router is my Juniper EX3300 switch that provides all inter-vlan routing.

I will do some research on creating a transit network.

Thank you do much on all the expert advice.

johnpoz · Aug 25, 2016, 3:14 PM

not a problem. As your network grows/expands beyond 1 flat network lots of things start coming into play that need to be taken into account.

For starters as you start adding downstream or daisy chained switches you need to worry about bottlenecks and or hairpins, if you add downstream routers asymmetrical routing comes into play as well. As you start to grow more away from just a core switch/router all at 1 spot do you need a distribution layer for your switches or just closet/access layer.

As it grows and you start to do failover or load balancing for your uplinks between your switches spanning tree and or loops become a possible issue, etc. etc. etc..

Having what your calling your core switch between your edge and an internal router and placements of devices on different vlans location and where most of your traffic flows needs to be taken into account when you do your layout so you don't have bottlenecks or multiple hairpins and asymmetrical routing..

For example might be better even with a transit network to put what your calling your core below your downstream.. Why do you have your nas on different vlan than your servers? Do your servers not access the storage and only users? Maybe it would be better to put your servers and nas all on same vlan so your not routing between them? And best to maybe be on the same switch so your not having to go through an uplink?

Would need to know your physical location of your servers/infrastructure type devices and where your users sit and where any closet switches are, etc. And what the major data flows are to best layout the network and vlans, etc. And then what security you would want/need between your segments. Is that juniper layer 4+? Can it do ACL's to filter/block traffic you don't want between your vlans? Or does it just route? I would assume you can do acl's there and filter traffic as you need too, etc.

As the network grows is when it all gets fun! ;)