Routing public netblocks locally

[jwelter99]

Hello,

At a co-location we have 3 public ip blocks.  These are all /28 blocks.  They are all routed by the co-lo to our racks over a pair of gig-e links (redundancy), and inside that rack we have a L3 switch for external traffic.

Connected to that L3 switch we have several Cisco ASA's and now a pFsense box.

The problem we have is that we configure the devices on the 3 ip-blocks to locally route between them to avoid bandwidth fee's which are charged when you cross the co-lo's provided gateways.  So traffic from ip block 1 to ip block 2 will be billed if it crosses the co-lo routers, but is free if routed inside our racks.  With the Cisco ASA's we simply setup static routes for these different IP blocks and it works fine.

The issue we are having is with pfSense and configuring it to use such local routes.  We can't seem to get this to work.

Any guidance would be greatly appreciated.

Thanks in advance!

[GruensFroeschli]

How are your static routes set up?

[jwelter99]

@GruensFroeschli:

How are your static routes set up?

I've tried it a few ways….

In the System->Static Routes I setup the routes.  As an example one of our public blocks is 70.200.26.128/28.  So I setup a route for that network range to use a interface local on that network as the gateway - in this case 70.200.26.135.  I also ensure that the pfSense box can see this interface and does have it listed in the ARP cache.  What is odd is that after setting this up I do not see the route listed in the Diag->Routes screen.

The above didn't work.

So I then tried to use policy-rules so that any traffic destined to 70.200.26.128/28 to use that as a gateway - the problem is that gateway isn't listed in the pull down list in the rules selection.

On the ASA's I set it up the same way, and it works to route locally TO the network the pFsense box is responsible for.  To make this work I just had to add a WAN rule allowing source ASA WAN to pFsense WAN.

Thanks!

[GruensFroeschli]

Did you configure the gateway on the interface-config-page?
If this is set, you should be able to select the interface in the dropdown in a rule.

Can you show a screenshot of your static route which is not working?

[jwelter99]

Hi, I think I may have explained this incorrectly.  Let me add more detail:

Assume the following 2 IP blocks:

block 1: 70.200.26.128/28, assigned to WAN of a Cisco ASA, gw: 70.200.26.129, ASA WAN if is 70.200.26.138
block 2: 70.200.29.208/28, assigned to WAN of pfSense 1.2.3 box, gw: 70.200.29.209, pfSense WAN if is 70.200.29.220

On Cisco ASA I have a route set:

IF: WAN
Destination: 70.200.29.208/28
Route: 70.200.29.220

On Cisco ASA traffic to 70.200.29.208/28 leaves Cisco WAN IF and goes directly to 70.200.29.220.  It does not traverse the gw for 70.200.29.208 set on the Cisco (which costs $$$).  I had to turn on the allow traffic local on interface in advanced to make this work, but it does work fine.  pfSense box sees traffic from source 70.200.26.138 to destination 70.200.29.220, I can see that in the logs when the allow traffic local on IF was off.  In ARP view on pfSense it is aware of 70.200.26.138.  On Cisco ASA ARP of 70.200.29.220 is known.

On the pfSense I have the following static route set:

IF: WAN
Destination: 70.200.26.128/28
Route: 70.200.29.138

The issue is this doesn't show up in the "diagnostic->routes" list.  It also doesn't show up as an option in the gateway field of rules to setup policy-routing.

So in essence all I want to do is force the pfSense box that any outgoing traffic on the WAN port to 70.200.26.128/28 doesn't use the default GW set for the WAN itself and routes directly to the Cisco.

I apologize if I've muddled some technical terms and such.  Just a lot of years experience on ASA's but now using pfSense as it is MUCH better in many regards.

[rikrobson]

@jwelter99:

So in essence all I want to do is force the pfSense box that any outgoing traffic on the WAN port to 70.200.26.128/28 doesn't use the default GW set for the WAN itself and routes directly to the Cisco.

I apologize if I've muddled some technical terms and such.  Just a lot of years experience on ASA's but now using pfSense as it is MUCH better in many regards.

I sounds like what you are tryng to do is make the pfsense do something outwith it's control.

in essence the pfsense box is just chucking stuff out it's WAN port. once done the switch is the key and should switch the data to the cisco's. if you want the pfsense box to do this then you need to make it route the traffic via another interface to the ciscos directly and setup static routes there.

what you are doing here is telling the PF box to route all traffic via it's wan interface to the switch. then wondering why the switch isn't routing.

with the static route will get the traffic to the switch, which is what would happens anyway then the switch is sending the traffic to the wan ans not the cisco.

[jwelter99]

Yes, I could change the gateways on the Cisco and ASA boxes to specify the L3 upstream switch and do the routing I need on it - but I was hoping it could be accomplished like on the Cisco ASA's by adding an extra route.  It also means using 2 public IP's for the router interfaces on the upstream switch.

I sounds like what you are tryng to do is make the pfsense do something outwith it's control.

in essence the pfsense box is just chucking stuff out it's WAN port. once done the switch is the key and should switch the data to the cisco's. if you want the pfsense box to do this then you need to make it route the traffic via another interface to the ciscos directly and setup static routes there.

what you are doing here is telling the PF box to route all traffic via it's wan interface to the switch. then wondering why the switch isn't routing.

with the static route will get the traffic to the switch, which is what would happens anyway then the switch is sending the traffic to the wan ans not the cisco.