Disconnects when routing through same interface

drifter104

Hi a Quick idea of the network

PFsense
Outside: 1.1.1.1
Inside: 10.10.10.1

Router A
E0: 10.10.10.254
E1: 10.10.20.1

In the config on the PFsense is a gateway for Router A and static route for the 10.10.20.0/24 via that gateway.
I've configured PFsense so that it doesn't apply firewall rules to traffic going in and out on the same card.

The problem I'm having is that with anything that requires a constant flow of data like RDP for example or SMB (during file transfers) disconnects randomly.
So for example RDP from 10.10.10.50 > 10.10.20.100
The tracert would be

10.10.10.50
10.10.10.1
10.10.10.254
10.10.20.100

I'm sure the problem is with the PFSense because if I put the following static route on my machine there are no issues (route add 10.10.20.0 mask 255.255.255.0 10.10.10.254) so basically bypassing the routing on the pfsense, I don't have the issues.
The tracert then would be

10.10.10.50
10.10.10.254
10.10.20.100

Any ideas?

johnpoz

10.10.10.50
10.10.10.1
10.10.10.254

Why would you have 3 hops all in the same network.. That doesn't seem correct. Can you draw this network..

drifter104

Hi

Of course here you go

Basic.jpg_thumb

johnpoz

Well that wouldn't be the trace route, .50 is not in the hop. If you hit pfsense as your gateway then that would be the first hop. So it would go

10.10.10.1
10.10.10.254
10.10.20.100

That can clearly be worked out better, I hate to hair pin connections like you have. Do you have another nic you can use for pfsense and use a transit network for you pfsense connection to router A? You could do it with vlans but then your still hairpinning.

Why do you need router A at all, just connect your 10.10.20 to another segment in pfsense.

you run into asynchronous issue with your type of setup.. See first pic

You have a few different options. If you have another nic in pfsense just hang the 2 segments direct off pfsense. You could vlan the other segment off the same interface. If you have requirements for that other router, then use a transit network say 192.168.0.0/29 so you don't have async routing problems, can be done with vlan if have limit of 1 nic in pfsense.

But best option is to use 2nd nic in pfsense for your 2 segments, other option is transit network to get to this other ruoter be it with 2nd nic in pfsense or vlans - see the different drawings, pic 2

asyncroute.png_thumb

options.png_thumb

drifter104

I'll see what I can do with those options.

That diagram was just the parts involved in this particular problem. I have 13 gateways and 20 static routes sigh (inherited) on that one pfsense.

I'd like to have a subnet per router, with a transit network connecting all the routers/firewalls (depending on the subnet) and only internet bound connections going via the pfsense, I'm just not in a position to do that at the moment. I've got as far as lifting out the previous firewall and replacing with pfsense but then came up against this issue, didnt appear in testing because it does connect and only disconnects at random.

Thanks for the help

phil.davis

You can do this while you sort out your network jungle; on pfSense LAN where you have the static route, also add Outbound NAT for traffic to 10.10.20.0/24 and NAT it to pfSense LAN IP. Then the returned traffic is delivered back to pfSense, which unNATs it and delivers it to the proper client. That makes the routing symmetric and thus keeps pfSense stateful firewall happy.

drifter104

@phil.davis:

You can do this while you sort out your network jungle; on pfSense LAN where you have the static route, also add Outbound NAT for traffic to 10.10.20.0/24 and NAT it to pfSense LAN IP. Then the returned traffic is delivered back to pfSense, which unNATs it and delivers it to the proper client. That makes the routing symmetric and thus keeps pfSense stateful firewall happy.

I think I did this originally but it made it difficult to secure some of the subnets (some have firewalls and acls) and it meant the source address was the LAN interface. I need it to be the real source otherwise (as I understood it) I'd have to set the IP of the source on the ACL to be the PFsense LAN address and this would mean opening the subnets to potentially everything. Another reason why I want a transit network that all routers/firewalls sit on.

Thanks for the suggestion though

johnpoz

"I have 13 gateways"

Why would you have so many gateways.. Do you have that many upstream wan type connections? You don't put gateways on "lan" interfaces - downstream connections. To get to downstream networks you add a route.

Sounds like you have a bit of mess. Ho many routers (what are they) do you have, how many segments? What is your total node count? If you don't mind posting up a more detailed picture happy to discuss options to clean it up, etc.

phil.davis

I think I did this originally but it made it difficult to secure some of the subnets (some have firewalls and acls) and it meant the source address was the LAN interface. I need it to be the real source otherwise (as I understood it) I'd have to set the IP of the source on the ACL to be the PFsense LAN address and this would mean opening the subnets to potentially everything.

For an interim measure you can put firewall rule/s on pfSense LAN to filter traffic from whatever pfSense LAN IPs you want to control that has destination in 10.10.20.0/24 - that gives you filtering at the point before the original client IP address is NATed into the pfSense's own LAN IP. Then let other firewall/router devices in the path pass everything from pfSense LAN IP, knowing that it has been pre-filtered by pfSense.

cmb

That's quite a mess indeed. While you clean it up, System>Advanced, Firewall/NAT, check "Bypass firewall rules for traffic on the same interface" will fix the problems inherent in asymmetric routing and trying to filter.

drifter104

@cmb:

That's quite a mess indeed. While you clean it up, System>Advanced, Firewall/NAT, check "Bypass firewall rules for traffic on the same interface" will fix the problems inherent in asymmetric routing and trying to filter.

I currently have that ticked but I still get the disconnects

@phil.davis:

For an interim measure you can put firewall rule/s on pfSense LAN to filter traffic from whatever pfSense LAN IPs you want to control that has destination in 10.10.20.0/24 - that gives you filtering at the point before the original client IP address is NATed into the pfSense's own LAN IP. Then let other firewall/router devices in the path pass everything from pfSense LAN IP, knowing that it has been pre-filtered by pfSense.

I don't think I've tried that, will look into it.

@johnpoz:

"I have 13 gateways"

Why would you have so many gateways.. Do you have that many upstream wan type connections? You don't put gateways on "lan" interfaces - downstream connections. To get to downstream networks you add a route.

Sounds like you have a bit of mess. Ho many routers (what are they) do you have, how many segments? What is your total node count? If you don't mind posting up a more detailed picture happy to discuss options to clean it up, etc.

Thanks for the offer, I've attached a more complete diagram (if you change your mind once seeing I understand :D )

What you said about gateways just from a purely config point of view raises a question. When I put in static routes through the pfsense webgui I have to "pick" an existing gateway. If I didn't add the gateway I couldn't add the static route could I? Or is it better/possible to add the static routes as the pfsense "OS" level?

All the "routers" on the diagrams are vyatta, all the "Firewalls" are ciscos PIX

Basic.jpg_thumb

doktornotor

@drifter104:

I've attached a more complete diagram

If I were to manage this mess, I'd find a new job. Good luck.

drifter104

This is an improvement. :-\

I've got 20 Pix 515 that I've removed the equation so far.

johnpoz

I don't get it?? Why would anyone set up something like that??

And you have multiple down stream routers for different segments as well? Is there anything below those routers? More routers? Oh so that is what you meant by gateways.. Why so many?? Where is the core of this network? So all your routers are running vyatta? On what hardware? Why would you not just put in a nice layer3 switch and be done?

So what pix, you mentioned a 515 you got rid of.. What are the existing ones? Why don't you just replace those all with 1 pfsense box? You can easy add multiple ports there. But I don't understand why you need so many segments? If you want the ext and internal - great that is 2 boxes ;) And then a L3 switch below there for your other segments. Sure set them up in HA if you want, etc. So say 4 boxes 2 ext, 2 internal and 2 L3 switches.

drifter104

@johnpoz:

I don't get it?? Why would anyone set up something like that??

And you have multiple down stream routers for different segments as well? Is there anything below those routers? More routers? Oh so that is what you meant by gateways.. Why so many?? Where is the core of this network? So all your routers are running vyatta? On what hardware? Why would you not just put in a nice layer3 switch and be done?

So what pix, you mentioned a 515 you got rid of.. What are the existing ones? Why don't you just replace those all with 1 pfsense box? You can easy add multiple ports there. But I don't understand why you need so many segments? If you want the ext and internal - great that is 2 boxes ;) And then a L3 switch below there for your other segments. Sure set them up in HA if you want, etc. So say 4 boxes 2 ext, 2 internal and 2 L3 switches.

Very poor advice from the IT consultant that put this all in many years ago. I think he just saw the £'s to be honest.
Each of the "C" subnets had 3 to 4 pix. So it would be WAN>Pix>Server>Pix>Server>Pix>Server<->Pix<->LAN with a different subnet on each side of each device.

We have some legacy contractual requirements to keep some networks seperated etc but certainly not the extent of where we are at. All the subnets are seperate VLANs already which makes moving towards reduced number of boxes less painfull (all switches are configured). We don't have any layer 3 switches though (at the moment but if I can make a good case for them it would help) all layer 2 Cisco 2960 (older type)

Most of the subnets are to seperate development, management, iscsi, voip, dmz/web servers, backend/sql servers

No routers behind the downstream routers

doktornotor

@drifter104:

Very poor advice from the IT consultant that put this all in many years ago. I think he just saw the £'s to be honest.
Each of the "C" subnets had 3 to 4 pix. So it would be WAN>Pix>Server>Pix>Server>Pix>Server<->Pix<->LAN with a different subnet on each side of each device.

You should have talked to your lawyers a couple of years ago, perhaps? WTF!!! :o :o :o :o :o

phil.davis

Seeing the later comments I did not even click to enlarge the diagram :'(

What you said about gateways just from a purely config point of view raises a question. When I put in static routes through the pfsense webgui I have to "pick" an existing gateway. If I didn't add the gateway I couldn't add the static route could I? Or is it better/possible to add the static routes as the pfsense "OS" level?

From System->Routing, Gateways tab, you add a gateway for each other router that is sitting on a locally-connected interface/subnet of the pfSense and is the next hop for some destination/s. Then you DO NOT make these the actual gateway of any Interface (i.e. in the Interfaces menu, DO NOT put any gateways against your LAN-style interfaces.
Then you add Statis Routes from the Routes tab - that tells pfSense which destination subnet are reached through which gateway. Then pfSense sets up the routing table for you.

drifter104

@phil.davis:

Seeing the later comments I did not even click to enlarge the diagram :'(

What you said about gateways just from a purely config point of view raises a question. When I put in static routes through the pfsense webgui I have to "pick" an existing gateway. If I didn't add the gateway I couldn't add the static route could I? Or is it better/possible to add the static routes as the pfsense "OS" level?

From System->Routing, Gateways tab, you add a gateway for each other router that is sitting on a locally-connected interface/subnet of the pfSense and is the next hop for some destination/s. Then you DO NOT make these the actual gateway of any Interface (i.e. in the Interfaces menu, DO NOT put any gateways against your LAN-style interfaces.
Then you add Statis Routes from the Routes tab - that tells pfSense which destination subnet are reached through which gateway. Then pfSense sets up the routing table for you.

Thats exactly what I've got, sorry I've not explained the configuration I done regarding these very well.

drifter104

@johnpoz:

I don't get it?? Why would anyone set up something like that??

And you have multiple down stream routers for different segments as well? Is there anything below those routers? More routers? Oh so that is what you meant by gateways.. Why so many?? Where is the core of this network? So all your routers are running vyatta? On what hardware? Why would you not just put in a nice layer3 switch and be done?

So what pix, you mentioned a 515 you got rid of.. What are the existing ones? Why don't you just replace those all with 1 pfsense box? You can easy add multiple ports there. But I don't understand why you need so many segments? If you want the ext and internal - great that is 2 boxes ;) And then a L3 switch below there for your other segments. Sure set them up in HA if you want, etc. So say 4 boxes 2 ext, 2 internal and 2 L3 switches.

With the absense of Layer 3 switches would the following be possible? Likely to cause any issues?

Green and Orange represent data flow along different VLans (only drawn 2 I have 25 in use) on the internal network. Blue would be a Vlan on the external side for traffic between subnets and would be on a private address range, red would be vlan for internet bound traffic, with the interface being public ips

Black lines are the physical connections and would pretty much all be trunk connections.

Hopefully that all makes sense, and thank you for your input

![Network and vlan.jpg](/public/imported_attachments/1/Network and vlan.jpg)
![Network and vlan.jpg_thumb](/public/imported_attachments/1/Network and vlan.jpg_thumb)