TCP responses / ACK routing problem on multiple networks

p-d-f

Hello mates,

our old Cisco router crashed and we set up a new router. pfSense is looking good, so we used it :-) (Tried it 2 weeks before and all was fine.)
Of course the problems started in productive environment.

All the several problems we have seem to have one reason: the response packets hop directly to the clients, not across the router.

Our basic network is:
192.168.0.0/24: infrastructure and several servers
192.168.6.0/24: clients
192.168.10.0/24: telecommunication services

The gateways are 0.5, 6.5 and 10.5, always the same pfSense router. Every Subnet has a separate physical interface on the router.
Because of problems with the old telecommunication server we cant really physically split the networks. So all switches of the networks are bound together.

A server on the .0 net has services that run in .0 and services that run in .6 (DHCP, VPN).
The telecommunication server has network cards for .6 and .11 (.11 doesnt matter in this case) networks. However the main IP is 192.168.10.10. The client phones use 10.10 to connect to the server. (software, VOIP like phones; don't ask, its really old and creepy and we are happy it works… or worked ^^)

The problem we have on the phones and for example a web page where you can upload files have the problem, that they don't work properly.
The phone disconnects after 2 to 6 minutes. A file can be uploaded up to ~ 62kbyte. A file via scp will be sent until 2112 kbytes and then stalls. SSH Session disconnects after a few minutes.

We tcpdumped a lot and what we saw was the following:
A .6 client sends something to the .0 server

We see the packages outgoing on the .6 machine and we see the ACKs received
We see the packages incoming on the .0 machine and the ACKs send
We see the packages incoming on the router but no ACKs from the .0 machine - but they arrive at the client

If we take the .6 interfaces from the .0 server away we also see the ACKs on the router and all is working fine. We could do proper routings on the .0 machine, but we can't do something like this on the telecommunications server.

If we reactivate our old Cisco router we had many, many other problems (because of hardware and whatever failures), but not the problems told above.

So the old router had no problems, the new has.
The question is now: how can we fix this? ;-)
We know the router doesn't receive ACKs, whats not really good but it seems to be normal in our case. Maybe one rule is enough to tell pfSense that he should not need to stop and let the packages flow without limitations.
We have the book here, we searched a lot for a few days now, and now we need your help.

We hope you can help us!

Greetings from Germany, and sorry for the bad language ;-)

cmb

What you're seeing there is the difference between a router and a stateful firewall. If you had a Cisco ASA in there, it wouldn't work for the same reasons. When you have dual homed hosts like that, the firewall only sees half the traffic, and it eventually starts to look like spoofed traffic. That scenario is impossible to work around in many stateful firewalls, though we do have an option with the "sloppy state" that lets you do so, it's generally preferable to not dual home hosts. You can work around by changing the rules on your internal networks to state type "sloppy", and any TCP flags. Also add floating rules (pass quick out on all your internal interfaces, sloppy state), that will let such traffic pass. Don't add such rules on your WANs, they shouldn't be necessary there, and that at least keeps tighter firewalling for Internet-initiated traffic.

p-d-f

Hello cmb,

thank you very, very much! Sloppy state was exactly what we are looking for. We saw that in the advanced settings, but we didn't know what it's doing. We added / edited the rules for our internal networks.
Our network works since 2 hours as it should be :-)

So thanks again, and have a nice day!

darnitol

Is it a good idea to turn on the Sloppy State option for all internal routing that doesn't require firewalling?

I shot a man in Reno, just to watch him die.

cmb

@darnitol:

Is it a good idea to turn on the Sloppy State option for all internal routing that doesn't require firewalling?

Never in normal circumstances. Only in unusual cases with multi-homed hosts or other cases for asymmetric routing. Both of those should be avoided in general.