Strange routing problem with static routes - ALIX hardware problem?
Today I had a strange problem and I was not able to resolve it, I worked around it, but I do not know why this worked.
ALIX-Board running pfsense 1.2.2 embedded doing the Internet traffic as default gateway for some Macs. Because the Macs have to reach a Lotus Notes server that is in a different network, I setup a static route to this network via a second router.
At first everything looked fine, I can ping the Lotus server - fine.
After a few hours my customer called and told me they have problems connecting to the Lotus server.
So I started to sniff the network traffic and everything looks fine, except that from time to time there were no answers from the Lotus server, the clients send packet after packet but the server did not answer. After some time the Lotus client runs into a timeout and told me to use another Lotus server (replik server), I clicked to use a different server (the customer has only one Lotus server, so the client connects to the same server again) and immediately was able to connect to the server to open mails. Round about 1 minute later the same problem - Lotus client got hung, after timeout - use different server - and Lotus works as expected for a few minutes.
After that I believed in a network problem at the Lotus server until I tried two different things:
1. ping the Lotus server while the network traffic got stuck - ping works without a problem, no lost paket at all!
2. setup a static route at the client computers and the problem was gone!
This is strange, right?
I used tcpdump at my pfsense router and what I can tell is every paket send by the clients reached my router and got send to the second router. But I am not very experienced in interpreting network traffic.
I wonder why is the problem gone if I setup static routes at the client computers and if am doing it the "right" way with a static route at the default gateway the network traffic periodically got stuck.
Maybe this is a hardware issue with ALIX and pfsense 1.2.2 embedded?
Did I miss something obvious?
BTW. I have no access to the second router
Check "Bypass firewall rules for traffic on the same interface" under System -> Advanced.
Hm, but why does it work for sometime and after a few seconds/minutes got stuck?
Why does ping works every time - because it is ICMP and this does not go through the firewall?
Does not really make sense to me - i would expect that it works or not but "works sometimes" is not very satisfying for a simple static route.
Well, next when I am at the site I try it, I remove the static route from the client and see what is happening. "Bypass firewall rules for traffic on the same interface" is already checked thanks to your advice.
You can't statefully filter that traffic properly with any stateful firewall, weird things will happen because the firewall can't see the entire conversation as it's asymmetrically routed.