Slow internal LAN web traffic with PFSense
-
Bump!
Just wanted to see if anyone thought this is a logical place to look before I take down the network for more maintenance this week.
-
Bump!
Just wanted to see if anyone thought this is a logical place to look before I take down the network for more maintenance this week.
IMO you should try to really isolate the problem first, before trying to fix it. There are too many variables. Follow the problem step by step from the clients to the web server; take a look at the logs in the web server, check where is it connecting and how, which services is it using. You should also check the switches logs. Focus on isolating and understanding the problem before trying to do anything else.
-
Thanks feadin,
There is nothing of note on the web application logs, traces from client to server, or the switch logs. The only thing that looked out of sorts was the entry I posted last from the state table. The web server is http/https only for the client side and all packets either route to the local LAN or through the pfSense system and out to the WAN. No additional information of merit is given. We have checked DNS, switches, etc. in diagnostics by replacement followed by testing. Nothing has helped. We can not recreate the problems seen in a test environment, and everything appears to work correctly when several users are on the web system at once for testing. Once all the users come on then the problems become evident. My suspicion of the NAT might be a false lead – this is why I am asking the community before I chase it. Taking down the working routing system to something that doesn't work correctly really ticks off the users and causes substantial downtime, so I have to dry lab and plan everything before going live with a test.
Bump!
Just wanted to see if anyone thought this is a logical place to look before I take down the network for more maintenance this week.
IMO you should try to really isolate the problem first, before trying to fix it. There are too many variables. Follow the problem step by step from the clients to the web server; take a look at the logs in the web server, check where is it connecting and how, which services is it using. You should also check the switches logs. Focus on isolating and understanding the problem before trying to do anything else.
-
Thanks feadin,
There is nothing of note on the web application logs, traces from client to server, or the switch logs. The only thing that looked out of sorts was the entry I posted last from the state table. The web server is http/https only for the client side and all packets either route to the local LAN or through the pfSense system and out to the WAN. No additional information of merit is given. We have checked DNS, switches, etc. in diagnostics by replacement followed by testing. Nothing has helped. We can not recreate the problems seen in a test environment, and everything appears to work correctly when several users are on the web system at once for testing. Once all the users come on then the problems become evident. My suspicion of the NAT might be a false lead – this is why I am asking the community before I chase it. Taking down the working routing system to something that doesn't work correctly really ticks off the users and causes substantial downtime, so I have to dry lab and plan everything before going live with a test.
What kind of connections and services use the webserver on it's server-side? Did you check those when problems start?
If you cannot reproduce this on a lab, I would start testing right on the client computer when the problem starts. Testing connectivity between client and web server, then connectivity between web server and every service and/or host it uses; like databases, dns, wins, even broadcasts. Go step by step. No point on keeping this on a basic network level only, check other levels as well as they are all interdependent. Even if the problem is at a basic network level, checking other levels allows you to isolate it much faster. After you isolate the problem the solution will be easy. I see no point on trying possible solutions blindly. -
Thank you everyone for your help - the system has been running for 1 week with no user reported problems. What I did was explicitly go to every 1:1 and port forward entry and disable NAT reflection. In advanced I disabled every reference to NAT reflection. All of the SYN:CLOSED entries in the state table dissipated after this. I noticed a number of FIN WAIT 2 entries, to attempt to resolve this I used the advice from another thread and set the packet timeout to 1 second for the web server routing entry - it was disastrous for performance and I had to revert. Despite a number of FIN WAIT2's in the state table, everything works fine now.
-
Good old NAT reflection. This is why split DNS is used. Internal IP are served to LAN and external are served to WAN originating connections.
That is what it sounded like you where doing, but since you turned off NAT reflection, it seems that it was not. -
Good old NAT reflection. This is why split DNS is used. Internal IP are served to LAN and external are served to WAN originating connections.
That is what it sounded like you where doing, but since you turned off NAT reflection, it seems that it was not.This is the strange thing–-dns on the inside resolved correctly for the webserver when we were still having problems----there must have been something hardcoded somewhere that caused the problem. Potentially this is in the PFsense box itself---it would just continue to try and unsuccessfully NAT data - causing timeouts. When the capability was disabled, with no other network changes, everything worked well
-
Not sure why. If the DNS returned internal address (assuming they are on the same subnet) then the traffic should never have gotten to the firewall at all. If you were going to DMZ from a LAN for instance, then it would go through the firewall, but NAT reflection would not have much to do here. You could even switch to advanced outbound NAT and not NATed for that traffic at all, just pure firewall and routing.
-
Not sure why. If the DNS returned internal address (assuming they are on the same subnet) then the traffic should never have gotten to the firewall at all. If you were going to DMZ from a LAN for instance, then it would go through the firewall, but NAT reflection would not have much to do here. You could even switch to advanced outbound NAT and not NATed for that traffic at all, just pure firewall and routing.
What it seemed like was happening was the web server was spending time trying to maintain dropped connections to the outside at the expense of inside connections - which should never touch the firewall. All internal machines used an internal DNS server that specified the IP for the web server that was on the same subnet. It looks like the symptoms we were seeing were indirectly related to the reflective NAT issue. For some reason there were tons of connections between the server and itself trying to loop back over an external address–-my best guess is that something somewhere was hardcoded to talk over that IP. But if that were the case, removing NAT reflection would not resolve the issue - it would still try and talk out and back and be blocked. I'm still at a loss to the exact mechanism of the problem but any speculation to help others in the future is welcome.
-
What it seemed like was happening was the web server was spending time trying to maintain dropped connections to the outside at the expense of inside connections - which should never touch the firewall. All internal machines used an internal DNS server that specified the IP for the web server that was on the same subnet. It looks like the symptoms we were seeing were indirectly related to the reflective NAT issue. For some reason there were tons of connections between the server and itself trying to loop back over an external address–-my best guess is that something somewhere was hardcoded to talk over that IP. But if that were the case, removing NAT reflection would not resolve the issue - it would still try and talk out and back and be blocked. I'm still at a loss to the exact mechanism of the problem but any speculation to help others in the future is welcome.
My guess would be that the html/php/asp is telling the client to go to http://<externalip>/internalpage.html/php/asp instead of ./internalpage.html/php.asp and as a result you where getting essentially redirected to the external ip instead of it using the internal ip from DNS. This happens sometimes when your webpage needs to load data from another page. This is generally the wrong way to setup a website IMO.</externalip>