Connection Drop after 10 Seconds, TCP, HTTP
-
Indeed, these hacks digging holes into your setup are just horrible and absolutely should not be required for anything.
-
They were not required before when i was using a Cisco 7507 at the gateway, when i moved this system where i have pfsense is when the issue first came around, but it was handlable and only intermittent untill the reports grew in size.
doktornotor, the fact that im looking for a better way to do this, in of itself denotes that im aware this is not ideal, so your post was not called for, if you arent going to contribute, please move along.
There should be no reason why you have to edit such settings. Did you take a look at pftop when your connections where active to see what the timeouts where in real time for your states??
I agree pftop would be able to help narrow the issue, if it were not for the fact that this network hosts 7 servers, a total of 27 websites. The one server the issue occurs on hosts 8 such sites, all on the same ports using apache virtualhosts if your familiar with it.(its not virtualization related) The number of states at peak times has hit 450,000.
This isnt a small 1 off network, this is at a datacenter, with a LOT of traffic, and the server in question being a 12 core(24 thread), 144 GB RAM monster box that handles MySQL for all the other servers as well as internet based systems using https apis.
not your average john boy setup to host a personal webpage from his basement on a extra pc.
-
My test file is here: http://pfmon.black-knights.org/test.php
I don't suppose you would share your code so I could test here eh?
Curious if you have tried 1:1 NAT in favor of port forwarding? ???
-
all that file does is:
while($i <= 30)
echo $1
$i = $i + 1;
sleep(11);it just sends numbers every 11 seconds to see if the connection is still alive.
if the browser counts all the way to 30, then the issue is fixed. if it stops for more than 11 seconds then its died.
-
"The number of states at peak times has hit 450,000."
So maybe your running into state exhaustion and pfsense is killing off the idle ones?
"The one server the issue occurs "
So you have other servers serving up stuff behind pfsense and this sort of thing doesn't happen with them? Why don't you isolate out this box or try and duplicate on test..
Dok is pointing out that what your doing is not a good idea, and that is very much so a valid contribution to the thread.. If someone like dok says its a bad idea - then its a BAD Idea!! And I agree what your doing is hack that should not have to be done… You got something else going on, what your doing is hiding the actual problem.
-
I really hate to state the obvious again, but – have you tried this with a physical machine?
-
the issue is solved, if it was a virtualization related issue i would not have solved it by changing the timeout of pfsense.
I think the source of the issue is this.
PFSense terminates sessions that are openning, if the machine behind pfsense doesnt respond within 10 seconds, period.
When apache/php is doing a large report processing job, it can take between 2 seconds for a small report, and 15-20 seconds for a large report.
if there was a problem in the virtualization, it would be affecting more than this 1 program.
This is not your average situation, this is a workload the likes of which you may not have seen before.
I agree this is not an ideal fix, but please doktornotor, please explain why this is a bad idea to you, from a technical standpoint, so maybe i can see your thought process for this assumption.
-
You know, because… well, this just happens to noone but you, pretty much.
PFSense terminates sessions that are openning, if the machine behind pfsense doesnt respond within 10 seconds, period.
Errr…. huh. No.
-
if your not going to back up your responses with anything technical, then find someone else to not help.
-
if your not going to back up your responses with anything technical, then find someone else to not help.
Your current fix includes lowering timeout values well below the defaults?
-
Actually making them longer, its seems as the stack is building the report it doesnt respond at all untill the report is actually complete, and then sends it.
but the sending was happenning just after the timeout, so the reponse from the stack was getting blocked from going out, as its state had already been dropped.
stack = MySQL, PHP, Apache.
sidenote: i only came to this conclusion after thouroughly testing all of the timeout settings in PHP, and apache, and nothing i did made any difference to the issue. and as i was able to confirm with a proof-of-concept test file that simulated the same delay but without actually doing anything, the identical behavior was seen, this it isnt a load issue. A php file that slept 11 seconds then printed a word, would never actually print anything, if the sleep was lowered to 9 seconds it reponded every time, and at 10 seconds it would respond intermittently because it was right on the wire timing wise. A touch command inserted into the php file after the print, revealed that even when it failed to respond it was indeed processing to completion but pfsense was not allowing the data out due to the closing of its state. Further evidenced when i saw a outbound denial in the firewall logs with a source port of 80 and from the webserver, meaning it was a response to an http request.
-
I wonder why tcp.first & tcp.open made an impact since I assume tcp.established should be the only relavent parameter.
I'm too curious to leave it alone but I guess if it works, it works.
Why did you change state tracking to sloppy (or none?)?
-
I wonder why tcp.first & tcp.open made an impact since I assume tcp.established should be the only relavent parameter.
I'm too curious to leave it alone but I guess if it works, it works.
Why did you change state tracking to sloppy (or none?)?
ive switched it back to sloppy for the moment, if it still works set back to normal then i will move it there permanently, i set it that way in troubleshooting tho.