Connection timeouts within pfsense DMZ setup involving static routes *SOLVED*
-
The issue: When I am using SSH or FTP (a connection oriented type of establishment) the connection works fine until I start to transfer more than an idle SSH session amount of data and then the connection times out. If the connection is idle it basically never times out.
The network topology of the pfSense DMZ setup.
{Internet} – [WAN-pfsense fw 1-LAN] - {DMZ network} - [LAN-pfsense fw 2-WAN] - {internal network 10.0.0.0/8}
1.) The machines in the DMZ network have their gateway set to the IP of FW1's LAN interface.
2.) FW1 has a static route for the 10.0.0.0/8 network on its LAN interface pointing to the IP of FW2's LAN interface.
3.) FW2 is doing 1:1 NATs on the WAN interface for the machines within the DMZ network.
Here is a tcpdump output on FW2 while I log into a machine via a SSH session in the DMZ network and perform a cat /var/log/messages on that machine via the SSH session.
tcpdump -n host 10.0.119.34
11:46:57.259484 IP 10.0.119.34.56016 > 10.251.0.222.22: S 4095225603:4095225603(0) win 5840 <mss 2="" 278521530="" 1460,sackok,timestamp="" 0,nop,wscale="">11:46:57.259937 IP 10.251.0.222.22 > 10.0.119.34.56016: S 662446507:662446507(0) ack 4095225604 win 5792 <mss 0="" 256441081="" 1460,sackok,timestamp="" 278521530,nop,wscale="">11:46:57.267713 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 1 win 1460 <nop,nop,timestamp 256441081="" 278521531="">11:46:57.268840 IP 10.251.0.222.22 > 10.0.119.34.56016: P 1:24(23) ack 1 win 5792 <nop,nop,timestamp 256441082="" 278521531="">11:46:57.277138 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 24 win 1460 <nop,nop,timestamp 256441082="" 278521534="">11:46:57.277431 IP 10.0.119.34.56016 > 10.251.0.222.22: P 1:41(40) ack 24 win 1460 <nop,nop,timestamp 256441082="" 278521534="">11:46:57.277674 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 41 win 5792 <nop,nop,timestamp 256441083="" 278521534="">11:46:57.279687 IP 10.251.0.222.22 > 10.0.119.34.56016: P 24:632(608) ack 41 win 5792 <nop,nop,timestamp 256441083="" 278521534="">11:46:57.283871 IP 10.0.119.34.56016 > 10.251.0.222.22: P 41:753(712) ack 24 win 1460 <nop,nop,timestamp 256441083="" 278521535="">11:46:57.322831 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 753 win 7120 <nop,nop,timestamp 256441088="" 278521535="">11:46:57.325768 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 632 win 1764 <nop,nop,timestamp 256441083="" 278521546="">11:46:57.328401 IP 10.0.119.34.56016 > 10.251.0.222.22: P 753:777(24) ack 632 win 1764 <nop,nop,timestamp 256441088="" 278521547="">11:46:57.328623 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 777 win 7120 <nop,nop,timestamp 256441088="" 278521547="">11:46:57.333219 IP 10.251.0.222.22 > 10.0.119.34.56016: P 632:784(152) ack 777 win 7120 <nop,nop,timestamp 256441089="" 278521547="">11:46:57.336558 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 784 win 2068 <nop,nop,timestamp 256441089="" 278521549="">11:46:57.338474 IP 10.0.119.34.56016 > 10.251.0.222.22: P 777:921(144) ack 784 win 2068 <nop,nop,timestamp 256441089="" 278521549="">11:46:57.372834 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 921 win 7120 <nop,nop,timestamp 256441093="" 278521549="">11:46:57.376211 IP 10.251.0.222.22 > 10.0.119.34.56016: P 784:1248(464) ack 921 win 7120 <nop,nop,timestamp 256441093="" 278521549="">11:46:57.389397 IP 10.0.119.34.56016 > 10.251.0.222.22: P 921:937(16) ack 1248 win 2372 <nop,nop,timestamp 256441093="" 278521562="">11:46:57.389685 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 937 win 7120 <nop,nop,timestamp 256441094="" 278521562="">11:46:57.396293 IP 10.0.119.34.56016 > 10.251.0.222.22: P 937:985(48) ack 1248 win 2372 <nop,nop,timestamp 256441094="" 278521564="">11:46:57.396533 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 985 win 7120 <nop,nop,timestamp 256441095="" 278521564="">11:46:57.396635 IP 10.251.0.222.22 > 10.0.119.34.56016: P 1248:1296(48) ack 985 win 7120 <nop,nop,timestamp 256441095="" 278521564="">11:46:57.403338 IP 10.0.119.34.56016 > 10.251.0.222.22: P 985:1049(64) ack 1296 win 2372 <nop,nop,timestamp 256441095="" 278521565="">11:46:57.404264 IP 10.251.0.222.22 > 10.0.119.34.56016: P 1296:1376(80) ack 1049 win 7120 <nop,nop,timestamp 256441096="" 278521565="">11:46:57.410721 IP 10.0.119.34.56016 > 10.251.0.222.22: P 1049:1961(912) ack 1376 win 2372 <nop,nop,timestamp 256441096="" 278521567="">11:46:57.413583 IP 10.251.0.222.22 > 10.0.119.34.56016: P 1376:2240(864) ack 1961 win 8208 <nop,nop,timestamp 256441097="" 278521567="">11:46:57.618577 IP 10.0.119.34.56016 > 10.251.0.222.22: P 1049:1961(912) ack 1376 win 2372 <nop,nop,timestamp 256441096="" 278521619="">11:46:57.619037 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 1961 win 8208 <nop,nop,timestamp 1="" 256441117="" 278521619,nop,nop,sack="" {1049:1961}="">11:46:57.623130 IP 10.251.0.222.22 > 10.0.119.34.56016: P 1376:2240(864) ack 1961 win 8208 <nop,nop,timestamp 256441118="" 278521619="">11:46:57.636974 IP 10.0.119.34.56016 > 10.251.0.222.22: P 1961:2921(960) ack 2240 win 2804 <nop,nop,timestamp 256441118="" 278521624="">11:46:57.637419 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 2921 win 10560 <nop,nop,timestamp 256441119="" 278521624="">11:46:57.678503 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2240:2272(32) ack 2921 win 10560 <nop,nop,timestamp 256441123="" 278521624="">11:46:57.685632 IP 10.0.119.34.56016 > 10.251.0.222.22: P 2921:2985(64) ack 2272 win 2804 <nop,nop,timestamp 256441123="" 278521636="">11:46:57.685926 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 2985 win 10560 <nop,nop,timestamp 256441124="" 278521636="">11:46:57.686143 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2272:2320(48) ack 2985 win 10560 <nop,nop,timestamp 256441124="" 278521636="">11:46:57.693874 IP 10.0.119.34.56016 > 10.251.0.222.22: P 2985:3433(448) ack 2320 win 2804 <nop,nop,timestamp 256441124="" 278521638="">11:46:57.695512 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2320:2368(48) ack 3433 win 12480 <nop,nop,timestamp 256441125="" 278521638="">11:46:57.709036 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2368:2464(96) ack 3433 win 12480 <nop,nop,timestamp 256441126="" 278521638="">11:46:57.742255 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2464 win 2804 <nop,nop,timestamp 256441125="" 278521650="">11:46:57.814891 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2464:2528(64) ack 3433 win 12480 <nop,nop,timestamp 256441137="" 278521650="">11:46:57.815069 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2528:2592(64) ack 3433 win 12480 <nop,nop,timestamp 256441137="" 278521650="">11:46:57.824054 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2528 win 2804 <nop,nop,timestamp 256441137="" 278521671="">11:46:57.824074 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2592 win 2804 <nop,nop,timestamp 256441137="" 278521671="">11:46:57.982336 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3433:3481(48) ack 2592 win 2804 <nop,nop,timestamp 256441137="" 278521709="">11:46:57.982750 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2592:2640(48) ack 3481 win 12480 <nop,nop,timestamp 256441153="" 278521709="">11:46:57.986600 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2640 win 2804 <nop,nop,timestamp 256441153="" 278521711="">11:46:58.176691 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3481:3529(48) ack 2640 win 2804 <nop,nop,timestamp 256441153="" 278521757="">11:46:58.177084 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2640:2688(48) ack 3529 win 12480 <nop,nop,timestamp 256441173="" 278521757="">11:46:58.185685 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2688 win 2804 <nop,nop,timestamp 256441173="" 278521759="">11:46:58.249719 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3529:3577(48) ack 2688 win 2804 <nop,nop,timestamp 256441173="" 278521775="">11:46:58.250074 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2688:2736(48) ack 3577 win 12480 <nop,nop,timestamp 256441180="" 278521775="">11:46:58.259945 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2736 win 2804 <nop,nop,timestamp 256441180="" 278521778="">11:46:58.413644 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3577:3625(48) ack 2736 win 2804 <nop,nop,timestamp 256441180="" 278521817="">11:46:58.414041 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2736:2784(48) ack 3625 win 12480 <nop,nop,timestamp 256441197="" 278521817="">11:46:58.418541 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2784 win 2804 <nop,nop,timestamp 256441197="" 278521818="">11:46:58.551741 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3625:3673(48) ack 2784 win 2804 <nop,nop,timestamp 256441197="" 278521851="">11:46:58.552095 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2784:2832(48) ack 3673 win 12480 <nop,nop,timestamp 256441210="" 278521851="">11:46:58.559782 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2832 win 2804 <nop,nop,timestamp 256441210="" 278521853="">11:46:58.670230 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3673:3721(48) ack 2832 win 2804 <nop,nop,timestamp 256441210="" 278521881="">11:46:58.670596 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2832:2880(48) ack 3721 win 12480 <nop,nop,timestamp 256441222="" 278521881="">11:46:58.674163 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2880 win 2804 <nop,nop,timestamp 256441222="" 278521883="">11:46:58.887187 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3721:3769(48) ack 2880 win 2804 <nop,nop,timestamp 256441222="" 278521935="">11:46:58.887578 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2880:2928(48) ack 3769 win 12480 <nop,nop,timestamp 256441244="" 278521935="">11:46:58.894124 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2928 win 2804 <nop,nop,timestamp 256441244="" 278521937="">11:46:59.090617 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3769:3817(48) ack 2928 win 2804 <nop,nop,timestamp 256441244="" 278521986="">11:46:59.090985 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2928:2976(48) ack 3817 win 12480 <nop,nop,timestamp 256441264="" 278521986="">11:46:59.099378 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 2976 win 2804 <nop,nop,timestamp 256441264="" 278521988="">11:46:59.183353 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3817:3865(48) ack 2976 win 2804 <nop,nop,timestamp 256441264="" 278522008="">11:46:59.183710 IP 10.251.0.222.22 > 10.0.119.34.56016: P 2976:3024(48) ack 3865 win 12480 <nop,nop,timestamp 256441274="" 278522008="">11:46:59.194407 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3024 win 2804 <nop,nop,timestamp 256441274="" 278522012="">11:46:59.443640 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3913:3961(48) ack 3024 win 2804 <nop,nop,timestamp 256441274="" 278522075="">11:46:59.443937 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 3865 win 12480 <nop,nop,timestamp 1="" 256441300="" 278522012,nop,nop,sack="" {3913:3961}="">11:46:59.455244 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3865:3913(48) ack 3024 win 2804 <nop,nop,timestamp 256441300="" 278522078="">11:46:59.455483 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 3961 win 12480 <nop,nop,timestamp 256441301="" 278522078="">11:46:59.455632 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3024:3072(48) ack 3961 win 12480 <nop,nop,timestamp 256441301="" 278522078="">11:46:59.460094 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3072 win 2804 <nop,nop,timestamp 256441301="" 278522080="">11:46:59.528461 IP 10.0.119.34.56016 > 10.251.0.222.22: P 3961:4009(48) ack 3072 win 2804 <nop,nop,timestamp 256441301="" 278522097="">11:46:59.528817 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3072:3120(48) ack 4009 win 12480 <nop,nop,timestamp 256441308="" 278522097="">11:46:59.534828 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3120 win 2804 <nop,nop,timestamp 256441308="" 278522098="">11:46:59.642178 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4009:4057(48) ack 3120 win 2804 <nop,nop,timestamp 256441308="" 278522125="">11:46:59.642551 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3120:3168(48) ack 4057 win 12480 <nop,nop,timestamp 256441319="" 278522125="">11:46:59.647737 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3168 win 2804 <nop,nop,timestamp 256441319="" 278522126="">11:46:59.741519 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4057:4105(48) ack 3168 win 2804 <nop,nop,timestamp 256441319="" 278522150="">11:46:59.741913 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3168:3216(48) ack 4105 win 12480 <nop,nop,timestamp 256441329="" 278522150="">11:46:59.748413 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3216 win 2804 <nop,nop,timestamp 256441329="" 278522152="">11:46:59.812647 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4105:4153(48) ack 3216 win 2804 <nop,nop,timestamp 256441329="" 278522168="">11:46:59.813023 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3216:3264(48) ack 4153 win 12480 <nop,nop,timestamp 256441337="" 278522168="">11:46:59.820264 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3264 win 2804 <nop,nop,timestamp 256441337="" 278522170="">11:46:59.992528 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4153:4201(48) ack 3264 win 2804 <nop,nop,timestamp 256441337="" 278522213="">11:46:59.992914 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3264:3312(48) ack 4201 win 12480 <nop,nop,timestamp 256441355="" 278522213="">11:47:00.000034 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3312 win 2804 <nop,nop,timestamp 256441355="" 278522215="">11:47:00.164105 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4201:4249(48) ack 3312 win 2804 <nop,nop,timestamp 256441355="" 278522255="">11:47:00.164627 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3312:3360(48) ack 4249 win 12480 <nop,nop,timestamp 256441372="" 278522255="">11:47:00.173214 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3360 win 2804 <nop,nop,timestamp 256441372="" 278522257="">11:47:00.699328 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4249:4297(48) ack 3360 win 2804 <nop,nop,timestamp 256441372="" 278522389="">11:47:00.700171 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3360:3408(48) ack 4297 win 12480 <nop,nop,timestamp 256441425="" 278522389="">11:47:00.707259 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3408 win 2804 <nop,nop,timestamp 256441425="" 278522391="">11:47:02.058449 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4297:4345(48) ack 3408 win 2804 <nop,nop,timestamp 256441425="" 278522729="">11:47:02.059037 IP 10.251.0.222.22 > 10.0.119.34.56016: P 3408:3456(48) ack 4345 win 12480 <nop,nop,timestamp 256441561="" 278522729="">11:47:02.063132 IP 10.251.0.222.22 > 10.0.119.34.56016: . 3456:4904(1448) ack 4345 win 12480 <nop,nop,timestamp 256441561="" 278522729="">11:47:02.064892 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 3456 win 2804 <nop,nop,timestamp 256441561="" 278522731="">11:47:02.065667 IP 10.251.0.222.22 > 10.0.119.34.56016: . 4904:6352(1448) ack 4345 win 12480 <nop,nop,timestamp 256441562="" 278522731="">11:47:02.065790 IP 10.251.0.222.22 > 10.0.119.34.56016: . 6352:7800(1448) ack 4345 win 12480 <nop,nop,timestamp 256441562="" 278522731="">11:47:02.070897 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 4904 win 3528 <nop,nop,timestamp 256441561="" 278522732="">11:47:02.071671 IP 10.251.0.222.22 > 10.0.119.34.56016: . 7800:9248(1448) ack 4345 win 12480 <nop,nop,timestamp 256441562="" 278522732="">11:47:02.074441 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 6352 win 4252 <nop,nop,timestamp 256441562="" 278522733="">11:47:02.074601 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 7800 win 4976 <nop,nop,timestamp 256441562="" 278522733="">11:47:02.075198 IP 10.251.0.222.22 > 10.0.119.34.56016: . 9248:10696(1448) ack 4345 win 12480 <nop,nop,timestamp 256441563="" 278522733="">11:47:02.075335 IP 10.251.0.222.22 > 10.0.119.34.56016: . 10696:12144(1448) ack 4345 win 12480 <nop,nop,timestamp 256441563="" 278522733="">11:47:02.080178 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 9248 win 5700 <nop,nop,timestamp 256441562="" 278522735="">11:47:02.080922 IP 10.251.0.222.22 > 10.0.119.34.56016: . 12144:13592(1448) ack 4345 win 12480 <nop,nop,timestamp 256441563="" 278522735="">11:47:02.081043 IP 10.251.0.222.22 > 10.0.119.34.56016: . 13592:15040(1448) ack 4345 win 12480 <nop,nop,timestamp 256441563="" 278522735="">11:47:02.082768 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 10696 win 6424 <nop,nop,timestamp 256441563="" 278522735="">11:47:02.083505 IP 10.251.0.222.22 > 10.0.119.34.56016: . 15040:16488(1448) ack 4345 win 12480 <nop,nop,timestamp 256441564="" 278522735="">11:47:02.083574 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 12144 win 7148 <nop,nop,timestamp 256441563="" 278522735="">11:47:02.084361 IP 10.251.0.222.22 > 10.0.119.34.56016: P 16488:17936(1448) ack 4345 win 12480 <nop,nop,timestamp 256441564="" 278522735="">11:47:02.091317 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 256441563="" 278522737="">11:47:02.092089 IP 10.251.0.222.22 > 10.0.119.34.56016: . 17936:19384(1448) ack 4345 win 12480 <nop,nop,timestamp 256441564="" 278522737="">11:47:02.092814 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 1="" 278522738="" 256441563,nop,nop,sack="" {15040:16488}="">11:47:02.093558 IP 10.251.0.222.22 > 10.0.119.34.56016: . 19384:20832(1448) ack 4345 win 12480 <nop,nop,timestamp 256441565="" 278522738="">11:47:02.094231 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 1="" 278522738="" 256441563,nop,nop,sack="" {15040:17936}="">11:47:02.094966 IP 10.251.0.222.22 > 10.0.119.34.56016: . 20832:22280(1448) ack 4345 win 12480 <nop,nop,timestamp 256441565="" 278522738="">11:47:02.097720 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 1="" 278522739="" 256441563,nop,nop,sack="" {15040:19384}="">11:47:02.098465 IP 10.251.0.222.22 > 10.0.119.34.56016: . 13592:15040(1448) ack 4345 win 12480 <nop,nop,timestamp 256441565="" 278522739="">11:47:02.099383 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 1="" 278522739="" 256441563,nop,nop,sack="" {15040:20832}="">11:47:02.101885 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 13592 win 7872 <nop,nop,timestamp 1="" 278522740="" 256441563,nop,nop,sack="" {15040:22280}="">11:47:02.102626 IP 10.251.0.222.22 > 10.0.119.34.56016: P 22280:23728(1448) ack 4345 win 12480 <nop,nop,timestamp 256441565="" 278522740="">11:47:02.106805 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 22280 win 8596 <nop,nop,timestamp 256441565="" 278522741="">11:47:02.107560 IP 10.251.0.222.22 > 10.0.119.34.56016: . 23728:25176(1448) ack 4345 win 12480 <nop,nop,timestamp 256441566="" 278522741="">11:47:02.109888 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 23728 win 9320 <nop,nop,timestamp 256441565="" 278522742="">11:47:02.110643 IP 10.251.0.222.22 > 10.0.119.34.56016: . 25176:26624(1448) ack 4345 win 12480 <nop,nop,timestamp 256441566="" 278522742="">11:47:02.110764 IP 10.251.0.222.22 > 10.0.119.34.56016: . 26624:28072(1448) ack 4345 win 12480 <nop,nop,timestamp 256441566="" 278522742="">11:47:02.112222 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 25176 win 10044 <nop,nop,timestamp 256441566="" 278522742="">11:47:02.112975 IP 10.251.0.222.22 > 10.0.119.34.56016: . 28072:29520(1448) ack 4345 win 12480 <nop,nop,timestamp 256441566="" 278522742="">11:47:02.115253 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 26624 win 10768 <nop,nop,timestamp 256441566="" 278522743="">11:47:02.115420 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 28072 win 11492 <nop,nop,timestamp 256441566="" 278522743="">11:47:02.116032 IP 10.251.0.222.22 > 10.0.119.34.56016: . 29520:30968(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522743="">11:47:02.116160 IP 10.251.0.222.22 > 10.0.119.34.56016: . 30968:32416(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522743="">11:47:02.116279 IP 10.251.0.222.22 > 10.0.119.34.56016: . 32416:33864(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522743="">11:47:02.117200 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 29520 win 12216 <nop,nop,timestamp 256441566="" 278522744="">11:47:02.117934 IP 10.251.0.222.22 > 10.0.119.34.56016: P 33864:35312(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522744="">11:47:02.122278 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 30968 win 12940 <nop,nop,timestamp 256441567="" 278522745="">11:47:02.122510 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 32416 win 13664 <nop,nop,timestamp 256441567="" 278522745="">11:47:02.122946 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 33864 win 14388 <nop,nop,timestamp 256441567="" 278522745="">11:47:02.123019 IP 10.251.0.222.22 > 10.0.119.34.56016: . 35312:36760(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522745="">11:47:02.123246 IP 10.251.0.222.22 > 10.0.119.34.56016: . 36760:38208(1448) ack 4345 win 12480 <nop,nop,timestamp 256441567="" 278522745="">11:47:02.123679 IP 10.251.0.222.22 > 10.0.119.34.56016: . 38208:39656(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522745="">11:47:02.124594 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 35312 win 15043 <nop,nop,timestamp 256441567="" 278522746="">11:47:02.125343 IP 10.251.0.222.22 > 10.0.119.34.56016: P 39656:41104(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522746="">11:47:02.125466 IP 10.251.0.222.22 > 10.0.119.34.56016: . 41104:42552(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522746="">11:47:02.129859 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 36760 win 14708 <nop,nop,timestamp 256441567="" 278522747="">11:47:02.130463 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 38208 win 14373 <nop,nop,timestamp 256441567="" 278522747="">11:47:02.130596 IP 10.251.0.222.22 > 10.0.119.34.56016: . 42552:44000(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522747="">11:47:02.131199 IP 10.251.0.222.22 > 10.0.119.34.56016: . 44000:45448(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522747="">11:47:02.131237 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 39656 win 14037 <nop,nop,timestamp 256441568="" 278522747="">11:47:02.132009 IP 10.251.0.222.22 > 10.0.119.34.56016: P 45448:46896(1448) ack 4345 win 12480 <nop,nop,timestamp 256441568="" 278522747="">11:47:02.132968 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 41104 win 13702 <nop,nop,timestamp 256441568="" 278522748="">11:47:02.133392 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 42552 win 13367 <nop,nop,timestamp 256441568="" 278522748="">11:47:02.133734 IP 10.251.0.222.22 > 10.0.119.34.56016: . 46896:48344(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522748="">11:47:02.134122 IP 10.251.0.222.22 > 10.0.119.34.56016: . 48344:49792(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522748="">11:47:02.138355 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 44000 win 13032 <nop,nop,timestamp 256441568="" 278522749="">11:47:02.138671 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 45448 win 12696 <nop,nop,timestamp 256441568="" 278522749="">11:47:02.138906 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 46896 win 12361 <nop,nop,timestamp 256441568="" 278522749="">11:47:02.139091 IP 10.251.0.222.22 > 10.0.119.34.56016: . 49792:51240(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522749="">11:47:02.139224 IP 10.251.0.222.22 > 10.0.119.34.56016: P 51240:52688(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522749="">11:47:02.139423 IP 10.251.0.222.22 > 10.0.119.34.56016: . 52688:54136(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522749="">11:47:02.139636 IP 10.251.0.222.22 > 10.0.119.34.56016: . 54136:55584(1448) ack 4345 win 12480 <nop,nop,timestamp 256441569="" 278522749="">11:47:02.142962 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 49792 win 13367 <nop,nop,timestamp 256441569="" 278522750="">11:47:02.143638 IP 10.0.119.34.56016 > 10.251.0.222.22: P 4345:4393(48) ack 51240 win 15112 <nop,nop,timestamp 256441569="" 278522750="">11:47:02.143701 IP 10.251.0.222.22 > 10.0.119.34.56016: . 55584:57032(1448) ack 4345 win 12480 <nop,nop,timestamp 256441570="" 278522750="">11:47:02.143823 IP 10.251.0.222.22 > 10.0.119.34.56016: . 57032:58480(1448) ack 4345 win 12480 <nop,nop,timestamp 256441570="" 278522750="">11:47:02.144199 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 54136 win 15714 <nop,nop,timestamp 256441569="" 278522751="">11:47:02.144389 IP 10.251.0.222.22 > 10.0.119.34.56016: . 58480:59928(1448) ack 4393 win 12480 <nop,nop,timestamp 256441570="" 278522750="">11:47:02.144989 IP 10.251.0.222.22 > 10.0.119.34.56016: . 59928:61376(1448) ack 4393 win 12480 <nop,nop,timestamp 256441570="" 278522751="">11:47:02.145112 IP 10.251.0.222.22 > 10.0.119.34.56016: . 61376:62824(1448) ack 4393 win 12480 <nop,nop,timestamp 256441570="" 278522751="">11:47:02.149156 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 59928 win 15836 <nop,nop,timestamp 256441569="" 278522752="">11:47:02.149659 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 62824 win 16022 <nop,nop,timestamp 256441570="" 278522752="">11:47:02.149948 IP 10.251.0.222.22 > 10.0.119.34.56016: P 62824:64272(1448) ack 4393 win 12480 <nop,nop,timestamp 256441570="" 278522752="">11:47:02.150071 IP 10.251.0.222.22 > 10.0.119.34.56016: . 64272:65720(1448) ack 4393 win 12480 <nop,nop,timestamp 256441570="" 278522752="">11:47:02.191804 IP 10.0.119.34.56016 > 10.251.0.222.22: . ack 65720 win 16022 <nop,nop,timestamp 256441570="" 278522763="">^
|___ This is when I stop seeing text scroll on my SSH session window.** I make some space on my terminal when the ssh session freezes then I exit the term window and get the below.
11:47:06.193367 IP 10.0.119.34.56016 > 10.251.0.222.22: F 4393:4393(0) ack 65720 win 16022 <nop,nop,timestamp 256441570="" 278523762="">11:47:06.232836 IP 10.251.0.222.22 > 10.0.119.34.56016: . ack 4394 win 12480 <nop,nop,timestamp 256441979="" 278523762="">** Also note: When my ssh session does freeze, it becomes 'unfrozen' after 30 to 60 seconds afterwards.
** On this particular machine within the DMZ that I logged in via a SSH session I even made sure to put these TCP Window Size parameters in for sysctl.
net.ipv4.tcp_wmem = 4096 16384 131072
net.ipv4.tcp_rmem = 4096 87380 174760Does anyone have any idea what is causing this issue I am experiencing?
Solution:
I originally thought it was one of the firewall's state tables but wasn't sure which one. I executed a pftctl -x misc on both firewalls and found the offending one. The on FW1 I enabled the 'Bypass firewall rules for traffic on the same interface" and I did not receive a time out while performing cat /var/log/message. I think I solved my problem!</nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></mss></mss>