LAN -> DMZ traffic slow responses

tgeorge06

From the LAN connecting to the server inside the DMZ results in slow responses in everything except webmin.

VNC hangs for 20-30 seconds then pops up the password box and everything works from there.

MySQL Workbench hangs for a good minute… then connects and is fine after that.

Loading websites from the webserver also hangs for 10-20 seconds, then loads finally.

Setup is pretty basic, 3 NIC's on the pfsense box... LAN, WAN, DMZ... the server is connected via crossover cable direct to DMZ on the pfsense box

I have a Proxy ARP setup for the webservers public IP with a port forward all traffic from wan on port 80 to the servers local ip on port 80.

I had an issue loading the websites on the webserver until i setup a port forward all traffic from LAN on port 80 to the servers local ip on port 80.

WAN.jpg_thumb

LAN.jpg_thumb

DMZ.jpg_thumb

NAT.jpg_thumb

wallabybob

Who is providing name service to the LAN clients when accessing the DMZ server?

If you are using DNS forwarding on pfSense you might getter results if you add an override entry for you DMZ server. (Then the LAN clients will get the name to IP address translation from pfSense rather than from some system out on the Internet.)

jimp

I agree with wallabybob, it sounds like a DNS resolution problem if it pauses for up to 45 seconds and then continues fine after that.

Ensure that not only all DNS servers are valid, but also aren't being blocked by firewall rules, etc. It could easily be a case of a misconfigured DNS setup, if you provide some more information about your setup in that regard we could advise how best to fix it.

Let us know what DNS servers you have under System > General, whether or not your client PCs are using the DNS forwarder on pfSense as their DNS server. And if they are not using the forwarder, what are they using, such as an AD Domain Controller (and if a DC, what are the forwarder IPs in there set to?)

tgeorge06

Well since tornados touched down here we dont have power im posting from my iphone.

What i do know is the dns im using is opendns, so they are 208.67.222.222 and 208.67.220.220. Omce we get power and i get everything running again i will try a few things. Thanks for your guys help i appreciate it.

tgeorge06

Everything is back online so i'm messing with it again..

What exactly or how exactly should i be adding rules into the firewall to let the dns go through if that is even the problem?

all lan machines are using the DNS Forwarder, but the server inside the DMZ is not, I have it set to manual specify 208.67.222.222 and 208.67.220.220. The LAN machines are using the same dns servers, just dhcp instead of manual.

I think that I should also note that i connect with vnc, mysql and webmin via local ip..192.168.5.10. I swapped the nameservers on the server to 192.168.1.1 instead of naming them manually to 208.67.222.222 and 208.67.220.220

jimp

So even now with all of the servers using the DNS forwarder, the connections still are slow to initialize?

We might need to see some packet captures on the LAN and DMZ sides to help any more.

tgeorge06

This is using pfsense packet capture and trying to connect to one of the websites the web server is hosting. When it finally connected and loaded I stopped the packet capture.

–------LAN INTERFACE--------

13:49:57.687087 IP 192.168.1.1.443 > 192.168.1.191.4443: tcp 1460
13:49:57.687643 IP 192.168.1.1.443 > 192.168.1.191.4443: tcp 1460
13:49:57.688947 IP 192.168.1.191.4443 > 192.168.1.1.443: tcp 0
13:49:57.890062 IP 192.168.1.191.4443 > 192.168.1.1.443: tcp 0
13:49:57.890127 IP 192.168.1.1.443 > 192.168.1.191.4443: tcp 726
13:49:57.899032 IP 192.168.1.191.4409 > 209.85.225.139.80: tcp 942
13:49:57.899092 IP 192.168.1.191.4409 > 209.85.225.139.80: tcp 403
13:49:57.963546 IP 209.85.225.139.80 > 192.168.1.191.4409: tcp 0
13:49:57.963863 IP 209.85.225.139.80 > 192.168.1.191.4409: tcp 303
13:49:58.090095 IP 192.168.1.191.4443 > 192.168.1.1.443: tcp 0
13:49:58.170069 IP 192.168.1.191.4409 > 209.85.225.139.80: tcp 0
13:49:58.731444 IP 192.168.1.191.4444 > 74.219.187.205.80: tcp 0
13:49:58.731700 IP 74.219.187.205.80 > 192.168.1.191.4444: tcp 0
13:49:58.732465 IP 192.168.1.191.4444 > 74.219.187.205.80: tcp 0
13:49:58.732933 IP 192.168.1.191.4444 > 74.219.187.205.80: tcp 418
13:49:58.733135 IP 74.219.187.205.80 > 192.168.1.191.4444: tcp 0
13:49:58.823646 IP6 fe80::b8c2:2c2f:ea99:fdb7.50570 > ff02::c.1900: UDP, length 146
13:50:00.145209 IP 192.168.1.50.1972 > 64.233.169.102.80: tcp 678
13:50:00.176253 IP 64.233.169.102.80 > 192.168.1.50.1972: tcp 0
13:50:00.272402 IP 64.233.169.102.80 > 192.168.1.50.1972: tcp 400
13:50:00.364520 IP 192.168.1.111.53823 > 209.85.225.95.443: tcp 0
13:50:00.364805 IP 192.168.1.111.53736 > 74.125.95.97.443: tcp 0
13:50:00.409953 IP 209.85.225.95.443 > 192.168.1.111.53823: tcp 0
13:50:00.410570 IP 74.125.95.97.443 > 192.168.1.111.53736: tcp 0
13:50:00.463188 IP 192.168.1.50.1972 > 64.233.169.102.80: tcp 0
13:50:00.538988 IP 192.168.1.124.50844 > 174.129.224.43.80: tcp 0
13:50:00.582227 IP 174.129.224.43.80 > 192.168.1.124.50844: tcp 0
13:50:00.583132 IP 192.168.1.124.50844 > 174.129.224.43.80: tcp 0
13:50:00.583837 IP 192.168.1.124.50844 > 174.129.224.43.80: tcp 651
13:50:00.658377 IP 174.129.224.43.80 > 192.168.1.124.50844: tcp 0
13:50:00.712526 IP 174.129.224.43.80 > 192.168.1.124.50844: tcp 417
13:50:00.712737 IP 174.129.224.43.80 > 192.168.1.124.50844: tcp 0
13:50:00.713134 IP 192.168.1.124.50844 > 174.129.224.43.80: tcp 0
13:50:00.714416 IP 192.168.1.124.50844 > 174.129.224.43.80: tcp 0
13:50:00.755800 IP 174.129.224.43.80 > 192.168.1.124.50844: tcp 0
13:50:01.823654 IP6 fe80::b8c2:2c2f:ea99:fdb7.50570 > ff02::c.1900: UDP, length 146
13:50:02.088462 IP 192.168.1.50.1972 > 64.233.169.102.80: tcp 683
13:50:02.120147 IP 64.233.169.102.80 > 192.168.1.50.1972: tcp 0
13:50:02.156225 IP 192.168.1.50.1972 > 64.233.169.102.80: tcp 0
13:50:02.252125 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.283191 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 0
13:50:02.283311 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.298187 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 955
13:50:02.331301 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 0
13:50:02.388361 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.388733 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.388996 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1004
13:50:02.389189 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.389333 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.389770 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.389891 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.390045 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 134
13:50:02.390339 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.390403 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 563
13:50:02.560831 IP 192.168.1.50.1974 > 72.14.209.99.80: tcp 0
13:50:02.575461 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.611280 IP 72.14.209.99.80 > 192.168.1.50.1974: tcp 0
13:50:02.611394 IP 192.168.1.50.1974 > 72.14.209.99.80: tcp 0
13:50:02.743036 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.743397 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1406
13:50:02.743630 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 992
13:50:02.743842 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.758629 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 1430
13:50:02.758801 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 243
13:50:02.759071 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.778714 IP 192.168.1.50.1974 > 72.14.209.99.80: tcp 796
13:50:02.820667 IP 72.14.209.99.80 > 192.168.1.50.1974: tcp 0
13:50:02.877216 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0
13:50:02.885438 IP 72.14.209.99.80 > 192.168.1.50.1974: tcp 1430
13:50:02.885593 IP 72.14.209.99.80 > 192.168.1.50.1974: tcp 155
13:50:02.885901 IP 192.168.1.50.1974 > 72.14.209.99.80: tcp 0
13:50:03.003274 IP 192.168.1.50.49971 > 65.24.0.168.53: UDP, length 43
13:50:03.021883 IP 65.24.0.168.53 > 192.168.1.50.49971: UDP, length 106
13:50:03.206294 IP 192.168.1.124.49782 > 208.111.158.53.27017: UDP, length 84
13:50:03.255437 IP 192.168.1.191.54924 > 72.165.61.187.27017: UDP, length 84
13:50:03.616477 IP 208.111.158.53.27017 > 192.168.1.124.49782: UDP, length 36
13:50:03.695374 IP 192.168.1.1.443 > 192.168.1.191.4443: tcp 37
13:50:03.695428 IP 192.168.1.1.443 > 192.168.1.191.4443: tcp 0
13:50:03.696558 IP 192.168.1.191.4443 > 192.168.1.1.443: tcp 0
13:50:03.709809 IP 72.165.61.187.27017 > 192.168.1.191.54924: UDP, length 36
13:50:04.258402 IP 192.168.1.50.50832 > 65.24.0.168.53: UDP, length 32
13:50:04.263069 IP 192.168.1.50.57753 > 65.24.0.168.53: UDP, length 32
13:50:04.267832 IP 192.168.1.50.60394 > 65.24.0.168.53: UDP, length 36
13:50:04.280198 IP 65.24.0.168.53 > 192.168.1.50.57753: UDP, length 62
13:50:04.285885 IP 65.24.0.168.53 > 192.168.1.50.60394: UDP, length 77
13:50:04.300912 IP 192.168.1.50.60364 > 65.24.0.168.53: UDP, length 38
13:50:04.308955 IP 192.168.1.50.64271 > 65.24.0.168.53: UDP, length 42
13:50:04.326221 IP 65.24.0.168.53 > 192.168.1.50.64271: UDP, length 58
13:50:04.345173 IP 192.168.1.50.64853 > 65.24.0.168.53: UDP, length 40
13:50:04.358281 IP 65.24.0.168.53 > 192.168.1.50.60364: UDP, length 68
13:50:04.411028 IP 192.168.1.50.60363 > 65.24.0.168.53: UDP, length 33
13:50:04.425973 IP 65.24.0.168.53 > 192.168.1.50.64853: UDP, length 70
13:50:04.430248 IP 65.24.0.168.53 > 192.168.1.50.60363: UDP, length 109
13:50:04.433650 IP 65.24.0.168.53 > 192.168.1.50.50832: UDP, length 62
13:50:04.438424 IP 192.168.1.50.64272 > 65.24.0.168.53: UDP, length 32
13:50:04.501975 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 1055
13:50:04.534217 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 0
13:50:04.573335 IP 65.24.0.168.53 > 192.168.1.50.64272: UDP, length 48
13:50:04.611655 IP 72.14.204.147.80 > 192.168.1.50.1973: tcp 215
13:50:04.788321 IP 192.168.1.50.1973 > 72.14.204.147.80: tcp 0

–------DMZ INTERFACE----------

13:53:11.442769 IP 66.69.99.191.59716 > 192.168.5.10.80: tcp 0
13:53:11.645237 IP 66.69.99.191.53432 > 192.168.5.10.80: tcp 0
13:53:12.802054 IP 192.168.1.191.4489 > 192.168.5.10.80: tcp 0
13:53:12.802190 IP 192.168.5.10.80 > 192.168.1.191.4489: tcp 0
13:53:12.803004 IP 192.168.1.191.4489 > 192.168.5.10.80: tcp 0
13:53:12.803384 IP 192.168.1.191.4489 > 192.168.5.10.80: tcp 415
13:53:12.803564 IP 192.168.5.10.80 > 192.168.1.191.4489: tcp 0
13:53:12.803887 IP 192.168.5.10.34735 > 192.168.1.1.53: UDP, length 44
13:53:13.024766 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 0
13:53:13.024907 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 0
13:53:13.129113 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 0
13:53:13.129581 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 623
13:53:13.129779 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 0
13:53:13.131175 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 1460
13:53:13.131298 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 1460
13:53:13.131421 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 1460
13:53:13.131439 IP 192.168.5.10.41048 > 192.168.1.1.53: UDP, length 41
13:53:13.247603 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 0
13:53:13.247765 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 390
13:53:13.256986 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.257117 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 0
13:53:13.257403 IP 97.93.0.35.58799 > 192.168.5.10.80: tcp 0
13:53:13.257526 IP 192.168.5.10.80 > 97.93.0.35.58799: tcp 0
13:53:13.357651 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 0
13:53:13.363600 IP 97.93.0.35.58798 > 192.168.5.10.80: tcp 380
13:53:13.369631 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.370002 IP 97.93.0.35.58799 > 192.168.5.10.80: tcp 0
13:53:13.370358 IP 97.93.0.35.58799 > 192.168.5.10.80: tcp 391
13:53:13.370519 IP 192.168.5.10.80 > 97.93.0.35.58799: tcp 0
13:53:13.370654 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 381
13:53:13.370829 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 0
13:53:13.371318 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.371440 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.371563 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.371581 IP 192.168.5.10.54023 > 192.168.1.1.53: UDP, length 41
13:53:13.372236 IP 192.168.5.10.80 > 97.93.0.35.58799: tcp 1460
13:53:13.372290 IP 192.168.5.10.80 > 97.93.0.35.58799: tcp 581
13:53:13.372330 IP 192.168.5.10.49112 > 192.168.1.1.53: UDP, length 41
13:53:13.399396 IP 192.168.5.10.80 > 97.93.0.35.58798: tcp 0
13:53:13.477894 IP 206.196.148.93.39975 > 192.168.5.10.80: tcp 0
13:53:13.478246 IP 206.196.148.93.39973 > 192.168.5.10.80: tcp 0
13:53:13.484706 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.484967 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.485089 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.485212 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.513967 IP 97.93.0.35.58799 > 192.168.5.10.80: tcp 0
13:53:13.597434 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.597706 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.597828 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.597951 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.604726 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.604966 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.605090 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.605213 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1460
13:53:13.628437 IP 97.93.0.35.58799 > 192.168.5.10.80: tcp 384
13:53:13.631508 IP 97.93.0.35.58805 > 192.168.5.10.80: tcp 0
13:53:13.631652 IP 192.168.5.10.80 > 97.93.0.35.58805: tcp 0
13:53:13.659373 IP 192.168.5.10.80 > 97.93.0.35.58799: tcp 0
13:53:13.712618 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.712856 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 1386
13:53:13.723147 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.726748 IP 97.93.0.35.58806 > 192.168.5.10.80: tcp 0
13:53:13.726891 IP 192.168.5.10.80 > 97.93.0.35.58806: tcp 0
13:53:13.727163 IP 97.93.0.35.58807 > 192.168.5.10.80: tcp 0
13:53:13.727314 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 0
13:53:13.730796 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 0
13:53:13.739714 IP 97.93.0.35.58805 > 192.168.5.10.80: tcp 0
13:53:13.745434 IP 97.93.0.35.58805 > 192.168.5.10.80: tcp 394
13:53:13.745638 IP 192.168.5.10.80 > 97.93.0.35.58805: tcp 0
13:53:13.746053 IP 192.168.5.10.80 > 97.93.0.35.58805: tcp 737
13:53:13.746077 IP 192.168.5.10.44224 > 192.168.1.1.53: UDP, length 41
13:53:13.827182 IP 97.93.0.35.58800 > 192.168.5.10.80: tcp 384
13:53:13.836259 IP 97.93.0.35.58806 > 192.168.5.10.80: tcp 0
13:53:13.841563 IP 97.93.0.35.58807 > 192.168.5.10.80: tcp 0
13:53:13.841986 IP 97.93.0.35.58807 > 192.168.5.10.80: tcp 385
13:53:13.842149 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 0
13:53:13.842290 IP 97.93.0.35.58806 > 192.168.5.10.80: tcp 382
13:53:13.842394 IP 192.168.5.10.80 > 97.93.0.35.58806: tcp 0
13:53:13.842559 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 1460
13:53:13.842682 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 1460
13:53:13.842732 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 498
13:53:13.842748 IP 192.168.5.10.43898 > 192.168.1.1.53: UDP, length 41
13:53:13.842931 IP 192.168.5.10.80 > 97.93.0.35.58806: tcp 1460
13:53:13.843015 IP 192.168.5.10.80 > 97.93.0.35.58806: tcp 981
13:53:13.843033 IP 192.168.5.10.55249 > 192.168.1.1.53: UDP, length 41
13:53:13.857968 IP 97.93.0.35.58805 > 192.168.5.10.80: tcp 387
13:53:13.859913 IP 192.168.5.10.80 > 97.93.0.35.58800: tcp 0
13:53:13.889907 IP 192.168.5.10.80 > 97.93.0.35.58805: tcp 0
13:53:13.966598 IP 97.93.0.35.58807 > 192.168.5.10.80: tcp 0
13:53:13.974100 IP 97.93.0.35.58806 > 192.168.5.10.80: tcp 0
13:53:13.974551 IP 97.93.0.35.58807 > 192.168.5.10.80: tcp 385
13:53:13.974825 IP 97.93.0.35.58806 > 192.168.5.10.80: tcp 390
13:53:14.009379 IP 192.168.5.10.80 > 97.93.0.35.58807: tcp 0
13:53:14.009395 IP 192.168.5.10.80 > 97.93.0.35.58806: tcp 0
13:53:14.646082 IP 192.168.5.10.80 > 152.216.11.5.53778: tcp 1380
13:53:14.646198 IP 192.168.5.10.80 > 152.216.11.5.53778: tcp 1380
13:53:14.646313 IP 192.168.5.10.80 > 152.216.11.5.53778: tcp 1380
13:53:14.734527 IP 152.216.11.5.53778 > 192.168.5.10.80: tcp 0
13:53:14.734780 IP 192.168.5.10.80 > 152.216.11.5.53778: tcp 1380
13:53:14.734898 IP 192.168.5.10.80 > 152.216.11.5.53778: tcp 1380

tgeorge06

Ok I went to the extreme and made a rule on each interface on the firewall rules… allow all on all interfaces.

Worked!

So i took the allow all off of the WAN and LAN and it's still working good.

Took it off of the DMZ and it is slowed down again.

What firewall rules am i missing on the DMZ rules? Screenshot posted on the first post.

wallabybob

The DMZ packet capture is not much help as an example of the problem because it spans only 13:53:11.442769 to 13:53:14.734898, less than 1.5 seconds.

I'm not sure that configuring your DMZ server directly to use OpenDNS is a great idea in this context. Some of the services on the server may invoke DNS to get a hostname for the IP address connecting to them. For a "local" (on your LAN) IP address OpenDNS isn't going to be able to help. I suggest you let your DMZ server get its DNS information from DHCP (then it should use pfSense's DNS forwarder through the IP address of the pfSense DMZ NIC).

After you took your "special" rule off the DMZ interface did you look at the firewall log to see if anything on the DMZ was reported?

tgeorge06

I've already switched it to let the DMZ machines grab the DNS from the pfsense dns forwarder.

as for the firewall rule, I didn't get a chance to look yesterday, I will be doing some more playing with firewall rules today and I will post back what I find for anybody else having the same issue as I was.

–--

Easy enough, compared firewall logs with and without the allow all rule to DMZ network and found that port 53 was being blocked.

Since port 53 is the DNS port using tcp/udp, i just created a rule allowing the DMZ to use port 53 tcp/udp to the DNS Forwarder.

TCP/UDP
Source: 192.168.5.10(Only DMZ Machine) Port: Any(because i seen it using multiple higher ports)
Destination: 192.168.1.1(DNS Forwarder) Port: 53