Strange issue with certain websites
-
Okay, not sure what I screwed up, or where exactly. New pfsense box, no firewall rules or anything, but certain websites have no route to them.
DNS resolves to the websites and the correct IP shows up (verified through a secondary connection) but the webpage fails to load.
Traceroute to the webpages fails at the pfsense box. Network is very simple, there's the comcast modem -> pfsense box -> switch -> LAN computers. If I take the pfsense box out of the equation and use the comcast modem for LAN routing (which I disable when the pfsense box in in line) pages load correctly.
Things I've already tried:
-
rebooting the pfsense box
-
-
clearing states to the ip addresses of the websites
-
changing dns servers and flushing dns cache
Some things that may have caused an error somewhere(?)
There's a 3rd interface that's not setup, maybe a route was setup at one point before I disabled it?
The comcast modem used to be the gateway, but now the pfsense router is the gateway. All the machines having issues see the pfsense box as the gateway
I had IPv6 routing enabled and then disabled itNot sure what else I'm missing here…
-
-
-
Have a look at the page below, you could try reducing your MTU
http://doc.pfsense.org/index.php/Unable_to_Access_Some_Websites
Hope that helps
-
Not an issue with the MTU.
-
To help better understand what is going on you could
-
Dump the routing table and display it here. Use the shell command netstat -rn to dump the routing table
-
Start a trace or packet capture on your WAN interface and traceroute to one of the sites that doesn't load. Post the traceroute command and response and the packet capture here. 50 packets will probably be more than enough. You might have to do the traceroute twice to get the 50 packets.
-
-
$ netstat -rn
Routing tablesInternet:
Destination Gateway Flags Refs Use Netif Expire
default 173.10.83.6 UGS 0 6618786 em1
10.1.10.0/24 link#2 UC 0 0 em0
10.1.10.3 00:1e:c9:cb:b7:3a UHLW 1 872 em0 1152
10.1.10.12 08:00:37:3f:e7:be UHLW 1 16 em0 1081
10.1.10.14 08:00:37:41:69:82 UHLW 1 15 em0 1114
10.1.10.32 00:1a:a0:97:df:e8 UHLW 1 3190 em0 1196
10.1.10.43 00:1a:a0:8a:7b:35 UHLW 1 286063 em0 1028
10.1.10.50 00:1f:f3:01:95:a8 UHLW 1 5 em0 478
10.1.10.51 f8:1e:df:d5:d6:e1 UHLW 1 15458 em0 198
10.1.10.52 00:1a:a0:97:d6:06 UHLW 1 11230 em0 529
10.1.10.56 00:1a:a0:8a:7b:1c UHLW 1 16509 em0 678
10.1.10.57 00:25:64:d6:cc:87 UHLW 1 7610 em0 1198
10.1.10.60 00:1a:a0:91:3a:fa UHLW 1 10343 em0 1193
10.1.10.159 70:1a:04:a1:be:bf UHLW 1 5323 em0 1182
10.1.10.160 00:1a:a0:95:7f:d5 UHLW 1 5084 em0 961
10.1.10.161 00:04:f2:21:1c:b5 UHLW 1 19 em0 343
10.1.10.162 00:25:64:d6:d0:3b UHLW 1 10123 em0 1185
10.1.10.163 00:1c:23:85:88:40 UHLW 1 103231 em0 973
10.1.10.164 00:1a:a0:9c:89:89 UHLW 1 2782 em0 679
10.1.10.165 00:1a:a0:95:7e:c5 UHLW 1 27473 em0 1199
10.1.10.166 00:08:5d:20:fe:2e UHLW 1 68 em0 891
10.1.10.170 00:1d:09:86:d6:8e UHLW 1 5157 em0 1191
10.1.10.172 00:21:9b:22:70:55 UHLW 1 2641315 em0 830
10.1.10.174 00:1a:a0:91:3b:50 UHLW 1 2447 em0 1198
10.1.10.179 00:21:9b:6d:e6:e7 UHLW 1 64916 em0 1196
10.1.10.180 00:04:f2:21:0d:ce UHLW 1 13 em0 596
10.1.10.183 00:21:9b:07:b9:ec UHLW 1 9120 em0 1068
10.1.10.184 00:04:f2:21:0d:f1 UHLW 1 13987 em0 782
10.1.10.185 00:25:64:03:72:80 UHLW 1 14951 em0 1193
10.1.10.186 00:25:64:03:71:66 UHLW 1 22727 em0 1196
10.1.10.187 00:25:64:d7:f0:44 UHLW 1 42470 em0 1183
10.1.10.188 00:25:64:02:b6:09 UHLW 1 34636 em0 1199
10.1.10.189 00:08:5d:10:bb:fa UHLW 1 1 em0 407
10.1.10.190 00:1a:a0:91:3b:37 UHLW 1 2187 em0 1182
10.1.10.191 00:1a:a0:9b:76:63 UHLW 1 244064 em0 723
10.1.10.192 00:21:9b:07:b9:d2 UHLW 1 59 em0 931
10.1.10.193 00:1a:a0:95:7c:f5 UHLW 1 39901 em0 1198
10.1.10.196 00:26:bb:d3:cb:25 UHLW 1 3 em0 359
10.1.10.198 00:24:e8:0e:f8:40 UHLW 1 9416 em0 1037
10.1.10.199 00:08:5d:20:fe:2f UHLW 1 24 em0 240
10.1.10.200 00:26:4a:c2:31:ee UHLW 1 2 em0 72
127.0.0.1 127.0.0.1 UH 0 0 lo0
168.0.0.0/5 link#3 UC 0 10947 em1
173.10.83.6 00:22:2d:39:e8:52 UHLW 2 4490 em1 1194Internet6:
Destination Gateway Flags Netif Expire
::1 ::1 UHL lo0
fe80::%em0/64 link#2 UC em0
fe80::21b:21ff:fe51:b70e%em0 00:1b:21:51:b7:0e UHL lo0
fe80::%em1/64 link#3 UC em1
fe80::21b:21ff:fe51:b713%em1 00:1b:21:51:b7:13 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:2::/32 link#2 UC em0
ff01:3::/32 link#3 UC em1
ff01:4::/32 ::1 UC lo0
ff02::%em0/32 link#2 UC em0
ff02::%em1/32 link#3 UC em1
ff02::%lo0/32 ::1 UC lo023:44:40.356305 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:44:43.355845 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:44:49.351261 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:01.355874 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:04.355426 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:10.356083 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:22.352719 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:25.352253 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:45:31.350794 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:01.271702 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:04.271514 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:10.280037 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:22.271794 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:25.278319 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:31.280738 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:41.695334 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:43.274387 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:43.696083 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:45.697063 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:46.274031 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:47.698178 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:49.699060 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:51.700074 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:52.277565 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:53.701191 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:55.702066 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:57.703067 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:46:59.704187 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:01.705080 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:03.706066 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:05.707196 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:07.708077 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:09.709063 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:11.710203 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:13.711075 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:15.712068 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:17.713198 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:19.714075 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:21.715081 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:23.716190 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:25.717070 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:27.718076 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:29.719259 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:31.720163 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:33.721069 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:35.722210 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:37.723108 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:39.724079 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:41.725209 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:43.726090 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:45.727078 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:47.728204 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:49.729074 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:51.730096 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:53.731199 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:55.732110 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:57.733080 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:47:59.734229 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:01.735089 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:03.736082 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:05.737224 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:07.738088 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:09.739080 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:11.740215 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:13.741081 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:15.742083 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:17.743211 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:19.744076 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:21.745115 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:23.746189 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:25.747076 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
23:48:27.748086 arp who-has www.bankofamerica.com tell 173-10-83-5-BusName-Washington.hfc.comcastbusiness.net
-
The arp trace shows the system thinks www.bankofamerica.com (171.161.161.173) is on the same subnet as em1. (I suspect this is unlikely :-) ). This suggests the network mask on em1 is not wide enough, for example the IP address of em1 is 173.10.86.x/4 when it should be 173.10.86.x/24 or 173.10.86.x/30.
Whats the output of the shell command ifconfig -a
-
$ ifconfig -a
bge0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:12:3f:37:41:1a
media: Ethernet autoselect (none)
status: no carrier
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:1b:21:51:b7:0e
inet6 fe80::21b:21ff:fe51:b70e%em0 prefixlen 64 scopeid 0x2
inet 10.1.10.254 netmask 0xffffff00 broadcast 10.1.10.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:1b:21:51:b7:13
inet6 fe80::21b:21ff:fe51:b713%em1 prefixlen 64 scopeid 0x3
inet 173.10.83.5 netmask 0xf8000000 broadcast 175.255.255.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33204</promisc></up,running></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast>
-
Yup, looks like that was the issue, somehow the subnet mask had been set to /5.