Strange connectivity problem with 1.2.3-RC2 embedded

  • I'm running the above version on a miniwall. It works fine except for one thing: I can't access the Google cache servers through the router. I can load the Google search page in a browser, but if I run a query and then click on a link to a cached page, the connection attempt just times out. I can't ping or traceroute the servers, either. This has been going on since I installed the router about a week ago. The firewall log shows nothing to or from Google being blocked. If I bypass the router and connect directly to my cable modem, with the same external IP address, there's no problem. It's definitely the router. My old router was a ZyWall, and it didn't have this behavior.

    I have no problem with any other website; just the nameless Google cache servers. I have no idea how to go about solving this. Thanks for any suggestions.

  • Are all the PCs behind pfsense having the same issue?
    Are you running any addons?

  • @tommyboy180:

    Are all the PCs behind pfsense having the same issue?
    Are you running any addons?

    Yes, and no.

  • @tommyboy180:

    Can you get to any of those links?

    No, neither.

  • I'm really curious,
    Can you get to

  • @tommyboy180:

    I'm really curious,
    Can you get to

    Times out.

  • Can you get to any address without DNS?
    such as

  • Yes, e.g. == Yours too.

  • Its almost like your pfsense box just doen't like the google cache server address.

    What do you have for firewall rules? Can we rule those out? I am at a loss of what it can be.

  • @tommyboy180:

    What do you have for firewall rules? Can we rule those out? I am at a loss of what it can be.

    Besides the default rule, I block only incoming multicast IGMP which comes from my ISP. I hesitated to post this problem here because it makes no sense, but it is real. I assumed Google itself was blocking me for some reason, but it isn't.

  • Yeah. We can rule out Google and we can rule out your ISP. I have no clue, I am waiting to see if a Hero member can shed some light.

    Your right it doesn't make sense.

  • Maybe do a tcpdump on your wan side and see if any of the packets are making it out or back. Comparing with what you see on the lan side tcpdump may shed more light.

  • On the WAN, nothing. On the LAN, a lot of this:

    arp who-has tell
    IP myhost.64801 > S 2290598658:2290598658(0) win 65535 

    This is interesting because is not my LAN. I'm on is the address pool of one of my two OpenVPN tunnels, which are bridged to the LAN using instructions I found around here somewhere. So the ARP packets are being directed to the wrong place. That seems to be a bug either in pfSense or in the BSD subsystem, but I still don't understand why it only happens with Google cache servers.

  • That is indeed very strange, since arp requests should only be generated on subnets you're connected to at layer 2, and that's obviously not the case here. I wonder if the way you've got the openvpn tunnel set up has pfsense thinking that subnet is connected to the vpn.

    What's ifconfig -a look like?

  • @ktims:

    What's ifconfig -a look like?

    #  ifconfig -a
    vr0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:88
    	inet netmask 0xffffff00 broadcast
    	inet6 fe80::20d:b9ff:fe17:6788%vr0 prefixlen 64 scopeid 0x1 
    	media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    vr1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:89
    	inet6 fe80::20d:b9ff:fe17:6789%vr1 prefixlen 64 scopeid 0x2 
    	inet netmask 0xfffff800 broadcast
    	media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    vr2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:8a
    	inet6 fe80::20d:b9ff:fe17:678a%vr2 prefixlen 64 scopeid 0x3 
    	media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    enc0: flags=0<> metric 0 mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
    	inet netmask 0xff000000 
    	inet6 ::1 prefixlen 128 
    	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
    pflog0: flags=100 <promisc>metric 0 mtu 33204
    pfsync0: flags=41 <up,running>metric 0 mtu 1460
    	pfsync: syncdev: lo0 syncpeer: maxupd: 128
    bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	ether ce:c8:e6:f7:f4:35
    	id 00:0d:b9:17:67:88 priority 32768 hellotime 2 fwddelay 15
    	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
    	root id 00:0d:b9:17:67:88 priority 32768 ifcost 0 port 0
    	member: tap1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 10 priority 128 path cost 2000000
    	member: tap0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 9 priority 128 path cost 2000000
    	member: vr0 flags=1e7 <learning,discover,stp,edge,autoedge,ptp,autoptp>ifmaxaddr 0 port 1 priority 128 path cost 200000 proto rstp
    	        role designated state forwarding
    	member: vr2 flags=1e7 <learning,discover,stp,edge,autoedge,ptp,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 200000 proto rstp
    	        role designated state forwarding
    tap0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	ether 00:bd:49:0c:00:00
    	inet6 fe80::2bd:49ff:fe0c:0%tap0 prefixlen 64 scopeid 0x9 
    	inet netmask 0xa000802 broadcast
    	Opened by PID 494
    tap1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	ether 00:bd:cb:0d:00:01
    	inet6 fe80::2bd:cbff:fe0d:1%tap1 prefixlen 64 scopeid 0xa 
    	inet netmask 0xa000902 broadcast
    	Opened by PID 522</up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></learning,discover,stp,edge,autoedge,ptp,autoptp></learning,discover,stp,edge,autoedge,ptp,autoptp></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast></up,running></promisc></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,promisc,simplex,multicast>

    vr0 is the LAN interface, vr1 is the WAN, and vr2 is the third Ethernet port bridge to vr0.

  • Aha:

    inet netmask 0xa000802 broadcast
    inet netmask 0xa000902 broadcast

    This means the subnet masks in use on the VPN interfaces are and, which is…wrong and will match a huge swath of Internet address space (and at a glance the Google address mentioned is included in this set). It should probably be 0xffffff00 or something similar, at the very least it should be all 1s followed by all 0s and no lower than 0xff000000, not a random binary number. I don't use OpenVPN myself though, so I'm not sure how you might fix this.

  • OK, I fixed this by changing the address pools of the VPN tunnels to a subnet closer to the LAN.

  • Indeed, this is a consequence of the bridging hack howto that someone posted to the doc site. The instructions leave you with a crazy mask on your tap interface that consumes a chunk of the Internet. There isn't any way around it right now, it's something I'm looking at accommodating in some fashion before 1.2.3-release.

  • Thanks, that will be a big improvement.

  • For now this could be "sorta" fixed by adding something like <shellcmd>ifconfig tap0</shellcmd> to your config file, but if the tunnel re-establishes itself after start up the mask will be reset again. I am using OpenVPN with bridging too and have seen the crazy netmask but so far it hasn't affected me. I just assumed that it wouldn't affect anything and left it alone, but apparently that might not always be the case. I'll have to keep an eye out for this if I see strange behavior.

    Fixing this before 1.2.3 would be amazing.

  • It depends on what you set for the remote network on the tap interface. By trial and error I found one that doesn't seem to cause interference with the part of the Internet that I use.

    If the Avahi package worked on nanobsd, I don't think bridging would be needed. But it doesn't.