Strange connectivity problem with 1.2.3-RC2 embedded

tommyboy180

Can you get to any address without DNS?
such as http://209.85.225.147/

Sostris

Yes, e.g. http://209.131.36.158/ == yahoo.com. Yours too.

tommyboy180

Its almost like your pfsense box just doen't like the google cache server address.

What do you have for firewall rules? Can we rule those out? I am at a loss of what it can be.

Sostris

@tommyboy180:

What do you have for firewall rules? Can we rule those out? I am at a loss of what it can be.

Besides the default rule, I block only incoming multicast IGMP which comes from my ISP. I hesitated to post this problem here because it makes no sense, but it is real. I assumed Google itself was blocking me for some reason, but it isn't.

tommyboy180

Yeah. We can rule out Google and we can rule out your ISP. I have no clue, I am waiting to see if a Hero member can shed some light.

Your right it doesn't make sense.

ktims

Maybe do a tcpdump on your wan side and see if any of the packets are making it out or back. Comparing with what you see on the lan side tcpdump may shed more light.

Sostris

On the WAN, nothing. On the LAN, a lot of this:

arp who-has px-in-f132.google.com tell 10.0.9.1
IP myhost.64801 > px-in-f132.google.com.http: S 2290598658:2290598658(0) win 65535

This is interesting because 10.0.9.1 is not my LAN. I'm on 192.168.1.0/24. 10.0.9.0/24 is the address pool of one of my two OpenVPN tunnels, which are bridged to the LAN using instructions I found around here somewhere. So the ARP packets are being directed to the wrong place. That seems to be a bug either in pfSense or in the BSD subsystem, but I still don't understand why it only happens with Google cache servers.

ktims

That is indeed very strange, since arp requests should only be generated on subnets you're connected to at layer 2, and that's obviously not the case here. I wonder if the way you've got the openvpn tunnel set up has pfsense thinking that subnet is connected to the vpn.

What's ifconfig -a look like?

Sostris

@ktims:

What's ifconfig -a look like?

#  ifconfig -a
vr0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:88
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::20d:b9ff:fe17:6788%vr0 prefixlen 64 scopeid 0x1 
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
vr1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:89
	inet6 fe80::20d:b9ff:fe17:6789%vr1 prefixlen 64 scopeid 0x2 
	inet 76.174.118.225 netmask 0xfffff800 broadcast 255.255.255.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
vr2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic>ether 00:0d:b9:17:67:8a
	inet6 fe80::20d:b9ff:fe17:678a%vr2 prefixlen 64 scopeid 0x3 
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
pflog0: flags=100 <promisc>metric 0 mtu 33204
pfsync0: flags=41 <up,running>metric 0 mtu 1460
	pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether ce:c8:e6:f7:f4:35
	id 00:0d:b9:17:67:88 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:0d:b9:17:67:88 priority 32768 ifcost 0 port 0
	member: tap1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 10 priority 128 path cost 2000000
	member: tap0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: vr0 flags=1e7 <learning,discover,stp,edge,autoedge,ptp,autoptp>ifmaxaddr 0 port 1 priority 128 path cost 200000 proto rstp
	        role designated state forwarding
	member: vr2 flags=1e7 <learning,discover,stp,edge,autoedge,ptp,autoptp>ifmaxaddr 0 port 3 priority 128 path cost 200000 proto rstp
	        role designated state forwarding
tap0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	ether 00:bd:49:0c:00:00
	inet6 fe80::2bd:49ff:fe0c:0%tap0 prefixlen 64 scopeid 0x9 
	inet 10.0.8.1 netmask 0xa000802 broadcast 255.255.255.253
	Opened by PID 494
tap1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	ether 00:bd:cb:0d:00:01
	inet6 fe80::2bd:cbff:fe0d:1%tap1 prefixlen 64 scopeid 0xa 
	inet 10.0.9.1 netmask 0xa000902 broadcast 255.255.255.253
	Opened by PID 522</up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></learning,discover,stp,edge,autoedge,ptp,autoptp></learning,discover,stp,edge,autoedge,ptp,autoptp></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></up,broadcast,running,simplex,multicast></up,running></promisc></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic></up,broadcast,running,promisc,simplex,multicast>

vr0 is the LAN interface, vr1 is the WAN, and vr2 is the third Ethernet port bridge to vr0.

ktims

Aha:

inet 10.0.8.1 netmask 0xa000802 broadcast 255.255.255.253
inet 10.0.9.1 netmask 0xa000902 broadcast 255.255.255.253

This means the subnet masks in use on the VPN interfaces are 160.0.9.2 and 160.0.8.2, which is…wrong and will match a huge swath of Internet address space (and at a glance the Google address mentioned is included in this set). It should probably be 0xffffff00 or something similar, at the very least it should be all 1s followed by all 0s and no lower than 0xff000000, not a random binary number. I don't use OpenVPN myself though, so I'm not sure how you might fix this.

Sostris

OK, I fixed this by changing the address pools of the VPN tunnels to a subnet closer to the LAN.

cmb

Indeed, this is a consequence of the bridging hack howto that someone posted to the doc site. The instructions leave you with a crazy mask on your tap interface that consumes a chunk of the Internet. There isn't any way around it right now, it's something I'm looking at accommodating in some fashion before 1.2.3-release.

Sostris

Thanks, that will be a big improvement.

Briantist

For now this could be "sorta" fixed by adding something like <shellcmd>ifconfig tap0 10.0.8.1/24</shellcmd> to your config file, but if the tunnel re-establishes itself after start up the mask will be reset again. I am using OpenVPN with bridging too and have seen the crazy netmask but so far it hasn't affected me. I just assumed that it wouldn't affect anything and left it alone, but apparently that might not always be the case. I'll have to keep an eye out for this if I see strange behavior.

Fixing this before 1.2.3 would be amazing.

Sostris

It depends on what you set for the remote network on the tap interface. By trial and error I found one that doesn't seem to cause interference with the part of the Internet that I use.

If the Avahi package worked on nanobsd, I don't think bridging would be needed. But it doesn't.