Changing Default Boot Device + Multi LAN with DMZ setup

stephenw10

Ok, here's something I see.
You'll notice the MAC address of the bridge on box is fake where as yours appears to be real. I exepct it to be fake as it doesn't have real hardware to get the MAC from. An issue that has cropped up in the past is that Windows systems can be confused by the fake MAC, especially because it can be generated as a new fake address on each boot. This causes systems >Win XP to label the network as a new untrusted environment and block any connections from them. Not sure if that applies to you.

Why can you not ping 192.168.2.2? Does it exist on your network?

Attempting to ping within the same subnet should not appear in the state table, I think we have a clue there. Are your clients receving the correct subnet mask?

Steve

Beaflag VonRathburg

All clients on the TP-Link 24 port switch are Windows computers. There's thirteen Windows 7 and three Windows 8.1 . My laptop does have Kubuntu 13.10 on it and has the same issue though so I'm not sure.

192.168.2.2 does not exist on my network so I used it as such to test that it would appear in the state table when I attempted to ping it. I haven't done anything of this before so I'm trying to be as thorough as possible.

Despite all the clients being bridged and on the same subnet I am still unable to ping anything, but pfsense. Regardless of the interface, device, or OS. When, I went through all of the computers attempting to determine what device .1.118 was I checked to make sure all computers and my TP-Link switch are on the 255.255.255.0 subnet. Using ifconfig and ipconfig confirmed that and all devices do report the subnet mask as being 255.255.255.0 .

stephenw10

Presumably you can ping other devices that are connected to the same switch?

I thought of one other thing I have that is non-standard on that box. I have IP fast forwarding enabled. That shouldn't make any difference at all but I'm running out of ideas. ::) It's a sysctl on the system tunables table. By default it's off because it breaks IPSec. Try setting it to 1.

You could try disabling the hardware offload options in System: Advanced: Networking: I can't really see how that would help either. :-\

Anyone else got any suggestions for a bridge that isn't forwarding packets? Or more accurately clients that seem to be sending all traffic to their gateway even that for their own subnet?

Time to run some packet captures and see what's up.

Steve

Beaflag VonRathburg

Power issues have been raping my life right now, but… While dealing with all of that crap I might have found something.

IP fast forwarding didn't do anything differently.

Windows 8.1 ipconfig shows different results than that of Windows 7. All of my Windows 7 computers show the subnet mask as 255.255.255.0 and my 8.1 machines do as well, but only for the section that says ethernet adapter to ethernet. Where it says something along the lines of local connection 1, 2, 3, 4, etc... it shows the subnet mask as 255.255.0.0 . I'm guessing that despite the bridge being setup and all the subnets of local hardware being set to 255.255.255.0 it is getting switched some where along the way.

stephenw10

A bad subnet mask is exactly the sort of this that might cause this. If your client machines don't realise they're in the same subnet as others on the bridge (or supposed to be as least) then they will send traffic via their gateway instead of directly. However for that to happen they would have to have a smaller subnet mask like 255.255.255.224 not a much larger one as you've discovered.

If that was the case then you might have a problem pinging other clients even on the same switch.

I don't have a Windows 8 machine to look at here so I'm not entirely sure what you're referring to. Do you have a screenshot or the output from ipconfig?

I assume you can't ping ping between two Win 7 clients on different switches though?

Steve

Beaflag VonRathburg

I can't ping anything even if they're sitting right next to each other on the same switch. This has been tested with on all of my switches with three different operating systems. Here is what I was speaking of in regards to Windows 8.1 showing odd subnet behavior.

When the bridge originally broke I had to keep using ipconfig release and renew to actually get it to connect automatically. That was due to me changing how the switches were connected around, but it might be a clue?

phil.davis

http://en.wikipedia.org/wiki/IP_address#IPv4_private_addresses - read the Address autoconfiguration section. Windows sets those 169.254 addresses when it does not get a DHCP response within "timeout?" seconds. So that indicates that the Windows client DHCP requests are not getting through to the pfSense DHCP server (or the replies are not getting back).

Beaflag VonRathburg

That makes sense. Again, I'm new at this so what should I start looking in to resolve this?

stephenw10

Ok so this is good problem solving clues we have here. :)
So you can't ping other machines even across the same switch when both are receiving IP addresses in the 192.168.1.X subnet from pfSense. That is interesting because that traffic doesn't go through pfSense at all, or at least it it shouldn't. Hence whatever is causing those machines not respond to the pings is local.

The ipconfig result from a Windows client shows, as Phil said, a load of interfaces with self assigned IP addresses along with one correctly assigned IP. I assume that you don't actually have 4 NICs in that box? What are showing up there as 'connections' I would interpret as what Windows does when it thinks you've connected it to a new wired network, it starts a new set of connection settings. Though I have no experience with Windows 8.

Theses two things together strongly point to my earlier hypothesis:@stephenw10:

An issue that has cropped up in the past is that Windows systems can be confused by the fake MAC, especially because it can be generated as a new fake address on each boot. This causes systems >Win XP to label the network as a new untrusted environment and block any connections from them.

You said that you've tried a Kubuntu machine as well and that behaved similarly which seemed to rule this out but perhaps you didn't try receving pings with Kubuntu or it has a personal firewall on it.

Check that Windows firewall hasn't assigned the network as 'public'. Change it 'private' or trusted (it'e been a while ::)).

The cause of this is that the pfSense bridge MAC address can change. If you've rebooted your pfSense box check the output of ifconfig again to see if the MAC has changed. If it has then you can get around it by spoofing the MAC to something fixed. I had forgotten earlier but if you look back at my ifconfig output where my bridge has an obviously fake MAC that is because I spoofed it to that to get around this very problem! ;)

Steve

Beaflag VonRathburg

I haven't had a lot of time to work on this, but I have been twiddling with it every once in a while. My main issue right now is getting my power to actually stay on. I have a main shutoff that is going bad and a local place wants $225 for the breaker alone… ::) . Back to pinging issues though as I still haven't figured out this issue. This is from the same computer with Windows 8.1 .

Before reboot:

After reboot:

ipconfig /all returns:

You can see it isn't resetting the MAC address on reboot, but each time it has setup a new connection it is. All connections are set to Home and private within Windows firewall.

Pfsense has been rebooted about a million times as I had six power outages today alone. Here is a new ifconfig output:

$ ifconfig
re0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:dd
	inet6 fe80::6a1c:a2ff:fe12:11dd%re0 prefixlen 64 scopeid 0x1 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
re1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:de
	inet6 fe80::6a1c:a2ff:fe12:11de%re1 prefixlen 64 scopeid 0x2 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
re2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 94:de:80:cc:8d:3b
	inet 67.190.224.33 netmask 0xfffffc00 broadcast 255.255.255.255
	inet6 fe80::96de:80ff:fecc:8d3b%re2 prefixlen 64 scopeid 0x3 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
re3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:db
	inet6 fe80::6a1c:a2ff:fe12:11db%re3 prefixlen 64 scopeid 0x4 
	nd6 options=1 <performnud>media: Ethernet autoselect (none)
	status: no carrier
re4: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:dc
	inet6 fe80::6a1c:a2ff:fe12:11dc%re4 prefixlen 64 scopeid 0x5 
	nd6 options=1 <performnud>media: Ethernet autoselect (none)
	status: no carrier
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33144
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:f2:fb:65:02:00
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::1:1%bridge0 prefixlen 64 scopeid 0xb 
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: re0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 1 priority 128 path cost 55
	member: re3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 4 priority 128 path cost 55
	member: re1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast>

This is the one I posted earlier:

$ ifconfig
re0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:dd
	inet6 fe80::6a1c:a2ff:fe12:11dd%re0 prefixlen 64 scopeid 0x1 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

re1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:de
	inet6 fe80::6a1c:a2ff:fe12:11de%re1 prefixlen 64 scopeid 0x2 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
re2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 94:de:80:cc:8d:3b
	inet XX.XXX.XXX.XX netmask 0xfffffc00 broadcast 255.255.255.255
	inet6 fe80::96de:80ff:fecc:8d3b%re2 prefixlen 64 scopeid 0x3 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex,master>)
	status: active

re3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:db
	inet6 fe80::6a1c:a2ff:fe12:11db%re3 prefixlen 64 scopeid 0x4 
	nd6 options=1 <performnud>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active

re4: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:1c:a2:12:11:dc
	inet6 fe80::6a1c:a2ff:fe12:11dc%re4 prefixlen 64 scopeid 0x5 
	nd6 options=1 <performnud>media: Ethernet autoselect (none)
	status: no carrier

plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500

enc0: flags=0<> metric 0 mtu 1536

pflog0: flags=100 <promisc>metric 0 mtu 33144

lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1

bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:f2:fb:65:02:00
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::1:1%bridge0 prefixlen 64 scopeid 0xb 
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: re0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 1 priority 128 path cost 55
	member: re3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 4 priority 128 path cost 55
	member: re1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex,master></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast>

stephenw10

The MAC I was referring to is on the pfSense bridge0 interface. It hasn't changed though which makes me think that's not the issue. Though you say the Windows box creates new connections every time pfSense is rebooted? It's hard to see but most of the connections on your Windows box appear to be TAP VPN connections rather than real Ethernet interfaces, does it actually have more than one real interface showing? Are those other interfaces something you have put there?

The only other thing I see that's different between your bridge setup and mine is that my bridge does not have an IPv6 address. Could this be an IPv6 issue? :-
Try setting the bridge interface to have IPv6 configuration type: 'none'.
Edit: The member interafce are also set to IPv6 type 'none'.
I can't see why that would be an issue to be honest but maybe something is trying route via IPv6.

Steve