Migration from 2100 to 4100

[mgi]

@johnpoz I should have been more precise about this, but basically, it's what I said in my first post...

• first, I spotted that DHCP either fails or takes long to kick in
• hosts configured with static IP or those that already got IP from DHCP before switching to bridge0 can communicate together even over the bridge with no issue
• I don't see bridge0 MAC in the ARP table on connected hosts

I don't think this is a loop. My network is only two hosts connected to the bridge, and chaining two switches (the bridge and AP) together on one port. None of my devices is connected to the wifi and the bridge at the same time.

I needed the network stable for work, so I didn't spend too much time troubleshooting the issue. I'll try again over the weekend and probably also contact the TAC.

[stephenw10]

Try running a pcap when you have a client trying to pull a dhcp lease.

You'll probably see either missing replies or missing responses somewhere.

You should probably spoof the MAC of bridge0 if you're running the dhcp server on it otherwise clients will see it as a new gateway after each boot.

Steve

[mgi]

@stephenw10 I ran a pcap today. I thought before that (as well) DHCP is part of the problem, but that looks ok.

The problem is that the box stops responding to ARP requests.

I was waiting with spoofing the MAC until I get evrything working. Anyway, I spoofed the MAC and the network started working. Then I restarted the interface on my laptop and the box stopped responding to ARPs again.

I can see the same even when I set static IP on my laptop.

This seems to be the same or at least very simillar issue. I’ll try to spoof the MAC on the bridge and all bridge-members later today and tomorrow even that sounds odd.

[stephenw10]

You only need to spoof it on the bridge interfaces because otherwise it uses a randomly generated MAC at each boot.

Are you using VLANs?

Steve

[mgi]

@stephenw10 That’s what I did, but then I found the post. It sounds strange but I might give it a shot.

I don’t use any VLANs. My home network is very simple - exactly as the diagram I posted.

[stephenw10]

Mmm, I can't see how it would help but easy to test.

[mgi]

@stephenw10 Tbh, me neither. I just found that post and I’m wondering if this might be a bug ¯\_(ツ)_/¯

I don’t want to mess with this anymore tonight. I’ll test again tomorrow morning even I have my doubts.

[mgi]

I tested the thing with same MAC on the bridge and all bridge members. No difference.

Anyway, something seems wrong as the bridge L3 interface does not respond to ARP requests.

[stephenw10]

But it only does it when the mesh wifi is connected to it?

Do you have STP enabled on the bridge?

[mgi]

@stephenw10 so it's even more weird :)

I tested again and only the port where the AP is connected seems to be affected. The other bridge ports seem to work ok.

I as well played with (R)STP combinations again and again, but no luck. I also tried to force the port to be edge, but that didn't help either.

There might be some incompatibility with the Eero mesh, but I'm not sure why; I can't see a reason why the port or ARP response should be blocked.

I had the APs connected to Ubiquiti ER-12, Juniper SRX and Netgate 2100 before. There was no issue, but all of those have HW switching.

[johnpoz]

Never understand why users continue to think bridging is the same as switch... If you want to switch, then use a switch ;)

Why do users always want to waste good discrete interfaces by trying to turn them into switch ports?

You have everything on a flat network, a 20$ 5 port gig switch would solve your problem ;) Make that $15 I see a netgear gs305 on amazon for 14.99 right now.

[mgi]

@johnpoz I know bridge and switch are not exactly the same, but there’s a “0” difference in my case :)

I already have a switch connected, that’s not the problem. The problem is that the bridge is not working as expected. I’m just trying to figure out if’s because of Eero or pfSense/Netgate.

In my case, using a switch is wasting ports on the 4100 because otherwise I don’t have use for those. I need 4100 only for the (IPSec) throughput.

[stephenw10]

Mmm, I agree that this could be trivially solved with the cheapest 5 port switch.

However I'm of the opinion that if you have interfaces you're not using then adding them to a bridge is not necessarily a bad use. But you need to be aware of the limitations of doing so. Mostly that it loads the firewall just to pass traffic between the bridged ports.

Mesh wifi is weird though, it's ot to be something in the ARP handling there.
Were you seeing ARP requests leave the client but never arrive at the bridge?

[johnpoz]

@mgi said in Migration from 2100 to 4100:

but there’s a “0” difference in my case :)

I would beg to differ, if you were using a switch we wouldn't be having this conversation and you wouldn't have spent how much time trying to get it to work ;)

because otherwise I don’t have use for those

Also going to differ on that opinion as well - just because you not currently using an interface doesn't mean its "wasted" it means you don't currently have a need for that interface.. I have ports open on my switch - are they "wasted" ;)

They are discrete interfaces - when you at some point? Decide to actually segment your network, they will be quite valuable..

I have TBs of space on my disks in my nas currently not in use - am I wasting it? I have currently not using all the ram in my PC, is that wasted? Room for growth is not wasted.. Now if you bought a 48 port switch and have 3 devices.. That might be wasteful ;) But a couple of interfaces on you router that are currently not being used because your only using 1 network segment is not the same thing.

[mgi]

@stephenw10 said in Migration from 2100 to 4100:

Mmm, I agree that this could be trivially solved with the cheapest 5 port switch.

Yes, that's true and that's what I'm doing for now. But I want to avoid having a dedicated switch if possible.

However I'm of the opinion that if you have interfaces you're not using then adding them to a bridge is not necessarily a bad use. But you need to be aware of the limitations of doing so. Mostly that it loads the firewall just to pass traffic between the bridged ports.

As I mentioned before, I need the bridge only for my APs (only one is connected directly), one RPi and a home GW.

I changed the net.link.bridge.pfil_member to 0 and net.link.bridge.pfil_bridge to 1. Do I have to configure FW rules for each bridge member explicitly even after changing those two knobs and assigning the bridge0 interface to the LAN "profile"?

Mesh wifi is weird though, it's ot to be something in the ARP handling there.
Were you seeing ARP requests leave the client but never arrive at the bridge?

Yeah, I agree mesh is weird, but usually works nice for home networks :)

This is from tcpdump on the Netgate.

00:18:08.679414 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.59, length 60
00:18:08.837574 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.16, length 60
00:18:08.841753 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.97, length 46
00:18:09.108861 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.60, length 60
00:18:09.151828 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.63, length 60
00:18:09.210252 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.15, length 60
00:18:09.236770 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.79, length 60
00:18:09.679341 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.59, length 60
00:18:09.839371 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.16, length 60
00:18:09.868070 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.97, length 46
00:18:10.109070 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.60, length 60
00:18:10.152127 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.63, length 60
00:18:10.260658 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.79, length 60
00:18:10.633207 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.15, length 60
00:18:10.679315 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.59, length 60
00:18:10.889649 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.97, length 46
00:18:11.108905 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.60, length 60

That really doesn't look good.

[mgi]

@johnpoz It's you right to differ :)

If I eventually decide that I need more segments (I don't see any reason for that in the foreseeable future) on my home network, I can always add a switch.

We can have a different understanding of what's "wasting" — anyway yes, I'm currently also wasting a 48 port switch from my lab.

{master:0}
root@ex-tmp-homenet> show chassis hardware | match "engine" 
Routing Engine 0          BUILTIN      BUILTIN           RE-EX4400-48T

I really don't have a problem adding a small switch to my network, but I'm also interested in why the bridge is not working and I came here to ask if anyone faced a similar issue with similar setup and similar equipment. That's all.

[johnpoz]

@mgi said in Migration from 2100 to 4100:

00:18:08.679414 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.254 tell 192.168.1.59, length 60

So pfsense is not answering arp for its IP.. Where exactly is that IP assigned - the bridge interface?

[mgi]

@johnpoz Yes, it seems so. The IP’s on the bridge0 interface.

I found this Reddit post. Sounds as my problem, but they never really solved it.

I need to have a look again when I have some time.

[johnpoz]

@mgi he did solve it ;)

"and is actually a better architecture anyway as I get to kick the bridge to the curb"

he got rid of the bridge hehhehehe

[stephenw10]

Are the MAC addresses in that pcap correct?

Is it using a multicast MAC address?

That will be blocked by default unless you add the appropriate tunable:
https://docs.netgate.com/pfsense/en/latest/install/upgrade-before-2.2.html?highlight=mesh#microsoft-load-balancing-open-mesh-traffic

Though I wouldn't expect that to be bridge specific.

Steve