(Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled

brandon3055 · Dec 26, 2018, 5:43 AM

So i have been banging my head against the wall for months now trying to figure out why unraid's networking just implodes when i turn on bridging. Eventually i concluded it must be an issue with the Realtek nic i'm using. I finally switched to a nic from the unraid recommended hardware list and... same problem.

After a lot more debugging i found the issue. I dont know much about this topic but my digging i have discovered that when most devices request an ip via DHCP they send a request to 255.255.255.255 with a source of 0.0.0.0. I assume this is what unraid does when bridging is disabled because everything works.

However when i enable bridging unraid switches to the "link local" domain (169.254.x.x) it sends its request to 169.254.255.255 with a source of 169.254.14.59.

Now i dont know if this is normal or a bug. Other routers seem ok with this but it turns out pfsense just completely blocks all traffic on the link local address thereby preventing unraid from ever getting a DHCP response. This fores unraid to use a fallback link local address (196.254.14.59 in my case)

It even shows in the pfsense log that the request is being blocked

Notice the source IP matches the ip unraid falls back to. Also a bunch more entries show up every time i connect the unraid box to the network so there is no doubt that this unraid being blocked.

Unfortunately all my research has told me that the rule that blocks link local traffic is hard coded into pfsense so i dont think there is anything i can do on that end to work around that.

This also may or may not have something todo with the fact that pfsense is not connected directly to the internet but instead to a dmz that eventually leads to the internet.

I have also posted this on the unraid forum

johnpoz · Dec 25, 2018, 1:31 PM

Huh?

You do understand that 169.254 is APIPA.. And yes many OSes will fall back to that when they can not get an IP from dhcp..

brandon3055 · Dec 25, 2018, 1:40 PM

@johnpoz said in pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Huh?

You do understand that 169.254 is APIPA.. And yes many OSes will fall back to that when they can not get an IP from dhcp..

I know that's what Microsoft calls it. But i really dont know much about it other than its the ip unraid is using when it requests an ip from pfsense and pfsense blocks all requests from that ip.

Edit: Unless im missing something. Maybe unraid is first failing to get a DHCP lease via 255.255.255.255 then falling back to APIPA at which point it may be attempting to access the network triggering pfsense to block in... That could explain what i am seeing in the log. But i get the same issue when i try ti use a static ip in unraid.

johnpoz · Dec 25, 2018, 3:20 PM

No that is not the IP it would request dhcp from - NEVER!! That is not dhcp discovery..

The IP it uses for the ivp4 link local would be under https://tools.ietf.org/html/rfc3927

When its sends a broadcast to 169.254.255.255 this is just a link local broadcast - part of it looking for other shit on its link local network.. What port is the broadcast too? Can can tell you what it was... Didnt you have a firewall log showing it was to netbios 137?

Has ZERO to do with it trying to get an IP address..

What exactly are you bridging and WHY?? Bridging should really always be a LAST choice sort of solution... And would always be some odd use case that is outside of normal network. Why do you think you need to create a bridge? And where exactly are you creating this bridge on pfsense or on your unraid box?

brandon3055 · Dec 25, 2018, 3:34 PM

Simply enabling bridging in unraid causes this issue. I need bridging enabled so that VM's running in unraid can be assigned their own ip's on the local network. I should point out that if i bypass pfsense and plug the unraid box directly into my dmz (which is just a cheap TPLink router setup with dhcp) both unraid and the vm running in unraid are assigned their own ip's and everything works perfectly. But thats obviously not much use to me if they can only access my DMZ it just shows this must be some how related to pfsense.

johnpoz · Dec 25, 2018, 3:49 PM

So you need to bridge the vm network to the physical interface.. That has zero to do with UNraids own IP..

You prob has something sending out link local broadcast, and yes pfsense will block that since its not the LAN net.. so it hits the default block rule..

You sure your VMs or your unraid don't get an IP? Look in your dhcpd on pfsense log for the discovery it gets and what IP it hands out..

What does your dhcpd log show on pfsense? Do other devices get dhcp from pfsense on the same network? Did you turn dhcpd off on pfsense? Why don't you forget the 169.254 entries in your firewall an look to what is or isn't working with dhcp.. You can then either make sure the VMs or unraid don't send out noise on linklocal - or that pfsense doesn't log it.

brandon3055 · Dec 25, 2018, 5:08 PM

I will have to continue investigating this in the morning but for now i will tell you what i know. Outside of this issue DHCP is working fine. I have phones, laptops, another system and a raspberry PI running rasplex all connected via DHCP and i have never had any issues. I also have a couple systems connected with static ip's and they work fine as well. These are all connected to the same pfsense box as unraid via a couple un-managed gigabit switches and a wireless AP. Unraid is connected to the same switch i am using to sent this message so i know there is not issue there.

I have not checked the dhcpd log (though that will be my next step in the morning) but thanks to unraid's gui boot mode i was able to login to the vm and confirm it is not getting an ip. I know the VM isnt sending out any noise because i get the same messages in the firewall log. But i think your probably rite What i am seeing in the firewall log is probably just a byproduct of some other issue that is preventing unraid from connecting to pfsense.

johnpoz · Dec 25, 2018, 5:22 PM

So if something doesn't get an IP it will set that link local range.. See the rfc I linked too. If it sets that address then all of the typical discovery stuff that a client sends out vs being in your actual lan network, say 192.168.1.0/24 so broadcast for that network would be 192.168.1.255, it sends out the brodcast address for the APIPA range of 169.254.0.0/16 which is 169.254.255.255 - which yes pfsense would block because its outside the LAN NET of 192.168.1.0/24 - but 192.168.1.255 traffic would not be logged in the firewall.

So yeah that traffic your seeing is related to the 169.254 address - so you need to figure out why something is not getting the IP in your network via dhcp.. You should be able to get some hint maybe from your dhcpd log.

But if pfsense never sees the discovery, it could never to the offer, then request then ack.. Look up how dhcp works.
https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol

brandon3055 · Dec 26, 2018, 2:02 AM

This is what i see in the DHCP log when i connect unraid

I also see the exact same messages when bridging is disabled and unraid is successfully assigned an ip.

But now i have a new problem. The last thing i tried last night was updating pfsense to the latest version via the auto update utility. That completed successfully but did not fix my problem.
Today i decided to backup my pfsense config and reset to factory defaults which i have done countless times with previous version. But this time with the latest version pfsense decided to brick itself... I have tried booting in single user mode, safe mod and i have tried booting with the previous kernel enabled. Nothing works. It looks like the boot process is hanging when it gets to "Trying to mount root from ufs:/<the device path>"

So now i need to re install pfsense which is something i really dont want to deal with at this time... WIll get back to you when i know more.

johnpoz · Dec 26, 2018, 4:10 AM

Is pfsense running on hardware or a VM?

brandon3055 · Dec 26, 2018, 4:35 AM

Ok so i got pfsense reinstalled and even with factory default settings there is no change.

So for now i am going to forget about DHCP and go back to debugging with a static ip because i assume its the same issue affecting both DHCP and static. I also really only care about static anyway.

Now i dont think i explained exactly what happens in this configuration so since its probably important.
If i set a static ip in unraid (192.168.1.133) unraid has access to LAN and devices on LAN can access unraid. The only thing unraid can not access is pfsense. If i try to ping pfsense from unraid i get
"From 192.168.1.133 icmp_seq=575 Destination Host Unreachable"
Similar thing if i try to ping unraid from pfsense just no response.
I dont see anything in the pfsense logs that looks related to this.

@johnpoz said in pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Is pfsense running on hardware or a VM?

Hardware.
Its running on a Pentium 4 dual core @3.20GHz with 1GB of ram. Its a pretty old system but for pfsense its fine... i think..

Edit: Once i get some better hardware for unraid i want to visualise pfsense (or switch to opensense) and run it in a vm under unraid with with a couple nics passed directly through to the pfsense vm. But thats assuming i can get this issue fixed.

johnpoz · Dec 26, 2018, 5:10 AM

Your problem seems to be something wrong with unraid..

So your default rules on pfsense for its lan 192.168.1.0/24 I take it pfsense is say 192.168.1.1? is any any - the default. If you set your unraid to be 192.168.1.133/24 and it can not ping pfsense then you have something wrong with your connectivity. When you ping for pfsense IP say 192.168.1.1 do you see an arp entry for it in unraids arp cache? Can pfsense ping 192.168.1.133 what is in pfsense arp cache for this .133 IP?

Your saying your VMS you create on unraid say 192.168.1.150/24 or something can ping pfsense and they can get to the internet through pfsense?

brandon3055 · Dec 26, 2018, 5:50 AM

And i figured it out.... Thankyou so much for suggesting i check the arp cache that lead me to this

Which is how i discovered that for whatever stupid reason pfsense sets its lan mac address to "00:00:00:00:00:00" by default. I mean why not just use the mac of the lan nic?

Anyway i added a mack address and suddenly EVERYTHING WORKS!!!

Thankyou for your help. Any idea why pfsense does not supply a valid mac on the lan interface by default?

Gertjan · Dec 26, 2018, 6:00 AM

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

why pfsense does not supply a valid mac on the lan interface by default?

pfSense isn't supplying MAC addresses to interfaces. It's the other way around.
Although users, that is, the 'admins', could change these MAC addresses.
Without being an expert here, I guess, when bridging interfaces, only one MAC from one interface is retained, all the others - members of the bridge, are discarded.

johnpoz · Dec 26, 2018, 11:20 AM

He is not bridging on pfsense... He is bridging on his unraid box for VMs

Why your pfsense box had 0 for mac have no idea.. That sure and the hell is not any sort of default thing.. Had you messed with bridges on pfsense? Had you played with mac cloning or setting the mac on pfsense?

You mean you set a static mac for pfsense on your unraid, or you had to add a mac to the interface on pfsense?

brandon3055 · Dec 26, 2018, 12:16 PM

When i discovered the issue it was a fresh unmodified install.

johnpoz · Dec 26, 2018, 12:35 PM

Have never heard of a nic not having a mac.. So what mac did you give it exactly? You just made one up?

6a:62:b2

And you did that where exactly?

What does your dmesg show for the nic as it boots?

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

brandon3055 · Dec 26, 2018, 2:25 PM

It is just am random mac from a random generator i found. I set it on the LAN interface page

If i leave it set to the default blank and then go to the interface assignments page i see this.

also

I rebooted the box a couple times but i could not catch the dmesg. Its nothing but a blur of white text that even my phone could not capture.

@johnpoz said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

Rite the one thing i did do was go through the initial basic setup which involved setting the IP.

Gertjan · Dec 26, 2018, 2:39 PM

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Its nothing but a blur of white text that even my phone could not capture.

?

You're an admin, so use the admin tools ^^
Go god mode (console) : option 8 and type the command

dmesg

Still to fast ?

Use this one :

dmesg | grep 'Ethernet'

If you found a " 00:00:00:00:00:00" then yes, things go downhill fast.

johnpoz · Dec 26, 2018, 2:49 PM

Yeah its going to need a mac that is for sure.. msk I assume that is a marvel nic.. What are the specifics of it? You should see that in the dmesg.

As stated from console if going by too fast you could also more the output

dmesg | more

Or grep for msk0

dmesg | grep msk0

Or from the web gui
Diagnostics / Command Prompt

run dmesg there..