(Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled

johnpoz

No that is not the IP it would request dhcp from - NEVER!! That is not dhcp discovery..

The IP it uses for the ivp4 link local would be under https://tools.ietf.org/html/rfc3927

When its sends a broadcast to 169.254.255.255 this is just a link local broadcast - part of it looking for other shit on its link local network.. What port is the broadcast too? Can can tell you what it was... Didnt you have a firewall log showing it was to netbios 137?

Has ZERO to do with it trying to get an IP address..

What exactly are you bridging and WHY?? Bridging should really always be a LAST choice sort of solution... And would always be some odd use case that is outside of normal network. Why do you think you need to create a bridge? And where exactly are you creating this bridge on pfsense or on your unraid box?

brandon3055

Simply enabling bridging in unraid causes this issue. I need bridging enabled so that VM's running in unraid can be assigned their own ip's on the local network. I should point out that if i bypass pfsense and plug the unraid box directly into my dmz (which is just a cheap TPLink router setup with dhcp) both unraid and the vm running in unraid are assigned their own ip's and everything works perfectly. But thats obviously not much use to me if they can only access my DMZ it just shows this must be some how related to pfsense.

johnpoz

So you need to bridge the vm network to the physical interface.. That has zero to do with UNraids own IP..

You prob has something sending out link local broadcast, and yes pfsense will block that since its not the LAN net.. so it hits the default block rule..

You sure your VMs or your unraid don't get an IP? Look in your dhcpd on pfsense log for the discovery it gets and what IP it hands out..

What does your dhcpd log show on pfsense? Do other devices get dhcp from pfsense on the same network? Did you turn dhcpd off on pfsense? Why don't you forget the 169.254 entries in your firewall an look to what is or isn't working with dhcp.. You can then either make sure the VMs or unraid don't send out noise on linklocal - or that pfsense doesn't log it.

brandon3055

I will have to continue investigating this in the morning but for now i will tell you what i know. Outside of this issue DHCP is working fine. I have phones, laptops, another system and a raspberry PI running rasplex all connected via DHCP and i have never had any issues. I also have a couple systems connected with static ip's and they work fine as well. These are all connected to the same pfsense box as unraid via a couple un-managed gigabit switches and a wireless AP. Unraid is connected to the same switch i am using to sent this message so i know there is not issue there.

I have not checked the dhcpd log (though that will be my next step in the morning) but thanks to unraid's gui boot mode i was able to login to the vm and confirm it is not getting an ip. I know the VM isnt sending out any noise because i get the same messages in the firewall log. But i think your probably rite What i am seeing in the firewall log is probably just a byproduct of some other issue that is preventing unraid from connecting to pfsense.

johnpoz

So if something doesn't get an IP it will set that link local range.. See the rfc I linked too. If it sets that address then all of the typical discovery stuff that a client sends out vs being in your actual lan network, say 192.168.1.0/24 so broadcast for that network would be 192.168.1.255, it sends out the brodcast address for the APIPA range of 169.254.0.0/16 which is 169.254.255.255 - which yes pfsense would block because its outside the LAN NET of 192.168.1.0/24 - but 192.168.1.255 traffic would not be logged in the firewall.

So yeah that traffic your seeing is related to the 169.254 address - so you need to figure out why something is not getting the IP in your network via dhcp.. You should be able to get some hint maybe from your dhcpd log.

But if pfsense never sees the discovery, it could never to the offer, then request then ack.. Look up how dhcp works.
https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol

brandon3055

This is what i see in the DHCP log when i connect unraid
alt text
I also see the exact same messages when bridging is disabled and unraid is successfully assigned an ip.

But now i have a new problem. The last thing i tried last night was updating pfsense to the latest version via the auto update utility. That completed successfully but did not fix my problem.
Today i decided to backup my pfsense config and reset to factory defaults which i have done countless times with previous version. But this time with the latest version pfsense decided to brick itself... I have tried booting in single user mode, safe mod and i have tried booting with the previous kernel enabled. Nothing works. It looks like the boot process is hanging when it gets to "Trying to mount root from ufs:/<the device path>"

So now i need to re install pfsense which is something i really dont want to deal with at this time... WIll get back to you when i know more.

johnpoz

Is pfsense running on hardware or a VM?

brandon3055

Ok so i got pfsense reinstalled and even with factory default settings there is no change.

So for now i am going to forget about DHCP and go back to debugging with a static ip because i assume its the same issue affecting both DHCP and static. I also really only care about static anyway.

Now i dont think i explained exactly what happens in this configuration so since its probably important.
If i set a static ip in unraid (192.168.1.133) unraid has access to LAN and devices on LAN can access unraid. The only thing unraid can not access is pfsense. If i try to ping pfsense from unraid i get
"From 192.168.1.133 icmp_seq=575 Destination Host Unreachable"
Similar thing if i try to ping unraid from pfsense just no response.
I dont see anything in the pfsense logs that looks related to this.

@johnpoz said in pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Is pfsense running on hardware or a VM?

Hardware.
Its running on a Pentium 4 dual core @3.20GHz with 1GB of ram. Its a pretty old system but for pfsense its fine... i think..

Edit: Once i get some better hardware for unraid i want to visualise pfsense (or switch to opensense) and run it in a vm under unraid with with a couple nics passed directly through to the pfsense vm. But thats assuming i can get this issue fixed.

johnpoz

Your problem seems to be something wrong with unraid..

So your default rules on pfsense for its lan 192.168.1.0/24 I take it pfsense is say 192.168.1.1? is any any - the default. If you set your unraid to be 192.168.1.133/24 and it can not ping pfsense then you have something wrong with your connectivity. When you ping for pfsense IP say 192.168.1.1 do you see an arp entry for it in unraids arp cache? Can pfsense ping 192.168.1.133 what is in pfsense arp cache for this .133 IP?

Your saying your VMS you create on unraid say 192.168.1.150/24 or something can ping pfsense and they can get to the internet through pfsense?

brandon3055

And i figured it out.... Thankyou so much for suggesting i check the arp cache that lead me to this

Which is how i discovered that for whatever stupid reason pfsense sets its lan mac address to "00:00:00:00:00:00" by default. I mean why not just use the mac of the lan nic?

Anyway i added a mack address and suddenly EVERYTHING WORKS!!!

Thankyou for your help. Any idea why pfsense does not supply a valid mac on the lan interface by default?

Gertjan

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

why pfsense does not supply a valid mac on the lan interface by default?

pfSense isn't supplying MAC addresses to interfaces. It's the other way around.
Although users, that is, the 'admins', could change these MAC addresses.
Without being an expert here, I guess, when bridging interfaces, only one MAC from one interface is retained, all the others - members of the bridge, are discarded.

johnpoz

He is not bridging on pfsense... He is bridging on his unraid box for VMs

Why your pfsense box had 0 for mac have no idea.. That sure and the hell is not any sort of default thing.. Had you messed with bridges on pfsense? Had you played with mac cloning or setting the mac on pfsense?

You mean you set a static mac for pfsense on your unraid, or you had to add a mac to the interface on pfsense?

brandon3055

When i discovered the issue it was a fresh unmodified install.

johnpoz

Have never heard of a nic not having a mac.. So what mac did you give it exactly? You just made one up?

6a:62:b2

And you did that where exactly?

What does your dmesg show for the nic as it boots?

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

brandon3055

It is just am random mac from a random generator i found. I set it on the LAN interface page

If i leave it set to the default blank and then go to the interface assignments page i see this.

also

I rebooted the box a couple times but i could not catch the dmesg. Its nothing but a blur of white text that even my phone could not capture.

@johnpoz said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

Rite the one thing i did do was go through the initial basic setup which involved setting the IP.

Gertjan

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Its nothing but a blur of white text that even my phone could not capture.

?

You're an admin, so use the admin tools ^^
Go god mode (console) : option 8 and type the command

dmesg

Still to fast ?

Use this one :

dmesg | grep 'Ethernet'

If you found a " 00:00:00:00:00:00" then yes, things go downhill fast.

johnpoz

Yeah its going to need a mac that is for sure.. msk I assume that is a marvel nic.. What are the specifics of it? You should see that in the dmesg.

As stated from console if going by too fast you could also more the output

dmesg | more

Or grep for msk0

dmesg | grep msk0

Or from the web gui
Diagnostics / Command Prompt

run dmesg there..

brandon3055

Using: dmesg | grep 'msk' (LAN)

mskc0: <Marvell Yukon 88E8055 Gigabit Ethernet> port 0xce00-0xceff mem 0xfdbfc000-0xfdbfffff irq 17 at device 0.0 on pci3
msk0: <Marvell Technology Group Ltd. Yukon EC Ultra Id 0xb4 Rev 0x02> on mskc0
msk0: Using defaults for TSO: 65518/35/2048
miibus1: <MII bus> on msk0
msk0: link state changed to DOWN
msk0: link state changed to UP
msk0: link state changed to DOWN
msk0: link state changed to UP
msk0: link state changed to DOWN
msk0: link state changed to UP

Using: dmesg | grep 're0' (WAN)

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xde00-0xdeff mem 0xfdcff000-0xfdcfffff,0xfdcf8000-0xfdcfbfff irq 16 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00200000
miibus0: <MII bus> on re0
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 80:1f:02:45:29:8f
re0: netmap queues/slots: TX 1/256, RX 1/256
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
re0: link state changed to DOWN
re0: link state changed to UP

Gertjan

Ok for the Realteck interface, is has a MAC :
@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

re0: Ethernet address: 80:1f:02:45:29:8f

The msk0 interface : no.

I tend to say : check with "Marvell Yukon 88E8055 Gigabit Ethernet" support how you can set it's original MAC back.

brandon3055

@gertjan said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

I tend to say : check with "Marvell Yukon 88E8055 Gigabit Ethernet" support how you can set it's original MAC back.

honestly i don't really care. Its working now and i have no problem with using a random generated mac for the LAN interface.
If this was caused by an issue with my specific nic then thats fine. The override fixes it so thats all that matters. I just initially assumed this was caused by pfsense.