(Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled

[johnpoz]

So you need to bridge the vm network to the physical interface.. That has zero to do with UNraids own IP..

You prob has something sending out link local broadcast, and yes pfsense will block that since its not the LAN net.. so it hits the default block rule..

You sure your VMs or your unraid don't get an IP? Look in your dhcpd on pfsense log for the discovery it gets and what IP it hands out..

What does your dhcpd log show on pfsense? Do other devices get dhcp from pfsense on the same network? Did you turn dhcpd off on pfsense? Why don't you forget the 169.254 entries in your firewall an look to what is or isn't working with dhcp.. You can then either make sure the VMs or unraid don't send out noise on linklocal - or that pfsense doesn't log it.

[brandon3055]

I will have to continue investigating this in the morning but for now i will tell you what i know. Outside of this issue DHCP is working fine. I have phones, laptops, another system and a raspberry PI running rasplex all connected via DHCP and i have never had any issues. I also have a couple systems connected with static ip's and they work fine as well. These are all connected to the same pfsense box as unraid via a couple un-managed gigabit switches and a wireless AP. Unraid is connected to the same switch i am using to sent this message so i know there is not issue there.

I have not checked the dhcpd log (though that will be my next step in the morning) but thanks to unraid's gui boot mode i was able to login to the vm and confirm it is not getting an ip. I know the VM isnt sending out any noise because i get the same messages in the firewall log. But i think your probably rite What i am seeing in the firewall log is probably just a byproduct of some other issue that is preventing unraid from connecting to pfsense.

[johnpoz]

So if something doesn't get an IP it will set that link local range.. See the rfc I linked too. If it sets that address then all of the typical discovery stuff that a client sends out vs being in your actual lan network, say 192.168.1.0/24 so broadcast for that network would be 192.168.1.255, it sends out the brodcast address for the APIPA range of 169.254.0.0/16 which is 169.254.255.255 - which yes pfsense would block because its outside the LAN NET of 192.168.1.0/24 - but 192.168.1.255 traffic would not be logged in the firewall.

So yeah that traffic your seeing is related to the 169.254 address - so you need to figure out why something is not getting the IP in your network via dhcp.. You should be able to get some hint maybe from your dhcpd log.

But if pfsense never sees the discovery, it could never to the offer, then request then ack.. Look up how dhcp works.
https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol

[brandon3055]

This is what i see in the DHCP log when i connect unraid

I also see the exact same messages when bridging is disabled and unraid is successfully assigned an ip.

But now i have a new problem. The last thing i tried last night was updating pfsense to the latest version via the auto update utility. That completed successfully but did not fix my problem.
Today i decided to backup my pfsense config and reset to factory defaults which i have done countless times with previous version. But this time with the latest version pfsense decided to brick itself... I have tried booting in single user mode, safe mod and i have tried booting with the previous kernel enabled. Nothing works. It looks like the boot process is hanging when it gets to "Trying to mount root from ufs:/<the device path>"

So now i need to re install pfsense which is something i really dont want to deal with at this time... WIll get back to you when i know more.

[johnpoz]

Is pfsense running on hardware or a VM?

[brandon3055]

Ok so i got pfsense reinstalled and even with factory default settings there is no change.

So for now i am going to forget about DHCP and go back to debugging with a static ip because i assume its the same issue affecting both DHCP and static. I also really only care about static anyway.

Now i dont think i explained exactly what happens in this configuration so since its probably important.
If i set a static ip in unraid (192.168.1.133) unraid has access to LAN and devices on LAN can access unraid. The only thing unraid can not access is pfsense. If i try to ping pfsense from unraid i get
"From 192.168.1.133 icmp_seq=575 Destination Host Unreachable"
Similar thing if i try to ping unraid from pfsense just no response.
I dont see anything in the pfsense logs that looks related to this.

@johnpoz said in pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Is pfsense running on hardware or a VM?

Hardware.
Its running on a Pentium 4 dual core @3.20GHz with 1GB of ram. Its a pretty old system but for pfsense its fine... i think..

Edit: Once i get some better hardware for unraid i want to visualise pfsense (or switch to opensense) and run it in a vm under unraid with with a couple nics passed directly through to the pfsense vm. But thats assuming i can get this issue fixed.

[johnpoz]

Your problem seems to be something wrong with unraid..

So your default rules on pfsense for its lan 192.168.1.0/24 I take it pfsense is say 192.168.1.1? is any any - the default. If you set your unraid to be 192.168.1.133/24 and it can not ping pfsense then you have something wrong with your connectivity. When you ping for pfsense IP say 192.168.1.1 do you see an arp entry for it in unraids arp cache? Can pfsense ping 192.168.1.133 what is in pfsense arp cache for this .133 IP?

Your saying your VMS you create on unraid say 192.168.1.150/24 or something can ping pfsense and they can get to the internet through pfsense?

[brandon3055]

And i figured it out.... Thankyou so much for suggesting i check the arp cache that lead me to this

Which is how i discovered that for whatever stupid reason pfsense sets its lan mac address to "00:00:00:00:00:00" by default. I mean why not just use the mac of the lan nic?

Anyway i added a mack address and suddenly EVERYTHING WORKS!!!

Thankyou for your help. Any idea why pfsense does not supply a valid mac on the lan interface by default?

[Gertjan]

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

why pfsense does not supply a valid mac on the lan interface by default?

pfSense isn't supplying MAC addresses to interfaces. It's the other way around.
Although users, that is, the 'admins', could change these MAC addresses.
Without being an expert here, I guess, when bridging interfaces, only one MAC from one interface is retained, all the others - members of the bridge, are discarded.

[johnpoz]

He is not bridging on pfsense... He is bridging on his unraid box for VMs

Why your pfsense box had 0 for mac have no idea.. That sure and the hell is not any sort of default thing.. Had you messed with bridges on pfsense? Had you played with mac cloning or setting the mac on pfsense?

You mean you set a static mac for pfsense on your unraid, or you had to add a mac to the interface on pfsense?

[brandon3055]

When i discovered the issue it was a fresh unmodified install.

[johnpoz]

Have never heard of a nic not having a mac.. So what mac did you give it exactly? You just made one up?

6a:62:b2

And you did that where exactly?

What does your dmesg show for the nic as it boots?

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

[brandon3055]

It is just am random mac from a random generator i found. I set it on the LAN interface page

If i leave it set to the default blank and then go to the interface assignments page i see this.

also

I rebooted the box a couple times but i could not catch the dmesg. Its nothing but a blur of white text that even my phone could not capture.

@johnpoz said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

BTW - ip 192.168.1.8 for pfsense would not be a fresh unmodified install.. 192.168.1.1 is the default IP that is on the lan interface.

Rite the one thing i did do was go through the initial basic setup which involved setting the IP.

[Gertjan]

@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

Its nothing but a blur of white text that even my phone could not capture.

?

You're an admin, so use the admin tools ^^
Go god mode (console) : option 8 and type the command

dmesg

Still to fast ?

Use this one :

dmesg | grep 'Ethernet'

If you found a " 00:00:00:00:00:00" then yes, things go downhill fast.

[johnpoz]

Yeah its going to need a mac that is for sure.. msk I assume that is a marvel nic.. What are the specifics of it? You should see that in the dmesg.

As stated from console if going by too fast you could also more the output

dmesg | more

Or grep for msk0

dmesg | grep msk0

Or from the web gui
Diagnostics / Command Prompt

run dmesg there..

[brandon3055]

Using: dmesg | grep 'msk' (LAN)

mskc0: <Marvell Yukon 88E8055 Gigabit Ethernet> port 0xce00-0xceff mem 0xfdbfc000-0xfdbfffff irq 17 at device 0.0 on pci3
msk0: <Marvell Technology Group Ltd. Yukon EC Ultra Id 0xb4 Rev 0x02> on mskc0
msk0: Using defaults for TSO: 65518/35/2048
miibus1: <MII bus> on msk0
msk0: link state changed to DOWN
msk0: link state changed to UP
msk0: link state changed to DOWN
msk0: link state changed to UP
msk0: link state changed to DOWN
msk0: link state changed to UP

Using: dmesg | grep 're0' (WAN)

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xde00-0xdeff mem 0xfdcff000-0xfdcfffff,0xfdcf8000-0xfdcfbfff irq 16 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00200000
miibus0: <MII bus> on re0
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 80:1f:02:45:29:8f
re0: netmap queues/slots: TX 1/256, RX 1/256
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
re0: link state changed to DOWN
re0: link state changed to UP

[Gertjan]

Ok for the Realteck interface, is has a MAC :
@brandon3055 said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

re0: Ethernet address: 80:1f:02:45:29:8f

The msk0 interface : no.

I tend to say : check with "Marvell Yukon 88E8055 Gigabit Ethernet" support how you can set it's original MAC back.

[brandon3055]

@gertjan said in (Solved) pfSense blocks static and DHCP ip requests from unraid when bridging is enabled:

I tend to say : check with "Marvell Yukon 88E8055 Gigabit Ethernet" support how you can set it's original MAC back.

honestly i don't really care. Its working now and i have no problem with using a random generated mac for the LAN interface.
If this was caused by an issue with my specific nic then thats fine. The override fixes it so thats all that matters. I just initially assumed this was caused by pfsense.

[johnpoz]

In my some 30 years in the biz, I can honestly say I do not recall ever seeing such a thing.. How could the nic not have a mac?

Does it get a mac if you boot some other os, so you use a different driver? Has to be some sort of issue with the driver? Or the thing is faulty out of the gate? I would be more concerned to be honest.. I wouldn't trust using it.. Until you figure out what exacty causes such a problem..

I have setup 10's of thousands of machines over the years - never seen such a thing.

Had you flashed the nics firmware at some point? What was on the machine before you installed pfsense on it?

It was working before with pfsense? What version of pfsense was that - the version of freebsd would of been different, etc.

Doing a google for all zeros mac and marvel does have quite a few hits... Not seeing any sort of rhyme or reason to what finding, most people just seem to get a new nic ;)

The only thing close is flashing a dd-wrt firmware on old router set the mac to same thing as everyone else using the firmware, and this would cause problems if someone else with your ISP had also flashed theirs and they got on first.. That took a bit to track down when saw.. Fix was to get the original mac back..

I wold have to say something wrong with the driver or the nic bad firmware flash or something in the factory... What you did by setting a mac is a work around.. But personally I would just get a different nic if you can not track down a reason to why mac went missing.

[brandon3055]

The nic is the built in nic on an old acer pc that looks like it originally came out of a school. I have been running the system with pfsense for about 3 years now and the nic has never caused any issues other than this one. This is only running my home network so nothing critical. It it does ever cause issues i will just throw in a different nic ore move it to a different system.