ARP Tables...Most Static Addresses are No Longer Permanent

ARAMP1

Everything is working well after updating to 2.7, just found it odd that most of my static addresses are now no longer permanent. There are a couple that are. Literally two. There seems to be no rhyme or reason for it.

Any ideas?

JKnott

@ARAMP1

ARP cache is not permanent, unless you create a static ARP entry. Is that what you're referring to? Or are you talking about static IP addresses?

johnpoz

@ARAMP1 did you click this on some of them when you created the reservation for a specific mac?

ARAMP1

@johnpoz Yes, every single one of them.

johnpoz

@ARAMP1 where did that mac come from? That isn't a listed mac..

ARAMP1

@johnpoz It's edited for the picture.

johnpoz

@ARAMP1 why? why not just hide the last 3 numbers if your worried? MACs are only layer 2 - unless that was a mac address of your wifi and worried about looking it up in some wifi db like wigle or something?

If I click on create static arp on one of my reservations - it switches to perm.

maybe something odd in the upgrade process - go in and save on your reservations - do they show up then as perm?

ARAMP1

I went back in and checked to see if the upgrade from 2.6 to 2.7 unclicked any of the boxes but it didn't. Figured there might be some setting that was added.

Everything still works. And really it's not that big of a deal. Just thought it was odd.

johnpoz

@ARAMP1 I agree it is odd, if shows static in the setting then it should be static - see my edit go in and click save on one of the entries that isn't showing static, but you have set too.. Does it switch to perm?

I just unchecked that one I had set as test and it went back to non static in the arp table. Maybe some weirdness in the upgrade process?

edit: maybe the device grabbed wrong IP and not its reservation - but if static entry was set then it wouldn't work, etc. But yeah it is odd if your setting shows it should create a static arp pair, and its not set.

edit2: So I looked through my reservations to see if I had previous set any to static - and I see the same thing.. This was previous set to create a static pair..

But as see its not - I recently updated to 23.05.1 so maybe something in the update the static pairing gets lost??

edit3: Well I do believe you have run into some sort of oddness for sure.. I went in and just hit save on the entry I had marked as static, and it didn't update the arp table. But unchecked static, saved, and then rechecked static and save and it switched to perm in the arp table..

Might want to file a redmine about it..

Let me look in redmine if I see anything related.

edit4: This seems relevant for sure

https://redmine.pfsense.org/issues/14374

Would have to reboot to see if happens then.. I could test on one of my VMs.. running 2.7, but there is a comment 3 days ago about seeing this in 2.7 as well..

ARAMP1

@johnpoz I went in and unclicked a few ARP static entry boxes, saved it, then went back in and clicked them. They're now showing up as permanent.

ARP Table 1.jpg

johnpoz

@ARAMP1 yeah see my edit this seems to be this bug

https://redmine.pfsense.org/issues/14374

I don't really feel like rebooting my box ;) I am working remote today.. But I could test it on my 2.7 vm..

ARAMP1

@johnpoz Just read through the link. Thanks.

I've rebooted a few times since upgrade and it didn't change anything.

johnpoz

@ARAMP1 so when you reboot the perm entries go away - that seems like what the bug is talking about.

ARAMP1

Ah...Yeah, just confirmed that's the case. Rebooted and all the permanent entries are gone (except for the VLAN interfaces themselves).

Well, thanks again @johnpoz for your help.

johnpoz

@ARAMP1 you might want to just chime in on that redmine, that you too are seeing it.. Or I can make a comment if you want, and link to this thread.

I made a comment and linked to this thread.

zkhcohen

I can also validate that Regression #14374 is present on my 3.7.0 installation.

I noticed that simply opening and saving each DHCP entry with static ARP assigned (without making any changes) will resolve the issue for that specific entry. Rebooting the DHCP service has no impact.

Another interesting note - when I create an XML backup of my pfSense config and inspect it, all the proper static ARP entries are present. In other words, the configuration is present, it's just not being initialized properly on reboot. Note that I haven't had a chance to test the secondary symptom -- the eventual erasure of static ARP entries for inactive devices.

Setting up an hourly cron job / shellcmd which sets "arp -s <ip> <mac>" for each entry is the best solution until a patch has been issued.

zkhcohen

Here's a script which leverages python3.11 to search for and set static arp entries:

Step 1. Create a shell script called "arp.sh" in the /usr/local/etc/rc.d/ directory:

#!/bin/sh

python3.11 - <<EOF
import xml.etree.ElementTree as ET
import subprocess

# Load the XML file
tree = ET.parse("/conf/config.xml")
root = tree.getroot()

# Extract the IP and MAC addresses using list comprehension
results = [{'mac': staticmap.find("mac").text, 'ip': staticmap.find("ipaddr").text}
           for staticmap in root.iter("staticmap")
           if staticmap.find("arp_table_static_entry") is not None]

# Execute the arp commands
for result in results:
    command = f"arp -s \"{result['ip']}\" \"{result['mac']}\""

    print("Command:", command)

    completed_process = subprocess.run(command, shell=True, executable="/bin/sh", capture_output=True, text=True)

    if completed_process.returncode == 0:
        print("Command executed successfully.")
        print("Output:", completed_process.stdout)
    else:
        print("Command execution failed.")
        print("Error:", completed_process.stderr)
EOF

Step 2. Run: chmod +x /usr/local/etc/rc.d/arp.sh
Step 3. In the pfSense UI, navigate to Services > Cron and create the following job:

0 	* 	* 	* 	* 	root 	/bin/sh /usr/local/etc/rc.d/arp.sh

Step 4. To execute the command manually, run /bin/sh /usr/local/etc/rc.d/arp.sh. Upon checking the pfSense UI Diagnostics > ARP Table, you should see that all of the entries are now "Permanent".

johnpoz

@zkhcohen so your script looks to what is in the arp table and makes them all static - ie perm? That would be a horrible idea

zkhcohen

@johnpoz No. It looks for a pre-existing permanent static arp mapping in the DHCP config and then reapplies it via the CLI.

zkhcohen

@johnpoz

A patch was just issued (extremely quickly, I might add), and I can confirm that it fixes the issue in my environment:

https://redmine.pfsense.org/projects/pfsense/repository/2/revisions/5082edf92795fe8266be49905fe4f07eb682449d