Arpwatch not downloading vendor ID's

Benjamin 3

Good day folks,

I am seeing Arpwatch fail to automatically update vendor IDs from http://standards-oui.ieee.org/oui/oui.csv despite the "Update vendors" checkbox being enabled in the settings. Manual updates using /usr/local/arpwatch/update-ethercodes work correctly, but the automatic update functionality appears broken.

Steps to Reproduce:

Install Arpwatch 0.2.3 on Netgate 4200 running pfSense 24.11.
Navigate to Services > Arpwatch > Settings.
Ensure the "Update vendors" checkbox is selected.
Restart the Arpwatch service.
Observe that vendor IDs in the Arpwatch database are not updated.
Manually run /usr/local/arpwatch/update-ethercodes.
Observe that vendor IDs are now correctly updated in the database.
Leave the "Update vendors" checkbox selected and observe that subsequent manual updates with /usr/local/arpwatch/update-ethercodes are overwritten.
Expected Behavior:

When the "Update vendors" checkbox is selected, Arpwatch should automatically download and update vendor IDs from the specified URL.

Actual Behavior:

Arpwatch fails to automatically update vendor IDs. Manual updates work correctly, but leaving the "Update vendors" option enabled causes these updates to be overwritten.

Workaround:

A cron job running /usr/local/arpwatch/update-ethercodes once a day with the "Update vendors" option disabled provides a temporary workaround.

Additional Information:

Platform: Netgate 4200
OS: pfSense 24.11
Arpwatch Version: 0.2.3

I am not seeing any any error messages related to vendor ID updates in the Arpwatch logs (I might not be checking for this correctly).

Any assistance greatly appreciated. Many thanks!

Gertjan

@Benjamin-3

Isn't that what is suggested when you install arpwatch :

....
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
=====
Message from arpwatch-3.7:

--
You can create an ethercodes.dat file by running this script:

    /usr/local/arpwatch/update-ethercodes

Here's a example crontab entry to update it every night:

    00 0 * * * root sleep `jot -r 1 0 600` ; /usr/local/arpwatch/update-ethercodes

The -m flag was removed. If you were using the -m watcher flag,
please switch to -w.
>>> Cleaning up cache...done.
Success

dennypage

@Gertjan Those are standard messages from the underlying arpwatch package (arpwatch-3.7), rather than the pfSense arpwatch package (pfSense-pkg-arpwatch-0.2.3).

The pfSense arpwatch package should be handling the update of the vendor database, but it appears to currently be broken. Perhaps a holiday project...

dennypage

It looks like the pfSense arpwatch package as a whole needs an update. A couple of things are out of date, in particular the concept that Update vendors is optional--arpwatch no longer ships a default database.

The database update problem stems from the fact that the package is standing on its head to support the ability to have non zero padded MAC addresses. Without compressed formats, it's an easy fix.

Given that the database itself is inherently zero padded, and that other subsystems such as DHCP use zero padded MAC addresses, does anyone see a reason to maintain the option of compressed MAC addresses?

johnpoz

@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..

dennypage

@johnpoz said in Arpwatch not downloading vendor ID's:

@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..

I'm not, but thinking about picking it up. It definitely needs a bit of TLC.

keyser

@dennypage That would be really excellent if you could find the time. I love what you have done on other packages you created/maintain :-)

darcey

There seems a problem retrieving the oui list with fetch preferred by /usr/local/arpwatch/update-ethercodes.

[2.7.2-RELEASE][root@fw.local.lan]/usr/local/arpwatch: fetch -q -o - https://standards-oui.ieee.org/oui/oui.csv
fetch: https://standards-oui.ieee.org/oui/oui.csv: Unknown HTTP error

Using wget, albeit on another machine, works. Maybe the remote site currently rejects requests with certain UAs? IIRC that was the case with pfBlocker a while back. I scp'd the file to pfsense and cat oui.csv | /usr/local/arpwatch/massagevendor > ethercodes.dat.

EDIT - just checked and it does look like default fetch UA is being blocked.

fireodo

@darcey said in Arpwatch not downloading vendor ID's:

EDIT - just checked and it does look like default fetch UA is being blocked.

Hi,

i do not use arpwatch but I also find out that ieee.org seams to block the user Agent that is provided by fetch (fetch libfetch/2.0). Using a other UA resolves the problem.

Regards,
fireodo

Gertjan

@fireodo
Nice catch.

fetch --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36' -v -o - https://standards-oui.ieee.org/oui/oui.csv

and done.

dennypage

@darcey There are several issues associated with the pfSense package module not using the update script provided by arpwatch itself:

It's using the wrong URL
The user agent (fetch) is blocked
When any form of error occurs, it overwrites the database with any empty file.

The reason behind that the pfSense package attempting its own retrieval is because it's trying to support compressed (non leading zero) MAC formats.

Benjamin 3

@dennypage thanks for the deep dive!!!

Thank you!

keyser

@dennypage Nice detective work!

Here's hoping that you can find the time to "modernize" the package so it becomes proper usable again. Not a critical package, but rather nice to have in smaller in L2 setups.