Arpwatch not downloading vendor ID's

Benjamin 3

Good day folks,

I am seeing Arpwatch fail to automatically update vendor IDs from http://standards-oui.ieee.org/oui/oui.csv despite the "Update vendors" checkbox being enabled in the settings. Manual updates using /usr/local/arpwatch/update-ethercodes work correctly, but the automatic update functionality appears broken.

Steps to Reproduce:

Install Arpwatch 0.2.3 on Netgate 4200 running pfSense 24.11.
Navigate to Services > Arpwatch > Settings.
Ensure the "Update vendors" checkbox is selected.
Restart the Arpwatch service.
Observe that vendor IDs in the Arpwatch database are not updated.
Manually run /usr/local/arpwatch/update-ethercodes.
Observe that vendor IDs are now correctly updated in the database.
Leave the "Update vendors" checkbox selected and observe that subsequent manual updates with /usr/local/arpwatch/update-ethercodes are overwritten.
Expected Behavior:

When the "Update vendors" checkbox is selected, Arpwatch should automatically download and update vendor IDs from the specified URL.

Actual Behavior:

Arpwatch fails to automatically update vendor IDs. Manual updates work correctly, but leaving the "Update vendors" option enabled causes these updates to be overwritten.

Workaround:

A cron job running /usr/local/arpwatch/update-ethercodes once a day with the "Update vendors" option disabled provides a temporary workaround.

Additional Information:

Platform: Netgate 4200
OS: pfSense 24.11
Arpwatch Version: 0.2.3

I am not seeing any any error messages related to vendor ID updates in the Arpwatch logs (I might not be checking for this correctly).

Any assistance greatly appreciated. Many thanks!

Gertjan

@Benjamin-3

Isn't that what is suggested when you install arpwatch :

....
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
=====
Message from arpwatch-3.7:

--
You can create an ethercodes.dat file by running this script:

    /usr/local/arpwatch/update-ethercodes

Here's a example crontab entry to update it every night:

    00 0 * * * root sleep `jot -r 1 0 600` ; /usr/local/arpwatch/update-ethercodes

The -m flag was removed. If you were using the -m watcher flag,
please switch to -w.
>>> Cleaning up cache...done.
Success

dennypage

@Gertjan Those are standard messages from the underlying arpwatch package (arpwatch-3.7), rather than the pfSense arpwatch package (pfSense-pkg-arpwatch-0.2.3).

The pfSense arpwatch package should be handling the update of the vendor database, but it appears to currently be broken. Perhaps a holiday project...

dennypage

It looks like the pfSense arpwatch package as a whole needs an update. A couple of things are out of date, in particular the concept that Update vendors is optional--arpwatch no longer ships a default database.

The database update problem stems from the fact that the package is standing on its head to support the ability to have non zero padded MAC addresses. Without compressed formats, it's an easy fix.

Given that the database itself is inherently zero padded, and that other subsystems such as DHCP use zero padded MAC addresses, does anyone see a reason to maintain the option of compressed MAC addresses?

johnpoz

@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..

dennypage

@johnpoz said in Arpwatch not downloading vendor ID's:

@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..

I'm not, but thinking about picking it up. It definitely needs a bit of TLC.

keyser

@dennypage That would be really excellent if you could find the time. I love what you have done on other packages you created/maintain :-)

darcey

There seems a problem retrieving the oui list with fetch preferred by /usr/local/arpwatch/update-ethercodes.

[2.7.2-RELEASE][root@fw.local.lan]/usr/local/arpwatch: fetch -q -o - https://standards-oui.ieee.org/oui/oui.csv
fetch: https://standards-oui.ieee.org/oui/oui.csv: Unknown HTTP error

Using wget, albeit on another machine, works. Maybe the remote site currently rejects requests with certain UAs? IIRC that was the case with pfBlocker a while back. I scp'd the file to pfsense and cat oui.csv | /usr/local/arpwatch/massagevendor > ethercodes.dat.

EDIT - just checked and it does look like default fetch UA is being blocked.

fireodo

@darcey said in Arpwatch not downloading vendor ID's:

EDIT - just checked and it does look like default fetch UA is being blocked.

Hi,

i do not use arpwatch but I also find out that ieee.org seams to block the user Agent that is provided by fetch (fetch libfetch/2.0). Using a other UA resolves the problem.

Regards,
fireodo

Gertjan

@fireodo
Nice catch.

fetch --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36' -v -o - https://standards-oui.ieee.org/oui/oui.csv

and done.

dennypage

@darcey There are several issues associated with the pfSense package module not using the update script provided by arpwatch itself:

It's using the wrong URL
The user agent (fetch) is blocked
When any form of error occurs, it overwrites the database with any empty file.

The reason behind that the pfSense package attempting its own retrieval is because it's trying to support compressed (non leading zero) MAC formats.

Benjamin 3

@dennypage thanks for the deep dive!!!

Thank you!

keyser

@dennypage Nice detective work!

Here's hoping that you can find the time to "modernize" the package so it becomes proper usable again. Not a critical package, but rather nice to have in smaller in L2 setups.

michmoor

@dennypage
Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review.

dennypage

@michmoor said in Arpwatch not downloading vendor ID's:

Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review.

Hey. Sorry, I haven't forgotten, and I have been working on it. But as a friend of mine once said, "Some home improvement projects may be larger than they appear." This is one of those cases, rather like avahi -> mdns-bridge. I should be able to do a write up about where I'm headed sometime next week. Hopefully, people will find their patience rewarded.

In the interim, if you want the arpwatch patch I'm currently running with in production I've included it below. Note that this patch requires the Zero padded ethernet addresses option to be enabled.

--- arpwatch.xml.org	2024-11-27 11:19:46.000000000 -0800
+++ arpwatch.xml	2024-12-24 13:07:50.974190000 -0800
@@ -120,7 +120,7 @@
 			<fielddescr>Update vendors</fielddescr>
 			<fieldname>update_vendors</fieldname>
 			<type>checkbox</type>
-			<description>Updates the ethernet vendor database, downloaded from http://standards-oui.ieee.org/oui/oui.csv.</description>
+			<description>Updates the ethernet vendor database, downloaded from https://standards-oui.ieee.org/oui/oui.csv.</description>
 		</field>
 		<field>
 			<fielddescr>Clear database</fielddescr>

--- arpwatch.inc.org	2024-11-27 11:19:46.000000000 -0800
+++ arpwatch.inc	2024-12-26 11:01:01.839497000 -0800
@@ -19,7 +19,7 @@
  */
 
 define('ARPWATCH_LOCAL_DIR', '/usr/local/arpwatch');
-define('ARPWATCH_ETHERCODES_URL', 'http://standards-oui.ieee.org/oui/oui.csv');
+define('ARPWATCH_ETHERCODES_URL', 'https://standards-oui.ieee.org/oui/oui.csv');
 define('ARPWATCH_SENDMAIL_PATH', '/usr/sbin/sendmail');
 define('ARPWATCH_SENDMAIL_PROXY', '/usr/local/arpwatch/sendmail_proxy.php');
 
@@ -128,9 +128,7 @@
 }
 
 function arpwatch_update_vendors($args) {
-	exec('/usr/bin/fetch -qo - '.ARPWATCH_ETHERCODES_URL.'|'
-	    .ARPWATCH_LOCAL_DIR.'/massagevendor '.$args.' >'
-	    .ARPWATCH_LOCAL_DIR.'/ethercodes.dat');
+	exec('/usr/local/arpwatch/update-ethercodes');
 }
 
 function arpwatch_clear_database() {
@@ -174,7 +172,7 @@
 
 			$entry = [
 				'ifname' => $ifname,
-				'ifdescr' => strtoupper($active_interface),
+				'ifdescr' => convert_friendly_interface_to_friendly_descr($active_interface),
 				'mac' => $mac,
 				'vendor' => $vendor,
 				'ip' => $ip,
@@ -194,14 +192,6 @@
 		}
 	}
 
-	usort($entries, function($e1, $e2){
-		if ($e1['ifdescr'] == $e2['ifdescr']) {
-			return 0;
-		}
-
-		return ($e1['ifdescr'] < $e2['ifdescr']) ? -1 : 1;
-	});
-
 	return $entries;
 }

michmoor

@dennypage no worries. no rush. Just wondering if i could provide a helping hand along the way. Appreciate yah !

dennypage

Took a bit longer than I expected, but hopefully this will reward your patience, and give you a good idea of where I'm headed:

Github: The ANDwatch daemon

There were just too many problems to overcome with arpwatch.

FWIW, it may be two or three weeks before I can do the pfSense package due to travel.

Benjamin 3

@dennypage this looks great! Thank you. Safe travels :-)

keyser

@dennypage This looks very exiting indeed. Thank you very much for investing your valuable time in creating such a great tool/package for all of us

dennypage

The submission for FreeBSD (upstream) is in. Not sure how long it will take, Usually it's pretty quick.

If anyone would like to give ANDwatch a spin before the pfSense UI is done, please let me know and I will send you a copy of the FreeBSD package.

I'll get to the pfSense UI package written when I return.