Arpwatch not downloading vendor ID's
-
@Gertjan Those are standard messages from the underlying arpwatch package (arpwatch-3.7), rather than the pfSense arpwatch package (pfSense-pkg-arpwatch-0.2.3).
The pfSense arpwatch package should be handling the update of the vendor database, but it appears to currently be broken. Perhaps a holiday project...
-
It looks like the pfSense arpwatch package as a whole needs an update. A couple of things are out of date, in particular the concept that Update vendors is optional--arpwatch no longer ships a default database.
The database update problem stems from the fact that the package is standing on its head to support the ability to have non zero padded MAC addresses. Without compressed formats, it's an easy fix.
Given that the database itself is inherently zero padded, and that other subsystems such as DHCP use zero padded MAC addresses, does anyone see a reason to maintain the option of compressed MAC addresses?
-
@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..
-
@johnpoz said in Arpwatch not downloading vendor ID's:
@dennypage your not the owner of the arpwatch package are you? If you could show it some love that would be fantastic - it needs a little tlc for sure..
I'm not, but thinking about picking it up. It definitely needs a bit of TLC.
-
@dennypage That would be really excellent if you could find the time. I love what you have done on other packages you created/maintain :-)
-
There seems a problem retrieving the oui list with
fetchpreferred by/usr/local/arpwatch/update-ethercodes.[2.7.2-RELEASE][root@fw.local.lan]/usr/local/arpwatch: fetch -q -o - https://standards-oui.ieee.org/oui/oui.csv fetch: https://standards-oui.ieee.org/oui/oui.csv: Unknown HTTP errorUsing
wget, albeit on another machine, works. Maybe the remote site currently rejects requests with certain UAs? IIRC that was the case with pfBlocker a while back. I scp'd the file to pfsense andcat oui.csv | /usr/local/arpwatch/massagevendor > ethercodes.dat.EDIT - just checked and it does look like default fetch UA is being blocked.
-
@darcey said in Arpwatch not downloading vendor ID's:
EDIT - just checked and it does look like default fetch UA is being blocked.
Hi,
i do not use arpwatch but I also find out that ieee.org seams to block the user Agent that is provided by fetch (fetch libfetch/2.0). Using a other UA resolves the problem.
Regards,
fireodo -
@fireodo
Nice catch.fetch --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36' -v -o - https://standards-oui.ieee.org/oui/oui.csv
and done.
-
@darcey There are several issues associated with the pfSense package module not using the update script provided by arpwatch itself:
- It's using the wrong URL
- The user agent (fetch) is blocked
- When any form of error occurs, it overwrites the database with any empty file.
The reason behind that the pfSense package attempting its own retrieval is because it's trying to support compressed (non leading zero) MAC formats.
-
@dennypage thanks for the deep dive!!!
Thank you!
-
@dennypage Nice detective work!
Here's hoping that you can find the time to "modernize" the package so it becomes proper usable again. Not a critical package, but rather nice to have in smaller in L2 setups.
-
@dennypage
Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review. -
@michmoor said in Arpwatch not downloading vendor ID's:
Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review.
Hey. Sorry, I haven't forgotten, and I have been working on it. But as a friend of mine once said, "Some home improvement projects may be larger than they appear." This is one of those cases, rather like avahi -> mdns-bridge. I should be able to do a write up about where I'm headed sometime next week. Hopefully, people will find their patience rewarded.
In the interim, if you want the arpwatch patch I'm currently running with in production I've included it below. Note that this patch requires the Zero padded ethernet addresses option to be enabled.
--- arpwatch.xml.org 2024-11-27 11:19:46.000000000 -0800 +++ arpwatch.xml 2024-12-24 13:07:50.974190000 -0800 @@ -120,7 +120,7 @@ <fielddescr>Update vendors</fielddescr> <fieldname>update_vendors</fieldname> <type>checkbox</type> - <description>Updates the ethernet vendor database, downloaded from http://standards-oui.ieee.org/oui/oui.csv.</description> + <description>Updates the ethernet vendor database, downloaded from https://standards-oui.ieee.org/oui/oui.csv.</description> </field> <field> <fielddescr>Clear database</fielddescr> --- arpwatch.inc.org 2024-11-27 11:19:46.000000000 -0800 +++ arpwatch.inc 2024-12-26 11:01:01.839497000 -0800 @@ -19,7 +19,7 @@ */ define('ARPWATCH_LOCAL_DIR', '/usr/local/arpwatch'); -define('ARPWATCH_ETHERCODES_URL', 'http://standards-oui.ieee.org/oui/oui.csv'); +define('ARPWATCH_ETHERCODES_URL', 'https://standards-oui.ieee.org/oui/oui.csv'); define('ARPWATCH_SENDMAIL_PATH', '/usr/sbin/sendmail'); define('ARPWATCH_SENDMAIL_PROXY', '/usr/local/arpwatch/sendmail_proxy.php'); @@ -128,9 +128,7 @@ } function arpwatch_update_vendors($args) { - exec('/usr/bin/fetch -qo - '.ARPWATCH_ETHERCODES_URL.'|' - .ARPWATCH_LOCAL_DIR.'/massagevendor '.$args.' >' - .ARPWATCH_LOCAL_DIR.'/ethercodes.dat'); + exec('/usr/local/arpwatch/update-ethercodes'); } function arpwatch_clear_database() { @@ -174,7 +172,7 @@ $entry = [ 'ifname' => $ifname, - 'ifdescr' => strtoupper($active_interface), + 'ifdescr' => convert_friendly_interface_to_friendly_descr($active_interface), 'mac' => $mac, 'vendor' => $vendor, 'ip' => $ip, @@ -194,14 +192,6 @@ } } - usort($entries, function($e1, $e2){ - if ($e1['ifdescr'] == $e2['ifdescr']) { - return 0; - } - - return ($e1['ifdescr'] < $e2['ifdescr']) ? -1 : 1; - }); - return $entries; } -
@dennypage no worries. no rush. Just wondering if i could provide a helping hand along the way. Appreciate yah !
-
Took a bit longer than I expected, but hopefully this will reward your patience, and give you a good idea of where I'm headed:
Github: The ANDwatch daemon
There were just too many problems to overcome with arpwatch.
FWIW, it may be two or three weeks before I can do the pfSense package due to travel.
-
@dennypage this looks great! Thank you. Safe travels :-)
-
@dennypage This looks very exiting indeed. Thank you very much for investing your valuable time in creating such a great tool/package for all of us
-
The submission for FreeBSD (upstream) is in. Not sure how long it will take, Usually it's pretty quick.
If anyone would like to give ANDwatch a spin before the pfSense UI is done, please let me know and I will send you a copy of the FreeBSD package.
I'll get to the pfSense UI package written when I return.
-
-
@michmoor said in Arpwatch not downloading vendor ID's:
I can try out the FreeBSD package.
Thanks. I've included the package below. Please let me know if you have any issues.
Here is the notification script if you want it:
#!/usr/bin/env php <?php require_once("notices.inc"); $timestamp=$argv[1]; $ifname=convert_real_interface_to_friendly_descr($argv[2]); $ipaddr=$argv[3]; $old_hwaddr=$argv[4]; $old_hwaddr_org=$argv[5]; $new_hwaddr=$argv[6]; $new_hwaddr_org=$argv[7]; $hostname = gethostbyaddr($ipaddr); $msg = "ANDwatch notificaton\n\n"; $msg .= sprintf("%22s: %s\n", "timestamp", $timestamp); $msg .= sprintf("%22s: %s\n", "interface", $ifname); $msg .= sprintf("%22s: %s\n", "hostname", $hostname); $msg .= sprintf("%22s: %s\n", "ip address", $ipaddr); $msg .= sprintf("%22s: %s %s\n", "old ethernet address", $old_hwaddr, $old_hwaddr_org); $msg .= sprintf("%22s: %s %s\n", "new ethernet address", $new_hwaddr, $new_hwaddr_org); notify_all_remote($msg); ?>I don't have anything to display a status page yet, but you can do a query via the command line like so:
andwatch-query <ifname>That will give you a report of all the latest IP mappings.
[Edit: Updated pkg to v1.0.1 to fix query bug with MAC addresses beginning with '0']
[Edit: Updated pkg to v1.1.0 to change record update / age behavior. Details on GitHub.]
