Arpwatch not downloading vendor ID's
-
There seems a problem retrieving the oui list with
fetchpreferred by/usr/local/arpwatch/update-ethercodes.[2.7.2-RELEASE][root@fw.local.lan]/usr/local/arpwatch: fetch -q -o - https://standards-oui.ieee.org/oui/oui.csv fetch: https://standards-oui.ieee.org/oui/oui.csv: Unknown HTTP errorUsing
wget, albeit on another machine, works. Maybe the remote site currently rejects requests with certain UAs? IIRC that was the case with pfBlocker a while back. I scp'd the file to pfsense andcat oui.csv | /usr/local/arpwatch/massagevendor > ethercodes.dat.EDIT - just checked and it does look like default fetch UA is being blocked.
-
@darcey said in Arpwatch not downloading vendor ID's:
EDIT - just checked and it does look like default fetch UA is being blocked.
Hi,
i do not use arpwatch but I also find out that ieee.org seams to block the user Agent that is provided by fetch (fetch libfetch/2.0). Using a other UA resolves the problem.
Regards,
fireodo -
@fireodo
Nice catch.fetch --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36' -v -o - https://standards-oui.ieee.org/oui/oui.csv
and done.
-
@darcey There are several issues associated with the pfSense package module not using the update script provided by arpwatch itself:
- It's using the wrong URL
- The user agent (fetch) is blocked
- When any form of error occurs, it overwrites the database with any empty file.
The reason behind that the pfSense package attempting its own retrieval is because it's trying to support compressed (non leading zero) MAC formats.
-
@dennypage thanks for the deep dive!!!
Thank you!
-
@dennypage Nice detective work!
Here's hoping that you can find the time to "modernize" the package so it becomes proper usable again. Not a critical package, but rather nice to have in smaller in L2 setups.
-
@dennypage
Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review. -
@michmoor said in Arpwatch not downloading vendor ID's:
Hey Denny. Any assistance you need when looking at this package? I dont know how far along you are in the review.
Hey. Sorry, I haven't forgotten, and I have been working on it. But as a friend of mine once said, "Some home improvement projects may be larger than they appear." This is one of those cases, rather like avahi -> mdns-bridge. I should be able to do a write up about where I'm headed sometime next week. Hopefully, people will find their patience rewarded.
In the interim, if you want the arpwatch patch I'm currently running with in production I've included it below. Note that this patch requires the Zero padded ethernet addresses option to be enabled.
--- arpwatch.xml.org 2024-11-27 11:19:46.000000000 -0800 +++ arpwatch.xml 2024-12-24 13:07:50.974190000 -0800 @@ -120,7 +120,7 @@ <fielddescr>Update vendors</fielddescr> <fieldname>update_vendors</fieldname> <type>checkbox</type> - <description>Updates the ethernet vendor database, downloaded from http://standards-oui.ieee.org/oui/oui.csv.</description> + <description>Updates the ethernet vendor database, downloaded from https://standards-oui.ieee.org/oui/oui.csv.</description> </field> <field> <fielddescr>Clear database</fielddescr> --- arpwatch.inc.org 2024-11-27 11:19:46.000000000 -0800 +++ arpwatch.inc 2024-12-26 11:01:01.839497000 -0800 @@ -19,7 +19,7 @@ */ define('ARPWATCH_LOCAL_DIR', '/usr/local/arpwatch'); -define('ARPWATCH_ETHERCODES_URL', 'http://standards-oui.ieee.org/oui/oui.csv'); +define('ARPWATCH_ETHERCODES_URL', 'https://standards-oui.ieee.org/oui/oui.csv'); define('ARPWATCH_SENDMAIL_PATH', '/usr/sbin/sendmail'); define('ARPWATCH_SENDMAIL_PROXY', '/usr/local/arpwatch/sendmail_proxy.php'); @@ -128,9 +128,7 @@ } function arpwatch_update_vendors($args) { - exec('/usr/bin/fetch -qo - '.ARPWATCH_ETHERCODES_URL.'|' - .ARPWATCH_LOCAL_DIR.'/massagevendor '.$args.' >' - .ARPWATCH_LOCAL_DIR.'/ethercodes.dat'); + exec('/usr/local/arpwatch/update-ethercodes'); } function arpwatch_clear_database() { @@ -174,7 +172,7 @@ $entry = [ 'ifname' => $ifname, - 'ifdescr' => strtoupper($active_interface), + 'ifdescr' => convert_friendly_interface_to_friendly_descr($active_interface), 'mac' => $mac, 'vendor' => $vendor, 'ip' => $ip, @@ -194,14 +192,6 @@ } } - usort($entries, function($e1, $e2){ - if ($e1['ifdescr'] == $e2['ifdescr']) { - return 0; - } - - return ($e1['ifdescr'] < $e2['ifdescr']) ? -1 : 1; - }); - return $entries; } -
@dennypage no worries. no rush. Just wondering if i could provide a helping hand along the way. Appreciate yah !
-
Took a bit longer than I expected, but hopefully this will reward your patience, and give you a good idea of where I'm headed:
Github: The ANDwatch daemon
There were just too many problems to overcome with arpwatch.
FWIW, it may be two or three weeks before I can do the pfSense package due to travel.
-
@dennypage this looks great! Thank you. Safe travels :-)
-
@dennypage This looks very exiting indeed. Thank you very much for investing your valuable time in creating such a great tool/package for all of us
-
The submission for FreeBSD (upstream) is in. Not sure how long it will take, Usually it's pretty quick.
If anyone would like to give ANDwatch a spin before the pfSense UI is done, please let me know and I will send you a copy of the FreeBSD package.
I'll get to the pfSense UI package written when I return.
-
-
@michmoor said in Arpwatch not downloading vendor ID's:
I can try out the FreeBSD package.
Thanks. I've included the package below. Please let me know if you have any issues.
Here is the notification script if you want it:
#!/usr/bin/env php <?php require_once("notices.inc"); $timestamp=$argv[1]; $ifname=convert_real_interface_to_friendly_descr($argv[2]); $ipaddr=$argv[3]; $old_hwaddr=$argv[4]; $old_hwaddr_org=$argv[5]; $new_hwaddr=$argv[6]; $new_hwaddr_org=$argv[7]; $hostname = gethostbyaddr($ipaddr); $msg = "ANDwatch notificaton\n\n"; $msg .= sprintf("%22s: %s\n", "timestamp", $timestamp); $msg .= sprintf("%22s: %s\n", "interface", $ifname); $msg .= sprintf("%22s: %s\n", "hostname", $hostname); $msg .= sprintf("%22s: %s\n", "ip address", $ipaddr); $msg .= sprintf("%22s: %s %s\n", "old ethernet address", $old_hwaddr, $old_hwaddr_org); $msg .= sprintf("%22s: %s %s\n", "new ethernet address", $new_hwaddr, $new_hwaddr_org); notify_all_remote($msg); ?>I don't have anything to display a status page yet, but you can do a query via the command line like so:
andwatch-query <ifname>That will give you a report of all the latest IP mappings.
[Edit: Updated pkg to v1.0.1 to fix query bug with MAC addresses beginning with '0']
[Edit: Updated pkg to v1.1.0 to change record update / age behavior. Details on GitHub.] -
Just because I hate packing...
Here are a couple of files that will get you going for a database query UI. Note that since there is no configuration you will need to hand edit the list of interfaces you are running ANDwatch on. The list is near the bottom of andwatch.inc.
/usr/local/pkg/andwatch.inc:
<?php /* * andwatch.inc * * part of pfSense (https://www.pfsense.org) * Copyright (c) 2025 Denny Page * All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ require_once("config.inc"); require_once("functions.inc"); require_once("util.inc"); require_once("service-utils.inc"); function andwatch_query_interfaces($ifnames) { $entries = array(); foreach($ifnames as $ifname) { $real_ifname = get_real_interface($ifname); $friendly_ifname = convert_friendly_interface_to_friendly_descr($ifname); $pipe = popen("/usr/local/bin/andwatch-query $real_ifname", 'r'); if ($pipe) { while ($line = fgets($pipe)) { list($date, $time, $age, $ipaddr, $hwaddr, $org) = sscanf(trim($line), '%s %s %s %s %s %[^$]s'); $hostname = gethostbyaddr($ipaddr); if ($hostname == $ipaddr) { $hostname = ""; } $entry = [ 'ifdesc' => $friendly_ifname, 'datetime' => "$date $time", 'age' => $age, 'hostname' => $hostname, 'ipaddr' => $ipaddr, 'hwaddr' => $hwaddr, 'org' => $org ]; $entries[] = $entry; } pclose($pipe); } } return $entries; } function andwatch_query_all() { $ifnames = array("lan"); //$ifnames = array("lan", "opt2", "opt3"); return andwatch_query_interfaces($ifnames); } ?>/usr/local/www/andwatch_database.php:
<?php /* * andwatch_database.php * * Copyright (c) 2025, Denny Page * All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ require_once("guiconfig.inc"); require_once("andwatch.inc"); $pgtitle = array(gettext('Status'), gettext('ANDwatch'), gettext('Database')); include("head.inc"); $entries = andwatch_query_all(); ?> <div class="panel panel-default"> <div class="panel-heading"><h2 class="panel-title"><?=gettext('Database')?></h2></div> <div class="panel-body table-responsive"> <table class="table table-striped table-hover table-condensed sortable-theme-bootstrap" data-sortable> <thead> <tr class="text-nowrap"> <th><?=gettext("Interface")?></th> <th><?=gettext("DateTime")?></th> <th><?=gettext("Hostname")?></th> <th><?=gettext("IP Address")?></th> <th><?=gettext("MAC Address")?></th> <th><?=gettext("MAC Organization")?></th> </tr> </thead> <tbody> <?php if (count($entries)) : ?> <?php foreach ($entries as $entry): ?> <tr class="text-nowrap"> <td><?=htmlspecialchars($entry['ifdesc'])?></td> <td><?=htmlspecialchars($entry['datetime'])?></td> <td><?=htmlspecialchars($entry['hostname'])?></td> <td><?=htmlspecialchars($entry['ipaddr'])?></td> <td><?=htmlspecialchars($entry['hwaddr'])?></td> <td><?=htmlspecialchars($entry['org'])?></td> </tr> <?php endforeach; ?> <?php else: ?> <tr> <td colspan="6"><?=gettext("No entries to display")?></td> </tr> <?php endif; ?> </tbody> </table> </div> <?php include("foot.inc"); ?> -
FYI, I've edited the original post to update the version of the package to 1.1.0. If you've pulled a prior version, please see the post containing the package to get an update.
