Arpwatch not downloading vendor ID's

dennypage · Feb 6, 2025, 5:58 PM

@michmoor said in Arpwatch not downloading vendor ID's:

I can try out the FreeBSD package.

Thanks. I've included the package below. Please let me know if you have any issues.

Here is the notification script if you want it:

#!/usr/bin/env php
<?php
require_once("notices.inc");

$timestamp=$argv[1];
$ifname=convert_real_interface_to_friendly_descr($argv[2]);
$ipaddr=$argv[3];
$old_hwaddr=$argv[4];
$old_hwaddr_org=$argv[5];
$new_hwaddr=$argv[6];
$new_hwaddr_org=$argv[7];

$hostname = gethostbyaddr($ipaddr);

$msg = "ANDwatch notificaton\n\n";
$msg .= sprintf("%22s: %s\n", "timestamp", $timestamp);
$msg .= sprintf("%22s: %s\n", "interface", $ifname);
$msg .= sprintf("%22s: %s\n", "hostname", $hostname);
$msg .= sprintf("%22s: %s\n", "ip address", $ipaddr);
$msg .= sprintf("%22s: %s %s\n", "old ethernet address", $old_hwaddr, $old_hwaddr_org);
$msg .= sprintf("%22s: %s %s\n", "new ethernet address", $new_hwaddr, $new_hwaddr_org);

notify_all_remote($msg);
?>

I don't have anything to display a status page yet, but you can do a query via the command line like so:

andwatch-query <ifname>

That will give you a report of all the latest IP mappings.

[Edit: Updated pkg to v1.0.1 to fix query bug with MAC addresses beginning with '0']
[Edit: Updated pkg to v1.1.0 to change record update / age behavior. Details on GitHub.]

andwatch-1.1.0.pkg.zip

dennypage · Feb 5, 2025, 9:59 PM

Just because I hate packing...

Here are a couple of files that will get you going for a database query UI. Note that since there is no configuration you will need to hand edit the list of interfaces you are running ANDwatch on. The list is near the bottom of andwatch.inc.

/usr/local/pkg/andwatch.inc:

<?php
/*
 * andwatch.inc
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2025 Denny Page
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

require_once("config.inc");
require_once("functions.inc");
require_once("util.inc");
require_once("service-utils.inc");

function andwatch_query_interfaces($ifnames)
{
    $entries = array();

    foreach($ifnames as $ifname) {
        $real_ifname = get_real_interface($ifname);
        $friendly_ifname = convert_friendly_interface_to_friendly_descr($ifname);
    
        $pipe = popen("/usr/local/bin/andwatch-query $real_ifname", 'r');
        if ($pipe) {
            while ($line = fgets($pipe)) {
                list($date, $time, $age, $ipaddr, $hwaddr, $org) = sscanf(trim($line), '%s %s %s %s %s %[^$]s');

                $hostname = gethostbyaddr($ipaddr);
                if ($hostname == $ipaddr) {
                    $hostname = "";
                }
 
                $entry = [
                    'ifdesc' => $friendly_ifname,
                    'datetime' => "$date $time",
                    'age' => $age,
                    'hostname' => $hostname,
                    'ipaddr' => $ipaddr,
                    'hwaddr' => $hwaddr,
                    'org' => $org
                ];
    
                $entries[] = $entry;
            }
            pclose($pipe);
        }
    }

    return $entries;
}

function andwatch_query_all()
{
    $ifnames = array("lan");
    //$ifnames = array("lan", "opt2", "opt3");

    return andwatch_query_interfaces($ifnames);
}

?>

/usr/local/www/andwatch_database.php:

<?php
/*
 * andwatch_database.php
 *
 * Copyright (c) 2025, Denny Page
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

require_once("guiconfig.inc");
require_once("andwatch.inc");

$pgtitle = array(gettext('Status'), gettext('ANDwatch'), gettext('Database'));
include("head.inc");

$entries = andwatch_query_all();
?>

<div class="panel panel-default">
    <div class="panel-heading"><h2 class="panel-title"><?=gettext('Database')?></h2></div>
    <div class="panel-body table-responsive">
        <table class="table table-striped table-hover table-condensed sortable-theme-bootstrap" data-sortable>
            <thead>
            <tr class="text-nowrap">
                <th><?=gettext("Interface")?></th>
                <th><?=gettext("DateTime")?></th>
                <th><?=gettext("Hostname")?></th>
                <th><?=gettext("IP Address")?></th>
                <th><?=gettext("MAC Address")?></th>
                <th><?=gettext("MAC Organization")?></th>
            </tr>
            </thead>
            <tbody>
            <?php if (count($entries)) : ?>
            <?php foreach ($entries as $entry): ?>
            <tr class="text-nowrap">
                <td><?=htmlspecialchars($entry['ifdesc'])?></td>
                <td><?=htmlspecialchars($entry['datetime'])?></td>
                <td><?=htmlspecialchars($entry['hostname'])?></td>
                <td><?=htmlspecialchars($entry['ipaddr'])?></td>
                <td><?=htmlspecialchars($entry['hwaddr'])?></td>
                <td><?=htmlspecialchars($entry['org'])?></td>
            </tr>
            <?php endforeach; ?>
            <?php else: ?>
            <tr>
                <td colspan="6"><?=gettext("No entries to display")?></td>
            </tr>
            <?php endif; ?>
            </tbody>
        </table>
</div>
<?php include("foot.inc"); ?>

dennypage · Feb 6, 2025, 6:19 PM

FYI, I've edited the original post to update the version of the package to 1.1.0. If you've pulled a prior version, please see the post containing the package to get an update.

dennypage · Feb 19, 2025, 9:59 PM

For those that are interested, here is a quick update.

Version 2.0.0 adds hostname for notify/query output, and support for user defined additions to the pcap filter. More detail can be found in the README.

I've included an updated pkg below, plus a tarball that contains these supporting files:

/usr/local/pkg/andwatch.inc
/usr/local/pkg/andwatch-notify.php
/usr/local/www/andwatch_database.php

Here is an example of how I am launching for my LAN interface:

/usr/local/bin/andwatchd -s -F 'not net fe80::0/10 and not net fc00::0/7' -n /usr/local/pkg/andwatch-notify.php ix0

I'm beginning work on the pfSense package now.

andwatch-2.0.0.pkg.zip
andwatch_pfsense_files.tgz

michmoor · Feb 24, 2025, 2:48 AM

@dennypage
Hey Denny,
Is it possible to add as part of the notification message what ports and/or vlan the arp message pertains to?
I sometimes get arpwatch messages about an IP that someone incorrectly self assigned to a machine and of course there’s a mac/arp conflict but it would be helpful to know which port this was seen on

dennypage · Feb 24, 2025, 4:02 PM

@michmoor said in Arpwatch not downloading vendor ID's:

Is it possible to add as part of the notification message what ports and/or vlan the arp message pertains to?

By “ports and/or vlan”, do you mean interface? If so, the interface is already part of the notification. If you mean something other than interface, please explain more.

Edit: This is what notifications look like with ANDwatch:

            timestamp: 2025-02-22 16:25:18
            interface: DEVICE
             hostname: myhost.mydomain
           ip address: 192.168.1.80
 new ethernet address: f8:b9:5a:37:1c:1e
     new ethernet org: LG Innotek
 old ethernet address: ac:f1:08:5e:85:24
     old ethernet org: LG Innotek

DEVICE is the interface name as seen in pfSense.

dennypage · Feb 24, 2025, 5:44 AM

Hey all, here is a version of the pfSense ANDwatch package for testing.

You'll need to have the andwatch package itself installed (posted a few days ago here) before installing the pfSense package.

Note that you can remove /usr/local/www/andwatch_database.php as it is no longer used.

Feedback welcome. Bonus points for finding bugs.

pfSense-pkg-ANDwatch-2.0.pkg.zip

michmoor · Feb 24, 2025, 3:27 PM

@dennypage
Yes I mean interface.
So it should say “igc3” or if using vlans “igc3.14”

dennypage · Feb 24, 2025, 4:09 PM

@michmoor said in Arpwatch not downloading vendor ID's:

So it should say “igc3” or if using vlans “igc3.14”

There are three levels of interface names in pfSense:

Friendly interface name -> Internal interface name -> Real interface name.

Some examples:

LAN -> lan -> ix0
GUEST -> opt2 -> igc0.2
DEVICE -> opt3 -> igc0.3

You can see the mapping in Status / Interfaces. The Internal and Real names are shown in parens following the Friendly name.

Pretty much everywhere in the GUI the Friendly name is the one used. If you go to Service / DHCP Server for instance, you will see "LAN", GUEST", "DEVICE", etc. Unless you are defining an interface you generally don't see/use the Internal or Real names. Friendly names are what administrators are most familiar with.

ANDwatch uses the Friendly name in the notification for this reason.

michmoor · Feb 24, 2025, 4:04 PM

@dennypage nice!
Thanks for the quick response Denny

patient0 · Feb 24, 2025, 5:20 PM

@dennypage I installed the two pkg on a 2.7.2 CE (on Proxmox), not sure that's supported at all.

Great work! Looks cool to me :o) ... some feedback:

there is no menu entry in the 'Status' menu, the status can only be access through the service 'ANDwatch'
there are two interface in my installation, LAN and TEST. On the LAN interface it shows a client and pfSense itself, all with the correct IP, MAC and hostname.
For the TEST network the connected client shows up correct but for the pfSense interface for network TEST the hostname is displayed as "(unknown)" (IP 200.1 in the picture). The other client with name "(unknown)" is disconnected.
maybe the icons search and clear icons are taller then they have to be :)

TEST with pfSense name as "unkown"
login-to-view

Missing Status menu entry
login-to-view

dennypage · Feb 24, 2025, 8:43 PM

@patient0 said in Arpwatch not downloading vendor ID's:

there is no menu entry in the 'Status' menu, the status can only be access through the service 'ANDwatch'

Something is really wrong. Did you refresh the pages after the install? There is no way to access Status from the Services / ANDwatch page--did you type in the URL by hand?

If refresh doesn't bring it up, check your config please. Look for the following:

                <menu>
                        <name>ANDwatch</name>
                        <tooltiptext>ANDwatch Settings</tooltiptext>
                        <section>Services</section>
                        <url>/andwatch.php</url>
                </menu>
                <menu>
                        <name>ANDwatch</name>
                        <tooltiptext>ANDwatch Status</tooltiptext>
                        <section>Status</section>
                        <url>/andwatch_status.php</url>
                </menu>

For the TEST network the connected client shows up correct but for the pfSense interface for network TEST the hostname is displayed as "(unknown)" (IP 200.1 in the picture). The other client with name "(unknown)" is disconnected.

If you log into your pfSense box, and run the command "host 10.99.200.104". This should respond with "Host 104.200.99.10.in-addr.arpa not found: 3(NXDOMAIN)"

maybe the icons search and clear icons are taller then they have to be :)

Very strange. You also don't have the icons. It should look identical to the Search panel on the DHCP Status page. Mine (24.11 & 25.03 beta) look like this:

For login-to-view

Two questions:

What browser are you using?
I don't have a 2.7.2 CE system. Can you run the following command on your system and send me the result please?

grep -i btn /usr/local/www/status_dhcp_leases.php

Thanks.

patient0 · Feb 24, 2025, 9:20 PM

@dennypage if it's too much work to support 2.7.2 CE just tell me, yes? Maybe it's easier to wait for 2.8.0 CE? Can only be a few months away.

Did you refresh the pages after the install? There is no way to access Status from the Services / ANDwatch page-

I did log out and in again, yes. The icon to access the status is in the service page, next to the "?":
login-to-view

check your config please. Look for the following

That section appears exactly like that in /usr/local/pkg/andwatch.xml. I assume Netgate made changes to the GUI framework in the newer versions.

If you log into your pfSense box, and run the command "host 10.99.200.104". This should respond with "Host 104.200.99.10.in-addr.arpa not found: 3(NXDOMAIN)"

You are right - although I was talking about 10.99.200.1 - pfSense itself on the TEST interface is not getting resolved. That way ANDwatch can't know the name, of course.

It should look identical to the Search panel on the DHCP Status page

I see, again probably changes to the GUI framework. I'll compare it with other package configs and see if I can see a difference.

What browser are you using?

Firefox 128.7.0esr on Debian 12

grep -i btn /usr/local/.../status_dhcp_leases.php

Below is output of the call:

[2.7.2-RELEASE][root@pfsense.home.arpa]/root: grep -i btn /usr/local/www/status_dhcp_leases.php
                                <a id="btnsearch" title="<?=gettext("Search")?>" class="btn btn-primary btn-sm"><i class="fa fa-search icon-embed-btn"></i><?=gettext("Search")?></a>
                                <a id="btnclear" title="<?=gettext("Clear")?>" class="btn btn-info btn-sm"><i class="fa fa-undo icon-embed-btn"></i><?=gettext("Clear")?></a>
        <a class="btn btn-info" href="status_dhcp_leases.php?all=0"><i class="fa fa-minus-circle icon-embed-btn"></i><?=gettext("Show Active and Static Leases Only")?></a>
        <a class="btn btn-info" href="status_dhcp_leases.php?all=1"><i class="fa fa-plus-circle icon-embed-btn"></i><?=gettext("Show All Configured Leases")?></a>
        <a class="btn btn-danger no-confirm" id="cleardhcp"><i class="fa fa-trash icon-embed-btn"></i><?=gettext("Clear All DHCP Leases")?></a>
        $("#btnsearch").prop('type', 'button');
        $("#btnclear").prop('type', 'button');
        $("#btnsearch").click(function() {
        $("#btnclear").click(function() {
                        $("#btnsearch").get(0).click();

Thanks you for taking the time!

dennypage · Feb 24, 2025, 9:36 PM

@patient0 said in Arpwatch not downloading vendor ID's:

That section appears exactly like that in /usr/local/pkg/andwatch.xml.

Can you check in your actual running config please? Like download it via Backup and see if both sections are in there?

@patient0 said in Arpwatch not downloading vendor ID's:

The icon to access the status is in the service page, next to the "?":

Ah, my bad. I forgot about the shortcuts.

@patient0 said in Arpwatch not downloading vendor ID's:

Below is output of the call:

Yep, that's got it. They changed from "fa" to "fa-solid" a couple of releases ago. Not sure why.

Anyway, if you edit andwatch_status.php and change occurrences of "fa-solid" to just "fa" it should look like it's supposed to.

patient0 · Feb 24, 2025, 10:36 PM

@dennypage said in Arpwatch not downloading vendor ID's:

Can you check in your actual running config please? Like download it via Backup and see if both sections are in there?

That section is not in the backup file, I'll investigate. Installed HAproxy and compared the config, looks really the same confused

dennypage · Feb 25, 2025, 8:20 AM

@patient0 You might try a re-install of the package.

patient0 · Feb 25, 2025, 8:20 AM

@dennypage I dove into pkg create and tried a few things. In the end it was simple as often:

On 2.7.2 CE in andwatch.xml the menu item name for "Status" can't be the same as for "Services". Renaming the <name> tag value of the menu item for Status from "ANDwatch" to "ANDwatch Status" resolved it. It still works works on 25.03-BETA :)

Btw: the value of the <name> tag right after the </copyright> tag seems to be lower case for the other packages I checked (HAproy, Shellcmd, Avahi, LCDproc). Made no functional difference though.

dennypage · Feb 25, 2025, 5:22 PM

@patient0 said in Arpwatch not downloading vendor ID's:

On 2.7.2 CE in andwatch.xml the menu item name for "Status" can't be the same as for "Services". Renaming the <name> tag value of the menu item for Status from "ANDwatch" to "ANDwatch Status" resolved it. It still works works on 25.03-BETA :)

Excellent catch. I'll change that. Thanks.

@patient0 said in Arpwatch not downloading vendor ID's:

Btw: the value of the <name> tag right after the </copyright> tag seems to be lower case for the other packages I checked (HAproy, Shellcmd, Avahi, LCDproc). Made no functional difference though.

I think that one is okay. There are a few on my system that aren't all lower case:

System Patches
RRD Summary
Netgate Firmware Upgrade

dennypage · Feb 25, 2025, 6:46 PM

@patient0 Here is an updated package. You should remove the prior package before installing the new version.

Note that it still has the "fa-solid", so you'll need to make that change again, except the file name to edit is now status_andwatch.php rather than andwatch_status.php in order to match naming convention.

pfSense-pkg-ANDwatch-2.0.pkg.zip

patient0 · Feb 25, 2025, 7:07 PM

@dennypage Thanks a lot, you are awesome :)

For the fa-solid to fa conversion I have created a patch and applied via System patches.

It's attached for reference if someone else would need it:

andwatch-2.0_pfSense-CE-272-status-CSS-fix.diff

In System / Patches, set "Path Strip Count" set to 0, leave the other settings as they are.