ARP 00:11:22:ab:cd:ee is using my IP address
I recently purchased a qotom q330G4 mini PC and installed pfsense 2.4.5-RELEASE (amd64) on it. Since then my "WAN" interface keeps disconnecting after ~15-20 mins of a fresh restart and I'm seeing something that can only be described as "flapping" of the WAN interface. For example, when trying to
ping 126.96.36.199, I see ~48-49% failed packets (out of ~600 transmitted packets)
The error seems to indicate that there is a "CIMSYS" based device that has the same IP as that of my firewall but I can assure you that there is no "CIMSYS" device connected on the "WAN" network. The only device I have on my so-called WAN interface is my pfsense box (and my audio system, a playbar and a sub, which I've yet to move behind the firewall (and both have static ips as well) )
Internet === ISP router (10.0.0.1) === (10.0.0.189) Pfsense === (192.168.10.0/24)lan/ wifi, (192.168.11.0/24)OPT1, (192.168.12.0/24)OPT2.
The pfsense box contains 4 interfaces (igb0,1,2,3). I've been using
Things that I have tried:
- I've verified the arp table on the pfsense box and I do not see the CIMSYS mac address anywhere in its table.
- I ran
arp -aon my laptop (while connected to the ISP router) and I do not see the CIMSYS entry in there either. The mac address corresponding to 10.0.0.189 is that of the pfsense.
- Moreover, I connected my laptop to the ISP router and ran
nmap -sP -PI -PT 10.0.0.1/24but couldn't find CIMSYS there either.
I took some packet captures on pfsense after I started seeing the WAN disconnection issue and filtering for
arppackets, this is what I see:
0.270436 Technico_xx:xx:xx Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.1 1.354452 Technico_xx:xx:xx Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.1 2.090278 CIMSYS_ab:cd:ee Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.189 2.090286 Gifa_yy:yy:yy CIMSYS_ab:cd:ee ARP 42 Gratuitous ARP for 10.0.0.189 (Reply) (duplicate use of 10.0.0.189 detected!) 2.350016 Technico_xx:xx:xx Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.1 3.349759 Technico_xx:xx:xx Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.1 4.434406 Technico_xx:xx:xx Broadcast ARP 60 Who has 10.0.0.246? Tell 10.0.0.1
Tehnicois the ISP router (masked last 3 octets)
Gifais the pfsense box (masked last 3 octets)
CIMSYSdevice mac address is displayed as-is.
We can see this "CIMSYS" device suddenly comes into picture repeating the ARP request from the ISP router and asking it to be reported to the IP of the Pfsense box.. this made me feel its the Pfsense box doing something unusual but at the same time, I see a GARP from Pfsense informing the CIMSYS device that it owns the
Also, the MAC address for the CIMSYS device is very interesting --
00:11:22:ab:cd:ee. I've been chasing ghosts since a few hours and I'm looking for some help to identify or narrow down the issue.
If this "CIMSYS" really want 10.0.0.189, let's him have it.
How does your pfSense WAN interface get's IP info ? DHCP ? Change it for static, or set up a static DHCP lease in your ISP router, so pfSense obtains 10.0.0.2 from it.
I had initially set pfsense to
10.0.0.2and got same issue.. I was like fine, I'll move my pfsense to
10.0.0.10.. again the same issue (both times, I verified and double checked that there is no other device on either IP addresses).
I thought lets prioritize the issue and let CIMSYS obtain the IP it wants.. so I configured pfsense to obtain IP via DHCP so that I can let ISP router's dhcp server give pfsense a suitable IP... but still.. the same error.
I have packet captures for when the IP was set to
10.0.0.10and it looks exactly the same.
Is your pfSense WAN connected directly with the ISP box or to a switch? Can you try without or another switch?
No, there is no switch between the ISP box and pfsense.
The ISP box has 2 ethernet interfaces and pfsense box is directly plugged in to one of them (and other one is not connected to anything)
The qotom box : what were you using before ? If could you use another box - ancient PC - VM, what ever, the "CIMSYS" fellows goes away ?
If so, who's, except for pfSense, "living" this "qotom" box ? Who makes it ?
I'm not sure about the qotom box tbh. It has a decent rep in home-labbers community as a quiet mini pc. The reviews on the amazon suggests quite a few people using it as their pfsense box (since its an i3 and offers AES-NI which aids in VPN especially).
I wish I could test things on a spare box. Unfortunately, I don't have another box lying around on which I can spin up a pfsense instance either virtual or on baremetal.
However, about VM, I guess I can get a bit creative and run a pfsense VM instance on my laptop and bridge the ISP connection via my laptop + a host-only network to it, then have another linux VM on that host-only network and see if it can reach the internet or if it faces a similar issue.
I'll try this experiment in about 9-10 hours and post the update here.
In the meantime, Im attaching some screenshots to aid the issue in the topic:
Cheers and thanks for suggestions!
And (as @Gertjan already stated)...check the BIOS settings. There is no reference to a management interface on their support page.
dotdash last edited by
Why don't you try switching the WAN port to igb1 and see if the problem goes away?
Wife is working from home and is using the only monitor I have so once she is done, I will try and take a look at the network settings in BIOS.
I have tried using "all" the interfaces for WAN (on separate occasions,
igb3) but keep getting the same issue every time. Exact same mac address for that "CIMSYS" thing..
Btw, quick update, I have been running pfsense in VM on virtualbox for about an hour or so and I'm not seeing this CIMSYS issue yet. So yeah, I think everything is pointing towards the qotom box being the issue. I'm surprised no one else is seeing this given its popularity as a pfsense box..
Okay, had the VM running pfsense the entire afternoon, not a single CIMSYS issue. Moreover, I've been trying to filter for that CIMSYS device's mac address in my ISP-router-provided-LAN-network (10.0.0.0/24 aka the WAN in this issue's context)
tshark -f "ether host 00:11:22:ab:cd:ee"but ... nothing, zip, zilch, nada !!!.
Issue is now most likely with the box.
I tried looking in the BIOS (its an American Megatrend bios) settings. I could see the 4 Intel
I211chipset based NICs and didn't find anything suspicious in their settings:
The gist is:
MAC Address: <matches what I'm seeing in pfsense. No reference to CIMSYS> WOL: enabled Link Speed: Auto Negotiated Adapter PBA: ?
The only strange part was that the
Adapter PBAis missing (seeing a ? next to it) but I think thats beyond this issue.
The only configurable parameter for me is the WOL and LinkSpeed. Im kind of certain it'll not help but I'll try flipping off WOL and see.
Wait wait.. Looks like I spoke too early..
Look what did I find:
❯ tshark -f "ether host 00:11:22:ab:cd:ee" Capturing on 'Wi-Fi: en0' 1 0.000000 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 188.8.131.52 2 2.457790 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 184.108.40.206 3 7.065529 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 220.127.116.11 4 8.294748 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 18.104.22.168 5 17.510315 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 22.214.171.124 6 37.479151 CIMSYS_ab:cd:ee → Broadcast ARP 42 Who has 10.0.0.205? Tell 126.96.36.199
Things. just. got. interesting!
en0 Wifi ?
Yeah, I connected my laptop to the ISP router's wifi and ran the capture on the LAN interface (which was WAN for the pfsense box) in the afternoon.
There are quite a few packets in the capture with destination IP in the ARP being some public IP addresses. It doesn't make sense. I think the issue is with the wife's laptop (at the time of capture, this IP was assigned to her laptop). She will investigate more tomorrow and I will post an update in this thread based on our findings.
But, fwiw, its "definitely" not a pfsense issue!
Kill de ISP router Wifi ?
I just remembered that I didn't close the loop here.
So, it turns out my wife's company uses some L2 VPN and due to a server misconfiguration, I was seeing the vpn client on her laptop misbehave. She raised a ticket with their IT and the rest is beyond our control.
As far as the issue in my network, after turning off the ISP router's wifi and putting all our devices behind pfsense box, I'm not seeing those issues any more.
Phew! The moment I was about to turn off the capture I saw the smoking gun. I was almost getting ready to call the device malicious and return it.
Thanks for helping look into this issue guys. Much appreciated!