ARP 00:11:22:ab:cd:ee is using my IP address

mohitsharma44

Hi folks,

I recently purchased a qotom q330G4 mini PC and installed pfsense 2.4.5-RELEASE (amd64) on it. Since then my "WAN" interface keeps disconnecting after ~15-20 mins of a fresh restart and I'm seeing something that can only be described as "flapping" of the WAN interface. For example, when trying to ping 8.8.8.8, I see ~48-49% failed packets (out of ~600 transmitted packets)

The error seems to indicate that there is a "CIMSYS" based device that has the same IP as that of my firewall but I can assure you that there is no "CIMSYS" device connected on the "WAN" network. The only device I have on my so-called WAN interface is my pfsense box (and my audio system, a playbar and a sub, which I've yet to move behind the firewall (and both have static ips as well) )

Current setup:

Internet === ISP router (10.0.0.1) === (10.0.0.189) Pfsense === (192.168.10.0/24)lan/ wifi, 
(192.168.11.0/24)OPT1, (192.168.12.0/24)OPT2.

The pfsense box contains 4 interfaces (igb0,1,2,3). I've been using igb3 interface for WAN, igb0 for LAN and igb2, 4 as OPT1 and OPT2.

Things that I have tried:

I've verified the arp table on the pfsense box and I do not see the CIMSYS mac address anywhere in its table.
I ran arp -a on my laptop (while connected to the ISP router) and I do not see the CIMSYS entry in there either. The mac address corresponding to 10.0.0.189 is that of the pfsense.
Moreover, I connected my laptop to the ISP router and ran nmap -sP -PI -PT 10.0.0.1/24 but couldn't find CIMSYS there either.

I took some packet captures on pfsense after I started seeing the WAN disconnection issue and filtering for arp packets, this is what I see:

0.270436	Technico_xx:xx:xx	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.1
1.354452	Technico_xx:xx:xx	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.1
2.090278	CIMSYS_ab:cd:ee	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.189
2.090286	Gifa_yy:yy:yy	CIMSYS_ab:cd:ee	ARP	42	Gratuitous ARP for 10.0.0.189 (Reply) (duplicate use of 10.0.0.189 detected!)
2.350016	Technico_xx:xx:xx	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.1
3.349759	Technico_xx:xx:xx	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.1
4.434406	Technico_xx:xx:xx	Broadcast	ARP	60	Who has 10.0.0.246? Tell 10.0.0.1

Tehnico is the ISP router (masked last 3 octets)
Gifa is the pfsense box (masked last 3 octets)
CIMSYS device mac address is displayed as-is.

We can see this "CIMSYS" device suddenly comes into picture repeating the ARP request from the ISP router and asking it to be reported to the IP of the Pfsense box.. this made me feel its the Pfsense box doing something unusual but at the same time, I see a GARP from Pfsense informing the CIMSYS device that it owns the 10.0.0.189 IP.

Also, the MAC address for the CIMSYS device is very interesting -- 00:11:22:ab:cd:ee. I've been chasing ghosts since a few hours and I'm looking for some help to identify or narrow down the issue.

Any ideas?

Gertjan

Hi,

If this "CIMSYS" really want 10.0.0.189, let's him have it.
How does your pfSense WAN interface get's IP info ? DHCP ? Change it for static, or set up a static DHCP lease in your ISP router, so pfSense obtains 10.0.0.2 from it.

mohitsharma44

Hi!
I had initially set pfsense to 10.0.0.2 and got same issue.. I was like fine, I'll move my pfsense to 10.0.0.10.. again the same issue (both times, I verified and double checked that there is no other device on either IP addresses).
I thought lets prioritize the issue and let CIMSYS obtain the IP it wants.. so I configured pfsense to obtain IP via DHCP so that I can let ISP router's dhcp server give pfsense a suitable IP... but still.. the same error.

I have packet captures for when the IP was set to 10.0.0.10 and it looks exactly the same.

Rico

Is your pfSense WAN connected directly with the ISP box or to a switch? Can you try without or another switch?

-Rico

mohitsharma44

Hi Rico,
No, there is no switch between the ISP box and pfsense.
The ISP box has 2 ethernet interfaces and pfsense box is directly plugged in to one of them (and other one is not connected to anything)

Gertjan

Hummm.
The qotom box : what were you using before ? If could you use another box - ancient PC - VM, what ever, the "CIMSYS" fellows goes away ?
If so, who's, except for pfSense, "living" this "qotom" box ? Who makes it ?

mohitsharma44

I'm not sure about the qotom box tbh. It has a decent rep in home-labbers community as a quiet mini pc. The reviews on the amazon suggests quite a few people using it as their pfsense box (since its an i3 and offers AES-NI which aids in VPN especially).

I wish I could test things on a spare box. Unfortunately, I don't have another box lying around on which I can spin up a pfsense instance either virtual or on baremetal.

However, about VM, I guess I can get a bit creative and run a pfsense VM instance on my laptop and bridge the ISP connection via my laptop + a host-only network to it, then have another linux VM on that host-only network and see if it can reach the internet or if it faces a similar issue.

I'll try this experiment in about 9-10 hours and post the update here.

In the meantime, Im attaching some screenshots to aid the issue in the topic:

Gateway status
System logs

Cheers and thanks for suggestions!

Gertjan

@mohitsharma44 said in ARP 00:11:22cd:ee is using my IP address:

CIMSYS

I tend to say this 'thing' is build into qotom. The "owner" of 00:11:22:xxxxx isn't Google. It's Chinese.
Does the box has a BIOS ? If so, start flipping off most things related to it's NIC's.

jmbraben

@mohitsharma44
As @Gertjan has stated...it really sounds like the "CIMSYS" is living in your box...currently you're using igb3 for WAN...what if you move WAN to another interface on the box?

And (as @Gertjan already stated)...check the BIOS settings. There is no reference to a management interface on their support page.

dotdash

Why don't you try switching the WAN port to igb1 and see if the problem goes away?

mohitsharma44

Wife is working from home and is using the only monitor I have so once she is done, I will try and take a look at the network settings in BIOS.

I have tried using "all" the interfaces for WAN (on separate occasions, igb0,1,2 and finally igb3) but keep getting the same issue every time. Exact same mac address for that "CIMSYS" thing..

Btw, quick update, I have been running pfsense in VM on virtualbox for about an hour or so and I'm not seeing this CIMSYS issue yet. So yeah, I think everything is pointing towards the qotom box being the issue. I'm surprised no one else is seeing this given its popularity as a pfsense box..

mohitsharma44

Okay, had the VM running pfsense the entire afternoon, not a single CIMSYS issue. Moreover, I've been trying to filter for that CIMSYS device's mac address in my ISP-router-provided-LAN-network (10.0.0.0/24 aka the WAN in this issue's context) tshark -f "ether host 00:11:22:ab:cd:ee" but ... nothing, zip, zilch, nada !!!.

Issue is now most likely with the box.
I tried looking in the BIOS (its an American Megatrend bios) settings. I could see the 4 Intel I211 chipset based NICs and didn't find anything suspicious in their settings:
The gist is:

MAC Address: <matches what I'm seeing in pfsense. No reference to CIMSYS>
WOL: enabled
Link Speed: Auto Negotiated
Adapter PBA: ?

The only strange part was that the Adapter PBA is missing (seeing a ? next to it) but I think thats beyond this issue.

The only configurable parameter for me is the WOL and LinkSpeed. Im kind of certain it'll not help but I'll try flipping off WOL and see.

mohitsharma44

Wait wait.. Looks like I spoke too early..
Look what did I find:

❯ tshark -f "ether host 00:11:22:ab:cd:ee"
Capturing on 'Wi-Fi: en0'
    1   0.000000 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150
    2   2.457790 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150
    3   7.065529 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150
    4   8.294748 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150
    5  17.510315 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150
    6  37.479151 CIMSYS_ab:cd:ee → Broadcast    ARP 42 Who has 10.0.0.205? Tell 3.218.96.150

Things. just. got. interesting!

Gertjan

en0 Wifi ?

mohitsharma44

Yeah, I connected my laptop to the ISP router's wifi and ran the capture on the LAN interface (which was WAN for the pfsense box) in the afternoon.

There are quite a few packets in the capture with destination IP in the ARP being some public IP addresses. It doesn't make sense. I think the issue is with the wife's laptop (at the time of capture, this IP was assigned to her laptop). She will investigate more tomorrow and I will post an update in this thread based on our findings.

But, fwiw, its "definitely" not a pfsense issue!

Gertjan

Kill de ISP router Wifi ?

mohitsharma44

I just remembered that I didn't close the loop here.

So, it turns out my wife's company uses some L2 VPN and due to a server misconfiguration, I was seeing the vpn client on her laptop misbehave. She raised a ticket with their IT and the rest is beyond our control.
As far as the issue in my network, after turning off the ISP router's wifi and putting all our devices behind pfsense box, I'm not seeing those issues any more.

Phew! The moment I was about to turn off the capture I saw the smoking gun. I was almost getting ready to call the device malicious and return it.

Thanks for helping look into this issue guys. Much appreciated!

Cheers!