Weird network issue not related to pfSense

KOM

I'm hoping someone can offer some suggestions as I'm starting to run out of ideas.

Among my stable of users are a development group that have very limited network knowledge. One of the devs has a Win7 workstation and vmW 11.1.2 with a number of virtual machines. The network scheme I use is that every dev user is given a block of 20 IP addresses to use for their various virtual machines. His local IP is 10.10.10.1 (PEGATRON NIC chipset), and the VMs use 10.10.10.2-10.10.10.20. Today, after applying Windows Updates and rebooting, his local network stack came up with the autoconfigure IP address of 167.x.y.z and a warning about an IP address conflict with 10.10.10.1. He has never had this error before. I keep tight control over my networks and IP conflicts just don't happen here. When I scan the network, 10.10.10.1 is alive with a MAC address that belongs to VMware instead of the expected PEGATRON, but you can't ping it or connect to it in any way. I unplugged his network cable and I could still see 10.10.10.1 on the network with his workstation as the hostname. I went around and audited every powered-on virtual machine for their MAC address and IP address, but found nothing conflicting. I finally changed his IP address to allow him to work, but this phantom IP is still lingering like a bad smell. Further scans show this IP is still active with the same hostname as the dev's workstation and with a VMware MAC.

I have no idea what's going on here.

phil.davis

Then you need to start in reverse with the scanning device - plug it into just an empty switch, run the scan, make sure it comes up with nothing. Then plug physical cables in until you find the offender (can use the half-split method if there are a lot of cables). Then go to the end of that cable and have a good look at that real system and VMs it is running.

KOM

But I don't think it's real. The LAN scan shows his workstation showing up as both IP addresses, but with different MACs (PEGATRON vs VMware). I'm no network genius, but I don't understand how this situation could even happen, where it appears that a virtual machine has decided to steal an IP address and hostname from a physical machine it lives on. From the image below, you can see the hostname listed twice with different MACs. All our physical machines are using PEGATRON chipset on the NIC, and all our VMs NICs are VMware. Note how one instance of ZERO at 10.10.10.1 has a VMware MAC and the other at 10.10.10.8 has a PEGATRON MAC.

My best guess, barring any other info, is that VMware's network driver somehow took a shit and is conflating the other NIC details and has assumed the host IP address. I don't believe for even a second that one of my users has been playing with network settings at the exact moment that my guy applied updates and rebooted.

ip.png_thumb

bennyc

Well, I'm not buying it that it would be the cause of a nic or a driver ;)

_below is a list of all registered OUIs in IEEE by VMware, Inc. :

00:50:56 - auto generated MAC by vCenter server
3) 00:0C:29 - auto generated MAC on ESXi hosts and products like VMware Workstation
(source: https://communities.vmware.com/message/2450865)_

So yes it is a vmware mac. But my first guess would be that his wks was converted (somewhere in time, as backup?), is running on some host somewhere and it only showed now?
If that's not the case, than I would think the hostname could be wrong.
You probably have better judgment over it than me (as you manage the network), but I don't trust resolved dns names (hostnames) very much.
Who or what does your dns? Did you flush your local dns cache before this test? And how about the aging/scavenging?

Happy hunting ;D

KOM

Who or what does your dns? Did you flush your local dns cache before this test?

DNS is not the issue here, it's the IP conflict with a phantom host.

Unfortunately, I'm not in a position to play around all day while this guy sits & spins. I'll try removing various elements until I can isolate the actual root cause near the end of the day.

bennyc

I understand very well DNS isn't the issue, but IMHO in this case it's unsure if it is helping in your troubleshooting.
I was merely pointing to the fact that your resolved hostname might not be correct.

A hint: use SPAN or RSPAN (or mirror-port or … depending on brand of your switch) to check if you see the mac on his interface on the switch.
If you know how much mac's are allowed to be learned on each port, you could also set port security (violations) on suspected ports.
You should be able to taccle this one... even without much access on his machine.

KOM

You should be able to taccle this one… even without much access on his machine.

Yeah, if I didn't have an idiot for a boss who thought it was a brilliant idea to save money by buying dumb switches. FML.

I just ran a small script to list all the MACs from my vCenter. Nothing there matches.

…

OK, I left this reply up while I tried something else and I solved it. Or, more to the point, I managed to undo my incompetence.

About a month ago, I setup an instance of ESXi 6.0 to do some IPSEC testing with a small dummy network featuring 2 pfSense nodes and a server behind each. I configured IPSEC and it was working like a charm. I even did some performance testing and it looked good. After that I forgot about it. Fast-forward to today when my dev guy reboots his machine and the magic IP conflict suddenly appears.

It turns out that I had given pfSense1 WAN the same IP address as my dev guy. My brain forgot that I was grabbing an IP from my userspace instead of any of the zillions of available subnets I could have used. Once I shut that machine down, everything was good. Now the question is why did I not get a conflict error right away? A month goes by and everyone is happy until this one Win7 box reboots. What's up with that?

bennyc

@KOM:

Yeah, if I didn't have an idiot for a boss who thought it was a brilliant idea to save money by buying dumb switches.

Hope he's not reading along? ;D

@KOM:

I just ran a small script to list all the MACs from my vCenter. Nothing there matches.

Hmm. Doesn't surprise me, the mac address didn't match a vCenter generated mac.
Hint, look for the bold in my first reply ;)

@KOM:

OK, I left this reply up while I tried something else and I solved it.

The most important reply of this topic! 8)

@KOM:

A month goes by and everyone is happy until this one Win7 box reboots. What's up with that?

My guess here, is that pfSense does not check for duplicate ip's on its assigned subnet, and Windows 7 does.
Other then that explenation, I know too little of your setup so that would be speculating heavily. (and I'm not very good at it. Sorry…)

KOM

My guess here, is that pfSense does not check for duplicate ip's on its assigned subnet, and Windows 7 does.

I inherited a 10.10.0.0/16 network (don't ask). As soon as I used 10.10.10.1 for pf1 WAN, it should have squawked about a conflict right there and then. Everything is in the broadcast domain.

I keep tight control over my networks and IP conflicts just don't happen here.

Somehow I just knew this claim would come back to haunt me.

bennyc

@KOM:

I keep tight control over my networks and IP conflicts just don't happen here.

Somehow I just knew this claim would come back to haunt me.

Hey, I was being nice, I didn't even mention it ;D
Just kidding. Relax, be happy that you figured it out…