ARP Moved from messages on virtualized pfsense

shinjuku

Hello,

I have a PC server on Debian 10 for a homelab, which I use to virtualize a pfsense using KVM.

The host is connected to the LAN network with the motherboard network interface.
I installed a Intel I350-t4 PCIe card to add 4 network interfaces to the server and use them on pfsense VM through bridge.

Then, the virtualized pfsense uses 3 ports of the PCIe card, first one is connected to the LAN, and the 2 others to 2 different WAN.

All interfaces are configured on the host as bridge to use them with KVM as bridge network and a dedicated MAC address.

The host network interfaces are configured like this :

eno1 MAC xx:xx:xx:c3:22:e0 inet manual (Connected on LAN)
enp1s0f0 MAC xx:xx:xx:0a:e6:ce inet manual (Connected on LAN)
enp1s0f1 MAC xx:xx:xx:0a:e6:cf inet manual (Connected on WAN1)
enp1s0f2 MAC xx:xx:xx:0a:e6:d0 inet manual (Connected on WAN2)

And all bridges :

br0 MAC xx:xx:xx:c3:22:e0 master eno1 inet static 192.168.100.102/24
br1 MAC xx:xx:xx:0a:e6:ce master enp1s0f0 inet manual
br2 MAC xx:xx:xx:0a:e6:ce master enp1s0f1 inet manual
br3 MAC xx:xx:xx:0a:e6:ce master enp1s0f2 inet manual

Pfsense is configured to use br1 as LAN interface like this :

vtnet0 MAC xx:xx:xx:56:99:01 inet static 192.168.100.111/24

All is working fine and the pfsense is the gateway of my local network and bring access to Internet to all my devices.

The behavior I want to understand is the following. If I check the ARP table on the pfsense, I can see this :

# arp 192.168.100.102
foobar.domain.com (192.168.100.102) at xx:xx:xx:c3:22:e0 on vtnet0 expires in 1017 seconds [ethernet]

The pfsense can see the host through the motherboard interface configured on LAN at 192.168.100.102 and the MAC address matches the MAC of the motherboard interface. All logic.

Then I force a delete on the arp table of the pfsense and then ping the host to update the ARP table :

# arp -d 192.168.100.102
192.168.100.102 (192.168.100.102) deleted

# arp 192.168.100.102
192.168.100.102 (192.168.100.102) -- no entry

# ping -c1 192.168.100.102
PING 192.168.100.102 (192.168.100.102): 56 data bytes
64 bytes from 192.168.100.102: icmp_seq=0 ttl=64 time=0.917 ms

--- 192.168.100.102 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.917/0.917/0.917/0.000 ms

# arp 192.168.100.102
foobar.domain.com (192.168.100.102) at xx:xx:xx:c3:22:e0 on vtnet0 expires in 1198 seconds [ethernet]

Nothing weird here, but when I do this, at the same moment I get the following message on pfsense syslog:

arp: 192.168.100.102 moved from xx:xx:xx:0a:e6:ce to xx:xx:xx:c3:22:e0 on vtnet0

It seems that for a very short time, the host motherboard interface uses the MAC address of the first ethernet of the PCIe card connected together on the same LAN (on the same switch).
OR, for a very short time, the pfsense sees the wrong MAC address for 192.168.100.102.

Maybe this is normal, but if so, I don't understand why this is normal and what I'm doing wrong. On the host, the motherboard interface are configured to get static IP address, so I really don't understand why pfsense detects the IP 192.168.100.102 on MAC xx:xx:xx:0a:e6:ce as this is a MAC of the PCIe card.

I can reproduce the arp message in syslog by deleting arp entry manually and force update as showed above, but this message appears each time the arp entry expired.

I pretty sure that those messages seems normal and I'm doing something wrong, or maybe just need to check the ARP Handling box in Advanced > Networking... I don't know, just want to understand because I like to learn how network works :)

Thank you for reading and thank you in advance for your help.

shinjuku

I did a Packet Capture on ARP to see what happen, and I can see that when I manually delete the entry 192.168.100.102, then the pfsense ask how is this IP address and both MAC address answer.

13:02:21.392609 ARP, Request who-has 192.168.100.102 tell 192.168.100.111, length 28
13:02:21.392899 ARP, Reply 192.168.100.102 is-at xx:xx:xx:0a:e6:ce, length 28
13:02:21.392954 ARP, Reply 192.168.100.102 is-at xx:xx:xx:c3:22:e0, length 46

But I don't know why. I think this is not related to pfsense, just KVM and/or network behavior that I don't understand.

shinjuku

Found it!

On the Debian host, I have to configure the parameter sysctl "arp_ignore". The possible value I found are :

0 - (default): reply for any local target IP address, configured on any interface
1 - reply only if the target IP address is local address configured on the incoming interface
2 - reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface
3 - do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied
4-7 - reserved
8 - do not reply for all local addresses

As I have 2 interfaces on the host which are connected on the same switch, both interface answer their MAC for the IP 192.168.100.102 (default behavior).

I changed the parameter to value 2 as I want that the IP address is configured on the incoming interface AND want to be on the same subnet.
So, I ran the following command on the debian host:

sysctl -w net.ipv4.conf.default.arp_ignore=2

And now, no more ARP message on the pfsense, and a new capture shows that only 1 interface answer to the ARP request.

13:14:53.331523 ARP, Request who-has 192.168.100.102 tell 192.168.100.111, length 28
13:14:53.331933 ARP, Reply 192.168.100.102 is-at xx:xx:xx:c3:22:e0, length 46

So, sorry all, this was not a pfsense problem, but I learnt something today :)

johnpoz

@shinjuku said in ARP Moved from messages on virtualized pfsense:

0 - (default): reply for any local target IP address, configured on any interface

So let me get this right - will have to look into this.

This setting means that if you have 2 interfaces.. Lets say - in completely different L2 networks. With different L3 ranges

int1, 192.168.100.100/24
int2, 192.168.200.200/24

And for whatever reason you see an arp for 192.168.200.200 on interface 1.. It will respond with the mac of interface 2? And with the mac of int1, even though int1 doesn't have the target IP on it?

That seems like a horrible "default".. I would think the default should be the setting 2..

Do you have say a Carp IP, or Vip or Loopback of some kind configured for this 100.102 address?

shinjuku

Good question.
I use another virtual machine on Virtualbox to simulate this case.

I attached 2 network interfaces on the VM both bridged to the same host's interface. So they are both connected on the same "virtual" switch and network.

My DHCP gives them an IP address, 192.168.100.141 and 192.168.100.226. The VM is like this :

# ip addr

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:78:d8:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.141/24 brd 192.168.100.255 scope global dynamic enp0s3
valid_lft 86394sec preferred_lft 86394sec
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:78:d9:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.226/24 brd 192.168.100.255 scope global dynamic enp0s8
valid_lft 3594sec preferred_lft 3594sec

So from the host, I can ping the VM on 192.168.100.141 and 192.168.100.226.
And if I do an arping (ARP request) on the first IP for example, yes, both interface answer their MAC :

# arping 192.168.100.141
ARPING 192.168.100.141
60 bytes from 08:00:27:78:d8:01 (192.168.100.141): index=0 time=402.140 usec
60 bytes from 08:00:27:78:d9:01 (192.168.100.141): index=1 time=455.373 usec

So, yes, the default behavior for ARP response on Linux can be tricky.
Hopefully, I think it is really rare to plug 2 network interfaces of a server to the same switch/network/subnet. I think this kind of configuration only happen when you do virtualization.

shinjuku

I didn't read carefully, and didn't see that in your question you ask for 2 separate subnets

int1, 192.168.100.100/24
int2, 192.168.200.200/24

Subnet1 192.168.100.0/24
Subnet2 192.168.200.0/24

So, I changed my test but the result is the same.

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:78:d8:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.141/24 brd 192.168.100.255 scope global dynamic enp0s3
valid_lft 86383sec preferred_lft 86383sec
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:78:d9:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.141/24 brd 192.168.200.255 scope global enp0s8
valid_lft forever preferred_lft forever

# arping 192.168.100.141
ARPING 192.168.100.141
60 bytes from 08:00:27:78:d9:01 (192.168.100.141): index=0 time=377.666 usec
60 bytes from 08:00:27:78:d8:01 (192.168.100.141): index=1 time=427.840 usec

But this is a configuration on virtualized environment, without any smart/manageable switch. All my PC/servers are connected to netgear non manageable gigabyte switch.
I think, when you have this kind of complex configuration (router, multi-subnet, ect.), you may need manageable switch which can separate VLAN and use spanning tree and others features like that. I don't really know because I don't have a manageable switch for the moment, but I'm looking to buy one to test :)