Forward traffic on same interface / VEPA bridging
I am operating a server hosting a set of services, each run in a separate Docker container. In addition, there is a KVM running pfSense acting as firewall. The firewall has a physical interface that is connected to the external network and a virtual network card that is connected to the internal container network, using MACVLAN Docker-side, so each container has its own IP address, but all of them are in the same subnet.
For security reasons, the containers need to be isolated and shall not be able to communicate with each other principally (just with the external network). For this, MACVLAN is configured in VEPA mode, which allows traffic from and to the parent device, but not to other addresses on the same parent device.
Now, I would like to allow specific traffic between specific containers, so pfSense must act as a VEPA bridge/router, considering the configured firewall rules (read, if incoming traffic on the internal interface matches a PASS rule it shall be forwarded to a host on the same interface / same subnet).
I can't seem to get that scenario working (no traffic between the hosts on the internal interface, traffic from and to the external network works as expected). Any ideas on how to proceed from here?
To be honest, I do not even known we this is not working in the first place.
If a packet arrives at the interface pfSense should forward the package according to the routing table, which will cause the package to go out on the same interface again.
Is there any configuration item the prevents such scenarios (like "filter traffic on own interface" or alike)?
Interestingly enough, pfSense does not even reply to the ARP request:
[root@server ~]# ip r default via 10.0.20.1 dev server proto static metric 410 10.0.20.0/24 dev server proto kernel scope link src 10.0.20.2 metric 410
21:52:49.651286 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:50.673895 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:51.697860 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28 21:52:52.721992 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28
I would assume a response with the MAC of the 10.0.20.0/24 interface. The trace was made on the firewall on that interface (PING from the firewall to 10.0.20.4 works as expected).
When adding the entry manually I can see the ICMP echo request, but no reply:
[root@server ~]# arp -s 10.0.20.4 02:42:0a:00:14:04 10.0.20.4 ether 02:42:0a:00:14:04 CM server
22:00:21.403515 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 1, length 64 22:00:22.450162 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 2, length 64 22:00:23.473790 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 3, length 64 22:00:24.497803 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 4, length 64
The interface has a
IPv4 ICMP PASS * *rule configured.