MAC-Address of outgoing packets

[wb-munzinger]

Hey there,

I use pfSense with CARP since quite some time without any problems on a DSL-Connection.
We recently purchased a second Internet-Connection through a cable-provider

The provider asked for the MAC-Addresses for the static IPs. I told him the ones of the carp addresses.
So:
IP1 - Firewall 1 Interface MAC-Address (aa:aa:aa:aa:aa:aa)
IP2 - Firewall 2 Interface MAC-Address (bb:bb:bb:bb:bb:bb)
IP3 - 00:00:5e:00:01:2d (vhid 45)
IP4 - 00:00:5e:00:01:2e (vhid 46)
and so on.

The provider only routes outgoing packets if they come from the correct IP-MAC combination.

Now the problems seems to be that outgoing packets aren't sent with the VIP-Mac (00:00:5e:00:01:2d) as source but the hardware interface MAC (aa:aa:aa:aa:aa:aa).

A Wireshark dump looks something like this:
1. foreign IP (Provider Router MAC) –------------------> IP3(00:00:5e:00:01:2d) Syn <-- Packet successfully reaches our server
2. foreign IP (Provider Router MAC) <-------------------- IP3(aa:aa:aa:aa:aa:aa) Ack <-- Provider doesn't route the packet because of the wrong MAC-Address

Is it the expected behavior of CARP-VIPs that they sent packets with the real Interfaces MAC-Address or is there a way to change this?
I tried the net.link.ether.inet.carp_mac sysctl but this seems to only change the behavior for ARP-packets.

I'm somehow stuck now and don't see a way to use the connection with any kind of HA.

pfSense version is 2.1-RC0.
Thanks in advance!

[jimp]

That is the expected behavior of CARP VIPs. Outgoing traffic comes from the NIC MAC, incoming goes to the CARP VIP MAC.

As far as I'm aware there isn't a way to change that, I've never heard of a setup being so strict that it's been a problem.

[b0rman]

I found same problem with MACs today :(
Will be glad for any ideas…

Seems like same problem here: http://forum.pfsense.org/index.php?topic=59060

[b0rman]

@jimp:

That is the expected behavior of CARP VIPs. Outgoing traffic comes from the NIC MAC, incoming goes to the CARP VIP MAC.

As far as I'm aware there isn't a way to change that, I've never heard of a setup being so strict that it's been a problem.

Why net.link.ether.inet.carp_mac key can't fix it?

[jimp]

I'm not 100% sure how that sysctl operates. It may only affect traffic sourced by the firewall for that IP, and not traffic leaving that has had NAT applied. (Basically it may vary depending on if it was generated by the OS, or routed via pf)

[b0rman]

So there is no chance to work with such kind of IP-MAC binding?

[wb-munzinger]

You can find the source of the net.link.ether.inet.carp_mac patch here:
https://github.com/pfsense/pfsense-tools/blob/master/patches/RELENG_8_3/carp_correct_mac.diff
As far as I understand, it only changes the MAC-Address of outgoing ARP-Pakets so it doesn't solve the problem.

We managed to get around the problem by putting a linux box between the pfsense firewall and the router.
It has no ip configuration but a bridge(like a hub) and rewrites the mac-addresses of the outgoing packets.
The configuration looks something like this:

brctl addbr br0
brctl addif br0 eth0 eth1 eth2

ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
ifconfig br0 up

ifconfig eth0 0.0.0.0 promisc
ifconfig eth1 0.0.0.0 promisc
ifconfig eth2 0.0.0.0 promisc

ebtables -t nat -A POSTROUTING -p IPv4 --ip-src <ip3>-j snat --to-src 00:00:5e:00:01:2d
ebtables -t nat -A POSTROUTING -p IPv4 --ip-src <ip4>-j snat --to-src 00:00:5e:00:01:2e</ip4></ip3>