@viragomann Thanks for your input!
Issue solved!
It was issue basically frame untagged on switch of particular VLAN, so after tagging it works and able to connect secondary and sync!

@hege Late reply on this topic, but relevant.

Hyper-V SR-IOV implementation does NOT support mac spoofing with SR-IOV

Technical;

Mac spoofing is required for CARP because the mac address is changed on outbound packets, that's part of CARP.

Hyper-V natively does not allow outbound packets through the virtual switch from a Hyper-V guest that does not have the exact same mac address as assigned to the virtual machine (unless you enable the "allow mac spoofing" checkbox.

SR-IOV technically can allow mac spoofing, this is all there in the IEEE specification for for this is to work, but, quite simply Microsoft Hyper-V doesn't implement it.

Therefore you need to enable "allow mac spoofing" and forego SR-IOV or VMQ network accelerate functions.

@derelict said in high availability w/ redundant layer 2 switches causing loop on my test network:

People call all sorts of things a "lagg."

Very true - its a kind of a catch all.. I was thinking lacp, which yeah you need a stack..

@viragomann Thanks for your help!

Problem isolated and solved

Working with a hosting provider and not having access to the underlying configuration layer means things get lost in translation.

The problem is/was Forged Transmits in the ESX environment that needed to be disabled so the CARP IP on the PFSense can create multiple MAC addresses and send/receive on these.

First paragraph here says it, but not having access and poor communication with hosting provider makes it difficult to debug by one self.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability-virtual.html?highlight=vmware

@pirateparley

last bump before giving up!

A bug for the issue has been raised.

@derelict said in Upgrade to 2.5.0, now seeing 224.0.0.18 CARP blocks:

@defunct78 It is your virtual environment improperly echoing back the CARP advertisements. They are being properly blocked by that rule.

https://kb.vmware.com/s/article/59235

https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability-virtual.html#changing-net-reversepathfwdcheckpromisc

That was it. Fixed the problem perfectly. Thanks.

Hello, realize this is an older thread but looking to gain insight on the subject as well.

I have a /26 public IP block, and currently use ProxyARP and 1:1 NAT to route traffic to Hyper-V VMs/web servers. I'll be adding subnets using VLANs to further isolate some new VMs. Is there any reason I should be using IP Aliases instead, or is ProxyARP fine for this application?

Thanks for any enlightenment!

FWIW, I looked through the startup scripts, and there is a script you can add - /etc/rc.custom_boot_early (make sure you chmod a+x on it). That seems to be a reasonable place for renaming interfaces, so I created it, and added, e.g.:

#!/bin/sh /sbin/ifconfig em0 name igb0 /sbin/ifconfig em1 name igb1

That seems to have done the trick for me. Clearly, I need to save this script and re-apply it on upgrades and such.

I have no idea what the official intent of this script is, but it would be nice if Netgate would formalize something like this - even if the onus is on us to maintain it across upgrades. To be clear, this appears to be part of pfSense, and not part of FreeBSD.

@mmangiante
You may simply do that. VIPs are basically independent from interface IPs. They may moreover cohere with the WAN gateway.

@mmangiante said in How safely change vip and their interface ip:

If I simply change the ip on the interfaces and then update the vip ip I have done all or I have to change every NAT rule, every page that use that ips, the ipsec vpn?

This depends on how you've configured your rules and services. If you used a variable as destination, for instance "WAN VIP", there is nothing to do. You only have to change the WAN VIP and you're ready. However, if you entered the IP explicitly, you will have to change it now as well.

Issue fixed, I juste forgot to check this on my backup node...

90fd1a45-7445-470d-b737-83c90cf19d05-image.png

Also make sure all the pfsense are not on persistent CARP maintenance mode.

@derelict I may have found the problem. Possibly a corrupt or failing disk.

I replaced the disk on the backup node today, rebuilt and and restored configs from a previous (recent) backup file. Everything looks fine now.

I will keep monitoring in case the problem reoccurs, but it may be something as simple as this.

A really strange symptom if it is in fact a failing disk. SMART status was OK, so perhaps some corruption from the recent power outage that took out my primary firewall disk.

For anyone else who may experience this issue, try rebooting with the disk repair option, and/or change out the disk and rebuild/restore.

Thanks for your help and guidance.

I Just wanted to update, came in today and just reset both machines to factory and started again, all seems to be working, fine. So I must have done something wrong or out of order. But thanks to all who commented.

@bennyc Thank you

@zerodeux You could have a single transit link to a layer 3 switch and have it route your 250 VLANs.

All in all, an HA firewall with 250 interfaces is going to be work. It is also going to generate heartbeat traffic for all the first-hop redundancy VIPs. That is true for CARP, VRRP, or HSRP.

This seems to be identical to:

https://forum.netgate.com/topic/161152/strange-problem-dhcp-failover-after-upgrade-to-2-5-0-xmlrpc-bug

Solution:
https://redmine.pfsense.org/issues/11519

@viragomann Thanks for your reply. Currently, I can't reach CARP IPs, I don't know where I'm wrong, CARP IPs of LAN is 172.16.100.4. I only can ping CARP IPs of WAN 10.84.100.4

and if I create master 10.84.3.2, slave 10.84.3.3 with VLAN 3. After set up that you can add 10.84.3.1 as CARP VIP on the master. I cannot ping as well.

@prk
You can do that all with Firewall > NAT > Outbound. Switch it into hybrid mod, then you can add rules to override the default behaviour (masquerading).

If you strict want to forward a public IP to a certain internal and have this internal IP use that public, you can use NAT 1:1 rules.

However, before you have to assign each IP out of the additional /29 subnet in Firewall > virtual IPs as type "IP Alias" to your WAN.

Still an issue with 2.5 by the way...

@jimp said in Can I find out the status of the CARP interface (BACKUP / MASTER) through a command?:

ifconfig -a | grep 'carp:'

This solution looks perfect, I only made one adjustment to get only the MASTER / BACKUP:

UserParameter = pfsense.carp.state, ifconfig -a | grep 'carp:' | cut -d '' -f2 | sed -n 1p

Sed is for taking only one CARP interface, it is very rare for one interface to be BACKUP while the others are MASTER, and vice versa.

PS: I don't know if I should close this post as resolved or how to do it if I should.

@lucazio

Hi,
what you want is net.inet.carp.preempt.

The preempt shold be enabled. That means if one interface is failing on a pfSense then ALL Interface do a failover not only one.

Also bare in mind I have seen some complications with carp and multicast on the esxi and the security settings of the protgroup / swtich. (Multicat - promismode / ARP address Change)

Once I was able to properly google for things I already know I didn't know I found this.

https://forum.netgate.com/topic/35849/accessing-wan-s-public-ip-from-the-lan-not-working-please-help/6

Split DNS worked like a charm for me!

Might need to enable reflection in the future but for now it the DNS method works fine.

@free4 said in Alert "XMLRPC method captive_portal_sync" in 2.5:

@jimp Oh ok

But wait...What's the point to backport the fix into RELENG_2_5_0 then ?

So it will be included in the next patch release, whenever that may be.

@mr_jinx
You can configure a failover group with the WAN gateway and the others box LAN interface.
So on the secondary you have to add the primarys LAN address as a gateway first. Then add a gateway failover group where you set the WAN GW as tier 1 and the pirmarys LAN IP as tier 2.
So now if the WAN GW is not accessible (cause the primary owns the WAN CARP) it goes out over the primary.

You can do the same on the primary with the secondarys LAN IP to retrieve updates when it's in CARP maintenance mode.

@viragomann
I already tried to build a chain, but dont like this aproach.
If one is temporary not reachable the update gets lost for all others in the chain and you'll not notice it.

@chrisnz
Hello, being of two distinct networks which, I think, should not be able to communicate with each other, the solution is to add an interface to the pfSense router, in your case not physical.
Since your switch is web managed the best thing you can do is to create a VLAN dedicated to the Guest network and use the switch for all your private connectivity. And only for those!
You will find everything you need in the pfSense and Netgear documentation, in the respective sections that talk about VLANs.
Googling I found this which looks a lot like the recommended solution:
pfSense router-on-a-stick VLAN configuration with a Netgear GS108E
I hope it will be useful to you.

@bimpe said in HA/CARP, with DHCP error:

https://forum.netgate.com/topic/106394/dhcp-not-working-properly-solved

The XMLRPC process will automatically add +100 to each skew when synchronizing the VIPs to the secondary node.

skew on second server with DHCP is more than 20 by ifconfig | grep carp ?

@koby-peleg-hen said in Another XMLRPC communication error:

ALL I want to achieve is 2 nodes on Heztner Cloud that can be sync between them for easy management

Sync is always primary to standby, never "to each other" or "between them". So I'd be careful with that. If you just want the config to be synced but no HA why sync at all? Just to have the same Aliases? If you don't run HA you commonly have other NICs/Interfaces or additional Interfaces and rules, syncing that to another node with a whole different setup makes no real sense to me?

@derelict said in VHIDs with two CARP HAs in the same LAN network?:

The CARP MAC address is derived from the VHID. This also applies to VRRP on the same segment.
You must use unique VHIDs on the same broadcast domain or you will experience MAC address collisions.

Hi, i will try it with unique VHIDs and let you know my results! Thanks for your fast help, regards Norbert!