@jimp Yes, once the secondary pfsense has some CARP virtual IP addresses, the error vanishes.

Deselecting the "Virtual IP" under "Select options to sync" does not work in the end, if one likes to also use DHCP failover.

@ivan-70 Have a read through https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html. There is a section on outbound NAT.

@huud
Try Status > Filter Reload.
Had a similar issue yesterday as I had a pass rule removed before, and this solved it.

@viragomann

I am going over our entire topic and my settings and documenting them so that I don't get lost in what has been done.

The question below was answered by another user at the beginning of the topic.

Even if I haven't put pfbackup on the network yet, the GW CARP VIP should work?

I must have grabbed the wrong ID when I copied/pasted, I edited in the correct one. The ID I had in there was the merge commit from when I merged that fix into the plus tree, not the publicly accessible one.

@kdmiller61

You've posted in the HA/CARP/VIPs forum.
Have a look at the OpenVPN forum.

You will find (yep, you can click, I made live easy today) many posts about NordVPN over there.
True, most if not all are posting if they have question, view are posting with "works great, this is what you should do".

Typically, your VPN should give you a dot opvn file file that looks like :

dev tun fast-io persist-key persist-tun nobind remote netherlands-amsterdam-ca-version-2.expressnetw.com 1195 remote-random pull comp-lzo no tls-client verify-x509-name Server name-prefix ns-cert-type server key-direction 1 route-method exe route-delay 2 tun-mtu 1500 fragment 1300 mssfix 1200 verb 3 cipher AES-256-CBC keysize 256 auth SHA512 sndbuf 524288 rcvbuf 524288 auth-user-pass <cert> -----BEGIN CERTIFICATE----- MIIDTjCCAregAwIBAgIDKzZvMA0GCSqGSIb3DQEBCwUAMIGFMQswCQYDVQQGEwJV .... removed .... H6bGeePdhvB4+ZbW7VMD7TE8hZhjhAL4F6yAP1EQvg3LDA== -----END CERTIFICATE----- </cert> <key> -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAqzmLfyjotrjAxnr96V4PI9UjuCf+BFVgxe7yXCq9o62Zag/8 .... removed .... p7/1CulE0z3O0sBekpwiuaqLJ9ZccC81g4+2j8j6c50rIAct3hxIxw== -----END RSA PRIVATE KEY----- </key> <tls-auth> # # 2048 bit OpenVPN static key # -----BEGIN OpenVPN Static key V1----- 48d9999bd71095b10649c7cb471c1051 .... removed .... 7aed27125592a7148d25c87fdbe0a3f6 -----END OpenVPN Static key V1----- </tls-auth> <ca> -----BEGIN CERTIFICATE----- MIIF+DCCA+CgAwIBAgIBATANBgkqhkiG9w0BAQ0FADCBhDELMAkGA1UEBhMCVkcx DDAKBgNVBAgMA0JWSTETMBEGA1UECgwKRXhwcmVzc1ZQTjETMBEGA1UECwwKRXhw .... removed .... 1xKbeC0kzhuN2gsL5UJaocBHyWK/AqwbeBttdhOCLwoaj7+nSViPxICObKrg3qav GNCvtwy/fEegK9X/wlp2e2CFlIhFbadeXOBr9Fn8ypYPP17mTqe98OJYM04= -----END CERTIFICATE----- </ca>

When you set up your client on pfSense, you use whatever you can find to enter into the GUI fields.
The remainder goes into the Custom Configuration :

1de618b5-ac49-474e-a6b9-119cdbe4b2f0-image.png

Before you begin, use your favourite web search site, and look up 10 examples that show you how 'others' did it.
Search for NordVPN examples.
with the pfSense version you use, that will be 2.6.0 CE or pfSense Plus 22.05 or more recent.
Find these : what are the identical steps. What are the steps that one example shows, and is absent in the other(s).

Also : look at the original Youtube > Netgate channel.

@cs_l
No. I prefer a unique VHID for each CARP VIP though, but for proper function they only have to be unique within an layer 2 network.

@viragomann

Status > DHCP Leases

Pool Status
7c9ddad5-07aa-4d91-ac82-fbaf5ca332b7-image.png

It is now as "recover state". Regardless if I put pfbackup on the network, with Failover peer IP configured, it is still in "recover state".

@jaredadams
An http frontend doesn't accept IP addresses for comprehensible reasons.
This might only work in tcp mode.

@viragomann

Good, so this is the way it should work then, then i can stop suspecting this was related to the issues at hand, thx !

@alcamar said in HAProxy Weiterleitung zum nextcloud-Server:

Kann das hier dokumentieren, falls es ähnliche unbedarfte wie mich in Zukunft gibt.

Warum nicht? Nachdem der Threadtitel schon darauf hinweist, könnten Leute das finden.

Allerdings ist deine Konstellation wohl eher eine Seltenheit. Wie gesagt, üblicherweise läuft ein Webserver heutzutage nicht in einem virtuellen Verzeichnis.
Wenn HAproxy würde ich alle Anfragen einfach weiterleiten lassen und den Rest den Backendserver machen lassen. Dafür gibt es jede Menge Anleitungen.

Ich kämpfe aber noch mit Zertikaten beim CALdav. Eigentlich müsste nur die pfsense Zertifikate jonglieren, oder?

So wäre es wünschenswert. Funktioniert leider nicht immer. Ich weiß aber nicht, wie das bei Nextcloud ist. Meine betreibe ich nicht hinter einem Proxy.

Aber bezüglich DAV und HAproxy habe ich schon Threads gesehen. Aber ich denke, hier würdest du mit den beiden Suchbegriffen im Netz rascher brauchbare Ergebnisse finden als hier.
Die könnte man dann auf die Konfiguration in der pfSense GUI "übersetzen".

Deinen Punkt hinsichtlich Sicherheit des Ports 443 habe ich mir für die nächsten Überlegungen vorgemerkt.

Wenn du es geschafft hast, dass HAproxy die Anfragen in das virtuelle Verzeichnis von Nextcloud leitet und keine anderen zulässt, sollte es soweit eh sicher sein. Abgesehen natürlich, dass dir klar sein muss, dass die Nextcloud im Internet steht und damit ordentliche Zugangspasswörter braucht und aktuell gehalten werden muss.

@mrpete Well, there you go. No CARP VIP no status.

If you have MASTER/MASTER then you need to fix the layer 2 between that interface on both nodes.

And here's something from the log...

DHCPDISCOVER from d0:94:66:4b:51:b6 via vtnet9: peer holds all free leases

I figured out that CARP uses the formula 3 * (advbase + (advskew / 256)) to detect that MASTER got down. This information is in OpenBSD ifconfig man.

Taken together, the advbase and advskew indicate how frequently, in seconds, the host will advertise the fact that it considers itself master of the virtual host. The formula is advbase + (advskew / 256). If the master does not advertise within three times this interval, this host will begin advertising as master.

So, from my dumps, as the interval that MASTER didn't advertise is about 5 seconds, I'd adjusted advbase to 2. Then, BACKUP will become MASTER only after 6 seconds without see advertisements from MASTER.

Although the firewall has a good hardware, it delays some seconds to apply new configurations. I think 6s for failover is fine here. It seems that advbase = 1 is not always possible.

Thanks!
:)

Thanks all for the suggestions. Digging into it...

After fiddling, I've found out that LACP must be turned off for the corresponding switch ports for it to work.

On Dell S4148T-ON the syntax for non-LACP link aggregation is:

interface ethernet 1/1/23 channel-group 123

while LACP link aggregation is:

interface ethernet 1/1/23 channel-group 123 mode active

My misunderstanding came from thinking link aggregation always involved LACP, even for static setups.

Next step will be to go to the data center, unplug cables, and see what happens, but at least, for now, STP does not detect loops and does not block traffic to the router.

@steveits thanks I will try that.
I will try to convert all my NAT CARP interface to IP aliases of the WAN.

@viragomann said in Can't get to Internet from LAN VIP:

the default gateway doesn't accept upstream traffic from this subnet

Yeah, I asked the data center this question (again), and that was it. 🙄 So apparently it was routing inbound but not allowing replies or outbound.

Thanks for being a sounding board.

@rcoleman-netgate OK thanks Ryan. That was my suspicion. Just trying to get all the rules set up before launch.

@jamiegb said in PfSense in Azure:

The solution is to use a load balancer for the inside interfaces with preference for the primary, using maybe a port probe on port 22 to check VM availability, and an external load balancer to perform a similar probe on the outside WAN interfaces of the PfSense VMs.

Tried this, but doesn't work this way. Not if the internal load balancer using the internal pfSense interfaces as backends.

Reason: the health probe source IP is always 168.63.129.16 for all load balancers (internal and external). Hence pfSense is sending responses to the inside interface health check out to WAN, to the default gateway.

The only workarounds I figured out are either

stating the outside pfSense interface as backend for the internal load balancer or assigning a gateway to the internal interface to lead pfSense to add the reply-to tag to the connection.

Also I could not find out, how to state a specific backend as preference. This option seems to be removed.
Instead Azure provides a so-called "gateway load balancer" nowadays, but this requires that the load balancer can communicate with the NVA by VxLAN protocol, which is sadly actually not supported by pfSense.

@seeking-sense said in Moving from 5 static IP to only 1. : (:

Are there any third party service that "tunnels" static / public IPv4 addresses? Likely it would be cost prohibitive if there is such an animal.

What do you want to tunnel and how should this work?

The thing is, there can only be a single service listen on the single port and IP.
So you have to declare what do your need exactly. What does this mean:

VM #1 Web, VM #2 Mail, VM #3 NAS, etc...

I guess you can run all these services on different ports on pfSense WAN address, apart from "web" (HTTP/S, port 80 and 443). The latter you can treat with the HAproxy package.
HAproxy can look into the HTTP host header and can redirect certain host names to different backend servers.
This works pretty well.

@jasjitchopra It used to be the case to sync states, but not since 22.01, see
https://docs.netgate.com/pfsense/en/latest/highavailability/pfsync.html#pfsync-and-physical-interfaces

Edit: to clarify, the hardware doesn't have to be identical for HA, but the interfaces/interface order needed to be. If the routers have the same packages and usage then it would help if they are reasonably close in spec.

@rico Thank you!
I configured the second WAN with SFP cable on IX0 interface

Ok everything is ok now.
The sync problem was a bad rule on pfsync interface.
Thanks again for your help and have a nice week end

You must allow for MAC Address changes, Promiscious MOde, and Forged Transmits on the port group to the VM for any interface that uses CARP. I created a single trunk portgroup that has these settings and only use it for my pfSense box.

@amoschb said in HA - NO ENOUGH WAN IP Addresses:

So the question is: if only 1 available WAN IP, can we build a HA pfsense?

Yes but the backup won't be active until the primary fails. It is also not supported by TAC so if you have issues, purchase TAC support, and come to us and we see that config we won't touch anything related to a HA issue with it.

Sometimes you can get your ISP to give you a single static and let you have two DHCP addresses (for the WAN on the two HAs) and go that route.

Hi
thanks to the help.

I have made a HA between Pfsense plus firewall and Pfsense CE for backup .
in production i have many problem like some network disconnection .
My pfsense plus firewall has a lot of rules , nat inbound and outbound, 46 VLAN , 6 public VIP .
My question is when deploying the HA , the backup firewall must be empty of rules befor activate HA to let the Xmlrpc sync rules automaticaly exept rules of sync interfaces ?
because i have exported config xml then i changed what i want like ip interfaces to match the prerequisties oh HA, but rules are the same on both firewall .

thank you