@Gabri-91 I have a dynamic connection in pppoe on vlan 835, I have performed all the steps but it doesn't connect using the carp interface.
I double checked all the steps and it seems to be ok, but it doesn't want to connect to the wan...

The "fun": I have one SG3100 without this issue and one WITH this issue.

Executing custom_php_install_command()... Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(783): eval() #5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158 PHP ERROR: Type: 1, File: /usr/local/pkg/haproxy/haproxy.inc, Line: 2158, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(783): eval() #5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrownpkg-static: POST-INSTALL script failed >>> Cleaning up cache... done. Success

The function in the error:

function use_transparent_clientip_proxying() { global $config; $a_backends = getarraybyref($config,'installedpackages','haproxy','ha_pools','item'); if (is_array($a_backends)) { foreach ($a_backends as $backend) { if ($backend["transparent_clientip"] == 'yes') { #### the line from the error message: line 2158 return true; } } } return false; }

@jimp

These two pfSenses are in the middle of network, the issue didn't affect interfaces faced to syslog server, syslog source set as local pfSense interface, not as CARP VIP. We see in syslog other messages like FW rules actions during the issue period, but not CARP-related ones.

@damianhl If it has ZFS there is a Disks widget that can expand to show details:
e80dceed-465d-4da2-9b03-30e91c0a4dcd-image.png

Not sure about hardware RAID, have never used it. Unless FreeBSD/pfSense includes a driver the pfSense OS will probably only be able to see what the BIOS shows it.

DHCPD LOG.txt

@decibel83 A problem at Layer 2 is the most common cause.

FYI- You can disable NAT and route without also disabling the firewall.

Firewall > NAT, Outbound tab, set it to Disable Outbound NAT and save/apply.

@james92 Yes a dumb switch is fine.

@davidredekop Interesting. I have never had to change anything in proxmox for CARP.

As an aside, while fc00::/7 is the ULA network space, fc00::/8 is currently undefined. fd00::/8 is proper ULA addressing. Recommend implementing RFC 4193 and randomly selecting a /48 for ULA usage.

i was able to track down a bit of a solution
we had disabled hardware offloads , this is now turned back on which make xmlrpc sync much quicker and lower load and cpu.

also

we have two wans, on each wan we had two openvpn servers listening for different purposes, 7-8 years ago we were told that its best to listen on localhost with each vpn server, then nat port forward each external port so that each wan can listen on the same server, it appears if we do this now, each time an xmlproc sync occurs it causes pfctl and the reload scripts to thrash and loop 3 or more times.
we this this occuring over and over with localhost
php-fpm[6973]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN

the solution now is listening on a single carp ip, this means were not able to openvpn in the backup wan, but atleast vpn works on master and backup servers, just not the backup wan

all xmlrpc sync is re-enabled and no CARP timeouts so far.......

@nocternal Yup, I'll be doing just that, super nice we can do "micro" patches like this.

Thanks again!!

@antoinef67 life saver! thanks!

Just to add for anyone else coming across this issue.

Adding a vlan and therefore triggering a configuration reload and mini failover, caused exactly the same issue. Which was not fixable with restoring a configuration backup or even a restart of both firewalls.

Applying this patch:
Fix automatic firewall rules for HA DHCP server failover (Requires reboot or filter reload to activate, Redmine #13965)

Fixed the issue with the DHCP server.

The issue showed in Status / DHCP Leases a permanent status of My State - 'Recover', as well as previously mentioned 'communication-interrrupted'

@derelict At the moment, I only have one firewall.

I will add the other one later when I have more WAN addresses.

@tony-soprano
Any switch should support CARP. The protocol simply sends out mulitcasts to talk to the other node. So both has only to be within the same L2 network.

@viragomann
I have read the Netgate documentation regarding Single Address CARP. This brief paragraph states there are significant challenges with updating, etc. I tried to look around for more information on this type of setup. What info I did encounter seemed to be people who were spoofing the WAN MAC Address on the secondary unit and using scripts to determine the interface state(ifup/ifdown) in an attempt to avoid collision.
I would be hesitant to put any real trust in hacks of this nature in a corporate production environment.

Edit: PPPoE is not being used.

@jakub_ Yes. The advertisements are sourced from the interface IP address and CARP MAC.

Not sure why you are seen advertisements from both the primary (advskew 0) and secondary (advskew 100) there.

@viragomann said in Different MAC addresses for virtual ip and WAN interface:

nce both need to communicate with the gateway, you need to state the correct mask for both. That's not gonna to work on different interfaces.

Yes, I am trying that in but I think it goes wrong somewhere it the MAC addresses because I set it VirtualIP but it does not work.

@philippe-richard Hi.

Yes. After converting a few CARP to IP alias, the problem disappear.
Note we have applied the patch #12827.

Hi again.

Enabled promiscuous mode in esxi and now it works.

@louis2

I have been trying 'to fix' the issue 'that the sftp-server sees the proxy address' and not the 'client address'

Since a proxy is forwarding a package, it is not strange that the server at the destination side normally sees the address of the proxy and not that of the client.

Luckily there are protocols which allows the proxy to forward the client address.

So the big question is how:

to enforce HA-proxy to forward the client IP and how to enforce the SSH-server to use the (added) client address

The config is as follows:

pfsense 2.7 actual build ha-proxy haproxy-develop in ssl /https(TCP-mode frontend listening to WAN-address (4/6) port 22 IPV4-frontend and an IPV6-frontend IPV4 and an IPV6 back-end bitvise (advanced) ssh-server (on windows)

That works, no problem apart of the 'lack of client address issue'.

I did a lot of searching on the internet and found options like:

‘option forwardfor’ (usable for the front and/or the back-end) options like 'send-proxy' and 'send-proxy-v2' and for bitvise 'proxy protocol'(disabled or required (default disabled)) and 'Enable UPnP gateway forwarding (on/off (default off))

After reading the links below I decided to try almost all possible options ...... nothing worked.

So if someone has a working config, I would love to know how 😊

I did google a lot. Here some links, which might or might not help (enough)

https://www.haproxy.com/documentation/hapee/latest/load-balancing/client-ip-preservation/add-x-forward-for-header/

https://www.reddit.com/r/PFSENSE/comments/108siet/forwarding_source_ip_from_haproxy/

https://forum.netgate.com/topic/159562/solved-haproxy-forward-client-ip

@louis2 said in HA-proxy: How to forward url-A to url-B?:

In my actual case port 80 and/or port 443 should be forwarded to the corresponding back-end, where the certificate is at the server, not on HA-proxy/pfSense.
(HA-proxy should be transparent for the SSL)

For access control I choose ^Host ends with^ ^.example.com^
PS I assume that 'http-request redirect' can also handle https !?

Sure, it can. But I'm in doubt that HAproxy can read the host header of an encrypted request without having the server certificate and private key for it.
It can read the SNI at its best. But this might only work in TCP mode and then not for unencrypted traffic.

So maybe you can configure a separate HTTP frontend listening on port 80 for unencrypted request. Here you can use a host header ACL doing the http-request redirect.
And configure an additonal SSL/HTTPS TCP mode frontend for port 443, encrypted traffic with an SNI ACL and redirect it to whatever you want.

Well,

I'm trying to replicate the issue in a test environment with a single pfSense box using CARP IPs and a spare Aruba CX switch. Of course I can't 😠

I guess my next step is to actually setup a HA setup and then remove the secondary and see what happens...

@roesh @jimp

I added another new CARP VIP, but the interface stays DOWN ?

@ivan-70 Have a read through https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html. There is a section on outbound NAT.

@huud
Try Status > Filter Reload.
Had a similar issue yesterday as I had a pass rule removed before, and this solved it.

@viragomann

I am going over our entire topic and my settings and documenting them so that I don't get lost in what has been done.

The question below was answered by another user at the beginning of the topic.

Even if I haven't put pfbackup on the network yet, the GW CARP VIP should work?

@kdmiller61

You've posted in the HA/CARP/VIPs forum.
Have a look at the OpenVPN forum.

You will find (yep, you can click, I made live easy today) many posts about NordVPN over there.
True, most if not all are posting if they have question, view are posting with "works great, this is what you should do".

Typically, your VPN should give you a dot opvn file file that looks like :

dev tun fast-io persist-key persist-tun nobind remote netherlands-amsterdam-ca-version-2.expressnetw.com 1195 remote-random pull comp-lzo no tls-client verify-x509-name Server name-prefix ns-cert-type server key-direction 1 route-method exe route-delay 2 tun-mtu 1500 fragment 1300 mssfix 1200 verb 3 cipher AES-256-CBC keysize 256 auth SHA512 sndbuf 524288 rcvbuf 524288 auth-user-pass <cert> -----BEGIN CERTIFICATE----- MIIDTjCCAregAwIBAgIDKzZvMA0GCSqGSIb3DQEBCwUAMIGFMQswCQYDVQQGEwJV .... removed .... H6bGeePdhvB4+ZbW7VMD7TE8hZhjhAL4F6yAP1EQvg3LDA== -----END CERTIFICATE----- </cert> <key> -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAqzmLfyjotrjAxnr96V4PI9UjuCf+BFVgxe7yXCq9o62Zag/8 .... removed .... p7/1CulE0z3O0sBekpwiuaqLJ9ZccC81g4+2j8j6c50rIAct3hxIxw== -----END RSA PRIVATE KEY----- </key> <tls-auth> # # 2048 bit OpenVPN static key # -----BEGIN OpenVPN Static key V1----- 48d9999bd71095b10649c7cb471c1051 .... removed .... 7aed27125592a7148d25c87fdbe0a3f6 -----END OpenVPN Static key V1----- </tls-auth> <ca> -----BEGIN CERTIFICATE----- MIIF+DCCA+CgAwIBAgIBATANBgkqhkiG9w0BAQ0FADCBhDELMAkGA1UEBhMCVkcx DDAKBgNVBAgMA0JWSTETMBEGA1UECgwKRXhwcmVzc1ZQTjETMBEGA1UECwwKRXhw .... removed .... 1xKbeC0kzhuN2gsL5UJaocBHyWK/AqwbeBttdhOCLwoaj7+nSViPxICObKrg3qav GNCvtwy/fEegK9X/wlp2e2CFlIhFbadeXOBr9Fn8ypYPP17mTqe98OJYM04= -----END CERTIFICATE----- </ca>

When you set up your client on pfSense, you use whatever you can find to enter into the GUI fields.
The remainder goes into the Custom Configuration :

1de618b5-ac49-474e-a6b9-119cdbe4b2f0-image.png

Before you begin, use your favourite web search site, and look up 10 examples that show you how 'others' did it.
Search for NordVPN examples.
with the pfSense version you use, that will be 2.6.0 CE or pfSense Plus 22.05 or more recent.
Find these : what are the identical steps. What are the steps that one example shows, and is absent in the other(s).

Also : look at the original Youtube > Netgate channel.

@cs_l
No. I prefer a unique VHID for each CARP VIP though, but for proper function they only have to be unique within an layer 2 network.

@viragomann

Status > DHCP Leases

Pool Status
7c9ddad5-07aa-4d91-ac82-fbaf5ca332b7-image.png

It is now as "recover state". Regardless if I put pfbackup on the network, with Failover peer IP configured, it is still in "recover state".