I have (hopefully) solved my immediate problem by marking the Tier 2 gateway in the group that we use most as "Disable Gateway Monitoring Action" so that if the Tier 1 gateway is down pfSense will never take the Tier 2 gateway down. This should be fine for our most used gateway group but it is inappropriate for other groups that we occasionally use. If/when we switch to using another gateway group I'll have to remember and change that setting on that gateway.

It seems to me that the various monitoring and threshold settings should be defined in the gateway group and would override those on the gateway, when the gateway is used as part of a group. That would allow me to configure each group as it makes sense and then switch between them with ease.

sound like you have asymmetrical mess if your gateway is going to be out your lan interface. Why don't you draw up your network and point out exactly what your trying to do..

ption use non-local gateway through interface specific route

How is it you would be hitting a "gateway" that is not on the same network?

@kpa Here is parts of my configs:
Client
0_1530578086947_5d158d98-e6c7-4eb1-ae50-31fef1ec71e6-изображение.png
Server ccd
0_1530578176972_e3ed8f4c-afaf-4e63-808a-c1c90d55680d-изображение.png
Server server.conf
0_1530578210375_a56d5260-f8e8-45d2-9835-f8f3b837f919-изображение.png
Server log before client connected
0_1530578288578_ced1dbe8-b9e6-48de-ac6f-f855f726f04c-изображение.png
Server log after client connected
0_1530578689401_c9afca24-742f-47a6-a23e-ca8501004b07-изображение.png
Client log

Jul 3 07:45:33 openvpn 60153 Initialization Sequence Completed Jul 3 07:45:33 openvpn 60153 /usr/local/sbin/ovpn-linkup ovpnc2 1500 1570 10.10.0.2 255.255.255.0 init Jul 3 07:45:33 openvpn 60153 /sbin/route add -net 10.10.0.0 10.10.0.1 255.255.255.0 Jul 3 07:45:33 openvpn 60153 /sbin/ifconfig ovpnc2 10.10.0.2 10.10.0.1 mtu 1500 netmask 255.255.255.0 up Jul 3 07:45:33 openvpn 60153 do_ifconfig, tt->did_ifconfig_ipv6_setup=0 Jul 3 07:45:33 openvpn 60153 TUN/TAP device /dev/tun2 opened Jul 3 07:45:33 openvpn 60153 TUN/TAP device ovpnc2 exists previously, keep at program end Jul 3 07:45:33 openvpn 60153 ROUTE_GATEWAY CLIENT_EX_IP/255.255.255.192 IFACE=em0 HWADDR=00:0c:29:6c:7e:79 Jul 3 07:45:33 openvpn 60153 Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:33 openvpn 60153 Incoming Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key Jul 3 07:45:33 openvpn 60153 Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:33 openvpn 60153 Outgoing Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: route-related options modified Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: --ifconfig/up options modified Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: timers and/or timeouts modified Jul 3 07:45:33 openvpn 60153 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.10.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.10.0.2 255.255.255.0' Jul 3 07:45:33 openvpn 60153 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1) Jul 3 07:45:32 openvpn 60153 [server] Peer Connection Initiated with [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:32 openvpn 60153 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA Jul 3 07:45:32 openvpn 60153 VERIFY OK: depth=0, C=RU, ST=TO, L=Tomsk, O=Kireva, OU=IT_dept, CN=server, name=oneandoneserver, emailAddress=winmasta@kireva.com Jul 3 07:45:32 openvpn 60153 VERIFY EKU OK Jul 3 07:45:32 openvpn 60153 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication Jul 3 07:45:32 openvpn 60153 Validating certificate extended key usage Jul 3 07:45:32 openvpn 60153 VERIFY KU OK Jul 3 07:45:32 openvpn 60153 VERIFY OK: depth=1, C=RU, ST=TO, L=Tomsk, O=Kireva, OU=IT_dept, CN=Kireva CA, name=oneandoneserver, emailAddress=winmasta@kireva.com Jul 3 07:45:31 openvpn 60153 TLS: Initial packet from [AF_INET]VPN_SERVER_EXT_IP:PORT, sid=446f96a7 9c4b7ab0 Jul 3 07:45:31 openvpn 60153 UDPv4 link remote: [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:31 openvpn 60153 UDPv4 link local (bound): [AF_INET]CLIENT_EXT_IP:0 Jul 3 07:45:31 openvpn 60153 Socket Buffers: R=[42080->42080] S=[57344->57344] Jul 3 07:45:31 openvpn 60153 TCP/UDP: Preserving recently used remote address: [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:31 openvpn 60153 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:31 openvpn 60153 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:31 openvpn 60153 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Jul 3 07:45:31 openvpn 60153 MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client2.sock Jul 3 07:45:31 openvpn 60082 library versions: OpenSSL 1.0.2m-freebsd 2 Nov 2017, LZO 2.10 Jul 3 07:45:31 openvpn 60082 OpenVPN 2.4.4 amd64-portbld-freebsd11.1 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Oct 8 2017 Jul 3 07:45:31 openvpn 60825 SIGTERM[hard,] received, process exiting Jul 3 07:45:31 openvpn 60825 /usr/local/sbin/ovpn-linkdown ovpnc2 1500 1570 10.10.0.2 255.255.255.0 init Jul 3 07:45:31 openvpn 60825 Closing TUN/TAP interface Jul 3 07:45:31 openvpn 60825 event_wait : Interrupted system call (code=4) Jul 3 07:45:29 openvpn 60825 MANAGEMENT: Client disconnected Jul 3 07:45:29 openvpn 60825 MANAGEMENT: CMD 'status 2' Jul 3 07:45:29 openvpn 60825 MANAGEMENT: CMD 'state 1' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: Client disconnected Jul 3 07:45:29 openvpn 60825 MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock Jul 3 07:45:29 openvpn 7621 MANAGEMENT: CMD 'status 2' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: CMD 'state 1' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock 111

Are the lines bonded?

Finally! For some reason it didn't work to set a rule allowing traffic to the destination IP for the proxy. Opening for port 80 to any destination fixed it!

@kpa said in Routing table with policy-based routing:

It's more like that the routing process uses information tagged on to the packets traversing the system to detect if a set of packets need special handling and bypasses the normal routing table when it sees those special tags. The firewall rules that match the incoming traffic apply these special tags to the incoming packets.

Gotcha, that makes more sense. Thanks for the explanation!

If I understand you correctly, you want users hitting the external link to be directed to the internal.

This is usually handled by a DNS service.

In pfSense if you are using the DNS Resolver, a host override should suffice.

Services -> DNS Resolver -> General Settings -> Host Overrides -> Add

Host: testcompany
Domain: sytes.net
IP Address: 192.168.88.88
Description: my site override

Save

Apply

Test

Of course this will redirect not only that page, site, http, but any request to that host incl https and any other protocol trying to hit that host name.

With above info from you I contacted again the ISP and it's finally clear... Had indeed to install LACP ( LAGG ) on OPT3 and OPT4 and all is working now in my test environment.
They do the VRRP on their side and just bring 2 cables to our rack (aggegration and redundancy in case of cable problem). So problem solved thanks to your help!
Highly appreciated @Derelict !
Thanks!
Jan

@kpa said in Domain/hostname based routing?:

All correct but the document makes no mention of policy based routing on the outgoing direction which is not possible in pfSense, normal rules or floating rules. PBR on the inbound direction works just fine with floating rules just like it does with normal rules.

Oh I just assume that PBR is just a firewall action like pass/drop so if you can apply firewall you can PBR. Looks like things are a bit more complex.

Anyway if Proxy2 is setup on a dedicate VM instead of pfsense then it should work? It might be a bit too complicated though.

Ok, it's working now that I've disabled the NAT rule, not sure what was wrong before...

@gr1pen said in OpenVPN routing issue?:

After comparing these two setups I found that pfSense seems to create a "client to server" config and not a "site to site" config when selecting "Peer to peer (SSL/TLS)" in the GUI. I have tried to recreate it and confirmed this...

Not a bug. As @kpa mentioned it creates a site-to-multi-site configuration by default in SSL/TLS mode.

If you want a basic site-to-site config with SSL/TLS you can do that, but you must manually define a tunnel network that has a /30 subnet mask so that it only includes two endpoints (pfSense and VyOS in this case).

Thanks a lot i will try it and give you a feedback as soon as i can.
👍

not sure if im supposed to manually create an SA for the bearer traffic (between 192.168.0.0/22 and 192.168.255.0/24) to go along with the SA I created between the BGP peer IPs?

I noticed I was not getting any encrypted traffic out my wan interface when trying to ping from 192.168.0.0/22 to 192.168.255.0/24, so I did add an additional SA between 192.168.0.0/22 and 192.168.255.0/24 in pfsense, and now I do see encrypted traffic when I ping, but still no routes in netstat -nr, so this leaves me a bit concerned as to whether/not Ill have good BGP routing resilience in the first place...

@tsho_admin
Yes, you need to add 10.2.1.0/24 to the phase 2 on site A as well, so that the IPSEC tunnel is aware of the addresses for the OpenVPN network.

no problem glad you got it sorted.. See how short threads can be when decent amount of info and drawing to show how all connected given ;)

Wish more posts were like yours for detailed information when asking for help.

@caltommo said in HELP APPRECIATED** 3G/4G Modem as WAN Interface?!:

Is there an alternative? It doesn’t have to be 100% reliable ...

You mean as unreliable as your main internet connection? Be prepared that it fails the exact moment your regular connection is down already.
There is no place for cheap when you need a backup for failsafe operation. Or vice versa, if it has to be cheap then it's not needed.

I had positive results with this device
https://www.amazon.co.uk/D-Link-DWR-921-Router-abnehmbare-Antennen/dp/B00BN36NMM