@michmoor Thanks so much for the response. After further digging, it turns out that I was just being dumb and not paying attention to what I was doing. I was jumping from adapter to adapter, connection to connection running around the building with my laptop. I didn't realize that the dock I had plugged into left me wireless.

Once I figured out I was chasing my tail and wired myself, I was able to all but max out the connection at 35MB/s. I also looked and one site is 500/500 and the other is 300/300., so the 35MB/s makes complete sense.

With that said, I'm going to bump up the second site to 500/500 on Monday. Any suggestions on how to pair the IPsec settings with the settings for the cryptographic hardware?

@aryanrai
Did you add firewall rules to the IPSec interfaces to allow access from the other site?

Or do you try to ping the LAN device in the other network? In this case you have to ensure that the device also allows access from a remote network. For testing disable its firewall.

Thanks @jimp - I found that out with additional reading, so changed tack and now am using OpenVPN with ESET Secure Authentication, which works well and provides convenient push authentication.

Just want to reply here my discoveries, to save people the hassle of attempting this to find out it does not work, there are two types of GRE tunnels, GRETAP and GRETUN, one supports layer 2 features such as broadcast/multicast and one does not, the PFSense implementation appears to use the later which does not support this feature, please see the following article to show the difference

https://developers.redhat.com/blog/2019/05/17/an-introduction-to-linux-virtual-interfaces-tunnels#:~:text=While%20GRE%20tunnels%20operate%20at,header%20in%20the%20inner%20header.

You would need a local UDP relay instead (on the client side) to instead allow the client to relay these broadcast message as unicast to a specific host, I struggled with this for Windows File Sharing (WS-Discovery) broadcast packets and ended up resorting to a script that auto maps all network drives on successful client connection, perhaps someone could get this working with a L2TP on top of Wireguard?

https://github.com/sparky3387/automapwireguard - Shameless plug of the automap script if someone else also needs this.........

That is expected and noted in the GUI:

c93e825d-26f0-4859-99eb-5d883a0f76d3-image.png

There is also more detail in the docs:

https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/configure-p1.html#ike-endpoint-configuration

@gorberus is the local unit able to route to internet using the second card?

@Topogigio the problem persists. After a few days pfSense stops binding the IP address on the established tunnel interface.
I've started to build a new opnSense gateway, but if there is some pfSense solution I'll be happy

@jto82 You may use OSPF, not static, than it will be easy

@Moses_Kabungo said in Setting up a site-to-site IPSec tunnel with a Vendor who needs to reach us via a public IP other than the WAN address:

no CHILD_SA built

From this error message, I"d assume that there is something wrong with the phase 2 configuration. But I don't know, what you're set there.

@HKFEVER
Fail, if I try to connect Office's pfsense IPsec from WIN11 through Home router gateway with NordVPN on!
OK, if I connect Office's pfsense IPsec from WIN11 through Home router gateway with NordVPN off :)

But then after connected:

WIN11's gateway becomes Office's pfsense default gateway. which don't exit out through Office's pfSense's NordVPN setup! If I un-checked "Use default gateway on remote network" in WIN11's ADVANCE TCP/IP Setting, then the gateway will become WIN11's NIC gateway. Which in theory, I can use NordVPN app in WIN11. I didn't try yet, as too busy :(

Here is the new question:
How can I set the WIN11's Internet request to go through "home or some cafeshop's" gateway to Office's pfSense and exit out to internet through Office pfsense's NordVPN setup?

I have send too long to figure out the rules in pfSense and still no go. May be need to find professional help :(

Okay, I took jimp's advice, and after some struggling with syntax, I was able to get past the NO_PROP error (to run into a different error right behind it).

Anyway, to help someone else with the NO_PROP error, I'll document what I did.

I looked in the /var/etc/ipsec/swanctl.conf file on the Netgate 4100 and found these two lines: proposals = aes256-sha256-modp1024 esp_proposals = aes256-sha1,aes256-sha256,aes256-sha384,aes256-sha512

Note that the syntax is very different from what was shown in the log file such as "AES_CBC_128".

I copied these into the corresponding fields in the network-manager-strongswan VPN settings.

On Ubuntu 22.04, it is in this location:

VPN Settings > Identity tab > Algorithms at the bottom

Check the box "Enable custom algorithm proposals"

In the IKE text input, I put:

aes256-sha256-modp1024

In the ESP text input, I put:

aes256-sha1;aes256-sha256;aes256-sha384;aes256-sha512

NOTE THAT THE COMMAS WERE REPLACED WITH SEMICOLONS! This caused me a bit of frustration until I accidentally mouse-overed the input label and saw that it said the list must be semi-colon-separated.

Anyway, with these changes, I now no longer get the NO_PROP error.

Now, I get a missing public key on the SSL certificate. If I can't solve that, I'll start a new thread.

Thanks, @jimp !

More logs still no success.

2100 Logs
Aug 28 23:10:56 charon 25380 05[NET] <201> received packet: from 24.51.235.3[4500] to 99.255.178.179[4500] (304 bytes)
Aug 28 23:10:56 charon 25380 05[ENC] <201> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Aug 28 23:10:56 charon 25380 05[IKE] <201> local endpoint changed from 99.255.178.179[500] to 99.255.178.179[4500]
Aug 28 23:10:56 charon 25380 05[IKE] <201> remote endpoint changed from 24.51.235.3[500] to 24.51.235.3[4500]
Aug 28 23:10:56 charon 25380 05[CFG] <201> looking for peer configs matching 99.255.178.179[99.255.178.179]...24.51.235.3[172.24.0.233]
Aug 28 23:10:56 charon 25380 05[CFG] <201> no matching peer config found
Aug 28 23:10:56 charon 25380 05[IKE] <201> received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Aug 28 23:10:56 charon 25380 05[ENC] <201> generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
Aug 28 23:10:56 charon 25380 05[NET] <201> sending packet: from 99.255.178.179[4500] to 24.51.235.3[4500] (80 bytes)
Aug 28 23:10:56 charon 25380 05[IKE] <201> IKE_SA (unnamed)[201] state change: CONNECTING => DESTROYING
Aug 28 23:11:01 charon 25380 10[CFG] vici client 603 connected
Aug 28 23:11:01 charon 25380 10[CFG] vici client 603 registered for: list-sa
Aug 28 23:11:01 charon 25380 10[CFG] vici client 603 requests: list-sas
Aug 28 23:11:01 charon 25380 06[CFG] vici client 603 disconnected
Aug 28 23:11:07 charon 25380 07[CFG] vici client 604 connected
Aug 28 23:11:07 charon 25380 07[CFG] vici client 604 registered for: list-sa
Aug 28 23:11:07 charon 25380 07[CFG] vici client 604 requests: list-sas
Aug 28 23:11:07 charon 25380 09[CFG] vici client 604 disconnected
Aug 28 23:11:12 charon 25380 13[CFG] vici client 605 connected
Aug 28 23:11:12 charon 25380 13[CFG] vici client 605 registered for: list-sa
Aug 28 23:11:12 charon 25380 07[CFG] vici client 605 requests: list-sas
Aug 28 23:11:12 charon 25380 12[CFG] vici client 605 disconnected

1100 Logs

Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> initiating IKE_SA con1[221] to 99.255.178.179
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> IKE_SA con1[221] state change: CREATED => CONNECTING
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> configured proposals: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> sending supported signature hash algorithms: sha256 sha384 sha512 identity
Aug 28 23:16:56 charon 80583 06[ENC] <con1|221> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Aug 28 23:16:56 charon 80583 06[NET] <con1|221> sending packet: from 172.24.0.233[500] to 99.255.178.179[500] (464 bytes)
Aug 28 23:16:56 charon 80583 06[NET] <con1|221> received packet: from 99.255.178.179[500] to 172.24.0.233[500] (472 bytes)
Aug 28 23:16:56 charon 80583 06[ENC] <con1|221> parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> received FRAGMENTATION_SUPPORTED notify
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> received SIGNATURE_HASH_ALGORITHMS notify
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> received CHILDLESS_IKEV2_SUPPORTED notify
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> selecting proposal:
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> proposal matches
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> received proposals: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> configured proposals: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> selected proposal: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> received supported signature hash algorithms: sha256 sha384 sha512 identity
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> local host is behind NAT, sending keep alives
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> reinitiating already active tasks
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> IKE_CERT_PRE task
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> IKE_AUTH task
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> authentication of '172.24.0.233' (myself) with pre-shared key
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> successfully created shared key MAC
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> proposing traffic selectors for us:
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> 192.168.2.0/24|/0
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> proposing traffic selectors for other:
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> 192.168.1.0/24|/0
Aug 28 23:16:56 charon 80583 06[CFG] <con1|221> configured proposals: ESP:AES_GCM_16_128/NO_EXT_SEQ, ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> establishing CHILD_SA con1{206} reqid 1
Aug 28 23:16:56 charon 80583 06[ENC] <con1|221> generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Aug 28 23:16:56 charon 80583 06[NET] <con1|221> sending packet: from 172.24.0.233[4500] to 99.255.178.179[4500] (304 bytes)
Aug 28 23:16:56 charon 80583 06[NET] <con1|221> received packet: from 99.255.178.179[4500] to 172.24.0.233[4500] (80 bytes)
Aug 28 23:16:56 charon 80583 06[ENC] <con1|221> parsed IKE_AUTH response 1 [ N(AUTH_FAILED) ]
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> received AUTHENTICATION_FAILED notify error
Aug 28 23:16:56 charon 80583 06[CHD] <con1|221> CHILD_SA con1{206} state change: CREATED => DESTROYING
Aug 28 23:16:56 charon 80583 06[IKE] <con1|221> IKE_SA con1[221] state change: CONNECTING => DESTROYING
Aug 28 23:17:17 charon 80583 06[KNL] creating acquire job for policy 172.24.0.233/32|/0 === 99.255.178.179/32|/0 with reqid {1}

Fixed by change IPSec to OpenVPN ( so even speed increased )

I've had no issues with IPSec on pf Plus at least, don't have a 2.7 system to test right now though, but that NAT setting normally shouldn't have to be adjusted.

Just out of curiosity, are you seeing any MAC auth errors?

I had an issue a while back, still not sure if it's solved or not (made a post with no responses) and haven't been able to test, but for some reason I was getting a ton of auth issues after updating pfSense to a newer version when it comes to IPSec, turned out that for some reason the option of using "My IP Address" wasn't properly authenticating and I had to manually specify the IP.

Anyway, seems like that's not related to your issue but just wanted to double check since it was something I ran across and never managed to solve.

Edit: here is that post I made: https://forum.netgate.com/topic/176502/had-to-manually-specify-identifier-ip-address-no-nat-involved-bug

Another edit: this does appear to have been resolved, just got it working when before it wouldn't.

Thank you @NOCling .

I've tried everything and I still have performance problems and packet loss.

Now the biggest problem is that after a while or the size of the traffic data, the traffic is lost and the stats = 0.

I realized that when it reaches 5Mb of traffic data it restarts phase 2 and resets the traffic, not letting it travel anymore. It is necessary to restart the VPN to resume traffic.

I don't know what else to change to solve this.

@mralvi22244
As I wrote, the above with BINAT in IPSec is meant for policy-based tunnel.

The last one is how I think, it has to be configured with VTI.
However, I'm unsure if it will work with the stated local / remote addresses, 192.168.227.253 / 10.10.10.10. Accordingly to the pfSense docs both addresses have be within a (transit) network. But yours obviously aren't. Don't think, that IPSec can do PPP.
But these are the data you stated.

@NOCling said in (yet another) IPsec throughput help request:

How do you move your Data?

rsync - have tried both an NFSv4 mount and over ssh (for testing purpose)

@rcoleman-netgate 2.png

@JustSumDad no need to mask RFC1918 addresss.

Ping is NOT a TCP/UDP action. It's ICMP. That's why they aren't passing.

@viragomann

It's working fine with leaving the Remote IKE port field blank.
Now the Server A is using the custom NAT-T port as intended.

Thanks for your help.

@viragomann I don't know when or why I set that, but I removed it and that appears to have resolved the issue.

Thank you!!! I don't think I'd have ever figured this out on my own.

LAN gateway2.png

@thomaspsimon said in IPSec with dual WAN:

@NOCling said in IPSec with dual WAN:

Do you have try enable the Mobike Option on both sitets?

tried, but no luck.

The issue here is Local Host is not changing from the failed WAN IP to the failover WAN IP automatically, without that it will not happen, if i am not wrong. Please see the screenshot.

Branch IPSec.JPG

it seems the link https://redmine.pfsense.org/issues/13076 talks about the same issue and an edit to rc.ipsec file fixes the issue.

But didn't get how to make that edit.

@iulianteodor
Lots of variables. You're going to need to add some more details. Policy based or routed? What's the other endpoint? Any NAT involved, etc...

@jimp said in 2 Clients Connecting from Same Public IP Fail:

Do you have hybrid or manual outbound NAT rules that setup static source port for 4500? That shouldn't be necessary and may be interfering with the clients.

NAT-T works fine with a randomized source port, so having outbound NAT preserve a static source port could break it for multiple clients.

I do not, but I suspect the provider at our DC does?

Reading up on it, it sounds like they’ve turned on some kind of IPSec pass through / helper feature on their side…which is not helpful!

@TheWaterbug

You'll need a few things, might seem like a lot but it's actually quite easy with the pfSense ipsec wizard!

Setup your VPN settings in pfSense for IPsec tunnel to use EAP TLS I.e. this is using a CA that you setup, a Cert for the IPsec Server, and a cert for each client. Download the config with the pfSense package "ipsec-profile-wizard" Test this config works by now loading that .mobileconfig onto your phone.

https://www.derman.com/blogs/iOS-IPSec-VPN-OnDemand-Setup

Section 3 there "3. Import the IPSec VPN configuration profile onto the iPhone" provides instructions for apple configurator, or just emailing it to yourself.

Once you've got the above working nicely, you know you have a secure VPN with cert based auth and you can add in the few line that I posted above to the .mobileconfig file, and then upload that to the phone in question.

N.B.

You MIGHT have to "Supervise" your iphone for this to work...
For "Always On VPN" you 100% need it supervised, but I THINK that with "On Demand VPN" you don't have to.

https://www.miradore.com/knowledge/ios/enable-supervised-mode-on-ios-device-using-apple-configurator/

That should get you started.

@stbellcom said in IPSec preformance:

Hello,

I currently have two Netgate 6100 setup in the lab connected to each other via 2.5gb/s and then running a IPSec VTI tunnel over this connection.

On each end I also have two pc's with 2.5gb/s running Iperf3 for testing.

About the best speed I can get is around 1.01 Gbits/sec though the Netgate spec says it should be around 1.8 Gbits/sec for the 6100

I have tried the recommended settings from this page:

https://docs.netgate.com/pfsense/en/latest/vpn/performance.html#optimal-encryption-settings

And a bunch of lesser secure with QAT | AES-NI enabled/disabled without much change.

Is it realistic to be getting 1.8 Gbits/sec in a lab setup and does anyone have recomendations on which encryption cipher to use to get raw speed though the VPN ?

Thanks

Can you get the 1.8 Gbps with just NAT?

i.e. get rid of the IPSEC tunnel, port forward iperf ports and just santiy check that the bandwidth is good for that?

Would be a good step to test to ensure the path is setup correctly for over gigabit speeds.

@TheWaterbug said in Mobile IPSec VPN On Demand from iOS/macOS?:

This may be more of an iOS question than a pfsense/IPSec question, but is there a known way to have an iOS/macOS device automatically connect to my pfsense 2.60CE IPSec endpoint, but only when attempting to connect to specific IP addresses inside that LAN?

For example I currently have my security camera system port forwarded in from ACMERocketCars.dyndns.org:80 to 192.168.50.3:80. That's on a separate subnet, firewall off from all my critical infrastructure, but it still seems a bit scary to have a machine widely exposed on the internet.

I already have a mobile IPSec tunnel set up that works from both my macOS devices and my iOS devices, but I have to "dial" it manually every time, which is inconvenient any time I want to just quickly check a camera.

Is there a recipe for creating a configuration file that I can load on my macOS and iOS that auto-dials my VPN connection if I attempt to access 192.168.50.3:80, and then drops the connection if there's no traffic in X minutes?

Yes take a read of this: https://github.com/nerd-one/VPN-OnDemand/blob/master/VPN%20OnDemand.mobileconfig

And my post here which shows where the code goes:

https://forum.netgate.com/topic/181588/ios-on-demand-vpn

No duplicates appeared over the weekend. Seems split connections is the way to go.