Thanks for the suggestion. Not the solution here, but was worth experimenting with.

Not yet, but soon.

Sorry for taking so long, I thought I checked to be notified for responses over email, I guess I didn't. Anyway, I had already done that, I'm starting to think my install is defective somehow. I think it's dropping information, I tried an internal VPN server and connections can't be made, they pass the firewall and are logged and reach the final server but this server reports it's missing stuff like username, I just nuked the server so I can't paste logs anymore but I found the same thing happening with VoIP traffic, no matter what I do I just can't get it to work. Something's off. This VPN server used to work before, now [if] the tunnel comes up, clients will be missing some information like default gateways and no traffic will pass. I tried with several ISPs yielding the same results and the only thing that's different is the gateway itself. I'm loving pfSense so far but I think I'll have to go back to my Ubiquiti gateway if this doesn't work. :( Thanks for you help anyway, I really appreciate it.

After much troubleshooting on and off the last few weeks, I think I have this working "good enough". Our old ASA 5505s handle failing over slightly more reliably, but in the end the cost and flexibility of the pfsense devices seem to be worth it. I think one of the main issues was DNS resolution. The firewall was unable to update the dynamic dns service when it failed over to the backup wan. One way to resolve that is to configure "Enable default gateway switching" under advanced, but that seemed to cause issues with a static route I had setup to allow the firewall itself to reach the remote lan of the ipsec tunnel. My resolution was to: 1. Make sure DNS servers are configured on all interfaces under System-> General Setup. (Why can't there be duplicates?) 2. Under Services->DNS Resolver, check "enable forwarding mode" 3. Set DPD to a lower retry rate to cause broken tunnel to be torn down quicker. 4. Enabled Cisco Extensions under vpn advanced. Not sure if this helped anything. 5. Installed cron and changed the rc.dyndns.update task to run every 3 minutes. This probably didn't fix anything, but it helped troubleshoot. Remaining issues: 1. When the primary WAN comes back online, the tunnel never seems to renegotiate on that circuit automatically (maybe im not waiting long enough?). This isn't critical in my view. 2. IPsec is currently using not very secure aggressive mode, more testing needed to check main mode. And maybe someday I'll be able to look into some sort of ipsec over gre connection that handles failures more gracefully, but that looks quite complex since I don't have much knowledge in the field. Hope this helps someone.

The ACL/Routing in Azure is allowing all traffic between pfSense and VM for troubleshooting purposes, so its definitely not getting blocked. I've had a capture running on the target for the last 24-hours and have not received a single packet from the remote side - it is getting lost in the Azure fabric, it seems. I have opened a ticket with Azure referencing this Forum Post to see if we can get some debugging done between the Azure Gateway and our target VM.

syncing, encryption and decryption

I have just checked, and there are no firewalls enabled on those machines.

@jimp: Look at the config that shows up in /var/etc/ipsec/ipsec.conf, especially "esp =", what does that look like? If you have multiple P2s on that P1, this bug might be a possibility: https://redmine.pfsense.org/issues/6263 I do have multiple P2s on the P1, so that bug is probably why I can't get it to use GCM (since I haven't gone through and disabled all of the other crypto types). Doesn't fully explain why the selection / ordering isn't working though.  The Palo Alto is supposed to be offering GCM first and then CBC last.  Only thing I can think of is that the GCM is failing for some reason, and it's falling back to CBC? Woah, this is kind of ugly.  I guess this is due to "auto" for the length, maybe combined with the fact that I have so many P2 entries (the same config line might be repeated for each P2)? esp = aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384,aes256-sha384-ecp384,aes192-sha384-ecp384,aes128-sha384-ecp384,aes256gcm128-sha384-ecp384,aes256gcm96-sha384-ecp384,aes256gcm64-sha384-ecp384! I'm seeing each P2 entry in there as individual "con1xxx" and each one has an identical esp= line. I could pare down to like 3-4 P2 entries if I had a way to "carve out" (exclude) individual /24s from a /16 tunnel… I think this is something strongswan can do but it isn't exposed in pfSense and I'm not about to start mucking around behind the scenes and breaking stuff.  ;) A routed /30 style tunnel would also do the trick (and I'd even prefer it)... I know strongswan can do this too (it's possible with Ubiquiti) but ISTR there are tunnel interface problems in BSD that make this not an option.

Thanks for your reply jimp. I will start a new thread.

Just to check, did you try this bit? https://doc.pfsense.org/index.php/L2TP/IPsec#Firewall_traffic_blocked_outbound I had a very similar problem and the sloppy state bit fixed the problem for me. It's not that a specific firewall rule is blocking something, it's the state handling that interferes on the L2TP virtual interface.

After helpful discussion on the irc - thank you rawtaz - the problem could be solved. Generally, I have to add an additional static route for the remote network. When I create this route, I could deactivate complete NAT handling and it works as expected.

Fixed. While doing a trace I realized that when a packet would leave a VM in site 10 it wouldn't make it past the core switch which does intervlan routing. I went digging into it and found out that when I was setting up the VM for site 30, interface vlan 1 on the switch received an IP from the pfSense LAN interface DHCP. So the core switch though that 10.10.30.0/24 was directly connected to VLAN1 instead of following the standard routing table. After flushing the IP on int vlan 1 everything started to work as expected.