Unexpected traffic hitting IPsec interface
-
I have a few s2s tunnels setup and the tunnels are working well, etc. But in the firewall logs I'm seeing a lot of weird entries for blocked traffic on interface IPsec:
SRC = 10.54.1.13:725
DST = 192.168.110.133:515The default rule on the IPsec interface is blocking this traffic, which is fine, but here's my concern:
There are no tunnels with P2s matching the source network. The dest network is one of my subnets (the subnet every remote end is connecting into) though the host is nonexistent. So my concern is how/why is this traffic hitting the IPsec interface to begin with? Shouldn't the only thing creating incoming traffic on this interface be traffic coming in from one of the established tunnels? Is there a way to determine which tunnel is producing this traffic on the interface?
-
You don't indicate what equipment is on the far side of the tunnel, nor what your IPSEC phase 2 settings are.
My best guess is that there is some equipment on the far side that is sending to 192.168.110.133 port 515, which is a well known port for print spooler, and that the phase 2 settings on the far side are sufficiently wide to allow that subnet in. -
AWS on the end of tunnels 1 & 2, Cisco ASAs on the end of tunnels 3 & 4, and pfSense on the end of tunnel 5.
The P2s for the tunnels as configured on my end look like this:
Local (me) <=> Remote (them)
Tunnel 1 & 2 (AWS redundant tunnels):
192.168.93.0/24 <=> 192.168.112.0/20
192.168.29.0/24 <=> 192.168.112.0/20
192.168.25.0/24 <=> 192.168.112.0/20
192.168.250.0/24 <=> 192.168.112.0/20
192.168.110.0/24 <=> 192.168.112.0/20
172.16.8.0/21 <=> 192.168.112.0/20Tunnel 3
192.168.110.0/24 <=> 192.168.145.72/29
Tunnel 4
192.168.110.0/24 <=> 192.168.145.112/28
Tunnel 5
192.168.110.0/24 <=> 192.168.199.0/30
192.168.97.0/24 <=> 192.168.199.0/30I'll be the first to admit I'm a software engineer in a smaller shop pretending to be a network engineer when needed so I'm a little confused when you say the P2 config on the other end is wide enough to let that subnet in. If my P2 doesn't let it in then the P2 wouldn't ever establish, would it? Or at the very least, the address doesn't match any of my P2 configs then the traffic should never make it as far as the IPsec interface, no?
But as you can see, the subnet here is not even close. It might be worth noting that tunnels 3-5 are required to NAT their traffic to the assigned subnet so maybe the other end on one of those ASAs isn't natting properly but then I still don't understand how traffic with such an IP (one that doesn't match any of my P2s) ever even hits the IPsec interface?
-
That's why you have rules defined on the IPSEC interface....
Basically, the far side encapsulated the traffic and sent it to the pfSense over the established tunnel. How it actually got into the tunnel in the first place would need to be explored on the far end.
The purpose of the P2 is to allow it to be used as a selector to build forwarding tables on the device so that it can figure out where to send the packet to a specific destination, but once the P2 is agreed upon during the negotiation phase, nothing is actually checking (maybe it should be, but clearly isn't) that the packet matches the criteria of the P2 once it arrives at the crypto engine. -
Interesting, ok, that's good to know. I assumed the P2 settings were a sort of allow/deny rule for traffic, but sounds like it's not.
So I've found out which peer is producing that traffic and with your explanation, it now makes sense how that traffic could end up on the interface. I found out through some documentation lying around pre NAT days for these tunnels so I know which peer is using 10.54.1.0/24 as their internal network.
But let's say I didn't have that documentation handy. Is there a way I could trace things within pfSense to figure out which of the tunnels is producing this traffic? Basically I'd just like to know how I might go about troubleshooting this if I didn't know which peer was producing the packets.
-
@Slugger said in Unexpected traffic hitting IPsec interface:
Basically I'd just like to know how I might go about troubleshooting this if I didn't know which peer was producing the packets.
Excellent question!
My understanding is that all packets are delivered to the kernel from the same place, so the encapsulation envelope information is lost by the time it is hitting the firewall.
Maybe turn on debug on IPsec and look at the logs?
See Advanced Settings > IPsec Logging Controls > IPsec traffic; you could try the diag or raw setting and have a look at what gets logged.
Keep in mind if you make that sort of change, I think it will restart the tunnels. -
You will not be able to tell from the firewall logs but you can tell from a packet capture. Every IPsec payload packet received will have an SPI associated with it. These are used to identify they key used to decrypt/authenticate the contents.
09:31:10.674116 (authentic,confidential): SPI 0xcfe3bb0d: UDP, length 807 09:31:10.721850 (authentic,confidential): SPI 0xc2dfeb43: UDP, length 531 09:31:10.735629 (authentic,confidential): SPI 0xcfe3bb0d UDP, length 808 09:31:10.798827 (authentic,confidential): SPI 0xc2dfeb43: UDP, length 468
Those SPIs are visible in-the-clear on the outside in the ESP packets. You could capture that and see what external host it is. You can also search the logs for them being negotiated to find out the peer that sent them:
Filter IPsec logs on
cfe3bb0d
Sep 19 02:23:25 charon 03[IKE] <con2000|25025> inbound CHILD_SA con2000{2374} established with SPIs c2dfeb43_i cfe3bb0d_o
con2000
is your tunnel identifier. -
So the general idea is I'm capturing on WAN then using the timestamp of the WAN packet to link it to the timestamp of the fw block entry on IPsec then searching the logs to find out which SPI generated the packet? If my understanding is correct, that's likely to work on lower traffic tunnels (which does apply here), but not so easy on tunnels with lots of traffic (because, for example, you could have many packets with the same timestamp as the fw block entry, even packets from every tunnel). Definitely makes sense though, now I have a debug method for the future. Thanks!
-
@Derelict Great suggestion, but as @Slugger pointed out, it might be tricky to do if there is a lot of traffic.
I had a look to see if there might be an easier way to track it down, and I think this might be one way:
You can use the Packet Capture on the IPSec interface, and specifying the mystery IP in the Host Address field. This is a special virtual interface, and it will show you both the SPI and the internal IPs at the same time.Here's an example:
09:00:49.106179 (authentic,confidential): SPI 0x770bd9e5: IP 10.71.77.10 > 10.0.128.10: ICMP echo request, id 24404, seq 0, length 76 09:00:49.108023 (authentic,confidential): SPI 0xcb496926: IP 10.0.128.10 > 10.71.77.10: ICMP echo reply, id 24404, seq 0, length 76
Armed with the SPI, in my case 0xcb496926 is the SPI for inbound traffic, go into the Status > IPSec > SADs tab and search for cb496926 (be sure to strip off the 0x); it will reveal the endpoint IP address. Here I can see that the source IP ending in .58 is the remote endpoint.
-
@awebster Yup, that works wonderfully. Just did it, confirmed what our docs were suggesting. Now to have a chat with that peer armed with sufficient evidence. :)
Thanks for all the help on this!