ipsec vti routing can only get to firewall, no clients
-
I'm trying to figure out why I'm not able to get to machines on the remote sites. Here is my config:
site a (my homelab) which has vlans\subnets that fall under the 10.37.0.0/16 space
site b (remote esxi box) which has subnets under the 10.38.0.0/16 space.
I have a 10.8.1.1/30 transit network defined. The connection is live and shows connected. I can ping the firewall IPs from both ends. I can also access the firewall from both ends. IE I can get to 10.38.1.1 pfsense at site b from my 10.37.20.2 client at site a.
I have a static route of 10.38.0.0/16 on site A pointed to the ipsec gateway
I have a static route of 10.37.0.0/16 on site B pointed to the ipsec gatewayI currently have them connected via a VTI routed IPSec connection. I can get to the site b pfsense web console at 10.38.1.1. Where I have problems is getting to a machine on site b at IP 10.38.70.10. I try to ssh to it and on the site a firewall I see a TCP:S block packet for port 22 from my client to the destination machine.
The VLAN\subnet at site A has a carte blanch allow any outbound IPv4. The IPSEC tab of the firewall rules on site b have a "allow anything in" rule as well. Why can I get to the .1 but not to any other machines? What am I doing wrong? thanks!
-
So I found if I put another "allow any" on my LAN on site a pointed to the site-to-site gateway I can get past the "TCP:S" error and then get a "TCP:PA" error. Should pfSense not be able to use the "default" gateway and know that based on the static route it should go over the gateway to begin with?
-
@realityman_ it's not clear what you mean. You need static routes from each site to the other, and you have them. pfSense will use that rules to route to the VPN, than standard routing to route encrypted packets to remote pfSense.
Do you have some NAT enabled over the VPN?
-
@Topogigio Apologies it was getting late haha
The only NAT I see that could affect this is for 1.8.1.2 but that looks like it's for the IPSEC negotiation:
The summarized version is. I have an established site-to-site IPSec route VTI connection. It shows as connected, but I can only get to the .1 addresses on the remote site, so from my site a to site b's 10.38.1.1 or 10.38.70.1 for example. When I try to SSH into a host on site B I get a TCP:S block on site A's firewall. I have a blanket "allow all" rule out of the VLAN 20 interface (where this request originates from) but it doesn't seem to be working with the "default" gateway. If I put another "allow all" and point the gateway to the IPSEC gateway it will get past the TCP:S but then fails with a TCP:PA.
Here is the config of my static routes on Site A:
Site B:
Here are my floating rules, I wouldn't think the Codel Queues would affect this since that's on the WAN interface, and not looking at the IPSEC:
-
You need some kind of rules as:
- source: 10.37.0.0/16
- destination: 10.38.0.0/16
- procotol: tcp/udp or icmp or all
and reverse on the other side
No rule in your sshot seems allowing a traffic from/to the VPN..
-
@Topogigio Here is my VLAN 20 rules for site A (this is where my request originates from)
Here is my IPSec rules on site A:
Hetzner DMZ is an alias to 10.38.70.0/24
Hetzner LAN is an alias to 10.38.1.0/24
Hetzner Tunnel is an alias to 10.8.1.1/30
Site B IPSec rules:
Here you can see where I initially tried the SSH. I get the "do you want to connect to this fingerprint?" and then as soon as I hit yes I get a time out. So it get's there at least initially, but dies.
Here are the states on the site b
-
This post is deleted! -
I think I may see the issue?
Site A:
Site B:
Should the site b subnet be in the outbound NAT of site A on the WAN interface and vice-a-versa? I would think not since those should go over the IPSec interface, right?
-
No NAT is required to route through VPN tunnel...
Your policies seem ok if VLAN70 net is the local VLAN of site B.
However if you have some kind of TCP session estabilished (the fngerprint request from SSH), firewall rules may be ok.
-
Right, there shouldn't be, so I was wondering if that should even be there? I didn't put it there.
VLAN70Net is a local DMZ VLAN on site A (10.37.70.0/24)
Wouldn't the site A IPSec rules only cover incoming traffic over the IPSec tunnel? The stuff outbound from site a would go through the interface it came from? So in my case I am initiating an SSH connection from a client on VLAN 20. The request would go through the VLAN 20 interface firewall rules, then be routed over the IPSec tunnel. Any requests coming from site B would hit the IPSec firewall interface rules, right?
-
It looks like I can ping hosts from site B into site A fine. It looks like the problem is on my site A (homelab) router side.
-
Here is my setup:
https://imgur.com/a/lBjOEJ5 -
Here is a TCP dump on my VLAN20 interface where my client I am trying to connect from lives, it just keeps repeating:
[2.4.5-RELEASE][admin@pfSense.localdomain]/root: tcpdump -ni lagg0.20 host 10.38.70.11
tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on lagg0.20, link-type EN10MB (Ethernet), capture size 262144 bytes
00:08:29.827060 IP 10.37.20.230.53844 > 10.38.70.11.22: Flags [S], seq 1814817855, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 393680583 ecr 0,sackOK,eol], length 0
00:08:30.908274 IP 10.37.20.230.53844 > 10.38.70.11.22: Flags [S], seq 1814817855, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 393681583 ecr 0,sackOK,eol], length 0
-
@realityman_ sorry, it's quite hard for me because you use rules in a different way as me (for example, I don't use floating rules).
It seems that ping can work in both direction (not TCP protocols, you have in site A for example incoming allowed from IPSEC only if it's ICMP).
I suggest to try on both sides, on the firewall rules:
- in IPSEC tab: from all to all, ICMP (or all or TCP based on what do you want test)
- in VLAN20 (or other local interface where SOURCE host is) tab: from all to all, ICMP (or all or TCP based on what do you want test).
After that, if it's work, you can arrange it with filters you need about source IP address and/or protocols or ports.
your routing table seems ok and so IPSEC VTIs
-
@Topogigio Yea the only reason I have floating rules is because I use codel on my WAN since I have a stupid small upload (40mbps) and when my lab backs up it will murder my latency.
Interesting thing happened. I restarted the VPN, and when I first started I was able to ping the host 10.38.70.11. I then tried SSHing to it, and I got the login prompt. The connection died once I sent my password though. So it does work for a period, but then stops.
Let me try what you suggested and see what happens. Thanks for your help!
-
@Topogigio tried your recommendation, still no traffic went to that host
-
@realityman_ my opinion this is not pfSense...
maybe do you have some dynamic firewall on the host the ban your IP?