Problem with TCP and GRE tunnel
-
Hey there!
It's my first time with pfsense and im experiencing one issue that i dont really know what I can do to solve.
So I setted up an home server and installed a VM with pfsense, then bought a VPS on a company and setted up another pf and created a GRE tunnel to route all traffic. I can send and recieve every packet between them, I can also ping every IP without any problem but when I try to ping or use curl on any dns it's really slow and sometimes dont even work (the packets still sending and recieving correctly)
The last thing i made was changing from DNS Resolve to Forward and it helped because i was able to ping dns's but when i try to use curl or access a webserver it's very very slow.
P.S -> I can see all traffic sending and recieving on both pf'sIf you guys need a screenshot of something specific tell me!
Thank you all in advance!
-
S stephenw10 moved this topic from NAT on
-
Do you see the traffic leaving from the correct interfaces?
With the correct NAT applied?
The symptoms sounds like an asymmetric routing issue to me.
Steve
-
@stephenw10 Hey, I think yes, i can send you some screenshots of that traffic.
Local pf -> Interface (GRE) at the time i do "curl ifconfig.me"
Local pf -> Interface (LAN) at the time i do "curl ifconfig.me"
Remote pf -> Interface (GRE) at the time i do "curl ifconfig.me"
Also i need to use DNS "185.113.141.1" because if I try google DNS or Cloudflare DNS's i cant even ping DNS's
-
Hmm, your local LAN network is using public IP addresses?
Are you NATing those outbound at the remote side?
I'd still guess this is an asymmetric routing issue. The target hosts are seeing the LAN side public IPs and try to reply directly rather than via the remote VPS.
-
@stephenw10
My lan is set to 185.113.141.1/24 but only the ips that my isp gave me are going through the gre tunnel without nat.
My goal with this is to make it easier to configure future IPs and not do 1:1If you want i can send some screenshots with smth that you think is wrong, and thank you in advance!
-
Is that a real subnet your ISP has given you though? A full /24 would normally be quite expensive!
If it is a real public subnet then target hosts will have a route back to it directly. You would need to NAT it at the VPS.
Unlss that subnet is in fact routed to your VPS instance?
-
@stephenw10 I just bought 4 ips not the entire subnet, it's just simulating on the local pf but only this 4 ips are being routed with the tunnel. Is there anything wrong in the traffic i sent?
-
Ok, where are those IP addresses routed to though? To the local pf or the remote pf at the VPC?
-
@stephenw10 In local PFsense there is a firewall rule specifying that each IP provided by the isp has the gre tunnel gateway and we do not nat those IPs which means they are leaving through the tunnel how they are. In the remote PFsense there is a static router for each IP address to go through the gre tunnel and I disabled nat in the remote PFsense for these IPs. I did pcap on the wan and gre on both pfsenses and the traffic is being sent and received on the local pfsense lan as well. The ping goes perfectly both from the outside to the inside and from the inside to the outside.I also did a traceroute from my PC to an IP and it is correct and a traceroute from a host with an IP from my isp to 1.1.1.1 and I see both going through the gre and reaching the destination.The dns on the host with an IP from my isp are 1.1.1.1 and 8.8.8.8 but when I ping a dns or curl a dns it takes a long time.However, I can see the response from 1.1.1.1 to the dns I want in the pcap in the local pfsense on the lan, the dns and the IP address of the requested dns. But on the host it takes a long time.
-
So those public IP addresses are globally routed to the VPC? You purchased them from the VPC provider?
If they are routed to the local pf by your ISP that then creates an asymmetric route.
-
@stephenw10 But the ISP is not making a static route, it's proxy arp. And yes the IPs are bought from the VPC provider
-
Ok, if the IPs are purchased from the VPC then they are presumably routed to the public IP there. So that should avoid any asymmetric routing.
So I would run a pcap on the WAN interface at the VPC end and see what's happening there for an ICMP and TCP connection.
-
@stephenw10
There's the pcap -> (WAN) on the VPC with "ping google.com" and "curl ifconfig.me"
-
Can you upload the actual pcap file here? https://nc.netgate.com/nextcloud/s/BynTbfz33wrToAm
I assume the curl fails?
-
@stephenw10 I'll do it in a sec, no the curl works but with a lot of delay
-
@stephenw10 Uploaded the pcap from VPC, WAN while doing a curl ifconfig.me
-
Hmm, weird there's a 30s gap between the initial syn/ack handshake and the http get request.
Does that same delay appear at local pf in a pcap there? It seems unlikely it would actually just wait which implies the initial request is being lost. Perhaps an MTU issue?
Is there any reason you chose to use GRE for this instead of something encrypted?
-
@stephenw10 Do you want a pcap on the local pf? If yes which interface? GRE?
An there's no reason to chose GRE, I just heard about it and decided to give it a try. -
The pcap should be identical on the local GRE and LAN interfaces. If it isn't that's probably where the problem is.
GRE has a few quirks that OpenVPN or route-mode IPSec do not. Since you're not NATing or encrypting though you shouldn't be hitting any of them.
-
@stephenw10 I think they are but maybe i miss something, do you want me to upload the pcap on the same URL?