Cannot access webservers through vpn that are on a different gateway
-
Hi to all,
I cannot figure out this, furthermore I'm not sure which side is the culprit, but I think that the issue is on the pfsense's side.
I have in my central offices two routers:- the first one is a mikrotik CCR1009-7G-1C-1S+ connected to internet through a very fast optical fiber provider(pppoe interface with static IP), this router has local IP 192.168.25.62. This router is an IPSEC/l2tp server for another office that is using a mikrotik 951G-2HnD and a very good VDSLwith local ip 192.168.26.1. Everything works and I can ping and reach everything on the
other side like webservers, filshares etc and I have a very good performance. - the second one is virtualized kvm pfsense connected to internet through another very fast optical fiber provider(pppoe interface with static IP), this router is a gateway only for a specific debian web sever 192.168.25.136 and a freepbx/asterisk distro 192.168.25.200. This router has local ip 192.168.25.76.
I have setup all routes and returning routes on both mikrotiks and pfsense and I can ping from remote mikrotik network 192.168.26.0/24 the debian webserver and freepbx, I can even register the phones to the freepbx server and I can reach the pfsense interface.. but for some reason I cannot open with my browser both the debian web server and the freepbx distro if I'm on the remote 192.168.26.0/24 network. Sometimes randomly it works, but after some minutes again I cannot reach any web server. I was thinking about mtu on the remote mikrotik side, but nothing seems to change lowering down the mtu values even if I change the values directly on the ethernet interface of workstations or servers. I have tried to add pass input/forward rules to see if something changes but with no results.
So the problem is that I can ping from both sides through the vpn, but I cannnot reach web servers that are using the pfsense gateway, only randomly it works.
here is tcpdump that I got on the webserver 192.168.25.136 when trying to open a page from remote 192.168.26.20 workstation
16:10:26.764026 IP 192.168.26.20.51372 > freepbx.sangoma.local.http: Flags [S], seq 976993469, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:26.764084 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:27.014422 IP 192.168.26.20.51373 > freepbx.sangoma.local.http: Flags [S], seq 2419313814, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:27.014452 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:27.764852 IP 192.168.26.20.51372 > freepbx.sangoma.local.http: Flags [S], seq 976993469, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:27.764895 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:28.014830 IP 192.168.26.20.51373 > freepbx.sangoma.local.http: Flags [S], seq 2419313814, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:28.014855 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:28.766953 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:29.166956 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:29.765511 IP 192.168.26.20.51372 > freepbx.sangoma.local.http: Flags [S], seq 976993469, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:29.765537 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:30.015698 IP 192.168.26.20.51373 > freepbx.sangoma.local.http: Flags [S], seq 2419313814, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:30.015742 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:31.766918 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:32.166957 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:33.765739 IP 192.168.26.20.51372 > freepbx.sangoma.local.http: Flags [S], seq 976993469, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:33.765783 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:34.016993 IP 192.168.26.20.51373 > freepbx.sangoma.local.http: Flags [S], seq 2419313814, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:34.017020 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:37.766944 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:38.166943 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:41.765829 IP 192.168.26.20.51372 > freepbx.sangoma.local.http: Flags [S], seq 976993469, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:41.765859 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:42.017096 IP 192.168.26.20.51373 > freepbx.sangoma.local.http: Flags [S], seq 2419313814, win 64240, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0 16:10:42.017117 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:49.966944 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:10:50.366943 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:11:05.966987 IP freepbx.sangoma.local.http > 192.168.26.20.51372: Flags [S.], seq 1214405327, ack 976993470, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 16:11:06.366951 IP freepbx.sangoma.local.http > 192.168.26.20.51373: Flags [S.], seq 1721642903, ack 2419313815, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0Any idea how to fix this?
many thanks - the first one is a mikrotik CCR1009-7G-1C-1S+ connected to internet through a very fast optical fiber provider(pppoe interface with static IP), this router has local IP 192.168.25.62. This router is an IPSEC/l2tp server for another office that is using a mikrotik 951G-2HnD and a very good VDSLwith local ip 192.168.26.1. Everything works and I can ping and reach everything on the
-
a very strange think is that if I run a ping or a tracert from a workstation to the remote web server, I will have access for some minutes to that specific webserver.. really strange look like something with opened states
-
Seems to me like an asymmetric routing issue.
As I understand your setup description, the webserver and the freepbx are in the 192.168.25.x network and they are using pfSense as default gateway. However, in this subnet is another router, which is the IPSec gateway and you are not able to access these two devices from remote?
Since these devices use another default gateway, they will send out responses to it, not back to the IPSec gateway.
You will have to add static routes to both for the remote network, pointing to the Mikrotik. Have you done that?A better solution would be to put them in a seperate subnet behind pfSense and set up a transit network between the two routers. However, doing that will also require to add an additinal IPSec P2 for the new subnet or enlarge the existing one so that it covers both subnets.
Alternativly you may do masquerading on packets destined for these devices on the Mikrotik.
But there is nothing you can do on pfSense, cause its not (should not be) involved in that traffic.
-
@viragomann thank you for the support
As I understand your setup description, the webserver and the freepbx are in the 192.168.25.x network and they are using pfSense as default gateway. However, in this subnet is another router, which is the IPSec gateway and you are not able to access these two devices from remote?
everything you wrote is correct, except the fact that I can ping, tracert and register extensions on those two devices from remote, but I do not have tcp traffic like http or ssh.
Since these devices use another default gateway, they will send out responses to it, not back to the IPSec gateway.
You will have to add static routes to both for the remote network, pointing to the Mikrotik. Have you done that?yes, I added all the necessary static routes on each device on the network both outward and return, otherwise I would not have had ping or registered extensions or even the tcpdump that I reported.
Sorry for this, but have you read my last update?
a very strange think is that if I run a ping or a tracert from a workstation to the remote web server, I will have access for some minutes to that specific webserver.. really strange look like something with opened states or arp tables -
@silvered-dragon
Are you able to ping the remote network from the webserver as well? -
@viragomann yes
Are you able to ping the remote network from the webserver as well?
yes! I think routing is perfect -
What I noticed now is that if I masquerade the remote networks on the central mikrotik behind the lan interface, things works. But honestly I'm not sure that masquerading the remote lan is a good practice.
-
@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
But honestly I'm not sure that masquerading the remote lan is a good practice.
Well that again screams asymmetrical.. You sniffing on the webserver doesn't tell you what gateway it sent it too.
If it doesn't send it back to pfsense, then its not going to work.
If your box your trying to get to is not using pfsense as its gateway, then you have couple of options. You either source nat the traffic so it looks like it came from pfsense IP, or your put a route on the dest box that tells it to use pfsense as the gateway to get back to your remote network..
-
@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
if I masquerade the remote networks on the central mikrotik behind the lan interface, things works.
So the only two reasons for failing without that I can think off are
- the route doesn't work
- the destination server itself blocks the access
Blocking access from outside its own subnet is the default behavior of system firewalls, however, a webserver should be configured to accept access from anywhere. I assume, the server is accessible from the internet.
@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
But honestly I'm not sure that masquerading the remote lan is a good practice.
The only one drawback is that you cannot identify the real source address on the destination device, as long as you do the masquerading only for the remote lan.
-
@viragomann said in Cannot access webservers through vpn that are on a different gateway:
the route doesn't work
Route doesn't work where? You can not put the route on the other gateway... It would have to be on the device your trying to talk to..
For starters if this other gateway is stateful he would not pass on your syn,ack because there is no state for it.. The correct solution if you want to bring this vpn in via pfsense vs your current edge is connect pfsense to your edge router via a transit network - as already mentioned by @viragomann
-
@johnpoz
The route on the destination device itself:@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
Since these devices use another default gateway, they will send out responses to it, not back to the IPSec gateway.
You will have to add static routes to both for the remote network, pointing to the Mikrotik. Have you done that?
yes, I added all the necessary static routes on each device on the network both outward and return, otherwise I would not have had ping or registered extensions or even the tcpdump that I reported. -
The logical answer to this is no he didn't do it correctly..
Because if he did - then it would be working..
-
Yeap, I think so.
-
Did he show us these routes? No
Did he show a sniff on pfsense - where the syn,ack was sent back to pfsense - No
But its work if he source nats - logical conclusion is host routing that he stated he did was not done, or was done wrong.
-
@johnpoz
I'm with you. -
Not sure why anyone would go that route anyway.. Why would you not just connect pfsense via a transit network to your edge router.. If you want to use pfsense for vpn.
Or better yet just replace whatever your using as your edge with pfsense, since clearly its vpn features is lacking that you want, or you would just be doing your vpn there.
Its like people like to cause themselves pain and issues, when there is such an easy solution..
-
@johnpoz as I wrote I can ping, I can see traffic going from the remote side to the webserver and viceversa, and most of all my extensions are registering and I can call and pass all rtp traffic, so this cannot be a routing problem. if I make a traceroute from both sides the routing is what I'm expecting see this:
this is from remote workstation to freepbx(that has pfsense as gw)
C:\Windows\system32>tracert 192.168.25.200 Traccia instradamento verso 192.168.25.200 su un massimo di 30 punti di passaggio 1 <1 ms <1 ms <1 ms 192.168.26.1 //this is the remote mikrotik 2 37 ms 37 ms 37 ms 10.0.25.1 //this is the l2pt server 3 38 ms 37 ms 37 ms 192.168.25.200 //freepbx reached Traccia completata.this is from the freepbx to remote workstation
[root@freepbx ~]# traceroute 192.168.26.20 traceroute to 192.168.26.20 (192.168.26.20), 30 hops max, 60 byte packets 1 192.168.25.62 (192.168.25.62) 0.301 ms 0.244 ms 0.204 ms //this is the central mikrotik router 2 10.0.25.2 (10.0.25.2) 37.522 ms 37.611 ms 37.647 ms //this is the l2pt remote client 3 192.168.26.20 (192.168.26.20) 38.085 ms * * //workstation reachedthis is the routing table on my pfsense you will notice that request to remote network 192.168.26.0/24 are routed through the mikrotik gw 192.168.25.62
default xx.xx.xx.xx UGS 1139654 1492 pppoe0 10.0.25.0/24 192.168.25.62 UGS 1 1500 vtnet1 10.0.40.0/24 10.0.40.2 UGS 49992 1500 ovpns1 10.0.40.1 link#9 UHS 0 16384 lo0 10.0.40.2 link#9 UH 2343778 1500 ovpns1 10.0.90.0/24 10.0.90.2 UGS 0 1500 ovpns2 10.0.90.1 link#10 UHS 0 16384 lo0 10.0.90.2 link#10 UH 0 1500 ovpns2 127.0.0.1 link#3 UH 1244 16384 lo0 192.168.25.0/24 link#2 U 754126 1500 vtnet1 192.168.25.76 link#2 UHS 0 16384 lo0 192.168.26.0/24 192.168.25.62 UGS 90979 1500 vtnet1 192.168.27.0/24 192.168.25.62 UGS 326 1500 vtnet1 195.96.194.18 link#8 UH 39459 1492 pppoe0 195.96.195.62 link#8 UHS 0 16384 lo0and this is the traffic in pfsense when trying to open freepbx webgui from remote workstation
23:40:27.265712 IP 192.168.25.200.80 > 192.168.26.20.50808: tcp 0 23:40:27.265751 IP 192.168.25.200.80 > 192.168.26.20.50809: tcp 0 23:40:27.515697 IP 192.168.25.200.80 > 192.168.26.20.50810: tcp 0 23:40:30.813344 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 733 23:40:30.813388 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 0 23:40:30.814584 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 733 23:40:30.814619 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 0 23:40:30.830751 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 799 23:40:30.830925 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 776 23:40:30.909501 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 0 23:40:30.909543 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 0 23:40:31.578588 IP 192.168.25.200.80 > 192.168.26.20.50810: tcp 0 23:40:31.578618 IP 192.168.25.200.80 > 192.168.26.20.50809: tcp 0 23:40:31.578639 IP 192.168.25.200.80 > 192.168.26.20.50808: tcp 0 23:40:35.266597 IP 192.168.25.200.80 > 192.168.26.20.50808: tcp 0 23:40:35.266619 IP 192.168.25.200.80 > 192.168.26.20.50809: tcp 0 23:40:35.516305 IP 192.168.25.200.80 > 192.168.26.20.50810: tcp 0 23:40:35.812030 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 733 23:40:35.812149 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 0 23:40:35.813357 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 733 23:40:35.813391 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 0 23:40:35.823793 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 794 23:40:35.824305 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 776 23:40:35.902929 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 0 23:40:35.902986 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 0 23:40:40.810130 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 733 23:40:40.810170 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 0 23:40:40.811211 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 733 23:40:40.811246 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 0 23:40:40.821519 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 800 23:40:40.823006 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 776 23:40:40.899932 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 0 23:40:40.900864 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 0 23:40:43.578720 IP 192.168.25.200.80 > 192.168.26.20.50810: tcp 0 23:40:43.578745 IP 192.168.25.200.80 > 192.168.26.20.50809: tcp 0 23:40:43.578757 IP 192.168.25.200.80 > 192.168.26.20.50808: tcp 0 23:40:45.820510 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 733 23:40:45.820563 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 0 23:40:45.821972 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 733 23:40:45.822014 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 0 23:40:45.833927 IP 192.168.25.76.80 > 192.168.26.20.50786: tcp 797 23:40:45.835247 IP 192.168.25.76.80 > 192.168.26.20.50787: tcp 776 23:40:45.923692 IP 192.168.26.20.50786 > 192.168.25.76.80: tcp 0 23:40:45.923742 IP 192.168.26.20.50787 > 192.168.25.76.80: tcp 0 -
I just updated my last answer cause the capture was wrong
-
@viragomann said in Cannot access webservers through vpn that are on a different gateway:
@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
if I masquerade the remote networks on the central mikrotik behind the lan interface, things works.
So the only two reasons for failing without that I can think off are
- the route doesn't work
- the destination server itself blocks the access
Blocking access from outside its own subnet is the default behavior of system firewalls, however, a webserver should be configured to accept access from anywhere. I assume, the server is accessible from the internet.
@silvered-dragon said in Cannot access webservers through vpn that are on a different gateway:
But honestly I'm not sure that masquerading the remote lan is a good practice.
The only one drawback is that you cannot identify the real source address on the destination device, as long as you do the masquerading only for the remote lan.
I'm 100% sure that there is no issue related on the servers side cause I created new vms with basic configuration, and I cannot access nothing in tcp even a simple debian+ssh
