Site to site ping with VLAN?

killmasta93

@marvosa Thanks for the reply, correct i fixed that part which i configured wrong but still cannot ping 192.168.60.100

this is my Server config of pfsense site 1

dev ovpns2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 181.143.xx.xx
ifconfig 192.168.90.1 192.168.90.2
lport 1195
management /var/etc/openvpn/server2.sock unix
max-clients 10
route 192.168.15.0 255.255.255.0
secret /var/etc/openvpn/server2.secret
compress

pfsense site 2

dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 181.57.xx.xxx
lport 0
management /var/etc/openvpn/client1.sock unix
remote 181.143.xx.xx 1195
ifconfig 192.168.90.2 192.168.90.1
route 192.168.0.0 255.255.252.0
route 192.168.60.0 255.255.255.0
secret /var/etc/openvpn/client1.secret 
compress 
resolv-retry infinite

site 2 pics
Screenshot from 2019-11-27 16-41-27.png

site 1 pics
Screenshot from 2019-11-27 16-42-20.png

marvosa

Configs look good, so once packets from site 2 hit the firewall that are destined for 192.168.60.0/24, they should get routed to site 1. Conversely, once packets from site 1 hit the firewall that are destined for 192.168.15.0/24, they should get routed to site 2.

At this point, it's all going to be troubleshooting, I would run a traceroute on both ends, see where the packets stop and troubleshoot from that point. Something is interrupting (or blocking) packet delivery, so every point along the data path will need to be traced and examined. Start checking routing tables, conflicting static routes, firewall logs, and re-examining firewall rules. I would also verify that there are no persistent routes on the endpoints you're testing with.

A couple of things I see that should be looked at:

On the LAN tab on site 2, that LAN net/192.168.60.0/24 is redundant (since there's a LAN net/any right below it), but since the rule is there... it does appear to give some insight. There do not appear to be any hits on that LAN net/192.168.60.0/24 rule, which suggests that traffic may not be making it to the firewall, so that will need to be looked at.
I'm not sure what the TELEFONIAIP tab is (maybe VLAN 60?) but if it's something that you're trying to hit from site 2, there also appear to be no hits on the any/any rule, so it looks like traffic isn't making it there.

We also have no insight into the network design at each end, so if there are any downstream routers, L3 switches, dual WAN, isolated networks behind a 2nd firewall, etc... that will all have to be looked at.

killmasta93

@marvosa Thank you for the reply, so troubleshooting this still cannot ping the VLAN 60.

On the LAN tab on site 2, that LAN net/192.168.60.0/24 is redundant (since there's a LAN net/any right below it), but since the rule is there... it does appear to give some insight. There do not appear to be any hits on that LAN net/192.168.60.0/24 rule, which suggests that traffic may not be making it to the firewall, so that will need to be looked at.

i added the rule thinking it was needed but i guess its not needed

I'm not sure what the TELEFONIAIP tab is (maybe VLAN 60?) but if it's something that you're trying to hit from site 2, there also appear to be no hits on the any/any rule, so it looks like traffic isn't making it there.

correct that is the VLAN60 but howcome no traffic even if i put on the VPN to be able to see it?

as for the traceroute
i ran from a server which has 192.168.15.253 on site and got this on the states of site 2 pfsense

LAN	udp	192.168.15.216:40450 -> 192.168.60.100:33469	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:40450 -> 192.168.60.100:33469	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:49537 -> 192.168.60.100:33470	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:49537 -> 192.168.60.100:33470	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:60136 -> 192.168.60.100:33471	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:60136 -> 192.168.60.100:33471	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:39725 -> 192.168.60.100:33472	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:39725 -> 192.168.60.100:33472	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:37392 -> 192.168.60.100:33473	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:37392 -> 192.168.60.100:33473	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:37383 -> 192.168.60.100:33474	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:37383 -> 192.168.60.100:33474	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:47801 -> 192.168.60.100:33475	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:47801 -> 192.168.60.100:33475	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:43818 -> 192.168.60.100:33476	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:43818 -> 192.168.60.100:33476	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:41138 -> 192.168.60.100:33477	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:41138 -> 192.168.60.100:33477	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:45075 -> 192.168.60.100:33478	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:45075 -> 192.168.60.100:33478	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:43683 -> 192.168.60.100:33479	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:43683 -> 192.168.60.100:33479	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:58588 -> 192.168.60.100:33480	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:58588 -> 192.168.60.100:33480	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:39537 -> 192.168.60.100:33481	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:39537 -> 192.168.60.100:33481	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:37876 -> 192.168.60.100:33482	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:37876 -> 192.168.60.100:33482	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B	
LAN	udp	192.168.15.216:45678 -> 192.168.60.100:33483	NO_TRAFFIC:SINGLE	1 / 0	60 B / 0 B	
ovpns2	udp	192.168.15.216:45678 -> 192.168.60.100:33483	SINGLE:NO_TRAFFIC	1 / 0	60 B / 0 B

and this is the traceroute

traceroute 192.168.60.100
traceroute to 192.168.60.100 (192.168.60.100), 30 hops max, 60 byte packets
 1  192.168.15.254 (192.168.15.254)  0.405 ms  0.397 ms  0.385 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

on the states on pf Sense site1 i never see any traffic on must i be missing something?

Thank you

marvosa

As you can see from the traceroute, the traffic never makes it to the tunnel. The question is why. If there's no added complexity behind PFsense on either end, the first thing I would do is start checking routing tables on both endpoints and both PFsense boxes.

If you post the routing tables we may be able to help. Also, are either of your endpoints dual-homed?

killmasta93

Thanks for the reply, thats correct thats what bugging me the rules are simple on both firewalls

This is my routing tables pfsense site 1

default	181.143.xx.xx	UGS	1010912193	1500	vtnet1	
8.8.8.8	181.143.xx.xx	UGHS	6852794	1500	vtnet1	
10.0.3.192/26	link#13	U	12109832	1500	vtnet0.20	
10.0.3.194	link#13	UHS	0	16384	lo0	
10.10.10.1	link#1	UHS	27686	16384	lo0	
10.10.10.1/32	link#1	U	0	1500	vtnet0	
127.0.0.1	link#5	UH	173093818	16384	lo0	
181.57.194.xx/30	link#3	U	20474263	1500	vtnet2	
181.57.194.xx	link#3	UHS	3	16384	lo0	
181.143.xx.xx/29	link#2	U	0	1500	vtnet1	
181.143.xx.xx	link#2	UHS	0	16384	lo0	
181.143.xx.xx/32	link#2	U	0	1500	vtnet1	
181.143.xx.xx	link#2	UHS	305	16384	lo0	
181.143.xx.xx/32	link#2	U	0	1500	vtnet1	
181.143.xx.xx	link#2	UHS	194	16384	lo0	
181.143.xx.xx/32	link#2	U	0	1500	vtnet1	
181.143.xx.xx	link#2	UHS	0	16384	lo0	
181.143.xx.xx/32	link#2	U	0	1500	vtnet1	
181.143.xx.xx	link#2	UHS	0	16384	lo0	
192.168.0.0/22	link#1	U	1163422258	1500	vtnet0	
192.168.3.254	link#1	UHS	1496054	16384	lo0	
192.168.5.0/24	link#17	U	4110530	1500	vtnet0.5	
192.168.5.2	link#17	UHS	0	16384	lo0	
192.168.7.32/27	link#8	U	279119732	1500	vtnet0.10	
192.168.7.62	link#8	UHS	0	16384	lo0	
192.168.7.64/28	link#22	U	0	1500	vtnet0.7	
192.168.7.70	link#22	UHS	0	16384	lo0	
192.168.12.0/24	link#10	U	21201511	1500	vtnet0.12	
192.168.12.254	link#10	UHS	0	16384	lo0	
192.168.15.0/24	192.168.90.2	UGS	324	1500	ovpns2	
192.168.50.0/24	link#18	U	745106863	1500	vtnet0.50	
192.168.50.2	link#18	UHS	0	16384	lo0	
192.168.51.0/24	link#20	U	39831878	1500	vtnet0.51	
192.168.51.2	link#20	UHS	0	16384	lo0	
192.168.52.0/24	link#19	U	395334421	1500	vtnet0.52	
192.168.52.2	link#19	UHS	0	16384	lo0	
192.168.60.0/24	link#21	U	3543023	1500	vtnet0.60	
192.168.60.254	link#21	UHS	1	16384	lo0	
192.168.90.1	link#25	UHS	0	16384	lo0	
192.168.90.2	link#25	UH	3	1500	ovpns2	
192.168.100.0/24	192.168.100.2	UGS	85978764	1500	ovpns1	
192.168.100.1	link#24	UHS	0	16384	lo0	
192.168.100.2	link#24	UH	3629147	1500	ovpns1	
208.69.xx.xx	181.57.194.xx	UGHS	6852765	1500	vtnet2

site 2 pfsense

127.0.0.1	link#4	UH	474	16384	lo0	
181.57.181.xx/30	link#2	U	647756	1500	vtnet1	
181.57.181.xx	link#2	UHS	0	16384	lo0	
192.168.0.0/22	192.168.90.1	UGS	7989	1500	ovpnc1	
192.168.15.0/24	link#1	U	46543	1500	vtnet0	
192.168.15.254	link#1	UHS	0	16384	lo0	
192.168.60.0/24	192.168.60.2	UGS	64	1500	ovpns2	
192.168.60.1	link#8	UHS	0	16384	lo0	
192.168.60.2	link#8	UH	18790	1500	ovpns2	
192.168.90.1	link#7	UH	0	1500	ovpnc1	
192.168.90.2	link#7	UHS	0	16384	lo0

marvosa

Site 1 shows 192.168.15.0/24 being routed towards 192.168.90.2 through the ovpns2 interface, which is site 2's end of the tunnel and is expected
Site 2 shows 192.168.0.0/22 being routed towards 192.168.90.1 through the ovpnc1 interface, which is site 1's end of the tunnel and is expected
However, Site 2 also shows 192.168.60.0/24 being routed towards 192.168.60.2 through the ovpns2 interface. <---Here's your issue... and it's the reason why traffic is never leaving site 2. Based on the site 2 config you posted, we should be seeing 192.168.60.0/24 being routed towards 192.168.90.1 on the ovpnc1 interface.

So, you have a conflict @ site 2 that is screwing up your routing. Per site 2's routing table, 192.168.60.0/24 is already being routed somewhere else in another server instance (ovpns2), which should correspond to server2.conf. You will have to find the server instance that's running at site 2 with the duplicate entry of 192.168.60.0/24 somewhere in the config and make adjustments accordingly.

killmasta93

Thanks for the reply, so the issue is on the site 2 pfsense?
so from what i understand that site 2 is routing the the network 192.168.60.0/24 in another server?
I checked on the site 2 pfSense box any rules that has that network but nothing. wouldn't that route be what i put on the site to site option to route?
My other question is why does it show ovpns2?

Thank you

killmasta93

o wait forget it
your right i forgot the pfsense 2 has VPN server with 60 o god i feel dump
sorry
thank you again

killmasta93

the new routing table on site 2 pfsense

default	181.57.181.xx	UGS	46042966	1500	vtnet1	
127.0.0.1	link#4	UH	494	16384	lo0	
181.57.181.xx/30	link#2	U	670282	1500	vtnet1	
181.57.181.xx	link#2	UHS	0	16384	lo0	
192.168.0.0/22	192.168.90.1	UGS	17521	1500	ovpnc1	
192.168.15.0/24	link#1	U	46559	1500	vtnet0	
192.168.15.254	link#1	UHS	0	16384	lo0	
192.168.80.0/24	192.168.80.2	UGS	0	1500	ovpns2	
192.168.80.1	link#8	UHS	0	16384	lo0	
192.168.80.2	link#8	UH	0	1500	ovpns2	
192.168.90.1	link#7	UH	0	1500	ovpnc1	
192.168.90.2	link#7	UHS	0	16384	lo0

Should i reboot because the ping is not working though

marvosa

Correct, the issue is at site 2. It looks like you made adjustments to ovpns2, which is the first step, so now we need to get the correct route added for 192.168.60.0/24. Rebooting will work, but we can use that as the last resort cause we should be able to resolve this without a reboot. I would try bouncing the tunnel first in one of two ways:

Status -> Services, then restart the service that is running the tunnel to site 1
VPN -> OpenVPN -> Servers, then edit the server that is running the tunnel to site 1 and re-save it.

After the tunnel is bounced, check the routing table on site 2... there should now be a route for 192.168.60.0/24 using 192.168.90.1 as the gateway on the ovpnc1 interface. If not, then go ahead and reboot site 2.

Once you see the route mentioned above, you should now be able to communicate across the tunnel to VLAN 60.

killmasta93

Thank you for the reply, that did the trick i rebooted and it started to work flawless
Thank you again for all the help