Forward /29 through gre tunnel and allocate public ips on hosts.

s_serra

Something strange has happened in the last few days. If I ping the ip 185.113.143.50, it goes through the remote pfsense and the local pfsense until it reaches the virtual machine and I get a response. When I ping from inside the virtual machine to the outside, it passes through local and remote pfsense but does not receive the ping response.

Below I send two pcaps the first is me pinging my pc and I get a response the second is me pinging the virtual machine and I don't get a response on the virtual machine. These two pcaps were made in the remote pfsense on the wan interface.

Thanks

11:46:37.039207 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 64165, offset 0, flags [none], proto ICMP (1), length 60)
    185.113.143.50 > 193.137.65.16: ICMP echo reply, id 1, seq 147, length 40
11:46:37.362149 00:00:5e:00:01:0d > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 185.113.143.50 tell 185.113.143.1, length 46
11:46:38.040709 00:00:5e:00:01:0b > 00:16:3c:d5:dc:45, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 120, id 15176, offset 0, flags [none], proto ICMP (1), length 60)
    193.137.65.16 > 185.113.143.50: ICMP echo request, id 1, seq 148, length 40
11:46:38.049015 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 64240, offset 0, flags [none], proto ICMP (1), length 60)
    185.113.143.50 > 193.137.65.16: ICMP echo reply, id 1, seq 148, length 40

11:42:13.334730 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 63699, offset 0, flags [DF], proto ICMP (1), length 84)
    185.113.143.50 > 1.1.1.1: ICMP echo request, id 59694, seq 28, length 64
11:42:14.358678 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 63794, offset 0, flags [DF], proto ICMP (1), length 84)
    185.113.143.50 > 1.1.1.1: ICMP echo request, id 59694, seq 29, length 64
11:42:15.382590 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 63795, offset 0, flags [DF], proto ICMP (1), length 84)
    185.113.143.50 > 1.1.1.1: ICMP echo request, id 59694, seq 30, length 64
11:42:16.406529 00:16:3c:d5:dc:45 > 00:00:5e:00:01:0b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 64029, offset 0, flags [DF], proto ICMP (1), length 84)
    185.113.143.50 > 1.1.1.1: ICMP echo request, id 59694, seq 31, length 64

stephenw10

Either something else is blocking it or 1.1.1.1 is just not responding.

Does that happen for any external IP you try to ping?

Clearly the route to 185.113.143.50 is still good since you are able to ping it.

Steve

s_serra

@stephenw10

Everything I try to ping to the outside gets no response.

In the vlan I just have this rule in the local pfsense. On the remote pfsense I have the firewall turned off, it's just for routing.

stephenw10

Try running a traceroute. It looks like something is blocking outbound traffic but it's not either of the pfSense boxes. The pcap shows the traffic leaving the remote WAN interface exactly as expected.

Steve

s_serra

@stephenw10

185.113.141.145 is my remote pfsense wan ip

stephenw10

I meant traceroute the other way, from 185.113.143.50 out to something. Where it's failing.

s_serra

@stephenw10

root@teste:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:b5:b6:9a:c0:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 185.113.143.50/28 brd 185.113.143.63 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f0b5:b6ff:fe9a:c0aa/64 scope link 
       valid_lft forever preferred_lft forever
root@teste:~# traceroute google.com
traceroute to google.com (142.250.184.174), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  8.151 ms  8.112 ms  8.087 ms
 2  gw-141.i4w.pt (185.113.141.1)  8.250 ms  8.252 ms  8.265 ms
 3  172.16.8.2 (172.16.8.2)  8.402 ms  8.373 ms  8.355 ms
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@teste:~#

root@teste:~# traceroute 185.83.212.22
traceroute to 185.83.212.22 (185.83.212.22), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  8.377 ms  8.336 ms  8.310 ms
 2  gw-141.i4w.pt (185.113.141.1)  8.664 ms  8.635 ms  8.611 ms
 3  NOS.AS2860.gigapix.pt (193.136.251.4)  9.810 ms  9.785 ms  9.949 ms
 4  * * *
 5  * * *
 6  pt1.cr1.as44222.net (185.83.212.11)  13.488 ms  13.176 ms  13.130 ms
 7  * * *
 8  * * *
 9  * * *

.......

stephenw10

Hmm, that 2nd trace looks like it's succeeding, just the target not responding.

Can it ping 185.83.212.22? That responds for me.

Steve

s_serra

@stephenw10

Traceroute is working strangely. Ping does not work.

Curl to ifconfig.me to check external ip doesn't work either.

root@teste:~# traceroute 185.83.212.22
traceroute to 185.83.212.22 (185.83.212.22), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  8.143 ms  8.094 ms  8.068 ms
 2  gw-141.i4w.pt (185.113.141.1)  8.473 ms  8.448 ms  8.424 ms
 3  NOS.AS2860.gigapix.pt (193.136.251.4)  9.113 ms  9.088 ms  9.173 ms
 4  * * *
 5  * * *
 6  pt1.cr1.as44222.net (185.83.212.11)  13.342 ms  13.255 ms  13.206 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@teste:~# ping 185.83.212.22
PING 185.83.212.22 (185.83.212.22) 56(84) bytes of data.
^C
--- 185.83.212.22 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8171ms

stephenw10

What about if you traceroute with ICMP?:

traceroute -I 185.83.212.22

That completes for me here.

s_serra

@stephenw10

It's very strange, sometimes my pc's ping to 185.113.143.50 works, other times it doesn't, without touching anything.

root@teste:~# traceroute -I 185.83.212.22
traceroute to 185.83.212.22 (185.83.212.22), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  8.193 ms  8.163 ms  8.156 ms
 2  gw-141.i4w.pt (185.113.141.1)  8.433 ms  8.427 ms  8.424 ms
 3  NOS.AS2860.gigapix.pt (193.136.251.4)  9.332 ms  9.376 ms  9.587 ms
 4  10.255.184.110 (10.255.184.110)  13.387 ms  13.381 ms  13.407 ms
 5  * * *
 6  pt1.cr1.as44222.net (185.83.212.11)  13.336 ms  13.000 ms  12.967 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@teste:~#

stephenw10

And still nothing shown in the firewall logs as blocked on either pfSense install?

s_serra

@stephenw10

in the local pfsense no, in the remote the firewall is disabled.

stephenw10

Hmm, well I see nothing in pfSense that would be causing a problem here and you say nothing changed there.

I can't ping 185.113.143.50 from here:

PING 185.113.143.50 (185.113.143.50) 56(84) bytes of data.
From 194.38.148.182 icmp_seq=1 Destination Host Unreachable

But I don't know if I should be able to.

If you can't ping into the routed subnet either that looks more like some routing issue. But it doesn't look like it's in pfSense because it can traceroute to something at least as far as the ISP.

Steve

s_serra

@stephenw10

The ip 185.113.143.49 is the ip of the vlan interface of the local pfsense and I think it's always working fine (the icmp is active you can ping it). The rest of the vms that have 185.113.143.49 as a gateway don't work well sometimes it works sometimes it doesn't.

stephenw10

Doesn't work from here:

PING 185.113.143.49 (185.113.143.49) 56(84) bytes of data.
From 194.38.148.182 icmp_seq=1 Destination Host Unreachable
From 194.38.148.182 icmp_seq=2 Destination Host Unreachable

Something filtering the source upstream?

tedquade

@stephenw10 works from my location:

C:\Users\Ted>ping 185.113.143.49

Pinging 185.113.143.49 with 32 bytes of data:
Reply from 185.113.143.49: bytes=32 time=215ms TTL=48
Reply from 185.113.143.49: bytes=32 time=209ms TTL=48
Reply from 185.113.143.49: bytes=32 time=215ms TTL=48
Reply from 185.113.143.49: bytes=32 time=225ms TTL=48

Ping statistics for 185.113.143.49:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 209ms, Maximum = 225ms, Average = 216ms

C:\Users\Ted>

stephenw10

Mmm, still failing here so it looks like something rejecting it for some sources.

Does your route go through 194.38.148.182?

tedquade

@stephenw10 It does not.

I attempted to post the complete traceroute but it was flagged as spam.

Ted

stephenw10

Mmm, this appears to be something in the route. I don't believe this is anything to do with either pfSense box.