Connectivity issue thru IPSEC Tunnel

BBcan177

I am having an issue with an Ubuntu Machine.

A Network 10.10.1.0/24

B Network 10.10.2.0/24

C Network 10.10.3.0/24

I have setup an Ipsec VPN tunnel from A - B, and A - C (all pfsense Boxes)

I have an Ubuntu Server on A network. An ubuntu machine on B network.

When I ping/ssh from the Ubuntu machine on B to A network, i am getting a Host Unreachable/Destination Host Unreachable
The Ubuntu machine can resolve the host and Ip as is confirmed with a DIG -x command. The Ubuntu machine on B can ping the local pfsense router and anything local or internet based. But it cant ping anything on the A network including the A router. All other devices have no issue. Just this one ubuntu machine.

I have no issue with connectivity between the A and C networks.

If I run this command on the Ubuntu machine in B network

sysctl -w net.ipv4.ip_forward=1

I can ping/ssh from A <-> B. The ubuntu machine has one NIC and two additional for a TAP monitoring system so they are set to

eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xx.xx.xx.xx Bcast:xx.xx.xx.255 Mask:255.255.255.0
inet6 addr: xxxx::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:750659 errors:0 dropped:0 overruns:0 frame:0
TX packets:460220 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:122675477 (122.6 MB) TX bytes:409259079 (409.2 MB)
Interrupt:19 Memory:f0180000-f01a0000

eth1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:4857110 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1017130172 (1.0 GB) TX bytes:0 (0.0 B)
Interrupt:16 Memory:f0280000-f02a0000

eth2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:16 Memory:f0300000-f0320000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:554233 errors:0 dropped:0 overruns:0 frame:0
TX packets:554233 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:783922279 (783.9 MB) TX bytes:783922279 (783.9 MB)

route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 xx.xx.xx.xx 0.0.0.0 UG 100 0 0 eth0
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0

So when the "sysctl -w net.ipv4.ip_forward=1" ping and ssh works but the traceroute doesnt seem as expected.
I dont understand how the machine is forwarding when only one NIC has an address?

PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.

From xx.xx.xx.xx: icmp_seq=1 Redirect Host(New nexthop: xx.xx.xx.xx)
64 bytes from xx.xx.xx.xx: icmp_req=1 ttl=63 time=46.8 ms

traceroute xx.xx.xx.xx (Traceroute from SO Sensor to SO Server)

traceroute to xx.xx.xx.xx (xx.xx.xx.xx), 30 hops max, 60 byte packets
1 xx.xx.xx.xx (xx.xx.xx.xx) 0.545 ms 0.532 ms 0.519 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

There are no Blocks in IPTables and UFW is set to allow the connectivity.
If anyone has any suggestions, I would appreciate it as I've tried several things to fix this issue without success.

BBcan177

I have found that following these instructions

https://doc.pfsense.org/index.php/Why_can't_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F

click System > Routing. On the Gateways, tab, click + and add a gateway using your LAN IP address (check the box to disable monitoring). Save/Apply,
then go to the Static Routes tab, click +, enter the remote VPN network in the "Destination Network" box, select the LAN IP gateway that was created before, and add a description
if you want, then Save/Apply.

Once i removed the manual route, rebooted pfSense. My ubuntu machine was able to communicate thru the tunnel.

Why this stopped working all of a sudden is a mystery.

Also, I am directing the pfSense Syslogs thru the tunnel to a remote syslog server and since removing the manual route, it is not working.

Any suggestions would be appreciated.