How to troubleshoot connection issues between two gateways?

solidak

Hey guys, I installed pfsense in a hosted virtualization solution (VirtualBox). My setup is basically:

Host
- Has 'Host-Only' adapter with address 172.16.1.2
pfsense
- Connected to physical network through "Bridged Networking"
- Connected to host by a 'Host-Only' adapter
- Connected to 'internal network X' as the gateway with IP 172.16.2.1
Arch Linux server
- Connected to 'internal network X' with a static IP of 172.16.2.2

This is just the basic setup. The idea is to have pfsense monitor traffic coming from and to Arch Linux image.

I got pretty much all protocols monitored and Arch Linux can connect to internet. I'd like to create an SSH tunnel from host -> Arch Linux image. So basically from 172.16.1.2 -> 172.16.2.2.

Initially, I defined it as a rule to 'Pass" SSH connections in between those two IPs but that did not work. The host hypervisor could not recognize the IP.

I removed the rule and tried to make everything pretty much allowed, but it still cannot recognize it.

My question is not really how to fix this issue but how to troubleshoot such networking issues in the first place. The only thing that came to mind was to run tracepath from the host hypervisor to 172.16.2.2 but it just printed _gateway twice and no replies...

Also, here is the results of route -n on the host hypervisor:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.2.1     0.0.0.0         UG    600    0        0 wlp2s0
172.16.1.0      0.0.0.0         255.255.255.0   U     0      0        0 vboxnet0
192.168.2.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp2s0

Also running ip route get 172.16.2.2 naturally got me:

172.16.2.2 via 192.168.2.1 dev wlp2s0 src 192.168.2.188 uid 1000
    cache

So it used the wireless network and did not understand a routing through the vboxnet0 host-only adapter towards pfsense.

So a request is going from host hypervisor (172.16.1.2) -> ArchLinux image (172.16.2.2). It should first go to the gateway 172.16.1.1 then another gateway 172.16.2.1 and then 172.16.2.2. All the routes there are clear, but I have no way of confirming that. How can one troubleshoot such an issue?

Sorry for my extended question but I promise I'll linger in the forums to at least pay back this question with two (hopefully helpful) answers to other people's questions :)

P.S: I'm following Tony Robinson's Building Virtual Machine Labs book.

johnpoz

So What are the rules on this host only network 172.16.1 interface in pfsense?

So I take it pfsense wan is your physical network and is what? 192.168.?, 10.?, 172.16-31.? Pfsense has public on its wan?

So this internal X network is your lan? A vm only network? It it can get through your pfsense vm to the internet..

Yeah really need to know what the rules are on this firewall interface your host only network.. And how you have it setup in pfsense. You didn't put a gateway on the interface did you. Did you set the mask to /24?

Can your host ping the pfsense IP 172.16.1.1?

solidak

@johnpoz said in How to troubleshoot connection issues between two gateways?:

So What are the rules on this host only network 172.16.1 interface in pfsense?
So I take it pfsense wan is your physical network and is what? 192.168.?, 10.?, 172.16-31.? Pfsense has public on its wan?
So this internal X network is your lan? A vm only network? It it can get through your pfsense vm to the internet..
Yeah really need to know what the rules are on this firewall interface your host only network.. And how you have it setup in pfsense. You didn't put a gateway on the interface did you. Did you set the mask to /24?
Can your host ping the pfsense IP 172.16.1.1?

I'll answer your questions one by one.

So What are the rules on this host only network 172.16.1 interface in pfsense?

The Host-Only network is the pfsense LAN network. The rules are:

Allow DNS from 172.16.1.0/24 to 172.16.1.1.
Allow NTP from 172.16.1.0/24 to 172.16.1.1.
Allow SSH from 172.16.1.2 (Host Hypervisor) to 172.16.2.2 (ArchLinux)
Block all IPv4+6 from any to RFC1918
Allow HTTPS from 172.16.1.0/24 to any.
Block everything else.

So I take it pfsense wan is your physical network and is what? 192.168.?, 10.?, 172.16-31.? Pfsense has public on its wan?

Yep, pfsense wan is my physical network (Bridged networking). Its 192.168.1.0/24.

So this internal X network is your lan? A vm only network? It it can get through your pfsense vm to the internet..

Internal network X is an 'internal network' in Virtualbox lingo, which acts as OPT1 and it is a vm-only network, yessir. Its in 172.16.2.x interface and it is connected to internet.

The Host-Only network is my LAN.

Yeah really need to know what the rules are on this firewall interface your host only network.. And how you have it setup in pfsense. You didn't put a gateway on the interface did you. Did you set the mask to /24?

I've posted the rules in this reply. There is no gateway in the interface. I didn't specify an 'upstream gateway address' for either the LAN (host-only) or OPT1 (vm internal network X). The subnet mask is 24, yes.

johnpoz

@solidak said in How to troubleshoot connection issues between two gateways?:

Allow SSH from 172.16.1.2 (Host Hypervisor) to 172.16.2.2 (ArchLinux)

That should allow it.. But simple to allow ping to see if you can ping it. But it could be running firewall on archlinux.. Do a simple sniff on pfsense do you see the ssh packet, do you see it go out the opt interface towards your archlinux?

Also keep in mind that screenshot is 1000 times better.. See it all the time users says/thinks they did X when they really did Y. Seen users click on block rfc on their lan, seen add bogon to their local interfaces, etc. And then wonder why something doesn't work.. Take a screenshot of your lan interface so we are sure they are right. And you have nothing in your floating tab right?

So your not allowing ping. Nor are you allowing access to your other network So how would anything your host get anywhere? And your specifically blocking it with this rule
Block all IPv4+6 from any to RFC1918

Your allow rule to any would never be seen since above that you specifically BLOCK to rfc1918.. Rules are evaluated top down, first rule to trigger wins, no other rules evaluated.

So if your trying to go to say 172.16.2.x That block rule stops it, and the allow rule below it never looked at.

solidak

@johnpoz Thanks for the prompt reply. I'm attaching some screenshots now:

I do have Squid Proxy turned on, yes, but its irrelevant since I disabled it for these tests and reverted my http_proxy environment variable.

Also, here is the output of ip a on the host hypervisor, if its relevant.

One thing to note is that I turned on ping and running ping 172.16.2.2 was unfruitful. Also, running a tcpdump -i em1 icmp showed nothing when pinging 172.16.2.2. It did however show an output when I pinged 172.16.1.1.

solidak

Alright, I figured out the issue. I was too stupid to notice but the issue was that my lovely host hypervisor routing table did not know what interface to use to connect to 172.16.2.0/24 subnet. So my solution was:

ip route add 172.16.2.0/24 via 172.16.1.1

and it worked. SSH, ICMP everything. My host hypervisor (172.16.1.2) was able to connect to the ArchLinux server running in a vm internal network (172.16.2.2).

Thanks @johnpoz for replying. This was pretty educational actually :)