Virtual IP unable to access VM (only ping)

McMurphy

I have pfSense hosted as a VM on OVH Cloud

I have a VM behind pfSense that I am having trouble accessing from a public ip.

I can ping the VM from the public IP but not view the GUI

After testing I believe the issue is pfSense

Port 80 on the VM can be accessed from LAN

I am unable to access port 80 from the virtual IP

I have created a NAT for the virtual IP and associated rule

patient0

@McMurphy WAN access to the GUI is of course denied per default because that's a security nightmare. But if you insist:

https://docs.netgate.com/pfsense/en/latest/recipes/remote-firewall-administration.html#i-don-t-care-about-security-how-do-i-open-access-to-the-gui

McMurphy

@patient0

Apologies. My post was ambiguous, since updated. It is not pfsense I am unable to access its a VM behind it.

patient0

@McMurphy I didn't look too closely into the pictures, my bad :)

Did you have a look at port forwarding? I would guess that is what you want:

https://docs.netgate.com/pfsense/en/latest/nat/port-forwards.html

McMurphy

@patient0

One pic shows the port forwarding

patient0

@McMurphy yes, but you don't set a port nor a protocol like mentioned in the guide.

McMurphy

@patient0

Correct. I am allowing all. This is why ping works even through it is not explicitly specified.

patient0

@McMurphy I have no idea how that should work. How can you ping your VM from the internet? How does the router know that that ping is for the VM?

Well, I can't answer that, I didn't think that this is possible, alle protocols and all ports.

McMurphy

@patient0

Port forwarding is setup. Why would that not work?

patient0

@McMurphy how do you test if a ping reaches your VM? Are you running a package capture on the VM and see it coming through.

As I said, someone with a better network understand can answer that. Maybe it's possible.

Why don't you start with just tcp and port 80 and if that works, expand the rule?

McMurphy

@patient0

I can ping the public IP and a response is received. The FW logs show the ICMP entry too. If I shutdown the VM then the ping stops responding.

I did originally start with only port 80 however expanded it to remove all variables.

Driving me crazy :)

mvbif

@McMurphy hello,
First dumb question, ovh firewall on vm is disabled?
Have you tried to capture traffic?
I would capture traffic from a sourc ip from interface wan , and from that source trying ping and telnet . Just to see if traffic is coming.
Then i would check capturing the same source ip on the interface where traffic should be rotaded ,

McMurphy

@mvbif

OVH firewall is disabled.

Ping works all the time so the port forwarding musty be correct and http works randomly

pfSense public IP is: 139.99.215.145
VM public IP is: 139.99.215.146

Not sure about the packet capture sorry

patient0

@McMurphy I have moved a 2.7.2 CE behind another pfSense in the lab and tested you scenario and it does work. But not from the pfSense GUI, with e.g. interface set to WAN. I had to use a external host to test.

Edit: What I didn't do in the first go was your WAN setup with a /32 public IP/netmask and a far away gateway (had it set the netmask to /24).
Done that now and I had to add the second IP as an virtual IP (Firewall / Virtual IPs) to the WAN interface. I assume because with the /32 netmask there is no routing to the second IP since outside of the first public IPs network (with /32 netmask). I was to fast to answer, I do see the tcp packages comming in but not going back.

And I did check your 2nd public ip for ping and tcp port 80 (around 10:30-ish UTC from IP 78.46.48.110). Ping did answer port 80 didn't. I assume there is a service responding to requests to tcp port 80?

Not sure about the packet capture sorry

That would be tcpdump (CLI tool) to capture traffic and it's per default installed on pfSense. The internet is full of infos about it if you are interested. "Wireshark" is a GUI version of it and both can read the package captures created by either of them.

For example if you want to check if a request to port 80 (tcp) reaches your pfSense on the second public IP on you public interface, e.g. called vtnet0 (tcpdump output onto the screen):

tcpdump -ni vtnet0 tcp and port 80

If you see your packages come in: 2nd step check if they get forwarded to your LAN interface (10.44.50.0/24 ?). For that use the same command as above but the LAN interface (let's say vtnet1)

tcpdump -ni vtnet1 tcp and port 80

And if you want to see if any requests reach your VM on port 80, proto tcp, you would run on the VM:

tcpdump -ni <LAN interface towards pfSense> tcp and port 80

McMurphy

@patient0
1st thing - thank you.

I have apache running on port 80 and it can be viewed on the local LAN

Packet capture on WAN

11:06:57.230459 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 0
11:06:57.230772 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:06:57.230934 IP 67.219.99.188.11949 > 139.99.215.146.80: tcp 0
11:06:57.231052 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:06:57.243767 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 0
11:06:57.246695 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:06:57.246816 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:06:57.247479 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:06:57.247491 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:06:57.247493 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 588
11:06:57.283545 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 588
11:06:57.464426 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:06:57.464565 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:06:57.507541 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:06:57.764478 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:06:57.764604 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:06:57.963530 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:06:58.230929 IP 67.219.99.188.11949 > 139.99.215.146.80: tcp 0
11:06:58.231064 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:06:58.364497 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:06:58.364651 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:06:58.859509 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:06:59.243511 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:06:59.564476 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:06:59.564632 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:00.230792 IP 67.219.99.188.11949 > 139.99.215.146.80: tcp 0
11:07:00.230970 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:07:00.651455 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:07:00.764485 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:07:00.764614 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:01.964500 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:07:01.964642 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:02.251399 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:07:02.252560 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:04.230984 IP 67.219.99.188.11949 > 139.99.215.146.80: tcp 0
11:07:04.231140 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:07:04.363323 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:07:04.365441 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:07:04.365561 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:04.975514 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 0
11:07:04.975641 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:08.459195 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:07:09.165397 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 429
11:07:09.165549 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 0
11:07:11.531098 IP 139.99.215.146.80 > 67.219.99.188.33572: tcp 1396
11:07:12.231906 IP 67.219.99.188.11949 > 139.99.215.146.80: tcp 0
11:07:12.232072 IP 139.99.215.146.80 > 67.219.99.188.11949: tcp 0
11:07:18.765465 IP 67.219.99.188.33572 > 139.99.215.146.80: tcp 0

Packet Capture on LAN

11:10:13.867771 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 0
11:10:13.868016 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 0
11:10:13.868216 IP 67.219.99.188.28621 > 10.44.50.104.80: tcp 0
11:10:13.868324 IP 10.44.50.104.80 > 67.219.99.188.28621: tcp 0
11:10:13.885834 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 0
11:10:13.895154 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 429
11:10:13.895258 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 0
11:10:13.895918 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 1396
11:10:13.895931 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 1396
11:10:13.895937 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 588
11:10:13.957886 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 588
11:10:14.127034 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 429
11:10:14.127128 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 0
11:10:14.189876 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 1396
11:10:14.427965 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 429
11:10:14.428067 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 0
11:10:14.665987 IP 10.44.50.104.80 > 67.219.99.188.49904: tcp 1396
11:10:14.865397 IP 67.219.99.188.28621 > 10.44.50.104.80: tcp 0
11:10:14.865512 IP 10.44.50.104.80 > 67.219.99.188.28621: tcp 0
11:10:15.029016 IP 67.219.99.188.49904 > 10.44.50.104.80: tcp 429

I am not sure what these packet captures show but I can see both WAN & LAN are responding to traffic on the public IP

patient0

@McMurphy If I try again from my lab on 78.46.48.110 I don't get any reply back.

Can you add your second public IP as an virtual IP for your WAN interface in pfSense? Without that there is no route for that IP in the routing table.
By doing that the public interface of your OVH instance gets a second IP on that interface.

I can see both WAN & LAN are responding to traffic on the public IP

Indeed it looks like it. What is the client OS you are testing from? I'm used to see a lot more info in the tcpdump output.

McMurphy

@patient0

I already have the 2nd IP added as a virtual IP on WAN in pfSense

I performed the packet capture using pfSense

Thinking this through overnight...

I have a 139.99.215.144/28 block of additional IPs from OVH and have allocated:

139.99.215.145 as the pfSense WAN IP
139.99.215.146 as a virtual IP on WAN

The WAN IP is enabled to allow remote admin of pfSense and the GUI loads without issue. I have removed the whitelisting to allow confirmation.

To remove (2) as a variable I have port forwarded 8443 on the WAN subnets to the Debain VM (port 80), this means I am able to access the Debain VM on:

139.99.215.145:8443
139.99.215.146:8443

Both IPs respond when using curl however slowly partially load the page before:
"curl: (56) Recv failure: Connection reset by peer"

mvbif

@McMurphy Hello,
So now seems that at network level the connection is working,
Now you should check on apache logs to see if it get's the request, and if there any error about.
Connection reset by peer, should be that you reached the server.