pfsense on a bridged VM on dedicated Hetzner

compuser

Hi I have a dedicated server on Hetzner with a main IP. I have installed proxmox and cut a VM on which I am running pfsense on a bridged mode. I have an additional IP whose MAC has been assigned to the pfsense' NIC. When I turn on pfsense, I am able to curl external domains from the shell, say

curl -4 -v google.com

without any issues for few minutes. After that one in 3-4 attempts succeeds. The rest freeze at

Trying 216.58.211.238:80

and finally timing out. FWIW, I am able to ping this IP without any issues from the pfsense shell. I am also able to curl from the host. I have not made any extra configuration on pfsense beyond the going through the initial wizard. I have just the WAN on pfsense and no LAN. On pfsense firewall logs, I see blocked outgoing TCP:RPA packets. From what I read this can be due to asymmetrical network. If I use a regular ubuntu VM in place of the pfsense VM, I am able to curl without issues. Which makes me believe there might be some configuration on pfSense that I am missing. How do I fix this intermittent network issue. Thanks for your time!

/etc/network/interfaces on the proxmox host

auto enp5s0
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address MainIP/26
        gateway MainIPGateway
        bridge_ports enp5s0
        bridge_stp off
        bridge_fd 0

patient0

@compuser from what you write, you followed the Hetzner docs "Additional IP Addresses / Use with virtualization with the bridged method" or Dedicated Server / Network / Bridged.

What is different from the documentation (referring to the second link) that you have not removed the enp5s0 part as they do. Hetzner writes (eth0 being the network interface in the doc):

"The configuration of eth0 is omitted without replacement."

compuser

@patient0 I had tried that too and I saw the same issue. I reached out to Hetzner and they pointed me to https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve. They have not been able to provide any more help though.

Anyways, I went back to your config suggestion and this is how my /etc/network/interfaces looks like now. I restarted the host for a good measure. Powered up pfsense and tried curl in shell. After few attempts, its again flaky. On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443

auto lo
iface lo inet loopback

iface lo inet6 loopback

#auto enp5s0
#iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address MainIP/26
        gateway  MainIPGateway
        bridge_hw enp5s0
        bridge_ports enp5s0
        bridge_stp off
        bridge_fd 1
        bridge_hello 2
        bridge_maxage 12

patient0

@compuser said in pfsense on a bridged VM on dedicated Hetzner:

On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443

Meaning pfSense is blocking the traffic; TCP:FPA blocked seems to indicate "It's out-of-state traffic, either from expired states or from asymmetric routing." (https://forum.netgate.com/post/292228)

Do you one WAN defined or multiple?

compuser

I have just one WAN. I havent yet made any changes to the installation except for the startup wizard.

patient0

@compuser mmh, I'm a bit out of ideas right now. Can't be asymmetic routing then.

Can you do a package capture for, like 208.123.73.209 (or any other external IP) on the WAN interface and check how the data flow is while you try to access the IP?
Is there a TCP:Sync and so on? You don't have to post the capture or if you do replace the public IP with some pattern.

compuser

@patient0 Thank you for looking into this. Yes there are Sync flags going out. Here is a packet capture when it fails.

https://pastebin.com/yNzb2Snv

compuser

I did a tcpdump on the proxmox host itself to capture the packets between the two sync flags from additional IP. From what I read. the host and gateway swap ICMP packets and then host pings the nameserver for reverse lookup but doesnt do anything after that.

https://pastebin.com/FuS0Sx9X

UPDATE: This is how it looks during a successful run. There is no reverse lookup and you can see the reply from destination.
https://pastebin.com/pDgkn111

compuser

I believe we have figured out the issue. Thank you so much! I was using hetzner firewall to allow incoming acks on 32678-65535 but looks ephemeral ports are not limited to it. In the failed cases it is <32k. I changed the firewall rule to 1024-65535 and now it works fine. Having said this what is a good rule for incoming acks in a stateless firewall?

patient0

@compuser said in pfsense on a bridged VM on dedicated Hetzner:

Having said this what is a good rule for incoming acks in a stateless firewall?

I wouldn't set up any firewall before pfSense, pfSense can handle it. There are default rules to block connections to port 0, for example.

And if you ever want to run a service behind pfSense, on port 80 let's say, you'd may forget about the Hetzner firewall and debug for a day before you remember that you blocked ports <1024.

But maybe that's only me who is forgetful :), happened with Hetzner cloud where I set Hetzner cloud firewall rules and forgot about it. And wondered by the f-ing connection to work when I configured the firewall on the cloud server.

compuser

Sounds good and thanks again for helping out!