pfsense on a bridged VM on dedicated Hetzner

patient0

@compuser from what you write, you followed the Hetzner docs "Additional IP Addresses / Use with virtualization with the bridged method" or Dedicated Server / Network / Bridged.

What is different from the documentation (referring to the second link) that you have not removed the enp5s0 part as they do. Hetzner writes (eth0 being the network interface in the doc):

"The configuration of eth0 is omitted without replacement."

compuser

@patient0 I had tried that too and I saw the same issue. I reached out to Hetzner and they pointed me to https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve. They have not been able to provide any more help though.

Anyways, I went back to your config suggestion and this is how my /etc/network/interfaces looks like now. I restarted the host for a good measure. Powered up pfsense and tried curl in shell. After few attempts, its again flaky. On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443

auto lo
iface lo inet loopback

iface lo inet6 loopback

#auto enp5s0
#iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address MainIP/26
        gateway  MainIPGateway
        bridge_hw enp5s0
        bridge_ports enp5s0
        bridge_stp off
        bridge_fd 1
        bridge_hello 2
        bridge_maxage 12

patient0

@compuser said in pfsense on a bridged VM on dedicated Hetzner:

On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443

Meaning pfSense is blocking the traffic; TCP:FPA blocked seems to indicate "It's out-of-state traffic, either from expired states or from asymmetric routing." (https://forum.netgate.com/post/292228)

Do you one WAN defined or multiple?

compuser

I have just one WAN. I havent yet made any changes to the installation except for the startup wizard.

patient0

@compuser mmh, I'm a bit out of ideas right now. Can't be asymmetic routing then.

Can you do a package capture for, like 208.123.73.209 (or any other external IP) on the WAN interface and check how the data flow is while you try to access the IP?
Is there a TCP:Sync and so on? You don't have to post the capture or if you do replace the public IP with some pattern.

compuser

@patient0 Thank you for looking into this. Yes there are Sync flags going out. Here is a packet capture when it fails.

https://pastebin.com/yNzb2Snv

compuser

I did a tcpdump on the proxmox host itself to capture the packets between the two sync flags from additional IP. From what I read. the host and gateway swap ICMP packets and then host pings the nameserver for reverse lookup but doesnt do anything after that.

https://pastebin.com/FuS0Sx9X

UPDATE: This is how it looks during a successful run. There is no reverse lookup and you can see the reply from destination.
https://pastebin.com/pDgkn111

compuser

I believe we have figured out the issue. Thank you so much! I was using hetzner firewall to allow incoming acks on 32678-65535 but looks ephemeral ports are not limited to it. In the failed cases it is <32k. I changed the firewall rule to 1024-65535 and now it works fine. Having said this what is a good rule for incoming acks in a stateless firewall?

patient0

@compuser said in pfsense on a bridged VM on dedicated Hetzner:

Having said this what is a good rule for incoming acks in a stateless firewall?

I wouldn't set up any firewall before pfSense, pfSense can handle it. There are default rules to block connections to port 0, for example.

And if you ever want to run a service behind pfSense, on port 80 let's say, you'd may forget about the Hetzner firewall and debug for a day before you remember that you blocked ports <1024.

But maybe that's only me who is forgetful :), happened with Hetzner cloud where I set Hetzner cloud firewall rules and forgot about it. And wondered by the f-ing connection to work when I configured the firewall on the cloud server.

compuser

Sounds good and thanks again for helping out!