pfsense on a bridged VM on dedicated Hetzner
-
@compuser from what you write, you followed the Hetzner docs "Additional IP Addresses / Use with virtualization with the bridged method" or Dedicated Server / Network / Bridged.
What is different from the documentation (referring to the second link) that you have not removed the
enp5s0
part as they do. Hetzner writes (eth0
being the network interface in the doc):"The configuration of eth0 is omitted without replacement."
-
@patient0 I had tried that too and I saw the same issue. I reached out to Hetzner and they pointed me to https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve. They have not been able to provide any more help though.
Anyways, I went back to your config suggestion and this is how my /etc/network/interfaces looks like now. I restarted the host for a good measure. Powered up pfsense and tried curl in shell. After few attempts, its again flaky. On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443
auto lo iface lo inet loopback iface lo inet6 loopback #auto enp5s0 #iface enp5s0 inet manual auto vmbr0 iface vmbr0 inet static address MainIP/26 gateway MainIPGateway bridge_hw enp5s0 bridge_ports enp5s0 bridge_stp off bridge_fd 1 bridge_hello 2 bridge_maxage 12
-
@compuser said in pfsense on a bridged VM on dedicated Hetzner:
On Pfsense firewall logs, I can see TCP:FPA blocked from AdditionalIP:51494 to 208.123.73.209:443
Meaning pfSense is blocking the traffic; TCP:FPA blocked seems to indicate "It's out-of-state traffic, either from expired states or from asymmetric routing." (https://forum.netgate.com/post/292228)
Do you one WAN defined or multiple?
-
I have just one WAN. I havent yet made any changes to the installation except for the startup wizard.
-
@compuser mmh, I'm a bit out of ideas right now. Can't be asymmetic routing then.
Can you do a package capture for, like 208.123.73.209 (or any other external IP) on the WAN interface and check how the data flow is while you try to access the IP?
Is there a TCP:Sync and so on? You don't have to post the capture or if you do replace the public IP with some pattern. -
@patient0 Thank you for looking into this. Yes there are Sync flags going out. Here is a packet capture when it fails.
https://pastebin.com/yNzb2Snv
-
I did a tcpdump on the proxmox host itself to capture the packets between the two sync flags from additional IP. From what I read. the host and gateway swap ICMP packets and then host pings the nameserver for reverse lookup but doesnt do anything after that.
https://pastebin.com/FuS0Sx9X
UPDATE: This is how it looks during a successful run. There is no reverse lookup and you can see the reply from destination.
https://pastebin.com/pDgkn111 -
I believe we have figured out the issue. Thank you so much! I was using hetzner firewall to allow incoming acks on 32678-65535 but looks ephemeral ports are not limited to it. In the failed cases it is <32k. I changed the firewall rule to 1024-65535 and now it works fine. Having said this what is a good rule for incoming acks in a stateless firewall?
-
@compuser said in pfsense on a bridged VM on dedicated Hetzner:
Having said this what is a good rule for incoming acks in a stateless firewall?
I wouldn't set up any firewall before pfSense, pfSense can handle it. There are default rules to block connections to port 0, for example.
And if you ever want to run a service behind pfSense, on port 80 let's say, you'd may forget about the Hetzner firewall and debug for a day before you remember that you blocked ports <1024.
But maybe that's only me who is forgetful :), happened with Hetzner cloud where I set Hetzner cloud firewall rules and forgot about it. And wondered by the f-ing connection to work when I configured the firewall on the cloud server.
-
Sounds good and thanks again for helping out!