1:1Nat, two public IPs for one server with one nic

leonidas-o

@rcoleman-netgate I think I could figure out the proxmox part. I played around with it a little bit, especially the post routing part, and while I entered through my main public IP A via wireguard, my IP outside was public IP B. So this makes sense for me right now:

auto enp0s31f6
iface enp0s31f6 inet static
  address 94.PUBLIC.IP.A
  netmask 255.255.255.192
  gateway 94.x.x.x
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  post-up ip addr add 94.PUBLIC.IP.B/26 dev $IFACE label $IFACE:0
  post-down ip addr del 94.PUBLIC.IP.B/26 dev $IFACE label $IFACE:0
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.PUBLIC.IP.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.PUBLIC.IP.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.PUBLIC.IP.B -j DNAT --to 10.10.10.3

iface enp0s31f6 inet6 static
  address 2a01:4f8:10b:2ee8::2
  netmask 64
  gateway fe80::1


auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/29
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.A
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.A
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.3/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.B
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.3/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.B

So now back at the 1:1 Nat issue/ Virtual IP. As you can see (vmbr10) I'm using 10.10.10.1/29 as the address and 10.10.10.2 for WAN. Now I thought, I will simply add 10.10.10.3 to the post routings and inside pfsense create a virtual IP (IP Alias) with 10.10.10.3 and a 1:1 Nat (WAN external: 10.10.10.3 to internal MY_VM_IP).
I don't know how virtual IPs (IP Alias) behave, right now it's not working but It has to be something in this direction, right?

P.S. 10.10.10.3 does not exists as a real interface of pfsense, just as virtual IP (IP Alias). Shouldn't I be able to route the packets to it like I do it with 10.10.10.2?

viragomann

@leonidas-o
Yes, an IP alias is just an additional IP on the interface. If assigned, pfSense answers ARP requests for this IP.

Run a packet capture on pfSense on WAN, enter 10.10.10.3 into the IP filter box, while you try to access the additional public IP from outside to see if the packets are forwarded correctly.

leonidas-o

@viragomann they are actually not routed correctly.

  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.x.x.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.x.x.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to 10.10.10.3

The prerouting here is not able to distinct between -d 94.x.x.A and -d 94.x.x.B and I don't know why. It always uses the rule with -d 94.x.x.A and routes to --to 10.10.10.2. Even if I manually type https://94.x.x.B into the browser, I end up at the traefik proxy which means the first rule of these three was used.

But for me the overview looks fine. Do you see anything, am I missing anything iptables specific?

iptables -t nat -L -n

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            94.x.x.A        multiport dports  !22,8006,179 to:10.10.10.2
DNAT       udp  --  0.0.0.0/0            94.x.x.A        multiport dports  !5405:5412,4789 to:10.10.10.2
DNAT       all  --  0.0.0.0/0            94.x.x.B         to:10.10.10.3

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
SNAT       all  --  10.10.10.2           0.0.0.0/0            to:94.x.x.A
SNAT       all  --  10.10.10.3           0.0.0.0/0            to:94.x.x.B

viragomann

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Even if I manually type https://94.x.x.A into the browser

You mean https://94.x.x.B, I guess.

Did you investigate this with packet capture on pfSense, or is this your assumption of what you see?

leonidas-o

@viragomann oh yes, I meant Ip address B. Corrected that typo.

No haven't checked that with packet capture as I ended Up at a "This connection is not private" site and when clicked on "view certificate" it shows "TRAEFIK DEFAULT CERT - self signed cert". Therefore it's pretty clear that it's traefik. If it would work either the 1:1 Nat works and the bbb service should be shown or simply timing out as firewall rules are maybe missing or whatever, but not traefik self signed cert.

I can still do a packet capture, maybe it reveals something helpful.

viragomann

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Therefore it's pretty clear that it's traefik

Agree, you see that traefik is responding. But you don't determine, why the packets go to traefik.
So is it on Proxmox or on pfSense??

leonidas-o

@viragomann I got the pfsense virtualised, so the entry point is proxmox (the Proxmox/Debian network/interfaces file controls to redirect "almost" everything to pfsense). Therefore I have to solve that iptables prerouting issue first. Right now, it doesn't make any sense to me, why it is ignoring the destination IPs.

viragomann

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Therefore I have to solve that iptables prerouting issue first.

Both of your public IPs go to the same pfSense interface, even each to a specific private IP, assumed it works properly.
But how did you determine, that Proxmox does the forwarding wrong without knowing which IP pfSense is seeing?

You can sniff the traffic either on pfSense WAN or on the Proxmox bridge, but without doing this you're standing in the dark at all.

leonidas-o

@viragomann I have a port forwarding for traefik:

Interface: WAN    Proto: TCP/UDP    Source addr: *    Source ports: *    Dest address: WAN address    Dest ports: 443 (HTTPS)    NAT IP: 10.1.1.32    NAT ports: 4430

Is uses "WAN address" (10.10.10.2) for destination address. I Don't have any port forwardings for the virtual IP (10.10.10.3), only the 1:1 Nat setting.

So, I would expect, that it forwards only the 10.10.10.2 traefik and would distinguish between the real Wan interface and the virtual IP. If it can't do that, how would you ever be able to achieve what I want?

viragomann

@leonidas-o
I know this rule, but I don't know your others.
I's better assure of something than do some assumptions of it. But this seems to feckless here.

leonidas-o

@viragomann yes sure, I agree with the assumptions point.
Do I actually need port forwardings even though when having a 1:1 NAT entry setup or should it be doing something like that?

I'm asking because I think I have something here. Your statement actually made me think:

"Both of your public IPs go to the same pfSense interface..."

I was wondering If I maybe need a port forwarding rule which is in front of the that mentioned traefik port forwarding. My thoughts were like, what if dest address as "WAN address" catches everything, so like all virtual IPs which belong to that WAN address. Would actually make sense, kind of.
And if you want to distinguish, you have to place a port forwarding for each of the VIPs (it is even available via a dropdown) in front of it like:

Interface: WAN    Proto: *    Source addr: *    Source ports: *    Dest address: 10.10.10.3    Dest ports: *    NAT IP: 10.1.1.57    NAT ports: *

I refreshed the page with https://94.x.x.B and was not seeing the traefik self signed cert, but the BBB services self signed cert. I think that's it, it could be that you are a true hero good sir.

But what is the 1:1 Nat in the end doing If still need port forwardings and firewall rules?

viragomann

@leonidas-o
No, the NAT 1:1 rule does the port forwarding, it doesn't need an additional rule for this if set properly.

An 1:1 rule on WAN forward packets destined to, say 10.10.10.3, to 10.1.1.57 and translates the source IP in upstream packets from 10.1.1.57 to 10.10.10.3.

If your rule doesn't work recheck the settings.

SteveITS

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

still need port forwardings and firewall rules

https://docs.netgate.com/pfsense/en/latest/nat/1-1.html
"All traffic initiated on the Internet destined for the specified public IP address on the mapping will be translated to the private IP address, then evaluated against the firewall ruleset on the inbound WAN interface. If matching traffic is permitted by the firewall rules to a target of the private IP address, it will be passed to the internal host."

leonidas-o

@viragomann @steveits thanks guys, yeah I'm starting to get a feeling for its behaviour. The most important part for me actually is: https://docs.netgate.com/pfsense/en/latest/nat/port-forwards.html#port-forwarding-and-1-1-nat

"Port forwards also take precedence over 1:1 NAT. If a port forward is defined on one external IP address forwarding a port to a host, and a 1:1 NAT entry is also defined on the same external IP address forwarding everything into a different host, then the port forward remains active and continues forwarding to the original host."

"Port forwards take precedence over 1:1 NAT", so as we found out that "WAN Address" matches the VIPS as well, I will have to change my port forward rules to just use the original WAN address 10.10.10.2 and ignore VIP (10.10.10.3).

I have to work on the BBB docker deployment atm, it is horrible to be honest, before I can test everything out. Will report back asap, but I think that's the key actually to get all working.

leonidas-o

Couldn't make BigBlueButton work behind pfsense/opnsense with 1:1 NAT + Reflection etc., so I gave up on that approach. I still found a solution assigning the second public IP directly to the BBB VM, which I documented here: https://serverfault.com/questions/1121061/assigned-second-public-ip-to-vm-from-outside-not-reachable/1121266#1121266