1:1Nat, two public IPs for one server with one nic

[leonidas-o]

Hello guys,

got a "little" issue with 1:1 Nat, feels like one little piece is missing. I hope someone can point me into the right direction.
From my provider, I ordered a second public IP address, which should be used for my VM with BigBlueButton.

On one of my Proxmox nodes, I'm using the following network/interfaces conf, which routes almost everything to my firewall, except the specified tcp (22,8006,179) and upd (5405:5412,4789) ports:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
  address 94.x.x.a
  netmask 255.255.255.192
  gateway 94.x.x.x
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2


iface enp0s31f6 inet6 static
  address 2a01:x:x:x::2
  netmask 64
  gateway fe80::1


auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j MASQUERADE
# WAN - Proxmox LAN

iface vmbr10 inet6 static
        address 2a01:x:x:x::3/127
        up ip route add 2a01:x:x:x::/64 via 2a01:x:x:x::4


auto vmbr11
iface vmbr11 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
# VM Net



source /etc/network/interfaces.d/*

So this network/interfaces conf file only contains the main public IP address in enp0s31f6. I hope that's correct so far. Don't know if I have to have anything here for the second public IP? For my second public IP address, I've done the following (https://docs.netgate.com/pfsense/en/latest/nat/1-1.html#configuring-a-1-1-nat-rule).

1. Setup a virtual IP

Virtual IP: 94.x.xx.b/32    Interface: WAN    Type: IP Alias

2. Setup 1:1 NAT

Interface: WAN    External: 94.xx.xx.b/32    Internal: 10.1.1.57    Destination: *

As a background info, my current firewall setup contains port forwardings, e.g. which forward all 443 traffic to my traefik proxy VM:
I had the following where source address was "*", so everything on 443 was forwarded to traefik.

Interface: WAN    Proto: TCP/UDP    Source addr: *    Source ports: *    Dest address: WAN address    Dest ports: 443 (HTTPS)    NAT IP: 10.1.1.32    NAT ports: 4430

My 1:1 Nat was not working (I ended up all the time at the traefik proxy), so I changed the source address from "*" to my main public IP address (94.xx.xx.a). The idea was everything which is accessing the server on Port 443 from the main public IP will be redirected to traefik. And now if accessing 443 from the second public IP, 1:1 Nat should simply redirect to my BBB VM.

Interface: WAN    Proto: TCP/UDP    Source addr: 94.xx.xx.a    Source ports: *    Dest address: WAN address    Dest ports: 443 (HTTPS)    NAT IP: 10.1.1.32    NAT ports: 4430

I don't have any other port forwardings or firewall rules which contain something with the second public IP (94.xx.xx.b).
BBB is working and can be accessed through its internal IP on port 443. When trying to access via the external domain, it loads like forever and dies.
So the question is, am I missing something, do I still need any port forwardings or rules to make 1:1 Nat work?

[rcoleman-netgate]

@leonidas-o Did you set up a VIP for the second IP address on your WAN?

[leonidas-o]

@rcoleman-netgate thanks for the quick reply. If VIP means virtual IP, then yes, done it like described in 1. Setup a virtual IP.
Or do you mean something else?

[rcoleman-netgate]

@leonidas-o Missed that.

Your destination for the VIPs should be the VIP not the WAN for NAT-port forwards.

For internal straight-through you will make the WAN firewall rule be the internal destination IP and port, so if VIP 1.2.3.4 goes to 192.168.1.23 for HTTPS the Destination would be 192.168.1.23.

[leonidas-o]

@rcoleman-netgate I'm not sure If I can follow you. I don't have any port forwardings for the second public IP which is setup as virtual IP (VIP) only for the main IP. I thought 1:1 NAT entry is enough, isn't it?
So I still need a port forwarding rule and I still need firewall rules allowing the traffic?

[leonidas-o]

I added a WAN firewall rule for the second public IP as source:

Proto: IPv4 *    Source: 94.xx.xx.b    Port: *    Dest: 10.1.1.57    Port: *    Gateway: *    Schedule: *

But still same behaviour.

[rcoleman-netgate]

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

So I still need a port forwarding rule and I still need firewall rules allowing the traffic?

Just doing the 1:1 means it will also communicate on that IP going out to the world... you need to still open ports to allow it through the firewall.

I might have overdone mine in 2014 when I switched to my ISP but here's my WAN firewall rule tab for my VIPs and the internal servers, the supporting VIP and 1:1 NAT pages.
login-to-view
login-to-view

login-to-view

[leonidas-o]

@rcoleman-netgate Okay I see, you got the visual IPs, so do I. Yours is USI and Proxy ARP, I'm simply using WAN and IP Alias.
You got the NAT 1:1 mappings, the external IP to internal IP and destination "*".
And finally you have some firewall rules. Okay, I thought if using 1:1 Nat it will automagically add the rules, so this is not the case and I have to add the rules, I see.

When I try to access the bbb service from a external connection/ not inside VPN, I see that the firewall is blocking for destination 10.10.10.2, which is the WAN address according to my network/interfaces conf:
Log:

Blocked -> WAN    171.x.x.x:48365    10.10.10.2:443	   tcp

Firewall rules:
So therefore my rule from above wasn't working, wrong source wrong destination:

Proto: IPv4 *    Source: 94.xx.xx.b    Port: *    Dest: 10.1.1.57    Port: *

When adding this, no blocking messages anymore, but can't still reach the bbb service:

Proto: IPv4 *    Source: *    Port: *    Dest: WAN address    Port: *

But this rule is actually not what I want, as it opens everything to WAN address, I would need something like this:

Proto: IPv4 *    Source: 94.xx.xx.b    Port: *    Dest: WAN address    Port: *

But this does not work of course, as the log clearly shows, that the source IP is the IP of a device and not the ip of my second public IP (VIP).

But now I'm confused, how can then my port forward for the main public IP from my initial post even work?

Interface: WAN    Proto: TCP/UDP    Source addr: 94.xx.xx.a    Source ports: *    Dest address: WAN address    Dest ports: 443 (HTTPS)    NAT IP: 10.1.1.32    NAT ports: 4430

It automatically creates a firewall rule, and the source address is set to the main public IP (94.xx.xx.a). I tested it and all my other services behind traefik/443 are still working. This doesn't make sense to me right now.

And also, do I need a port forwarding as well? If so, what is the 1:1 Nat entry actually doing in the end?

[Ycldoro]

This post is deleted!

[leonidas-o]

I have the feeling that I'm missing something in my proxmox network/interfaces conf file to make 1:1 Nat to work. The conf file has only knowledge about 94.x.x.A (The main public IP address) but not about 94.x.x.B (the second public IP address).

So this is the "Masquerading (NAT) with iptables" configuration according the proxmox docs (https://pve.proxmox.com/wiki/Network_Configuration).

[leonidas-o]

This post is deleted!

[leonidas-o]

@rcoleman-netgate I think I could figure out the proxmox part. I played around with it a little bit, especially the post routing part, and while I entered through my main public IP A via wireguard, my IP outside was public IP B. So this makes sense for me right now:

auto enp0s31f6
iface enp0s31f6 inet static
  address 94.PUBLIC.IP.A
  netmask 255.255.255.192
  gateway 94.x.x.x
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  post-up ip addr add 94.PUBLIC.IP.B/26 dev $IFACE label $IFACE:0
  post-down ip addr del 94.PUBLIC.IP.B/26 dev $IFACE label $IFACE:0
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.PUBLIC.IP.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.PUBLIC.IP.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.PUBLIC.IP.B -j DNAT --to 10.10.10.3

iface enp0s31f6 inet6 static
  address 2a01:4f8:10b:2ee8::2
  netmask 64
  gateway fe80::1


auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/29
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.A
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.A
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.3/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.B
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.3/32' -o enp0s31f6 -j SNAT --to-source 94.PUBLIC.IP.B

So now back at the 1:1 Nat issue/ Virtual IP. As you can see (vmbr10) I'm using 10.10.10.1/29 as the address and 10.10.10.2 for WAN. Now I thought, I will simply add 10.10.10.3 to the post routings and inside pfsense create a virtual IP (IP Alias) with 10.10.10.3 and a 1:1 Nat (WAN external: 10.10.10.3 to internal MY_VM_IP).
I don't know how virtual IPs (IP Alias) behave, right now it's not working but It has to be something in this direction, right?

P.S. 10.10.10.3 does not exists as a real interface of pfsense, just as virtual IP (IP Alias). Shouldn't I be able to route the packets to it like I do it with 10.10.10.2?

[viragomann]

@leonidas-o
Yes, an IP alias is just an additional IP on the interface. If assigned, pfSense answers ARP requests for this IP.

Run a packet capture on pfSense on WAN, enter 10.10.10.3 into the IP filter box, while you try to access the additional public IP from outside to see if the packets are forwarded correctly.

[leonidas-o]

@viragomann they are actually not routed correctly.

  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.x.x.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.x.x.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to 10.10.10.3

The prerouting here is not able to distinct between -d 94.x.x.A and -d 94.x.x.B and I don't know why. It always uses the rule with -d 94.x.x.A and routes to --to 10.10.10.2. Even if I manually type https://94.x.x.B into the browser, I end up at the traefik proxy which means the first rule of these three was used.

But for me the overview looks fine. Do you see anything, am I missing anything iptables specific?

iptables -t nat -L -n

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            94.x.x.A        multiport dports  !22,8006,179 to:10.10.10.2
DNAT       udp  --  0.0.0.0/0            94.x.x.A        multiport dports  !5405:5412,4789 to:10.10.10.2
DNAT       all  --  0.0.0.0/0            94.x.x.B         to:10.10.10.3

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
SNAT       all  --  10.10.10.2           0.0.0.0/0            to:94.x.x.A
SNAT       all  --  10.10.10.3           0.0.0.0/0            to:94.x.x.B

[viragomann]

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Even if I manually type https://94.x.x.A into the browser

You mean https://94.x.x.B, I guess.

Did you investigate this with packet capture on pfSense, or is this your assumption of what you see?

[leonidas-o]

@viragomann oh yes, I meant Ip address B. Corrected that typo.

No haven't checked that with packet capture as I ended Up at a "This connection is not private" site and when clicked on "view certificate" it shows "TRAEFIK DEFAULT CERT - self signed cert". Therefore it's pretty clear that it's traefik. If it would work either the 1:1 Nat works and the bbb service should be shown or simply timing out as firewall rules are maybe missing or whatever, but not traefik self signed cert.

I can still do a packet capture, maybe it reveals something helpful.

[viragomann]

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Therefore it's pretty clear that it's traefik

Agree, you see that traefik is responding. But you don't determine, why the packets go to traefik.
So is it on Proxmox or on pfSense??

[leonidas-o]

@viragomann I got the pfsense virtualised, so the entry point is proxmox (the Proxmox/Debian network/interfaces file controls to redirect "almost" everything to pfsense). Therefore I have to solve that iptables prerouting issue first. Right now, it doesn't make any sense to me, why it is ignoring the destination IPs.

[viragomann]

@leonidas-o said in 1:1Nat, two public IPs for one server with one nic:

Therefore I have to solve that iptables prerouting issue first.

Both of your public IPs go to the same pfSense interface, even each to a specific private IP, assumed it works properly.
But how did you determine, that Proxmox does the forwarding wrong without knowing which IP pfSense is seeing?

You can sniff the traffic either on pfSense WAN or on the Proxmox bridge, but without doing this you're standing in the dark at all.

[leonidas-o]

@viragomann I have a port forwarding for traefik:

Interface: WAN    Proto: TCP/UDP    Source addr: *    Source ports: *    Dest address: WAN address    Dest ports: 443 (HTTPS)    NAT IP: 10.1.1.32    NAT ports: 4430

Is uses "WAN address" (10.10.10.2) for destination address. I Don't have any port forwardings for the virtual IP (10.10.10.3), only the 1:1 Nat setting.

So, I would expect, that it forwards only the 10.10.10.2 traefik and would distinguish between the real Wan interface and the virtual IP. If it can't do that, how would you ever be able to achieve what I want?