Multi WAN, TO VM NAS with seperate Firewall

jochenmehlich

Hi,

im using for virtulization Proxmox and my problem is, that i cant use the proxmox firewall - because it generates (why ever) random macs - which arent supported by my hoster. So i have to setup a pfsense proxmox network and every traffic has to go through the pfsense vm.
Now I have the problem (2 servers are in the new network), that everytime only one server is responding - the other can go out, but income traffic cant reach it.
How do I have to configure pfSense for it?

viragomann

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

Now I have the problem (2 servers are in the new network), that everytime only one server is responding

On requests from the internet?

Do you have multiple public IPs? If so, do get routed them to the primary IP? If no, have you assigned them to pfSense.

What are the four NICs between vmbr0 and pfSense supposed to mean?

jochenmehlich

@viragomann
yes every vm has its own public ip. The four NICs are because the hoster (hetzner) give me it with macs so that im getting it with dhcp, i dont know how else to do it.
Do you have an idea how to do it in proxmox?

viragomann

So instead of giving static IP addresses to you, the ISP gives you MACs and you can pull the IPs from DHCP? Strange.
Or is there still a possibility to assign the IPs manually?

So obviously this is meant to spoof the MACs in the Proxmox VM settings.

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

my problem is, that i cant use the proxmox firewall - because it generates (why ever) random macs - which arent supported by my hoster

What do you mean by that?
The Proxmox VM MAC? You can enter here what ever you want.

If you want to us pfSense for security reasons however, you can put it in the line in bridge mode.

But if it will work with the four WAN NICs depends on the IPs you get. If they are within different subnets and have different gateways it should work though. However, if you have only one gateway, pfSense will respond to all requests from the same NIC (MAC).

jochenmehlich

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

Or is there still a possibility to assign the IPs manually?

Yes, I could give them the static things. One problem could be that one of the four ips has a other gateway (unluckly the pfsense ip).

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

What do you mean by that?
The Proxmox VM MAC? You can enter here what ever you want.

Unfortunately not, they scan the MACs on the switch port, so I can only use the ones I got from the hoster. This just happened when I used the Proxmox firewall.

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

But if it will work with the four WAN NICs depends on the IPs you get. If they are within different subnets and have different gateways it should work though. However, if you have only one gateway, pfSense will respond to all requests from the same NIC (MAC).

Actually it is planned that the VMs are only reachable / send over their IP. Unfortunately I don't know how to use multiple IPs over one NIC

viragomann

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

Unfortunately I don't know how to use multiple IPs over one NIC

Firewall > Virtual IPs
Here you can assign further IPs to one interface. Use type IP alias.

However, if you get routed the additional IPs to you primary that isn't necessary. That's why I asked before.

If you want each VM to be accessible and also go out with a unique IP set up a NAT 1:1 for each.

jochenmehlich

I configured it now so:
Dashboard:

Nat 1:1:

and Virtual IPs:

put when i try to ping to one of the server, i dont get any response. What did I wrong?

jochenmehlich

One Problem could be, that the Ip of the PfSense VM has a completly other gateway as the from the others VM. Would it be fixed, if I setup a second NIC?

jochenmehlich

Falls du willst, können wir gerne auch auf Deutsch schreiben.

viragomann

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

Falls du willst, können wir gerne auch auf Deutsch schreiben.

Vom Interesse des Forum her sollte am Ende jedenfalls hier eine Lösung deponiert werden, sofern es eine gibt.

viragomann

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

One Problem could be, that the Ip of the PfSense VM has a completly other gateway as the from the others VM. Would it be fixed, if I setup a second NIC?

You've set the virtual IPs with a /32 mask. Consider that the gateway must be within the subnet of the IPs, cause if you access one pfSense sends responses back with that source IP.

So you got a second gateway from the provider for use with the additional IP as I understand? Have you already added this to pfSense?
However, if it can work with the gateway outside the subnet depends on the GW configuration.

jochenmehlich

The main Reason why I'm setting this up, is because i want to have for each vm a seperate firewall.

I hope it is not so critical to post now the public IPs, because it's very hard to check if every ip is censored. So i have following IPs.

Projectserver
IP: 144.76.93.247
Gateway: 144.76.93.225
Netmask: 255.255.255.224
Broadcast: 144.76.93.255

Publicserver
IP: 144.76.93.245
Gateway: 144.76.93.225
Netmask: 255.255.255.224
Broadcast: 144.76.93.255

Safeserver:
IP: 144.76.93.234
Gateway: 144.76.93.225
Netmask: 255.255.255.224
Broadcast: 144.76.93.255

PfSense:
IP: 136.243.196.130
Gateway: 136.243.196.129
Netmask: 255.255.255.248
Broadcast: 136.243.196.135

So I have

a WAN-Interface (136.243.196.130)
a LAN-Interface (172.16.0.1)
a OPT1-Interface (144.76.93.247)

All Traffic from the VMs and for the VMs should go through opt1.
It's static with the gateway with the CDIR 27.

Alias-IP

Nat 1:1

Firewalls:

Floating is empty.

What did I wrong? Because i can reach the projectserver but not the safeserver.

viragomann

@jochenmehlich
It's not really recommended to post the real public IPs here. The info that all except pfSense are in the same subnet with a separate gateway would be sufficient.

Why did you set a destination in the 1:1 NAT for the Safeserver? The own public IP makes no sense at all.
However, that won't be the problem, I think.

Your firewall rules page shows the interfaces "Safeserver" and "Publicserver". Are the VMs really connected to separate interfaces on pfSense? That cannot work, at least not with the server IPs shown in the NAT rules.

jochenmehlich

@viragomann

It's not really recommended to post the real public IPs here

Ok, thanks for the information

Why did you set a destination in the 1:1 NAT for the Safeserver?

I think it would fix my problem.

The own public IP makes no sense at all.

How could I fix it, that every vm has it own public ip?

Your firewall rules page shows the interfaces "Safeserver" and "Publicserver". Are the VMs really connected to separate interfaces on pfSense?

No, they are not connected, that are the old interfaces (from beginning, but not plugged in (link down via ProxMox)

That cannot work, at least not with the server IPs shown in the NAT rules.

How can I do it?

viragomann

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

Why did you set a destination in the 1:1 NAT for the Safeserver?

I think it would fix my problem.

The destination should be any if there are no special reasons for to set specific one.

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

How could I fix it, that every vm has it own public ip?

The rest of the 1:1 NAT rule is fine. One external IP to one internal.

@jochenmehlich said in Multi WAN, TO VM NAS with seperate Firewall:

No, they are not connected, that are the old interfaces (from beginning, but not plugged in (link down via ProxMox)

That cannot work, at least not with the server IPs shown in the NAT rules.

How can I do it?

Ok, if the server are both on LAN within the same subnet there is no problem.

Test the access again with the corrected NAT rule.

jochenmehlich

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

The destination should be any if there are no special reasons for to set specific one.

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

The rest of the 1:1 NAT rule is fine. One external IP to one internal.

Okay :D

@viragomann said in Multi WAN, TO VM NAS with seperate Firewall:

Ok, if the server are both on LAN within the same subnet there is no problem.

They should be.

jochenmehlich

its not working. the vms can go out.
only project is reachable.

here is the virtual-ip configuration

and the interface

Did i something wrong? Or should I ask my hoster, if any feature is missing?

viragomann

Are you sure, the safeserver is responding to the access from the internet?

To investigate use the packet capture tool from the Diagnostic menu on pfSense.
Take a capture on LAN interface while trying to access the Safeserver from outside.
If you see nothing take a capture on OPT1 to check if the packets arrives pfSense.

jochenmehlich

In OPT1 its showing the ping request

00:07:12.007358 IP 5.100.XXX.XXX > 144.76.93.234: ICMP echo request, id 6, seq 4, length

but there is nothing in lan

jochenmehlich

so i thing, something with the redirection is broke. (maybe the nat 1:1)?