Drastically Slow internet Speed for VMs/CTs Behind pfSense on Proxmox
-
Hello pfSense community,
I'm experiencing a significant issue with internet speeds for my VMs and LXC containers running on a Proxmox host, which are routed through a pfSense VM. While the Proxmox host itself gets excellent speeds, clients behind pfSense have an extremely low internet speed this cause issue for our gitlab runner. I'm hoping someone can help me pinpoint the cause.
My Setup:
1. Proxmox Host Network Configuration:
My Proxmox host has the following bridge configurations:- vmbr0: Connected to my physical network, assigned 141.75.x.x/24 (public/external network).
- vmbr1: Internal network for pfSense WAN, assigned 10.0.0.0/24.
- vmbr2: Internal network for pfSense LAN and my VMs/CTs, assigned 10.1.0.0/24.
2. Proxmox Host iptables POSTROUTING Rules:
I have the following NAT rules on the Proxmox host:Chain POSTROUTING (policy ACCEPT 2981K packets, 191M bytes) pkts bytes target prot opt in out source destination 1255K 86M MASQUERADE all -- any vmbr0 10.1.0.0/24 anywhere 37M 2788M MASQUERADE all -- any vmbr0 10.0.0.0/24 anywhere3. pfSense VM Network Configuration:
-
LAN Interface: 10.1.0.252 (connected to vmbr2 on Proxmox)
-
WAN Interface: 10.0.0.252 (connected to vmbr1 on Proxmox)
4. pfSense Routing and NAT Configuration:
-
Gateways (System / Routing / Gateways):
- WANGW: Interface WAN, Gateway 10.0.0.1, Monitor IP 10.0.0.1 (Default IPv4)
- LANGW1: Interface LAN, Gateway 10.1.0.1, Monitor IP 10.1.0.1
- LAN Interface Configuration (Static IPv4 Configuration):
- IPv4 Address: 10.1.0.252 / 24
- IPv4 Upstream gateway: LANGW1 - 10.1.0.1
- Outbound NAT (Firewall / NAT / Outbound):
The Problem: Speed Test Results
-
Speed test from Proxmox Host:
- Download: 9351.72 Mbps
- Upload: 953.98 Mbps
(Excellent speeds, as expected)
-
Speed test from LXC/VM (connected to vmbr2, behind pfSense):
- Download: 3721.76 Mbps
- Upload: 0.16 Mbps
Given this configuration, what could be causing the slow speed connection
Thank you in advance for your help.
-
@zikou You don't put a gateway on your LAN, remove that and test again.
And you can't give pfSense its own interface, you have to NAT through the proxmox interface? That is cheap. -
@Bob-Dig those are the configuration that I found in pfsense (previous teammates) that configured pfsense, Im devops engineer so my knowledge in networking is average
so I should put upstream gateway for the LAN interface to None also do I need to configure NAT in this case -
@zikou 10G is pretty hardware dependent on pfSense but obviously the upload is wrong.
Normally an internal network doesn't have a gateway. pfSense will consider that a WAN interface. And then it needs to decide to which gateway/WAN to route packets that aren't for its networks. Were they trying to set up a static route?
-
Mmm, your auto outbound NAT rules show that the WAN interface is not configured with a gateway on it and the LAN is. That's the opposite of what I would expect to see there.
Are you expecting pfSense to NAT that traffic from LAN side hosts leaving the WAN? It doesn't have to be NAT'd there is the upstream router (Proxmox here) is NATing traffic from the LAN subnet. And it looks like you do have a masq rule there for it.
However I would bet you have some sort of asymmetric routing in that upload path given that very low rate.
Do you see unexpected blocked traffic in the firewall log? -
@stephenw10 so what should I do, upstream for LAN to none and add NAT outbound for WAN
-
@zikou Normally one does not need to change Outbound NAT routing unless you're trying to do something specific like have certain traffic use an alias IP.
It also looks like you're trying to map traffic on the WAN interface in your screenshot?
I suggest leaving Outbound NAT at Automatic.
A LAN network does not have a gateway because pfSense should be routing any traffic for other networks to its gateway, i.e. towards the Internet. So unless you have two Internet connections there should be only one gateway.
Can you answer about the checksum setting?
-
@SteveITS yeah the checksum is already configured
I set the upstream of LAN to none still low upload
speedtest Speedtest by Ookla [error] Error: [0] Cannot read from socket: Server: WEMACOM Telekommunikation GmbH - Frankfurt (id: 54577) ISP: OVHcloud Idle Latency: 1.56 ms (jitter: 0.12ms, low: 1.53ms, high: 1.76ms) Download: 2525.11 Mbps (data used: 1.6 GB) 3.73 ms (jitter: 0.82ms, low: 1.54ms, high: 7.64ms) Upload: FAILED cy: 1.55 ms [error] Cannot write: -
@zikou said in Drastically Slow internet Speed for VMs/CTs Behind pfSense on Proxmox:
I set the upstream of LAN to none
What about on the WAN interface config? Does that have the upstream gateway set?
If so you should then see the auto outbound NAT rules generated for the LAN subnet to WAN translation.
But, as I said, the rules you have in proxmox make it look like it's NATing the LAN subnet there and if so you don't need any outbound NAT in pfSense. However that would require a route in Proxmox.
So check in Proxmox does it have a route to the LAN subnet via the pfSense WAN? If it's routing via something else that may cause the asymmetry. -
@stephenw10 here the route table in proxmox host
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 141.55.x.x 0.0.0.0 UG 0 0 0 vmbr0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr1
10.1.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr2
141.75.x.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0WAN interface
here the output of traceroute in vm
traceroute google.com
traceroute to google.com (172.217.168.78), 30 hops max, 60 byte packets
1 * * 10.1.0.252 (10.1.0.252) 0.217 ms
2 10.1.0.1 (10.1.0.1) 0.076 ms 0.144 ms 0.136 ms
3 141.75.x.x (141.95.x.x) 0.540 ms 0.569 ms 0.603 ms
4 10.164.161.142 (10.164.161.142) 0.416 ms 10.164.161.144 (10.164.161.144) 0.327 ms 10.164.161.142 (10.164.161.142) 0.404 ms -
Ok, two problems there:
The gateway is not set on the WAN interface in pfSense. Which means no auto outbound NAT rules are generated for it.
The traffic route from the client is not going through pfSense at all. It goes to the pfSense LAN and is redirected to 10.1.0.1 which is some other router in the LAN subnet. It never goes to the pfSense WAN side.
And that is where your asymmetric route is because reply traffic just goes from that other router to the client directly which means pfSense never sees it and hence will be blocking traffic.
What is 10.1.0.1? I don't see it in the Proxmox routing table.
Unless there's some other subnet behind that you need access to you should remove that as a gateway in pfSense. Make sure the WAN gateway is set as default and is on the WAN dircetly.
However as I said before it looks like whoever set this up expected hosts to use the LAN directly because it has a MASQ rule for that subnet. Normally that would not exist is clients had to use pfSense to get external access.
-
@stephenw10 there is a dhcp server the gateway for vm that are in vmb2 is 10.1.0.252
10.1.0.1 is the IP address assigned to your vmbr2 bridge on the Proxmox host. It's not a separate physical router; it's the Proxmox host's interface that directly connects to the 10.1.0.0/24 network.
ip route show on proxmox host
ip route show
default via 141.75.x.x dev vmbr0 proto kernel onlink
10.0.0.0/24 dev vmbr1 proto kernel scope link src 10.0.0.1
10.1.0.0/24 dev vmbr2 proto kernel scope link src 10.1.0.1
141.75.x.x/24 dev vmbr0 proto kernel scope link src 141.75.x.x -
Ok if that subnet on vmbr1 should be isolated by the pfSense VM then I would not expect to see an IP address assigned to it in Proxmox.
The way you have it setup now clients in that subnet can just use Proxmox directly and bypass pfSense.
-
@stephenw10 I dont know what to do to solve the issue because there are some production vm if I touch something it will affect those vms
-
The first thing to do is make sure pfSense itself is using the WAN side gateway as it's default route. That will stop it sending redirects to the client to the LAN side gateway. As I said I would remove/disable the LAN side gateway entirely since it should never be used as far as I can see.
Of course that is assuming that the expected route of traffic from LAN side clients is through pfSense.
I would guess that those client VMs/containers existed before pfSense was added and used that LAN gateway. Then someone added pfSense to get more control over the traffic but never finished setting up the routing.
-
OP, if you follow this you cannot go wrong, plain and simple: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
