Inter VLAN routing Iperf results 0.00 bits per second when target has multi LAN

Delegator5042

Disclaimer and purpose

I have replicated the same behavior on the other *sense. For pf I used the latest available community edition.
Firewall settings were kept on default and an Allow all rule was put in place on all interfaces (to rule out bad firewall rules).

What I am after is an explanation and whether a setup like this is non-standard.
I honestly feel pretty silly for not being able to find the explanation, the closest answer I found was on the OPNsense forum [url]https://forum.opnsense.org/index.php?topic=35157.0[/url] but the eventual conclusion was that the person just gave up on VLANs.

Setup

I made a simple topology for myself that I hope is clear enough
The server has public facing services on VLAN10.
Management of the server and the services is done on VLAN100
alt text

The problem: Zero upload speed, but normal download speed

I noticed secure copy (scp) could not upload files to VLAN10 despite being able to connect and use ssh.
Iperf3 then gave me a very weird result: 0.00 bits/s

iperf3 -c 192.168.10.10
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.1.10 port 58322 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   128 KBytes   105 Kbits/sec    5             sender
[  5]   0.00-10.04  sec  65.0 KBytes  53.1 Kbits/sec                  receiver

iperf Done.

There are three ways I get normal speeds

1: Removing the VLAN100 interface from the dual LAN

This suggests to me that the upload is being (partially) incorrectly routed.
I also noticed that if I changed VLAN100 to a different VLAN (20) on the single-LAN or mult-LAN side (but not both), the speed also returns to normal. Meaning that as long as VLAN100 is present on both devices, the single VLAN device cannot upload to the server's VLAN10.

So multi LAN is not the real issue, the real issue is multi LAN where I try to connect from a device whose VLAN is also setup on the target machine but is not the target for the file transfer.
What I just don't understand is why.

2: Reverse the iperf3 test (iperf3 -s)

Seems logical given the previous point. The target is the device with only one (V)LAN.

3: Turning off the firewall

Of course very silly, but it at least tells me that there is some traffic rejection going on, though I don't see anything in my Firewall Live View.

Conclusion

As an addition to the disclaimer, I think I just fundamentally misunderstand something that makes the setup act this way. I figured a setup like this is pretty common for at least homelabs: server with a network for its services, and a network where its managment gui/ssh is listening on.
In any case I can put the laptop on VLAN10 and only go to VLAN100 when I need to configure stuff, but in my (admittedly limited) experience with VLANs I have not yet encountered an issue like this.

johnpoz

@Delegator5042 said in Inter VLAN routing Iperf results 0.00 bits per second when target has multi LAN:

setup like this is non-standard.

Yeah you have an asymmetrical setup there.. So if your laptop talks to vlan 10 its routed for going there but the answer will come from the vlan 100 interface on your server.. Because the server has a leg in the vlan 100 network, why would it need to send traffic to pfsense to get back to vlan 100, its on vlan 100.

Asymmetrical traffic flow is pretty much never going to be optimal sort of setup..

And you throw in a stateful firewall and now you can run into even more issues..

Delegator5042

@johnpoz
Thank you for taking a look!
Your explanation makes sense to me and the image helps a lot. It also explains why I didn't see firewall (deny) logs.

A question I have floating around is if I could still make it work somehow by returning the laptop to vlan10 but then also have it capable of swapping to vlan100 when I want to do management. But there's probably enough explanation for that online already.

Once more, thank you very much.