The PVLANs are tagged, but as you said, the issue is multiple devices and trunking in play here. Using the HA pair, two switches and the 10Gbps trunk means I either need pfSense to understand PVLANs or the Cisco switch to do the work for it. I am opening a ticket with Cisco to see if this model can do what is needed. Thanks for the response.

Yeah I have no idea what he is talking about either.. If your not going to use vlan capable switches, then yeah we talked about just different interfaces and native networks connected to their own switches, etc.

If your going to force traffic out a gateway on the rules - then NO your not going to other local networks.

Again DRAWING!!!! if you want anyone to understand what your wanting to do.

@tman222 said in Jumbo Frames not working on L3in 10Gbit network:

Hi @LaUs3r - have you tried running an iperf3 test across the firewall (i.e. between two VLAN's or subnets) to see how many packets per second it handle with PF enabled? That might be a good first step to see where the theoretical transfer limits are (and would leave out any impact storage might have on slowing down the transfer speed). Check out this link:

https://bsdrp.net/documentation/technical_docs/performance#where_is_the_bottleneck

You can use netstat to monitor number of packets being transferred while running an iperf3 test across the firewall (i.e. between two hosts in different VLAN's or subnets). Then reduce the MSS and see where you hit a bottleneck (i.e. the number of packets no longer increase as you increase the number of parallel iperf3 streams)

Hope this helps.

So, today I performed some iperf tests. my i3 cannot take more than 8Gbit/s which is fine for me. I now have transfer rates of approx. 800MB/s :-)
Interestingly, on my other PC where WinPcap is installed, I only get around 550MB/s.

So, thank you all for your valuable inputs and help!
Cheers guys

No idea why you would have two pfSense nodes for two WANs.

That would require the hosts know how to route to either node based on where they are trying to do.

A much more common way to handle it is to connect both WANs to different interfaces on the outside node, create a gateway group, and policy route to it for the inside host traffic.

https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html

I perused this thread and cannot get a clear picture of what you are trying to do based on your descriptions.

A proper network diagram would probably be in order since the design sounds rather unconventional.

Yes, that's what I get with the /25 setting. Thanks again.

Ok I get by myself, I just forget this :

Annotation 2019-04-04 135646.png

Et voila !

If i understood correctly is to create a host overide

but host overide to put host mail and ip to return the WAN ip?

Note quite. You want to configure a host override for mail.mydomain.com and point it at the internal IP of your mail server (i.e. 192.168.3.150). Right now, your clients are trying to connect to mail.mydomain.com, which is resolving to a public IP, then being routed accordingly out your WAN and relying on NAT reflection to redirect the traffic back thru the firewall. Once the host override is configured, when your clients initiate a connection to mail.mydomain.com, the DNS query will get resolved locally to 192.168.3.150 and then traffic will get sent to the mail server directly vs. being routed out the WAN interface.

At this point, two things will now happen more efficiently:

DNS queries for mail.mydomain.com will be resolved locally by PFsense instead of being forwarded to a server on the internet for resolution.

When clients initiate connections to your mail server, the traffic will be sent directly to the mail server instead of relying on a "hack" that loops traffic through the firewall after it hits the WAN interface.

then create an explict firewall rule? to allow all guest VLAN to access 192.168.3.150?

Almost, but even more explicit. You don't want to allow all guest VLAN traffic access to 192.168.3.150... you only want to allow traffic sourced from the guest VLAN and destined to 192.168.3.150 using email ports.

Finally got this solved! Previous admin never documented anything so I was just stumbling around. There's a layer 3 switch doing VLAN routing (originally it was just passing the routing off to the old router). So I reconfigured the layer 3 switch to handle the VLANs, pfSense just handles everything out to the internet. So we're all good!

First off, thank you for you help. Been going nuts trying to figure all this out, but i think I finally got it working (all systems are getting IPs in the correct subnet for their VLAN from pfSense). Now to see if I have a basic understanding:

Untagged: So by listing eth2 as untagged in VLAN 10

Untaggedd traffic going IN eth2 is added to VLAN 10 (this is because PVID on eth2 is set to 10)

Traffic going OUT eth2 has tags stripped because eth2 is untagged (we are saying that the device connected to eth2 does not expect tags)

Tagged: This is the one that confuses me. Basically when traffic enters on eth1, the switch checks for what VLANs eth1 is a "tagged" member of. If the traffic has a VLAN ID that matches a VLAN eth1 is tagged in, the traffic is allowed to pass. If the traffic has no tag or has a tag with a VLAN ID that doesn't match a VLAN eth1 is a tagged member of, then the traffic is dropped. Any traffic exiting eth1 keeps its pre-existing tag or is tagged with the PVID if no tag exists.

So my device can send untagged traffic to eth2, have it tagged as VLAN 10, and then sent out eth1 to pfSense. In order to get any traffic back from pfSense tagged as VLAN 10, I must have eth1 as a "tagged" member of VLAN 10.

This all sound correct?

Verify there is an outbound NAT in place (and configured correctly) for traffic sourced from VLAN 60 on the VPN interface

This was the problem. Thank you so much. I feel a little stupid right about now ;-)

https://www.youtube.com/watch?v=NgRy14rYhV8
https://forum.netgate.com/topic/138167/sg-3100-lagg

I got this figured out finally. I'm working on this from about 700 miles away so I was lacking some crucial documentation which I got when I asked for some pictures of the network rack. There's a managed Cisco SG300 switch that connects the pfSense firewalls which has never been configured. I thought I was going crazy when sh cdp neighbor was only showing a single MAC and not two firewalls. I had assumed both firewalls were plugged directly into the Catalyst 2960 which had a left over port description that wasn't updated.

Thanks all - we are good!

https://www.netgate.com/resources/videos/configuring-netgate-appliance-integrated-switches-on-pfsense-244.html
Enjoy ☺

-Rico

UPnP has no idea whether it is on a VLAN or not.

@johnpoz
My drawing above is the existing config. The Netgear switch is completely unused. I would be fine using that if need be for the VLAN or port 5 or 6, although from Example 1 here, which I was going to loosely follow for the VLAN tagging, it appears to me, I should be able to remove a port from VLAN 1.

https://www.tp-link.com/us/faq-788.html

Tim

Ok I finally was able to get the management interface to work, but in a backwards way. I added an ipv4 interface with VLAN 10 and rather than setting the IP I let it get it from DHCP. It did, and it worked fine! I then put a static IP mapping in pfSense for the MAC and forced a refresh of the management interface and now it has the IP I wanted all along. I then changed the IP over to "static" on the switch.

My question is why did it work via DHCP, but not via just setting it up as a static mapping? The only thing I can think of is that when I add the interface manually with a static IP it is doing something weird with the default route or not updating the default route and it's trying to access the management VLAN over a different interface. Anyone seen this before?

@johnpoz said in Browsing nfs share in other VLAN not working.:

@herman said in Browsing nfs share in other VLAN not working.:

When the family is watching a movie and I start to copy large files the movie stutters or even stops. And the kids do not appreciate that :-)

If your loading up the server, might not matter if your on another interface or not if your sucking up the I/O to a shared disk or CPU of the machine. If your on a switch you doing something between A and B doesn't effect traffic between C and D..

Adding another interface on B, will only solve the problem of stuttering a streaming movie if the only problem was saturation of the interface.. But if your working with the same disk that the movie is streaming from... Your issue might not be network bandwidth it could be your hitting the I/O of the disk limits, or the cpu of the server streaming, etc..

Thank you @johnpoz for the info... I wil dig in it.