Why is PFSense adding so much to my packet headers?

[youcangetholdofjules]

Hi All, I have 2 internet connections - one that goes via my PFSense box, one that bypasses it (for emergency use only, and PFSense is far from the only security feature, both then go into a Mikrotik switch and my internet router is an Edgerouter. Anyway - blah blah)

Problem I have is when I route everything through the PFSense box, my MTU size goes down to sub 1100, but if I bypass the PFSense bos it jumps to a more healthy 1466.

PFSense:

PFSense bypassed:

I have been horsing around with this a bit recently, but no idea what I did to create this mess - it was around 1400 through the PFSense a few days back, but every time I adjusted the MTU size down on my PC, the PFSense box adjusted that down even further and the fragmentation starts at sub 1100 bytes now - help! The reason I started playing around with this was that Teams was cutting out regularly and I was pulling my hair out - my original MTU size was fragmenting. (for reference my internet connection is 900Mbps - so its not like there is a bottleneck there....)

I know I am doing something stupid, usually the act of writing this out is the 4x2 that clocks me on the head and I think to myself "silly boy, there was an easy fix, aren't you glad you didn't make a goat of yourself in front of everyone" but alas I am that goat today. Any ideas?

Thanks

[youcangetholdofjules]

@youcangetholdofjules one other thing I am seeing when all traffic is routed through PFSense is this:

Direct out (bypassing the PFSense box) - never see this.

Any ideas?

Thanks

[bmeeks]

@youcangetholdofjules said in Why is PFSense adding so much to my packet headers?:

Hi All, I have 2 internet connections - one that goes via my PFSense box, one that bypasses it

This is a possible contributor to asymmetric routing (meaning outgoing and incoming packets sometimes taking different routes) if you have both routes online at the same time.

Is one unplugged and only connected when needed, or are both "hot" all the time? If the latter, I would suspect you might have asymmetric routing.

Here is a brief Cisco article on the issue: https://www.cisco.com/web/services/news/ts_newsletter/tech/chalktalk/archives/200903.html#:~:text=What%20is%20Asymmetric%20Routing%3F,in%20Layer%2D3%20routed%20networks.

Not saying you 100% for sure have this problem, but there is a possibility when you have multiple WAN connections, and care must be taken to ensure the routing is consistent.

[Gertjan]

@youcangetholdofjules

If you have a solid "900 mbit" you will see no spreading what so ever when doing this test :
https://www.waveform.com/tools/bufferbloat

Or, I'll bet you will see some random spreading and even spikes while up and downloading.