Network issue with very small frames (tcp, padded)
-
So to be clear in a pcap on the WAN side you do not see the rogue 00s appended to the packet?
Have you tried disabling pf-scrub?
It obviously shouldn't do that but it's an easy test.Steve
-
@stephenw10
The 0 bytes are in the middle of a frame when looking at the WAN side.
When doing the test locally, there of course is no wan trace.
Just the file which is defect (we ran it in a script calling fetch in a loop and then checking for the correct checksum).
And yes, we switched off the scrub for test, the hardware options, anything we could think of.
It does not happen that often if we increase the read buffers, but this is potentially because the small packet arrives because of a tcp window full event or something similar.
The point here is not the small frame, but that the code seems to forward the ethernet padding to the application layer. It just happens more often if the read buffers are smaller.
We got another hint that the hardware could be a reason, since the machine is somewhat older, but anyway the padding should not get forwarded. -
@stephenw10
Another thing: the final file has the original size ! The bytes added in between some frames, are missing at the end. So the total len of tcp stream is miscalculated. This is weird. -
Hmm, maybe I've misunderstood how you're testing here then.
The file is being served from a server in the new VM environment which is behind pfSense?
And the client fetching that file and getting the wrong result is on a different internal interface? -
@stephenw10
We initially ran into the issue when some clients reported wrong files.
We then tested if we could reproduce the wrong download from our office network.
This was possible. From that point of view, the wrong bytes were in the middle of a random frame for unknown reason.We then checked with different internal servers (Linux mostly) with wget if the error could be reproduced (which was not the case).
Then we did package traces on the pfsense while performing the download via WAN and found the small frames < 60 bytes, padded, which occurred on the LAN side from time to time, leading to the problematic result.
This setup is Client -> Internet -> WAN -> haproxy -> LAN -> virtual env/Server (Apache)At the end, we tried to reproduce this on the firewall itself (in a shell) in order to eleminate
factors like haproxy and reduce the number of components in line.
So the latest test was just pfsense local shell on pfsense-> LAN -> virtualized server.Nothing else in between but switches (huge MTU) and the hosting environment.
The situation is occurring less often in a local environment, but was occurring irregularly (sometimes after 50 tries, sometimes after 400 tries).
The reason why the error did not happen to other machines, I can only guess that either these do not have an issue with the small padded frames, or never get them, because of better capabilities to handle fast responses in buffer (so no TCP window alerts, which seem to play a role here).
We have other (older) servers in place which should be converted into the virtual environment, but these have 1GBit NICs, which does not seem to trigger the issue. (this is an old setup running smoothly for a number of years). We have stopped the conversion for the moment until
this unexpected issue is solved.On the other machines in the lan segment, we only have tcpdump to trace the traffic and that one is not really comparable, contains very big frames (8k to 16k) because the place of tracing is different from a pfsense packet trace (as being said, MTU is 1500 everywhere, so the output is taken "after" the traversal of the segmentation possibly).
It could be that an old NIC (Intel I350 copper is detected, igb driver, but that can be a lot of types, not an uncommon card type) is a reason, I now ordered new hardware, but it is not logical to me that hardware should be the reason for such an issue.
We have a failover machine with identical hardware and were able to reproduce the issue there. So it is not a defect of the NIC.
We also have an "internal" firewall (pfsense, same version) present, but that one is running with a 10GBit NIC and we were not able to reproduce the situation here. Maybe because it never gets such small frames or because the driver correctly handles the padding (ix Driver here). -
Hmm, so where is the MTU change, packet re-assembly happening? If it is?
Does the other pfSense with the 10G NIC have larger frames enabled?
Are you able to test a different pfSense version a client?
Did you test a connection through pfSense but without HAProxy?
-
@stephenw10
Hi,
we now have installed new hardware and started to run in the first place the same software version (community edition, 2.6). There, we were able to reproduce the issue relatively fast.
So it is not a hardware (completely other NIC) issue.
We temporarily upgraded to dev version with bad results (firewall was unstable, so we did not proceed with the dev release).
Now, we upgraded to pfsense+ / FreeBsd 14, and found that the problematic situation cannot be reproduced any more.So this case seems to be solved.
-
Hmm, well that's good I guess even if I hate 'solutions' like that.
If you never know what the root cause was you can't fix it if it ever returns. -
@stephenw10
I agree, certainly would feel better if it would have a solution in form of a fix somewhere in the kernel/pf code, where some weird code fragment would treat the tcp length wrongly when padding occurs for whatever reason.
We very much like the product and have been working for many years with the community edition which really is a very stable platform.
Maybe I will check the code differences between the tcp stacks of 12.3 and 14 when I find some time, but I am not that good in low level C programming, and this is certainly an issue to be tracked by an expert who is more experienced in the code and in the low level ethernet/ip/tcp/pf subject.
Just to mention it, the error situation could not be reproduced by NAT only, so simple package forwarding which is running mostly in the IP layer leaves the stream in a correct flow to the client, at least we were not able to reproduce the behaviour.
But we need the load balancing, which has been relayd for many years before haproxy was replacing the functionality. haproxy was producing the error in both tcp and http mode. Executing a "fetch" to the url on the command line also did produce the error.
I got the impression that the issue is somewhere in the local tcp / socket code, because both haproxy and fetch are user mode applications, while NAT is probably done mostly inside the kernel IP code and probably does not touch the applications socket layer in the same way.
Of course there is the pf code somewhere in between, which is reorganizing the packets which also could be a reason(but we switched off scrubbing and the error was present after his action). But this is only an assumption. -
Yes, that seems more likely something in the base TCP stack since forwarding still passed pf but does not terminate TCP sessions.
A lot changed between 2.6 and 23.01 (or 2.7) because of the rebase to FreeBSD 14. I'll see if any of our developers are aware of anything that might have caused this.Steve
