NAT not always working
-
i've got a strange problem with my amd64 pfsense 2.0.
i've got several external ip's (using ip aliases) and i'm forwarding the required ports to my servers. it all works quite well.however i've got a problem with "fileuploads". if i try to upload a file larger than ~55kb, the upload fails.
in my example i try to upload a file onto a webserver. i found out about the problem because i had troubles with attachments in emails. same situation there. if the attachment gets to big, i cannot send / receive it.i've already tried to modify the firewall optimization (changed to conservative) and i also tried to disable scrubbing.
the thing that freaks me out, is that according to the logs everything is fine (no packets dropped or blocked).
any ideas on my problem / anyone else with the same problem?
greetings
-
First guess, you need to lower your MSS on WAN. Try 1400.
-
I tried it, but it didn't work. I also tried lowering the MTU - but also with no success.
I did 2 tcpdumps. The first with a "small" file (i think it was 52KB big) that was working and a second one with a "big" file (56KB or something) that failed.
I've uploaded the dumps. Maybe someone could have a look. I would really appreciate it.
http://dl.dropbox.com/u/5448701/pfsense_working_small_file
http://dl.dropbox.com/u/5448701/pfsense_notworking_big_fileI did some more dumps.
Tried to upload a file from an external client to a http server behind fw.WORKING 52KB upload
–----------------------------
52KB Fileupload captured on LAN interface (tcpdump -nN -i re0_vlan3 net INTERNAL_SERVER)
http://dl.dropbox.com/u/5448701/52KB_LAN_INTERFACE
52KB Fileupload captured on WAN interface (tcpdump -nN -i em0 net EXTERNAL_CLIENT and not port 1194)
http://dl.dropbox.com/u/5448701/52KB_WAN_INTERFACENOT WORKING 66KB upload
66KB Fileupload captured on LAN interface (tcpdump -nN -i re0_vlan3 net INTERNAL_SERVER)
http://dl.dropbox.com/u/5448701/66KB_LAN_INTERFACE
66KB Fileupload captured on WAN interface (tcpdump -nN -i em0 net EXTERNAL_CLIENT and not port 1194)
http://dl.dropbox.com/u/5448701/66KB_WAN_INTERFACEbtw. i'm using following nics:
WAN: em0: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfea00000-0xfea7ffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci1
LAN (re0): rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus0</rtl8169s></intel(r)> -
The text output isn't really adequate to analyze that particular scenario, if you can save that to a pcap (-w <filename>) and upload it, that will be much more telling.</filename>
-
Ok… I did so. Sorry it took quite a while cause I was not at home.
Here are the 2 files.
working.cap (53.93KB JPG) http://dl.dropbox.com/u/5448701/working.cap
not_working.cap (67.9KB JPG) http://dl.dropbox.com/u/5448701/not_working.cap -
It's exactly what I alluded to in my first reply, large packets aren't getting through somewhere. Guessing that capture was on WAN since it sounds like you're NATing, what does it look like from LAN at the same time?
-
Now it's getting weird… Almost all captured packets on the lan-if have invalid header checksums. Can sombody explain that to me please. Shouldn't the nat rewrite the checksums to match the translated ip??? (RFC1631 or something)
Working file (53.93KB)
WAN http://dl.dropbox.com/u/5448701/working_wan.cap
LAN http://dl.dropbox.com/u/5448701/working_lan.capNot working file (67.9KB)
WAN http://dl.dropbox.com/u/5448701/not_working_wan.cap
LAN http://dl.dropbox.com/u/5448701/not_working_lan.capAnother strange thing I noticed is that I can receive e-Mails (from external) with big attachments (just tried a 5MB mail). But I still cannot send an large email (from outside via my internal mailserver)
-
did you check limited attach size on your mail server?
-
The Mailserver is working fine. (It works from inside the lan and with the old fw)
But thanks :)I did some other tests with my old freebsd fw (ipfw + natd). The old firewall is rewriting the header checksum (after rewriting the Dst), while the new Pfsense seems to forget about it. I really hope someone of you guys could explain that to me / help me. Another strange thing is that the MSS seems to change (from 1460 on WAN to 1360 on LAN) on the Pfsense.
I really appreciate all of you help!
Tell me what infos you need i'd be happy to provide them. -
You truncated the capture to 96Bytes, Im like 99% sure you can not validate checksums when you truncate.
Also if your offloading to your nic, checks sums can be wacky – I normally turn off checksum validation in wireshark because of this.
I would redo you captures without truncating.
-
You won't have checksums at all when capturing on any host with hardware checksum offloading enabled. That's added by the NIC right before it hits the wire. It's normal (Wireshark even gives you a note as such). You do have the correct checksum there, 0x0000 is what it really is at that stage. With a 96 byte snap length you will get the full IP and TCP headers (plus some), which will have the full checksums.
So the second capture shows the firewall is sending all those packets out of LAN that disappear somewhere. Short of the checksum that gets added by the hardware, what you're capturing on LAN is what is on the wire. It's passing it to the internal host.
Those captures show it's more than just large packets not getting through, SYNs are even getting retransmitted multiple times. Unless you had some odd filter on that capture, you have some broken routing somewhere. Note there is no traffic going from 192.168.200.16 back to any outside IP on that LAN capture. Which probably means that host's default gateway is set to something else, which isn't going to work correctly.
-
OMFG!
THANK YOU CMB!!!!!!
What a $!%%#**! I simply screwed up my routing. Now it is working like charm!
Thank you a million times. I was just to stupid :)You guys are awesome! Thank you all for your help!
-
Hi all.. What do you mean with ï screewed my routing"? Thank You
-
Hi all.. What do you mean with ï screewed my routing"? Thank You
There are lots of ways to screw routing. If yours is screwed, it's likely screwed in different ways than this one. :) Start a new thread to describe your issue.
In this case, it came down to:
@cmb:Note there is no traffic going from 192.168.200.16 back to any outside IP on that LAN capture. Which probably means that host's default gateway is set to something else, which isn't going to work correctly.