NAT issues with multi-WAN

Briantist · Jun 15, 2008, 5:40 PM

it's the return path where the packets are being sent to the gateway when they shouldn't be.

generally means the netmask is wrong but I am not sure how VIPs (and associated netmask) work in this…

now this is weird:

11:54:47.266968 00:50:8b:cf:a2:6a > 00:30:94:01:e7:90, length 62: (tos 0x0, ttl 63, id 38111, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3490: S, cksum 0xbcfc (correct), 3571569917:3571569917(0) ack 2188962038 win 5840 <mss 1460,nop,nop,sackok="">11:54:49.492704 00:50:8b:cf:a2:6a > 00:20:ed:91:f7:04, length 62: (tos 0x0, ttl 63, id 1468, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3479: S, cksum 0x6e76 (correct), 2180526778:2180526778(0) ack 2586252549 win 5840 <mss 1460,nop,nop,sackok="">2 consecutive packets, both from 12.34.56.24 both to 12.34.56.241 but (looking at the MAC address ) one goes to the gateway (ie the T1 modem) the other goes to the correct server.

all I can say is "wtf?". this problem really needs a guru</mss></mss>

I didn't see your edit until now. That is weird; I didn't notice it before. I also checked the original packet capture data to make sure I didn't accidentally paste an incorrect MAC address, and I have confirmed that I did not (what you're seeing is correct). This makes it all the more weird though; I thought I had a consistent, repeatable issue here (and I do, kind of), but this one little packet is seriously making me wonder… I think it increases the likelihood that this problem is something I've done (whether it's in pfSense or not) to cause this problem. I can't think of what that might be though. Again, the help is really appreciated. Thanks everyone.

GruensFroeschli · Jun 20, 2008, 2:24 PM

Sorry for not writing back sooner.
I'm having some problems with faulty hardware (part of my network-test-enviroment just died) and i havent had the time to replace it.
I'll be in holiday for a week now.
I hope the replacement parts i ordered are here when i get home.

Briantist · Jun 20, 2008, 3:06 PM

That sucks, sorry to hear about your hardware. Thanks for the update though, I've been checking the thread multiple times per day.

Briantist · Jul 5, 2008, 6:17 PM

GruensFroeschli, please tell me you haven't forgotten about me! :'(

Anyone have any ideas? Any experiences with this?

GruensFroeschli · Jul 5, 2008, 6:24 PM

No i havent forgotten about you.
In fact i'm working on it right now ;)

But i think i've run into another problem with NAT.
Trying to reproduce it right now >_<

Will probably write back later today. (if i dont go crazy)

GruensFroeschli · Jul 5, 2008, 8:48 PM

Ok i think i tried everything.
But i havent been able to reproduce your problem.

Here is what i did:


		(WAN - public IP)
		ADSL-modem/router
		(LAN - 192.168.20.1/29)
		   |
		   |
192.168.20.4/29	   |
test-client-----switch--------------------------(WAN - 192.168.20.5/29)
		   |				pfSense2
		   |				(LAN - 192.168.40.1/24)
		   |				   |
		   |				   |
	(WAN - 192.168.20.6/29)			   |
	pfSense1 (OPT1 - 192.168.40.2/24)--------switch----------test-client/server
	(LAN - 10.0.0.1/24)					192.168.40.200/24
		   |
		   |
		server
		10.0.0.10/24

I can access the server from the 192.168.20.4 test-client as expected if i connect to 192.168.20.6.
I can access the server as well if i connect to 192.168.20.5

What you described is, that if the gateway on OPT1 is set you can no longer access the server from (in my case) the 192.168.40.x/24 range.
This worked for me.

I'm not sure what the problem in your case could be >_<

GruensFroeschli · Jul 5, 2008, 9:10 PM

Schnittstelle: 192.168.40.200 –- 0x3
Physikalische Adresse . . . . . . : 00-03-25-09-91-19 <<–- test-client/server
Internetadresse Physikal. Adresse Typ
192.168.40.1 00-02-44-8f-03-ae dynamisch <<–- Gateway
192.168.40.2 00-0d-b9-05-67-25 dynamisch <<–- OPT-interface
@OK:

22:54:31.781241 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 56430, offset 0, flags [DF], proto: TCP (6), length: 48) 192.168.40.200.1596 > 192.168.40.2.80: S, cksum 0x30be (correct), 1713509472:1713509472(0) win 65535

@STRANGE:

22:54:31.782478 00:0d:b9:05:67:25 > 00:02:44:8f:03:ae, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 127, id 31917, offset 0, flags [DF], proto: TCP (6), length: 48) 192.168.40.2.80 > 192.168.40.200.1596: S, cksum 0x7865 (correct), 1117681065:1117681065(0) ack 1713509473 win 65535

@OK:

22:54:31.782905 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 56431, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.40.200.1596 > 192.168.40.2.80: ., cksum 0xa461 (correct), 1:1(0) ack 1 win 65535

@OK:

22:54:31.819768 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 591: (tos 0x0, ttl 128, id 56432, offset 0, flags [DF], proto: TCP (6), length: 577) 192.168.40.200.1596 > 192.168.40.2.80: P, cksum 0x6236 (correct), 1:538(537) ack 1 win 65535

@STRANGE:

22:54:31.822462 00:0d:b9:05:67:25 > 00:02:44:8f:03:ae, ethertype IPv4 (0x0800), length 299: (tos 0x0, ttl 127, id 4765, offset 0, flags [DF], proto: TCP (6), length: 285) 192.168.40.2.80 > 192.168.40.200.1596: P, cksum 0xc9bd (correct), 1:246(245) ack 538 win 64998

@OK:

22:54:31.945686 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 56435, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.40.200.1596 > 192.168.40.2.80: ., cksum 0xa248 (correct), 538:538(0) ack 246 win 65290

Obviously i can reproduce that traffic is being sent to the wrong MAC.

But i'm wondering why it's working here…..

GruensFroeschli · Jul 5, 2008, 9:29 PM

Ok i went at it with wireshark again. (see attachment)
The downloaded content is the page on http://psymia.mine.nu
Everything seems to be in order…. (see frame 6)

But now i'm wondering why the capture from pfSense itself differs from the capture with wireshark.

??? ??? ???

test_capture.pcap.txt

Briantist · Jul 18, 2008, 3:35 PM

GruensFroeschli, thanks so much for putting time into this. I'm sorry I haven't been able to do anything with it; I had to move suddenly, and things have been really crazy. I have had no time whatsoever to devote to this, and it might be a while before I can try again. I will resume working on this problem though, and I'll let you know how it turns out. Thanks again!

eri-- · Jul 18, 2008, 4:08 PM

Can any of you describe what the issue is, if any, so i can give a looka t it?

Briantist · Jul 19, 2008, 5:44 PM

ermal, I'm confused. What information do you need that is not in the thread? I think we've been really descriptive.