Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    NAT issues with multi-WAN

    Scheduled Pinned Locked Moved Routing and Multi WAN
    28 Posts 4 Posters 10.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sai
      last edited by

      might be a netmask problem.. any chance that one of the netmask/gateway combination on the server in the OPT subnet is wrong? or overlaps/conflicts with the VIP defined on the firewall?

      1 Reply Last reply Reply Quote 0
      • B
        Briantist
        last edited by

        I'm not sure exactly what you mean, sai. The OPT1 interface is set to 12.34.56.24/24. The VIP is defined as 12.34.56.25/32 (Single Address; in this case I assumed that the netmask is used to determine the quantity of IP addresses for which you want ProxyARP). Really the VIP config should be moot here because the problem behaves exactly the same on OPT1's interface address.

        The server that is assigned a public IP directly which I am using for testing, has 12.34.56.241/24 (and .242/24 but that seems irrelevant here). Its gateway is 12.34.56.1 but again this machine is correctly sending its packets to pfSense; it's the return path where the packets are being sent to the gateway when they shouldn't be.

        Is there anything else I should check specifically?

        1 Reply Last reply Reply Quote 0
        • S
          sai
          last edited by

          it's the return path where the packets are being sent to the gateway when they shouldn't be.

          generally means the netmask is wrong but I am not sure how VIPs (and associated netmask) work in this…

          now this is weird:

          11:54:47.266968 00:50:8b:cf:a2:6a > 00:30:94:01:e7:90, length 62: (tos 0x0, ttl  63, id 38111, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3490: S, cksum 0xbcfc (correct), 3571569917:3571569917(0) ack 2188962038 win 5840 <mss 1460,nop,nop,sackok="">11:54:49.492704 00:50:8b:cf:a2:6a > 00:20:ed:91:f7:04, length 62: (tos 0x0, ttl  63, id 1468, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3479: S, cksum 0x6e76 (correct), 2180526778:2180526778(0) ack 2586252549 win 5840 <mss 1460,nop,nop,sackok="">2 consecutive packets, both from 12.34.56.24 both to 12.34.56.241 but (looking at the MAC address ) one goes to the gateway (ie the T1 modem) the other goes to the correct server.

          all I can say is "wtf?".  this problem really needs a guru</mss></mss>

          1 Reply Last reply Reply Quote 0
          • B
            Briantist
            last edited by

            :sigh: I always get the crazy problems no one has ever heard of.

            Keep in mind this doesn't really relate to the VIP/ProxyARP/1:1 NAT, since the problem happens on the interface address.

            I have looked at the routing table, and it seems fine to me. I don't pretend to understand all of the fields, but it looks okay to me:

            12.34.56 link#3 UC 0 0 1500 fxp2
            12.34.56.1 00:30:94:01:e7:90 UHLW 1 24400 1500 fxp2 1179
            12.34.56.3 00:20:ed:66:79:34 UHLW 1 124 1500 fxp2 1148
            12.34.56.241 00:20:ed:91:f7:04 UHLW 1 26009 1500 fxp2 1192

            (this is just the part of the routing table with 12.34.56 addresses)

            1 Reply Last reply Reply Quote 0
            • GruensFroeschliG
              GruensFroeschli
              last edited by

              I'll set a test-network up today evening and try to recreate your problem.
              This seems to be a really strange problem.

              We do what we must, because we can.

              Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

              1 Reply Last reply Reply Quote 0
              • B
                Briantist
                last edited by

                Thanks a lot; I really appreciate it. If you have any additional questions, you contact me directly at brian-NATissue@briantist.com. That goes right to my phone as well so I can probably reply quickly if I don't need to be at a computer to answer your question.

                1 Reply Last reply Reply Quote 0
                • B
                  Briantist
                  last edited by

                  @sai:

                  it's the return path where the packets are being sent to the gateway when they shouldn't be.

                  generally means the netmask is wrong but I am not sure how VIPs (and associated netmask) work in this…

                  now this is weird:

                  11:54:47.266968 00:50:8b:cf:a2:6a > 00:30:94:01:e7:90, length 62: (tos 0x0, ttl  63, id 38111, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3490: S, cksum 0xbcfc (correct), 3571569917:3571569917(0) ack 2188962038 win 5840 <mss 1460,nop,nop,sackok="">11:54:49.492704 00:50:8b:cf:a2:6a > 00:20:ed:91:f7:04, length 62: (tos 0x0, ttl  63, id 1468, flags [DF], proto: TCP (6), length: 48) 12.34.56.24.22 > 12.34.56.241.3479: S, cksum 0x6e76 (correct), 2180526778:2180526778(0) ack 2586252549 win 5840 <mss 1460,nop,nop,sackok="">2 consecutive packets, both from 12.34.56.24 both to 12.34.56.241 but (looking at the MAC address ) one goes to the gateway (ie the T1 modem) the other goes to the correct server.

                  all I can say is "wtf?".  this problem really needs a guru</mss></mss>

                  I didn't see your edit until now. That is weird; I didn't notice it before. I also checked the original packet capture data to make sure I didn't accidentally paste an incorrect MAC address, and I have confirmed that I did not (what you're seeing is correct). This makes it all the more weird though; I thought I had a consistent, repeatable issue here (and I do, kind of), but this one little packet is seriously making me wonder… I think it increases the likelihood that this problem is something I've done (whether it's in pfSense or not) to cause this problem. I can't think of what that might be though. Again, the help is really appreciated. Thanks everyone.

                  1 Reply Last reply Reply Quote 0
                  • GruensFroeschliG
                    GruensFroeschli
                    last edited by

                    Sorry for not writing back sooner.
                    I'm having some problems with faulty hardware (part of my network-test-enviroment just died) and i havent had the time to replace it.
                    I'll be in holiday for a week now.
                    I hope the replacement parts i ordered are here when i get home.

                    We do what we must, because we can.

                    Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                    1 Reply Last reply Reply Quote 0
                    • B
                      Briantist
                      last edited by

                      That sucks, sorry to hear about your hardware. Thanks for the update though, I've been checking the thread multiple times per day.

                      1 Reply Last reply Reply Quote 0
                      • B
                        Briantist
                        last edited by

                        GruensFroeschli, please tell me you haven't forgotten about me!  :'(

                        Anyone have any ideas? Any experiences with this?

                        1 Reply Last reply Reply Quote 0
                        • GruensFroeschliG
                          GruensFroeschli
                          last edited by

                          No i havent forgotten about you.
                          In fact i'm working on it right now ;)

                          But i think i've run into another problem with NAT.
                          Trying to reproduce it right now >_<

                          Will probably write back later today. (if i dont go crazy)

                          We do what we must, because we can.

                          Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                          1 Reply Last reply Reply Quote 0
                          • GruensFroeschliG
                            GruensFroeschli
                            last edited by

                            Ok i think i tried everything.
                            But i havent been able to reproduce your problem.

                            Here is what i did:

                            
		(WAN - public IP)
		ADSL-modem/router
		(LAN - 192.168.20.1/29)
		   |
		   |
192.168.20.4/29	   |
test-client-----switch--------------------------(WAN - 192.168.20.5/29)
		   |				pfSense2
		   |				(LAN - 192.168.40.1/24)
		   |				   |
		   |				   |
	(WAN - 192.168.20.6/29)			   |
	pfSense1 (OPT1 - 192.168.40.2/24)--------switch----------test-client/server
	(LAN - 10.0.0.1/24)					192.168.40.200/24
		   |
		   |
		server
		10.0.0.10/24

                            I can access the server from the 192.168.20.4 test-client as expected if i connect to 192.168.20.6.
                            I can access the server as well if i connect to 192.168.20.5

                            What you described is, that if the gateway on OPT1 is set you can no longer access the server from (in my case) the 192.168.40.x/24 range.
                            This worked for me.

                            I'm not sure what the problem in your case could be >_<

                            pfSense1(main)_OPT_config.JPG
                            pfSense1(main)_OPT_config.JPG_thumb
                            pfSense1(main)_port_forwards.JPG
                            pfSense1(main)_port_forwards.JPG_thumb
                            pfSense1(main)_FW_WAN.JPG
                            pfSense1(main)_FW_WAN.JPG_thumb
                            pfSense1(main)_FW_OPT.JPG
                            pfSense1(main)_FW_OPT.JPG_thumb
                            pfSense2_port_forwards.JPG
                            pfSense2_port_forwards.JPG_thumb
                            pfSense2_FW_WAN.JPG
                            pfSense2_FW_WAN.JPG_thumb

                            We do what we must, because we can.

                            Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                            1 Reply Last reply Reply Quote 0
                            • GruensFroeschliG
                              GruensFroeschli
                              last edited by

                              Schnittstelle: 192.168.40.200 –- 0x3
                              Physikalische Adresse . . . . . . : 00-03-25-09-91-19  <<–- test-client/server
                                Internetadresse      Physikal. Adresse    Typ
                                192.168.40.1          00-02-44-8f-03-ae    dynamisch    <<–- Gateway
                                192.168.40.2          00-0d-b9-05-67-25    dynamisch <<–- OPT-interface
                              @OK:

                              22:54:31.781241 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 56430, offset 0, flags [DF], proto: TCP (6), length: 48) 192.168.40.200.1596 > 192.168.40.2.80: S, cksum 0x30be (correct), 1713509472:1713509472(0) win 65535

                              @STRANGE:

                              22:54:31.782478 00:0d:b9:05:67:25 > 00:02:44:8f:03:ae, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 127, id 31917, offset 0, flags [DF], proto: TCP (6), length: 48) 192.168.40.2.80 > 192.168.40.200.1596: S, cksum 0x7865 (correct), 1117681065:1117681065(0) ack 1713509473 win 65535

                              @OK:

                              22:54:31.782905 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 56431, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.40.200.1596 > 192.168.40.2.80: ., cksum 0xa461 (correct), 1:1(0) ack 1 win 65535

                              @OK:

                              22:54:31.819768 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 591: (tos 0x0, ttl 128, id 56432, offset 0, flags [DF], proto: TCP (6), length: 577) 192.168.40.200.1596 > 192.168.40.2.80: P, cksum 0x6236 (correct), 1:538(537) ack 1 win 65535

                              @STRANGE:

                              22:54:31.822462 00:0d:b9:05:67:25 > 00:02:44:8f:03:ae, ethertype IPv4 (0x0800), length 299: (tos 0x0, ttl 127, id 4765, offset 0, flags [DF], proto: TCP (6), length: 285) 192.168.40.2.80 > 192.168.40.200.1596: P, cksum 0xc9bd (correct), 1:246(245) ack 538 win 64998

                              @OK:

                              22:54:31.945686 00:03:25:09:91:19 > 00:0d:b9:05:67:25, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 56435, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.40.200.1596 > 192.168.40.2.80: ., cksum 0xa248 (correct), 538:538(0) ack 246 win 65290

                              Obviously i can reproduce that traffic is being sent to the wrong MAC.

                              But i'm wondering why it's working here…..

                              We do what we must, because we can.

                              Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                              1 Reply Last reply Reply Quote 0
                              • GruensFroeschliG
                                GruensFroeschli
                                last edited by

                                Ok i went at it with wireshark again. (see attachment)
                                The downloaded content is the page on http://psymia.mine.nu
                                Everything seems to be in order…. (see frame 6)

                                But now i'm wondering why the capture from pfSense itself differs from the capture with wireshark.

                                ??? ??? ???

                                test_capture.pcap.txt

                                We do what we must, because we can.

                                Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                                1 Reply Last reply Reply Quote 0
                                • B
                                  Briantist
                                  last edited by

                                  GruensFroeschli, thanks so much for putting time into this. I'm sorry I haven't been able to do anything with it; I had to move suddenly, and things have been really crazy. I have had no time whatsoever to devote to this, and it might be a while before I can try again. I will resume working on this problem though, and I'll let you know how it turns out. Thanks again!

                                  1 Reply Last reply Reply Quote 0
                                  • E
                                    eri--
                                    last edited by

                                    Can any of you describe what the issue is, if any, so i can give a looka t it?

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      Briantist
                                      last edited by

                                      ermal, I'm confused. What information do you need that is not in the thread? I think we've been really descriptive.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.