Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Dual (2)WAN / Multi (9)LAN Routing Issue with Public IP's

    Routing and Multi WAN
    2
    38
    3901
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      Disturbed1 last edited by

      having trouble with routing…

      have attached a diagram...

      i use to run two ubnt edge lite routers, i wanted to remove them and merge both upstreams into one box...

      so now everything is connected as in attached diagram...

      2 problems…

      problem 1, can't get 204.101.*.208/29 working unless i check routing only platform (once checked i loose nat'ed clients) (SOLVED POST#11)

      problem 2, blue subnets can't communicate to other blue subnets… (SOLVED POST#32)

      firewall/rules set to allow all..

      VIP/Proxy ARP set for both upstreams…

      system/routing default gateway isp1

      pink subnets firewall rules set GW default...

      blue subnets firewall rules set GW as isp2...

      can pfSense do this or will i need to remove nat'ed clients subnets to another pfsense box to allow route only platform?

      when connected to blue subnet and check at google whats my ip, the right ip appears but can't communicate to other blue subnets

      when connected to pink subnet and check at google whats my ip, only the 67.69..254 shows even when connected in 204.101..208/29 pool

      can post more info/screenshots as requested if need be….

      not sure what i'm missing here but after many searches on forum and google, i've broke down and posted here :(

      when i was using the two ubnt egdemax lites eveything was communicating fine between the two different upstreams... so i can only guess i missed something somewheres in pfsense.....

      EDIT: new issue noticed, goto post  Reply #33 https://forum.pfsense.org/index.php?topic=83413.30

      1 Reply Last reply Reply Quote 0
      • Derelict
        Derelict LAYER 8 Netgate last edited by

        can pfSense do this or will i need to remove nat'ed clients subnets to another pfsense box to allow route only platform?

        It will only NAT if there are NAT rules.  You will absolutely have to enable manual outbound NAT and only have rules for the networks/interfaces you want to NAT for.

        Any possibility you can start smaller during some dead time and work one WAN/one LAN at a time until get get more familiar with what pfSense needs?

        1 Reply Last reply Reply Quote 0
        • D
          Disturbed1 last edited by

          i will give a try at 4/5am to add nat rules & switch to manual nat… (production enviro)

          before this setup i had...

          isp2 i've had on pfsense hardware before, everything was great, then tried the edgemax...  worked but missed features pfsense had...

          isp1 i had two pfsense vm's, first was route only platform, second used for nat'n clients behind single ip from /29 pool...

          surffered hvac failure in room, edgemax baked, vm mechine toasted raid card... etc etc etc...

          so as a last minute get everything back online i decided to go multi wan single hardware... i figured i missed something, thx for pointing out the manual outbound....

          will post update in morning....

          1 Reply Last reply Reply Quote 0
          • Derelict
            Derelict LAYER 8 Netgate last edited by

            Ok.  You will do it in the opposite order though - enable manual outbound NAT then tweak the rules.

            I am pretty sure you can enable manual outbound any time.  All of the rules that are placed there by the automatic process will be there so there should be no  change in behavior until you start tweaking rules.  Also keep in mind that you can just disable the ones you don't want instead of deleting them.

            You should see two rules for each LAN interface.  One with a static port for IPsec and one for everything else.  I'd just duplicate those for interfaces you want to NAT for.

            1 Reply Last reply Reply Quote 0
            • D
              Disturbed1 last edited by

              good morning…

              so it's 5am here... (5:30am by time i post this)

              i set manual out bound, set nat rules...

              connected to pink ip pool, googled whats my ip, still came up with pink gw ip.... when connected to blue ip pools (tested three subnets) proper ip showed...

              :(

              so since i was here, i checked route only platform... and two things happened... google showed the ip from pink ip pool, and blue subnets could talk to each other... but nat'ed clients no access to net....

              so i'm a little baffled here....

              edit: also rebooted just incase... same resault after....

              1 Reply Last reply Reply Quote 0
              • Derelict
                Derelict LAYER 8 Netgate last edited by

                Screen shots of Firewall->NAT->Outbound and Firewall->Rules (Interfaces) please.  You have something not right.  Routing Only Platform is not what you want.

                1 Reply Last reply Reply Quote 0
                • D
                  Disturbed1 last edited by

                  ka… snapshoots attached..... firewall rules same for both nat'ed subnets...




                  1 Reply Last reply Reply Quote 0
                  • Derelict
                    Derelict LAYER 8 Netgate last edited by

                    Why are you natting your public IPs?

                    1 Reply Last reply Reply Quote 0
                    • D
                      Disturbed1 last edited by

                      no reason, forgot to remove it…. trail and error trying different things.... cleared it out now... just 172.16.5.0/172.16.10.0 in there now...

                      1 Reply Last reply Reply Quote 0
                      • Derelict
                        Derelict LAYER 8 Netgate last edited by

                        OK.  And what, specifically, isn't working now?  Let's work one LAN/WAN interface at a time.

                        1 Reply Last reply Reply Quote 0
                        • D
                          Disturbed1 last edited by

                          first problem…

                          isp1,

                          204.101.*.208/29 subnet...

                          when connected to ip in that pool, i goto google and search whats my ip.... should show 204.101..209, but shows isp1 pfsense gw ip 67.69..254....

                          lan_bell = 204.101.*.208/29 subnet






                          1 Reply Last reply Reply Quote 0
                          • Derelict
                            Derelict LAYER 8 Netgate last edited by

                            With those NAT rules it would do that.  Without them it should not.  Did you clear states after deleting the NAT rules?  You can clear only the states in question by filtering on 204.101.*.209.

                            I wouldn't have the Proxy ARP VIPs.  I'd have type Other - if you need any at all.  Out of curiosity, what is the IP address of the LAN_BELL interface?

                            1 Reply Last reply Reply Quote 0
                            • D
                              Disturbed1 last edited by

                              @Derelict:

                              With those NAT rules it would do that.  Without them it should not.  Did you clear states after deleting the NAT rules?  You can clear only the states in question by filtering on 204.101.*.209.

                              I wouldn't have the Proxy ARP VIPs.  I'd have type Other - if you need any at all.  Out of curiosity, what is the IP address of the LAN_BELL interface?

                              my bad for typo… 204.101..209 should have been 204.101..208/29 meaning a ip from that pool....

                              ka i changed VIP to other... checked and is now working.....  THANK YOU….  i swear i tried that once but must have over looked....

                              i will try changing the other isp2 vip setting in a minute and see if that changes the block between isp2's smaller subnets....

                              1 Reply Last reply Reply Quote 0
                              • D
                                Disturbed1 last edited by

                                okay, so both VIPs are set to Other, and seem to be okay…. connected to different pools and whats my ip was correct in all tests from all pools....

                                while i was connected to pool 216.185..192/26 i tried to access email server in 216.185..160/27 pool with no success.... while connected to a pool fed from isp1, i could access with no prob... when tried from a pool feed by isp2 only time i could access email was while i was inside same pool as server....

                                here attached are the screenshots of firewall rules for both...




                                1 Reply Last reply Reply Quote 0
                                • Derelict
                                  Derelict LAYER 8 Netgate last edited by

                                  Check all your netmasks and gateways.  What happens when it fails?  Anything in your firewall logs?

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    Disturbed1 last edited by

                                    all netmasks correct… doubled checked with online subnet calculator...

                                    here ping resaults from 216.185..201/26 pool pinging 216.185.166/27

                                    C:\Users\chrism>ping 216.185.*.166
                                    
                                    Pinging 216.185.*.166 with 32 bytes of data:
                                    Reply from 216.185.*.1: TTL expired in transit.
                                    Reply from 216.185.*.1: TTL expired in transit.
                                    Reply from 216.185.*.1: TTL expired in transit.
                                    Reply from 216.185.*.1: TTL expired in transit.
                                    
                                    Ping statistics for 216.185.*.166:
                                        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
                                    

                                    packet capture:

                                    216.185.*.1 > 216.185.*.201: ICMP time exceeded in-transit, length 36
                                    	(tos 0x0, ttl 1, id 6454, offset 0, flags [none], proto UDP (17), length 56)
                                    

                                    checked system logs / firewall, tried both source and Destination with ip 216.185.*.201, clear, nothing there…..

                                    1 Reply Last reply Reply Quote 0
                                    • Derelict
                                      Derelict LAYER 8 Netgate last edited by

                                      What routes have you put in System->Routing ??  You probably want to get rid of everything.

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        Disturbed1 last edited by

                                        zero….

                                        screen shot of gateways in post #10


                                        1 Reply Last reply Reply Quote 0
                                        • Derelict
                                          Derelict LAYER 8 Netgate last edited by

                                          When you traceroute it what IPs is it bouncing between?

                                          1 Reply Last reply Reply Quote 0
                                          • D
                                            Disturbed1 last edited by

                                            now this is where it gets funny….

                                            216.185.*.201 is from Canada(CA) in region North America
                                            
                                            TraceRoute from Network-Tools.com to 216.185.*.201 [*************]
                                            Hop	(ms)	(ms)	(ms)		     IP Address	Host name
                                            1 	  Timed out 	  Timed out 	  Timed out 	     	  -  
                                            2 	  19 	  22 	  20 	     4.69.158.145	 ae-205-3605.edge4.chicago2.level3.net  
                                            3 	  25 	  19 	  19 	     4.69.158.145	 ae-205-3605.edge4.chicago2.level3.net  
                                            4 	  24 	  24 	  24 	     4.28.68.22	  -  
                                            5 	  49 	  49 	  49 	     199.212.168.186	 ge8-2.hcap7-tor.bb.allstream.net  
                                            6 	  41 	  41 	  41 	     216.13.105.170	 216-13-105-170.dedicated.allstream.net  
                                            7 	  42 	  42 	  42 	     66.207.112.74	 bb1-core-bra-kaa-g11-v3983.fibrewired.ca  
                                            8 	  46 	  45 	  48 	     216.185.*.110	 mercuri.ca  
                                            9 	  39 	  40 	  40 	     67.69.*.254	  -  
                                            10 	  Timed out 	  Timed out 	  Timed out 	     	  -  
                                            11 	  Timed out 	  Timed out 	  Timed out 	     	  -  
                                            

                                            216.185..110 don't belong to me, but belongs to my isp2 upstream… 67.69..254 is my pfsense isp1 gw.....

                                            used http://network-tools dot com to get this

                                            isp2 should have went 216.185..110 then to 216.185..1 which is the isp2 gw i connect to... then 216.185.*.2 which is my pfsense box

                                            1 Reply Last reply Reply Quote 0
                                            • Derelict
                                              Derelict LAYER 8 Netgate last edited by

                                              No.  I meant from inside.  Usually when TTLs expire in your situation you have a routing loop.

                                              Traceroute to .166 from .201

                                              1 Reply Last reply Reply Quote 0
                                              • D
                                                Disturbed1 last edited by

                                                tracert from 201 to 166

                                                C:\Users\chrism>tracert 216.185.*.166
                                                
                                                Tracing route to ******************************** [216.185.*.166]
                                                over a maximum of 30 hops:
                                                
                                                  1     3 ms    29 ms    11 ms  216.185.*.1
                                                  2     2 ms     3 ms     3 ms  216.185.*.1
                                                  3     4 ms     5 ms     4 ms  216.185.*.1
                                                  4     5 ms     5 ms     4 ms  216.185.*.1
                                                  5     5 ms     4 ms     5 ms  216.185.*.1
                                                  6    12 ms    75 ms    34 ms  216.185.*.1
                                                  7     7 ms     8 ms     7 ms  216.185.*.1
                                                  8     6 ms     4 ms     7 ms  ^C
                                                

                                                just repeats till i ctrl c…

                                                1 Reply Last reply Reply Quote 0
                                                • Derelict
                                                  Derelict LAYER 8 Netgate last edited by

                                                  You have something configured wrong.  What's your IPV4 routing table?  What are all your interfaces configured like?  I don't know how secret you think your IP address is but it's probably getting pretty tedious masking it.

                                                  What interface is configured as .1?  What's its netmask?  I'm guessing here.  you're going to have to figure out why pfSense keeps routing back to .1.

                                                  1 Reply Last reply Reply Quote 0
                                                  • Derelict
                                                    Derelict LAYER 8 Netgate last edited by

                                                    Just referenced your diagram again.  .1 is your ISP gateway.  That doesn't make any sense because traceroute hop1 should be the pfSense interface facing that segment.

                                                    ETA: Hmm.  pfSense is invisible in my traceroutes.  But only when I NAT.

                                                    1 Reply Last reply Reply Quote 0
                                                    • D
                                                      Disturbed1 last edited by

                                                      i hear ya… i've been poking away at this for some time too...

                                                      i don't think this will help... but when i did test at 5am using route only platform, all subnets could cross talk with no problems...

                                                      1 Reply Last reply Reply Quote 0
                                                      • Derelict
                                                        Derelict LAYER 8 Netgate last edited by

                                                        Just forget about Route only platform. It will not do what you need.  It also turns off all firewalling and makes all your public IPs wide open.  What you're doing isn't that complicated.  I think you got a little clicky clicky and have something in there that's wrong - somewhere.  What are your NAT rules currently?  What's your IPv4 routing table?

                                                        1 Reply Last reply Reply Quote 0
                                                        • D
                                                          Disturbed1 last edited by

                                                          ka… and yes... i give up on masking lol

                                                          heres screen shots


                                                          1 Reply Last reply Reply Quote 0
                                                          • Derelict
                                                            Derelict LAYER 8 Netgate last edited by

                                                            See those two routes for 216.185.64.6 and 216.185.75.161 with a gateway of 216.185.75.1?

                                                            Those (particularly the 161) are probably your problem.  Somewhere pfSense has been told to send everything destined for 216.185.75.161 out to your ISP's .1 address.

                                                            1 Reply Last reply Reply Quote 0
                                                            • D
                                                              Disturbed1 last edited by

                                                              the only places i can think of that happening would be system:gateways and/or firewall rule interface gateway set to netoptiks… with out that there it wanted to route out isp1...

                                                              1 Reply Last reply Reply Quote 0
                                                              • Derelict
                                                                Derelict LAYER 8 Netgate last edited by

                                                                Without that there it will route out whatever your default gateway is.

                                                                1 Reply Last reply Reply Quote 0
                                                                • D
                                                                  Disturbed1 last edited by

                                                                  ka, so we've come to a couple ideas where the possible problem may be…

                                                                  i'll make changes to .160/27 so that subnet has gateway 161 and not 190,

                                                                  other

                                                                  maybe bgp...

                                                                  will post in morning with resaults from subnet restructuring, and if that don't resolve, presue the bgp...

                                                                  anyone else with any ideas please feel free to jump in...

                                                                  Thank you Derelict for the time u spent... huge help in long run... :) solved my VIP issue...

                                                                  1 Reply Last reply Reply Quote 0
                                                                  • Derelict
                                                                    Derelict LAYER 8 Netgate last edited by

                                                                    i'll make changes to .160/27 so that subnet has gateway 161 and not 190

                                                                    The only way that should make any difference is if something on the LAN thinks .161 should be the the default gateway.  Like I said in the PM, there's no reason not to use .190 as the interface address/gateway as long as everything on the LAN knows that's the case (just like with .161). Most people use the first IP in the subnet but that's just convention, not a requirement by any means.

                                                                    1 Reply Last reply Reply Quote 0
                                                                    • D
                                                                      Disturbed1 last edited by

                                                                      SOLVED by accident….

                                                                      interface settings: IPv4 Upstream Gateway: changed from none to isp2 gw

                                                                      and

                                                                      changed firewall rules for subnets from isp2. "gateway" was set as netoptiks (isp2) changed back to default....

                                                                      after doing this was able to get communication between lans(subnets).....

                                                                      4 months banging head on desk... :)

                                                                      1 Reply Last reply Reply Quote 0
                                                                      • D
                                                                        Disturbed1 last edited by

                                                                        possible related issue here….

                                                                        so after thinking everything was all good, i saw something strange and doesn't look right....

                                                                        from a subnet on isp2 (blue) i ran a traceroute and this was the resault...

                                                                        C:\Users\chrism>tracert 8.8.8.8
                                                                        Tracing route to google-public-dns-a.google.com [8.8.8.8]
                                                                        over a maximum of 30 hops:
                                                                          1    <1 ms    <1 ms    <1 ms  office [192.168.0.2]
                                                                          2     6 ms     3 ms     4 ms  host31.indicativesolutions.com [216.185.75.190]
                                                                          3     8 ms     9 ms    11 ms  67.69.244.253
                                                                          4    10 ms    10 ms    11 ms  tcore3-kitchener06_TenGigE0-10-0-3.net.bell.ca [64.230.111.82]
                                                                          5    10 ms    11 ms    10 ms  tcore3-toronto63_pos1-5-0-0.net.bell.ca [64.230.50.49]
                                                                          6     9 ms    11 ms    12 ms  tcore3-torontoxn_HundredGigE0-8-0-0.net.bell.ca[64.230.50.7]
                                                                          7     9 ms    18 ms    12 ms  bx1-torontoxn_et1-0-0.net.bell.ca [64.230.97.157]
                                                                          8     9 ms    10 ms     9 ms  72.14.221.233
                                                                          9    48 ms    74 ms     9 ms  216.239.47.114
                                                                         10    20 ms    19 ms    21 ms  216.239.46.160
                                                                         11    52 ms    34 ms    35 ms  64.233.174.88
                                                                         12    32 ms    34 ms    32 ms  216.239.46.193
                                                                         13     *        *        *     Request timed out.
                                                                         14    32 ms    34 ms    31 ms  google-public-dns-a.google.com [8.8.8.8]
                                                                        Trace complete.
                                                                        C:\Users\chrism>
                                                                        

                                                                        216.185.75.190 should not have routed out 67.69.244.253 (<-belongs to isp1 pink) but rather should have stayed in isp2 gw which is 216.185.75.1…

                                                                        any suggestions????????

                                                                        1 Reply Last reply Reply Quote 0
                                                                        • Derelict
                                                                          Derelict LAYER 8 Netgate last edited by

                                                                          What are the firewall rules for the interface on which 216.185.75.190 can be found?

                                                                          Which WAN is set as your default gateway?

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • D
                                                                            Disturbed1 last edited by

                                                                            default wan is isp1bell (pink)

                                                                            when i set firewall rules for blue(isp2) subnets to default i can cross talk but wrong outbound, when firewall rules set gw as netoptiks(isp2) they route outbound proper but can't cross talk…

                                                                            1 Reply Last reply Reply Quote 0
                                                                            • Derelict
                                                                              Derelict LAYER 8 Netgate last edited by

                                                                              Seems like it shouldn't do that.

                                                                              You might try creating something like a local_nets alias, put the /24 in it (or whatever local networks you want to "cross talk" with) and put a pass rule on each LAN interface from "THAT_INTERFACE net" to local_nets with the default gateway (*/none).

                                                                              Follow that with a pass any any any rule with the desired egress gateway set.

                                                                              1 Reply Last reply Reply Quote 0
                                                                              • D
                                                                                Disturbed1 last edited by

                                                                                i'll give it a try…. will report back later...

                                                                                1 Reply Last reply Reply Quote 0
                                                                                • First post
                                                                                  Last post

                                                                                Products

                                                                                • Platform Overview
                                                                                • TNSR
                                                                                • pfSense
                                                                                • Appliances

                                                                                Services

                                                                                • Training
                                                                                • Professional Services

                                                                                Support

                                                                                • Subscription Plans
                                                                                • Contact Support
                                                                                • Product Lifecycle
                                                                                • Documentation

                                                                                News

                                                                                • Media Coverage
                                                                                • Press
                                                                                • Events

                                                                                Resources

                                                                                • Blog
                                                                                • FAQ
                                                                                • Find a Partner
                                                                                • Resource Library
                                                                                • Security Information

                                                                                Company

                                                                                • About Us
                                                                                • Careers
                                                                                • Partners
                                                                                • Contact Us
                                                                                • Legal
                                                                                Our Mission

                                                                                We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                                                Subscribe to our Newsletter

                                                                                Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                                                © 2021 Rubicon Communications, LLC | Privacy Policy