Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Dual (2)WAN / Multi (9)LAN Routing Issue with Public IP's

    Scheduled Pinned Locked Moved Routing and Multi WAN
    38 Posts 2 Posters 5.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      Disturbed1
      last edited by

      zero….

      screen shot of gateways in post #10

      staticroutes.png
      staticroutes.png_thumb

      1 Reply Last reply Reply Quote 0
      • DerelictD Offline
        Derelict LAYER 8 Netgate
        last edited by

        When you traceroute it what IPs is it bouncing between?

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • D Offline
          Disturbed1
          last edited by

          now this is where it gets funny….

          216.185.*.201 is from Canada(CA) in region North America

TraceRoute from Network-Tools.com to 216.185.*.201 [*************]
Hop	(ms)	(ms)	(ms)		     IP Address	Host name
1 	  Timed out 	  Timed out 	  Timed out 	     	  -  
2 	  19 	  22 	  20 	     4.69.158.145	 ae-205-3605.edge4.chicago2.level3.net  
3 	  25 	  19 	  19 	     4.69.158.145	 ae-205-3605.edge4.chicago2.level3.net  
4 	  24 	  24 	  24 	     4.28.68.22	  -  
5 	  49 	  49 	  49 	     199.212.168.186	 ge8-2.hcap7-tor.bb.allstream.net  
6 	  41 	  41 	  41 	     216.13.105.170	 216-13-105-170.dedicated.allstream.net  
7 	  42 	  42 	  42 	     66.207.112.74	 bb1-core-bra-kaa-g11-v3983.fibrewired.ca  
8 	  46 	  45 	  48 	     216.185.*.110	 mercuri.ca  
9 	  39 	  40 	  40 	     67.69.*.254	  -  
10 	  Timed out 	  Timed out 	  Timed out 	     	  -  
11 	  Timed out 	  Timed out 	  Timed out 	     	  - 

          216.185..110 don't belong to me, but belongs to my isp2 upstream… 67.69..254 is my pfsense isp1 gw.....

          used http://network-tools dot com to get this

          isp2 should have went 216.185..110 then to 216.185..1 which is the isp2 gw i connect to... then 216.185.*.2 which is my pfsense box

          1 Reply Last reply Reply Quote 0
          • DerelictD Offline
            Derelict LAYER 8 Netgate
            last edited by

            No.  I meant from inside.  Usually when TTLs expire in your situation you have a routing loop.

            Traceroute to .166 from .201

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • D Offline
              Disturbed1
              last edited by

              tracert from 201 to 166

              C:\Users\chrism>tracert 216.185.*.166

Tracing route to ******************************** [216.185.*.166]
over a maximum of 30 hops:

  1     3 ms    29 ms    11 ms  216.185.*.1
  2     2 ms     3 ms     3 ms  216.185.*.1
  3     4 ms     5 ms     4 ms  216.185.*.1
  4     5 ms     5 ms     4 ms  216.185.*.1
  5     5 ms     4 ms     5 ms  216.185.*.1
  6    12 ms    75 ms    34 ms  216.185.*.1
  7     7 ms     8 ms     7 ms  216.185.*.1
  8     6 ms     4 ms     7 ms  ^C

              just repeats till i ctrl c…

              1 Reply Last reply Reply Quote 0
              • DerelictD Offline
                Derelict LAYER 8 Netgate
                last edited by

                You have something configured wrong.  What's your IPV4 routing table?  What are all your interfaces configured like?  I don't know how secret you think your IP address is but it's probably getting pretty tedious masking it.

                What interface is configured as .1?  What's its netmask?  I'm guessing here.  you're going to have to figure out why pfSense keeps routing back to .1.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • DerelictD Offline
                  Derelict LAYER 8 Netgate
                  last edited by

                  Just referenced your diagram again.  .1 is your ISP gateway.  That doesn't make any sense because traceroute hop1 should be the pfSense interface facing that segment.

                  ETA: Hmm.  pfSense is invisible in my traceroutes.  But only when I NAT.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • D Offline
                    Disturbed1
                    last edited by

                    i hear ya… i've been poking away at this for some time too...

                    i don't think this will help... but when i did test at 5am using route only platform, all subnets could cross talk with no problems...

                    1 Reply Last reply Reply Quote 0
                    • DerelictD Offline
                      Derelict LAYER 8 Netgate
                      last edited by

                      Just forget about Route only platform. It will not do what you need.  It also turns off all firewalling and makes all your public IPs wide open.  What you're doing isn't that complicated.  I think you got a little clicky clicky and have something in there that's wrong - somewhere.  What are your NAT rules currently?  What's your IPv4 routing table?

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • D Offline
                        Disturbed1
                        last edited by

                        ka… and yes... i give up on masking lol

                        heres screen shots

                        currentnatoutbound.png
                        currentnatoutbound.png_thumb

                        1 Reply Last reply Reply Quote 0
                        • DerelictD Offline
                          Derelict LAYER 8 Netgate
                          last edited by

                          See those two routes for 216.185.64.6 and 216.185.75.161 with a gateway of 216.185.75.1?

                          Those (particularly the 161) are probably your problem.  Somewhere pfSense has been told to send everything destined for 216.185.75.161 out to your ISP's .1 address.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • D Offline
                            Disturbed1
                            last edited by

                            the only places i can think of that happening would be system:gateways and/or firewall rule interface gateway set to netoptiks… with out that there it wanted to route out isp1...

                            1 Reply Last reply Reply Quote 0
                            • DerelictD Offline
                              Derelict LAYER 8 Netgate
                              last edited by

                              Without that there it will route out whatever your default gateway is.

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • D Offline
                                Disturbed1
                                last edited by

                                ka, so we've come to a couple ideas where the possible problem may be…

                                i'll make changes to .160/27 so that subnet has gateway 161 and not 190,

                                other

                                maybe bgp...

                                will post in morning with resaults from subnet restructuring, and if that don't resolve, presue the bgp...

                                anyone else with any ideas please feel free to jump in...

                                Thank you Derelict for the time u spent... huge help in long run... :) solved my VIP issue...

                                1 Reply Last reply Reply Quote 0
                                • DerelictD Offline
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  i'll make changes to .160/27 so that subnet has gateway 161 and not 190

                                  The only way that should make any difference is if something on the LAN thinks .161 should be the the default gateway.  Like I said in the PM, there's no reason not to use .190 as the interface address/gateway as long as everything on the LAN knows that's the case (just like with .161). Most people use the first IP in the subnet but that's just convention, not a requirement by any means.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • D Offline
                                    Disturbed1
                                    last edited by

                                    SOLVED by accident….

                                    interface settings: IPv4 Upstream Gateway: changed from none to isp2 gw

                                    and

                                    changed firewall rules for subnets from isp2. "gateway" was set as netoptiks (isp2) changed back to default....

                                    after doing this was able to get communication between lans(subnets).....

                                    4 months banging head on desk... :)

                                    1 Reply Last reply Reply Quote 0
                                    • D Offline
                                      Disturbed1
                                      last edited by

                                      possible related issue here….

                                      so after thinking everything was all good, i saw something strange and doesn't look right....

                                      from a subnet on isp2 (blue) i ran a traceroute and this was the resault...

                                      C:\Users\chrism>tracert 8.8.8.8
Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:
  1    <1 ms    <1 ms    <1 ms  office [192.168.0.2]
  2     6 ms     3 ms     4 ms  host31.indicativesolutions.com [216.185.75.190]
  3     8 ms     9 ms    11 ms  67.69.244.253
  4    10 ms    10 ms    11 ms  tcore3-kitchener06_TenGigE0-10-0-3.net.bell.ca [64.230.111.82]
  5    10 ms    11 ms    10 ms  tcore3-toronto63_pos1-5-0-0.net.bell.ca [64.230.50.49]
  6     9 ms    11 ms    12 ms  tcore3-torontoxn_HundredGigE0-8-0-0.net.bell.ca[64.230.50.7]
  7     9 ms    18 ms    12 ms  bx1-torontoxn_et1-0-0.net.bell.ca [64.230.97.157]
  8     9 ms    10 ms     9 ms  72.14.221.233
  9    48 ms    74 ms     9 ms  216.239.47.114
 10    20 ms    19 ms    21 ms  216.239.46.160
 11    52 ms    34 ms    35 ms  64.233.174.88
 12    32 ms    34 ms    32 ms  216.239.46.193
 13     *        *        *     Request timed out.
 14    32 ms    34 ms    31 ms  google-public-dns-a.google.com [8.8.8.8]
Trace complete.
C:\Users\chrism>

                                      216.185.75.190 should not have routed out 67.69.244.253 (<-belongs to isp1 pink) but rather should have stayed in isp2 gw which is 216.185.75.1…

                                      any suggestions????????

                                      1 Reply Last reply Reply Quote 0
                                      • DerelictD Offline
                                        Derelict LAYER 8 Netgate
                                        last edited by

                                        What are the firewall rules for the interface on which 216.185.75.190 can be found?

                                        Which WAN is set as your default gateway?

                                        Chattanooga, Tennessee, USA
                                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                        1 Reply Last reply Reply Quote 0
                                        • D Offline
                                          Disturbed1
                                          last edited by

                                          default wan is isp1bell (pink)

                                          when i set firewall rules for blue(isp2) subnets to default i can cross talk but wrong outbound, when firewall rules set gw as netoptiks(isp2) they route outbound proper but can't cross talk…

                                          1 Reply Last reply Reply Quote 0
                                          • DerelictD Offline
                                            Derelict LAYER 8 Netgate
                                            last edited by

                                            Seems like it shouldn't do that.

                                            You might try creating something like a local_nets alias, put the /24 in it (or whatever local networks you want to "cross talk" with) and put a pass rule on each LAN interface from "THAT_INTERFACE net" to local_nets with the default gateway (*/none).

                                            Follow that with a pass any any any rule with the desired egress gateway set.

                                            Chattanooga, Tennessee, USA
                                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.