Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    Two Bridges, two WAN and firewall rules problem

    Scheduled Pinned Locked Moved Firewalling
    18 Posts 2 Posters 5.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      podilarius
      last edited by

      Sorry, I would also like the rules from the other bridge also.

      1 Reply Last reply Reply Quote 0
      • M Offline
        morpheus747
        last edited by

        @podilarius:

        Sorry, I would also like the rules from the other bridge also.

        Are the same of the image an all 4 bridge interfaces gtd_1 and 2 and entel_1 and 2

        :/ all pass…

        1 Reply Last reply Reply Quote 0
        • P Offline
          podilarius
          last edited by

          Do you have keep states turned off?

          1 Reply Last reply Reply Quote 0
          • M Offline
            morpheus747
            last edited by

            @podilarius:

            Do you have keep states turned off?

            I test with keep states, sloppy states and none    and NOT WORK :s

            1 Reply Last reply Reply Quote 0
            • P Offline
              podilarius
              last edited by

              What kind of NAT are you doing? I think I will setup a test with bridging to see …

              This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
              I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
              I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.

              1 Reply Last reply Reply Quote 0
              • M Offline
                morpheus747
                last edited by

                @podilarius:

                What kind of NAT are you doing? I think I will setup a test with bridging to see …

                This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
                I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
                I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.

                there is no NAT i have Dell poweredge r710 with 6 NICs
                and TWO ISP
                GTD and ENTEL

                i have several Ip blocks in every ISPs comes from 2 fiber optics mostly /29 nets

                the connection is some like this
                GTD  –-> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN
                ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN
                pfsense(nic5) ---> SWITCH LAN (i use this nic with 192.168.1.250 ip to control pfsense webadministrator

                so i don't have any NAT configured just 4 NIC cards  2 joined with a bridge for GTD provider and the other two with other bridge for the other ISP.

                i configure my servers to route default for one or other with this script:

                the script purpose if when the packet come from entel provider (190.151.x.x) go with default gateway to the corresponding default gateway of the providers.

                ip route add table T1 default via 190.196.32.89
                ip route add table T2 default via 190.151.71.161

                ip rule add from 190.196.32.93 table T1
                ip rule add to 190.196.32.93 table T1

                ip rule add from 190.151.71.171 table T2
                ip rule add from 190.151.71.171 table T2

                ip route flush cache

                the script works i can probe with a traceroute here.
                [root@correo ~]# traceroute -s 190.151.71.171 8.8.8.8
                traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
                1  200.111.166.121 (200.111.166.121)  1.790 ms  1.625 ms  1.837 ms  <–-- DEFAULT GATEWAY OF ENTEL provider
                2  192.168.90.173 (192.168.90.173)  20.690 ms  20.612 ms  21.075 ms

                traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
                1  190.196.32.89 (190.196.32.89)  2.491 ms  2.457 ms  2.448 ms  <--- DEFAULT GATEWAY OF GTD provider
                2  190.196.126.126 (190.196.126.126)  2.452 ms  2.446 ms  2.435 ms
                3  cn1.ge0-0-3.str2.gtdinternet.com (201.238.238.69)  3.742 ms  3.735 ms  3.798 ms

                so all servers are configured the same with their respective providers the script work

                when a client OUTSIDE MY NETWORK try to upload files if the packet come from entel is responsed to entel gateway
                and if the IP packet come from GTD the server response from GTD. The system works fine no problem

                so in my diagram
                (CLIENT HERE) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN <-->(server here) THIS WORK
                (or client here) ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN <--> (server here) THIS WORK

                all work fine if the client start the connection

                but if i have server1 and server2 in MY NETWORK and try to connect one with other PFSENSE block the trafic

                (REACH GTD ROUTER) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) <-->SWITCH WAN  <--- (IF TRAFFIC START HERE trying to go out to server2)
                (back from entel) -->ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN  ????? BLOCKED after ~48KBs of transfer....

                so the problem if here i think if a routing problem maybe, pfsense don't understand why one packet goes out from one bridge and enter from the other bridge and is confused...

                some ideas ????
                i will provide some screenshots of my CFGs

                thanks

                Pantallazo.png
                Pantallazo.png_thumb
                Pantallazo-1.png
                Pantallazo-1.png_thumb
                Pantallazo-2.png
                Pantallazo-2.png_thumb
                Pantallazo-3.png
                Pantallazo-3.png_thumb
                Pantallazo-4.png
                Pantallazo-4.png_thumb
                Pantallazo-5.png
                Pantallazo-5.png_thumb

                1 Reply Last reply Reply Quote 0
                • M Offline
                  morpheus747
                  last edited by

                  @podilarius:

                  What kind of NAT are you doing? I think I will setup a test with bridging to see …

                  This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
                  I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
                  I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.

                  there is no NAT i have Dell poweredge r710 with 6 NICs
                  and TWO ISP
                  GTD and ENTEL

                  i have several Ip blocks in every ISPs comes from 2 fiber optics mostly /29 nets

                  the connection is some like this
                  GTD  –-> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN
                  ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN
                  pfsense(nic5) ---> SWITCH LAN (i use this nic with 192.168.1.250 ip to control pfsense webadministrator

                  so i don't have any NAT configured just 4 NIC cards  2 joined with a bridge for GTD provider and the other two with other bridge for the other ISP.

                  i configure my servers to route default for one or other with this script:

                  the script purpose if when the packet come from entel provider (190.151.x.x) go with default gateway to the corresponding default gateway of the providers.

                  ip route add table T1 default via 190.196.32.89
                  ip route add table T2 default via 190.151.71.161

                  ip rule add from 190.196.32.93 table T1
                  ip rule add to 190.196.32.93 table T1

                  ip rule add from 190.151.71.171 table T2
                  ip rule add from 190.151.71.171 table T2

                  ip route flush cache

                  the script works i can probe with a traceroute here.
                  [root@correo ~]# traceroute -s 190.151.71.171 8.8.8.8
                  traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
                  1  200.111.166.121 (200.111.166.121)  1.790 ms  1.625 ms  1.837 ms  <–-- DEFAULT GATEWAY OF ENTEL provider
                  2  192.168.90.173 (192.168.90.173)  20.690 ms  20.612 ms  21.075 ms

                  traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
                  1  190.196.32.89 (190.196.32.89)  2.491 ms  2.457 ms  2.448 ms  <--- DEFAULT GATEWAY OF GTD provider
                  2  190.196.126.126 (190.196.126.126)  2.452 ms  2.446 ms  2.435 ms
                  3  cn1.ge0-0-3.str2.gtdinternet.com (201.238.238.69)  3.742 ms  3.735 ms  3.798 ms

                  so all servers are configured the same with their respective providers the script work

                  when a client OUTSIDE MY NETWORK try to upload files if the packet come from entel is responsed to entel gateway
                  and if the IP packet come from GTD the server response from GTD. The system works fine no problem

                  so in my diagram
                  (CLIENT HERE) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN <-->(server here) THIS WORK
                  (or client here) ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN <--> (server here) THIS WORK

                  all work fine if the client start the connection

                  but if i have server1 and server2 in MY NETWORK and try to connect one with other PFSENSE block the trafic

                  (REACH GTD ROUTER) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) <-->SWITCH WAN  <--- (IF TRAFFIC START HERE trying to go out to server2)
                  (back from entel) -->ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN  ????? BLOCKED after ~48KBs of transfer....

                  so the problem if here i think if a routing problem maybe, pfsense don't understand why one packet goes out from one bridge and enter from the other bridge and is confused...

                  some ideas ????
                  i will provide some screenshots of my CFGs





                  thanks

                  1 Reply Last reply Reply Quote 0
                  • P Offline
                    podilarius
                    last edited by

                    So the servers have 2 NICs one in each providers IP subnet? If that is the case, then pfSense should never be reached, it should just the NIC in the correct subnet. Are you just creating routing rules on the servers themselves? Are you using firewall rules on the servers themselves?

                    1 Reply Last reply Reply Quote 0
                    • M Offline
                      morpheus747
                      last edited by

                      No the IPTABLES of the servers was flushed, i can see the packets pass thoug the firewall usign the packet capture function, if i go to system->advanced->firewall/NAT->disable all packet filtering. and check this option all work fine but i don't have firewall rules working :/ so the servers was blocked by the firewall

                      1 Reply Last reply Reply Quote 0
                      • P Offline
                        podilarius
                        last edited by

                        I saw your firewall rules. You have an allow all at the top, so anything should be able to pass. What I am saying is that if you have 2 NIC in each server, one on either WAN subnet, then pfSense should not be consulted when trying to go from server to server (WAN1 to WAN2). It should just use the NIC in that subnet. If you are doing something weird with the server routing, you might be preventing it from working as it should.
                        What do you have set for your outbound NAT? Please make sure it is set to manual (not auto) and there are no rules except for the LAN subnet. When you turn off firewalling, it disables NAT also.

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          morpheus747
                          last edited by

                          I don't have any nat i use the firewall like a filter bridge of my optic fibers.

                          In the servers i have 1 or 2 nics not all servers have two providers connected, but if server 1 is connected to GTD and server 2 connected with ENTEL if i send mails or something work but if i attach files to mail and send, or transfer more than ~48Kb the pfsense block the traffic.

                          this is my map of connections

                          SERVER1 –->SWITCH<--- SERVER2
                                                |        |
                                                |        | FO (ENTEL AND GTD)
                                              PFSENSE  <---- HERE IS THE BRIDGES one bridge for entel and other for GTD providers...
                                                |        |  <--FO external providers...

                          this is the nat screenshot now...

                          Pantallazo-9.png
                          Pantallazo-9.png_thumb

                          1 Reply Last reply Reply Quote 0
                          • P Offline
                            podilarius
                            last edited by

                            On GTD_1 and ENTEL_1 on that allow everything rule at the top, is keep state on or off in the advanced options?

                            1 Reply Last reply Reply Quote 0
                            • M Offline
                              morpheus747
                              last edited by

                              Now states are in keep states, but i try with sloppy and not work, when i try with NONE i lose conection at all in the bridge.

                              Some idea?

                              1 Reply Last reply Reply Quote 0
                              • P Offline
                                podilarius
                                last edited by

                                I would do a tcpdump at each NIC to see where the packets are getting to before they are dropped. You might want to also log dropped packets. Are you seeing any drops or other errors in the system logs?

                                1 Reply Last reply Reply Quote 0
                                • P Offline
                                  podilarius
                                  last edited by

                                  Ran across this today:

                                  http://forum.pfsense.org/index.php/topic,50711.0/topicseen.html

                                  See step number 2. Did you setup this advanced option:
                                  2. VERY IMPORTANT: As mentioned at http://forum.pfsense.org/index.php?topic=30653.0, go to the 'System -> Advanced -> System Tunables'  and set net.link.bridge.pfil_bridge from 'default' to '1'

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.