Two Bridges, two WAN and firewall rules problem



  • Hi i have two bridges, so 4 ifaces, 1 bridge is used for one ISP, the second for the other ISP.
    i have servers behind this with a public IP all in a switch and i use rules to block some traffic.
    I have problems when one server of the ISP1 trying to UPLOAD files to the ISP2, the firewall probably drop packets because if i disable firewall rules all work well, but when i active the rules the problem start, i check the firewall rules and y put all in GREEN, pass all traffic in all the 4 interfaces but the problem still there. How can i solve this problem? i try bypass firewall rules, i try other various options.

    NOTE: The clients don't have any problem, they can upload files because the connection start in bridge1 and pass to bridge1 again, but when the packets start in bridge1 and enter in bridge2 then is blocked by "ghost" i don't know.

    Thanks



  • I don't suppose that you could post a sanitized copy/screen shot of your rules?



  • I don't think if necesary :/ all interfaces was pass all trafic * * * for all protocols… so any difference.
    but i will provide one if this solve something...
    the screenshot is here... check
    http://www.image-share.com/ipng-1502-122.html

    Note: i'm try conservative, latency... and is not work :/ i'm try Clear invalid DF bits instead of dropping the packets
    Disables the PF scrubbing option which can sometimes interfere with NFS and PPTP traffic.
    well almost any option and combinations...

    thanks

    @podilarius:

    I don't suppose that you could post a sanitized copy/screen shot of your rules?



  • Sorry, I would also like the rules from the other bridge also.



  • @podilarius:

    Sorry, I would also like the rules from the other bridge also.

    Are the same of the image an all 4 bridge interfaces gtd_1 and 2 and entel_1 and 2

    :/ all pass…



  • Do you have keep states turned off?



  • @podilarius:

    Do you have keep states turned off?

    I test with keep states, sloppy states and none    and NOT WORK :s



  • What kind of NAT are you doing? I think I will setup a test with bridging to see …

    This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
    I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
    I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.



  • @podilarius:

    What kind of NAT are you doing? I think I will setup a test with bridging to see …

    This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
    I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
    I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.

    there is no NAT i have Dell poweredge r710 with 6 NICs
    and TWO ISP
    GTD and ENTEL

    i have several Ip blocks in every ISPs comes from 2 fiber optics mostly /29 nets

    the connection is some like this
    GTD  –-> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN
    ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN
    pfsense(nic5) ---> SWITCH LAN (i use this nic with 192.168.1.250 ip to control pfsense webadministrator

    so i don't have any NAT configured just 4 NIC cards  2 joined with a bridge for GTD provider and the other two with other bridge for the other ISP.

    i configure my servers to route default for one or other with this script:

    the script purpose if when the packet come from entel provider (190.151.x.x) go with default gateway to the corresponding default gateway of the providers.

    ip route add table T1 default via 190.196.32.89
    ip route add table T2 default via 190.151.71.161

    ip rule add from 190.196.32.93 table T1
    ip rule add to 190.196.32.93 table T1

    ip rule add from 190.151.71.171 table T2
    ip rule add from 190.151.71.171 table T2

    ip route flush cache

    the script works i can probe with a traceroute here.
    [root@correo ~]# traceroute -s 190.151.71.171 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
    1  200.111.166.121 (200.111.166.121)  1.790 ms  1.625 ms  1.837 ms  <–-- DEFAULT GATEWAY OF ENTEL provider
    2  192.168.90.173 (192.168.90.173)  20.690 ms  20.612 ms  21.075 ms

    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
    1  190.196.32.89 (190.196.32.89)  2.491 ms  2.457 ms  2.448 ms  <--- DEFAULT GATEWAY OF GTD provider
    2  190.196.126.126 (190.196.126.126)  2.452 ms  2.446 ms  2.435 ms
    cn1.ge0-0-3.str2.gtdinternet.com (201.238.238.69)  3.742 ms  3.735 ms  3.798 ms

    so all servers are configured the same with their respective providers the script work

    when a client OUTSIDE MY NETWORK try to upload files if the packet come from entel is responsed to entel gateway
    and if the IP packet come from GTD the server response from GTD. The system works fine no problem

    so in my diagram
    (CLIENT HERE) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN <-->(server here) THIS WORK
    (or client here) ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN <--> (server here) THIS WORK

    all work fine if the client start the connection

    but if i have server1 and server2 in MY NETWORK and try to connect one with other PFSENSE block the trafic

    (REACH GTD ROUTER) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) <-->SWITCH WAN  <--- (IF TRAFFIC START HERE trying to go out to server2)
    (back from entel) -->ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN  ????? BLOCKED after ~48KBs of transfer....

    so the problem if here i think if a routing problem maybe, pfsense don't understand why one packet goes out from one bridge and enter from the other bridge and is confused...

    some ideas ????
    i will provide some screenshots of my CFGs

    thanks














  • @podilarius:

    What kind of NAT are you doing? I think I will setup a test with bridging to see …

    This might be just a routing problem. I am guessing that your 2 ISPs are in seperate subnets.
    I think it should be routing to the ISP default gateway and then over to your second bridge through the second ISP. If you have told it not to NAT then the system will not know how to route back.
    I think we are going to need more information here. NAT rules, route rules and such ... the firewall rules themselves seem to be fine. Just don't understand how it is working with firewall disabled.

    there is no NAT i have Dell poweredge r710 with 6 NICs
    and TWO ISP
    GTD and ENTEL

    i have several Ip blocks in every ISPs comes from 2 fiber optics mostly /29 nets

    the connection is some like this
    GTD  –-> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN
    ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN
    pfsense(nic5) ---> SWITCH LAN (i use this nic with 192.168.1.250 ip to control pfsense webadministrator

    so i don't have any NAT configured just 4 NIC cards  2 joined with a bridge for GTD provider and the other two with other bridge for the other ISP.

    i configure my servers to route default for one or other with this script:

    the script purpose if when the packet come from entel provider (190.151.x.x) go with default gateway to the corresponding default gateway of the providers.

    ip route add table T1 default via 190.196.32.89
    ip route add table T2 default via 190.151.71.161

    ip rule add from 190.196.32.93 table T1
    ip rule add to 190.196.32.93 table T1

    ip rule add from 190.151.71.171 table T2
    ip rule add from 190.151.71.171 table T2

    ip route flush cache

    the script works i can probe with a traceroute here.
    [root@correo ~]# traceroute -s 190.151.71.171 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
    1  200.111.166.121 (200.111.166.121)  1.790 ms  1.625 ms  1.837 ms  <–-- DEFAULT GATEWAY OF ENTEL provider
    2  192.168.90.173 (192.168.90.173)  20.690 ms  20.612 ms  21.075 ms

    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
    1  190.196.32.89 (190.196.32.89)  2.491 ms  2.457 ms  2.448 ms  <--- DEFAULT GATEWAY OF GTD provider
    2  190.196.126.126 (190.196.126.126)  2.452 ms  2.446 ms  2.435 ms
    cn1.ge0-0-3.str2.gtdinternet.com (201.238.238.69)  3.742 ms  3.735 ms  3.798 ms

    so all servers are configured the same with their respective providers the script work

    when a client OUTSIDE MY NETWORK try to upload files if the packet come from entel is responsed to entel gateway
    and if the IP packet come from GTD the server response from GTD. The system works fine no problem

    so in my diagram
    (CLIENT HERE) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) -->SWITCH WAN <-->(server here) THIS WORK
    (or client here) ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN <--> (server here) THIS WORK

    all work fine if the client start the connection

    but if i have server1 and server2 in MY NETWORK and try to connect one with other PFSENSE block the trafic

    (REACH GTD ROUTER) GTD  ---> to pfsense(nic1)  (bridge_gtd here) (nic3) <-->SWITCH WAN  <--- (IF TRAFFIC START HERE trying to go out to server2)
    (back from entel) -->ENTEL --> to pfsense(nic2)  (bridge_entel here) (nic4) -->SWITCH WAN  ????? BLOCKED after ~48KBs of transfer....

    so the problem if here i think if a routing problem maybe, pfsense don't understand why one packet goes out from one bridge and enter from the other bridge and is confused...

    some ideas ????
    i will provide some screenshots of my CFGs





    thanks



  • So the servers have 2 NICs one in each providers IP subnet? If that is the case, then pfSense should never be reached, it should just the NIC in the correct subnet. Are you just creating routing rules on the servers themselves? Are you using firewall rules on the servers themselves?



  • No the IPTABLES of the servers was flushed, i can see the packets pass thoug the firewall usign the packet capture function, if i go to system->advanced->firewall/NAT->disable all packet filtering. and check this option all work fine but i don't have firewall rules working :/ so the servers was blocked by the firewall



  • I saw your firewall rules. You have an allow all at the top, so anything should be able to pass. What I am saying is that if you have 2 NIC in each server, one on either WAN subnet, then pfSense should not be consulted when trying to go from server to server (WAN1 to WAN2). It should just use the NIC in that subnet. If you are doing something weird with the server routing, you might be preventing it from working as it should.
    What do you have set for your outbound NAT? Please make sure it is set to manual (not auto) and there are no rules except for the LAN subnet. When you turn off firewalling, it disables NAT also.



  • I don't have any nat i use the firewall like a filter bridge of my optic fibers.

    In the servers i have 1 or 2 nics not all servers have two providers connected, but if server 1 is connected to GTD and server 2 connected with ENTEL if i send mails or something work but if i attach files to mail and send, or transfer more than ~48Kb the pfsense block the traffic.

    this is my map of connections

    SERVER1 –->SWITCH<--- SERVER2
                          |        |
                          |        | FO (ENTEL AND GTD)
                        PFSENSE  <---- HERE IS THE BRIDGES one bridge for entel and other for GTD providers...
                          |        |  <--FO external providers...

    this is the nat screenshot now...




  • On GTD_1 and ENTEL_1 on that allow everything rule at the top, is keep state on or off in the advanced options?



  • Now states are in keep states, but i try with sloppy and not work, when i try with NONE i lose conection at all in the bridge.

    Some idea?



  • I would do a tcpdump at each NIC to see where the packets are getting to before they are dropped. You might want to also log dropped packets. Are you seeing any drops or other errors in the system logs?



  • Ran across this today:

    http://forum.pfsense.org/index.php/topic,50711.0/topicseen.html

    See step number 2. Did you setup this advanced option:
    2. VERY IMPORTANT: As mentioned at http://forum.pfsense.org/index.php?topic=30653.0, go to the 'System -> Advanced -> System Tunables'  and set net.link.bridge.pfil_bridge from 'default' to '1'


Locked