Problems routing opt3 and opt4 through 2nd Wan (opt1)



  • Hey folks,
    I've been wrestling with a multiwan problem for a while and could really use some help.

    Setup:
    WAN1 –> 72.251.230.x / 29 (have 5 statics) FiOS connection
    OPT1 --> DHCP FiOS connection
    LAN --> 10.1.1.x /24
    Opt3 (PublicWiFi) --> 10.1.2.x / 24  should allow traffic to the net but not LAN (open AP) w/ captive portal
    Opt 4 (VODDMZ) --> 10.1.20.x /24  used to allow Verizon's router to connect to the FiOS TV guide and VOD service. This has to go out over the OPT1 connection (per verizon's spec) and until I get it working, my TV service is essentially broken.

    Outbound NAT --> Automatic (had been manual, but I've been told that with RC3 auto should work)

    I have a rule on the LAN side that sends all traffic out over OPT1 (except VoIP and Work VPN which goes out over WAN). That rule works fine. IPchicken.com and wdc.speakeasy.net report that I am using the OPT1 connection.

    Opt3 and Opt4 are setup identically (with the exception that OPT3 has captive portal).
    Proto  Source  Port  Destination  Port  Gateway  Schedule  Description

    Block * * * LAN net * *   BLOCK WLAN->Lan

    pass  * * * * * FiOSr           All traffic out over FiOSR  (opt1)

    However, this does not work. When I change the gateway to Default (WAN), it works fine.

    If anyone has any thoughts, I'd be very appreciative! Thanks in advance



  • As an additional note, when on the Opt3 (wifi) network, the captive portal does not work when the gateway is set to Opt1…it does work when its set to WAN



  • could you make a diagram?
    and provide screenshots of your rules form all interfaces and NAT.

    it is really really hard to understand a setup when it's only described in words >_<
    and rules copied as text give only a vague pointer on how they are set up.



  • Here's my stab at a diagram
    Let me know if this clears things up at all or just muddies the waters :)

    On the LAN side:
    I have rules that determine which traffic (work, VoIP, etc) go out over the WAN (Business FiOS)connection, the rest goes out over my residential connection

    Opt2 and Opt3 are essentially the same
    Opt2 has a public AP with captive portal enabled, should go out over Opt1 (residential FiOS)
    Opt3 has no captive portal and MUST go out over Opt1 (residential fios) so that the connected TVs can get their guide, etc.

    Rules:
    LAN

    PublicWiFi (OPT2)

    VODDMZ (Opt3)

    Thanks again for the help. I'm looking forward to learning what I have done wrong or missed.



  • One more note:
    Since setting up the 2nd WAN (Opt1, Residential FiOS), my site-to-site IPsec VPN has been flaky. It will often work after rebooting the router, but then for only about 24 hours. The tunnel is showing as UP and the logs look normal, but I cannot route between the two networks.

    I'm not sure its directly related to this issue, but wanted to toss it out in case it pointed to something else.



  • Rules are processed from top to down.
    If a rule catches the rest of the rules below is no longer considered.

    to LAN
    you have block rules below allow some allow rules.
    is that desired?

    to VODDMZ

    on your OPT3 Interface you have a rule (the second) with as source * destination * and gateway *
    this means ALL traffic is being routed according to the routingtable (–> * as gateway means that the routing table is used)
    your last rule that has as Gateway your second WAN will never be processed.

    If you just delete the second rule the traffic from your DODDMZ should go out the second WAN.

    to PublicWiFi
    your block rule is below the allow rule.
    it will never be applied (should be in first place like on the VODDMZ interface).

    i'm not sure if that solves your problem.

    when you do a traceroute from within your VODDMZ where does the traffic stop?



  • GruensForeschili - Thank you so much!
    While I've probably trashed any geek credibility I ever had, I swear I did know that rules were processed from the top down. Sometimes you stare at this stuff so long that you totally miss the obvious.

    I'm going to do some testing, but I am confident that removing the 2nd rule will fix my problem on the VODDMZ subnet.

    On the PublicWiFi subnet, I moved the block rule down for testing to see if that would get captive portal working, but I have since moved it back to be the first rule.

    I'm off to do some testing with my fingers crossed - thanks again, I really appreciate the 2nd set of eyes!



  • Ok, after some testing, I can report some success…not total, but some :)
    GruensForeschili was right on the mark with the problem for VODDMZ.
    Removing the 2nd rule got things moving. Lookups were failing but in the DHCP server I put the IPs for OpenDNS and that solved everything.

    PublicWiFi is still broken. I cannot ping the gateway (OPT2 interface) and lookups fail (using either DNS frowarder or OpenDNS).
    I also never get prompted to authenticate against the captive portal.

    Here are the updated screenshots, I could use another set of eyes again to see if I've missed something. Thanks in advance!

    VODDMZ Rules

    PublicWiFi

    Captive Portal setup



  • could you try to get it working first with
    out the restrictions you want? (second wan, no access anywhere else)

    have only an allow all rule with default routing table.
    try it first without CP.
    If that works you can go on from there.

    If not: do you get an IP on a client on WLAN?
    can you ping the gateway directly? after you can ping the gateway can you ping the WAN2 interface?
    if not you should see something in the logs.



  • @GruensFroeschli:

    could you try to get it working first with
    out the restrictions you want? (second wan, no access anywhere else)

    have only an allow all rule with default routing table.
    try it first without CP.
    If that works you can go on from there.

    If not: do you get an IP on a client on WLAN?
    can you ping the gateway directly? after you can ping the gateway can you ping the WAN2 interface?
    if not you should see something in the logs.

    It does in fact work if I use the default gateway (WAN). I get captive portal auth and get out to the net.

    I do get an IP that is clearly coming from PFsense (and not a misconfigured AP for instance).

    I cannot ping either the gateway or WAN2 interface when I have my rule enabled.



  • Just to be sure…..cause i don't know what all those names stands for so bare with me.
    Wan and Opt1 are Wan connection while opt3,4 are LAN connection (where did opt2 go?) right? :)

    1. On Lan nic's don't use * as source but Lan net or ip adresse

    2. There seems to be a limit to multi wan setups. So the lan2, lan3, lan4 gateway has to be the default one. while the lan can be set to wan2 ( someone correct me if I'm wrong ). 
    So from what i can see you need to switch your wan connections so FiOSr will be your wan.



  • @Perry:

    Just to be sure…..cause i don't know what all those names stands for so bare with me.
    Wan and Opt1 are Wan connection while opt3,4 are LAN connection (where did opt2 go?) right? :)

    1. On Lan nic's don't use * as source but Lan net or ip adresse

    2. There seems to be a limit to multi wan setups. So the lan2, lan3, lan4 gateway has to be the default one. while the lan can be set to wan2 ( someone correct me if I'm wrong ). 
    So from what i can see you need to switch your wan connections so FiOSr will be your wan.

    Perry - you are correct about the setup. Actually, I mis-typed… Opt2 and Opt3 are LAN connections Opt 4 and Opt 5 are currently unused

    WAN - STATIC 71.230.xx.zz
    OPT1 - DHCP ISP  71.232.aaa.bbb

    LAN - 10.1.1.x /24
    Opt2 - 10.1.2.x /24
    Opt3 - 10.1.20.x /24

    I 'll change the LAN rules, but not sure why that matters since LAN is working fine.

    As for your 2nd point, my testing seems to confirm that I can use OPT1 as the gateway for the other lan connections.
    The VODDMZ (opt3) subnet is not working perfectly and routing out over OPT1 (FiOSR)
    Its the Opt2 with the captive portal that is the problem - it works if I disable captive portal, but I really need that enabled since that subnet is home to an open access point.



  • Quick update:

    Opt2 is not using the correct gateway, but its totally bypassing captive portal.

    To get it to work I had to add a rule that allowed all traffic to reach the interface address, as shown below.
    If I disable that rule, then routing to the internet breaks.


Log in to reply