VLAN Issue



  • Hey experts  ;)

    Today I have done some random test in my environment. I saw that the VLAN's doesn't work correctli with IPv4 and IPv6.

    Environment:

    One WAN Interface IPv4 and IPV6
    One LAN Interface IPv4 and IPv6
    VLAN3 to VLAN5 Interfaces

    LAN Range 172.16.1.0/16
    VLAN Ranges 10.0.0.0/8 –>  /28 subnet per VLAN

    Have a look at the attachment

    The problem is that from a machine in the VLAN I can reach (ping,http etc.) machines in an other vlan. To test if the VLANs are working correctly (vmware esxi + cisco switches) I replace the IP of a machine in the VLAN3 with one IP from the VLAN4 range. Neither I can reach the machine in the VLAN4 and the VLAN4 gateway (10.0.0.46). So the VLAN are working correctly.

    If I do a traceroute from the machine in the VLAN4 to that one in the VLAN3 I saw that I can passtrough to the VLAN4 gateway. (picture) I tried that also with IPv6 --> same problem

    Two months ago i tried that out at it was working like a charm. I updated the pfsense box to the last patch today without any success (i386)

    Any idea ??

    Tanks for your help  ;)

    regards

    supermega






  • Seems to be the same issue this guy found:

    http://forum.pfsense.org/index.php/topic,61017.0.html



  • I also tried out different combinations with the rules on the interfaces without any success  :(



  • Okay I got the right problem now.

    It is exactly the same problem like the guy had in the thread posted above.

    Can someone tell me the right rules for one of my vlan.

    I want that a vlan can only reach the wan with ipv4 and ipv6 I don't understand exactly how to configure this rules (src, dst).



  • Technically you should not have to configure any rules, as pf should deny implicitly. Could you post your current rules, the contents of /tmp/rules.debug, and the output of```
    pfctl -sr



  • My rules for my guest VLAN currently look like this:

    Make sure that you flush your states (I prefer to do it in command line just to be safe with pfctl -F state) and verify that your pfctl -sr output is what you expect it to be.



  • Thank you timthetortoise :)

    Its working  :)

    I will setup a lot of vlans and I have to do so much rules per interface.

    Why something changed in the new snapshots of pfsense 2.1 ? Before everything between the interfaces was automatically blocked without any rule.



  • I'm having the opposite issue, LOL! I have my setup set to allow OPT1 to any and I can't connect to IP addresses on the LAN interface. em0_vlan69 is LAN em0_vlan15 is OPT1



  • @supermega:

    Why something changed in the new snapshots of pfsense 2.1 ? Before everything between the interfaces was automatically blocked without any rule.

    Hasn't changed ever, a new interface has no rules which means no traffic allowed initiated from that interface, same as always. Initiated into that network is controlled by the source interface's rules.



  • @cmb:

    @supermega:

    Why something changed in the new snapshots of pfsense 2.1 ? Before everything between the interfaces was automatically blocked without any rule.

    Hasn't changed ever, a new interface has no rules which means no traffic allowed initiated from that interface, same as always. Initiated into that network is controlled by the source interface's rules.

    No CMB, there's something seriously wrong in the latest builds. The same rules that USED to work fine don't work anymore on VLAN's to allow traffic between interfaces. I can't communicate with the LAN (which is a VLAN) from other interfaces (including the OpenVPN interface). Something's definitely seriously broken - this always used to work.



  • Can you post your /tmp/rules.debug and screenshots of the gui(or config.xml) to see what is happening htere?



  • I got it to work by adding a new rule I don't think I needed before. I added a rule on the LAN interface to allow ANY to LAN SUBNET. I don't think I had a rule like that before but I do now realize I may have had allow any to any. I don't recall, but should a rule like that be needed on the LAN side? Thank you for your help Ermal. It definitely seems something's changed slightly.



  • It technically shouldn't be necessary. Established connections should be allowed through. I noticed the same thing with my OpenVPN tunnel and chalked it up to a misconfiguration on my part. I'll have to try setting a rule for it today.



  • @timthetortoise:

    It technically shouldn't be necessary. Established connections should be allowed through. I noticed the same thing with my OpenVPN tunnel and chalked it up to a misconfiguration on my part. I'll have to try setting a rule for it today.

    I wasn't just noticing it on the OpenVPN but also on another VLAN. Something seems to have changed in the default rules and it now needs explicitly stated. What would be most helpful Ermal and I will get it posted right away.



  • I'm not noticing that, to be honest - my rules allow LAN1 to LAN2 on the LAN1 side, but not the LAN2 side, and it works correctly (can pass traffic from LAN1 to LAN2, not vice-versa). I have no rules regarding LAN1 in LAN2, except to disallow ingress traffic on LAN2 to LAN1 (again, already established connections should not be affected). These could be completely different issues, but it's very odd nonetheless.



  • @timthetortoise:

    I'm not noticing that, to be honest - my rules allow LAN1 to LAN2 on the LAN1 side, but not the LAN2 side, and it works correctly (can pass traffic from LAN1 to LAN2, not vice-versa). I have no rules regarding LAN1 in LAN2, except to disallow ingress traffic on LAN2 to LAN1 (again, already established connections should not be affected). These could be completely different issues, but it's very odd nonetheless.

    Yes, also limiters are broken I forgot to note that I found that - with in/out limiters set up on my VLAN's (yes, in is masked by source IP and out by destination IP) no traffic passes whatsoever. Something is definitely seriously broken in recent builds, could it be something that is just one thing that relates to the parsing of firewall rules? Sadly, I have no way to tell when it broke since I hadn't upgraded in 60 days and when I did, I wiped the box and started over.



  • If you refuse to provide feedback i will just mark you as a troll an lock this subject.

    Decide on your own if you want to continue and give feedback or just troll around.



  • @ermal:

    If you refuse to provide feedback i will just mark you as a troll an lock this subject.

    Decide on your own if you want to continue and give feedback or just troll around.

    What on earth are you talking about? I said above - given what's going on, what would be most useful? I'm happy to send you anything you need to help troubleshoot!



  • I seem to have got my limiters working again, it seems that was an unrelated thing (I forgot limiters have NEEDED defined schedules for quite a long time now, forgot why they do).

    What files would be useful to help determine why I'm needed extra rules to communicate between VLAN's? I'd be happy to send anything you need or provide access to the system if I'm not able to get it.



  • @markuhde:

    What files would be useful to help determine why I'm needed extra rules to communicate between VLAN's? I'd be happy to send anything you need or provide access to the system if I'm not able to get it.

    Have you already provided the requested information:
    @ermal:

    Can you post your /tmp/rules.debug and screenshots of the gui(or config.xml) to see what is happening htere?

    If so, where?



  • No I haven't because the forum doesn't let you post files with those extensions. I'd be happy to email those files to Ermal (or other devs) though!

    Edit, I just tried posting screenshots of my rules but the forum just timed out and didn't let me post those either. Today, I'm having issues connecting to one of my WAN interfaces but that may simply be my Internet connection at work blocking it - I have a HORRIBLE Internet connection here.



  • @markuhde:

    No I haven't because the forum doesn't let you post files with those extensions.

    It won't stop you posting the contents of the requested file(s) within a reply (within a quote block or code block to distinguish the contents from commentary).



  • @wallabybob:

    @markuhde:

    No I haven't because the forum doesn't let you post files with those extensions.

    It won't stop you posting the contents of the requested file(s) within a reply (within a quote block or code block to distinguish the contents from commentary).

    Good point I didn't want to flood the forum but if that's the best way to get those to you guys here they are. Also, not being able to access one of my WAN interfaces was just, as I suspected, my work blocking the IP address. So the only issue on-topic to this thread seems to be the communication between VLAN's, except, instead of being unable to block traffic I had to specifically allow it on the other interface. Here are the files Ermal requested:

    rules.debug:

    set limit tables 3000
    set optimization aggressive
    set timeout { adaptive.start 0, adaptive.end 0 }
    set limit states 612000
    set limit src-nodes 612000
    
    #System aliases
    
    loopback = "{ lo0 }"
    MONTANADIGITAL = "{ em1 }"
    STAFFNETWORK = "{ em0_vlan69 }"
    GUESTNETWORK = "{ em0_vlan15 }"
    SOUNDTECH = "{ em0_vlan763 }"
    CENTURYLINK = "{ pppoe0 }"
    OpenVPN = "{ openvpn }"
    
    #SSH Lockout Table
    table <sshlockout>persist
    table <webconfiguratorlockout>persist
    #Snort tables
    table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
    table <vpn_networks>{ 172.21.16.0/24 }
    table <negate_networks>{ 172.21.16.0/24 }
    
    # User Aliases 
    table <badstuff>persist
    badstuff = "<badstuff>"
    blockedports = "{   31337  27374  6969  6881:6889 }"
    fastbucket = "{   443  53 }"
    midbucket = "{   5222:5223  5228:5230 }"
    slowbucket = "{   5050  5190  1863  6665:6669  6679:6697  5631:5632  5800  6891:6901  1725  2302  3074  88  3724  6112  20:22  110  143  465  587  993  995 }"
    
    # Gateways
    GWWANGW = " route-to ( em1 72.250.187.1 ) "
    GWCENTURYLINK_PPPOE = " route-to ( pppoe0 72.160.61.1 ) "
    GWbothDSL = "  route-to { ( em1 72.250.187.1 ) ( pppoe0 72.160.61.1 )  }  round-robin  "
    GWCenturyLinkPreferred = "  route-to { ( pppoe0 72.160.61.1 )  }  "
    
    set loginterface em0_vlan69
    
    set skip on pfsync0
    
    scrub on $MONTANADIGITAL all    fragment reassemble
    scrub on $STAFFNETWORK all    fragment reassemble
    scrub on $GUESTNETWORK all    fragment reassemble
    scrub on $SOUNDTECH all    fragment reassemble
    scrub on $CENTURYLINK all    fragment reassemble
    
    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/*"
    nat-anchor "natrules/*"
    
    # Outbound NAT rules
    
    # Subnets to NAT 
    tonatsubnets	= "{ 172.21.12.0/22 172.16.16.0/22 192.168.41.0/24 172.21.16.0/24 127.0.0.0/8 0.0.0.0  }"
    nat on $MONTANADIGITAL  from $tonatsubnets port 500 to any port 500 -> 72.250.187.21/32 port 500  
    nat on $MONTANADIGITAL  from $tonatsubnets to any -> 72.250.187.21/32 port 1024:65535  
    
    nat on $CENTURYLINK  from $tonatsubnets port 500 to any port 500 -> 72.160.61.87/32 port 500  
    nat on $CENTURYLINK  from $tonatsubnets to any -> 72.160.61.87/32 port 1024:65535  
    
    # Load balancing anchor
    rdr-anchor "relayd/*"
    # TFTP proxy
    rdr-anchor "tftp-proxy/*"
    # NAT Inbound Redirects
    rdr on em1 proto tcp from any to 72.250.187.21 port 8034 -> 172.21.12.34 port 80
    
    # Setup Squid proxy redirect
    no rdr on em0_vlan15 proto tcp from any to { 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 } port 80
    no rdr on em0_vlan15 proto tcp from any to { 172.21.12.0/22, 172.16.16.0/22, 192.168.41.0/24 } port 80
    rdr on em0_vlan15 proto tcp from any to !(em0_vlan15) port 80 -> 127.0.0.1 port 3128
    
    # UPnPd rdr anchor
    rdr-anchor "miniupnpd"
    
    anchor "relayd/*"
    anchor "openvpn/*"
    anchor "ipsec/*"
    # Block all IPv6
    block in log quick inet6 all label "Block all IPv6"
    block out log quick inet6 all label "Block all IPv6"
    #---------------------------------------------------------------------------
    # default deny rules
    #---------------------------------------------------------------------------
    block in log inet all label "Default deny rule IPv4"
    block out log inet all label "Default deny rule IPv4"
    block in log inet6 all label "Default deny rule IPv6"
    block out log inet6 all label "Default deny rule IPv6"
    
    # IPv6 ICMP is not auxilary, it is required for operation
    # See man icmp6(4)
    # 1    unreach         Destination unreachable
    # 2    toobig          Packet too big
    # 128  echoreq         Echo service request
    # 129  echorep         Echo service reply
    # 133  routersol       Router solicitation
    # 134  routeradv       Router advertisement
    # 135  neighbrsol      Neighbor solicitation
    # 136  neighbradv      Neighbor advertisement
    pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state
    
    # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state
    
    # We use the mighty pf, we cannot be fooled.
    block quick inet proto { tcp, udp } from any port = 0 to any
    block quick inet proto { tcp, udp } from any to any port = 0
    block quick inet6 proto { tcp, udp } from any port = 0 to any
    block quick inet6 proto { tcp, udp } from any to any port = 0
    
    # Snort package
    block quick from <snort2c>to any label "Block snort2c hosts"
    block quick from any to <snort2c>label "Block snort2c hosts"
    
    # SSH lockout
    block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"
    
    # webConfigurator lockout
    block in log quick proto tcp from <webconfiguratorlockout>to any port 443 label "webConfiguratorlockout"
    block in quick from <virusprot>to any label "virusprot overload table"
    antispoof for em1
    antispoof for em0_vlan69
    # allow access to DHCP server on STAFFNETWORK
    pass in quick on $STAFFNETWORK proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $STAFFNETWORK proto udp from any port = 68 to 172.21.12.1 port = 67 label "allow access to DHCP server"
    pass out quick on $STAFFNETWORK proto udp from 172.21.12.1 port = 67 to any port = 68 label "allow access to DHCP server"
    antispoof for em0_vlan15
    # allow access to DHCP server on GUESTNETWORK
    pass in quick on $GUESTNETWORK proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $GUESTNETWORK proto udp from any port = 68 to 172.16.16.1 port = 67 label "allow access to DHCP server"
    pass out quick on $GUESTNETWORK proto udp from 172.16.16.1 port = 67 to any port = 68 label "allow access to DHCP server"
    antispoof for em0_vlan763
    # allow access to DHCP server on SOUNDTECH
    pass in quick on $SOUNDTECH proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $SOUNDTECH proto udp from any port = 68 to 192.168.41.1 port = 67 label "allow access to DHCP server"
    pass out quick on $SOUNDTECH proto udp from 192.168.41.1 port = 67 to any port = 68 label "allow access to DHCP server"
    antispoof for pppoe0
    
    # loopback
    pass in on $loopback inet all label "pass IPv4 loopback"
    pass out on $loopback inet all label "pass IPv4 loopback"
    pass in on $loopback inet6 all label "pass IPv6 loopback"
    pass out on $loopback inet6 all label "pass IPv6 loopback"
    # let out anything from the firewall host itself and decrypted IPsec traffic
    pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
    pass out route-to ( em1 72.250.187.1 ) from 72.250.187.21 to !72.250.187.0/24 keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to ( pppoe0 72.160.61.1 ) from 72.160.61.87 to !72.160.61.87/32 keep state allow-opts label "let out anything from firewall host itself"
    # make sure the user cannot lock himself out of the webConfigurator or SSH
    pass in quick on em0_vlan69 proto tcp from any to (em0_vlan69) port { 443 80 22 } keep state label "anti-lockout rule"
    
    # User-defined rules follow
    
    anchor "userrules/*"
    pass  in  quick  on $OpenVPN  from any to any keep state  label "USER_RULE: OpenVPN Tech Remotes wizard"
    pass  in  quick  on $MONTANADIGITAL reply-to ( em1 72.250.187.1 ) inet proto tcp  from any to any port 443 flags S/SA keep state  label "USER_RULE"
    pass  in  quick  on $MONTANADIGITAL reply-to ( em1 72.250.187.1 ) inet proto tcp  from any to any port 22 flags S/SA keep state  label "USER_RULE"
    pass  in  quick  on $MONTANADIGITAL reply-to ( em1 72.250.187.1 ) inet proto { tcp udp }  from any to any port 1723 keep state  label "USER_RULE"
    pass  in  quick  on $MONTANADIGITAL reply-to ( em1 72.250.187.1 )  proto udp  from any to 72.250.187.21 port 1194 keep state  label "USER_RULE: OpenVPN Tech Remotes wizard"
    block  in  quick  on $STAFFNETWORK inet proto { tcp udp }  from any to any port $blockedports  label "USER_RULE"
    pass  in  quick  on $STAFFNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 1,2)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $STAFFNETWORK  $GWbothDSL inet proto { tcp udp }  from any to any port $fastbucket keep state  dnpipe ( 1,2)  label "USER_RULE"
    pass  in  quick  on $STAFFNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 3,4)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $STAFFNETWORK  $GWbothDSL inet proto { tcp udp }  from any to any port $midbucket keep state  dnpipe ( 3,4)  label "USER_RULE"
    pass  in  quick  on $STAFFNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 5,6)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $STAFFNETWORK  $GWCenturyLinkPreferred inet proto { tcp udp }  from any to any port $slowbucket keep state  dnpipe ( 5,6)  label "USER_RULE"
    pass  in  quick  on $STAFFNETWORK inet from 172.21.12.0/22  to <negate_networks>keep state  dnpipe ( 1,2)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $STAFFNETWORK  $GWbothDSL inet from 172.21.12.0/22 to any keep state  dnpipe ( 1,2)  label "USER_RULE: Default allow LAN to any rule"
    pass  in  quick  on $STAFFNETWORK  $GWbothDSL inet from any to 172.21.12.0/22 keep state  label "USER_RULE: allow any to staff network"
    pass  in  quick  on $GUESTNETWORK inet proto { tcp udp }  from any to 172.21.12.0/22 keep state  label "USER_RULE"
    pass  in  quick  on $GUESTNETWORK inet proto { tcp udp }  from any to 172.16.16.1/22 keep state  label "USER_RULE"
    pass  in  quick  on $GUESTNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 3,4)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $GUESTNETWORK  $GWbothDSL inet proto { tcp udp }  from any to any port $fastbucket keep state  dnpipe ( 3,4)  label "USER_RULE"
    pass  in  quick  on $GUESTNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 5,6)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $GUESTNETWORK  $GWbothDSL inet proto { tcp udp }  from any to any port $midbucket keep state  dnpipe ( 5,6)  label "USER_RULE"
    pass  in  quick  on $GUESTNETWORK inet proto { tcp udp }  from any  to <negate_networks>keep state  dnpipe ( 7,8)  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in  quick  on $GUESTNETWORK  $GWCenturyLinkPreferred inet proto { tcp udp }  from any to any port $slowbucket keep state  dnpipe ( 7,8)  label "USER_RULE"
    pass  in  quick  on $SOUNDTECH inet proto tcp  from any to 172.21.12.0/22 flags S/SA keep state  label "USER_RULE"
    pass  in  quick  on $SOUNDTECH inet proto { tcp udp }  from any to 192.168.41.1/24 keep state  label "USER_RULE"
    pass  in  quick  on $SOUNDTECH inet proto { tcp udp }  from any to any keep state  dnpipe ( 1,2)  label "USER_RULE"
    pass  in  quick  on $CENTURYLINK reply-to ( pppoe0 72.160.61.1 ) inet proto tcp  from any to any port 22 flags S/SA keep state  label "USER_RULE"
    pass  in  quick  on $CENTURYLINK reply-to ( pppoe0 72.160.61.1 ) inet proto tcp  from any to any port 443 flags S/SA keep state  label "USER_RULE"
    pass  in  quick  on $CENTURYLINK reply-to ( pppoe0 72.160.61.1 ) inet proto { tcp udp }  from any to any port 1723 keep state  label "USER_RULE"
    
    # Automatic Pass rules for any delegated IPv6 prefixes through dynamic IPv6 clients
    
    # VPN Rules
    anchor "tftp-proxy/*"
    
    # Setup squid pass rules for proxy
    pass in quick on em0_vlan15 proto tcp from any to !(em0_vlan15) port 80 flags S/SA keep state
    pass in quick on em0_vlan15 proto tcp from any to !(em0_vlan15) port 3128 flags S/SA keep state</negate_networks></negate_networks></negate_networks></negate_networks></negate_networks></negate_networks></negate_networks></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></badstuff></badstuff></negate_networks></vpn_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> 
    

    (I'm at the size limit I guess my post got cut short so I'll post config.xml in the next post)



  • Use sites like pastie.org etc for posting text data



  • @ermal:

    Use sites like pastie.org etc for posting text data

    I tried that and config.xml was over their 64 K limit but here's a link to it as a document on Google Drive. Hopefully this is helpful:

    https://docs.google.com/document/d/13gilNZkUWH0sBi39visLSX4QUC5PlzmdF7-vraIwa6k/edit?usp=sharing



  • Could someone post their netstat -rn output on a box with a working OpenVPN config?

    Edit: nevermind, found out one of my switches' configs got changed by someone who doesn't like to inform other people why they're doing stupid stuff, so my route was gone. I am not having the VPN/firewall issue.



  • @timthetortoise:

    Could someone post their netstat -rn output on a box with a working OpenVPN config?

    Edit: nevermind, found out one of my switches' configs got changed by someone who doesn't like to inform other people why they're doing stupid stuff, so my route was gone. I am not having the VPN/firewall issue.

    That's great, I'm wondering if my issue has to do with the cheap MicroTik switch the firewall is connected to. I was successfully talking from OpenVPN to LAN and now it's quit working again. Here is my netstat -rn if it's useful:

    [2.1-BETA1][admin@glacierfire.glaciercamp]/root(1): netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            72.250.187.1       UGS         0   684815    em1
    72.160.38.1        link#11            UH          0    54778 pppoe0
    72.160.38.69       link#11            UHS         0        0    lo0
    72.250.187.0/24    link#3             U           0   113694    em1
    72.250.187.21      link#3             UHS         0        0    lo0
    127.0.0.1          link#6             UH          0     1363    lo0
    172.16.16.0/22     link#9             U           0  1520856 em0_vl
    172.16.16.1        link#9             UHS         0        0    lo0
    172.21.12.0/22     link#8             U           0   311550 em0_vl
    172.21.12.1        link#8             UHS         0        0    lo0
    172.21.16.0/24     172.21.16.2        UGS         0    15315 ovpns1
    172.21.16.1        link#12            UHS         0        0    lo0
    172.21.16.2        link#12            UH          0        0 ovpns1
    192.168.41.0/24    link#10            U           0       33 em0_vl
    192.168.41.1       link#10            UHS         0        0    lo0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               ::1                           UH          lo0
    fe80::%re0/64                     link#1                        U           re0
    fe80::d227:88ff:feba:d2e%re0      link#1                        UHS         lo0
    fe80::%em0/64                     link#2                        U           em0
    fe80::204:23ff:feac:cf84%em0      link#2                        UHS         lo0
    fe80::%em1/64                     link#3                        U           em1
    fe80::204:23ff:feac:cf85%em1      link#3                        UHS         lo0
    fe80::%lo0/64                     link#6                        U           lo0
    fe80::1%lo0                       link#6                        UHS         lo0
    fe80::%em0_vlan69/64              link#8                        U      em0_vlan
    fe80::d227:88ff:feba:d2e%em0_vlan69 link#8                        UHS         lo0
    fe80::%em0_vlan15/64              link#9                        U      em0_vlan
    fe80::d227:88ff:feba:d2e%em0_vlan15 link#9                        UHS         lo0
    fe80::%em0_vlan763/64             link#10                       U      em0_vlan
    fe80::d227:88ff:feba:d2e%em0_vlan763 link#10                       UHS         lo0
    fe80::%pppoe0/64                  link#11                       U        pppoe0
    fe80::d227:88ff:feba:d2e%pppoe0   link#11                       UHS         lo0
    fe80::%ovpns1/64                  link#12                       U        ovpns1
    fe80::d227:88ff:feba:d2e%ovpns1   link#12                       UHS         lo0
    ff01::%re0/32                     fe80::d227:88ff:feba:d2e%re0  U           re0
    ff01::%em0/32                     fe80::204:23ff:feac:cf84%em0  U           em0
    ff01::%em1/32                     fe80::204:23ff:feac:cf85%em1  U           em1
    ff01::%lo0/32                     ::1                           U           lo0
    ff01::%em0_vlan69/32              fe80::d227:88ff:feba:d2e%em0_vlan69 U      em0_vlan
    ff01::%em0_vlan15/32              fe80::d227:88ff:feba:d2e%em0_vlan15 U      em0_vlan
    ff01::%em0_vlan763/32             fe80::d227:88ff:feba:d2e%em0_vlan763 U      em0_vlan
    ff01::%pppoe0/32                  fe80::d227:88ff:feba:d2e%pppoe0 U        pppoe0
    ff01::%ovpns1/32                  fe80::d227:88ff:feba:d2e%ovpns1 U        ovpns1
    ff02::%re0/32                     fe80::d227:88ff:feba:d2e%re0  U           re0
    ff02::%em0/32                     fe80::204:23ff:feac:cf84%em0  U           em0
    ff02::%em1/32                     fe80::204:23ff:feac:cf85%em1  U           em1
    ff02::%lo0/32                     ::1                           U           lo0
    ff02::%em0_vlan69/32              fe80::d227:88ff:feba:d2e%em0_vlan69 U      em0_vlan
    ff02::%em0_vlan15/32              fe80::d227:88ff:feba:d2e%em0_vlan15 U      em0_vlan
    ff02::%em0_vlan763/32             fe80::d227:88ff:feba:d2e%em0_vlan763 U      em0_vlan
    ff02::%pppoe0/32                  fe80::d227:88ff:feba:d2e%pppoe0 U        pppoe0
    ff02::%ovpns1/32                  fe80::d227:88ff:feba:d2e%ovpns1 U        ovpns1
    
    


  • Can you run wireshark or tcpdump on the machine you're trying to get traffic to to verify that it's not getting through? Or do you know for a fact that none of the traffic is touching it?



  • @timthetortoise:

    Can you run wireshark or tcpdump on the machine you're trying to get traffic to to verify that it's not getting through? Or do you know for a fact that none of the traffic is touching it?

    Sadly I can't, since every "machine" on the network on a regular basis I control is a Wi-Fi AP or a small smart switch. I could, next time I'm on site, put a machine on it running wireshark and attempt to VPN in from my iPad… there's an OVPN iPad client now. I'll do that next time I'm on site if I get a chance to.



  • I honestly can't find anything wrong with your firewall or routing tables. Could you post a simplified version of your topology when you get a chance? If you have an intermediary L3 device, I'm thinking your issue may be similar to what was going on with me, but I don't believe at this point that it's pf's fault.



  • Okay here's the basic text description:

    WAN 1 is Montana Digital (mtdig.net) 72.250.187.21
    WAN 2 is CenturyLink PPPoE
    both feed into pfSense
    Local side of pfSense is one port - three VLAN's. VLAN 69 - 172.21.12.1/22, VLAN 15 - 172.16.16.1/22, VLAN 763 - 192.168.41.1/24

    This feeds a MikroTik RB 250/GS 5-port switch.
    One of these ports feeds a Raspberry Pi (and is a VLAN 69 access port) for RADIUS
    The rest are trunk ports of which two are used (one free port):
    One feeds a Trendnet PoE 8-port switch which feeds an EnGenius ENH-200, a Trendnet TEW-653AP and a Buffalo WHR-HP-G300N

    One feeds another building which has a small 8-port ASUS Gigabit Unmanaged Switch which in turn feeds:
    A EnGenius EAP-600
    A Buffalo WHR-HP-G300N
    An EnGenius ENH-202
    An Ubiquiti Nanostation M5 in WDS-AP mode

    The Nanostation M5 has three clients in WDS-STA mode, which feed:

    Nanostation Loco M5 #1:
    Another ASUS unmanaged switch connected to:
    EnGenius ENH-202
    EnGenius ENH-500
    EnGenius EAP-300

    Nanostation Loco M5 #2:
    Another MikroTik RB/250GS feeding:
    Trunk: Trendnet PoE switch connected to an EnGenius EAP600 (more devices to come, LOL)
    Access 763: Audio DSP (forgot the make/model)
    Access 763: Buffalo WHR-HP-G300N

    Nanostation Loco M5 #3:
    Buffalo WHR-HP-G300N

    All except the AP for the sound techs are on receiving all VLAN's and set to management on VLAN 69. This has all worked very well historically, I reinstalled pfSense and upgraded from a snap several months old when I needed to add a new NIC to support the second  WAN connection. (which itself is a pain, I didn't know Squid didn't load balance so I'm trying to learn how to do that, but that's unrelated obviously)



  • No intermediary L3 devices, can't get traffic that originates on the VPN side through, I'm stumped. Unless your default gateway is wrong on your machines (I'm assuming it's the pfSense IP for each respective VLAN), I can't find a reason your configuration isn't working without knowing your network/config more in-depth. Good luck with it!



  • @timthetortoise:

    No intermediary L3 devices, can't get traffic that originates on the VPN side through, I'm stumped. Unless your default gateway is wrong on your machines (I'm assuming it's the pfSense IP for each respective VLAN), I can't find a reason your configuration isn't working without knowing your network/config more in-depth. Good luck with it!

    Default gateway is the pfSense IP for VLAN_69 on everything. I think I've tracked down the problem as an OpenVPN issue though more than a firewall issue. All that testing was with the OpenVPN client for Windows which had been working and suddenly stopped with no config changes (couldn't get traffic to anything). I just installed the client for iPad and for Android. Lots of random disconnects, especially on the Android version, but both 95% work - definitely enough to be usable for my purposes. I changed NOTHING on the Windows client from when it was working though. I sure wish the OpenVPN setup on pfSense worked with Fedora (it doesn't even let you try because there's no certificate password).

    So, it seems the only issue I'm having that's actually possibly VLAN caused (I still need to figure out load balancing Squid but that's obviously a topic for another thread) is need for the seemingly redundant allow from any rule on the LAN side (since as said, established states should just be allowed).

    Thanks for the good wishes on sorting this out :) I really am trying to be as useful as I can be with this.



  • Can you make a network diagram? gliffy.com . Are you seeing the mac address of the default gateways on the various subnets? (arp -a in windows will show you your arp table) Not sure how to do it in FreeBSD. I'm thinking that you might have a switching loop on your network. Also are your trunk ports tagged vlan ports. Are you sending tagged traffic to a end device? Most computers and access points will drop tagged traffic unless they are configured to deal with the extra 4 bytes. I like a puzzle, looking forward to seeing your diagram.



  • @mikeisfly:

    Can you make a network diagram? gliffy.com . Are you seeing the mac address of the default gateways on the various subnets? (arp -a in windows will show you your arp table) Not sure how to do it in FreeBSD. I'm thinking that you might have a switching loop on your network. Also are your trunk ports tagged vlan ports. Are you sending tagged traffic to a end device? Most computers and access points will drop tagged traffic unless they are configured to deal with the extra 4 bytes. I like a puzzle, looking forward to seeing your diagram.

    I'm adding a few AP's and a couple more little MikroTik switches to create access ports NEXT weekend but as of today, here's how it looks:

    http://www.gliffy.com/go/publish/4535154/

    The access points are configured to deal with tagged traffic as we run separate SSID's on each VLAN. Nobody is plugging into a trunk port directly (nobody gets to come in and plug into ANY port anymore for that matter since the switches that accomplished that broke)

    P.S. As I noted this same basic setup used to work perfectly fine without any "extra" rules. Thanks a ton for your help! It seems to be mostly working now, I think all my OpenVPN issues are actually my internet connection where I live.



  • Okay if you said it's working then I will leave enough alone. I know you said it worked before but from your diagram you are using unmanaged switches with shouldn't pass tagged traffic. No if they are on access ports assigned to specific vlans then there should be no problem there. However if you send tagged traffic into a access port most unmanaged switches I would think would drop it. Just something to look at. Just wanted to offer my input, let me know if I could be of any further assistance? By the way I have a EnGenius EAP 600 Access Point in my house, that is a sweet piece of gear.



  • @mikeisfly:

    Okay if you said it's working then I will leave enough alone. I know you said it worked before but from your diagram you are using unmanaged switches with shouldn't pass tagged traffic. No if they are on access ports assigned to specific vlans then there should be no problem there. However if you send tagged traffic into a access port most unmanaged switches I would think would drop it. Just something to look at. Just wanted to offer my input, let me know if I could be of any further assistance? By the way I have a EnGenius EAP 600 Access Point in my house, that is a sweet piece of gear.

    MOST unmanaged switches pass tagged traffic just fine. They don't do anything with the tags, nor do they need to in this application. Some will drop tagged frames, but they generally don't - and shouldn't. They aren't smart enough to know what the tags are, and if behaving properly they just ignore them.

    It seems to be working okay now, though I had to roll back last night's pfSense snap - I made a new thread for that!

    Yes, the EAP600 is very nice, I was a beta tester for them and they're fantastic for the price. They've got a removable-antenna version the ECB600 coming soon.



  • This affects this particular install so little, I haven't had time to worry about it - but I still can't get traffic between interfaces at all most of the time (sometimes it works, the extra rule had nothing to do with it, it's actually rather random, and hasn't worked in weeks). I'm on the latest snapshot tonight and… nothing. Firewall to any address is fine. Any address to another interface's firewall address is fine. Any LAN interface out to the Internet is fine. But pinging a machine on another LAN interface or connecting to it in any way? Just silently blocked (it doesn't show up in the firewall logs, it just doesn't work).

    I'm at a total loss for why - my rules should definitely be allowing this traffic to the best of my understanding.


Log in to reply