Bandwidth limiting not working: Is this a bug, or am I overlooking something?



  • Right now, I'm not sure if I'm being dense and overlook something really silly and fundamental, or if this is a bug.
    Here's the situation: There's one physical interface connected to the internet, the WAN link. That WAN link carries a tunnelbroker interfacec for IPv6 traffic (all of which is routed through that tunnel) and an IPv4 IPSec link, over which all IPv4 traffic is routed to the public interface. Thus the WAN traffic should always be WAN6 traffic plus IPSec traffic plus protocol overhead.

    The WAN interface should be bandwidth limited to 15Mbit/s, and for good measure, I also have a firewall rule that limits the bandwidth of ESP triaffic bandwidth from the WAN interface IP to the IPSec remote endpoint IP address to 15Mbit/s.
    So despite of what I think should be TWO bottlenecks in series that each one of them should make sure that traffic on the WAN/IPSec interface should not go above 15Mbit/s, I end up with bandwidth exceeding that by a factor of two…

    (...until something breaks the IPSec link, an issue that I have raised in a separate thread: http://forum.pfsense.org/index.php/topic,64603.0.html

    Can someone please point out to me what I'm doing wrong, if anything or help me confirm this is a bug, which ever is applicable?

    And before someone asks which version/build:

    Version 2.1-RC0 (amd64)
    built on Mon Jul 22 03:26:48 EDT 2013
    FreeBSD 8.3-RELEASE-p8

    PS: bug in the BBS software: if a post fails because the attachments take up too much storage, and then one tries to resubmit the posting with changed attachments, it claims it's a duplicate post, and throws an error, even though double-checking reveals the post has not been made….
    ![Traffic Graph 2013-07-24 at 02.56.26.png](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 02.56.26.png)
    ![Traffic Graph 2013-07-24 at 02.56.26.png_thumb](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 02.56.26.png_thumb)
    ![Traffic Graph 2013-07-24 at 03.00.26.png](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 03.00.26.png)
    ![Traffic Graph 2013-07-24 at 03.00.26.png_thumb](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 03.00.26.png_thumb)
    ![Traffic Graph 2013-07-24 at 03.05.24.png](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 03.05.24.png)
    ![Traffic Graph 2013-07-24 at 03.05.24.png_thumb](/public/imported_attachments/1/Traffic Graph 2013-07-24 at 03.05.24.png_thumb)



  • Here traffic shaping on the WAN interface:

    ![Traffic Shaper.png](/public/imported_attachments/1/Traffic Shaper.png)
    ![Traffic Shaper.png_thumb](/public/imported_attachments/1/Traffic Shaper.png_thumb)



  • The firewall rules and the corresponding limiters:

    ![Limiter Rules.png](/public/imported_attachments/1/Limiter Rules.png)
    ![Limiter Rules.png_thumb](/public/imported_attachments/1/Limiter Rules.png_thumb)
    ![Limiter Out.png](/public/imported_attachments/1/Limiter Out.png)
    ![Limiter Out.png_thumb](/public/imported_attachments/1/Limiter Out.png_thumb)
    ![Limiter In.png](/public/imported_attachments/1/Limiter In.png)
    ![Limiter In.png_thumb](/public/imported_attachments/1/Limiter In.png_thumb)



  • and lastly the details of the first of the firewall rules:

    ![Limiter Rule 1.png](/public/imported_attachments/1/Limiter Rule 1.png)
    ![Limiter Rule 1.png_thumb](/public/imported_attachments/1/Limiter Rule 1.png_thumb)



  • and lastly the details of the second of the firewall rules:

    ![Limiter Rule 2.png](/public/imported_attachments/1/Limiter Rule 2.png)
    ![Limiter Rule 2.png_thumb](/public/imported_attachments/1/Limiter Rule 2.png_thumb)



  • Interface firewall rules apply to traffic arriving at the interface. Your second firewall rule (source=WAN address) won't apply because you shouldn't have traffic arriving at the WAN interface with source address = WAN address.

    I suspect something like that rule is probably needed as a Floating rule (not an interface rule), quick, direction=out.



  • @wallabybob:

    Interface firewall rules apply to traffic arriving at the interface. Your second firewall rule (source=WAN address) won't apply because you shouldn't have traffic arriving at the WAN interface with source address = WAN address.

    Hm, OK. I can see this model work with rules for outgoing traffic being on the LAN interface and for incoming traffic being on the WAN interface. But the VPN is created ON THE PFSENSE BOX.
    So the only interface being ever involved is the WAN interface, both incoming and outgoing, hence I set it up this way. So this is a bit of a mind trick…

    @wallabybob:

    I suspect something like that rule is probably needed as a Floating rule (not an interface rule), quick, direction=out.

    Thanks for the suggestion. Which makes me wonder: Why do we need anything but the floating rules in the first place? There's a section in the floating rules editor where one can specify interfaces, directions, etc.
    So why not just simplify the entire pfSense system and program the entire firewall with floating rules? Or in other words: why would I put anything in an interface specific rule, when I can put the same thing in a floating rule with the interface(s) and direction specified?

    EDIT: still get throughput well over 20Mbit/s after following that suggestion :(

    The problem here is, in my mind, that the outgoing traffic I want to bandwidth limit is arriving on the LAN NIC, and goes out over the "IPSec NIC". So I could add a limiter to each and every LAN rule, but that might limit each type of traffic matched by the various rules, and not the total bandwidth.
    Or I could try to add limiting rules to the IPSec "interface", but what does traffic "arriving" at the IPSec interface mean? Arriving from the LAN, or from the WAN, or both?

    Oh, and I still don't get it why limiting the bandwidth of the WAN interface with the traffic shaper to 15Mbit/s doesn't work, or is that also only incoming traffic? Can't really limit the incoming traffic on the LAN interface, because that would also limit traffic going from the LAN to the DMZ or out the WAN6.

    I feel stupid right now…



  • This topic seems to have gone cool if not positively cold.

    Another topic I read recently got me thinking about this problem. The other topic was about some firewall rule attributes (e.g. maximum connections and connection rate) being applied only to TCP rules (protocol=TCP) and not rules that included TCP (e.g. protocol=ANY). Perhaps there is a problem (GUI limitation or technical) with applying limiters to non TCP traffic.

    Would it be "good enough" to apply a limiter (of say 14Mbps or a little less) to TCP traffic from LAN to public destinations?



  • @wallabybob:

    This topic seems to have gone cool if not positively cold.

    Another topic I read recently got me thinking about this problem. The other topic was about some firewall rule attributes (e.g. maximum connections and connection rate) being applied only to TCP rules (protocol=TCP) and not rules that included TCP (e.g. protocol=ANY). Perhaps there is a problem (GUI limitation or technical) with applying limiters to non TCP traffic.

    Would it be "good enough" to apply a limiter (of say 14Mbps or a little less) to TCP traffic from LAN to public destinations?

    Thanks for the suggestion, if I take IPv4 traffic only, that would likely do it, assuming e.g. Apple isn't using some non-TCP based protocol for AppStore software update downloads, or iTunes downloads (both of which quickly spike up to 30Mbps).
    Unfortunately, it doesn't seem to work, either. :(
    I made both a LAN and a floating rule to that effect, (source address LAN subnet, destination NOT LAN subnet, IPv4, TCP, plus of course the limiters) => still traffic going up to 30Mbps.

    Of course, while that's somewhat annoying, equally dubious is the fact that an IPSec connection would be that fragile. I mean, after all, this is the internet, so completely aside from my little ZyWall P1 being a bit overworked with this, all sorts of things could happen even if that weren't the case: dropped packets, slow links, etc.
    So for thinks to silently fail, remain "up", but no longer pass data, rather than either recovering, or going down and reestablishing the link, is a bit odd. Which makes me wonder if the IPSec spec is so vulnerable, or if there's an implementation bug either in the IPSec tools used by pfSense or in the ZyWall. On the latter, we of course have no influence, but the former, there would be a chance to fix things, if this whole scenario is the result of a bug.



  • @rcfa:

    Unfortunately, it doesn't seem to work, either. :(
    I made both a LAN and a floating rule to that effect, (source address LAN subnet, destination NOT LAN subnet, IPv4, TCP, plus of course the limiters) => still traffic going up to 30Mbps.

    Did you remember to reset states before trying your test (see Diagnostics -> States, click on Reset States)?

    Are you generating only IPv4 traffic for the test? (That firewall rule won't apply to any traffic going over the tunnel for IPv6.)



  • @wallabybob:

    @rcfa:

    Unfortunately, it doesn't seem to work, either. :(
    I made both a LAN and a floating rule to that effect, (source address LAN subnet, destination NOT LAN subnet, IPv4, TCP, plus of course the limiters) => still traffic going up to 30Mbps.

    Did you remember to reset states before trying your test (see Diagnostics -> States, click on Reset States)?

    Are you generating only IPv4 traffic for the test? (That firewall rule won't apply to any traffic going over the tunnel for IPv6.)

    Nope, didn't reset states, but I would have assumed that new downloads create new connections/states, since in this case it's not the VPN link itself that gets throttled, only the traffic going over it. But I can try again.
    As for only IPv4 traffic: maybe not, but that shouldn't matter, because IPv6 traffic should get routed over the tunnelbroker gateway, so it will use the same WAN port, but it won't use the IPSec link.
    Since I graph on the dashboard WAN, WAN6 and IPSec whereby WAN should always be the sum of the WAN6 and IPSec plus protocol overhead, it's easy to see if any significant part of the traffic would be IPv6, which wasn't the case.
    I guess I'll reset the states once, and try some other download, just for the sake of completeness…
    ...should against expectations things then no longer shoot well over the 15Mbit/s limit set, I'll report back the success. Not very optimistic, though.



  • Argh, brainfart on my side: what you suggest won't help. That limits the UPLOAD, but that's already "limited" by the speed of my FiOS link, which is limited to 15Mbit/s up, it's the download link which is "too fast" with 50Mbit/s, and my attempts are to limit the connection such as to be symmetrical.
    But the download link is ESP traffic on the WAN interface, which is what isn't being limited.

    I can try again adding limiting to the IPSec rules, but since limiting works on the "incoming" traffic of an interface, I'm not sure what's considered "incoming" in the case of IPSec: since the tunnel has two ends, one being the WAN the other being the LAN…
    ...I'll see if any limiters there will do something. If so, I'll report back.

    EDIT: No, nothing. As it stands I have yet to see any effect of limiting on anything.



  • I setup a limiter in pfSense a little while ago. It was on a pfSense 2.0.3 system and seemed to work fine. It restricted a single system on my LAN to 700kbps while other systems on the LAN could download at near full link speed (a bit over 2.4Mbps).

    If you post the output of the following commands I'll compare with my working system to see if I can spot a significant difference.```
    more /tmp/rules.limiter
    more /tmp/rules.limits
    more /tmp/rules.debug
    kldstat

    
    @rcfa:
    
    > I can try again adding limiting to the IPSec rules, but since limiting works on the "incoming" traffic of an interface, I'm not sure what's considered "incoming" in the case of IPSec: since the tunnel has two ends, one being the WAN the other being the LAN…
    > ...I'll see if any limiters there will do something. If so, I'll report back.
    
    I expected the firewall rule I suggested would have worked by limiting TCP traffic over over the IPSEC link. not by limiting the IPSEC traffic itself.
    
    In my rules.debug the first LAN rule is
    
    > pass  in  quick  on $LAN  proto { tcp udp }  from  $Luke to  ! $Privatesubnets keep state  dnpipe ( 3, 2)  label "USER_RULE: Traffic Limiting"
    
    which I'm guessing assigns matching connections to dummynet pipes 3 and 2\. These pipes appear to be setup by /tmp/rules.limiter:
    
    > pipe 1 config  bw 1000Kb mask src-ip 0xffffffff
    > pipe 2 config  bw 700Kb
    > pipe 3 config  bw 200Kb
    
    As I have been writing this up I realise that that the limiter I suggested would work only on connections initiated from the LAN side. In the load you have been running, are the connections initiated from the pfSense LAN side?


  • @wallabybob:

    As I have been writing this up I realise that that the limiter I suggested would work only on connections initiated from the LAN side. In the load you have been running, are the connections initiated from the pfSense LAN side?

    Most of the traffic is initiated on the LAN as in "I click the download button on the LAN side", but of course, the bulk of the data comes in from the internet, and that incoming direction is the fat pipe that needs a throttle to slow it down so it doesn't overwhelm the poor little ZyWALL on the far end of the IPSec tunnel.

    Here the requested output, only minimally altered by regex replacing the LAN addresses, etc.

    #/root(5): cat /tmp/rules.limiter
    
    pipe 1 config  bw 15Mb burst 15Mb
    
    pipe 2 config  bw 15Mb burst 15Mb
    
    pipe 3 config  bw 15Mb burst 15Mb mask src-ip6 /128 src-ip 0xffffffff
    
    pipe 4 config  bw 15Mb burst 15Mb mask dst-ip6 /128 dst-ip 0xffffffff
    
    #/root(6): cat /tmp/rules.limits
    set limit tables 3000
    set limit table-entries 400000
    set optimization normal
    set timeout { adaptive.start 0, adaptive.end 0 }
    set limit states 512000
    set limit src-nodes 512000
    #/root(7): cat /tmp/rules.debug
    set limit tables 3000
    set limit table-entries 400000
    set optimization normal
    set timeout { adaptive.start 0, adaptive.end 0 }
    set limit states 512000
    set limit src-nodes 512000
    
    #System aliases
    
    loopback = "{ lo0 }"
    WAN = "{ em0 }"
    LAN = "{ lagg1 }"
    DMZ = "{ lagg2 }"
    WAN6 = "{ gif0 }"
    pptp = "{ pptp }"
    IPsec = "{ enc0 }"
    WANGRP = "{ WANGRP }"
    
    #SSH Lockout Table
    table <sshlockout>persist
    table <webconfiguratorlockout>persist
    #Snort tables
    table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
    table <bogonsv6>persist file "/etc/bogonsv6"
    table <negate_networks># User Aliases 
    table <pfblockerblocklistde>persist file "/var/db/aliastables/pfBlockerblocklistde.txt"
    pfBlockerblocklistde = "<pfblockerblocklistde>"
    table <anveosip>persist
    AnveoSIP = "<anveosip>"
    table <easyruleblockhostsenc0>{   219.159.184.54/32  184.82.107.116/32  139.194.105.5/32  95.132.52.161/32  60.173.11.204/32  122.227.98.206/32  37.139.2.18/32  58.221.60.156/32  64.118.75.20/32  87.241.219.147/32  199.85.205.94/32  188.190.98.6/32  166.111.7.196/32  212.102.17.153/32  184.22.190.62/32  184.82.28.76/32  184.22.120.206/32  198.20.69.74/32  62.75.130.185/32  188.95.234.6/32  210.31.177.197/32 } 
    EasyRuleBlockHostsENC0 = "<easyruleblockhostsenc0>"
    table <easyruleblockhostsopt2>{   2607:f8b0:400e:c02::6c/128 } 
    EasyRuleBlockHostsOPT2 = "<easyruleblockhostsopt2>"
    table <easyruleblockhostswan>{   122.166.51.37/32  188.190.98.6/32  17.158.8.89/32  205.188.170.15/32  64.12.95.72/32  173.194.79.108/32  205.188.155.221/32  205.188.170.12/32  64.12.95.69/32  62.75.130.185/32  93.157.98.204/32  93.82.255.94/32 } 
    EasyRuleBlockHostsWAN = "<easyruleblockhostswan>"
    table <ipv4tunnelremote>{   111.222.333.444 } 
    IPv4TunnelRemote = "<ipv4tunnelremote>"
    table <ipv6gateways>{   209.51.161.14  2001:470:1f06:356::1  2001:470:1f06:356::2 } 
    IPv6Gateways = "<ipv6gateways>"
    table <ipv6tunnelremote>{   209.51.161.14 } 
    IPv6TunnelRemote = "<ipv6tunnelremote>"
    SIP_ports = "{   5060  5061  5010 }"
    table <snortwhitelist>{   209.51.161.14  72.9.149.69  67.212.84.21  176.9.39.206 } 
    SnortWhitelist = "<snortwhitelist>"
    
    # Gateways
    GWWANGW = " route-to ( em0 96.253.50.1 ) "
    GWWAN6GW = " route-to ( gif0 2001:470:1f06:356::1 ) "
    
    set loginterface lagg1
    
    set skip on pfsync0
    
    scrub from any to <vpn_networks>max-mss 1400
    scrub on $WAN all  random-id  fragment reassemble
    scrub on $LAN all  random-id  fragment reassemble
    scrub on $DMZ all  random-id  fragment reassemble
    scrub on $WAN6 all  random-id  fragment reassemble
    
     altq on  em0 hfsc bandwidth 15Mb queue {  qACK,  qDefault,  qVoIP,  qOthersHigh,  qOthersLow  } 
     queue qACK on em0 bandwidth 19.666% hfsc (  ecn  , linkshare 19.666%  )  
     queue qDefault on em0 bandwidth 9.833% hfsc (  ecn  , default  )  
     queue qVoIP on em0 bandwidth 32Kb hfsc (  ecn  ,  realtime 256Kb )  
     queue qOthersHigh on em0 bandwidth 9.833% hfsc (  ecn  , linkshare 9.833%  )  
     queue qOthersLow on em0 bandwidth 4.9165% hfsc (  ecn  , linkshare 4.9165%  )  
    
     altq on  lagg1 hfsc bandwidth 15Mb queue {  qLink,  qInternet  } 
     queue qLink on lagg1 bandwidth 20% qlimit 500 hfsc (  ecn  , default  )  
     queue qInternet on lagg1 bandwidth 15728.64Kb hfsc (  ecn  , linkshare 15728.64Kb  , upperlimit 15728.64Kb  )  {  qACK,  qVoIP,  qOthersHigh,  qOthersLow  } 
     queue qACK on lagg1 bandwidth 19.96% hfsc (  ecn  , linkshare 19.96%  )  
     queue qVoIP on lagg1 bandwidth 32Kb hfsc (  ecn  ,  realtime 256Kb )  
     queue qOthersHigh on lagg1 bandwidth 9.98% hfsc (  ecn  , linkshare 9.98%  )  
     queue qOthersLow on lagg1 bandwidth 4.99% hfsc (  ecn  , linkshare 4.99%  )  
    
    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/*"
    nat-anchor "natrules/*"
    
    # Outbound NAT rules
    
    # Subnets to NAT 
    table <tonatsubnets>{ 123.45.67.0/24 10.0.13.0/24 10.0.66.1/32 10.0.66.1/32 10.0.66.2/31 10.0.66.4/30 127.0.0.0/8 0.0.0.0  }
    nat on $WAN  from <tonatsubnets>port 500 to any port 500 -> 96.253.50.123/32 port 500  
    nat on $WAN  from <tonatsubnets>to any -> 96.253.50.123/32 port 1024:65535  
    
    # Load balancing anchor
    rdr-anchor "relayd/*"
    # TFTP proxy
    rdr-anchor "tftp-proxy/*"
    rdr pass on lagg1 proto udp from any to 123.45.67.254 port 69 -> 127.0.0.1 port 69
    nat on lagg1 from 127.0.0.1 to any -> 123.45.67.254 port 1024:65535 
    rdr pass on lagg2 proto udp from any to 10.0.13.254 port 69 -> 127.0.0.1 port 69
    nat on lagg2 from 127.0.0.1 to any -> 10.0.13.254 port 1024:65535 
    # UPnPd rdr anchor
    rdr-anchor "miniupnpd"
    
    anchor "relayd/*"
    anchor "openvpn/*"
    anchor "ipsec/*"
    #---------------------------------------------------------------------------
    # default deny rules
    #---------------------------------------------------------------------------
    block in log inet all label "Default deny rule IPv4"
    block out log inet all label "Default deny rule IPv4"
    block in log inet6 all label "Default deny rule IPv6"
    block out log inet6 all label "Default deny rule IPv6"
    
    # IPv6 ICMP is not auxilary, it is required for operation
    # See man icmp6(4)
    # 1    unreach         Destination unreachable
    # 2    toobig          Packet too big
    # 128  echoreq         Echo service request
    # 129  echorep         Echo service reply
    # 133  routersol       Router solicitation
    # 134  routeradv       Router advertisement
    # 135  neighbrsol      Neighbor solicitation
    # 136  neighbradv      Neighbor advertisement
    pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state
    
    # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state
    
    # We use the mighty pf, we cannot be fooled.
    block quick inet proto { tcp, udp } from any port = 0 to any
    block quick inet proto { tcp, udp } from any to any port = 0
    block quick inet6 proto { tcp, udp } from any port = 0 to any
    block quick inet6 proto { tcp, udp } from any to any port = 0
    
    # Snort package
    block quick from <snort2c>to any label "Block snort2c hosts"
    block quick from any to <snort2c>label "Block snort2c hosts"
    
    # SSH lockout
    block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"
    
    # webConfigurator lockout
    block in log quick proto tcp from <webconfiguratorlockout>to any port 443 label "webConfiguratorlockout"
    block in quick from <virusprot>to any label "virusprot overload table"
    pass in quick on { lagg2 } proto tcp from any to { 10.0.13.254 } port { 8001 8000 } keep state(sloppy)
    pass out quick on { lagg2 } proto tcp from any to any flags any keep state(sloppy)
    # block bogon networks
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in log quick on $WAN from <bogons>to any label "block bogon IPv4 networks from WAN"
    block in log quick on $WAN from <bogonsv6>to any label "block bogon IPv6 networks from WAN"
    antispoof for em0
    # block anything from private networks on interfaces with the option set
    antispoof for $WAN
    block in log quick on $WAN from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8"
    block in log quick on $WAN from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8"
    block in log quick on $WAN from 100.64.0.0/10 to any label "Block private networks from WAN block 100.64/10"
    block in log quick on $WAN from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12"
    block in log quick on $WAN from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16"
    block in log quick on $WAN from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7"
    # allow our DHCP client out to the WAN
    pass in on $WAN proto udp from any port = 67 to any port = 68 label "allow dhcp client out WAN"
    pass out on $WAN proto udp from any port = 68 to any port = 67 label "allow dhcp client out WAN"
    # Not installing DHCP server firewall rules for WAN which is configured for DHCP.
    # block bogon networks
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in log quick on $LAN from <bogons>to any label "block bogon IPv4 networks from LAN"
    block in log quick on $LAN from <bogonsv6>to any label "block bogon IPv6 networks from LAN"
    antispoof for lagg1
    # allow access to DHCPv6 server on LAN
    # We need inet6 icmp for stateless autoconfig and dhcpv6
    pass quick on $LAN inet6 proto udp from fe80::/10 to fe80::/10 port = 546 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 546 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 547 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from ff02::/16 to fe80::/10 port = 547 label "allow access to DHCPv6 server"
    pass in quick on $LAN inet6 proto udp from fe80::/10 to 2001:470:88e1:ffff:ffff:ffff:ffff:ffff port = 546 label "allow access to DHCPv6 server"
    pass out quick on $LAN inet6 proto udp from 2001:470:88e1:ffff:ffff:ffff:ffff:ffff port = 547 to fe80::/10 label "allow access to DHCPv6 server"
    # block bogon networks
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in log quick on $DMZ from <bogons>to any label "block bogon IPv4 networks from DMZ"
    block in log quick on $DMZ from <bogonsv6>to any label "block bogon IPv6 networks from DMZ"
    antispoof for lagg2
    # allow access to DHCP server on DMZ
    pass in quick on $DMZ proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $DMZ proto udp from any port = 68 to 10.0.13.254 port = 67 label "allow access to DHCP server"
    pass out quick on $DMZ proto udp from 10.0.13.254 port = 67 to any port = 68 label "allow access to DHCP server"
    # block bogon networks
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in log quick on $WAN6 from <bogons>to any label "block bogon IPv4 networks from WAN6"
    block in log quick on $WAN6 from <bogonsv6>to any label "block bogon IPv6 networks from WAN6"
    # block anything from private networks on interfaces with the option set
    antispoof for $WAN6
    block in log quick on $WAN6 from 10.0.0.0/8 to any label "Block private networks from WAN6 block 10/8"
    block in log quick on $WAN6 from 127.0.0.0/8 to any label "Block private networks from WAN6 block 127/8"
    block in log quick on $WAN6 from 100.64.0.0/10 to any label "Block private networks from WAN6 block 100.64/10"
    block in log quick on $WAN6 from 172.16.0.0/12 to any label "Block private networks from WAN6 block 172.16/12"
    block in log quick on $WAN6 from 192.168.0.0/16 to any label "Block private networks from WAN6 block 192.168/16"
    block in log quick on $WAN6 from fc00::/7 to any label "Block ULA networks from WAN6 block fc00::/7"
    
    # loopback
    pass in on $loopback inet all label "pass IPv4 loopback"
    pass out on $loopback inet all label "pass IPv4 loopback"
    pass in on $loopback inet6 all label "pass IPv6 loopback"
    pass out on $loopback inet6 all label "pass IPv6 loopback"
    # let out anything from the firewall host itself and decrypted IPsec traffic
    pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
    pass out route-to ( em0 96.253.50.1 ) from 96.253.50.123 to !96.253.50.0/24 keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to ( gif0 2001:470:1f06:356::1 ) inet6 from 2001:470:1f06:356::2 to !2001:470:1f06:356::2/64 keep state allow-opts label "let out anything from firewall host itself"
    pass out on $IPsec all keep state label "IPsec internal host to host"
    # make sure the user cannot lock himself out of the webConfigurator or SSH
    pass in quick on lagg1 proto tcp from any to (lagg1) port { 443 22 } keep state label "anti-lockout rule"
    # PPTPd rules
    pass in on $WAN proto tcp from any to 96.253.50.123 port = 1723 modulate state label "allow pptpd 96.253.50.123"
    pass in on $WAN proto gre from any to any keep state label "allow gre pptpd"
    
    # User-defined rules follow
    
    anchor "userrules/*"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto tcp  from any to 96.253.50.123 port 22 flags S/SA keep state  label "USER_RULE: WAN console access"
    # returning at dst  == "/" label "USER_RULE: WAN console access"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto tcp  from any to 96.253.50.123 port 443 flags S/SA keep state  label "USER_RULE: WAN web configurator access"
    # returning at dst  == "/" label "USER_RULE: WAN web configurator access"
    pass  quick  on {  WANGRP  em0  } inet proto icmp  from $IPv6TunnelRemote to any keep state  label "USER_RULE: HE Tunnelbroker connectivity check"
    pass  quick  on {  WANGRP  lagg1  lagg2  gif0  } inet6 proto ipv6-icmp  from any to any keep state  label "USER_RULE: ICMP Packet Too Big (Type 2) [needs fixing]"
    pass  in  quick  on {  em0  } reply-to ( em0 96.253.50.1 ) inet proto esp  from $IPv4TunnelRemote to 96.253.50.123  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  out  quick  on {  em0  }  $GWWANGW inet proto esp  from 96.253.50.123 to $IPv4TunnelRemote  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  in  on {  lagg1  } inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting outgoing"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  enc0  } inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  quick  on {  WANGRP  lagg1  lagg2  gif0  } inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto { tcp udp }  from any to any port 53 keep state  label "USER_RULE: DNS"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet6 proto { tcp udp }  from any to any port 53 keep state  label "USER_RULE: DNS"
    pass  in log  quick  on {  lagg1  } inet proto tcp  from 123.45.67.0/24  to <negate_networks>flags S/SA keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in log  quick  on {  lagg1  }  $GWWANGW inet proto tcp  from 123.45.67.0/24 to any port 80 flags S/SA keep state  label "USER_RULE: HTTP"
    pass  out log  quick  on {  lagg1  } inet proto tcp  from 123.45.67.0/24  to <negate_networks>flags S/SA keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  out log  quick  on {  lagg1  }  $GWWANGW inet proto tcp  from 123.45.67.0/24 to any port 80 flags S/SA keep state  label "USER_RULE: HTTP"
    match    proto udp  from $AnveoSIP to any  queue (qVoIP)  label "USER_RULE: Connections From Upstream SIP Server"
    match    proto udp  from any to $AnveoSIP  queue (qVoIP)  label "USER_RULE: Connections To Upstream SIP Server"
    match    on {  em0  }  proto tcp  from any to any port 3389 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other MSRDP outbound"
    match    on {  em0  }  proto tcp  from any to any port 5899 >< 5931 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other VNC outbound"
    match    on {  em0  }  proto tcp  from any to any port 3283 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AppleRemoteDesktop1 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5900 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AppleRemoteDesktop2 outbound"
    match    on {  em0  }  proto udp  from any to any port 3283  queue (qOthersHigh)  label "USER_RULE: m_Other AppleRemoteDesktop3 outbound"
    match    on {  em0  }  proto udp  from any to any port 5900  queue (qOthersHigh)  label "USER_RULE: m_Other AppleRemoteDesktop4 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5631 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other pcany1 outbound"
    match    on {  em0  }  proto udp  from any to any port 5632  queue (qOthersHigh)  label "USER_RULE: m_Other pcany2 outbound"
    match    on {  em0  }  proto tcp  from any to any port 6666 >< 6671 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5222 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5223 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5269 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5190 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other ICQ1 outbound"
    match    on {  em0  }  proto udp  from any to any port 5190  queue (qOthersHigh)  label "USER_RULE: m_Other ICQ2 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5190 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AIM outbound"
    match    on {  em0  }  proto tcp  from any to any port 1723 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other PPTP outbound"
    match    on {  em0  }  proto gre  from any to any  queue (qOthersHigh)  label "USER_RULE: m_Other PPTPGRE outbound"
    match    on {  em0  }  proto tcp  from any to any port 7999 >< 8101 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other STREAMINGMP3 outbound"
    match    on {  em0  }  proto tcp  from any to any port 554 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other RTSP1 outbound"
    pass  in  quick  on $WANGRP inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $WANGRP inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in  quick  on $IPsec inet from $EasyRuleBlockHostsENC0 to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $IPsec inet proto tcp  from !123.45.67.0/24 to 123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: Limit incoming IPSec traffic"
    pass  in  quick  on $IPsec inet from any to any  dnpipe ( 1,2)  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $pptp inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $pptp inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $WAN reply-to ( em0 96.253.50.1 )  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    block return  in log  quick  on $WAN reply-to ( em0 96.253.50.1 )  from any to $pfBlockerblocklistde  label "USER_RULE: pfBlockerblocklistde auto rule"
    block  in  quick  on $WAN reply-to ( em0 96.253.50.1 ) inet from $EasyRuleBlockHostsWAN to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $WAN reply-to ( em0 96.253.50.1 ) inet from any to any  dnpipe ( 3,4)  label "USER_RULE: TEMP pass all IPv4"
    block  in log  quick  on $LAN  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    pass  in log  quick  on $LAN inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: slow IPv4 TCP traffic leaving the LAN"
    pass  in  quick  on $LAN inet from any to any  dnpipe ( 3,4)  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $LAN inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $DMZ  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    pass  in  quick  on $DMZ inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $DMZ inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $WAN6  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    block return  in log  quick  on $WAN6  from any to $pfBlockerblocklistde  label "USER_RULE: pfBlockerblocklistde auto rule"
    block  in  quick  on $WAN6 inet6 from $EasyRuleBlockHostsOPT2 to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $WAN6 inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    
    # VPN Rules
    pass out on $WAN  route-to ( em0 96.253.50.1 )  proto udp from any to 111.222.333.444 port = 500 keep state label "IPsec: PUBLIC_IP-LAN-NET-tunnel - outbound isakmp"
    pass in on $WAN  reply-to ( em0 96.253.50.1 )  proto udp from 111.222.333.444 to any port = 500 keep state label "IPsec: PUBLIC_IP-LAN-NET-tunnel - inbound isakmp"
    pass out on $WAN  route-to ( em0 96.253.50.1 )  proto esp from any to 111.222.333.444 keep state label "IPsec: PUBLIC_IP-LAN-NET-tunnel - outbound esp proto"
    pass in on $WAN  reply-to ( em0 96.253.50.1 )  proto esp from 111.222.333.444 to any keep state label "IPsec: PUBLIC_IP-LAN-NET-tunnel - inbound esp proto"
    anchor "tftp-proxy/*"
    anchor "miniupnpd"
    
    #/root(8): kldstat
    Id Refs Address            Size     Name
     1   18 0xffffffff80100000 15658c0  kernel
     2    1 0xffffffff81666000 27a8     coretemp.ko
     3    1 0xffffffff81812000 133e50   zfs.ko
     4    1 0xffffffff81946000 1fcd     opensolaris.ko
     5    1 0xffffffff81948000 a066     dummynet.ko
    #/root(9):</negate_networks></negate_networks></bogonsv6></bogons></bogonsv6></bogons></bogonsv6></bogons></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></tonatsubnets></tonatsubnets></tonatsubnets></vpn_networks></snortwhitelist></snortwhitelist></ipv6tunnelremote></ipv6tunnelremote></ipv6gateways></ipv6gateways></ipv4tunnelremote></ipv4tunnelremote></easyruleblockhostswan></easyruleblockhostswan></easyruleblockhostsopt2></easyruleblockhostsopt2></easyruleblockhostsenc0></easyruleblockhostsenc0></anveosip></anveosip></pfblockerblocklistde></pfblockerblocklistde></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> 
    


  • A quick look at your rules.debug shows:

    pass  out  quick  on {  em0  }  $GWWANGW inet proto esp  from 96.253.50.123 to $IPv4TunnelRemote  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  in  on {  lagg1  } inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting outgoing"

    I have quoted the two rules to point out an important distinction. The first rule has the quick attribute which means that rule matching stops for any traffic that matches the rule. The second rule (for TCP traffic arriving on the LAN interface, lagg1) doesn't have the quick attribute so rule matching continues until the last rule (or the next "quick" rule) that matches, probably effectively bypassing the limiter. (I haven't checked the following LAN rules.)

    I suspect you defined this rule as a floating rule rather than a LAN interface rule. (Interface rules seem to get the quick attribute). I suggest you delete this rule and then add a similar LAN interface rule as the first interface rule, reset states then test the limiter is applied. The LAN interface rules might then need some tweaking to get the ordering and limiting as desired.



  • @wallabybob:

    A quick look at your rules.debug shows:

    pass  out  quick  on {  em0  }  $GWWANGW inet proto esp  from 96.253.50.123 to $IPv4TunnelRemote  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  in  on {  lagg1  } inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting outgoing"

    I have quoted the two rules to point out an important distinction. The first rule has the quick attribute which means that rule matching stops for any traffic that matches the rule. The second rule (for TCP traffic arriving on the LAN interface, lagg1) doesn't have the quick attribute so rule matching continues until the last rule (or the next "quick" rule) that matches, probably effectively bypassing the limiter. (I haven't checked the following LAN rules.)

    I suspect you defined this rule as a floating rule rather than a LAN interface rule. (Interface rules seem to get the quick attribute). I suggest you delete this rule and then add a similar LAN interface rule as the first interface rule, reset states then test the limiter is applied. The LAN interface rules might then need some tweaking to get the ordering and limiting as desired.

    Well, I tried various things, for one I added the "Quick" attribute, which was an oversight.
    I also have a LAN interface rule, I think like you suggested.

    Here is the latest set of rules:

    # User-defined rules follow
    
    anchor "userrules/*"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto tcp  from any to 96.253.50.123 port 22 flags S/SA keep state  label "USER_RULE: WAN console access"
    # returning at dst  == "/" label "USER_RULE: WAN console access"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto tcp  from any to 96.253.50.123 port 443 flags S/SA keep state  label "USER_RULE: WAN web configurator access"
    # returning at dst  == "/" label "USER_RULE: WAN web configurator access"
    pass  quick  on {  WANGRP  em0  } inet proto icmp  from $IPv6TunnelRemote to any keep state  label "USER_RULE: HE Tunnelbroker connectivity check"
    pass  quick  on {  WANGRP  lagg1  lagg2  gif0  } inet6 proto ipv6-icmp  from any to any keep state  label "USER_RULE: ICMP Packet Too Big (Type 2) [needs fixing]"
    pass  in  quick  on {  em0  } reply-to ( em0 96.253.50.1 ) inet proto esp  from $IPv4TunnelRemote to 96.253.50.123  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  out  quick  on {  em0  }  $GWWANGW inet proto esp  from 96.253.50.123 to $IPv4TunnelRemote  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting"
    pass  in  quick  on {  lagg1  } inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: IPSec bandwidth limiting outgoing"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  enc0  } inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  quick  on {  WANGRP  lagg1  lagg2  gif0  } inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet proto { tcp udp }  from any to any port 53 keep state  label "USER_RULE: DNS"
    pass  quick  on {  WANGRP  em0  lagg1  lagg2  gif0  pptp  enc0  } inet6 proto { tcp udp }  from any to any port 53 keep state  label "USER_RULE: DNS"
    pass  in log  quick  on {  lagg1  } inet proto tcp  from 123.45.67.0/24  to <negate_networks>flags S/SA keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  in log  quick  on {  lagg1  }  $GWWANGW inet proto tcp  from 123.45.67.0/24 to any port 80 flags S/SA keep state  label "USER_RULE: HTTP"
    pass  out log  quick  on {  lagg1  } inet proto tcp  from 123.45.67.0/24  to <negate_networks>flags S/SA keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
    pass  out log  quick  on {  lagg1  }  $GWWANGW inet proto tcp  from 123.45.67.0/24 to any port 80 flags S/SA keep state  label "USER_RULE: HTTP"
    match    proto udp  from $AnveoSIP to any  queue (qVoIP)  label "USER_RULE: Connections From Upstream SIP Server"
    match    proto udp  from any to $AnveoSIP  queue (qVoIP)  label "USER_RULE: Connections To Upstream SIP Server"
    match    on {  em0  }  proto tcp  from any to any port 3389 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other MSRDP outbound"
    match    on {  em0  }  proto tcp  from any to any port 5899 >< 5931 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other VNC outbound"
    match    on {  em0  }  proto tcp  from any to any port 3283 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AppleRemoteDesktop1 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5900 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AppleRemoteDesktop2 outbound"
    match    on {  em0  }  proto udp  from any to any port 3283  queue (qOthersHigh)  label "USER_RULE: m_Other AppleRemoteDesktop3 outbound"
    match    on {  em0  }  proto udp  from any to any port 5900  queue (qOthersHigh)  label "USER_RULE: m_Other AppleRemoteDesktop4 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5631 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other pcany1 outbound"
    match    on {  em0  }  proto udp  from any to any port 5632  queue (qOthersHigh)  label "USER_RULE: m_Other pcany2 outbound"
    match    on {  em0  }  proto tcp  from any to any port 6666 >< 6671 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5222 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5223 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5269 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other IRC outbound"
    match    on {  em0  }  proto tcp  from any to any port 5190 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other ICQ1 outbound"
    match    on {  em0  }  proto udp  from any to any port 5190  queue (qOthersHigh)  label "USER_RULE: m_Other ICQ2 outbound"
    match    on {  em0  }  proto tcp  from any to any port 5190 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other AIM outbound"
    match    on {  em0  }  proto tcp  from any to any port 1723 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other PPTP outbound"
    match    on {  em0  }  proto gre  from any to any  queue (qOthersHigh)  label "USER_RULE: m_Other PPTPGRE outbound"
    match    on {  em0  }  proto tcp  from any to any port 7999 >< 8101 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other STREAMINGMP3 outbound"
    match    on {  em0  }  proto tcp  from any to any port 554 flags S/SA  queue (qOthersHigh,qACK)  label "USER_RULE: m_Other RTSP1 outbound"
    pass  in  quick  on $WANGRP inet from any to any  dnpipe ( 1,2)  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $WANGRP inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in  quick  on $IPsec inet from $EasyRuleBlockHostsENC0 to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $IPsec inet proto tcp  from !123.45.67.0/24 to 123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: Limit incoming IPSec traffic"
    pass  in  quick  on $IPsec inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: Limit outgoing IPSec traffic"
    pass  in  quick  on $IPsec inet from any to any  dnpipe ( 1,2)  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $pptp inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $pptp inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $WAN reply-to ( em0 96.253.50.1 )  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    block return  in log  quick  on $WAN reply-to ( em0 96.253.50.1 )  from any to $pfBlockerblocklistde  label "USER_RULE: pfBlockerblocklistde auto rule"
    block  in  quick  on $WAN reply-to ( em0 96.253.50.1 ) inet from $EasyRuleBlockHostsWAN to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $WAN reply-to ( em0 96.253.50.1 ) inet from any to any  dnpipe ( 3,4)  label "USER_RULE: TEMP pass all IPv4"
    block  in log  quick  on $LAN  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    pass  in log  quick  on $LAN inet proto tcp  from 123.45.67.0/24 to !123.45.67.0/24 flags S/SA keep state  dnpipe ( 1,2)  label "USER_RULE: slow IPv4 TCP traffic leaving the LAN"
    pass  in  quick  on $LAN inet from any to any  dnpipe ( 3,4)  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $LAN inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $DMZ  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    pass  in  quick  on $DMZ inet from any to any  label "USER_RULE: TEMP pass all IPv4"
    pass  in  quick  on $DMZ inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"
    block  in log  quick  on $WAN6  from $pfBlockerblocklistde to any  label "USER_RULE: pfBlockerblocklistde auto rule"
    block return  in log  quick  on $WAN6  from any to $pfBlockerblocklistde  label "USER_RULE: pfBlockerblocklistde auto rule"
    block  in  quick  on $WAN6 inet6 from $EasyRuleBlockHostsOPT2 to any  label "USER_RULE: Easy Rule: Blocked from Firewall Log View"
    pass  in  quick  on $WAN6 inet6 from any to any  label "USER_RULE: TEMP pass all IPv6"</negate_networks></negate_networks> 
    

    I found another solution to the original problem of the IPSec link going down, which however does nothing to solve the mystery of the limiting not working: Since the pfSense box is hooked up to the rest of the LAN through a little 8-port smart ethernet switch, I first just forced the interface speed down to 10Mbit/s, then  decided to look if that switch as bandwidth limiting, which it turns out to have. Unfortunately only in certain fixed increments, so I can't limit to the 15Mbit/s I originally wanted to, but I can limit to 20Mbit/s and that seems to be sufficient, because the peaks into higher speeds don't happen, and so far the IPSec link has remained stable with this, which at least shows that it's a traffic speed issue.

    So we can continue to see if we can make limiting work, if that helps potentially debug some issue with pfSense, because the 15Mbit/s are lower than the 20Mbit/s I limit to in the ethernet switch, so it's still easily visible if things take effect or not.

    On a somewhat related note: you keep telling me to reset the states, but when ever I look, there is an empty state table. Could it be that IPSec simply sucks all traffic past any rules, and therefore there never are any states in the first place, and none of the filters ever get applied? Or is there an issue with the web GUI just not showing any states?



  • @rcfa:

    I found another solution to the original problem of the IPSec link going down, which however does nothing to solve the mystery of the limiting not working: Since the pfSense box is hooked up to the rest of the LAN through a little 8-port smart ethernet switch, I first just forced the interface speed down to 10Mbit/s, then  decided to look if that switch as bandwidth limiting, which it turns out to have. Unfortunately only in certain fixed increments, so I can't limit to the 15Mbit/s I originally wanted to, but I can limit to 20Mbit/s and that seems to be sufficient, because the peaks into higher speeds don't happen, and so far the IPSec link has remained stable with this, which at least shows that it's a traffic speed issue.

    So we can continue to see if we can make limiting work, if that helps potentially debug some issue with pfSense, because the 15Mbit/s are lower than the 20Mbit/s I limit to in the ethernet switch, so it's still easily visible if things take effect or not.

    No sooner than I claim that things are working with a workaround, they get stuck again… Must be one of Murphy's laws be at work.

    Well, things stayed stable for close to a day, but then the link got stuck a few times in series with the same old symptoms: shows as "up" while passing no traffic. So I guess 20Mbit/s is still too fast a limit, but a lot more stable than before. I may have to limit to 10Mbit/s until I can convince pfSense to cooperate, because the Switch doesn't give me any speed between 10 or 20Mbit/s as an option, which is a bummer, but better than nothing.



  • I would add some logging to the rules to attempt to determine which rule the traffic is matching.