Traffic Shaper Wizard / ICMP queue problem

athurdent · Jan 12, 2017, 7:35 AM

Creating a fresh topic for this. :)
I never actually used the traffic shaper, always created everything by hand if needed. So I must be doing something wrong here.

Latest 2.4 snapshot as of now. Mostly vanilla Proxmox KVM created for testing pfSense 2.4 beta, 2 em interfaces.

em0 LAN
em1 WAN

Checked via viconfig, no ezshaper section in the config.

Used the Multi WAN / LAN Wizard and kept everything at default except for Upload (20MBit) / Download (400MBit) and chose ICMP High Prio.
After finishing the wizard I rebooted.

Verified the rules:

[2.4.0-BETA][root@pfSense-beta]/root: cat /tmp/rules.debug | grep outbound
match    on {  em1  }  proto icmp  from any to any  queue (qOthersHigh)  label "USER_RULE: m_Other ICMP outbound"

[2.4.0-BETA][root@pfSense-beta]/root: pfctl -vvsr | grep -A2 outbound
@70(0) match on em1 proto icmp all label "USER_RULE: m_Other ICMP outbound" queue qOthersHigh
  [ Evaluations: 378       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: pid 30474 State Creations: 0     ]

Ping www.heise.de through the test VM:

[2.4.0-BETA][root@pfSense-beta]/root: tcpdump -nvvi em1 icmp and host www.heise.de
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:10:09.790054 IP (tos 0x0, ttl 63, id 31661, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.178.22 > 193.99.144.85: ICMP echo request, id 57153, seq 1, length 64
08:10:09.816643 IP (tos 0x0, ttl 246, id 19159, offset 0, flags [DF], proto ICMP (1), length 84)
    193.99.144.85 > 192.168.178.22: ICMP echo reply, id 57153, seq 1, length 64
08:10:10.791350 IP (tos 0x0, ttl 63, id 31737, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.178.22 > 193.99.144.85: ICMP echo request, id 57153, seq 2, length 64
08:10:10.814755 IP (tos 0x0, ttl 246, id 47076, offset 0, flags [DF], proto ICMP (1), length 84)
    193.99.144.85 > 192.168.178.22: ICMP echo reply, id 57153, seq 2, length 64
08:10:11.792482 IP (tos 0x0, ttl 63, id 31896, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.178.22 > 193.99.144.85: ICMP echo request, id 57153, seq 3, length 64
08:10:11.816634 IP (tos 0x0, ttl 246, id 17809, offset 0, flags [DF], proto ICMP (1), length 84)
    193.99.144.85 > 192.168.178.22: ICMP echo reply, id 57153, seq 3, length 64

Result:

[2.4.0-BETA][root@pfSense-beta]/root: pfctl -vvsr | grep -A2 outbound
@70(0) match on em1 proto icmp all label "USER_RULE: m_Other ICMP outbound" queue qOthersHigh
  [ Evaluations: 467       Packets: 1         Bytes: 84          States: 0     ]
  [ Inserted: pid 30474 State Creations: 0     ]

[2.4.0-BETA][root@pfSense-beta]/root: pfctl -vvsq
queue qACK on em1 priority 6 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qDefault on em1 priority 3 priq( red ecn default )
  [ pkts:       1172  bytes:      80801  dropped pkts:     52 bytes:   4385 ]
  [ qlength:   0/ 50 ]
  [ measured:     3.1 packets/s, 1.49Kb/s ]
queue qOthersHigh on em1 priority 4 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qOthersLow on em1 priority 2 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qLink on em0 priority 2 qlimit 500 priq( red ecn default )
  [ pkts:       2134  bytes:    1249152  dropped pkts:      3 bytes:   4542 ]
  [ qlength:   0/500 ]
  [ measured:     4.6 packets/s, 12.87Kb/s ]
queue qACK on em0 priority 6 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qOthersHigh on em0 priority 4 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue qOthersLow on em0 priority 3 priq( red ecn )
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]

qOthersHigh stays empty.

Creating a rule by hand prevents the ruleset from loading:

pass  in  quick  on $LAN inet proto icmp  from any to any icmp-type any tracker 1484205722 keep state  queue (qOthersHigh)  label "USER_RULE: ICMP q test"

Result:

There were error(s) loading the rules: /tmp/rules.debug:141: syntax error - The line in question reads [141]: pass in quick on $LAN inet proto icmp from any to any icmp-type any tracker 1484205722 keep state queue (qOthersHigh) label "USER_RULE: ICMP q test"
@ 2017-01-12 08:22:05

Again, I might be doing something wrong here, never used traffic shaping for a long time. :)

chrcoluk · Jan 12, 2017, 5:30 PM

Remove icmp-type any as that is invalid PF syntax.

So change the new rule to

pass  in  quick  on $LAN inet proto icmp  from any to any tracker 1484205722 queue (qOthersHigh)  label "USER_RULE: ICMP q test"

Bear in mind if you want icmp prioritised both ways due to the lack of keep state (even tho syntax is accepted) then you also need a rule on the WAN interface.

here is my icmp rules

pass  in  quick  on $LAN inet proto icmp  from any to any icmp-type { echorep,echoreq,timex,trace,unreach } tracker 1483894554  no state  queue (qICMP)  label "USER_RULE: icmp"
pass  in  quick  on $LAN inet6 proto ipv6-icmp  from any to any icmp6-type { echorep,echoreq,timex,toobig,unreach } tracker 1483894614  no state  queue (qICMP)  label "USER_RULE: icmp6"

and for other direction

pass  in  quick  on $WAN reply-to ( igb0 fe80::a2f3:e4ff:fe7f:4230 ) inet6 proto ipv6-icmp  from any to any icmp6-type { echorep,echoreq,timex,toobig,unreach } tracker 1483286402 keep state  queue (qICMP)  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( igb0 x.x.x.x) inet proto icmp  from any to any icmp-type { echorep,echoreq } tracker 1482936755 keep state  queue (qICMP)  label "USER_RULE: ICMP Ping Request"

the x.x.x.x is wan gateway ip, I generated that rule in the GUI I personally would not have used that exact syntax by hand as quick in on $WAN without the reply-to is sufficient. pfSense also added my link-local gateway on ipv6.

athurdent · Jan 12, 2017, 5:38 PM

Uhm, no. It is perfectly fine to use keep state on every protocol, this is how pf was designed.

chrcoluk · Jan 12, 2017, 6:21 PM

what is the result then with the fixed syntax?

you can keep keep state on with it only in one direction but just be aware it wont catch all icmp traffic.

Nullity · Jan 12, 2017, 6:31 PM

@chrcoluk:

what is the result then with the fixed syntax?

you can keep keep state on with it only in one direction but just be aware it wont catch all icmp traffic.

The response packets will be statefully allowed and if they leave through qArbitrary on LAN they will be assigned to qArbitrary on WAN, I think. (But "prioritizing" ingress ICMP packets that have already arrived on the LAN gateway is usually nonsensical if the intent is to improve latency.)

athurdent · Jan 12, 2017, 6:41 PM

chrcoluk,

I have only tried to reproduce your problems and posted my own findings in the first post, I hope with the necessary information for everyone else to reproduce. I do not use the traffic shaper for any of my installations ATM.

So, I think we should just wait and see if anyone thinks there might really be a problem with 2.4 beta or maybe correct my faulty test setup from post one. :)

chrcoluk · Jan 12, 2017, 6:49 PM

So in the end you did not try to get the rule you added working?

Also I have never diagnosed the problem I reported heavily with 2.2 so I dont know if it is unique to 2.4.

chrcoluk · Jan 12, 2017, 6:54 PM

@Nullity:

@chrcoluk:

what is the result then with the fixed syntax?

you can keep keep state on with it only in one direction but just be aware it wont catch all icmp traffic.

The response packets will be statefully allowed and if they leave through qArbitrary on LAN they will be assigned to qArbitrary on WAN, I think. (But "prioritizing" ingress ICMP packets that have already arrived on the LAN gateway is usually nonsensical if the intent is to improve latency.)

I have not really tested the impact on ICMP with ingress shaping, but I have with ssh packets. So e.g. if I run a dslreports speedtest using the cable settings which uses over 20 download threads, and then hold down a key in a ssh session I see the cursor jumping around (which means packet loss), however if the ssh traffic is on a higher inbound queue, it becomes smooth as none of its packets are getting dropped, so ingress shaping is at least effective on tcp traffic.

athurdent · Jan 13, 2017, 9:41 AM

@chrcoluk:

Remove icmp-type any as that is invalid PF syntax.

Filed a bugreport for this, phil.davis has fixed it and created a pull request. https://github.com/pfsense/pfsense/pull/3377
Should be in one of the next snapshots I think. Thanks again, Phil!

chrcoluk · Jan 13, 2017, 4:22 PM

also someone has repeated the other problem I found and files his own report.