Just an update, go this working by moving the rule to the LAN tab
used the ack queue for SSH interactive and used the main queue for SCP
$ cat /tmp/rules.debug <snip>pass in quick on $LAN proto { tcp udp } from any to any port 22 keep state queue (qP2P,qOthersHigh) dnpipe ( 4, 3) label "USER_RULE: Prioritise SSH not SCP traffic"</snip>