OK, I understand much of the issue a bit better now.
The real problem is related to internal/external DNS. (See below)
Devices on different subnets/VLANs have different DNS rules...
Turns out the stuff that's borked has DHCP-level DNS override for that subnet, specifying external (184.108.40.206) DNS for devices on that subnet...
With the result that those devices don't ever get the internal IP address for the (mqtt) server in question... (my pfSense DNS resolver has host overrides for internal servers)
And apparently the packets get royally screwed up:
<internal IP A> sends SYN to <external WAN address>
then the <SYN ACK> reply from the server is
<internal server IP> to <internal IP A>
which simply does NOT work.
Edit: belay the question below. I've now learned about NAT Reflection, and @johnpoz' well informed disgust at any use of it ;)
Alternative question: how do we have DNS handle different subnets / VLANs in different ways? In essence, after local host overrides, some VLANs would be processed by forwarding to our filtered DNS service, while others would use unfiltered generic DNS.
Any idea how to implement that? I don't see that as a feature in the GUI at least.
Bottom line question... is there a correct way to configure the ability for internal IP's to access NAT'd devices by way of the external WAN IP address?
If so, perhaps I don't need the host overrides in the first place.
(The Rst stuff is, I think, because I wasn't looking for the packets in the right place ;) ... now I'm testing on a laptop with wifi rather than a phone, so I can run WireShark and see exactly what's happening...)