Confused: how to correctly allow specific inter-VLAN traffic?

MrPete

I thought I understood this stuff. Clearly I do not :(
My situation:

Multiple VLANs each on a separate subnet
I have interface groups for "trusted" and "public"
I have a default rule on the public group to block all traffic to the trusted group

Now, for example, I want to allow anybody to reach a specific IP in the trusted group, with alias alias_mqtt (an MQTT broker, surprise surprise ;) )

It doesn't work.

Perhaps there's something unusual about the websocket protocol? What I see is this:

Without adding additional rules, I get firewall logs of Default Deny Rule IPV4, TCP:R, from my device to the MQTT broker on the stated port.
tcpdump on both the source and target subnets captures just a bunch of Rst packets. ?????
If I add another rule to specifically allow R packets... those go through but still the connection is not made.

I don't have much more hair to tear out ;)

johnpoz

@mrpete said in Confused: how to correctly allow specific inter-VLAN traffic?:

TCP:R, from my device to the MQTT broker on the stated port.

Well that is RST - that is the client saying done, really done!! with this connection.. If the state was already closed on pfsense because it saw a fin, then yeah that would be blocked by pfsense because it is out of state.

A connection is made with SYN (S) packets, the client wanting to talk to IP X sends a SYN, the server would send back a SA (syn,ack).. If your not seeing those in your sniffs then no connections would never work.

stateful firewall is only going to open a state when it sees a SYN, and the traffic, ie port and IPs are allowed.. No other flag IE SA or Fin or Fin,ACK or R (rst) is going to open a state to allow traffic.

MrPete

@johnpoz That's been my understanding as well.

So why oh why am I only seeing Rst packets. Hmmmm...

Fortunately I can do some same-subnet experiments and watch... and when on same-subnet it all works as expected S, SA, and lots of data...

So I'm still stumped on why I'm not able to open a cross-subnet hole for this. I've got many other services working across subnets...

johnpoz

@mrpete have no idea.. but sure never going to work if you only ever see RST.. Which has nothing to do with pfsense. If your client never sends a syn when trying to talk.. How could it ever work?

MrPete

@johnpoz
OK, I understand much of the issue a bit better now.

The real problem is related to internal/external DNS. (See below)
Devices on different subnets/VLANs have different DNS rules...
Turns out the stuff that's borked has DHCP-level DNS override for that subnet, specifying external (8.8.8.8) DNS for devices on that subnet...
With the result that those devices don't ever get the internal IP address for the (mqtt) server in question... (my pfSense DNS resolver has host overrides for internal servers)
And apparently the packets get royally screwed up:
<internal IP A> sends SYN to <external WAN address>
then the <SYN ACK> reply from the server is
<internal server IP> to <internal IP A>
which simply does NOT work.

Edit: belay the question below. I've now learned about NAT Reflection, and @johnpoz' well informed disgust at any use of it ;)

Alternative question: how do we have DNS handle different subnets / VLANs in different ways? In essence, after local host overrides, some VLANs would be processed by forwarding to our filtered DNS service, while others would use unfiltered generic DNS.

Any idea how to implement that? I don't see that as a feature in the GUI at least.

~~Bottom line question... is there a correct way to configure the ability for internal IP's to access NAT'd devices by way of the external WAN IP address?~~

~~If so, perhaps I don't need the host overrides in the first place.~~

(The Rst stuff is, I think, because I wasn't looking for the packets in the right place ;) ... now I'm testing on a laptop with wifi rather than a phone, so I can run WireShark and see exactly what's happening...)