Weird failed connections

[KOM]

I am trying to audit exactly which ports our server software uses so as to better assist our customers when configuring their own firewalls.  Our developers use high-level RPC function calls to do everything and they don't understand the underlying plumbing going on.  When you try to connect to a server, here is what happens:

On the first server, you select itself and its data disk.  You then specify the remote server and its data disk.  A group of the two is then created.

With both servers on the same subnet, everything works as expected every time.  When I have first server on LAN and second server on OPT1 (both nets have Allow All rule), they can talk but they fail to create a group.  There is nothing in the firewall log.  I thought maybe static ports were the issue and manually added a static port outbound NAT rule for both LAN and OPT1.  No change.

LAN 10.10.0.0/16  pfsense @ 10.10.5.1
OPT1 192.168.10.0/24  pfsense @ 192.168.10.1

Server1 ip 10.10.0.160 / mask 16 / gw 10.10.5.1
Server2 ip 192.168.10.2 / mask 24 / gw 192.168.10.1

I'm sure I'm missing something dumb or got my NAT rule wrong.

[johnpoz]

Why don't you just sniff on one of the servers directly or both of them when they are on the same network to see what they are doing.. For you know they are trying to use multicast which is not being logged nor would it pass, etc.

[KOM]

We have other customers that have routed servers without this issue.  I didn't think any direct server to server RPC comms required multicast.

[johnpoz]

I agree with you - but you stated this

"they don't understand the underlying plumbing going on."

So unless you do, you have no idea what they are doing - right?  You say it works when both on the same lan.. So look to see what is going on when on the lan, then you can make your firewall rules to allow this, etc.

You for sure would not need to do any sort of natting here - since local networks to pfsense do not nat between each other.