Lan rules have NOTHING to do with unsolicited traffic TO the server.. Since the server is not creating the connection.
Rules are evaluated as the traffic enters an interface from the network the interface is connected too, towards pfsense.
If your vpn can talk to everything on this lan network, except this server I would look to as already mentioned firewall on this server.