Better logging & RPC Traffic

[boomam]

Hi,
Is there anyway to get PFSense to show me better logs?
As ive been trying to diagnose an issue for a few days now but without the ability to really drill down into the logs like i can with TMG or ISA, its hard to tell where the fault lies!

Basically im fairly certain that RPC traffic, or traffic to a specific IP is getting blocked by one of the PFSense boxes either side of a Site-to-site link, but the logs on either if i filter it via IP address, lists nothing.
When i KNOW that traffic to or from those IPs are passing through both pfsenses!

Any ideas on either anyone?

Thanks in advance.

[jimp]

It only logs what it's told to log, which by default is only blocked traffic not matching any other rules.

If you want it to log passes, edit your pass rules, check the log box, and then try the connections again.

You could also do packet captures, check the states table, etc.

[boomam]

Thank you for the prompt reply, i'm editing the rules now to see whats going on…will post back if i solve it (unlikely), and post anyway if i dont. :-p

[boomam]

Although thinking about it, unless im being daft, if theres nothing in the default log rule (basically blocks), then nothing will be getting blocked will it?

[jimp]

Correct. If it is being blocked, but not logging, it would have to be hitting a block rule that you created without the log option set.

Otherwise, if it did hit the default rule and was blocked, then it would have made a log entry (assuming you did not disable the default block logging on  the log settings tab…)

[boomam]

This is very strange.

I can see the traffic going over the two PF's in the firewall log.
Yet the servers are acting like the traffic isn't going across.

If i do a wireshark on the local network, i can see the packets, ranging from ports 49000-59000 all hitting the box fine.
But if i then use Wireshark to monitor incoming connections from the remote site, i get only 3 packets coming across.

I'm completely stumped.

Anyone got any ideas?

[boomam]

This is annoying, not even getting conclusive results!
Firewall log shows the traffic, packet capture says there's nothing!

What the hell is going on!

##EDIT##
Silly question, but does each pf need a static route on them pointing traffic for the subnet on the other pfsense?

[phil.davis]

If you are using OpenVPN for site-to-site and have a simple network (e.g. LAN at each end, connected to pfSense, and OpenVPN site-to-site link between the 2 pfSense) then putting the correct subnets in the local and remote network fields in the OpenVPN server and client settings will make the correct routes appear. You can see the routing from Diagnostics->Routes. If you have a routing problem, then it is likely that nothing can connect across the VPN.

[boomam]

Ive exhausted every possibility with windows based diagnostics of my issue.
And now im of the opinion that it is pfsense that is as fault here.

Baring in mind that RPC traffic is generally a random port on windows network, is there any way to get pfsense to both display the logs of whats happening and allow the ports across the site link?

As reference, on both PFsenses, theres a rule on the OpenVPN interface that allows traffic from the remote subnet to the local LAN, and a rule on the LAN interface that allows traffic from the local LAN to the remote subnet.
There is also a rule on the openvpn interface on both PF's that blocks broadcast traffic (in theory) from the DCs to 255.255.255.255.

Anyone got any ideas?

[boomam]

How would I go about ensuring that ALL traffic no matter what protocol, port, etc; is passed over the openvpn site link without being blocked?
With the only thing I'd want blocking would be broadcast traffic and DHCP traffic?

As I'm wondering if the random nature of RPC and the ports its using is causing this…

[jimp]

If your rules are set to pass any protocol from/to your subnets, then pfSense wouldn't be blocking it, even if the ports are random.

If pfSense is blocking anything, it shows up in the firewall log, assuming it's not hitting your own block rules that don't have 'log' checked.

If you're curious, check the 'log' box on all your rules on the VPN and watch as you try to connect. Odds are, the firewall is passing everything that tries to go over the VPN.

Some other things to check:

• If you have multiple WANs and use policy routing, make sure that VPN traffic has a pass rule above any rule with a gateway set on it.
• Check your network settings on both systems (client and server) to make sure that Windows firewall is off or is at least considering the current network as a private or work network.
• Confirm traffic flow using the states table and packet captures

[boomam]

@jimp:

If your rules are set to pass any protocol from/to your subnets, then pfSense wouldn't be blocking it, even if the ports are random.

So unless I've set a block, it wouldn't block any traffic on openvpn?

• Confirm traffic flow using the states table and packet captures

Huh?

So if we go on the logic that as I've set non block rules on either pfsense that nothing over the openvpn link is blocked, what other causes could we be looking at for this issue. As the only consistent fault is that if the computer is at a remote site over the VPN, RPC services won't connect.
Would changing the VPN to something more basic help?

[stephenw10]

Can you get any sort of traffic between the two boxes (RDP client and server)?

Generally speaking if there is nothing appearing in logs as blocked but things are still not working I next look for a routing problem. Like, for example, the server sees the incoming requests to open a session but can't respond because, for whatever reason, it doesn't have a route back to the client.

Steve

Edit: Ooops, read RPC as RDP.  :-[ General advice still stands.

[boomam]

Every other type of traffic flows fine, from smb, to Rdp, ping, etc;
So I'm not sure, with those working, that its a routing issue as data is being passed back.
Its just RPC traffic, in this case the observes fault is with computers at the remote site being able to request certs from the CA.

[boomam]

Bump.
Any ideas?

Thanks.

[jimp]

Set all of your rules to log, see what turns up between those two PCs in the logs, pass or block.

Try to get a packet capture of the traffic on both sides, see what portions of the traffic show up on either side, if at all.

[boomam]

@jimp:

Set all of your rules to log, see what turns up between those two PCs in the logs, pass or block.

Try to get a packet capture of the traffic on both sides, see what portions of the traffic show up on either side, if at all.

Already done all that.
Its hard to get it to capture RPC traffic as the ports are random each time.
Assuming that the traffic would be seen in the "allow" rule as previously mentioned, i would assume it would show in a log.

[jimp]

It's not difficult to capture if you filter by IP and not port. No matter what port was being sent, it would still be from the same source IP to the same destination IP.

Yes, if your allow rules all log, and your block rules all log, then any traffic seen by the firewall would be logged (pass or block) - if you never see traffic hit the firewall then it wasn't sent to the firewall.

[boomam]

Running a packet capture on the remote Pf on its LAN interface, filtering on the computer im testing from.
The test to see if traffic is going across is a ping to a computer on the main site.
ICMP Packets are showing, now to test other protocols…

[boomam]

I can see lots of traffic from the test source to the test destination, ranging along a large variety of ports!
I'll do the same test on the primary PF now too…

##EDIT##
On the primary PF, i can see the traffic coming in over the OpenVPN interface.

Is there a way to show what in the capture is blocked? Or allowed?

The MSDTC test program shows that the test works from Primary server to remote server, but not remote to primary.