Cannot get the new packet flow to export flows

j.koopmann

Hi,

I enabled both options in the menu, created an exporter and saved. My expectation would have been to see outgoing packets to port 2055 from the firewall and my graylog to receive them....but.... nothing. No packets are being sent. No errors. Rebooted. pflow kernel module is loaded. I am at a loss.

What am I missing?

kprovost

Make sure there are new states to be exported. States that were created prior to enabling pflow will not be exported.

State information is also not created until the state is cleaned up (i.e. the connection is finished), and there may be another delay of up to 30 seconds on top of that.

j.koopmann

@kprovost Well I am creating new states all the time. I waited now 12h and not a single packet.... I was aware of the necessity of state changes but after a reboot new states would have been there as well. .-) So that is not it unfortunately.

kprovost

It may be worth looking at your firewall log, in case that's causing the packets to get dropped.

You can confirm that the pflow code is getting called with dtrace -n 'fbt::export_pflow:'. That'll generate output like this when states are cleaned up and handed to pflow for export:

dtrace: description 'fbt::export_pflow:' matched 2 probes
CPU     ID                    FUNCTION:NAME
  1  86463               export_pflow:entry
  1  86464              export_pflow:return
  1  86463               export_pflow:entry
  1  86464              export_pflow:return

Terminate with Ctrl+C.

j.koopmann

@kprovost Thanks.

I can see the dtrace output and can confirm that part. But still no packets being sent out.

I use my LAN interface as a source in the exporter and send packets to another server within that LAN segment. In theory no firewall rule should block this should it? LAN is a VLAN on a lagg. Not sure if this makes a difference. I tried a management interface as well which is a physical interface with no change. tcpdump on the pfsense itself does not see packets on 2055 on any interface I checked.

kprovost

@j-koopmann said in Cannot get the new packet flow to export flows:

In theory no firewall rule should block this should it?

That depends on your firewall rules. It's perfectly possible to block outbound packets from the device itself.
The default rules should allow this, and I'd also expect such drops to get logged.

It's also worth checking that things are configured correctly.

Try pflowctl -l -v and netstat -s -p pflow.
The first will show us what pflow targets are set up, the second should show us counters of how many flows and packets have been transmitted.

j.koopmann

@kprovost

# pflowctl -l -v
pflow0: version 5 domain 1 src 10.10.101.254 dst 10.10.101.133:2055
	socket: connected
# netstat -s -p pflow
pflow:
	1563165 flows sent
	52106 packets sent
	0 send failed due to mbuf memory error
	0 send error
#

Same subnet. On the LAN interface. No blocking rule. Last ruleset for that interface is "allow all"...

kprovost

That all looks good. Did you check the filter log for dropped packets to the pflow port?

We can probe a bit deeper with this script (save it on the pfsense box, chmod +x it, then execute it).

#!/usr/sbin/dtrace -s

BEGIN
{
    self->in = 0;
}

fbt::pflow_output_process:entry
{
    printf(" ");
    self->in = 1;
}

fbt::pflow_output_process:return
{
    printf(" ");
    self->in = 0;
}

fbt::sosend:entry
/ self->in == 1 /
{
    printf("Send to %s:%d", inet_ntoa(&((struct sockaddr_in *)arg1)->sin_addr.s_addr), ntohs(((struct sockaddr_in *)arg1)->sin_port));
}

fbt::sosend:return
/ self->in == 1 /
{
    printf("=> %d", arg1);
}

That'll show us every time the timeout function that actually transmits gets called, and it'll also show us where it sends packets to, and if that succeeds.

Expect output like

dtrace: script './pflow.d' matched 5 probes
CPU     ID                    FUNCTION:NAME
  8  84190       pflow_output_process:entry
  8  54845                     sosend:entry Send to 10.0.2.1:2055
  8  54846                    sosend:return => 0
  8  84191      pflow_output_process:return

j.koopmann

@kprovost let me start out with two things:

Many thanks for your assistance. I really appreciate it.
I obviously need more coffee... The target IP was a typo. I have not spotted that earlier since all my tcpdump were filtering for port 2055 only and of course I would have suspected to see outgoing packets of some sort (which I still do not understand why this was not the case).

However after fixing the typo netflows are being received now and I feel extremely stupid for not having that spotted earlier. Many thanks! Apologies!