Can't get port forward to work correctly.

undertaker666 · Mar 10, 2022, 4:56 PM

    92.14.118.168.34931 > 192.168.55.100.59372: [udp sum ok] UDP, length 1425
18:52:09.786523 ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 64, id 44038, offset 0, flags [none], proto UDP (17), length 48)
    192.168.55.100.59372 > 92.14.118.168.34931: [udp sum ok] UDP, length 20
18:52:09.787142 , ethertype IPv4 (0x0800), length 590: (tos 0x48, ttl 53, id 45721, offset 0, flags [none], proto UDP (17), length 576)
    181.214.206.157.46682 > 192.168.55.100.59372: [udp sum ok] UDP, length 548
18:52:09.787395  ethertype IPv4 (0x0800), length 70: (tos 0x0, ttl 64, id 32769, offset 0, flags [none], proto UDP (17), length 56)
    192.168.55.100.59372 > 181.214.206.157.46682: [udp sum ok] UDP, length 28
18:52:09.788122  ethertype IPv4 (0x0800), length 590: (tos 0x48, ttl 53, id 45722, offset 0, flags [none], proto UDP (17), length 576)
    181.214.206.157.46682 > 192.168.55.100.59372: [udp sum ok] UDP, length 548

Here's a capture of the traffic coming in to that machine on that port. TCP is non-existent, it's all UDP for some reason. Even though the client is set to use TCP and UTP (which is UDP).
Other than that, I don't see anything else out of this capture.

viragomann · Mar 10, 2022, 5:15 PM

@undertaker666
So the traffic seems to flow well. That's all pfSense can do.

Actually I didn't even have a port forward on my previous router, I had UPnP enabled on my client and everything worked.

Would be worth to mention. Maybe UPnP opens more than only one port.

I'm using a BitTorrent client behind pfSense. I've forwarded two ports, one is TCP for talking to other client, the other UDP for the tracker, and it works flawlessly.

However, it's also possible to enable UPnP on pfsense, but that's only recommended if you know what you do and restrict the access to known clients only.

undertaker666 · Mar 10, 2022, 5:18 PM

@viragomann Yeah UPnP is not secure, and it's better to stick to just 1 port.

I use QBittorrent, and there's only one port that can be set, so both use the same port, but only UDP is used for some reason.

undertaker666 · Mar 13, 2022, 4:46 AM

@undertaker666 OK, did a bit more digging, and found out that pfBlockerNG was ignoring the rule order, and was still blocking the traffic that matched that NAT Rule.

Turned it off, but still not seeding well, so did a port test from both pfSense and outside the network using an online service.

External service said the port was closed, pfSense said the port was closed when the destination was the WAN IP, but when it was the LAN IP it said it was open.

Now why would that be?

lolipoplo · Mar 13, 2022, 9:34 AM

@undertaker666

easiest way to debug, run tcpdump on pflog0, you are going to see all of the blocked packets and according to which rule they are blocked, if they are indeed blocked.

tcpdump -nettti pflog0 port 8010 and then run an external syn scan on 8010

viragomann · Mar 13, 2022, 10:01 AM

@undertaker666 said in Can't get port forward to work correctly.:

Turned it off, but still not seeding well

Maybe QBittorrent needs the outbound port to be static.
Many consumer routers does this, but pfSense use random outbound ports by default. You may have to add an outbound NAT rule to achieve this.

pfSense said the port was closed when the destination was the WAN IP, but when it was the LAN IP it said it was open.

You cannot use the port check on pfSense for the firewall itself. You can only check other destinations.

lolipoplo · Mar 13, 2022, 2:48 PM

@viragomann if you have port forwarding working, outbound NAT doesn't matter. Making it static just helps with hole punching

viragomann · Mar 13, 2022, 2:48 PM

@lolipoplo said in Can't get port forward to work correctly.:

if you have port forwarding working, outbound NAT doesn't matter.

Some programs need this like several games. Maybe QBittorrent as well. I don't know how it works, as I mentioned above.
But it's for sure that QBittorrent also make upstream connections and these have nothing to do with forwarding at all.

So a presume, you're knowing well QBittorrent and can possibly give more reliable infos.

undertaker666 · Mar 13, 2022, 5:42 PM

@viragomann @lolipoplo said in Can't get port forward to work correctly.:

tcpdump -nettti pflog0 port 8010 and then run an external syn scan on 8010

Actually, once I turned off pfBlockerNG, parsec managed to connect to a host game. So those ports are fine.

The problem is with the torrent, it's better, it's actually seeding now, and it actually reached 300 KB/s, but it does not stay at those speeds, and there's more downtime than actual seeding.

Maybe QBittorrent as well

Well, I already had outbound set up, so that's not what's stopping it. The question is why is pfBlockerNG ignoring the rule order, and even with it turned off, why are connections not sticking as they used to?

lolipoplo · Mar 13, 2022, 8:06 PM

@undertaker666
So have you tried pflog as I suggested?

Actually run tcpdump on wan port at the same time to compare incoming and rule matching

undertaker666 · Mar 14, 2022, 3:53 AM

@lolipoplo No, because you suggested running the tcpdump on the parsec port, and that is solved by just turning off pfBlockerNG.

I could run the tcpdump on the qbittorrent port and see what happens.

undertaker666 · Mar 14, 2022, 4:19 AM

@undertaker666

tcpdump -nettti pppoe0 port 59372 -vv
tcpdump: listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
 00:00:00.000000 AF IPv4 (2), length 56: (tos 0x2,ECT(0), ttl 63, id 28582, offset 0, flags [none], proto TCP (6), length 52)
    My-WAN.31533 > 172.16.1.0.59372: Flags [SEW], cksum 0xbecc (correct), seq 2428356104, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
 00:00:00.013465 AF IPv4 (2), length 48: (tos 0x2,ECT(0), ttl 254, id 6409, offset 0, flags [DF], proto TCP (6), length 44)
    172.16.1.0.59372 > My-WAN.31533: Flags [S.], cksum 0x9772 (correct), seq 4044710442, ack 2428356105, win 4200, options [mss 1400], length 0
 00:00:00.000241 AF IPv4 (2), length 44: (tos 0x0, ttl 63, id 28583, offset 0, flags [none], proto TCP (6), length 40)
    My-WAN.31533 > 172.16.1.0.59372: Flags [.], cksum 0xc3ca (correct), seq 1, ack 1, win 64400, length 0
 00:00:00.003992 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28584, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.302961 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28585, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.394904 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28586, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.699004 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28587, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.589524 AF IPv4 (2), length 56: (tos 0x2,ECT(0), ttl 63, id 36468, offset 0, flags [none], proto TCP (6), length 52)
    My-WAN.60033 > 172.16.2.0.59372: Flags [SEW], cksum 0x442d (correct), seq 3544223184, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
 00:00:00.013934 AF IPv4 (2), length 48: (tos 0x2,ECT(0), ttl 254, id 9436, offset 0, flags [DF], proto TCP (6), length 44)
    172.16.2.0.59372 > My-WAN.60033: Flags [S.], cksum 0x3e54 (correct), seq 3004006065, ack 3544223185, win 4200, options [mss 1400], length 0
 00:00:00.000178 AF IPv4 (2), length 44: (tos 0x0, ttl 63, id 36469, offset 0, flags [none], proto TCP (6), length 40)
    My-WAN.60033 > 172.16.2.0.59372: Flags [.], cksum 0x6aac (correct), seq 1, ack 1, win 64400, length 0
 00:00:00.005371 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36470, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.385924 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36471, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.304076 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28593, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.095936 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36472, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.218016 AF IPv4 (2), length 44: (tos 0x0, ttl 254, id 10875, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.1.0.59372 > My-WAN.31533: Flags [R.], cksum 0xbf57 (correct), seq 1, ack 1, win 0, length 0
 00:00:00.476972 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36477, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:01.204971 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36480, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.323059 AF IPv4 (2), length 44: (tos 0x0, ttl 254, id 13669, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.2.0.59372 > My-WAN.60033: Flags [R.], cksum 0x6639 (correct), seq 1, ack 1, win 0, length 0

Here's a small output of that command.

The weird thing is, when I do canyouseeme, it shows me another public IP than what is shown in the WAN interface on my pfSense box.
Both are public IPs, and I tested both, and now they say closed.

lolipoplo · Mar 14, 2022, 4:19 AM

@undertaker666

if you aren't willing to do tcpdump on pflog0 you can't see how your packets get blocked

undertaker666 · Mar 14, 2022, 4:39 AM

@lolipoplo Look up

lolipoplo · Mar 14, 2022, 4:53 AM

@undertaker666 where?

undertaker666 · Mar 14, 2022, 5:05 AM

@undertaker666 said in Can't get port forward to work correctly.:

@undertaker666

tcpdump -nettti pppoe0 port 59372 -vv
tcpdump: listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
 00:00:00.000000 AF IPv4 (2), length 56: (tos 0x2,ECT(0), ttl 63, id 28582, offset 0, flags [none], proto TCP (6), length 52)
    My-WAN.31533 > 172.16.1.0.59372: Flags [SEW], cksum 0xbecc (correct), seq 2428356104, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
 00:00:00.013465 AF IPv4 (2), length 48: (tos 0x2,ECT(0), ttl 254, id 6409, offset 0, flags [DF], proto TCP (6), length 44)
    172.16.1.0.59372 > My-WAN.31533: Flags [S.], cksum 0x9772 (correct), seq 4044710442, ack 2428356105, win 4200, options [mss 1400], length 0
 00:00:00.000241 AF IPv4 (2), length 44: (tos 0x0, ttl 63, id 28583, offset 0, flags [none], proto TCP (6), length 40)
    My-WAN.31533 > 172.16.1.0.59372: Flags [.], cksum 0xc3ca (correct), seq 1, ack 1, win 64400, length 0
 00:00:00.003992 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28584, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.302961 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28585, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.394904 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28586, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.699004 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28587, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.589524 AF IPv4 (2), length 56: (tos 0x2,ECT(0), ttl 63, id 36468, offset 0, flags [none], proto TCP (6), length 52)
    My-WAN.60033 > 172.16.2.0.59372: Flags [SEW], cksum 0x442d (correct), seq 3544223184, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
 00:00:00.013934 AF IPv4 (2), length 48: (tos 0x2,ECT(0), ttl 254, id 9436, offset 0, flags [DF], proto TCP (6), length 44)
    172.16.2.0.59372 > My-WAN.60033: Flags [S.], cksum 0x3e54 (correct), seq 3004006065, ack 3544223185, win 4200, options [mss 1400], length 0
 00:00:00.000178 AF IPv4 (2), length 44: (tos 0x0, ttl 63, id 36469, offset 0, flags [none], proto TCP (6), length 40)
    My-WAN.60033 > 172.16.2.0.59372: Flags [.], cksum 0x6aac (correct), seq 1, ack 1, win 64400, length 0
 00:00:00.005371 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36470, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.385924 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36471, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.304076 AF IPv4 (2), length 205: (tos 0x0, ttl 63, id 28593, offset 0, flags [none], proto TCP (6), length 201)
    My-WAN.31533 > 172.16.1.0.59372: Flags [P.], cksum 0xc832 (correct), seq 1:162, ack 1, win 64400, length 161
 00:00:00.095936 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36472, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.218016 AF IPv4 (2), length 44: (tos 0x0, ttl 254, id 10875, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.1.0.59372 > My-WAN.31533: Flags [R.], cksum 0xbf57 (correct), seq 1, ack 1, win 0, length 0
 00:00:00.476972 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36477, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:01.204971 AF IPv4 (2), length 497: (tos 0x0, ttl 63, id 36480, offset 0, flags [none], proto TCP (6), length 493)
    My-WAN.60033 > 172.16.2.0.59372: Flags [P.], cksum 0x62af (correct), seq 1:454, ack 1, win 64400, length 453
 00:00:00.323059 AF IPv4 (2), length 44: (tos 0x0, ttl 254, id 13669, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.2.0.59372 > My-WAN.60033: Flags [R.], cksum 0x6639 (correct), seq 1, ack 1, win 0, length 0

Here's a small output of that command.

The weird thing is, when I do canyouseeme, it shows me another public IP than what is shown in the WAN interface on my pfSense box.
Both are public IPs, and I tested both, and now they say closed.

here

lolipoplo · Mar 14, 2022, 5:07 AM

@undertaker666 I only see pppoe0, where's pflog0?

undertaker666 · Mar 14, 2022, 5:12 AM

@lolipoplo pflog0 was empty, no matches.

lolipoplo · Mar 14, 2022, 5:20 AM

@undertaker666

pfsense has logging on for all of the default block/reject rules

If pflogs is empty, this probably means your port forwarding is working provided you do not have silent block/reject rules.

one more sanity check, go to your associated pass rules for nat and enable logging, then listen to pflog0 again to make sure they are matched

undertaker666 · Mar 14, 2022, 5:52 PM

@lolipoplo Still empty, and I could see almost nothing hits the rule in the auth screen as well.

Even though I ran a check from an online service to see if it's open or not, it did not register, and the test came back as closed.

Something is very weird here. I already did such a check before that returned open, so I'm not sure what happened.