Filter needs manual reload to open incoming ports

phil.davis

This might be the same underlying problem as reporting in thread https://forum.pfsense.org/index.php/topic,71555.0.html - perhaps all the messages sent to check_reload_status are failing to get delivered/actioned?
In the logs anywhere are there messages like:

Jan 18 21:54:25 	apinger: command (/usr/local/sbin/pfSctl -c 'service reload dyndns OPT1GW' -c 'service reload ipsecdns' -c 'service reload openvpn OPT1GW' -c 'filter reload' ) exited with status: 255

(They won't be from "apinger" but from other software components trying to do 'filter reload')?

doktornotor

@phil.davis:

This might be the same underlying problem as reporting in thread https://forum.pfsense.org/index.php/topic,71555.0.html - perhaps all the messages sent to check_reload_status are failing to get delivered/actioned?

Have the same feeling.

jimp

Try it with Thu Jan 23 17:15:05 EST 2014 or later.

doktornotor

@jimp:

Try it with Thu Jan 23 17:15:05 EST 2014 or later.

Made things much worse, in fact it fails to complete upgrade. After reboot it freezes indefinitely on

fetch: http://files.pfsense.org/lists/fullbogons-ipv4.txt: No address record
fetch: http://files.pfsense.org/lists/fullbogons-ipv6.txt: No address record

I already pointed out that these bogon lists got broken with the website upgrade many days ago.

jimp

Hmm, those URLs load fine for me, and I didn't have any trouble upgrading a couple VMs just now. There was a problem with that server on the day you posted the other message, and again overnight, but it's OK at the moment.

doktornotor

@jimp:

Hmm, those URLs load fine for me, and I didn't have any trouble upgrading a couple VMs just now. There was a problem with that server on the day you posted the other message, and again overnight, but it's OK at the moment.

The only thing I can ever download from the page since the website upgrade is bogon-bn-nonagg.txt and the MD5 files. Anything else never loads. Is 2001:500:b::1 the IP for this? In that case, afraid it's completely broken by https://redmine.pfsense.org/issues/2762 and plain unusable here with IPv6. Tried with multiple boxes, multiple providers, simply never ever loads. Fragmented packets blocked by firewall. As noted on the above linked bug, I have severe issues with all pfSense sites due to this and need an urgent workaround for the pf stupidity with dropping completely legit traffic. >:(

maverick_slo

Huh works fine for me aswell…
I upgraded 2 boxes without a hitch...

jimp

Hmm, that IP wouldn't be it. That's a DNS server somewhere in Canada. c0.org.afilias-nst.info

$ host files.pfsense.org
files.pfsense.org has address 66.111.2.167
files.pfsense.org has IPv6 address 2610:1c0:1:25::55

doktornotor

Well, whatever….


# wget http://files.pfsense.org/lists/fullbogons-ipv4.txt
--2014-01-24 16:02:27--  http://files.pfsense.org/lists/fullbogons-ipv4.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
HTTP request sent, awaiting response...

Zzzzzzzzzzzzzzzzzzzz.


# wget http://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 16:03:38--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
HTTP request sent, awaiting response...

Zzzzzzzzzzzzzzzzzzzz.


# wget http://files.pfsense.org/lists/bogon-bn-nonagg.txt
--2014-01-24 16:04:13--  http://files.pfsense.org/lists/bogon-bn-nonagg.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 185 [text/plain]
Saving to: `bogon-bn-nonagg.txt'

100%[====================================================================================================================================================================================================>] 185         --.-K/s   in 0s

2014-01-24 16:04:14 (7.91 MB/s) - `bogon-bn-nonagg.txt' saved [185/185]

WTF?!

The other issue here obviously being that this failure should not freeze boot, created a new issue for that - https://redmine.pfsense.org/issues/3412

jimp

Yeah that is odd. I'll have to check and see if anything else changed on there.

jimp

Out of curiosity, are you able to fetch those over HTTPS instead of HTTP?

doktornotor

@jimp:

Out of curiosity, are you able to fetch those over HTTPS instead of HTTP?

Does not seem so… In fact, I cannot even fetch the bogon-bn-nonagg.txt one in that way. Double WTF.


# wget --no-check-certificate https://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 16:40:30--  https://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:443... connected.

Zzzzzzzzzzzzzzzzzzzz.


# wget --no-check-certificate https://files.pfsense.org/lists/bogon-bn-nonagg.txt
--2014-01-24 16:41:30--  https://files.pfsense.org/lists/bogon-bn-nonagg.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:443... connected.

Zzzzzzzzzzzzzzzzzzzz.

Just in case it might help, the debug one:


# wget -v http://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 16:46:50--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
HTTP request sent, awaiting response... ^C
[admin@nas ~]# wget --help | grep deb
  -d,  --debug               print lots of debugging information.
[admin@nas ~]# wget -d http://files.pfsense.org/lists/fullbogons-ipv6.txt
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 16:47:14--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Caching files.pfsense.org => 2610:1c0:1:25::55 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
Created socket 3.
Releasing 0x01945598 (new refcount 1).

---request begin---
GET /lists/fullbogons-ipv6.txt HTTP/1.0
User-Agent: Wget/1.12 (linux-gnueabi)
Accept: */*
Host: files.pfsense.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...

Zzzzzzzzzzzzzzzzzzzz.


# wget -d --no-check-certificate https://files.pfsense.org/lists/fullbogons-ipv6.txt
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 16:48:17--  https://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Caching files.pfsense.org => 2610:1c0:1:25::55 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:443... connected.
Created socket 3.
Releasing 0x00627948 (new refcount 1).
Initiating SSL handshake.

Zzzzzzzzzzzzzzzzzzzz.


# wget -d http://files.pfsense.org/lists/bogon-bn-nonagg.txt
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 16:49:44--  http://files.pfsense.org/lists/bogon-bn-nonagg.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Caching files.pfsense.org => 2610:1c0:1:25::55 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:80... connected.
Created socket 3.
Releasing 0x01a61598 (new refcount 1).

---request begin---
GET /lists/bogon-bn-nonagg.txt HTTP/1.0
User-Agent: Wget/1.12 (linux-gnueabi)
Accept: */*
Host: files.pfsense.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 24 Jan 2014 15:49:44 GMT
Content-Type: text/plain
Content-Length: 185
Last-Modified: Tue, 21 Jan 2014 06:01:01 GMT
Connection: keep-alive
ETag: "52de0d1d-b9"
Accept-Ranges: bytes

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 185 [text/plain]
Saving to: `bogon-bn-nonagg.txt'

100%[====================================================================================================================================================================================================>] 185         --.-K/s   in 0s

2014-01-24 16:49:44 (8.44 MB/s) - `bogon-bn-nonagg.txt' saved [185/185]

Worked.


# wget -d --no-check-certificate https://files.pfsense.org/lists/bogon-bn-nonagg.txt
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 16:50:40--  https://files.pfsense.org/lists/bogon-bn-nonagg.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::55, 66.111.2.167
Caching files.pfsense.org => 2610:1c0:1:25::55 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::55|:443... connected.
Created socket 3.
Releasing 0x01a8c948 (new refcount 1).
Initiating SSL handshake.

Zzzzzzzzzzzzzzzzzzzz.
:o :o :o

doktornotor

Well… and guess what - it just works over IPv4. HTTP or HTTPS does not matter.


# wget -4 -d --no-check-certificate https://files.pfsense.org/lists/fullbogons-ipv6.txt
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 17:04:29--  https://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 66.111.2.167
Caching files.pfsense.org => 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|66.111.2.167|:443... connected.
Created socket 3.
Releasing 0x00fe6740 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00fe6a10
certificate:
  subject: /OU=Domain Control Validated/CN=*.pfsense.org
  issuer:  /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
WARNING: cannot verify files.pfsense.org's certificate, issued by `/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2':
  Unable to locally verify the issuer's authority.

---request begin---
GET /lists/fullbogons-ipv6.txt HTTP/1.0
User-Agent: Wget/1.12 (linux-gnueabi)
Accept: */*
Host: files.pfsense.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 24 Jan 2014 16:04:29 GMT
Content-Type: text/plain
Content-Length: 738156
Last-Modified: Tue, 21 Jan 2014 06:01:01 GMT
Connection: keep-alive
ETag: "52de0d1d-b436c"
Accept-Ranges: bytes

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt'

100%[===================================================================================================================================================================================================>] 738,156      730K/s   in 1.0s

2014-01-24 17:04:31 (730 KB/s) - `fullbogons-ipv6.txt' saved [738156/738156]


# wget -4 -d http://files.pfsense.org/lists/fullbogons-ipv6.txt
DEBUG output created by Wget 1.12 on linux-gnueabi.

URI encoding = `ANSI_X3.4-1968'
--2014-01-24 17:04:57--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 66.111.2.167
Caching files.pfsense.org => 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|66.111.2.167|:80... connected.
Created socket 3.
Releasing 0x003eb580 (new refcount 1).

---request begin---
GET /lists/fullbogons-ipv6.txt HTTP/1.0
User-Agent: Wget/1.12 (linux-gnueabi)
Accept: */*
Host: files.pfsense.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 24 Jan 2014 16:04:57 GMT
Content-Type: text/plain
Content-Length: 738156
Last-Modified: Tue, 21 Jan 2014 06:01:01 GMT
Connection: keep-alive
ETag: "52de0d1d-b436c"
Accept-Ranges: bytes

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt.1'

100%[===================================================================================================================================================================================================>] 738,156      354K/s   in 2.0s

2014-01-24 17:05:00 (354 KB/s) - `fullbogons-ipv6.txt.1' saved [738156/738156]

jimp

Can you clear your DNS cache (if you have one) and try again? It should be on ::56 now. cmb noticed some issues routing to ::55 but ::56 seems to be OK at the moment.

doktornotor

Well yes, but the files are not there… :D


#  wget -v http://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 18:03:04--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::56, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::56|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2014-01-24 18:03:05 ERROR 404: Not Found.

P.S. Thanks for looking into the problem! 8)

gogol

@doktornotor:

Well yes, but the files are not there… :D

Yes, they are…, oh no they are not..., ah there they are again..., oh my, gone again. :-[

Edit: I was finally able to download packages by disabling my ipv6 tunnel. Weird. Probably Pfsense was connecting ipv6 which didn't work?

doktornotor

@gogol:

Yes, they are…, oh no they are not..., ah there they are again..., oh my, gone again. :-[
[/quote]

Yes, they are, right now. ;D

IPv6:


# wget -v http://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 19:29:30--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::57, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::57|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt'

100%[===================================================================================================================================================================================================>] 738,156      222K/s   in 3.3s

2014-01-24 19:29:33 (222 KB/s) - `fullbogons-ipv6.txt' saved [738156/738156]

# wget -v --no-check-certificate https://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 19:29:49--  https://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 2610:1c0:1:25::57, 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|2610:1c0:1:25::57|:443... failed: Connection refused.
Connecting to files.pfsense.org (files.pfsense.org)|66.111.2.167|:443... connected.
WARNING: cannot verify files.pfsense.org's certificate, issued by `/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt.1'

100%[===================================================================================================================================================================================================>] 738,156      744K/s   in 1.0s

2014-01-24 19:29:50 (744 KB/s) - `fullbogons-ipv6.txt.1' saved [738156/738156]

IPv4:


# wget -4 -v http://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 19:31:06--  http://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|66.111.2.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt.3'

100%[===================================================================================================================================================================================================>] 738,156      523K/s   in 1.4s

# wget -4 -v --no-check-certificate https://files.pfsense.org/lists/fullbogons-ipv6.txt
--2014-01-24 19:30:41--  https://files.pfsense.org/lists/fullbogons-ipv6.txt
Resolving files.pfsense.org (files.pfsense.org)... 66.111.2.167
Connecting to files.pfsense.org (files.pfsense.org)|66.111.2.167|:443... connected.
WARNING: cannot verify files.pfsense.org's certificate, issued by `/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 738156 (721K) [text/plain]
Saving to: `fullbogons-ipv6.txt.2'

100%[===================================================================================================================================================================================================>] 738,156      721K/s   in 1.0s

2014-01-24 19:30:43 (721 KB/s) - `fullbogons-ipv6.txt.2' saved [738156/738156]

All good now, thanks jimp and else everyone involved! 8) 8) 8)

Guest

@jimp:

Try it with Thu Jan 23 17:15:05 EST 2014 or later.

It was the same issue after the upgrade.

The filter needed a manual reload and the apinger service needed to be restarted.

Guest

Same for. 8.3-RELEASE-p14 FreeBSD 8.3-RELEASE-p14 #1: Sat Jan 25 11:19:23 EST 2014 root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64

maverick_slo

Huh I don`t have this problems since THU snapshot…
Today I upgraded to SAT snapshot and all is OK...