[SOLVED] 2.4.4-p3 Weird time and network phenomena on Odroid H2

EveningStarNM

UPDATE: Please see my reply to this post at the end. This is problem caused by FreeBSD not reading the clock on the Odroid H2 correctly. A solution was found.

Note: I'm editing this from its original version because I've done some more investigation and have found some strange things.

I have two pfSense machines configured behind a Centurylink DSL modem serving two disjoint LANs. (I've also tried this behind a Comcast cable modem).

The older pfsense device (call it "pf1") has worked well for years. The new one ("pf2") is a fresh installion on an Odroid H2, a quad-core Intel Celeron J4105 processor, 4GB, 64GB eMMC machine with two gigabit ports.

Resolver is configured identically on both pfsense machines. The only other configuration on pf2 is DHCP for the LAN, and the usual initial settings.

WEIRD PHENOMENON #1

I can get to the internet from workstations behind both pfsense machines. However, while pf1 can check for packages, pf2 cannot. For instance cannot g Package Manager always returns the message "Unable to retrieve package information".

I have read the pinned threads and some others that describe problems similar to this, but none of the solutions work. (For instance, I'm not behind an upstream proxy. This is actually a very simple setup. Even the firewall rules are still the defaults.) I've tried different external DNS servers (Cloudflare, Comcast, Centurylink) and changed Resolver's settings appropriately. Ive tried every permutation of Resolver's settings I can think of. One post suggested upgrading pkg, so I opened a command prompt and tried "pkg upgrade pkg". This was the result:

Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
pkg: https://pkg.pfsense.org/pfSense_v2_4_4_amd64-core/meta.txz: Authentication error
repository pfSense-core has no meta file, using default settings
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
pkg: https://pkg.pfsense.org/pfSense_v2_4_4_amd64-core/packagesite.txz: Authentication error
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
pkg: https://pkg.pfsense.org/pfSense_v2_4_4_amd64-pfSense_v2_4_4/meta.txz: Authentication error
repository pfSense has no meta file, using default settings
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
Certificate verification failed for /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.netgate.com
34405284920:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
pkg: https://pkg.pfsense.org/pfSense_v2_4_4_amd64-pfSense_v2_4_4/packagesite.txz: Authentication error
Unable to update repository pfSense
Error updating repositories!

I'm aware that pkg.pfsense.org is reached through a SRV record and redirects to files01.netgate.com, but I don't think name resolution has anything to do with it. That leads to:

WEIRD PHENOMENON #2

If I log on to the pfSense web console and use Diagnostica \ DNS Lookup for any FQDN on the internet, Resolver returns the correct address.

However, if I use the nslookup from the command line using pfsense as the server, lookups fail.

Since DNS lookups sometimes fail if the time is off, I did some more investigating. This led to the final (so far)

EXTREMELY WEIRD PHENOMENON #3

The time is set correctly in the machine's UEFI BIOS. However, the time shown in pfSense or from the command line (using DATE) is always eleven hours behind.

I am in the America/Denver time zone, which I have configured in System \ General Settings. I might understand a 7 hour difference, but eleven? (It was also at this point that I discovered that pfsense can't contact any NTP servers.)

Another weird part of this is that, if I run DATE from the command line, wait ten seconds, and run DATE again, it reports only a three- or four-second difference in time.

I have no idea what's going on here. The computer itself appears to be okay (it runs Windows 10 PE just fine), but pfsense isn't getting the right time from the system clock, or it's interpreting it incorrectly. I think I have to solve this problem before I can solve any of the others, but I'm stumped.

Meanwhile, the other pfsense devise, which runs the same version but on different hardware, just keeps humming along.

I'll be grateful for any help.

jimp

Issue #1 may be related to the clock -- It's failing to validate the certificate, which may mean that not just your time but the date may be off.

Issue #2 -- Never use nslookup for DNS testing. It's antiquated. Use tools like drill, dig, and host. There is not enough information to go by here, though. The output from those commands would be more useful -- Though again if you have clock issues, and the system is using resolver mode, and DNSSEC is enabled, then the clock problem will cause this to fail as well.

Issue #3 is probably either hardware or a compatibility issue between your hardware and FreeBSD. First step is a BIOS update. Second step might be to try a 2.5.0 snapshot to see if FreeBSD 12 copes better there. You could probably also try fiddling with the timecounter sysctls, that should be in the docs, but ultimately it's probably low quality hardware to blame here.

Given the symptoms I'd say all the problems are related, and it's the clock to blame.

EveningStarNM

First of all, /I'm/ antiquated. (I mean, I know how to use ip, but I often find nano more flexible and useful.) If it does what I need it to do, I'm fine with it.

But I think you're right about the clock. I just don't know why it's having a problem under FreeBSD but not Windows. I'm going to talk to Hardkernel to see if they've got any ideas, but I don't have much hope that they can help.

EveningStarNM

SOLVED

FreeBSD does not read the hardware clock correctly from the Odroid H2. This causes all sorts of weird problems.

johnsond in the Odroid forum tested the board and found a solution, all of which he described in this post: https://forum.odroid.com/viewtopic.php?t=33911#p261986

Go to System > Advanced > System Tunables.
Add New
Tunable Name: kern.timecounter.hardware
Value: ACPI-fast
Description: TSC-low doesn't work on Odroid-H2

I've testing this on two Odroid H2s. After rebooting, the boards were able to sync with NTP, the time was displayed correctly, and pfSense could contact its repository.