error - IP packet with unknown IP version=15 seen

seewolf · Aug 26, 2021, 10:57 AM

after the latest update to 2.5.2, openvpn in pfsense started to fill whole logs with the message
'IP packet with unknown IP version=15 seen'
and we can not figure why.

we did not change anything before/after the update regarding the openvpn configuration.

the compression is not the issue. it was left blank in the client configuration and pushed from the server. but even when i added the compression configuration in the client conf file (same as on the server side of course), it did not help.
regardless if i push or do not push the compression configuration.

the biggest issue is that the errors start even when no one is connected, after i restart the server service.

i updated another firewall that has the same server configurations, and that one is fine.

could anyone point us in the right direction to figure what is going on?

thank you for your help :).

jimp · Aug 26, 2021, 12:38 PM

Can you post the full and exact log message?

Normally that kind of message is due to the NIC receiving corrupt packets, indicating a problem that could be more hardware-related (either on the firewall or whatever it's connected to, switch, modem, cabling, etc) but that isn't always the case.

It could be that the upgrade didn't fully run or complete and you have a mismatch between the kernel and other parts of the system, too.

I would take a backup and wipe/reload that system to be sure it's on a consistent state with 2.5.2.

seewolf · Aug 26, 2021, 4:09 PM

thank you for a quick reply.

the log is full of
Aug 26 17:32:24 openvpn 47390 IP packet with unknown IP version=15 seen

out of 200mb of logs, cca 194mb are just that line (the pid changes of course).
both of the openvpn servers instances (the first one is a legacy server waiting to be killed, but we cant do that before we sort this out) experience the same issue after the update.

and that is a problem because when we need to find a certain entry in the logs, we need to download the full log, and then search 200mb text file for the entry.

in the rest of the interfaces gui, there are no errors on those interfaces.
and on the same master switch there is another pfsense box connected, and it has no error in the openvpn logs (and the configuration is the same as the new server instance).

we can try a reinstall, but folks will not be happy :).
some are working remotely and some are working from the office/lab.
the update was gradual, from 2.4.3, to 2.5.0, to 2.5.1, to 2.5.2.. and after the last update this started (or the one before, cant be 100 percent certain).

before the reinstall, any other suggestions?

jimp · Aug 26, 2021, 7:26 PM

Searching around that is almost certainly due to compression settings between the client and server. The important thing to note is that not only do the configuration options chosen for compression matter, but the OpenVPN version of both is also significant.

For example, the default behavior of OpenVPN changed between OpenVPN 2.4.x and OpenVPN 2.5.x so something like omitting the directive will mean different things in each case.

So the first thing to do is compare the exact generated/in-use configuration of both server and client and see what shows in both.

seewolf · Aug 26, 2021, 7:37 PM

the servers are configured (this is the new option from pfsense 2.5.x i think)
Decompress incomming, do not compress outgoing (Asymmetric)

and compression is
No LZO Compression [Legacy style, comp-lzo no]

all clients are probably 2.4.x

the config file did not contain 'comp-lzo no', but the compression was pushed from the server.
then i added the 'comp-lzo no' to the config file just for the test. nothing happened.

i tried all permutations of 'Allow compression' and 'Compression' settings that made sense, and the most i did was that the clients could not connect anymore.

the biggest question is, why does the error start when i restart openvpn server service, and no one is connected. just at random at some point it starts.
or the error starts when the server is running, but no one is connected. i kick/wait for no one to be connected, and still, the error persists.

for example, at this moment no one is connected, and the logs are full of
Aug 26 21:30:01 openvpn 47390 IP packet with unknown IP version=15 seen

jimp · Aug 26, 2021, 7:43 PM

On the server side what you probably want is:

Allow Compression: Asymmetric
Compression: Disable Compression [Omit Preference]

And on the clients either remove the comp-lzo statements or maybe try comp-lzo no, depending on the version.

As to why you are seeing those log messages when nobody is connected, I'm not sure. Something has to be sending the packets into OpenVPN for that to be logged.

seewolf · Aug 26, 2021, 7:51 PM

lets try right now :).

allow compression - asymmetric as it was.
compression - disable compression

the 'comp-lzo no' is still in the config file of a colleague who will try to connect.
his client is 2.4.11

and we get this
Aug 26 21:47:43 openvpn 23648 PUBLIC_IP:62943 WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'
Aug 26 21:47:44 openvpn 23648 USERNAME/PUBLIC_IP:62943 IP packet with unknown IP version=15 seen

same thing but different? i see that now compression is not the same.

the only other warning i see is
Aug 26 21:47:43 openvpn 23648 PUBLIC_IP:62943 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1549', remote='link-mtu 1550'

also, when no one is connected i dont see failed or successful connection attempts when the errors start, that is what is wierd.

seewolf · Aug 26, 2021, 8:12 PM

small update.

i reverted the settings, and no one tried to connect since the services started (and is not currently connected).

i get this again
Aug 26 22:00:01 openvpn 12215 IP packet with unknown IP version=15 seen
and
Aug 26 22:00:19 openvpn 49182 IP packet with unknown IP version=15 seen

both server (the legacy one and the new one) instances show that.

the legacy one is
12215 root 20 0 17M 6324K select 6 0:00 0.00% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn{openvpn}

and the new one is
49182 root 20 0 19M 6580K select 4 0:00 0.00% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2/config.ovpn{openvpn}

seewolf · Aug 26, 2021, 8:18 PM

small update 2

when a client connects, like now, we get this
Aug 26 22:10:12 openvpn 49182 USERNAME/PUBLICIP:51634 MULTI_sva: pool returned IPv4=172.20.100.2, IPv6=(Not enabled)
Aug 26 22:10:12 openvpn 71409 user 'USERNAME' authenticated
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 [USERNAME] Peer Connection Initiated with [AF_INET]PUBLICIP:51634
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_GUI_VER=OpenVPN_GUI_11
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_TCPNL=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_COMP_STUBv2=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_COMP_STUB=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_LZO=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_LZ4v2=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_LZ4=1
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_CIPHERS=AES-256-GCM
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_NCP=2
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_PROTO=2
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_PLAT=win
Aug 26 22:10:12 openvpn 49182 141.136.159.18:51634 peer info: IV_VER=2.4.11

no warnings or errors.
the pid 71409 is probably freeradius who authenticates the users on the new server instance.

and then a thousnd
Aug 26 22:15:30 openvpn 49182 IP packet with unknown IP version=15 seen

because this is not directly connected to a specific client and happens when no one is using the service, i think it it something on the server side, but i cant figure out what.
more so because i have the exact same config on other firewalls and everything works.

this is going for months now and i could not figure it out by myself.

seewolf · Mar 24, 2023, 8:02 PM

I fixed the problem.

The firewall died during the night from friday to saturday, so naturally I needed to build a new one on sunday.

After a clean reinstall, I again started having the same error
openvpn xxxxx IP packet with unknown IP version=15 seen
Endlessly filling the logs, and killing the SSD-s.

It seems the ntopng is the culprit.
After disabling ntopng, the errors stopped. And after enabling ntopng, the errors started again, even when there are no clients connected, and the errors start and stop at random intervals.

I am currently testing running ntopng but without OpenVPN interfaces selected.
For now it seems to be working as expected.

So it seems running ntopng with OpenVPN interfaces selected causes the OpenVPN server to have endless errors, even when everything else is working fine.

Now I am waiting for monday so we have some user traffic, but judging by the short test I am currently conducting, it should work.
Hope this helps someone with a similar problem.