Pfsense states and unwanted IPv6 addresses

[bateau009]

Hello all,

As you can see in the attachment snapshot, I am experiencing weird (at least to some extent) States output referencing IPv6 addresses.  On the right of the snapshot you'll se the "States New" output referencing the same protocol connections of the "States" output on the left but with a "(null)" definition in place of the IP addresses. Only a few states are listed with correct IPv4 addresses.

I am wondering what all that means and how to go back to plain IPv4 states, partly because I suspect that this could have been triggered by some sort of hidden issues. We have been using pfsense since 1.2.3 days and only recently it came out with that unwanted behavior.

I am using pfsense 2.0.1 here with no IPv6 addresses enabled on the box. Also "Allow IPv6" in the Networking Tab of the Advanced dropdown menu is unchecked.

thanks!
bateau

[cmb]

It's not parsing the states correctly since 2.0.x's code there doesn't support IPv6. What do those states look like in the output of "pfctl -ss"? That will help track down why you're seeing that.

[bateau009]

Its output seems to be all mixed with non-ascii data and it screws up the terminal:

# pfctl -ss
all carp 192.168.2.253 -> 224.0.0.18       SINGLE:NO_TRAFFIC
Pr| <c8>ipcomp ::192.168.0.226 <- ::224.0.0.18 <- ::2.112.2.0       NO_TRAFFIC:NO_TRAFFIC
Pr|ȸ̱Ɛ<8B>{ <c8>152 0:0:ac1f:1fd:: <- 0:0:e000:12:: <- ::270:200:0:c00:616c:6c00[47308]       NO_TRAFFIC:NO_TRAFFIC
<b8>̱ <c6>ip ::[57344] <- ::[624] <- 0:c00:616c:6c00::b8cc:b1c6       255:0
<c8><f5><96>Ȑ<bb><8A>Ƹ܈<c8>^A 204 ::e000:12:0:0 <- ::270:200:0:c00[24940] <- 0:0:b8cc:b1c6::       NO_TRAFFIC:NO_TRAFFIC
<90><9B><93> <c6>176 570c:3fb4::[500] <- 211:200:0:800:616c:6c00:682a:8ac8[16617] <- ::c8f5:aac8:68ca:94c8:a064:a7c8       NO_TRAFFIC:NO_TRAFFIC
P2VȀЄ <c8>idpr ::7b:0:211:100 <- 616c:6c00:a094:3fc5:18d8:b7c6::[61510] <- c815:79c8:90cb:a2c6::       191:0
 114 7b:0:211:200:1:700:616c:6c00[57533] <- d0:84c8:100:0:78c3:7dc8:a0e4:8dc6[47164] <- ::24.168.95.200       0:255
 narp 0:c00:616c:6c00:f0e6:b9c6:50a2:94c6[256] <- 18c8:a4c8:f0e6:b9c6:681a:91c6::[36987] <- 7813:acc8:2861:78c8::c8a5:8ac6[53298]       NO_TRAFFIC:NO_TRAFFIC
 ip ::78b3:99c6:100:0:683a:54c8[6264] <- 686a:99c8::[51413] <- ::683a:54c8:d072:7cc8:36:bfc4       NO_TRAFFIC:NO_TRAFFIC
 ip 100:0:78b3:99c6:c805:53c8:c8a5:8ac6 <- 1888:99c8:78a3:a5c8:7883:9ec8:100:0[37003] <- 48a6:8ac6:36:bfc4::       0:191
 ip ::c8f5:aac8:0:0:b8ec:b1c6[37115] <- 90:61c8:100:0:4089:96c6:e83a:54c8[54] <- ::100       NO_TRAFFIC:NO_TRAFFIC

<fe><87> <a7>ip ::b83c:86c8:0:0:28d1:7ec8 <- 7813:61c8:108c:7dc8:36:bfc4:10:566c[47] <- 0:0:1020:484::       NO_TRAFFIC:NO_TRAFFIC
> 254 68ea:55c8:40a9:9cc8::1838:a0c6[49289] <- 36:bfc4:10:566c:2f:566c::[4128] <- ::[58370]       NO_TRAFFIC:NO_TRAFFIC
 xns-idp ::78c3:56c8:f813:61c8:36:bfc4[49597] <- c2d4:94a::2940:482:0:0 <- ::5a6c:d41c:fe6c:d51c       167:0
 ip 9838:a0c6:36:bfc4:c1bd:94a:c2d4:94a <- 2940:482:: <- 5a6c:d41c:fe6c:d51c::d016:487       56:0
<ff><ff><ff><ff><ff><ff><ff><ff>¸G <fe>ip 61f4:8488:620b:8588::2940:482 <- ::7.220.6.209[43996] <- 0:0:d016:487::       NO_TRAFFIC:NO_TRAFFIC
 255 0:0:2940:482:: <- ::7dc:6d1:abdc:7d1:0:0[53270] <- ::[65535]       NO_TRAFFIC:NO_TRAFFIC
 ip ::[59678] <- 69c4:6473::802:987:0:0 <- ::ffff:ffff:ffff:ffff[65535]       71:167
 ip ::7c80:118d:8cc4:118d <- d016:487:: <- ffff:ffff:ffff:ffff:ffff:ffff:c2b8:47fe       0:128
 ip 7c80:118d:8cc4:118d::d016:487 <- ::255.255.255.255[65535] <- ffff:ffff::       NO_TRAFFIC:NO_TRAFFIC
 ip 0:0:2e00:487:: <- ::ffff:ffff:ffff:ffff:ffff:ffff[49848] <- ::0.51.191.196       NO_TRAFFIC:NO_TRAFFIC
 ip ::[65535] <- ffff:ffff:ffff:ffff:: <- ::[5632]       0:71
 ip ::ffff:ffff:ffff:ffff[65535] <- c2b8:47fe::[51] <- ::700:0:0:0[1280]       MULTIPLE:NO_TRAFFIC
 ip ffff:ffff:ffff:ffff:ffff:ffff:: <- :: <- 700::500:0:0:0[24577]       108:0
ESC^D ip ffff:ffff:c2b8:47fe:: <- 33:bfc4::500:0 <- 400::f800:0:0:0[53248]       86:0
 ipencap :: <- 0:0:500::400:0 <- f800::d000:0:0:0[50253]       NO_TRAFFIC:NO_TRAFFIC
Q#<a7>X^B<ea>N<fd><ed><c5>T<f6>O7 <e4><d5>ip 0:0:33:bfc4::[512] <- 0:0:600::7000:0 <- 6801::7b4d:100:6304:0[34626]       122:2
%#<88>j <c0><a8>idpr ::200:0:0:0[1536] <- 0:0:7000::6801:0 <- 7b4d:100:6304:0:8742:1e00::       136:108
 idpr f00::e00:0:0:0[13388] <- 0:0:e002::e44b:100[39941] <- c043:1e00::5123:a758[746]       135:170
 ip 2800::f908:0:0:0[3161] <- ::1e4b:100:b50d:0:d94b:1e00 <- ::5123:a758:2ea:c1ee:2523:886a[55752]       127:0
¸G^G ip f908::c59:0:0:0[7755] <- b50d:0:d94b:1e00::[20771] <- 2eb:dcc4:c4fc:1a38:c0a8:7::       191:153
 184 7800::c45:100:8a0c:0[44618] <- ::5123:a758:2ec:f058[65407] <- c0a8:7::[80]       93:0
f <f2>ip 573e:100:5713:0:7b51:1e00:: <- 5123:a758:2ec:f059:ff7f:df36:2eb:ffc7 <- ::66f2:0:2eb:ffc7       4:168
 iso-ip 7b51:1e00::5123:a758[749] <- ff7f:df36:c0a8:7:: <- 50:0:c2b8:4707::       0:151
 ip ::5123:a758:2ed:11e9:ff7f:df36[54575] <- ::235.14.0.0[54575] <- ::235.14.0.0[49320]       0:191
^B^F^A ip 2ed:9e9d:a49:74d7:c0a8:4:: <- ::3e3:0:c2b8:4704:0:0 <- ::3e3:0:b0ce:bcee:0:0       35:71
all tcp 176.206.188.238:50281 -> 192.168.0.4:995       ESTABLISHED:ESTABLISHED
ȕ<89><c6>^A ipcomp ::50:0:c2b8:4715 <- ::50:0:7b7d:4731 <- ::41bd:0:206:100       NO_TRAFFIC:NO_TRAFFIC
 skip 41bd:0:7b7d:4731:: <- 41bd:0:c0a8:c:: <- 50:0:206:200:1:200:616c:6c00[10385]       NO_TRAFFIC:NO_TRAFFIC
<f0>^F^? <c8>larp ::0.80.0.0[20249] <- ::39.61.0.0[518] <- 0:200:616c:6c00:a024:9ec8:683a:62c8       255:162
<f0>ƭȠT<aa><c8>(<b1><99> <c6>visa ::273d:0:c0a8:7:0:0 <- ::50:0:206:200:1:200[24940] <- 685a:a7c8:f076:b5c7::f076:b5c7       0:4
(<a1>`<c8>^A 217 c0a8:4::[42275] <- 206:100:0:200:616c:6c00:18a8:9ac8[41028] <- ::a0c4:80c8:4089:91c6:4099:84c8       NO_TRAFFIC:NO_TRAFFIC
xõ<c7>p'V <c8>sctp ::19:0:206:200[1] <- 616c:6c00:30:a7c8:28b1:9ec8::[10401] <- 30:a7c8:4049:93c6::b83c:66c8       NO_TRAFFIC:NO_TRAFFIC
 ipv6-icmp f26e:0:206:100:0:200:</c8></c7></c8></a1></c6></b1></c8></aa></f0></c8></f0></c6></f2></a8></c0></d5></e4></f6></c5></ed></fd></ea></a7></fe></ff></ff></ff></ff></ff></ff></ff></ff></a7></fe></c8></c6></c8></bb></f5></c8></c6></b8></c8></c8>

[jimp]

Looks like a kernel/world mismatch.

Random guess: are you running a hacom ALIX unit with a hard drive?

They shipped a number of those installed with the kernel on a separate boot partition that doesn't get updated by a firmware update, so you end up running  a new base OS with an old kernel.

If that is not the case, then try manually switching to the SMP kernel as described here:
http://doc.pfsense.org/index.php/Switching_Kernels

[bateau009]

@jimp:

Looks like a kernel/world mismatch.

Random guess: are you running a hacom ALIX unit with a hard drive?

They shipped a number of those installed with the kernel on a separate boot partition that doesn't get updated by a firmware update, so you end up running  a new base OS with an old kernel.

If that is not the case, then try manually switching to the SMP kernel as described here:
http://doc.pfsense.org/index.php/Switching_Kernels

It is already running an SMP kernel on a plain old i386 machine (Pentium4 cpu). It has always been running SMP kernels.

On january 25 pfsense has been upgraded to the latest 2.0.2 release. "upgrade_log.txt" reveales some suspicious lines, though:

x ./tmp/pre_upgrade_command
Firmware upgrade in progress...
Installing /root/latest.tgz.
./var/empty/: Can't set user=0/group=0 for var/emptyCan't update time for var/empty
./boot/loader.rc: Could not unlink
tar: Error exit delayed from previous errors.
Image installed /root/latest.tgz.
. changed
        modification time expected Fri May 18 18:19:43 2012 found Fri Jan 25 17:27:06 2013 modified

Furthermore it seems that the running kernel belongs to an old build, confirming your guess about a kernel/world mismatch:

# uname -a
FreeBSD pfmaster.ouverture.lan 8.1-RELEASE-p6 FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 17:53:00 EST 2011     root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  i386

This seems odd to me because the contents of the /boot/kernel directory all resamble the contents of /kernels/kernel_SMP.gz from 2.0.2 release (dated December 7, 2012).
Could it be the loader actually loading an old kernel from the disk?

[jimp]

Try looking at these:

fdisk

ls -l /dev/ad*  (or  da* if you have SCSI drives)

boot0cfg -v /dev/ad0  (Or whatever your drive is)

mount

Odds are there is a second partition somewhere that is actually doing the boot. The full dmesg log would show it also, watch where it says it's mounting root from compared to what the boot device is.

[bateau009]

@jimp:

Try looking at these:

fdisk

ls -l /dev/ad*   (or  da* if you have SCSI drives)

boot0cfg -v /dev/ad0   (Or whatever your drive is)

mount

Odds are there is a second partition somewhere that is actually doing the boot. The full dmesg log would show it also, watch where it says it's mounting root from compared to what the boot device is.

Eventually, I found out that we have a degraded array (ad4+ad6) and the pfsense upgrade process likely didn't correctly update the boot loader of the running disk. Indeed, booting from the degraded component (ad4) it loads the new kernel, although with the old pfsense version(2.0.1). After a few reboots with both the array components online (really don't know exactly how, though) it has now loaded the newer kernel alongside the upgraded pfsense version and now pf states output went back to normal.

Name    Status  Components
mirror/pfSenseMirror  DEGRADED  ad6

Last question: How may I tell pfsense to update the bootloader on both disks?
Thanks for your help and patience.

[jimp]

fix the array (remove the bad drive from the array and readd) as described at http://doc.pfsense.org/index.php/Create_a_Software_RAID1_%28gmirror%29 and then you'll have no issues updating.