Yet another sk0 (Dlink DGE-530T) driver issue on pfSense 2.0

[Ventolin]

Hi,

I think if the interface was cycling between DOWN and UP the interrupt moderation tuning parameter might be helpful, but in my case the interface goes down and stays down, so even a long time might not work.

It's worth a try though, where / how can I configure this?

Secondly, I have been looking at /etc/rc.linkup and it looks to have the exact event I'm seeing in my logs:

function handle_argument_group($iface, $argument2) {
	global $config;	

	$ipaddr = $config['interfaces'][$iface]['ipaddr'];
	if (is_ipaddr($ipaddr) || empty($ipaddr)) {
		log_error("Hotplug event detected for {$iface} but ignoring since interface is configured with static IP ({$ipaddr})");
		interfaces_staticarp_configure($iface);
		$iface = get_real_interface($iface);
		interfaces_bring_up($iface);
		if ($argument2 == "start" || $argument2 == "up")
			send_event("interface newip {$iface}");

There are other interesting functions in there to bring the interface down or up depending on the second variable.

I might not be reading this correctly, but it looks like it tries to bring the interface up after logging the error.  This is what I'm looking for if it does do that, but how do I get this to run?  It's almost as if the script is doing the opposite of ignoring the event as it says in the log line and goes ahead and tries to bring the interface up anyway.

I am confused!

This is the first time I'm looking at pfSense code and I'm way to new to this to try and play with the code, so can you or anyone offer any advice or suggestions?

Thanks in advance,

Vent

[stephenw10]

@Ventolin:

It's worth a try though, where / how can I configure this?

You configure it using the sysctl command:

sysctl dev.skc.0.int_mod

Should give the current value.

sysctl dev.skc.0.int_mod 500

Sets it to 500.

To set it permanently you can add this as a new value in System: Advanced: System Tunables:

Re-writing parts of pfSense is beyond my usual level of tinkering ;) It shouldn't have to do this. If the interface were actually going down and then coming back up it would reload no problem. Adding code to reset the interface when it sees a hotplug event would cause flapping if you actually unplugged the cable. Perhaps if you posted some of your logs it would be helpful. Also the relevant sections from the boot log:

 cat /var/log/dmesg.boot | grep sk0

And maybe:

pciconf -lv | grep sk0

Steve

[Ventolin]

Hi,

Thanks Steve for your help again.

sysctl dev.skc.0.int_mod

returns 100

sysctl -w dev.skc.0.int_mod=500

sets it to 500
(quick man sysctl was needed for correct syntax :P)

From the logs, this is what I get…

Dec  1 18:51:17 myfirewall check_reload_status: Linkup starting sk0
Dec  1 18:51:17 myfirewall kernel: sk0: link state changed to DOWN
Dec  1 18:51:24 myfirewall php: : Hotplug event detected for lan but ignoring since interface is configured with static IP (192.168.0.3)

After that, nothing until I got hope and typed:

ifconfig sk0 UP

and got this in the logs:

Dec  1 22:25:30 myfirewall check_reload_status: Reloading filter
Dec  1 22:26:31 myfirewall check_reload_status: Reloading filter
Dec  1 22:32:06 myfirewall check_reload_status: Linkup starting sk0
Dec  1 22:32:06 myfirewall kernel: sk0: link state changed to UP
Dec  1 22:32:17 myfirewall php: : Hotplug event detected for lan but ignoring since interface is configured with static IP (192.168.0.3)
Dec  1 22:32:17 myfirewall check_reload_status: rc.newwanip starting sk0
Dec  1 22:32:29 myfirewall php: : rc.newwanip: Informational is starting sk0.
Dec  1 22:32:29 myfirewall php: : rc.newwanip: on (IP address: 192.168.0.3) (interface: lan) (real interface: sk0).

and then some stuff about routing, apinger restarting, dnsmasq starting etc.

From dmseg.boot:

skc0: <marvell gigabit="" ethernet=""> port 0xd400-0xd4ff mem 0xd8000000-0xd8003fff irq 10 at device 9.0 on pci0
skc0: Marvell Yukon Gigabit Ethernet rev. (0x1)
sk0: <marvell semiconductor,="" inc.="" yukon=""> on skc0
miibus0: <mii bus=""> on sk0
e1000phy0: <marvell 88e1011="" gigabit="" phy=""> PHY 0 on miibus0
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
skc0: [ITHREAD]
skc1: <d-link dge-530t="" gigabit="" ethernet=""> port 0xe000-0xe0ff mem 0xd8004000-0xd8007fff irq 11 at device 15.0 on pci0
skc1: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
sk1: <marvell semiconductor,="" inc.="" yukon=""> on skc1
miibus3: <mii bus=""> on sk1
e1000phy1: <marvell 88e1011="" gigabit="" phy=""> PHY 0 on miibus3
e1000phy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
skc1: [ITHREAD]
skc2: <d-link dge-530t="" gigabit="" ethernet=""> port 0xe400-0xe4ff mem 0xd8008000-0xd800bfff irq 10 at device 17.0 on pci0
skc2: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
sk2: <marvell semiconductor,="" inc.="" yukon=""> on skc2
miibus4: <mii bus=""> on sk2
e1000phy2: <marvell 88e1011="" gigabit="" phy=""> PHY 0 on miibus4
e1000phy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
skc2: [ITHREAD]</marvell></mii></marvell></d-link></marvell></mii></marvell></d-link></marvell></mii></marvell></marvell>

I have 3 gigabit nics in there and 2 10/100 nics (not Marvell/Yukon), a total of 5 nics.
sk0 is currently a brand new SMC 9452TX gigabit card, as mentioned before.  I put the extra cards in there for quick testing of other nics to see if I could isolate the problem but all 3 cards do the same thing (even when installed 1 at a time) which led me to suspect the sk driver.

From pciconf:

skc0@pci0:0:9:0:        class=0x020000 card=0xb45210b8 chip=0x432011ab rev=0x12 hdr=0x00
    class      = network
    subclass   = ethernet
skc1@pci0:0:15:0:       class=0x020000 card=0x4c001186 chip=0x4c001186 rev=0x11 hdr=0x00
    class      = network
    subclass   = ethernet
skc2@pci0:0:17:0:       class=0x020000 card=0x4c001186 chip=0x4c001186 rev=0x11 hdr=0x00
    class      = network
    subclass   = ethernet

If a script ran triggered by the hotplug event I don't mind the consequence of the interface 'flapping' if I unplug the network cable; I'm only ever doing that when the box is off and I'm chaning network cards.  Very much the lesser of 2 evils.

Thank you very much for your help,

Regards,

Vent

[stephenw10]

Hmm, well I'm at the end of diagnostic skills I'm afraid.
My own interfaces:

skc0: <marvell gigabit="" ethernet="">port 0xc000-0xc0ff mem 0xd042c000-0xd042ffff irq 16 at device 0.0 on pci5
skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9)
sk0: <marvell semiconductor,="" inc.="" yukon="">on skc0
e1000phy4: <marvell 88e1011="" gigabit="" phy="">PHY 0 on miibus4

skc0@pci0:5:0:0:	class=0x020000 card=0x43201148 chip=0x432011ab rev=0x13 hdr=0x00</marvell></marvell></marvell> 

Slightly different from yours and working perfectly. Not much help to you though!

Steve

[Ventolin]

Hi,

Just to report sysctl -w dev.skc.0.int_mod=500 didn't work, though I got 20 days uptime this time.

Have updated to pfsense 2.0.1 now, maybe that will be better.

Would really like to know how to write that script.

Would be even better if the sk driver was updated or fixed.  How do I make a bug report?

To be honest, I'm either going to put a cheap realtek nic in or build a second pfsense box and use CARP failover between them.

Will report back if any further issues.

[stephenw10]

It's unlikely a bug report on 2.0.X would be acted on at this point since builds of 2.1 on FreeBSD 9 are now getting close. CMB recently posted 'by the end of the year'.
These will be the first builds for testing only and will probably have many bugs but will have much newer drivers. And by running these and reporting the bugs you will be helping out everyone.  :)

Steve

Edit: However now I can't find where I read that! A tweet maybe? Maybe it wasn't CMB.  ::) Fairly sure I did read it though!

[Ventolin]

Hi,

Well 2.0.1 lasted 2 days of uptime, so the sk bug is still there, perhaps unsurprisingly.

So, I did some digging and learnt how to write a shell script which simply pings an ip on the LAN 3 times (with a delay of 1 second) and then does an ifconfig sk0 DOWN then ifconfig sk0 UP if the ping fails.  There's a 2 second wait between the DOWN and UP commands in case the driver needs a bit of time to work.

I then hooked the script up to cron so it could run every minute.

So, here's a quick guide on how to keep your breaking sk driver working.

1. Make a new file called ifcheck.sh in /usr/local/bin

2. copy and paste the following code in:

#!/bin/sh
#set logfile here or uncomment second line for no logging
LOGFILE=/tmp/ifcheck.log
#LOGFILE=/dev/null

#Set primary interface/ip to check here
IF1=sk0
IP1=192.168.0.10

#add more interfaces/ips here
#IF2=rl0
#IP2=192.168.1.254

#uncomment next line for debugging
#echo $(date) "pinging interfaces..." >> $LOGFILE

ping -c 3 -t 1 $IP1 > /dev/null 2>&1 || (echo $(date) "$IF1 DOWN, bouncing..." >> $LOGFILE && /sbin/ifconfig $IF1 down && sleep 2 && /sbin/ifconfig $IF1 up && echo $(date) "$IF1 set to UP" >> $LOGFILE)
#ping -c 3 -t 1 $IP2 > /dev/null 2>&1 || (echo $(date) "$IF2 DOWN, bouncing..." >> $LOGFILE && /sbin/ifconfig $IF2 down && sleep 2 && /sbin/ifconfig $IF2 up && echo $(date) "$IF2 set to UP" >> $LOGFILE)

3. reconfigure the script to your chosen interface name and ip address - you can uncomment the second ping and variables to test another interface and extend to test more.

I used the Easy Editor ee from the console shell prompt to write the script instead of the Virtually Impossible vi editor because I didn't want to take 10 years to master vi to do a simple edit when I could spend 5 seconds doing the same thing in ee :P

4. from a shell prompt type the following to make the script executable:

chmod +755 /usr/local/bin/ifconfig.sh

5. from the web interface add the cron package

6. Add a new entry to the cron table:
minute */1
hour *
mday *
month *
wday *
who root
command /usr/local/bin/ifcheck.sh

7. Save

8. To test, uncomment the debugging line like this:

#uncomment next line for debugging
echo $(date) "pinging interfaces..." >> $LOGFILE

This will write to a log file: /tmp/ifcheck.log

The cron settings will mean this will fire every minute.  You'll want to comment back the debug line once you're satisfied the script is working to only have errors in the log file and to save space.

You could probably use:

/usr/bin/nice -n20 /usr/local/bin/ifcheck.sh
```in the cron entry which I think will make the process run with a lower priority, but I haven't tested that yet.

The script should be self explanatory.  If the ping fails, ping will return an output that will trigger everything after the || to run.

OK, standard disclaimer, if it breaks your system sorry about that, use at your own risk.

To write the script I did quite a bit of googling and took bits of code from other slightly more complicated scripts and kept things simple.

If anyone has any suggestions to make it better, please just add your thoughts and/or improvements, it will be very welcome.  Feel free to use / modify as you wish.

Thanks everyone, esp stephenw10.

Vent

[Ventolin]

OK, this version logs to the syslog as well as the logfile:

#!/bin/sh
#set logfile here or uncomment second line for no logging
LOGFILE=/tmp/ifcheck.log
#LOGFILE=/dev/null

#Set primary interface/ip to check here
IF1=sk0
IP1=192.168.0.10

#add more interfaces/ips here
#IF2=rl0
#IP2=192.168.1.254

#uncomment next line for debugging
echo $(date) "Pinging interfaces..." >> $LOGFILE
#logger -t ifcheck Pinging interfaces...

ping -c 3 -t 1 $IP1 > /dev/null 2>&1 || (logger -t ifcheck $IF1 DOWN, bouncing... && echo $(date) "$IF1 DOWN, bouncing..." >> $LOGFILE && /sbin/ifconfig $IF1 down && sleep 2 && /sbin/ifconfig $IF1 up && logger -t ifcheck $IF1 set to UP && echo $(date) "$IF1 set to UP" >> $LOGFILE)
#ping -c 3 -t 1 $IP2 > /dev/null 2>&1 || (logger -t ifcheck $IF2 DOWN, bouncing... && echo $(date) "$IF2 DOWN, bouncing..." >> $LOGFILE && /sbin/ifconfig $IF2 down && sleep 2 && /sbin/ifconfig $IF2 up && logger -t ifcheck $IF2 set to UP && echo $(date) "$IF2 set to UP" >> $LOGFILE)

[Ventolin]

This version only logs to the Syslog and no log file:

#!/bin/sh

#Set primary interface/ip to check here
IF1=sk0
IP1=192.168.0.10

#add more interfaces/ips here
#IF2=rl0
#IP2=192.168.1.254

#uncomment next line for debugging
#logger -t ifcheck Pinging interfaces...

ping -c 3 -t 1 $IP1 > /dev/null 2>&1 || (logger -t ifcheck $IF1 DOWN, bouncing... && /sbin/ifconfig $IF1 down && sleep 2 && /sbin/ifconfig $IF1 up && logger -t ifcheck $IF1 set to UP)
#ping -c 3 -t 1 $IP2 > /dev/null 2>&1 || (logger -t ifcheck $IF2 DOWN, bouncing... && /sbin/ifconfig $IF2 down && sleep 2 && /sbin/ifconfig $IF2 up && logger -t ifcheck $IF2 set to UP)

Hope that helps someone,

Regards,

Vent

[thestealth]

I found this post as I have the exact same problem. The posted solution works for my needs.

Thanks Ventolin!!! ;D ;D