(possibly) broken drivers (ral and rum)

[wallabybob]

I've just installed pfSense 2.0-BETA1 built on Sun Feb 21 05:56:33 EST 2010 FreeBSD 8.0-STABLE.

The system crashed on configuring and "Apply changes" to ral0 from the WEB GUI. Same outcome with rum0  but panic in a different place and with a different backtrace.

I'll try to reproduce with FreeBSD 8.0 RELEASE.

[jimp]

Did it work on previous snapshots?

I have a test box with a ral card in it, and it seems to be OK with today's snap. I tried changing a couple settings and applying and it didn't panic on me.

[cmb]

@wallabybob:

I'll try to reproduce with FreeBSD 8.0 RELEASE.

We're not on 8.0 release anymore, a proper test will need to be using RELENG_8.

[wallabybob]

@jimp:

Did it work on previous snapshots?

Don't know. I haven't looked at the pfSense repository but its been quite a while since there has been any significant changes to ral in the FreeBSD CVS repository so I'm thinking the problem could be in the interface between device driver and infrastructure code.

I have a test box with a ral card in it, and it seems to be OK with today's snap. I tried changing a couple settings and applying and it didn't panic on me.

I can reproduce the crashes reliably on a 800MHz VIA C3 system.  The backtrace in the most recent "ral" crash shows a kernel page fault (accessing 0xc3bd3008) early in ieee80211_free_node() called through 4 intermediate functions from softclock(). The backtrace in the "rum" crash shows a kernel page fault (accessing 0xffff) in ieee80211_getcapinfo() called through 4 intermediate functions from taskqueue_run(). In both cases the crash happens in an "asynchronous" operation - maybe running on a multi-CPU system masks the problem. Whats your system?

[jimp]

My ral card is a PCI card (Some variety of Linksys, I forget the model number) and it's in a P4, 2.6GHz box, with hyperthreading disabled.

[wallabybob]

My ral card is a PCI card: Gigabyte GN-WPKG

It doesn't seem to be readily available anymore.

I'll log these as issues on redmine.

[wallabybob]

@cmb:

We're not on 8.0 release anymore, a proper test will need to be using RELENG_8.

If I recall correctly RELENG_8 is a branch in the repository. How often does pfSense sync with the FreeBSD repository?

[jimp]

Perhaps it's a slightly different chip?

ral0: <ralink technology="" rt2560=""> mem 0xdffcc000-0xdffcdfff irq 19 at device 11.0 on pci0
ral0: MAC/BBP RT2560 (rev 0x04), RF RT2525
ral0: [ITHREAD]

ral0@pci0:0:11:0:       class=0x028000 card=0x00321737 chip=0x02011814 rev=0x01 hdr=0x00
    class      = network</ralink>

Mine still hasn't panic'd, and I've rebooted it a few times and poked around in the WebGUI, saving settings.

[wallabybob]

Here's mine:
ral0: <ralink technology="" rt2560="">mem 0xef000000-0xef001fff irq 7 at device 8.0 on pci0
ral0: MAC/BBP RT2560 (rev 0x04), RF RT2525
ral0: [ITHREAD]

ral0@pci0:0:8:0: class=0x028000 card=0xe9321458 chip=0x02011814 rev=0x01 hdr=0x00

Looks similar.

@jimp:

Mine still hasn't panic'd, and I've rebooted it a few times and poked around in the WebGUI, saving settings.

Mine has panic'd every time I've clicked on "Apply Changes" for the ral0 interface. IF the changes get written to the config file then the system panics at the same location in ieee80211_free_node() on startup. I've been able to work around this by booting into single user mode, editing the config file to remove opt1 details then rebooting.</ralink>

[jimp]

Do you have another system to try those cards in? Or perhaps reorder the cards in the PCI slots?

Since ours appear to use the same chip, that leads me to suspect some other hardware interaction is involved.

[wallabybob]

@jimp:

Do you have another system to try those cards in? Or perhaps reorder the cards in the PCI slots?

My system has only a single PCI slot (mini-ITX motherboard). To avoid the hassle of removing and replacing the card to test things I've been concentrating on the problem with the rum driver since the USB NIC is far easier to remove and replace than the PCI card. If I'm lucky it will turn out to be the same or very simiar problem with both drivers.

Using the USB NIC the system reliably panics on clicking on Apply all changes IF the interface mode is set to Access Point but doesn't panic if the interface mode is set to Infrastructure.

Here's the latest panic on applying all changes to the rum interface:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address  = 0xffff
fault code            = supervisor read, page not present
instruction pointer    = 0x20:0xc0a777ab
stack pointer          = 0x28:0xd50ddba4
frame pointer          = 0x28:0xd50ddbb0
code segment          = base 0x0, limit 0xfffff type 16
                        DPL 0, pres 1, def32 1, gram 1
processor eflags      = interrupt enabled, resume, IOPL=0
current process        = 0 (rum0 taskq)

[jimp]

It may help to get some ifconfig output from that card when plugged in, before you configure it all the way.

Something like:

ifconfig -m rum0_wlan0

Or just

ifconfig -m rum0

And perhaps:

ifconfig rum0_wlan0 list caps

[wallabybob]

I currently have rum0 configured in Infrastructure mode rather than Access Point. Here's the output you requested:

ifconfig rum0_wlan1 list caps

drivercaps=2181e501 <sta,ibss,hostap,txpmgt,shslot,shpreamble,monitor,wpa1,wpa2,bgscan># ifconfig -m rum0_wlan1
rum0_wlan1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:1b:11:b5:c1:c8
inet6 fe80::21b:11ff:feb5:c1c8%rum0_wlan1 prefixlen 64 scopeid 0xa
nd6 options=3 <performnud,accept_rtadv>media: IEEE 802.11 Wireless Ethernet autoselect mode 11b
status: no carrier
supported media:
media OFDM/54Mbps mode autoselect
media OFDM/48Mbps mode autoselect
media OFDM/36Mbps mode autoselect
media OFDM/24Mbps mode autoselect
media OFDM/18Mbps mode autoselect
media OFDM/12Mbps mode autoselect
media OFDM/9Mbps mode autoselect
media OFDM/6Mbps mode autoselect
media DS/11Mbps mode autoselect
media DS/5.5Mbps mode autoselect
media DS/2Mbps mode autoselect
media DS/1Mbps mode autoselect
media OFDM/54Mbps mode 11g
media OFDM/48Mbps mode 11g
media OFDM/36Mbps mode 11g
media OFDM/24Mbps mode 11g
media OFDM/18Mbps mode 11g
media OFDM/12Mbps mode 11g
media OFDM/9Mbps mode 11g
media OFDM/6Mbps mode 11g
media DS/11Mbps mode 11g
media DS/5.5Mbps mode 11g
media DS/2Mbps mode 11g
media DS/1Mbps mode 11g
media autoselect mode 11g
media DS/11Mbps mode 11b
media DS/5.5Mbps mode 11b
media DS/2Mbps mode 11b
media DS/1Mbps mode 11b
media autoselect mode 11b
media autoselect mode autoselect
ssid zzzz channel 1 (2412 Mhz 11b)
country US authmode WPA2/802.11i privacy ON deftxkey UNDEF txpower 0
bmiss 7 scanvalid 450 bgscan bgscanintvl 300 bgscanidle 250
roam:rssi 7 roam:rate 1 roaming MANUAL

ifconfig -m rum0

rum0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 2290
ether 00:1b:11:b5:c1:c8
media: IEEE 802.11 Wireless Ethernet autoselect mode 11b
status: associated
supported media:
media OFDM/54Mbps mode autoselect mediaopt monitor
media OFDM/54Mbps mode autoselect mediaopt hostap
media OFDM/54Mbps mode autoselect mediaopt adhoc
media OFDM/54Mbps mode autoselect
media OFDM/48Mbps mode autoselect mediaopt monitor
media OFDM/48Mbps mode autoselect mediaopt hostap
media OFDM/48Mbps mode autoselect mediaopt adhoc
media OFDM/48Mbps mode autoselect
media OFDM/36Mbps mode autoselect mediaopt monitor
media OFDM/36Mbps mode autoselect mediaopt hostap
media OFDM/36Mbps mode autoselect mediaopt adhoc
media OFDM/36Mbps mode autoselect
media OFDM/24Mbps mode autoselect mediaopt monitor
media OFDM/24Mbps mode autoselect mediaopt hostap
media OFDM/24Mbps mode autoselect mediaopt adhoc
media OFDM/24Mbps mode autoselect
media OFDM/18Mbps mode autoselect mediaopt monitor
media OFDM/18Mbps mode autoselect mediaopt hostap
media OFDM/18Mbps mode autoselect mediaopt adhoc
media OFDM/18Mbps mode autoselect
media OFDM/12Mbps mode autoselect mediaopt monitor
media OFDM/12Mbps mode autoselect mediaopt hostap
media OFDM/12Mbps mode autoselect mediaopt adhoc
media OFDM/12Mbps mode autoselect
media OFDM/9Mbps mode autoselect mediaopt monitor
media OFDM/9Mbps mode autoselect mediaopt hostap
media OFDM/9Mbps mode autoselect mediaopt adhoc
media OFDM/9Mbps mode autoselect
media OFDM/6Mbps mode autoselect mediaopt monitor
media OFDM/6Mbps mode autoselect mediaopt hostap
media OFDM/6Mbps mode autoselect mediaopt adhoc
media OFDM/6Mbps mode autoselect
media DS/11Mbps mode autoselect mediaopt monitor
media DS/11Mbps mode autoselect mediaopt hostap
media DS/11Mbps mode autoselect mediaopt adhoc
media DS/11Mbps mode autoselect
media DS/5.5Mbps mode autoselect mediaopt monitor
media DS/5.5Mbps mode autoselect mediaopt hostap
media DS/5.5Mbps mode autoselect mediaopt adhoc
media DS/5.5Mbps mode autoselect
media DS/2Mbps mode autoselect mediaopt monitor
media DS/2Mbps mode autoselect mediaopt hostap
media DS/2Mbps mode autoselect mediaopt adhoc
media DS/2Mbps mode autoselect
media DS/1Mbps mode autoselect mediaopt monitor
media DS/1Mbps mode autoselect mediaopt hostap
media DS/1Mbps mode autoselect mediaopt adhoc
media DS/1Mbps mode autoselect
media OFDM/54Mbps mode 11g mediaopt monitor
media OFDM/54Mbps mode 11g mediaopt hostap
media OFDM/54Mbps mode 11g mediaopt adhoc
media OFDM/54Mbps mode 11g
media OFDM/48Mbps mode 11g mediaopt monitor
media OFDM/48Mbps mode 11g mediaopt hostap
media OFDM/48Mbps mode 11g mediaopt adhoc
media OFDM/48Mbps mode 11g
media OFDM/36Mbps mode 11g mediaopt monitor
media OFDM/36Mbps mode 11g mediaopt hostap
media OFDM/36Mbps mode 11g mediaopt adhoc
media OFDM/36Mbps mode 11g
media OFDM/24Mbps mode 11g mediaopt monitor
media OFDM/24Mbps mode 11g mediaopt hostap
media OFDM/24Mbps mode 11g mediaopt adhoc
media OFDM/24Mbps mode 11g
media OFDM/18Mbps mode 11g mediaopt monitor
media OFDM/18Mbps mode 11g mediaopt hostap
media OFDM/18Mbps mode 11g mediaopt adhoc
media OFDM/18Mbps mode 11g
media OFDM/12Mbps mode 11g mediaopt monitor
media OFDM/12Mbps mode 11g mediaopt hostap
media OFDM/12Mbps mode 11g mediaopt adhoc
media OFDM/12Mbps mode 11g
media OFDM/9Mbps mode 11g mediaopt monitor
media OFDM/9Mbps mode 11g mediaopt hostap
media OFDM/9Mbps mode 11g mediaopt adhoc
media OFDM/9Mbps mode 11g
media OFDM/6Mbps mode 11g mediaopt monitor
media OFDM/6Mbps mode 11g mediaopt hostap
media OFDM/6Mbps mode 11g mediaopt adhoc
media OFDM/6Mbps mode 11g
media DS/11Mbps mode 11g mediaopt monitor
media DS/11Mbps mode 11g mediaopt hostap
media DS/11Mbps mode 11g mediaopt adhoc
media DS/11Mbps mode 11g
media DS/5.5Mbps mode 11g mediaopt monitor
media DS/5.5Mbps mode 11g mediaopt hostap
media DS/5.5Mbps mode 11g mediaopt adhoc
media DS/5.5Mbps mode 11g
media DS/2Mbps mode 11g mediaopt monitor
media DS/2Mbps mode 11g mediaopt hostap
media DS/2Mbps mode 11g mediaopt adhoc
media DS/2Mbps mode 11g
media DS/1Mbps mode 11g mediaopt monitor
media DS/1Mbps mode 11g mediaopt hostap
media DS/1Mbps mode 11g mediaopt adhoc
media DS/1Mbps mode 11g
media autoselect mode 11g mediaopt monitor
media autoselect mode 11g mediaopt hostap
media autoselect mode 11g mediaopt adhoc
media autoselect mode 11g
media DS/11Mbps mode 11b mediaopt monitor
media DS/11Mbps mode 11b mediaopt hostap
media DS/11Mbps mode 11b mediaopt adhoc
media DS/11Mbps mode 11b
media DS/5.5Mbps mode 11b mediaopt monitor
media DS/5.5Mbps mode 11b mediaopt hostap
media DS/5.5Mbps mode 11b mediaopt adhoc
media DS/5.5Mbps mode 11b
media DS/2Mbps mode 11b mediaopt monitor
media DS/2Mbps mode 11b mediaopt hostap
media DS/2Mbps mode 11b mediaopt adhoc
media DS/2Mbps mode 11b
media DS/1Mbps mode 11b mediaopt monitor
media DS/1Mbps mode 11b mediaopt hostap
media DS/1Mbps mode 11b mediaopt adhoc
media DS/1Mbps mode 11b
media autoselect mode 11b mediaopt monitor
media autoselect mode 11b mediaopt hostap
media autoselect mode 11b mediaopt adhoc
media autoselect mode 11b
media autoselect mode autoselect mediaopt monitor
media autoselect mode autoselect mediaopt hostap
media autoselect mode autoselect mediaopt adhoc
media autoselect mode autoselect

I have had this same card successfully operating in Access Point mode under pfSense 1.2.3 Release.</up,broadcast,running,simplex,multicast></performnud,accept_rtadv></up,broadcast,running,simplex,multicast></sta,ibss,hostap,txpmgt,shslot,shpreamble,monitor,wpa1,wpa2,bgscan>

[wallabybob]

If I apply all changes after setting the rum interface into adhoc mode I see a similar crash (at ieee80211_getcapinfo+0x56) but with slightly different stacktrace.

Summary: Infrastructure mode is ok, Access Point and adhoc cause crashes.

[jimp]

If that's the case, then it's either going to end up being a driver bug, or a bug with your particular chip.

Not sure you'd get any real response on a driver bug here, but if you can reproduce that with a stock FreeBSD 8 installation, you could report it directly to FreeBSD.

[wallabybob]

I'm happy to report it as FreeBSD bug. What does a click on Apply all changes map into? At most a couple of ifconfig shell commands? If so, what?

[jimp]

Some ifconfig commands and it writes out a config for hostapd.

Check the contents of /tmp and /var/etc - I'm not sure if it still does, but it used to write out a script with the ifconfig commands in it. I know the hostapd config should be in /var/etc

[wallabybob]

Thanks for the hints jimp. /etc/inc/interfaces.php writes the hostapd and wpa_supplicant configuration and associated shell commands into /tmp/rum0_wlan_setup.sh but it doesn't save the other ifconfig commands anywhere. Nonetheless, reading through the code I worked out a fairly simple set to demonstrate the problem.

The following shell commands are sufficient to crash my FreeBSD 8.0 Release system and my pfSense 2.0 BETA box.

ifconfig wlan create wlandev rum0 wlanmode adhoc bssid

ifconfig wlan0 mode 11b

ifconfig wlan0 up mode 11b ssid Bree protmode off -mediaopt hostap mediaopt adhoc -hidessid -pureg -puren -apbridge -mediaopt turbo -wme authmode open wepmode off

There is a noticeable pause between the shell prompt after the last command and the system crash.

I'll arrange for a FreeBSD problem report to be submitted.

[ShadowFlare]

Try changing that last command to:

ifconfig wlan0 up mode 11b ssid Bree -apbridge -wme authmode open wepmode off

See if it still does the same with that.  You could also try removing bssid, but that was added after you already saw the issue on your system and is likely unrelated.  You could even try just a very basic command:

ifconfig wlan0 up mode 11b ssid Bree

If that still doesn't work, also try leaving off the mode 11b.

[wallabybob]

Thanks for the suggestions. The two commands:

ifconfig wlan create wlandev rum0 wlanmode adhoc bssid

ifconfig wlan0 up ssid Bree

are sufficient to crash the pfSense system.

[ShadowFlare]

I'm guessing it probably is the same without bssid, since you reported this before it was added to the commands used (and probably only applies for hostap mode anyway).  Removing that, you would pretty much have it reduced to the bare minimum commands to get it operational.

[wallabybob]

The ral problem I reported earlier in this thread is also easily reproduced. The two commands:

ifconfig wlan create wlandev ral0 wlanmode hostap bssid

ifconfig wlan0 up ssid Bree

are sufficient to crash the pfSense system, again after a pause of no more than a few seconds after the second command completes.

The same two commands with hostap replaced by adhoc have a similar result.

When I issued the two commands

ifconfig wlan create wlandev ral0 bssid

ifconfig wlan0 up ssid Bree

the system successfully continued for some minutes until I forced a reboot.

[wallabybob]

ral update:

I have further investigated the ral problem and found that ral wasn't interrupting, even when in infrastructure mode. Close inspection showed the PCI card wasn't seated evenly in the PCI socket; the edge closer to the back panel protruded a little more than the edge further from the back panel. So I evened out it and rebooted and the system did not crash on clicking Apply all changes in any of the three modes: ad-hoc, access point or Infrastructure.

However when the NIC was configured in Infrastructure mode it repeatedly reported "device timout" and my netbook saw no evidence of a signal corresponding to the ssid I had configured.  I saw a problem report about this ( http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/117655 ) but I thought the explanation was unconvincing. I'm using a 90W PicoPS and a 48W switch mode PS (12V @ 4A) so I suppose the surge current capability of this is a fair bit lower than that of a more standard computer power supply. I can get a pretty cheap card with a more recent ral chip (rt2561) so I might get that to try out.