Pfsense Nokıa IP380 Install
-
In FreeBSD the calculated IRQ for a PCI device can be overridden by an entry in /boot/loader.conf.local (preferred since it won't be changed by a pfSense install) or /boot/loader.conf of the form:```
hw.pci1.2.3.INTA.irg=15If you have a suitable IPSO installation or the startup output from a previous IPSO boot you should be able to determine the IRQs of each of the NICs. The challenge is then to map from the IPSO device name (e.g. eth-1, eth-s1p2) to FreeBSD device name (e.g. fxp1, dc2). -
I give a try to your solution by modifying the /boot/loader.conf but it makes no difference.
It seems that the entry I write in the file are not affecting the kernel.
I use the following config:
hw.pci3.5.0.dc0.irq=12
hw.pci3.6.0.dc1.irq=13I checked the pci addresse swith dmesg.
ALex
-
It seems that the entry I write in the file are not affecting the kernel.
I use the following config:
hw.pci3.5.0.dc0.irq=12
hw.pci3.6.0.dc1.irq=13You should have used INTA rather than dc0 and dc1. See my post about this!
It might be helpful to have a bit of background on PCI devices. Each PCI device can use up to 4 interrupt request lines. They are called INTA, INTB, INTC, INTD. Most devices use INTA. The PCI interrupt request lines are connected to IRQs according to the whim of the motherboard designer. The /boot/loader.conf.local line```
hw.pci3.5.0.INTA.irq=12As far as I know, there is no code that would act on a line like:``` hw.pci3.5.0.dc0.irq=12The bus, device and function numbers you have used look plausible but I have no idea if they are correct.
Edit: I had the parameters in the loader variable name incorrect and consequently the suggested variables and values in this reply won't work.
-
Hi
First of all, my system is running even with interrupt storm on irq10.
The mapping of the NICs IPSO <-> pfSense can be deduced from the bootlog of IPSO, below:
eth-s1p1 <-> dc0
eth-s1p2 <-> dc1
eth-s2p1 <-> dc2
eth-s2p2 <-> dc3
eth1 <-> fxp0
eth2 <-> fxp1
eth3 <-> fxp2
eth4 <-> fxp3This is confirmed by the experience of connecting stuff to these ports. The four first NICs, the Digital ones, are the left ones: eth-s1p2 stands for slot 1 port 2, the other ones are the Intel NICs.
From the same bootlog, I can extract the INTA, INTB, etc. and the corresponding irq, but not the PCI bus number. I don't know what "rev" and "onboard" stand for.
pcidec0 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 on pci1:0:0 fxp0 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int d irq 6 onboard 1 netlog:eth1 .. Ethernet address 0:a0:8e:78:c1:b4 fxp1 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int a irq 10 onboard 2 netlog:eth2 .. Ethernet address 0:a0:8e:78:c1:b5 fxp2 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int b irq 11 onboard 3 netlog:eth3 .. Ethernet address 0:a0:8e:78:c1:b6 fxp3 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int c irq 12 onboard 4 netlog:eth4 .. Ethernet address 0:a0:8e:78:c1:b7 pcidec1 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 slot 1 tulip0 <digital dc21143="" fast="" ethernet=""> rev 65 int b irq 11 slot 1 port 1 netlog:eth-s1p1 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:ac netlog:eth-s1p1 .. enabling 10baseT/UTP port in half duplex mode tulip1 <digital dc21143="" fast="" ethernet=""> rev 65 int c irq 15 slot 1 port 2 netlog:eth-s1p2 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:ad netlog:eth-s1p2 .. enabling 10baseT/UTP port in half duplex mode pcidec2 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 slot 2 tulip2 <digital dc21143="" fast="" ethernet=""> rev 65 int b irq 15 slot 2 port 1 netlog:eth-s2p1 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:b0 netlog:eth-s2p1 .. enabling 10baseT/UTP port in half duplex mode tulip3 <digital dc21143="" fast="" ethernet=""> rev 65 int c irq 12 slot 2 port 2 netlog:eth-s2p2 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:b1 netlog:eth-s2p2 .. enabling 10baseT/UTP port in half duplex mode</digital></digital></intel></digital></digital></intel></intel></intel></intel></intel></intel>For giving values for the PCI bus, I suppose I have to look into the bootlog of pfSense this time:
pci0: <serial bus,="" usb="">at device 15.2 (no driver attached) pcib1: <serverworks nb6635="" 3.0le="" host="" to="" pci="" bridge="">pcibus 1 on motherboard pci1: <pci bus="">on pcib1 pcib2: <pci-pci bridge="">at device 0.0 on pci1 pci2: <pci bus="">on pcib2 fxp0: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xec00-0xec3f mem 0xc0100000 -0xc0100fff,0xc0120000-0xc013ffff irq 11 at device 3.0 on pci2 fxp0: Disabling dynamic standby mode in EEPROM fxp0: New EEPROM ID: 0xfffd fxp0: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9 miibus0: <mii bus="">on fxp0 inphy0: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: [ITHREAD] fxp1: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe800-0xe83f mem 0xc0140000 -0xc0140fff,0xc0160000-0xc017ffff irq 11 at device 4.0 on pci2 fxp1: Disabling dynamic standby mode in EEPROM fxp1: New EEPROM ID: 0xfffd fxp1: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9 miibus1: <mii bus="">on fxp1 inphy1: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: [ITHREAD] fxp2: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe400-0xe43f mem 0xc0180000 -0xc0180fff,0xc01a0000-0xc01bffff irq 11 at device 5.0 on pci2 fxp2: Disabling dynamic standby mode in EEPROM fxp2: New EEPROM ID: 0xfffd fxp2: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9 miibus2: <mii bus="">on fxp2 inphy2: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus2 inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp2: [ITHREAD] fxp3: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe000-0xe03f mem 0xc01c0000 -0xc01c0fff,0xc01e0000-0xc01fffff irq 11 at device 6.0 on pci2 fxp3: Disabling dynamic standby mode in EEPROM fxp3: New EEPROM ID: 0xfffd fxp3: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9 miibus3: <mii bus="">on fxp3 inphy3: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus3 inphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp3: [ITHREAD] pcib3: <pci-pci bridge="">at device 1.0 on pci1 pci3: <pci bus="">on pcib3 dc0: <intel 10="" 21143="" 100basetx="">port 0xdc00-0xdc7f mem 0xc0200000-0xc02003ff irq 10 at device 5.0 on pci3 miibus4: <mii bus="">on dc0 bmtphy0: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus4 bmtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto dc0: [ITHREAD] dc1: <intel 10="" 21143="" 100basetx="">port 0xd800-0xd87f mem 0xc0200400-0xc02007ff irq 10 at device 6.0 on pci3 miibus5: <mii bus="">on dc1 bmtphy1: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus5 bmtphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto dc1: [ITHREAD] pcib4: <pci-pci bridge="">at device 2.0 on pci1 pci4: <pci bus="">on pcib4 dc2: <intel 10="" 21143="" 100basetx="">port 0xcc00-0xcc7f mem 0xc0300000-0xc03003ff irq 10 at device 5.0 on pci4 miibus6: <mii bus="">on dc2 bmtphy2: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus6 bmtphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto dc2: [ITHREAD] dc3: <intel 10="" 21143="" 100basetx="">port 0xc800-0xc87f mem 0xc0300400-0xc03007ff irq 10 at device 6.0 on pci4 miibus7: <mii bus="">on dc3 bmtphy3: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus7 bmtphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto dc3: [ITHREAD]</bcm5221></mii></intel></bcm5221></mii></intel></pci></pci-pci></bcm5221></mii></intel></bcm5221></mii></intel></pci></pci-pci></i82555></mii></intel></i82555></mii></intel></i82555></mii></intel></i82555></mii></intel></pci></pci-pci></pci></serverworks></serial>So, cross referencing both bootlogs, I suppose I have to put next settings in /boot/loader.conf.local (not in device.hints, by the way):
hint.pci2.3.0.INTD.irq="6" hint.pci2.4.0.INTA.irq="10" hint.pci2.5.0.INTB.irq="11" hint.pci2.6.0.INTC.irq="12" hint.pci3.5.0.INTB.irq="11" hint.pci3.6.0.INTC.irq="15" hint.pci4.5.0.INTB.irq="15" hint.pci4.6.0.INTC.irq="12"I am going to try later and I'll tell the results. If this doesn't make sense, please stop me before I risk to make my system unbootable.
-
By me if I check, I got the following information about the nics:
dmesg | grep dc ubsec0: Broadcom 5802 dc0: <intel 10="" 21143="" 100basetx=""> port 0xdc00-0xdc7f mem 0xc0200000-0xc02003ff irq 10 at device 5.0 on pci3 miibus4: <mii bus=""> on dc0 dc0: [ITHREAD] dc1: <intel 10="" 21143="" 100basetx=""> port 0xd800-0xd87f mem 0xc0200400-0xc02007ff irq 10 at device 6.0 on pci3 miibus5: <mii bus=""> on dc1 dc1: [ITHREAD]</mii></intel></mii></intel> -
Based on the posting by pierref of the IPSO output (which gives the actual IRQs for the NICs), I suggest```
hw.pci3.5.0.INTA.irg=11
hw.pci3.6.0.INTA.irg=15if the two NICs are in slot 1 andhw.pci3.5.0.INTA.irg=15
hw.pci3.6.0.INTA.irg=12The two NICs on the plugin cards are downstream of a PCI bridge which "swizzles" the interrupt lines. This may add a complication which I haven't accounted for. Please try the appropriate suggestion and report back the outcome. A further complication is that I'm not sure if the startup output reports the IRQ from the PCI configuration registers (wrong in many cases on IPSO systems) or the actual IRQ the software considers the device using. The output of pfSense shell command``` vmstat -i ```is probably the best report. -
Hi,
So I ried with the folowing both version without any result:
hint.pci3.5.0.INTB.irq="12" hint.pci3.6.0.INTC.irq="13"and
hw.pci3.5.0.INTB.irq=12
hw.pci3.6.0.INTC.irq=13 -
I tried:
hint.pci2.3.0.INTD.irq="6" hint.pci2.4.0.INTA.irq="10" hint.pci2.5.0.INTB.irq="11" hint.pci2.6.0.INTC.irq="12" hint.pci3.5.0.INTB.irq="11" hint.pci3.6.0.INTC.irq="15" hint.pci4.5.0.INTB.irq="15" hint.pci4.6.0.INTC.irq="12"and still have interrupt storm detected on "irq10:"; throttling interrupt source and something related to stray irq7.
The output of vmstat -i is
interrupt total rate irq0: clk 53059 99 irq4: uart0 13408 25 irq7: 14 0 stray irq7 14 0 irq10: dc0 dc1 dc2+ 744325 1396 irq11: ubsec0 fxp0* 51603 96 irq14: ata0 3684 6 Total 866107 1624I will try the suggestion of wallabybob and post the results here.
-
Posting the output of pciconf -lv of my configuration still having interrupt storm on irq10, in case somebody is inspired to fix it.
hostb0@pci0:0:0:0: class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 hdr=0x00 class = bridge subclass = HOST-PCI hostb1@pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 hdr=0x00 class = bridge subclass = HOST-PCI cbb0@pci0:0:9:0: class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 hdr=0x02 class = bridge subclass = PCI-CardBus cbb1@pci0:0:9:1: class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 hdr=0x02 class = bridge subclass = PCI-CardBus ubsec0@pci0:0:10:0: class=0x0b4000 card=0x00000000 chip=0x580214e4 rev=0x01 hdr=0x00 class = processor isab0@pci0:0:15:0: class=0x060100 card=0x02001166 chip=0x02001166 rev=0x50 hdr=0x00 class = bridge subclass = PCI-ISA atapci0@pci0:0:15:1: class=0x01018a card=0x00000000 chip=0x02111166 rev=0x00 hdr=0x00 class = mass storage subclass = ATA none0@pci0:0:15:2: class=0x0c0310 card=0x02201166 chip=0x02201166 rev=0x04 hdr=0x00 class = serial bus subclass = USB pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI pcib3@pci0:1:1:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI pcib4@pci0:1:2:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI fxp0@pci0:2:3:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp1@pci0:2:4:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp2@pci0:2:5:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp3@pci0:2:6:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet dc0@pci0:3:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc1@pci0:3:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc2@pci0:4:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc3@pci0:4:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernetAny help would be appreciated.
Pierre
-
Hi,
It's atrange that all these config are not afecting the irq maping!
The question is: is it possible to force a config?Pierre do you have a running iposo? If yes do you have checked if there is some kernel option set?
Alex
-
Hi Alex
I don't have any IPSO running no more. I can provide you the full output of the bootlog I saved some weeks ago, before I installed pfSense. See the attached file: that is all what I kept. I also have an image of the disk, but it is several GB big.
Meilleures salutations.
Pierre
-
Pierre and Alex, My apologies. My hasty reading of the source code led me to misinterpret the parameters in the loader variable assignments I posted earlier. I have done some testing and confirmed I can change the reported interrupts on my system through the mechanism I have suggested (after correction of the parameters).
I haven't seen any reports that either of you have followed exactly my previous suggestions. I hope we can all pay closer attention to the details.
Lets start with the fxps:
| Device | IPSO irq | FreeBSD irq |
| eth1/fxp0 | 6 | 11 |
| eth2/fxp1 | 10 | 11 |
| eth3/fxp2 | 11 | 11 |
| eth4/fxp3 | 12 | 11 |Suppose eth2/fxp1 is in use. It interrupts on irq10 but FreeBSD thinks it interrupts on irq11 so FreeBSD will call the interrupt handler for eth2/fxp1 on an irq11 interrupt. If eth2/fxp1 interrupts then FreeBSD will call the irq10 handlers (none of which is the actual eth2/fxp1 handler) none of which will clear the eth2/fxp1 interrupt condition so irq10 interrupt will happen again and again: irq10 storm!
The following lines in /boot/loader.conf.local should fix this:```
hw.pci0.2.3.INTA.irq=6
hw.pci0.2.4.INTA.irq=10
hw.pci0.2.5.INTA.irq=11
hw.pci0.2.6.INTA.irq=12where the value of the variable hw.pci0.2.3.INTA.irq specifies the interrupt line to use for the device with PCI address (PCI domain=0, PCI bus =2, PCI device number on bus = 3). Please try this, reboot after updating /boot/loader.conf.local and verify the irqs reported in the dmesg output and the output of
vmstat -iEdit: Corrected typo in lines for /boot/loader.conf.local -
Hi wallabybob
I am really happy to read your post. You look like the first person understanding exactly what's happening here. Alex and I are more Linux men and noobs in the world of pfSense/FreeBSD.
I am ready to test what you are proposing, but I will probably not be able to do it today, because of other priorities.
I will make my file /boot/loader.conf.local look like:
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12correcting the tipo on the last line: INTA/irq -> INTA.irq, and I will post the ouptput of vmstat -i
Regards.
Pierre
-
Hi,
I tried it but it's still not working as expected.
With the following loader.conf.local
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12 hw.pci0.3.5.INTA.irq=13 hw.pci0.3.6.INTA.irq=15I get these results
#vmstat -i interrupt total rate irq0: clk 32550 99 irq4: uart0 676 2 irq7: 21 0 stray irq7 21 0 irq8: rtc 41663 127 irq10: fxp1 1002 3 irq11: ubsec0 fxp2 32516 99 irq14: ata0 2177 6 Total 110626 339vmstat -ai interrupt total rate ??? 0 0 irq0: clk 33828 99 stray irq0 0 0 irq1: 0 0 stray irq1 0 0 irq3: uart1 0 0 stray irq3 0 0 irq4: uart0 728 2 stray irq4 0 0 irq5: 0 0 stray irq5 0 0 irq6: fxp0 0 0 stray irq6 0 0 irq7: 22 0 stray irq7 22 0 irq8: rtc 43300 127 stray irq8 0 0 irq9: 0 0 stray irq9 0 0 irq10: fxp1 1002 2 stray irq10 0 0 irq11: ubsec0 fxp2 33794 99 stray irq11 0 0 irq12: fxp3 0 0 stray irq12 0 0 irq13: dc0 0 0 stray irq13 0 0 irq14: ata0 2195 6 stray irq14 0 0 irq15: dc1 ata1 0 0 stray irq15 0 0 Total 114891 338The irq mapping has changed but the NICs are not working anymore and pfsense is slower as before.
I can configure each interface as usual but for example the local one is no more responding and the WAN gets no address.The good point is that the irq storm disappear …
Is our irq mapping wrong or is something other?
Alex
-
The irq mapping has changed but the NICs are not working anymore and pfsense is slower as before.
I can configure each interface as usual but for example the local one is no more responding and the WAN gets no address.The good point is that the irq storm disappear …
Is our irq mapping wrong or is something other?
Thanks for the data. Looks like some progress is being made. What is the front panel designation of the NICs you are using? and for for what purpose? (which is LAN? WAN? etc). From the interrupt counts it looks as if only two NICs are in use since only two have non-zero interrupt counts.
correcting the tipo on the last line: INTA/irq -> INTA.irq, and I will post the ouptput of vmstat -i
Thanks. Not a good start in attention to details on my part. I'll correct the error in the original post.
-
Currently I'm only using fxp0(wan) and fxp1(lan), I'm testing the box :D
But I have tried to use the other port for the lan interface with the same result.Alex
-
Currently I'm only using fxp0(wan) and fxp1(lan), I'm testing the box
But isn't there a front panel designation: eth1? eth2? etc.
fxp1 and fxp2 are the only NICs with non-zero interrupt counts so I suspect you are really using fxp1 and fxp2 but that you have done a mental translation from ethx (x=1, 2, 3, 4) to fxpn (n=0, 1, 2, 3). Correct?
One of the earlier replies gave a mapping from the ethx to the fxpn but with no mention of how that was derived. I'm suspecting that mapping is not correct. I have vague memory of some Nokia IP? boxes where the electrical ordering of NICs on PCI buses didn't correspond to the front panel ordering but its over 4 years ago that I worked on the Nokia IPxxx boxes.
-
Sorry, you are right I made a mental mapping
On the front panel I have Eth1 to Eth4.
I have checked and I have the following correspondence:Eth1 : fxp0
Eth2 : fxp1
Eth3 : fxp2
Eth4 : fxp3In the welcome screen of pfsense I get the same information about the up links.
And for my tests I'm using Eth1(fxp0) and Eth2(fxp1) for the wan and lan interface.
Alex
-
I have checked and I have the following correspondence:
Eth1 : fxp0
Eth2 : fxp1
Eth3 : fxp2
Eth4 : fxp3How did you determine this mapping is correct rather than (say) eth1 <-> fxp3 … ?
In the welcome screen of pfsense I get the same information about the up links.
Do you mean pfSense reports fxp0 and fxp1 up? (I don't know how pfSense would know about eth1 and eth2).
Do you see the problem: you say you are using fxp0 and fxp1 but fxp1 and fxp2 are the NICs with non-zero interrupt counts. Maybe your translation from the front panel label to fxpn is not correct? Maybe you moved one or more cables after booting? Can you account for the anomaly?
-
Hello wallabybob
I will make my file /boot/loader.conf.local look like:
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12With these settings I promised you to test, the irq10 storm is worse than ever and there was no connectivity on the NICs. I removed /boot/loader.conf.local and now, I can reboot and have connectivity. But the storm is still there. Up to now, I only use fxp0 (WAN) and fxp1 (LAN), just like Alex. We will have to try something different.