Pfsense Nokıa IP380 Install
-
By me if I check, I got the following information about the nics:
dmesg | grep dc ubsec0: Broadcom 5802 dc0: <intel 10="" 21143="" 100basetx=""> port 0xdc00-0xdc7f mem 0xc0200000-0xc02003ff irq 10 at device 5.0 on pci3 miibus4: <mii bus=""> on dc0 dc0: [ITHREAD] dc1: <intel 10="" 21143="" 100basetx=""> port 0xd800-0xd87f mem 0xc0200400-0xc02007ff irq 10 at device 6.0 on pci3 miibus5: <mii bus=""> on dc1 dc1: [ITHREAD]</mii></intel></mii></intel> -
Based on the posting by pierref of the IPSO output (which gives the actual IRQs for the NICs), I suggest```
hw.pci3.5.0.INTA.irg=11
hw.pci3.6.0.INTA.irg=15if the two NICs are in slot 1 andhw.pci3.5.0.INTA.irg=15
hw.pci3.6.0.INTA.irg=12The two NICs on the plugin cards are downstream of a PCI bridge which "swizzles" the interrupt lines. This may add a complication which I haven't accounted for. Please try the appropriate suggestion and report back the outcome. A further complication is that I'm not sure if the startup output reports the IRQ from the PCI configuration registers (wrong in many cases on IPSO systems) or the actual IRQ the software considers the device using. The output of pfSense shell command``` vmstat -i ```is probably the best report. -
Hi,
So I ried with the folowing both version without any result:
hint.pci3.5.0.INTB.irq="12" hint.pci3.6.0.INTC.irq="13"and
hw.pci3.5.0.INTB.irq=12
hw.pci3.6.0.INTC.irq=13 -
I tried:
hint.pci2.3.0.INTD.irq="6" hint.pci2.4.0.INTA.irq="10" hint.pci2.5.0.INTB.irq="11" hint.pci2.6.0.INTC.irq="12" hint.pci3.5.0.INTB.irq="11" hint.pci3.6.0.INTC.irq="15" hint.pci4.5.0.INTB.irq="15" hint.pci4.6.0.INTC.irq="12"and still have interrupt storm detected on "irq10:"; throttling interrupt source and something related to stray irq7.
The output of vmstat -i is
interrupt total rate irq0: clk 53059 99 irq4: uart0 13408 25 irq7: 14 0 stray irq7 14 0 irq10: dc0 dc1 dc2+ 744325 1396 irq11: ubsec0 fxp0* 51603 96 irq14: ata0 3684 6 Total 866107 1624I will try the suggestion of wallabybob and post the results here.
-
Posting the output of pciconf -lv of my configuration still having interrupt storm on irq10, in case somebody is inspired to fix it.
hostb0@pci0:0:0:0: class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 hdr=0x00 class = bridge subclass = HOST-PCI hostb1@pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 hdr=0x00 class = bridge subclass = HOST-PCI cbb0@pci0:0:9:0: class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 hdr=0x02 class = bridge subclass = PCI-CardBus cbb1@pci0:0:9:1: class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 hdr=0x02 class = bridge subclass = PCI-CardBus ubsec0@pci0:0:10:0: class=0x0b4000 card=0x00000000 chip=0x580214e4 rev=0x01 hdr=0x00 class = processor isab0@pci0:0:15:0: class=0x060100 card=0x02001166 chip=0x02001166 rev=0x50 hdr=0x00 class = bridge subclass = PCI-ISA atapci0@pci0:0:15:1: class=0x01018a card=0x00000000 chip=0x02111166 rev=0x00 hdr=0x00 class = mass storage subclass = ATA none0@pci0:0:15:2: class=0x0c0310 card=0x02201166 chip=0x02201166 rev=0x04 hdr=0x00 class = serial bus subclass = USB pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI pcib3@pci0:1:1:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI pcib4@pci0:1:2:0: class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 hdr=0x01 class = bridge subclass = PCI-PCI fxp0@pci0:2:3:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp1@pci0:2:4:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp2@pci0:2:5:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet fxp3@pci0:2:6:0: class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 hdr=0x00 class = network subclass = ethernet dc0@pci0:3:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc1@pci0:3:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc2@pci0:4:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernet dc3@pci0:4:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00 class = network subclass = ethernetAny help would be appreciated.
Pierre
-
Hi,
It's atrange that all these config are not afecting the irq maping!
The question is: is it possible to force a config?Pierre do you have a running iposo? If yes do you have checked if there is some kernel option set?
Alex
-
Hi Alex
I don't have any IPSO running no more. I can provide you the full output of the bootlog I saved some weeks ago, before I installed pfSense. See the attached file: that is all what I kept. I also have an image of the disk, but it is several GB big.
Meilleures salutations.
Pierre
-
Pierre and Alex, My apologies. My hasty reading of the source code led me to misinterpret the parameters in the loader variable assignments I posted earlier. I have done some testing and confirmed I can change the reported interrupts on my system through the mechanism I have suggested (after correction of the parameters).
I haven't seen any reports that either of you have followed exactly my previous suggestions. I hope we can all pay closer attention to the details.
Lets start with the fxps:
| Device | IPSO irq | FreeBSD irq |
| eth1/fxp0 | 6 | 11 |
| eth2/fxp1 | 10 | 11 |
| eth3/fxp2 | 11 | 11 |
| eth4/fxp3 | 12 | 11 |Suppose eth2/fxp1 is in use. It interrupts on irq10 but FreeBSD thinks it interrupts on irq11 so FreeBSD will call the interrupt handler for eth2/fxp1 on an irq11 interrupt. If eth2/fxp1 interrupts then FreeBSD will call the irq10 handlers (none of which is the actual eth2/fxp1 handler) none of which will clear the eth2/fxp1 interrupt condition so irq10 interrupt will happen again and again: irq10 storm!
The following lines in /boot/loader.conf.local should fix this:```
hw.pci0.2.3.INTA.irq=6
hw.pci0.2.4.INTA.irq=10
hw.pci0.2.5.INTA.irq=11
hw.pci0.2.6.INTA.irq=12where the value of the variable hw.pci0.2.3.INTA.irq specifies the interrupt line to use for the device with PCI address (PCI domain=0, PCI bus =2, PCI device number on bus = 3). Please try this, reboot after updating /boot/loader.conf.local and verify the irqs reported in the dmesg output and the output of
vmstat -iEdit: Corrected typo in lines for /boot/loader.conf.local -
Hi wallabybob
I am really happy to read your post. You look like the first person understanding exactly what's happening here. Alex and I are more Linux men and noobs in the world of pfSense/FreeBSD.
I am ready to test what you are proposing, but I will probably not be able to do it today, because of other priorities.
I will make my file /boot/loader.conf.local look like:
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12correcting the tipo on the last line: INTA/irq -> INTA.irq, and I will post the ouptput of vmstat -i
Regards.
Pierre
-
Hi,
I tried it but it's still not working as expected.
With the following loader.conf.local
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12 hw.pci0.3.5.INTA.irq=13 hw.pci0.3.6.INTA.irq=15I get these results
#vmstat -i interrupt total rate irq0: clk 32550 99 irq4: uart0 676 2 irq7: 21 0 stray irq7 21 0 irq8: rtc 41663 127 irq10: fxp1 1002 3 irq11: ubsec0 fxp2 32516 99 irq14: ata0 2177 6 Total 110626 339vmstat -ai interrupt total rate ??? 0 0 irq0: clk 33828 99 stray irq0 0 0 irq1: 0 0 stray irq1 0 0 irq3: uart1 0 0 stray irq3 0 0 irq4: uart0 728 2 stray irq4 0 0 irq5: 0 0 stray irq5 0 0 irq6: fxp0 0 0 stray irq6 0 0 irq7: 22 0 stray irq7 22 0 irq8: rtc 43300 127 stray irq8 0 0 irq9: 0 0 stray irq9 0 0 irq10: fxp1 1002 2 stray irq10 0 0 irq11: ubsec0 fxp2 33794 99 stray irq11 0 0 irq12: fxp3 0 0 stray irq12 0 0 irq13: dc0 0 0 stray irq13 0 0 irq14: ata0 2195 6 stray irq14 0 0 irq15: dc1 ata1 0 0 stray irq15 0 0 Total 114891 338The irq mapping has changed but the NICs are not working anymore and pfsense is slower as before.
I can configure each interface as usual but for example the local one is no more responding and the WAN gets no address.The good point is that the irq storm disappear …
Is our irq mapping wrong or is something other?
Alex
-
The irq mapping has changed but the NICs are not working anymore and pfsense is slower as before.
I can configure each interface as usual but for example the local one is no more responding and the WAN gets no address.The good point is that the irq storm disappear …
Is our irq mapping wrong or is something other?
Thanks for the data. Looks like some progress is being made. What is the front panel designation of the NICs you are using? and for for what purpose? (which is LAN? WAN? etc). From the interrupt counts it looks as if only two NICs are in use since only two have non-zero interrupt counts.
correcting the tipo on the last line: INTA/irq -> INTA.irq, and I will post the ouptput of vmstat -i
Thanks. Not a good start in attention to details on my part. I'll correct the error in the original post.
-
Currently I'm only using fxp0(wan) and fxp1(lan), I'm testing the box :D
But I have tried to use the other port for the lan interface with the same result.Alex
-
Currently I'm only using fxp0(wan) and fxp1(lan), I'm testing the box
But isn't there a front panel designation: eth1? eth2? etc.
fxp1 and fxp2 are the only NICs with non-zero interrupt counts so I suspect you are really using fxp1 and fxp2 but that you have done a mental translation from ethx (x=1, 2, 3, 4) to fxpn (n=0, 1, 2, 3). Correct?
One of the earlier replies gave a mapping from the ethx to the fxpn but with no mention of how that was derived. I'm suspecting that mapping is not correct. I have vague memory of some Nokia IP? boxes where the electrical ordering of NICs on PCI buses didn't correspond to the front panel ordering but its over 4 years ago that I worked on the Nokia IPxxx boxes.
-
Sorry, you are right I made a mental mapping
On the front panel I have Eth1 to Eth4.
I have checked and I have the following correspondence:Eth1 : fxp0
Eth2 : fxp1
Eth3 : fxp2
Eth4 : fxp3In the welcome screen of pfsense I get the same information about the up links.
And for my tests I'm using Eth1(fxp0) and Eth2(fxp1) for the wan and lan interface.
Alex
-
I have checked and I have the following correspondence:
Eth1 : fxp0
Eth2 : fxp1
Eth3 : fxp2
Eth4 : fxp3How did you determine this mapping is correct rather than (say) eth1 <-> fxp3 … ?
In the welcome screen of pfsense I get the same information about the up links.
Do you mean pfSense reports fxp0 and fxp1 up? (I don't know how pfSense would know about eth1 and eth2).
Do you see the problem: you say you are using fxp0 and fxp1 but fxp1 and fxp2 are the NICs with non-zero interrupt counts. Maybe your translation from the front panel label to fxpn is not correct? Maybe you moved one or more cables after booting? Can you account for the anomaly?
-
Hello wallabybob
I will make my file /boot/loader.conf.local look like:
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12With these settings I promised you to test, the irq10 storm is worse than ever and there was no connectivity on the NICs. I removed /boot/loader.conf.local and now, I can reboot and have connectivity. But the storm is still there. Up to now, I only use fxp0 (WAN) and fxp1 (LAN), just like Alex. We will have to try something different.
-
Hi wallabybob
I promised you:
I will make my file /boot/loader.conf.local look like:
hw.pci0.2.3.INTA.irq=6 hw.pci0.2.4.INTA.irq=10 hw.pci0.2.5.INTA.irq=11 hw.pci0.2.6.INTA.irq=12I am sorry to tell you these settings are worse than before: the irq10 storm was heavier, and the NIC's were no longer working. The only ports I am using now are fxp0 (WAN) and fxp1 (LAN), just as Alex. If you want me to try something different, I am ready to do it.
Thanks in advance.
-
I can not run, unfortunately. I get an error when installing the USB
-
Hi Alex and wallabybob
I found a set of entries in /boot/loader.conf.local where the NICs are working, at least fxp0 and fxp1, and no irq10 storm. Here is my file:
hw.pci0.2.3.INTD.irq="6" hw.pci0.2.4.INTA.irq="10" hw.pci0.2.5.INTB.irq="11" hw.pci0.2.6.INTC.irq="12" hw.pci0.3.5.INTB.irq="11" hw.pci0.3.6.INTC.irq="15" hw.pci0.4.5.INTB.irq="15" hw.pci0.4.6.INTC.irq="12"Basically, these are the values collected from IPSO but in the correct format: lines starting with hw and not hint, correct order of parameters for PCI domain/bus/device_number.
I will connect more switches to the ports in the hope that all the NICs will go on working, and I will inform you.
Oh yes: and this is the output of # vmstat -i
interrupt total rate
irq0: clk 16552 99
irq4: uart0 560 3
irq7: 23 0
stray irq7 23 0
irq10: fxp1 dc0 dc* 2158 13
irq11: ubsec0 fxp0* 16507 99
irq14: ata0 2765 16
Total 38588 232There is something strange with this output: irq6, irq12 and irq15 are not set as I expected, but… it works.
Anyway, I hope these setting will also work for Alex: since he has a subset of my NICs in his configuration, this should do the job for him too.
Regards.
Pierre
-
Sorry, other commitments mean I will have very limited time to devote to this for at least the next week, maybe the next two weeks.
There are a number of inconsistencies in the accounts so far:
1. Alex and Pierre both say they are using fxp0 and fxp1 but fxp1 and fxp2 show significant interrupt rates for Alex while Pierre sees significant interrupt rates for fxp0 and fxp1.
2. Pierre reports interrupt storm on irq10 with my suggested modifications to /boot/loader.conf.local but the IPSO startup reports only one device on irq10: fxp1 and fxp1 was assigned to irq10 in my suggested modifications.
Maybe the boxes have different motherboards. Maybe the IPSO startup output we have is from a different box
From memory, the front panels of these devices designate the onboard (not on expansion slot) NICs as eth1, eth2, eth3 and eth4 but both Pierre and Alex report devices fxp0 and fxp1 in use. To get this to work correctly we need appropriate IPSO source code (used to be available from Nokia on request if I recall correctly; don't know about Checkpoint who took over the firewalls from Nokia) or an accurate association between front panel port name, FreeBSD device name and IRQ. Both Pierre and Alex have made associations between front panel port names and FreeBSD device names. I want to know how these associations were derived. If they have been assumed then they can't be trusted and they will need to be derived from the source code or worked out by experiment..
If anyone can track down the source code for a version of IPSO that supported IP380 I'll take a look at it to try to derive the association I requested.
There is something strange with this output: irq6, irq12 and irq15 are not set as I expected, but… it works.
All the hw.pci….irq entries for fxp NICs and dc NICs need to specify INTA because these devices request interrupts on their INTA line. The INTx reported by IPSO refer to interrupt lines on the motherboard which is different from interrupt lines the actual devices use. More detailed explanation will have to wait.