Pfsense Nokıa IP380 Install

psykok

hi,

Merci je suis bien en france :D

I have performed some test today on the box and pfsense is running without any problem.
I can assign all my 4 NICs. The configuration is kept over reboot or power failure by me.
I have not changed anything in the config.xml, the MAC addresses are generated automatically when I’m configure an interface.

I know that Nokia has released several version of the IP380. Currently I have two different version (grey and white box), both are now running pfsense. I have only some hardware issues with the grey box … it’s not always booting : (
Maybe there is some difference between the NICs.

Alex

pierref

Hello Alex

I also have an IP380 with a grey box. I have 8 NICs. fxp[0-3] (Intel EtherExpress Pro 10/100B Ethernet) at the right and dc[0-3] (Digital DC21143 Fast Ethernet) at the left. The Intel ones are running. The Digital ones not yet. Worse: sometimes, I have an interrupt storm detected on "irq10", which is the bios irq of the dc[0-3] NICs, and my webconfigurator becomes non responsive, needing a reboot at the serial console. It is possible to unplug the Digital NICs.

I could redesign my LAN for 4 NICs and forget about the 4 NICs at the left, but it is pity…

I suppose you don't have these Digital DC21143 Fast Ethernet in your configuration.

Anyway, the system is running and that's the most important.

Vive la France.

Pierre

psykok

Hi,

I have 6 NICS in my grey box, the 4 fxp and 2 other on the first extension card.

Unfortunately I have some trouble with this version, I think something is going wron on the hardware level because the box is not booting correctly.

On the other box I have only the 4 standard ports.

But for your problem if the Digital interfaces are visible using the command ifconfig the system has loaded some drivers for it.
If the driver is good working, that’s another point.

For the irq storm did you try to modify something in the bios?

Alex

psykok

So my gray box booted again and I have check the config of the additional NIC card.
Same error as you with the IRQ, the two port are detected but not working, the 4 default NICs are working without any problem.

I will check if there is some way to change the irq mapping on the firewall but I remember that the bios is a special light version with a very few available options.

pierref

When the system was running under IPSO, the Digital NICs used irq 11, 12 and 15.

If there is any way to assign other irq's to the Digital NIC's (through the device.hints file), it could solve the problem.

I would try something like adding some lines in the /conf/device.hints file, like

hint.dc.0.irq="10"
hint.dc.1.irq="12"
hint.dc.2.irq="13"
hint.dc.3.irq="15"

avoiding the irq that are already listed in the output of vmstat -i

Does this make sense? I am a software man, not a hardware guru.

pierref

Anyway, my entries in the device.hints file are not taken into consideration. Issuing vmstat -i shows the dc card is still using irq 10.

psykok

Did you check the bios if some irq setting is available?

I don't know if there is a way to force the irq for these nics.

Alex

wallabybob

In FreeBSD the calculated IRQ for a PCI device can be overridden by an entry in /boot/loader.conf.local (preferred since it won't be changed by a pfSense install) or /boot/loader.conf of the form:```
hw.pci1.2.3.INTA.irg=15


If you have a suitable IPSO installation or the startup output from a previous IPSO boot you should be able to determine the IRQs of each of the NICs. The challenge is then to map from the IPSO device name (e.g. eth-1, eth-s1p2) to FreeBSD device name (e.g. fxp1, dc2).

psykok

I give a try to your solution by modifying the /boot/loader.conf but it makes no difference.

It seems that the entry I write in the file are not affecting the kernel.
I use the following config:
hw.pci3.5.0.dc0.irq=12
hw.pci3.6.0.dc1.irq=13

I checked the pci addresse swith dmesg.

ALex

wallabybob

@psykok:

It seems that the entry I write in the file are not affecting the kernel.
I use the following config:
hw.pci3.5.0.dc0.irq=12
hw.pci3.6.0.dc1.irq=13

You should have used INTA rather than dc0 and dc1. See my post about this!

It might be helpful to have a bit of background on PCI devices. Each PCI device can use up to 4 interrupt request lines. They are called INTA, INTB, INTC, INTD. Most devices use INTA. The PCI interrupt request lines are connected to IRQs according to the whim of the motherboard designer. The /boot/loader.conf.local line```
hw.pci3.5.0.INTA.irq=12


As far as I know, there is no code that would act on a line like:```
hw.pci3.5.0.dc0.irq=12

The bus, device and function numbers you have used look plausible but I have no idea if they are correct.

Edit: I had the parameters in the loader variable name incorrect and consequently the suggested variables and values in this reply won't work.

pierref

Hi

First of all, my system is running even with interrupt storm on irq10.

The mapping of the NICs IPSO <-> pfSense can be deduced from the bootlog of IPSO, below:

eth-s1p1 <-> dc0
eth-s1p2 <-> dc1
eth-s2p1 <-> dc2
eth-s2p2 <-> dc3
eth1 <-> fxp0
eth2 <-> fxp1
eth3 <-> fxp2
eth4 <-> fxp3

This is confirmed by the experience of connecting stuff to these ports. The four first NICs, the Digital ones, are the left ones: eth-s1p2 stands for slot 1 port 2, the other ones are the Intel NICs.

From the same bootlog, I can extract the INTA, INTB, etc. and the corresponding irq, but not the PCI bus number. I don't know what "rev" and "onboard" stand for.

pcidec0 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 on pci1:0:0                
fxp0 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int d irq 6 onboard 1     
netlog:eth1 .. Ethernet address 0:a0:8e:78:c1:b4                               
fxp1 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int a irq 10 onboard 2    
netlog:eth2 .. Ethernet address 0:a0:8e:78:c1:b5                               
fxp2 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int b irq 11 onboard 3    
netlog:eth3 .. Ethernet address 0:a0:8e:78:c1:b6                               
fxp3 <intel 10="" etherexpress="" pro="" 100b="" ethernet=""> rev 9 int c irq 12 onboard 4    
netlog:eth4 .. Ethernet address 0:a0:8e:78:c1:b7                               
pcidec1 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 slot 1                     
tulip0 <digital dc21143="" fast="" ethernet=""> rev 65 int b irq 11 slot 1 port 1       
netlog:eth-s1p1 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:ac         
netlog:eth-s1p1 .. enabling 10baseT/UTP port in half duplex mode               
tulip1 <digital dc21143="" fast="" ethernet=""> rev 65 int c irq 15 slot 1 port 2       
netlog:eth-s1p2 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:ad         
netlog:eth-s1p2 .. enabling 10baseT/UTP port in half duplex mode               
pcidec2 <intel 21154be="" 64-bit="" pci-pci="" bridge=""> rev 0 slot 2                     
tulip2 <digital dc21143="" fast="" ethernet=""> rev 65 int b irq 15 slot 2 port 1       
netlog:eth-s2p1 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:b0         
netlog:eth-s2p1 .. enabling 10baseT/UTP port in half duplex mode               
tulip3 <digital dc21143="" fast="" ethernet=""> rev 65 int c irq 12 slot 2 port 2       
netlog:eth-s2p2 .. Generic 2114x DC21143 pass 4.1 -- 00:a0:8e:78:c1:b1         
netlog:eth-s2p2 .. enabling 10baseT/UTP port in half duplex mode</digital></digital></intel></digital></digital></intel></intel></intel></intel></intel></intel>

For giving values for the PCI bus, I suppose I have to look into the bootlog of pfSense this time:

pci0: <serial bus,="" usb="">at device 15.2 (no driver attached)
pcib1: <serverworks nb6635="" 3.0le="" host="" to="" pci="" bridge="">pcibus 1 on motherboard
pci1: <pci bus="">on pcib1
pcib2: <pci-pci bridge="">at device 0.0 on pci1
pci2: <pci bus="">on pcib2
fxp0: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xec00-0xec3f mem 0xc0100000
-0xc0100fff,0xc0120000-0xc013ffff irq 11 at device 3.0 on pci2
fxp0: Disabling dynamic standby mode in EEPROM
fxp0: New EEPROM ID: 0xfffd
fxp0: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9
miibus0: <mii bus="">on fxp0
inphy0: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: [ITHREAD]
fxp1: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe800-0xe83f mem 0xc0140000
-0xc0140fff,0xc0160000-0xc017ffff irq 11 at device 4.0 on pci2
fxp1: Disabling dynamic standby mode in EEPROM
fxp1: New EEPROM ID: 0xfffd
fxp1: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9
miibus1: <mii bus="">on fxp1
inphy1: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: [ITHREAD]
fxp2: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe400-0xe43f mem 0xc0180000
-0xc0180fff,0xc01a0000-0xc01bffff irq 11 at device 5.0 on pci2
fxp2: Disabling dynamic standby mode in EEPROM
fxp2: New EEPROM ID: 0xfffd
fxp2: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9
miibus2: <mii bus="">on fxp2
inphy2: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus2
inphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp2: [ITHREAD]
fxp3: <intel 10="" 100="" 82559er="" embedded="" ethernet="">port 0xe000-0xe03f mem 0xc01c0000
-0xc01c0fff,0xc01e0000-0xc01fffff irq 11 at device 6.0 on pci2
fxp3: Disabling dynamic standby mode in EEPROM
fxp3: New EEPROM ID: 0xfffd
fxp3: EEPROM checksum @ 0xff: 0xffff -> 0xbbb9
miibus3: <mii bus="">on fxp3
inphy3: <i82555 10="" 100="" media="" interface="">PHY 1 on miibus3
inphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp3: [ITHREAD]
pcib3: <pci-pci bridge="">at device 1.0 on pci1
pci3: <pci bus="">on pcib3
dc0: <intel 10="" 21143="" 100basetx="">port 0xdc00-0xdc7f mem 0xc0200000-0xc02003ff irq
 10 at device 5.0 on pci3
miibus4: <mii bus="">on dc0
bmtphy0: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus4
bmtphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc0: [ITHREAD]
dc1: <intel 10="" 21143="" 100basetx="">port 0xd800-0xd87f mem 0xc0200400-0xc02007ff irq
 10 at device 6.0 on pci3
miibus5: <mii bus="">on dc1
bmtphy1: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus5
bmtphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc1: [ITHREAD]
pcib4: <pci-pci bridge="">at device 2.0 on pci1
pci4: <pci bus="">on pcib4
dc2: <intel 10="" 21143="" 100basetx="">port 0xcc00-0xcc7f mem 0xc0300000-0xc03003ff irq
 10 at device 5.0 on pci4
miibus6: <mii bus="">on dc2
bmtphy2: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus6
bmtphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc2: [ITHREAD]
dc3: <intel 10="" 21143="" 100basetx="">port 0xc800-0xc87f mem 0xc0300400-0xc03007ff irq
 10 at device 6.0 on pci4
miibus7: <mii bus="">on dc3
bmtphy3: <bcm5221 10="" 100basetx="" phy="">PHY 1 on miibus7
bmtphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc3: [ITHREAD]</bcm5221></mii></intel></bcm5221></mii></intel></pci></pci-pci></bcm5221></mii></intel></bcm5221></mii></intel></pci></pci-pci></i82555></mii></intel></i82555></mii></intel></i82555></mii></intel></i82555></mii></intel></pci></pci-pci></pci></serverworks></serial>

So, cross referencing both bootlogs, I suppose I have to put next settings in /boot/loader.conf.local (not in device.hints, by the way):

hint.pci2.3.0.INTD.irq="6"
hint.pci2.4.0.INTA.irq="10"
hint.pci2.5.0.INTB.irq="11"
hint.pci2.6.0.INTC.irq="12"
hint.pci3.5.0.INTB.irq="11"
hint.pci3.6.0.INTC.irq="15"
hint.pci4.5.0.INTB.irq="15"
hint.pci4.6.0.INTC.irq="12"

I am going to try later and I'll tell the results. If this doesn't make sense, please stop me before I risk to make my system unbootable.

psykok

By me if I check, I got the following information about the nics:

dmesg | grep dc
ubsec0: Broadcom 5802
dc0: <intel 10="" 21143="" 100basetx=""> port 0xdc00-0xdc7f mem 0xc0200000-0xc02003ff irq 10 at device 5.0 on pci3
miibus4: <mii bus=""> on dc0
dc0: [ITHREAD]
dc1: <intel 10="" 21143="" 100basetx=""> port 0xd800-0xd87f mem 0xc0200400-0xc02007ff irq 10 at device 6.0 on pci3
miibus5: <mii bus=""> on dc1
dc1: [ITHREAD]</mii></intel></mii></intel>

wallabybob

Based on the posting by pierref of the IPSO output (which gives the actual IRQs for the NICs), I suggest```

hw.pci3.5.0.INTA.irg=11
hw.pci3.6.0.INTA.irg=15

if the two NICs are in slot 1 and

hw.pci3.5.0.INTA.irg=15
hw.pci3.6.0.INTA.irg=12


The two NICs on the plugin cards are downstream of a PCI bridge which "swizzles" the interrupt lines. This may add a complication which I haven't accounted for. Please try the appropriate suggestion and report back the outcome. A further complication is that I'm not sure if the startup output reports the IRQ from the PCI configuration registers (wrong in many cases on IPSO systems) or the actual IRQ the software considers the device using. The output of pfSense shell command```
vmstat -i
```is probably the best report.

psykok

Hi,

So I ried with the folowing both version without any result:

hint.pci3.5.0.INTB.irq="12"
hint.pci3.6.0.INTC.irq="13"

and

hw.pci3.5.0.INTB.irq=12
hw.pci3.6.0.INTC.irq=13

pierref

I tried:

hint.pci2.3.0.INTD.irq="6"
hint.pci2.4.0.INTA.irq="10"
hint.pci2.5.0.INTB.irq="11"
hint.pci2.6.0.INTC.irq="12"
hint.pci3.5.0.INTB.irq="11"
hint.pci3.6.0.INTC.irq="15"
hint.pci4.5.0.INTB.irq="15"
hint.pci4.6.0.INTC.irq="12"

and still have interrupt storm detected on "irq10:"; throttling interrupt source and something related to stray irq7.

The output of vmstat -i is

interrupt                          total       rate
irq0: clk                          53059         99
irq4: uart0                        13408         25
irq7:                                 14          0
stray irq7                            14          0
irq10: dc0 dc1 dc2+               744325       1396
irq11: ubsec0 fxp0*                51603         96
irq14: ata0                         3684          6
Total                             866107       1624

I will try the suggestion of wallabybob and post the results here.

pierref

Posting the output of pciconf -lv of my configuration still having interrupt storm on irq10, in case somebody is inspired to fix it.

hostb0@pci0:0:0:0:      class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 
hdr=0x00
    class      = bridge
    subclass   = HOST-PCI
hostb1@pci0:0:0:1:      class=0x060000 card=0x00000000 chip=0x00091166 rev=0x06 
hdr=0x00
    class      = bridge
    subclass   = HOST-PCI
cbb0@pci0:0:9:0:        class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 
hdr=0x02
    class      = bridge
    subclass   = PCI-CardBus
cbb1@pci0:0:9:1:        class=0x060700 card=0x00000000 chip=0xac1c104c rev=0x01 
hdr=0x02
    class      = bridge
    subclass   = PCI-CardBus
ubsec0@pci0:0:10:0:     class=0x0b4000 card=0x00000000 chip=0x580214e4 rev=0x01 
hdr=0x00
    class      = processor
isab0@pci0:0:15:0:      class=0x060100 card=0x02001166 chip=0x02001166 rev=0x50 
hdr=0x00
    class      = bridge
    subclass   = PCI-ISA
atapci0@pci0:0:15:1:    class=0x01018a card=0x00000000 chip=0x02111166 rev=0x00 
hdr=0x00
    class      = mass storage
    subclass   = ATA
none0@pci0:0:15:2:      class=0x0c0310 card=0x02201166 chip=0x02201166 rev=0x04 
hdr=0x00
    class      = serial bus
    subclass   = USB
pcib2@pci0:1:0:0:       class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 
hdr=0x01
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:1:1:0:       class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 
hdr=0x01
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:1:2:0:       class=0x060400 card=0x00000000 chip=0xb1548086 rev=0x00 
hdr=0x01
    class      = bridge
    subclass   = PCI-PCI
fxp0@pci0:2:3:0:        class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 
hdr=0x00
    class      = network
    subclass   = ethernet
fxp1@pci0:2:4:0:        class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 
hdr=0x00
    class      = network
    subclass   = ethernet
fxp2@pci0:2:5:0:        class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 
hdr=0x00
    class      = network
    subclass   = ethernet
fxp3@pci0:2:6:0:        class=0x020000 card=0x00000000 chip=0x12098086 rev=0x09 
hdr=0x00
    class      = network
    subclass   = ethernet
dc0@pci0:3:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00
    class      = network
    subclass   = ethernet
dc1@pci0:3:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00
    class      = network
    subclass   = ethernet
dc2@pci0:4:5:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00
    class      = network
    subclass   = ethernet
dc3@pci0:4:6:0: class=0x020000 card=0x61e00040 chip=0x00191011 rev=0x41 hdr=0x00
    class      = network
    subclass   = ethernet

Any help would be appreciated.

Pierre

psykok

Hi,

It's atrange that all these config are not afecting the irq maping!
The question is: is it possible to force a config?

Pierre do you have a running iposo? If yes do you have checked if there is some kernel option set?

Alex

pierref

Hi Alex

I don't have any IPSO running no more. I can provide you the full output of the bootlog I saved some weeks ago, before I installed pfSense. See the attached file: that is all what I kept. I also have an image of the disk, but it is several GB big.

Meilleures salutations.

Pierre

mac-addresses-nokia-ip380.txt

wallabybob

Pierre and Alex, My apologies. My hasty reading of the source code led me to misinterpret the parameters in the loader variable assignments I posted earlier. I have done some testing and confirmed I can change the reported interrupts on my system through the mechanism I have suggested (after correction of the parameters).

I haven't seen any reports that either of you have followed exactly my previous suggestions. I hope we can all pay closer attention to the details.

Lets start with the fxps:

| Device | IPSO irq | FreeBSD irq |
| eth1/fxp0 | 6 | 11 |
| eth2/fxp1 | 10 | 11 |
| eth3/fxp2 | 11 | 11 |
| eth4/fxp3 | 12 | 11 |

Suppose eth2/fxp1 is in use. It interrupts on irq10 but FreeBSD thinks it interrupts on irq11 so FreeBSD will call the interrupt handler for eth2/fxp1 on an irq11 interrupt. If eth2/fxp1 interrupts then FreeBSD will call the irq10 handlers (none of which is the actual eth2/fxp1 handler) none of which will clear the eth2/fxp1 interrupt condition so irq10 interrupt will happen again and again: irq10 storm!

The following lines in /boot/loader.conf.local should fix this:```

hw.pci0.2.3.INTA.irq=6
hw.pci0.2.4.INTA.irq=10
hw.pci0.2.5.INTA.irq=11
hw.pci0.2.6.INTA.irq=12

where the value of the variable hw.pci0.2.3.INTA.irq specifies the interrupt line to use for the device with PCI address (PCI domain=0, PCI bus =2, PCI device number on bus = 3). Please try this, reboot after updating /boot/loader.conf.local and verify the irqs reported in the dmesg output and the output of
vmstat -i


Edit: Corrected typo in lines for /boot/loader.conf.local

pierref

Hi wallabybob

I am really happy to read your post. You look like the first person understanding exactly what's happening here. Alex and I are more Linux men and noobs in the world of pfSense/FreeBSD.

I am ready to test what you are proposing, but I will probably not be able to do it today, because of other priorities.

I will make my file /boot/loader.conf.local look like:

hw.pci0.2.3.INTA.irq=6
hw.pci0.2.4.INTA.irq=10
hw.pci0.2.5.INTA.irq=11
hw.pci0.2.6.INTA.irq=12

correcting the tipo on the last line: INTA/irq -> INTA.irq, and I will post the ouptput of vmstat -i

Regards.

Pierre