Wired Memory grows rapidly when rule with limiter applied

maxk

I have following problem – when firewall rule with limiter is applied – system wired memory grows rapidly. If I have no rules with limiter – that’s all good with memory. On the enclosed picture I have shown how memory grows. Have anyone had such problem? May be – someone have any ideas to solve this problem?

pfsense_memory_overload.JPG_thumb

maxk

According to my previous post – here is my firewall rule. May be – this rule is incorrect?
Why does such rule can cause memory grow rapidly (only manual restart can save system)?

![limiter + rule.JPG](/public/imported_attachments/1/limiter + rule.JPG)
![limiter + rule.JPG_thumb](/public/imported_attachments/1/limiter + rule.JPG_thumb)

eri--

You just need to upgrade.

maxk

@ermal:

You just need to upgrade.

I have tested few versions. In the last one the same error. (Rules were identical to previous post) See attachment. Can anybody help to solve problem?

![pfsense - last version.JPG](/public/imported_attachments/1/pfsense - last version.JPG)
![pfsense - last version.JPG_thumb](/public/imported_attachments/1/pfsense - last version.JPG_thumb)

stompro

Interesting, I'm seeing the same thing, but just didn't notice it since it was taking a month or more for the machine to lock up from the memory being used up.

"ipfw pipe show" doesn't look out of the ordinary. Hmm.

I'm using a freebsd 7.2 snapshot from aug (2.0 alpha-alpha) on the machine I'm seeing the problem on. (Alix) So it has been around since then at least.
Josh

maxk

@stompro:

Interesting, I'm seeing the same thing, but just didn't notice it since it was taking a month or more for the machine to lock up from the memory being used up.

"ipfw pipe show" doesn't look out of the ordinary. Hmm.

I'm using a freebsd 7.2 snapshot from aug (2.0 alpha-alpha) on the machine I'm seeing the problem on. (Alix) So it has been around since then at least.
Josh

Stompro, thanks for your comment. I have found out following dependency - the memory will overflow as fast, as speed limit rule will be applied to traffic. I have tested following – I created rule for 1 PC only with speed limit 1 Mbit. And I started downloading large file – memory started overflow rapidly. So It took 6 hours to overflow memory (wired memory increased from 10M up to 96 M). And If I will surf the WEB only on the same PC, it take more than 10 hours (sometimes more than 1 day) PFsense memory to overflow. Do you have any ideas?

Here is info from 2.0-ALPHA-ALPHA
built on Wed Dec 2 08:57:36 EST 2009
FreeBSD 8.0-RELEASE
(Server was after reboot and no traffic passed through it )

$ ipfw pipe show
00001: 10.000 Mbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail
burst: 0 Byte
00003: 1.000 Mbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail
burst: 0 Byte
00004: 1.000 Mbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail
burst: 0 Byte

And that is from another server where wired memory raised from 16M up to 45M (3 rules with speed upload/download limit applied there)

$ ipfw pipe show
00001: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
46 ip 192.168.1.35/0 0.0.0.0/0 342311 32657861 0 0 0
00002: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
51 ip 0.0.0.0/0 192.168.1.35/0 461574 492522695 0 0 0
00003: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
44 ip 192.168.1.34/0 0.0.0.0/0 93299 10170752 0 0 13
00004: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
50 ip 0.0.0.0/0 192.168.1.34/0 52530 11098282 0 0 25
00005: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
34 ip 192.168.1.37/0 0.0.0.0/0 463376 44730982 0 0 0
00006: 1.000 Mbit/s 0 ms 50 sl. 1 queues (64 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
53 ip 0.0.0.0/0 192.168.1.37/0 252901 33586381 0 0 0
00007: 1.000 Mbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail
00008: 1.000 Mbit/s 0 ms 50 sl. 0 queues (64 buckets) droptail

Waiting for ideas…

stompro

No solution but here is some more info.

I have one firewall that is not showing the problem.
2.0-ALPHA-ALPHA
built on Sat Jul 25 23:59:13 EDT 2009
FreeBSD 7.2-RELEASE-p2
CPU Type Pentium III/Pentium III Xeon/Celeron (Old Dell XPS system)
Platform pfSense
Interfaces: DC0, XL0, XL1

Two that are showing the problem.

One:
2.0-ALPHA-ALPHA
built on Sat Aug 22 01:39:53 UTC 2009
FreeBSD 7.2-RELEASE-p3
CPU Type VIA Nehemiah (1Ghz Abitech FX5620)
Platform nanobsd
Interfaces: RL0… (Realtech)

Two:
2.0-ALPHA-ALPHA
built on Sat Aug 22 01:39:53 UTC 2009
FreeBSD 7.2-RELEASE-p3
Platform nanobsd
CPU Type Geode(TM) Integrated Processor by AMD PCS (Alix)
Interfaces: VR0...

So the two that are showing the problem are running an Aug 22 build. The one that is not is running an older build. So it could be a bug introduced after July 25th.

The two that are showing the problem have different NICs, so it is probably not driver dependent.

Two different X86 cpus, so that probably isn't related.

Both are running the same dummynet pipes.

I will try a bleeding edge snapshot when I have a chance.
Thanks
Josh

maxk

@stompro:

No solution but here is some more info.

I have one firewall that is not showing the problem.
2.0-ALPHA-ALPHA
built on Sat Jul 25 23:59:13 EDT 2009
FreeBSD 7.2-RELEASE-p2
CPU Type Pentium III/Pentium III Xeon/Celeron (Old Dell XPS system)
Platform pfSense
Interfaces: DC0, XL0, XL1

Thanks for comment. I have checked site http://snapshots.pfsense.org/ - there's no versions with FreeBSD 7 there. In other words - folder http://snapshots.pfsense.org/FreeBSD_7/ is empty. Stompro, do you have distributive of 2.0-ALPHA-ALPHA built on Sat Jul 25 23:59:13 EDT 2009? Can you upload it anywhere (Rapidshare for example)? I will check it too… I have checked folders http://snapshots.pfsense.org/FreeBSD_8_0/ and http://snapshots.pfsense.org/FreeBSD_RELENG_8_0/pfSense_HEAD/livecd_installer/ folders - there's no old versions there...Can anybody help to find old PFsense version? Thanks.

cmb

We will not provide any old versions, 7.x versions of 2.0 are long gone. You'll just have to wait until things get fixed in FreeBSD 8.

maxk

@cmb:

We will not provide any old versions, 7.x versions of 2.0 are long gone. You'll just have to wait until things get fixed in FreeBSD 8.

May be - you have old version of PFSense on FreeBSD 8 (2.0-ALPHA-ALPHA built on Sat Jul 25 23:59:13 EDT 2009 FreeBSD 8)?
Where I can download it? I only want to help PFSense project to solve problem more quickly. So if you will let to download some old version - i can help to find where exactly bug appeared.

stompro

@cmb:

We will not provide any old versions, 7.x versions of 2.0 are long gone. You'll just have to wait until things get fixed in FreeBSD 8.

CMB,
Can you provide some guidance on how Maxk and I should help to get this particular bug figured out. Should we try a vanilla Freebsd 8 install to see if the problem is there, and then file a bug with the FreeBSD project if it is? Should we open a bug report on redmine? Is this a known issue that is being worked on and we should just chill for a few weeks?
Thanks
Josh

maxk

@stompro:

@cmb:

We will not provide any old versions, 7.x versions of 2.0 are long gone. You'll just have to wait until things get fixed in FreeBSD 8.

CMB,
Can you provide some guidance on how Maxk and I should help to get this particular bug figured out. Should we try a vanilla Freebsd 8 install to see if the problem is there, and then file a bug with the FreeBSD project if it is? Should we open a bug report on redmine? Is this a known issue that is being worked on and we should just chill for a few weeks?
Thanks
Josh

Can anybody from administrators answer my and Stompro's question? We want to help to the whole project but need your comments to do it.

sullrich

Please try the latest snapshot.

stompro

Scott, I tested the 12/14/09 snapshot and it appears to have fixed the problem on my test system. With 512mb of ram, the wired memory initially climbs to 39M and then stays there. I don't know what this fix was so I don't know who to thank, but thank you for letting us know that we should test it again.

Hey maxk, did it work for you also?
Josh

maxk

@stompro:

Scott, I tested the 12/14/09 snapshot and it appears to have fixed the problem on my test system. With 512mb of ram, the wired memory initially climbs to 39M and then stays there. I don't know what this fix was so I don't know who to thank, but thank you for letting us know that we should test it again.

Hey maxk, did it work for you also?
Josh

I have tested version pfSense-2.0-ALPHA-20091221-2130.iso and still can see the problem.

PFSense_test1Q.gif_thumb

maxk

Additionally here is System activity. You can see - wired memory too large. Stompro, what exactly snapshot have you tested from 12/14/09? Have you checked RRD graphs in snapshot from 12/14/09? Are they working? In pfSense-2.0-ALPHA-20091221-2130 RRD graphs show nothing. Have you tested BandwidthD in snapshot from 12/14/09? Is it working? In pfSense-2.0-ALPHA-20091221-2130 BandwidthD installs fine but can not start as a service. Have you seen such problem? Thanks.

PFSense_test2Q.gif_thumb

stompro

I think I spoke too soon, I tried a new snapshot on Monday, and as of today from light testing the wired memory usage has grown to 57M.

2.0-BETA1 Built on Sun Mar 21 02:30:16 EDT 2010
FreeBSD 8.0-Stable
nanobsd

Can anyone point me to where I can find more info about what causes this problem?
Thanks
Josh