Path for driver patches into pfSense

stephenw10

Hi All,
I seem to be suffering from some confusion.
What is the path by which new code in FreeBSD enters pfSense?
I realise that since pfSense is based on FreeBSD 8.1 new code from 8.2 or 9 would only be included if it was specifically ported. However what about bug fixes from 8 stable?
Looking through the pfSense source code (https://github.com/bsdperimeter/pfsense) it's obvious that not all of the FreeBSD code is there. Does that mean it gets pulled from the FreeBSD repo at build time?

Specifically I'm interested in a bug fix that was applied to the msk(4) driver on Jun 24.
The new code was merged with both 8 stable and 7 stable but does that mean it will find it's way into pfSense automatically? ???

Steve

stephenw10

None of the head honchos with a yes or no? :-\

Steve

Gloom

There is no way to run something like freebsd-update to follow build revisions as there is with a full install of FreeBSD. PFSense does not do a compile at install time and all the packages are prebuilt.

I believe that when it reaches release status it is frozen and the incremental releases contain any updates the team feel are needed. If all the FreeBSD code was pulled down at build it would effectively be a FreeBSD box which sort of defeats the point.
You can always take the driver from a compatible FreeBSD build and replace the one on PFSense.

stephenw10

@Gloom:

PFSense does not do a compile at install time and all the packages are prebuilt.

Right but presumably it is compiled at build time. I.e. the snapshot server is compiling the pfSense source each day to generate to current build. Since the github repo doesn't appear to have all the code necessary to do that, just the additional pfSense stuff and patches, some of the code must be pulled from FreeBSD. No?

Now it may be that code is pulled from a source that is frozen in which case no updates would get through.

@Gloom:

You can always take the driver from a compatible FreeBSD build and replace the one on PFSense.

Good point. It's probably worth trying that before petitioning to have it included in pfSense.
Without the source code it can be a bit tricky knowing which driver version you're actually using.

Steve

jimp

You seem to be missing the tools repo.

https://github.com/bsdperimeter/pfsense-tools

http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso

stephenw10

I did look through there but didn't find answers.
Why is there no search facility on github? >:(
Is the source code for the msk driver used in the current pfSense there?

I take it that the answer to my original question is; no, the driver patches will not automatically get into pfSense?

It seems as though I'm missing some large piece of the puzzle here. :-\

Edit:
Reading through:
http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso (interesting stuff)
I see that:

Invoke the Apply patches option.
This will fetch the version of FreeBSD required and apply our patches

jimp

It's all in the tools repo, and the source from FreeBSD.

That said, if you can get an updated if_msk.ko compiled on an 8.1 system from an updated driver source, you can likely use that via loader.conf.local to override the in-kernel driver.

We've done this before for customers with quirky em/igb cards before.

stephenw10

Hmmm, after a few painful hours of reading my eyes are hurting. ::)

Perhaps someone would be kind enough to outline what steps I might take to generate a new kernel module.

It's easy enough to generate a kernel module:

cd /usr/src/sys/modules/msk
make

The result of that is, of course, the same as the built-in kernel driver when building using 8.1.

So I update the source code and make a newer version. The new code has been backported to 8.2, will it run on 8.1?
How can I update the source code? Can I update just the msk code? :-
Can I generate a diff and apply it?

Steve

Edit: That didn't work, can't compile 8.2 source code on 8.1. :(

stephenw10

Hmm, well this post looks promising. However there seem to be a few typos in there which are killing me.
It didn't work for me as is but with a few tweaks it compiled. However I think I've probably ended up with something compiled for 8.2. We'll see…

Steve

Edit: Yep, kernel mismatch.

stephenw10

Following on from here.

Attempting to make the driver using the 8.1 /usr/src/sys as SYSDIR gives:


admin@.box/usr/msk# make
Warning: Object directory not changed from original /usr/msk
@ -> /usr/src/sys
cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc   -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common  -mno-align-long-strings -mpreferred-stack-boundary=2  -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -fformat-extensions -c if_msk.c
if_msk.c: In function 'msk_miibus_statchg':
if_msk.c:537: error: 'IFM_ETH_RXPAUSE' undeclared (first use in this function)
if_msk.c:537: error: (Each undeclared identifier is reported only once
if_msk.c:537: error: for each function it appears in.)
if_msk.c:540: error: 'IFM_ETH_TXPAUSE' undeclared (first use in this function)
cc1: warnings being treated as errors
if_msk.c: In function 'msk_attach':
if_msk.c:1663: warning: implicit declaration of function 'mii_attach'
if_msk.c:1663: warning: nested extern declaration of 'mii_attach'
if_msk.c:1664: error: 'BMSR_DEFCAPMASK' undeclared (first use in this function)
if_msk.c:1664: error: 'MII_OFFSET_ANY' undeclared (first use in this function)
if_msk.c: In function 'mskc_attach':
if_msk.c:1912: error: 'MIIF_FORCEPAUSE' undeclared (first use in this function)
if_msk.c:1916: error: 'MIIF_MACPRIV0' undeclared (first use in this function)
*** Error code 1

Stop in /usr/msk.

Clearly something isn't included which should be.
The Makefile for that is:


# $FreeBSD: stable/8/sys/modules/msk/Makefile 165140 2006-12-13 02:37:48Z yongari $

.PATH: ${.CURDIR}/../../dev/msk

KMOD=   if_msk
SRCS=   if_msk.c device_if.h bus_if.h pci_if.h
SRCS+=  miibus_if.h miidevs.h rdcphy.c

.include

I've added the extra files but as sources rather than objects to be built. :-\

Steve

Edit: I should read some of this stuff first! rdcphy.c is nothing to do with the msk driver. ::)

wallabybob

What were you most recently trying to compile: msk from 8.2? msk from head? What is the revision information in the file? Have you applied your own patches? How did you get your file(s)?

And what are you building on? Stock 8.1?

I don't immediately have access to a FreeBSD system (maybe tomorrow) to investigate the undefined symbols but from the look of them I would expect them to have been standard definitions for some time. Maybe the definition location moved in the 8.1 to 8.2 transition.

stephenw10

Thanks for reading. :)
That is the result of trying to compile msk from stable/8/
The file info is:

__FBSDID("$FreeBSD: stable/8/sys/dev/msk/if_msk.c 223400 2011-06-22 01:54:49Z yongari $");

I haven't applied any patches. The source code was checked out from /stable/8/ using SVN two days ago.

I am using stock FreeBSD 8.1_rel to build on.

If the file locations had changed that might explain it.

I have attempted to replicate the process used in the linked post however I now realise that a number of those steps are specific to that driver.
I have created a new folder holding the msk code as well as some other files to build in. Mostly I assume that was done so as not to corrupt the source tree of either the 8.1 or stable/8/ code.

Setting the environment variable SYSDIR to /usr/stable/8/src/sys (where I checked out the code) allows the module to build but results in a module for 8.2 (or whatever kernel stable is at).
Setting it back to /usr/src/sys results in the error above.

If you would rather I delete everything and start again to avoid any confusion I'm more than happy to do that. ;)

Steve

Edit: As a test I attempted to replicate the procedure outlined in the link FreeNAS post exactly (creating a vte module) but resulted in a similar list of errors. I believe it's likely I'm modifying the Makefile incorrectly. Or things have changed dramatically since January.

wallabybob

IFM_ETH_RXPAUSE, IFM_ETH_TXPAUSE, BMSR_DEFCAPMASK, MII_OFFSET_ANY and MII_FORCEPAUSE are not defined in the FreeBSD 8.1 source tree. (I didn't check the other undefined symbols reported.) IFM_ETH_RXPAUSE and IFM_ETH_TX_PAUSE are now defined in sys/net/if_media.h

From the comments in the revision history it appear some driver specific code for handling PHYs was made more generic: kernel interfaces changed in the 8.1 to 8.2 transition.

I expect you will have to hand edit the change you want back to the 8.1 release code. In the CVS web interface revision 1.64.2.20.2.1 was the revision for 8.1 release, 1.64.2.27 is the current head of the 8.x branch and seems to have invisible changes (e.g. spaces to tabs) from 1.64.2.26. Revision 1.64.2.23 looks to have the changes for the generic PHY support. Its possible subsequent changes rely on new (to 8.2) kernel interfaces.

I don't have time now to investigate further.

stephenw10

Thankyou very much for looking at that. :)
Time to get stuck into the code then!
It's interesting then that I couldn't get the VTE driver to compile, replicating the steps on the FreeNAS forum, since that post was in Jan 2011 and the 8.2 release was in Dec 2010. :-
In that post it was also necessary to include the relevant PHY code. That would certainly tie in with a mii re-write.
I'm still unsure about the Makefile. I seem to be unable to find a good resource describing the function of the different lines. Clearly if more code has to be included/compiled/built then the Makefile needs to be modified in order to automate that process.
As an example given the Makefile:


# $FreeBSD: stable/8/sys/modules/vte/Makefile 216829 2010-12-31 00:21:41Z yongari $

.PATH: ${.CURDIR}/../../dev/vte

KMOD=   if_vte
SRCS=   if_vte.c device_if.h bus_if.h pci_if.h miibus_if.h

.include

And the instruction:

edit Makefile to add 'miidevs.h rdcphy.c' to the list of files to build

Where should those lines be added?
Thanks again.

Steve

Metu69salemi

Not sure at all, but could it be in target?

wallabybob

@stephenw10:

And the instruction:

edit Makefile to add 'miidevs.h rdcphy.c' to the list of files to build

Where should those lines be added?

They shouldn't. You are extrapolating too much from the template you are using. The template is for building a module from sources of a completely new driver that didn't exist in the FreeBSD source tree. You are wanting to build an driver (msk) that already exists in the FreeBSD src tree.

rdcphy.c now exists in the FreeBSD source tree in src/sys/dev/mii. A quick examination shows it claims to be support code for the RDC R6040 10/100 phy. As best I know that phy has nothing to do with msk so don't complicate your challenge by adding stuff you don't need.

stephenw10

Yes, I realise that rdcphy.c is nothing to do with the msk driver (which appears to use e1000phy) I was just using that as an example.
I expected to be able to compile the if_vte.ko module as per those instructions since the code change between now and Jan doesn't seem so dramatic. Since I cannot compile that then either I doing it wrong (very likely) or the code has changed much more than I realise. Or it never worked in the first place and that poster was making the whole thing up! The only part of those instructions that confused me was modifying the Makefile.

It seems likely that some other files will need building as well and adding them to the Makefile seems like the right way to do it.

I appreciate all your help and patience with this. Clearly this is beyond my usual level of tinkering!

Steve

Edit: Actually there is significant code change in if_vte since Jan.

wallabybob

@stephenw10:

I expected to be able to compile the if_vte.ko module as per those instructions since the code change between now and Jan doesn't seem so dramatic.

A "simple" code change can hide considerable complexity. For example, a change in the name of a function called at first seems a simple change. However the "new name" function may exist in the "new" kernel but not the "old". The "new" function may call quite a number of other "new" functions. What appears at first to be a simple backport problem has now become a much more challenging backport problem.

Your compile of vte driver might have proceeded rather further if you had checked out its sources as of the date of the post rather than "now".

stephenw10

Yes, in fact the changes were signoficant.

Reading through the threads for the various bug reports that went into the driver patch for msk it seems that in fact there are samples of code in many of them and a lot dating from around the time of 8.1 rel. I think, as you said, I need to find exactly which code change is relavent to my NIC and patch the 8.1 source manually.

There is also a possibility of updating the 'firmware' in the nic to remove the problem condition in the first place. I'm looking into it.

Thanks again.

Steve