Compiling pfsnse and some more thoughts

[motionthings]

This is my first post on this forum!
My name is Simon, I live in Fredrikstad, Norway and I'm 40 years old.

rant (this rant is all my fault for not RTFM. but I'm doing it anyway)

I'm not going to link to a lot of other posts and documentation of my claims in this post.
You just have to trust me when I say I have done my fair bit of research on the subject I'm writing about in this one.

I am a first time pfsense and freebsd user (I know some linux).

I started trying to install pfsense on a new box I have, a few days ago.
But it turns out my mainboard has an Atheros AR8161 (ath1) which is not supported by your current build (freebsd 10?).
But is supported in freebsd 11.

So I thought I would just build the kernel myself.

But the loops I have to jump through to get the tools I need to compile this thing from source is bordering on ridiculous. (This is just my personal opinion. I'm a novice)
https://forum.pfsense.org/index.php?topic=83785.0
https://forum.pfsense.org/index.php?topic=76132.0

Loading the driver as a module with «if_alc_load="YES"» does not work (for me)

PS. I just ordered/bought a couple of compatible gigabit intel cards. All is good :-)

/rant

Now that my rant is over I would like some advice from you on my "home" network.
I have a "live" diagram here: https://cacoo.com/diagrams/lGWugiSw4Ud27JYA
A static one here: https://www.dropbox.com/s/nh8pvmpnsm3swhs/Network%20Diagram.png?dl=0

I made a post on reddit here: https://www.reddit.com/r/AskNetsec/comments/37hes4/homenetwork/
I have made some changes in my design since then^^ The links I post here are current.

Would love some feedback from someone here too?

[divsys]

So have you managed to get yourself an operational pfSense box yet?

There's any number of discussions on the forums about the hoops required to compile your own copy and the various reasons for those hoops.

I won't rehash it other than to say if you've got some reasonable hardware (and I think you do) that's being held up because of a NIC driver, get another NIC with a builtin driver.

PS. I just ordered/bought a couple of compatible gigabit intel cards. All is good :-)

See? We think alike  ;)

As far as the rest of your setup goes, my one suggestion would be to separate out the IP cams into their own subnet.
I've done more than a few small-medium (up to 50 camera) security installs and I would never setup the cameras to co-exist on the customer's data LAN.
Too much possibility of either cameras or PC's stepping on one another.

Put another NIC in the pfSense box (or go with VLANs) and create a new subnet (192.168.30.0/24 ?).
Move the cameras and the NVR onto that subnet and you can create simple rules to allow/deny access as needed.
The main recording functions will be handled as normal by the NVR and you only pass video traffic through pfSense when you need to view or playback.

Welcome to pfSense!

[KOM]

Jesus, most SMBs I've seen have less-complicated networks!  I must say that I haven't seen too many people running their own SNMP/Syslog servers at home.  5 APs?  How big is your house?

[firewalluser]

Norwegians are all paper millionaires according to the latest financial data due to their Govt's fiscal responsibilities.

[motionthings]

Sorry again for the rant. And sorry for posting to a link on reddit instead of writing my question here.
I'll do that now.

I built my (got the parts) pfsense box on Friday, and I was looking forward to "nerding out", setting it up this weekend.
It is of course my fault for not checking the HCL first. (I actually thought the mainboard I ordered had two intel NIC's)

Sorry again. My bad.

But, a couple of good Intel cards are in the mail, and I have next week off :-)

Back to my "home" network.
https://www.dropbox.com/s/nh8pvmpnsm3swhs/Network%20Diagram.png?dl=0

It's just something I'm putting together with some leftover hardware I have lying around.

I'm doing it primarily to (re)learn SNMP, Syslog, Snort and pfSense. It's an experiment.
I took some "Linux CBT" classes back in 2005. But trust me when I say, I have forgotten everything I learnt there.

FreeBSD is also "new" to me.

Hardware:

• Jupiter: pfSense-box: Intel-i3, 8GB-RAM, 2x Intel Gigabit NIC (No room for more cards! I'll have to get the Atheros card working at some point).

• Europa: Switch, Zyxel GS-1900-8HP

• Ganymede: Snort NDIS, Some Athlon 64, 4GB RAM, 2x Intel Gigabit NIC

• Io: SNMP and Syslog, Some Athlon 64, 4GB RAM, 2x Intel Gigabit NIC (I'm running this as a VM right now, for testing)

• Callisto: Router, Wifi AP, Asus RTN66U

• Amalthea: Seagate Central NAS

• Metis, Adrastea, Thebe: Wifi AP, Ubiquiti Nanostation M2

• Leda: Router, Wifi AP, Linksys TL-WR941N

• VideoRig: Some Athlon 64, 8GB-RAM, Radeon GPU

• IP- cameras: Some cheap ass china 640x480 cameras. Approx 8Mbps pr. camera.

Everything that can is running dd-wrt.

Unused hardware:

As of right now I'm still in the planning stage of this network.
The pfSense box is sort of the core of my network, so I need that to be running smooth first.

Someone said that bridging my pfSense box directly to my ISP was a bad thing?
I "only" have a 150/150 Mbps connection. Shouldn't my Intel-i3 be able to handle this?

I have a couple of Linksys/Cisco RVL200's lying around. And I was thinking of putting that in front of my pfSense box.
Yes, I know it's only a 100 Mbps box.
Do I need to do that?

@divsys.
There is no such thing as wireless security or surveillance. I know this.
The only important cameras I have are my two PoE-wired outside cameras. (Sricam ap004).
I'm thinking my NVR can handle this.

Thank you for your feedback, and welcome :-)

@KOM
LOL!
I'm doing this to learn :-)
My three Nanostation M2's are directional antennas. I live in a "geographically challenged" place, and I'm trying to cover a big area.
I have made a somewhat 180° coverage area with my Nanostations.
Have a map: https://www.google.com/maps/d/viewer?mid=zxVnNek7hjWg.kPRp7cjdww3Q&usp=sharing
I only have one wireless AP inside my house (RT-N66U)

@firewalluser
That's only on paper :-(

I have a couple of thoughts/plans for my network.

First of all, it's cool that I have my own home wifi covering a large area :-)

I was thinking about giving everyone over 65 years old in my neighborhood free internet.
But then I got the thought that this would probably have to include their grandchildren too.
And then I could probably end up with some legal problems.
(I could of course do some MAC-filtering or something (Radius?). I'm still thinking about this)

A third thought I have is to offer remote support in my coverage area. I'll set up some "open" wifi SSID's (Radius) named something like "Computer problems, connect here", redirecting you to a webpage that contains information about my services.

Well, this turned out to be a long post.
But I'm not currently configuring my pfSense box :-p
Off to read some more documentation.

Have a wonderful Sunday…

[divsys]

Sounds like a fun project!

In the words of someone else on the Forum: Think big, but start small.

Try this one step at a time.

I think your on the right track, but definitely consider subdividing your design into different subnets to give you smaller "chunks" to work with.

If you're seriously considering handing out WiFi in the neighborhood (a laudable goal) then that's definitely it's own subnet.
Cameras are another one, and your own internal LAN is a third.
Personally I like to put my home Wifi on its own subnet as well, but that's me.

So I would plan on at least three subnets plus your WAN connection.
If you haven't got four NICs, your Zyxel switch supports VLANS which makes it easy to separate out your networks.

The more I look at your setup, the more a VLAN implementation makes sense.

There is no such thing as wireless security or surveillance. I know this.
The only important cameras I have are my two PoE-wired outside cameras. (Sricam ap004).
I'm thinking my NVR can handle this.

Yah wired is good in general, but especially for cameras.
The security issue is one thing, but the reliability of the network is another and my earlier comment(s) about separating the networks still stands.

One side thought, you might want to consider looking at DigitalWatchdog's SpectrumVMS software for your setup.
The software is free and the NVR runs on both Windows and Linux boxes.
The only cost is a one time license per camera (~$85/camera?) to allow recording.
You can load the software and get a free 30 day trial license to test it out with your cameras.
Given your NVR hardware I think this could be a very good fit especially as Ubuntu on that box will probably give you some reasonable performance.

Good luck, let us know how it goes….

[motionthings]

VLANS it is :-)

The reason I did not go for VLANS is that I also have an IPTV service coming into my house (multicast). And as long as you are on the same LAN you also get free tv on your phone/tablet.
This is of course illegal! So I'm not doing that ;-)

And thanks for the NVR software link.

I have been looking for an Open Source NVR that runs on linux for a long time.
But so far nothing seems to match iSpy.
https://alternativeto.net/software/ispy/

I actually limit some of my cameras to record in 10 fps, and that seems to be fine.

For the last 1,5 years I have been running iSpy on an old pentium Core 2 laptop. And this seems to be doing fine.
As of now it contains over 90000 video files :-) Attachment 1.

Here are some examples (All wireless cameras):
http://motionthings.no/upload/3_2014-07-22_20-17-20.mp4 (My grandfather)
http://motionthings.no/upload/4_2014-08-27_10-22-38.mp4 (Me, getting a cup of coffee*)
http://motionthings.no/upload/4_2014-09-24_17-45-15.mp4 (Me, in my "Batcave")
http://motionthings.no/upload/5_2014-08-01_09-17-01.mp4 (Always post a video of cats! This is after all the internet)

The audio is pretty good!

I'll have a serious look at what you're recommending.

edit
*This camera is for catching cats on my kitchen counter.
At first I played a foghorn sound every time there were cats on the counter, but that almost gave my grandfather a heart attack.
Luckily his hearing is not what it used to be, so I replaced the sound with a 15KHz sound that the cats, but not my grandfather can hear :-) I can promise you that they run out of the kitchen pretty fast when I play the sound.

And since I now am using a sound that my grandfather cannot hear I can automate this process.
"For every motion alarm in kitchen, play sound 15Khz.mp3"
/edit

I am doing a "buildlog" of this entire project. Hoping that it can land me some work within networking.
Using it as an addition to my CV :-)

In other words. I would love to talk alot about every step on my "journey".

This thread will be referenced in the documentation. Along with all my other questions elsewhere.

PS. I'm really looking forward to getting my NIC's tomorrow :-)

[motionthings]

Well FML!

Got my compatible intel NIC cards delivered two days ago (Yay).

Guess what I woke up too. No internet connection!
Attachment 1 and 2.

Turns out there was a problem with my fiber. All good now!
pfSense is running!
Attachment 3

Starting a new thread with a buildlog if anyone wants it?

edit
Sorry for the ginormous attachments again.
/edit

edit2
In my third attachment (see the red arrow). pfSense asked for two DNS servers. I said, use yourself (192.168.3.1), and google (8.8.8.8) as DNS.
The two that are already there I got from my ISP via DHCP.

Isn't my first entry 192.168.3.1 as DNS unnecessary/redundant? It already uses loopback interface for DNS?
/edit2


[motionthings]

Hmm. I only pay for a 100/100 line.
Took a speedtest today:

Lets hope they don't find out :-)

[Nullity]

@motionthings:

Hmm. I only pay for a 100/100 line.
Took a speedtest today:

Lets hope they don't find out :-)

Whoah.  :o

I felt kinda lucky that I was paying for 6 and getting 7.2, lol. :\

:)

[motionthings]

This is after I got my bridged connection for the first time :-)

I was thinking that this was some sort of cached result. I know my ISP is using alot of cache servers.
But I FTP'd into one of my webhotels (in Norway). And I got the same results :-)

Shhh. Don't tell anyone! Ever!!

I'm afraid that this will balance out over time, and that I will end up with a 100/100 line in a few days.

They probably opened up everything when they were diagnosing my connection.
They replaced a 3-inch piece of fiber that was "spliced" badly.

Hoping it never changes, but I'm not optimistic…

[stephenw10]

You can probably get that Atheros NIC going with this kernel module: https://forum.pfsense.org/index.php?topic=78932.msg434620#msg434620
That code is in the alc driver though so you might also try the alc module from FreeBSD 10 stable.

Steve

[motionthings]

Thanks Steve. I have tried that a couple of times, but always ended up with a machine that would not boot.

Did it again yesterday. Without making a backup first  :'( Overconfidence is a bitch!

My steps for making a non bootable box ;)

• Copy 'if_alc.ko' from a FreeBSD 11 ISO to /boot/kernel/ (*)

• chmod -x 'if_alc.ko'

• edit 'loader.conf' to include the line 'if_alc_load="YES"' (**)

• I tried copying the same file from a running FreeBSD 11 install too, with the AR8161 NIC working.
  ** Don't use the webinterface editor! This put some "artifacts" in my 'loader.conf' file. Fixed it with 'ee'. I should probably report this as a bug.

Here is what I end up with after these steps (Video of my box "booting" in slow-motion):
https://www.dropbox.com/s/ncx880gyj4d17m5/2015-06-16%2000.05.34.mp4?dl=0

Last time I just reformatted the box.
But now I have actually done some meaningful configuration, that I would hate to loose.

PICNIC

OK

• Boot single user mode. Not Working

• Boot safe mode. Not Working

• No options are working! (load/unload modules, list modules. Nothing is working)

OK

• Boot from CD (pfSense)

• Mount harddrive to /mnt/

• edit 'loader.conf' (remove 'if_alc_load"YES"')

• remove 'if_alc.ko' from /boot/kernel/

Reboot.

Results… Same as the video above^^ A "slow-motion" boot.

So, what is happening here?
What am I doing wrong, or forgetting?

I'm now running a live pfSense CD :)

I have read this: https://forum.pfsense.org/index.php?topic=88511.msg488963#msg488963
And many other posts about the AR8161

How can I get my system to boot again? I don't care about the Atheros NIC at the moment.

Simon

PS. "He who laughs last, probably has made a backup!"

edit
Lesson learned:
Never think; "I'll setup backup once configuration is done"
Instead think; "Setup backup before configuration"
/edit

[stephenw10]

Hmm, I wouldn't expect a FreeBSD 11 module to load. Perhaps when those other users tried it was still close enough to 10 to work.
Have you tried the alx module I linked to? That was compiled against FreeBSD 10, others have reported success with that.
Try a module from a FreeBSD 10 recent snapshot that should have that code in it: http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/10.1/

Steve

[motionthings]

Thanks again Steve.

But I don't really need it at the moment.
My original network plan was to use the third NIC (Atheros) on my pfSense box to send traffic to Snort.
https://cacoo.com/diagrams/y2rMw37kzlzcHzZy

Now that I have found out that Snort runs just fine on my pfSense box, I don't need the third network card anymore :)

And given my previous experience (ending up with a non bootable box) I don't want to experiment with this on a "live" system.

I'll just wait till pfSense gets updated to FreeBSD 11.
Maybe then I will do something fun with it.

Simon

[jimbrown]

@motionthings:

VLANS it is :-)

The reason I did not go for VLANS is that I also have an IPTV service coming into my house (multicast). And as long as you are on the same LAN you also get free tv on your phone/tablet.
This is of course illegal! So I'm not doing that ;-)

And thanks for the NVR software link.

I have been looking for an Open Source NVR that runs on linux for a long time.
But so far nothing seems to match iSpy.
https://alternativeto.net/software/ispy/

I actually limit some of my cameras to record in 10 fps, and that seems to be fine.

For the last 1,5 years I have been running iSpy on an old pentium Core 2 laptop. And this seems to be doing fine.
As of now it contains over 90000 video files :-) Attachment 1.

Here are some examples (All wireless cameras):
http://motionthings.no/upload/3_2014-07-22_20-17-20.mp4 (My grandfather)
http://motionthings.no/upload/4_2014-08-27_10-22-38.mp4 (Me, getting a cup of coffee*)
http://motionthings.no/upload/4_2014-09-24_17-45-15.mp4 (Me, in my "Batcave")
http://motionthings.no/upload/5_2014-08-01_09-17-01.mp4 (Always post a video of cats! This is after all the internet)

The audio is pretty good!

I'll have a serious look at what you're recommending.

edit
*This camera is for catching cats on my kitchen counter.
At first I played a foghorn sound every time there were cats on the counter, but that almost gave my grandfather a heart attack.
Luckily his hearing is not what it used to be, so I replaced the sound with a 15KHz sound that the cats, but not my grandfather can hear :-) I can promise you that they run out of the kitchen pretty fast when I play the sound.

And since I now am using a sound that my grandfather cannot hear I can automate this process.
"For every motion alarm in kitchen, play sound 15Khz.mp3"
/edit

I am doing a "buildlog" of this entire project. Hoping that it can land me some work within networking.
Using it as an addition to my CV :-)

In other words. I would love to talk alot about every step on my "journey".

This thread will be referenced in the documentation. Along with all my other questions elsewhere.

PS. I'm really looking forward to getting my NIC's tomorrow :-)

Thanks for sharing this. openipcam is another nice little alternative which provides open source web cam.

[wricaurte]

Hi,

Nice project, I Have something like that at home but smaller, only 5 cameras (cheap chinese foscam), asterisk voip, video+audio streaming (Plex + SqueezeBox). Some Apple TV, Some Kodi boxes, a Supermicro 1U Rangeley Atom server with some virtual machines and a Qnap 2 bay NAS.

For NVR I use Blueiris (http://blueirissoftware.com/). It is only 59.95 and you can add as many cameras as your hardware can handle. The nice thing is that if you want you can buy the apps for your smartphone (iOS and Android), you can set push notifications, email notifications and more. You can access the cameras and recordings from internet publishing the Blueiris Web Portal.

I hope this helps you to evaluate options.

Regards.

[motionthings]

Thank to everyone taking the time to read and respond to my overly long posts.

As of now everything is working great!
But I always have new questions….

"The Dude" is all of a sudden picking up a node with gigabits of traffic on a node ending with *.255 (see attachment).

Is this something internal to pfSense? DNS (ubound)?

My actual pfSense box is 192.168.3.1.

Any and all suggestions appreciated :)

Simon