Thanks, that's exactly what it was. My static mapping wasn't setup properly. I left 192.168.2.10 in the range of the DHCP pool and it was assigned to my IPOD touch.
It's all setup properly now.
Thanks!

There should be some UK companies that make rackmount servers using Supermicro boards.  That would be my first choice since they tend to integrate dual Intel NIC's.

A quick Google search brought these guys up on the first page:
http://www.sentralsystems.com/superintel.html

I have no idea how good they are.  But given the choice between a Dell or a Supermicro with Intel, I'd choose the latter.

This is "normal" DHCP traffic, where Comcast is responding to a request for a lease, which could be any computer connected to the same head end as you.

Cheers.

As I said, no suggestion should go untried, so I dd'ed the pfSense nanoBSD image to my SSD.
And basically it works fine!

However, even if the / filesystem is mounted read-only (RO), it seems to be mounted read-write (RW) every now and then.
I notice a lot of calls to conf_mount_rw() in /usr/local/www

I guess this works nice with a CF card: In general RO, but when needed RW.
But the root filesystem is thus not truly RO.

So in "my" case it does not work as needed…

I might play around a bit and try to mount /etc from a different partion etc...

The only way to do this would be with a proxy of some kind. Squid would work for HTTP transparently, but not HTTPS. If you want to do that, the clients would have to either hardcode the proxy settings or you could setup WPAD and they can use proxy autoconfigure.

Even squid won't get the MAC address, though, just IP, date/time, and URL.

Even if you could write some sort of DPI tool that would log URLs, it would still only work for HTTP.

Another way around this is to give all your clients public IP addresses (which may not be feasible), and then just keep a record of who was assigned which public IP when (PPPoE would help you here, if you forced auth).

Squid shouldn't be too bad performance-wise if you don't really have it caching, just logging.

Just add rules above that firewall rule to block access to the networks you don't want to be accessed.

Just to elaborate on the previous reply because the question didn't make the context plain. TFTP on "local" subnets (routing between source and destination but not NAT) shouldn't be any problem. TFTP through NAT (e.g. to Internet) requires a TFTP proxy as discussed earlier in this topic.

Hi,

well I finally got everything working (regarding the VLANs) and I was also able to determine what went wrong.
I'm now running a dedicated machine for PFS and ESX is on its own.

My first mistake was that I wasn't sure what access or general port was on the switch and my second was that first I didn't set the port that contained the VLANs as trunk. So after I created VLANs on PFS and attached them to the OPT1 interface (interface is used only for vlans) and set the port this cable was connected to on the switch as trunk. I precoded to tag this port on every VLAN I needed on the switch and added access ports to the appropriate VLAN. It started working right away without any restarts or reboots of PFS or the switches.

Now the ESX is a bit of a different story. For the VMs on the ESX I created a virtual switch, each with corresponding VLAN tags and connected it via trunk to the switch. Then I added the VMs to the appropriate virtual switch and changed the IPs on them and everything started to work as it should. I'm still not sure if I could have set the virtual switch to 4095 and setup VLANs on each VM separately, but since it's working it doesn't make much sense to start medaling with it now.  :)

Anyway thank you all for your help and I hope that anyone with similar problems might benefit from this information here. I'm also attaching a diagram of my network topology for reference (sorry but it's not very good, but I think it illustrates the network).

By the way for example if I have setup OPT2 as a second LAN and it is working what happens if I attach a few VLANs to the same interface as OPT2 and then set the port on the switch as trunk. Will OPT2 still work and fall into the default vlan on the switch 1 and all other VLANs to tagged the appropriate VLANs. Would this work? It works the ESX any non tagged traffic falls into vlan1 on the switch. Or is it a better idea to leave only VLANs on the PFS nic without the non VLAN traffic? And when you attach VLANs to a nic in PFS is that nic automatically marked as trunk?

Bye

network_diagram.jpg
network_diagram.jpg_thumb

If you're on nanobsd, you could set the boot slice to the alternate and reboot, and it would be back to normal (but with your config, of course). That's under Diagnostics > NanoBSD

Power loss normally isn't an issue with embedded, as the device is read-only most of the time. Even so, it's rare that a file gets corrupted.

The newest release version of pfSense is 1.2.3-release which is what you should be using in production.

Adding something like iBlocklist is a feature coming in 2.0 which is currently in Beta.  It's not quite ready for production, but its getting very close.

The updated ones may have carried over files from the older version, but should have updated all core files. The update kernels are built separately from the ones that go onto the ISOs.

The firewall logs are also updated by the Dashboard package, you may have installed that in some or all of the boxes.

Key words from your post are:
@rwalker:

I.e. segment A - 10.10.10.x has 3 servers, server 1 is requesting from server 2 & 3 which are in a cluster.

Correct approach would be: to access this cluster Server1 should use 10.10.10.y IP assigned to serv2&3 cluster.

If you still want Public IP then probably you should try NAT reflection.

Hardware: My connection is 5500/720. I run squid, freeswitch, and do lots of traffic shaping, with voip and torrents on the LAN. A 500 MHz Geode is more than enough hardware for this. I could handle 3 times the speed comfortably.

Bandwidth: The more the better, obviously, but when I worked for a wISP a couple years back I observed that we were able to oversell our bandwidth about 30x without too much trouble. In other words, we could sell 1000/300 to 90 clients on a 3000/1000 connection before our pipe really started to max out. Using that math, I would estimate that you could provide 1000/1000 to roughly 60 clients on a 2/2 mpbs pipe before performance would start to suffer at the client's end.

Try another location - before you spend time chasing red herrings you need to be able to narrow down the issues.

As for VPN - try this about setting up site to site with OpenVPN.  Before you do that though, do check that it is happening from more than just your office.

Some computer on your net is not configured with static IP and can not receive IP through DHCP, so it picks up some 169.254.x.x IP and then sends arp requests. pfSense's kernel receiving this arp requests understands that according to its config it is impossible to have this IP on this interface and reacts with the message in log.
Do tcpdump, find out mac of this computer, find computer and fix it.