Hello, I found my answer: ensure the clocks are synced correctly.  one had ntp turned off, and the wrong timezone set. Just like the last line in the "configuration synchronization problems" section of the 2.1 book. –jason

I looked this up long ago, so I hope my memory serves. Basically, in BSD, the packets get to the kernel and then firewall decides on out to deal with it. So basically you are blocking outgoing connections and not inbound connections. So when you put in a WAN rule, you are putting in an allow out rule to the internal network. You will need to google BSD networking/routing/firewalling to get more details.

Mm not sure if it is similar then, as I was using 2x pfSense. So not CentOS as OS….

OK, so in my test setup at home, I have to use my 192.168.1.127 for its WAN IP (instead of the public 67.x IP's that I will have at my colo when it goes into production) and I have bridged the WAN/LAN interfaces. Gateway of 192.168.1.1 is setup on the WAN interface. Now this should allow me to use 192.168.1.0 ip's within my the network behind the pfsense device. Correct? Now, with that bridge setup, how do I give the 192.168.0.0 subnet/vlan access to the internet?

my upstream provider uses a cisco gateway, when i asked them to show me how they routed the #.#.236.0/24 subnet to me, i was dumbfounded with how simple he made it look… my static ip assigned to wan is #.#.232.12/24 (pfsense box) from his CLi, he typed in a single command telling the cisco gateway to route any requests to 236.0/24 to be routed to 232.12/24, the pfsense box was then configured with virtual ip / proxy arp, and from there i would assign 236.0/24 ip to lan hosts.... so, at this point, i have broken the /24 into smaller subnets (/30,/29,/25) creating new vlan interfaces, and configuring each vlan interface with /29 or /30... disabled dhcp within that vlan giving the host control over ip assignment to me this is a waste of ip's being i loose two ip's to subnet id/broadcast just to provide a /29 or /30 to a single host.... is pfsense capable of doing what my upstream provider did? per say, route requests to #.#.236.10-16/24 to their wan ip like he did thru cli?

Pushing 1Gbit/s through FW+NAT isn't very hard.  A modern Celeron or i3 will have no issues. What exactly are the "network issues" that you're experiencing?

I found a workaround which is not ideal but allows me to move on. I noticed that in fact i couldn't ping the LAN VIP if i was connected in WiFi to the active CARP appliance. So basically instead of having two active hotspots, I just had to make sure the hotspot was disabled on the active CARP appliance. To do so here is what I did: 1 - Modify /etc/devd.conf to point to custom rc.carp scripts ... # CARP notify hooks. This will call carpup/carpdown with the # interface (carp0, carp1) as the first parameter. notify 100 {     match "system"          "IFNET";     match "type"            "LINK_UP";     match "subsystem"          "[a-zA-Z0-9_]+_vip[0-9]+";     action "/etc/rc.carpmaster.custom $subsystem"; }; notify 100 {     match "system"          "IFNET";     match "type"            "LINK_DOWN";     match "subsystem"          "[a-zA-Z0-9_]+_vip[0-9]+";     action "/etc/rc.carpbackup.custom $subsystem"; }; ... 2 - Create /etc/rc.carpmaster.custom #!/bin/sh /etc/rc.carpmaster $1 ifconfig ath0_wlan0 down /usr/local/sbin/pfSsh.php playback svc restart racoon 3 - Create /etc/rc.carpbackup.custom #!/bin/sh /etc/rc.carpbackup $1 ifconfig ath0_wlan0 up So basically, when an appliance becomes master: It runs the usual carp scripts It disables the hotspot It restarts racoon (to make sure IPSEC connection are restarted) when an appliance becomes master: It runs the usual carp scripts It enables the hotspot Having the same SSID and both appliance not too far from one another makes it almost transparent to end-users (about 30 seconds service interruption in case of a CARP failover).

AFTER your box is up and running (not so well as you write), call the ISP and ask them to clear the cable modem's ARP table.  If it starts working (for a little while), then you are seeing the same thing I'm seeing which only started within the last few weeks. Somehow, the router in the cable modem puts the interface's actual ip in its ARP table with the first VIP mac address.  Then, it puts the VIP's mac address in it's table for the pf router's actual interface card ip. The result is no traffic gets through.  Traffic addressed on the link to the vip has the interface address and so it gets dropped.  Traffic addressed to the VIP on the link has the interface's IP and that gets dropped.  I still don't have an answer.

This turned out to be a simple mistake on my part combined with a UI problem. When creating the SYNC interfaces - I had clicked "Insert my local MAC address" in the MAC address field, thinking it would populate using the NIC's MAC address, not the machine I am logging in from. When I realized it didn't, I blanked the field, assuming it would revert to its own MAC if the field was blank. Not so. My originally entered MAC address remained with the interface even after the field was blanked and saved, resulting in a loop. Changing the MAC addresses such that they do not conflict solved the issue.

1- Provided they are used with a CARP VIP or subnets routed to a CARP VIP, yes 2a- Yes, CARP VIP or IP alias w/CARP VIP as its interface 2b- Yes, provided your routed subnet is routed via your CARP VIP 3- Proxy ARP won't work with failover, it would cause an IP conflict. All it does is listen for ARP requests for the IPs it is given and answer with the firewall's MAC on the appropriate interface. That's really all there is to it.  See here for more info.

This is also related to hiding everything behind the CARP VIP: You don't want to use the CARP VIP to do the gateway pinging, otherwise your CARP BACKUP hosts won't know if they are connected. The pings go out, but with the source address rewritten as the CARP VIP so the ping responses come back to the VIP, not the CARP BACKUP host. So you need to add a NO NAT for these ping packets to make the gateway pinging work properly.

yes, you can submit a feature request (don't assign it or give it a target though).

I've figured it out!  Despite looking like everything is fine after hitting apply, the system acts really strangely until you disable CARP and reenable.

Yes you just have to ensure you use a different VHID for each VIP on the same interface