Ok what am I missing?

Mad Professor

I'm trying to setup carp failover in virtual box guests. I got everything somewhat setup, trying my best in a double nat environment,

pfmaster lan 10.10.10.1
pfmaster wan 192.168.0.62

pfslave lan 10.10.10.2
pfslave wan 192.168.0.63

virtual ip for lan 10.10.10.5
virtual ip for wan 192.168.0.64

I'm having a hard time comprehending this guide from
https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_%28CARP%29#pfSense_Hardware_Redundancy_with_CARP

Setting up advanced outbound NAT

Enable advanced outbound NAT in Firewall -> NAT -> Outbound -> Enable advanced outbound NAT. Click save.

Edit the automatically added rule for LAN. Pick a shared CARP virtual IP address as the Translation IP address. Give the item a description and click Save.

Setting DHCP Server to use CARP LAN IP Address

On both firewalls, visit Services -> DHCP Server. Click on the LAN tab. Set the default gateway to 192.168.1.3. Click save.

It also may be a good idea to enable failover DHCP. Enter 192.168.1.2 in the failover peep box on the primary and 192.168.1.1 on the backup server. Click save.

Now for Outbound NAT, I don't know if it's needs to be the LAN Carp or something else

and when I set the GW in the dhcp tab, I lose internet access and can't ping 10.10.10.5 but pings fine with 10.10.10.1/2

What am I doing wrong?

viragomann

Now for Outbound NAT, I don't know if it's needs to be the LAN Carp or something else

The WAN CARP, of course. That's the IP the outbound packets are translated to.

and when I set the GW in the dhcp tab, I lose internet access and can't ping 10.10.10.5 but pings fine with 10.10.10.1/2

Have you set the failover peer IP in DHCP servers LAN tab?

Mad Professor

@viragomann:

Now for Outbound NAT, I don't know if it's needs to be the LAN Carp or something else

The WAN CARP, of course. That's the IP the outbound packets are translated to.

Ok I set it to that, but nothing.

and when I set the GW in the dhcp tab, I lose internet access and can't ping 10.10.10.5 but pings fine with 10.10.10.1/2

Have you set the failover peer IP in DHCP servers LAN tab?

Is it the LAN IP or the SYNC IP? cause I'm doing the sync ip of 1.1.1.2, does it need to be 192.168.0.2? Every time I do the sync ip, the dhcp stops working and the firewall becomes inaccessible period, even setting the windows guest to static won't let me back in the network.

Gotta go do factory reset on the both.
I'll post screenshots of what I'm doing, hopefully we can figure what the "F" I'm doing.

Mad Professor

Found this in my system logs in the DHCP tab.


Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.88: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.89: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.90: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.91: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.92: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.93: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.94: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.95: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.96: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.97: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.98: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: lease 10.0.0.99: no subnet.
Apr 16 21:27:01 	pfSense dhcpd: Corrupt lease file - possible data loss!
Apr 16 21:27:01 	pfSense dhcpd: Wrote 73 leases to leases file.
Apr 16 21:27:01 	pfSense dhcpd: Listening on BPF/em1/08:00:27:d7:d3:95/10.0.0.0/24
Apr 16 21:27:01 	pfSense dhcpd: Sending on BPF/em1/08:00:27:d7:d3:95/10.0.0.0/24
Apr 16 21:27:01 	pfSense dhcpd: Sending on Socket/fallback/fallback-net
Apr 16 21:27:01 	pfSense dhcpd: failover peer dhcp_lan: I move from normal to startup
Apr 16 21:27:01 	pfSense dhcpd: failover peer dhcp_lan: peer moves from normal to partner-down
Apr 16 21:27:01 	pfSense dhcpd: failover peer dhcp_lan: I move from startup to recover
Apr 16 21:27:01 	pfSense dhcpd: Sent update request all message to dhcp_lan
Apr 16 21:27:01 	pfSense dhcpd: failover peer dhcp_lan: peer update completed.
Apr 16 21:27:01 	pfSense dhcpd: failover peer dhcp_lan: I move from recover to recover-wait
Apr 16 21:27:02 	pfSense dhcpd: peer dhcp_lan: disconnected
Apr 16 21:27:05 	pfSense dhcpd: peer dhcp_lan: disconnected
Apr 16 21:27:12 	pfSense dhcpd: DHCPINFORM from 10.0.0.100 via em1
Apr 16 21:27:12 	pfSense dhcpd: DHCPACK to 10.0.0.100 (08:00:27:8c:b7:7f) via em1
Apr 16 21:28:36 	pfSense dhcpd: Received signal 15, initiating shutdown.

What the fuck am I doing wrong?

dotdash

Are you using a dedicated network for the sync traffic?
The DHCP configuration on the slave should be identical except for the failover peer would be 10.0.0.1
Try stopping the DHCP service on both boxes, then start the master, then the slave. Check the DHCP logs again.
I'm sure you cleared the 'block private networks' box on your WAN interfaces…

Mad Professor

@dotdash:

Are you using a dedicated network for the sync traffic?
The DHCP configuration on the slave should be identical except for the failover peer would be 10.0.0.1
Try stopping the DHCP service on both boxes, then start the master, then the slave. Check the DHCP logs again.
I'm sure you cleared the 'block private networks' box on your WAN interfaces…

Yes sync has a virtualbox private network intnet1 and lan has intnet.

And I have "block privates networks" clear on both master and slave.

I keep reading and watching youtube video of other people speaking Spanish able to successfully do it. I guess I need to bang my head on the wall for a a bit.

Mad Professor

I'm gonna call it quits.

I suspect there is something wrong with the dhcpd, specifically gateway and dns and how it interacts with virtual IP. I did a packet capture on the lan interface and send packets to the LAN CARP IP, and I can see the packets make it to the lan interface but from there, they don't show up in the firewall logs, it just if the packets were silently dropped.

On the other hand CARP works, or well the HA sync works, I can see the slave take the role when the master goes down, but other than that, a bit worthless if you can't switch the connections over seamlessly.

Virtual IPs kind of work, well they work from the pfsense side with the GUI ping, you can ping google.com from all the virtual IPs, but no one on the network can resolve google.com when using the GW/DNS with a VIP
DHCP failover works somewhat, I kept finding either the Master being in recover and claiming it's peer is partner down, and on the slave it's saying same the thing that the master is partner down and he's trying to recover. It made no sense, only a slave reboot set it back to normal until the next client renew it's lease then both would claming that it's peer was down and it's trying to recover. I've checked the VIP for adskew, master 0 and slave 100.
It made no sense.

Another thing, seems to me that Automatic Outbound and Advanced Manual Outbound I didn't see a damn difference between the two, so either it's completely broken, or automatic outbound does damn good job at setting the nat rules.

I followed the guide and read up on quite a few threads and watch about 6 videos on youtube and done it exactly the same way as they did and I still couldn't get it to work.

I've only managed to get it to work once and I don't even know how I did it. I haven't been able to reproduce it.

bshack

It may be something in the networking stack on the hosting box for the VM's. I was having all kinds of issues with doing this testing in a VM enviroment and finally just grabbed a couple of boxes and tested on physical machines. Followed the setup guide and bang - it works like a champ!

Thanks,
Brian

Mad Professor

I think you're right, I was messing with promiscuous mode on virtualbox nics and everything started working some what fine but still wasn't working as intended.

What I wanted to do was set this up on my server which is running a virtual instance of pfsense for my network at home, lately my server has been having issues and I kept breaking stuff and the internet goes down for few hours; and it becomes difficult to fix things when you don't have the resources of the internet and have to rely on a mobile data plan from your phone. But anyways what I wanted to do was, I had a physical box which use to be my old pfsense router burning 80watts 24/7 which is why I went to virtual setup. Anyways I wanted to CARP to this box so I can take down the server for maintenance and still have internet and not interrupt anybody in the home, who may be playing video games, watching netflix so on.

The server runs CentOS 6.5 with KVM, and pfsense utilizes virtio drivers. I can't find anything for promiscuous mode settings for KVM, even though a web search suggest to acknowledge that a promiscuous mode setting does exist. I just have no idea how yet, although I haven't dug deep into it yet.