I have two masters

mgiammarco

Hello I have two pfsense 1.4rc that do dual wan. They have for network card each: lan, wan, wan2 and carp. Carp is a direct connection with cross cable. The master one show master for each virtual ip. The slave one show master too for WAN (wrong!) and backup (right!) for wan2 and lan.

It seems to work well anyway but sometimes I have some strange network problem that I stop shutting down slave.

Please help me!

Thanks in advance for any help.

Mario

hoba

Something at your WAN seems to block the CARP-broadcasts so that your Machines can't see each other at the WAN interface. If the backupmachine can't see the heartbeat of the master it goes to master itself as it thinks the real master has died. Check your configuration at WAN. If you plugin a Crossovercable between the WAN interfaces of both machines (with no switch and nothing in between) does the backup machine drop the master state? If yes something is wrong with your Switch/Connection at WAN. If not it most likely is a configuration problem and we need some further details but I would first do the simple cable test.

mgiammarco

Are you saying that the dedicated network interface for carp is not enough? I can connect two carp directly with crossover but the wan interface connect to the hdsl router with some switches. In the wan lan I have only two pfsense and the modem. I will try direct connection just for debugging but I supposed that dedicated carp was enough.

hoba

You have some CARP-Broadcasts on any interface that has a virtual CARP IP. This way it will as well detect link failures and not only complete machine failures. Have a look at http://www.countersiege.com/doc/pfsync-carp/ if you are interested how it works in detail. Actually the dedicated Sync interface is only for synchronizing the statetables which is done by the pfSync-Protocol.

mgiammarco

Thanks I have read it and now picture is clear (but I am not happy).

I have no spare cross cables so now I have rebooted master and slave and I have disconnected their wan interfaces from the dell switch and I connected them and only them to a new intelllinet 8 port gigabit switch. Slave is always master.
I do not understand now why wan2 and lan are ok: infact them are on the same dell switch (different vlan obviously). Using same switch I expected same result: all working or none working.

Anyway could you suggest me a switch that accept carp multicast?

I write here pfsync hashes after reboot and after some time:

master

1b1f2768
8fa1ba67
f0104f21
f10bb0ea

0a0fe2e6
0b9e5310
1b1f2768
4a7698da
639103a5
6aa7e8a4
f0104f21
f10bb0ea

slave

8fa1ba67
f0104f21

0b9e5310
54648b0b
639103a5
6aa7e8a4
8cd0c167
8fa1ba67
f0104f21

hoba

Please recheck your configuration and make sure your vlans are setup correctly and the vhids of the CARP virtual IPs at both machines are identical for the same IP and all CARP IPs are assigned to the correct interfaces. This is definately some kind of cnfiguration mismatch as the other carp IPs work fine with the same switch (either on the switch or the pfSense for this CARP vip).

Also have a look at the systemlogs if you see something obvious there.

mgiammarco

I have removed the vip that causes problems. I have created it again with vhid 10 (it was vhid 1). Log shows no errors (apparently). I have tried with another switch and vlan. I still have two masters (and after a bit now network goes crazy: drops packets etc.)

What can I do now?

An interesting thing: last month we had old switches without vlan. So I had to put master and slave and all they subnet and all their virtual ip on the same physical lan (sigh…). After a bit of time with all on same lan windows machines got crazy: they put out a strange error saying that 192.168.10.201 (the virtual ip with two masters) was not partecipating with smb networks. The a broadcast storm killed the net.

So I bought vlan switches to put 192.168.10.0/24 on a different vlan. But this virtual ip does not want to work well. Other two works.

Can it help for the problem solution?

Thanks again.

hoba

What nics are you using? Is this one CARP VIP that behaves strange on a nic of a different nic vendor? Maybe it's the nic or the driver of this special nic? If this one nic is different from the others can you try to get one more nic of the same type like the ones are that work fine and retest with that?

mgiammarco

I have changed master computer with a dell. Reinstalled from scratch pfsense 1.2 final. Power up slave: all interfaces now are marked backup. It seemed to work. But after a bit (I repeat, not immediately) voip phone stopped working and I experienced stange network behaviour.

So I powered down slave. Today I powered it up again and it shows again the dual master problem only on wan interface.

The strange thing is that now correctly internal network works (only wan network has problems). But voip phones in internal network do not work (voice with heavy distortion).

I am using packet capture: in wan network master and slave send vrrp2 advertisement (it is a ucarp packet, but also YOUR packet capture says it is vrrp….).

I have saved .cap file but it is xml/unreadable, is there an utility to esasily browse .cap files?

Can it be a firewall problem? I have checked wan and wan2 firewall and I have no specific rules for multicast/carp. Do I need them?

hoba

If there is vrrp traffic on the same broadcastdomain make sure the vrrp and the carp vhids are different!

mgiammarco

The only vrrp traffic is the pfense one. The pfsense packet sniffer says it is vrrp2 but it is obvious that it is ucarp.

Anyway looking at firewall I have noticed that by mistake wan2 ports were all open. So now I have closed all ports in wan2. I rebooted slave and now all network is working. But DUAL MASTER is there yet. The only thing I have done is closing wan2 ports. There is dual master (and dual advertisement) but no more network problems: voip phones work, people surf internet.

Can someone tell me what is happening?
Can someone tell me what packets ucarp sends when all is ok?

Thanks again.

eri--

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574

could by any chance this help you?

sullrich

Normally our XMLRPC sync facilities would prevent this if you are syncing the configuration from the primary to secondary in the CARP settings tab.

mgiammarco

@ermal:

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574

could by any chance this help you?

Thanks for this, I will check for it but I have discovered another thing I will check before: I put it in next reply.

mgiammarco

Today I have discovered an important thing.

I was trying to migrate openvpn from tcp to udp. I have changed the firewall rule to accept udp and also openvpn type to udp.

But it does not work!

This time I started serious debugging with packet trace. I have checked that udp packets arrive to pfsense ip. I have also checked that openvpn daemon listen to right port and right ip for udp packets.

But openvpn does not work.

Why this matter for my carp problem? Because looking packet trace I have discovered that carp udp packets are blocked by firewall! So I immediately added a rule in the firewall to enable carp protocol. It did not work. So I enabled a rule to pass all udp packets. It did not work.

How can I enable udp in openvpn and carp on my wan interface? Wan2 works… at least for carp, I have no openvpn on wan2.

Thanks in advance!

Mario

mgiammarco

I have finally solved the problem. I post here the solution because it is a common error that can happen (I am asking pfsense programmers to modify pfsense behaviour):

in wan (and only in wan, not wan2) there is a setting to "block private networks". It is suggested to check it, but nobody warns that blocking private networks block also vrrp advertisement!!!!!!!!!!!

Now I will investigate on the openvpn not working on udp problem.