WAN works only for few seconds - IRRELEVANT

fernandopsilveira

Right now I have in my hands the following:

2 pfSense boxes with CARP as MASTER and BACKUP (let's call them A and B);
Each one has two WANs, 1 IP for each WAN interface + 1 CARP IP for each interface (all public IPs);
Each one has 6 VLANs, 1 IP for each VLAN + 1 CARP IP for each VLAN;

Box A:
WAN: xxx.xxx.xxx.108
WAN2: yyy.yyy.yyy.108

Box B:
WAN: xxx.xxx.xxx.109
WAN2: yyy.yyy.yyy.109

CARP:
WAN: xxx.xxx.xxx.106
WAN2: yyy.yyy.yyy.104

WAN gateway: xxx.xxx.xxx.105
WAN2 gateway: yyy.yyy.yyy.105

On B machine everything works perfectly. On A machine WAN doesn't work for more then few seconds. WAN2 works like a charm.

Even when WAN gateway is shown as down, I can access A using WAN CARP IP (WAN IP doesn't work, though).

Because of that, I manually got one VLAN down on A (ifconfig <if>down) so B assumes as MASTER. The WAN issue still happens when A is BACKUP.

WAN works for a few seconds when I set its configuration (IP, Gateway, etc), and then gateway is shown as down, doesn't reach anything

I was monitoring using route -n monitor.

The following shows up when I set the interface:

got message of size 172 on Tue Jun  9 16:44:36 2015
RTM_DELETE: Delete Route: len 172, pid: 0, seq 0, errno 0, flags:
locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 xxx.xxx.xxx.96  (255) ffff ffff f0ff

got message of size 116 on Tue Jun  9 16:44:36 2015
RTM_DELADDR: address being removed from iface: len 116, metric 0, flags: <up>sockaddrs: <netmask,ifp,ifa,brd>
 255.255.255.240 em0:e0.e0.c8.89.b1.3 xxx.xxx.xxx.108 xxx.xxx.xxx.111

got message of size 116 on Tue Jun  9 16:44:36 2015
RTM_DELETE: Delete Route: len 116, pid: 0, seq 0, errno 0, flags: <up,gateway,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 default default default

got message of size 104 on Tue Jun  9 16:44:36 2015
RTM_DELADDR: address being removed from iface: len 104, metric 0, flags:
sockaddrs: <netmask,ifp,ifa,brd>
 default em0:e0.e0.c8.89.b1.3 default default

got message of size 116 on Tue Jun  9 16:44:36 2015
RTM_NEWADDR: address being added to iface: len 116, metric 0, flags:
sockaddrs: <netmask,ifp,ifa,brd>
 255.255.255.240 em0:e0.e0.c8.89.b1.3 xxx.xxx.xxx.108 xxx.xxx.xxx.111

got message of size 172 on Tue Jun  9 16:44:36 2015
RTM_ADD: Add Route: len 172, pid: 0, seq 0, errno 0, flags: <up>locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 xxx.xxx.xxx.96  (255) ffff ffff f0ff

got message of size 144 on Tue Jun  9 16:44:36 2015
RTM_NEWMADDR: new multicast group membership on iface: len 144, 
sockaddrs: <gateway,ifp,ifa>
 1.0.5e.0.0.1 em0:e0.e0.c8.89.b1.3 224.0.0.1</gateway,ifp,ifa></dst,gateway,netmask></up></netmask,ifp,ifa,brd></netmask,ifp,ifa,brd></dst,gateway,netmask></up,gateway,static></netmask,ifp,ifa,brd></up></dst,gateway,netmask>

After a few seconds, WAN stops working, doesn't ping anything anymore and the following shows up:

got message of size 200 on Tue Jun  9 16:44:50 2015
RTM_GET: Report Metrics: len 200, pid: 43178, seq 1, errno 0, flags: <up,gateway,done,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask,ifp,ifa>
 default yyy.yyy.yyy.105 default em1:e0.e0.c8.89.b1.4 yyy.yyy.yyy.108

got message of size 128 on Tue Jun  9 16:44:50 2015
RTM_CHANGE: Change Metrics or flags: len 128, pid: 43439, seq 1, errno 0, flags: <up,gateway,done,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 default xxx.xxx.xxx.105 default

got message of size 180 on Tue Jun  9 16:44:50 2015
RTM_GET: Report Metrics: len 180, pid: 43769, seq 1, errno 3, flags: <up,gateway,static>locks:  inits: 
sockaddrs: <dst,netmask,ifp>
 :: :: 

got message of size 200 on Tue Jun  9 16:44:51 2015
RTM_GET: Report Metrics: len 200, pid: 44830, seq 1, errno 0, flags: <up,gateway,done,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask,ifp,ifa>
 default xxx.xxx.xxx.105 default em0:e0.e0.c8.89.b1.3 xxx.xxx.xxx.108

got message of size 180 on Tue Jun  9 16:44:51 2015
RTM_GET: Report Metrics: len 180, pid: 44972, seq 1, errno 3, flags: <up,gateway,static>locks:  inits: 
sockaddrs: <dst,netmask,ifp>
 :: :: 

got message of size 128 on Tue Jun  9 16:45:08 2015
RTM_CHANGE: Change Metrics or flags: len 128, pid: 75185, seq 1, errno 0, flags: <up,gateway,done,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 default yyy.yyy.yyy.105 default

got message of size 180 on Tue Jun  9 16:45:08 2015
RTM_GET: Report Metrics: len 180, pid: 75481, seq 1, errno 3, flags: <up,gateway,static>locks:  inits: 
sockaddrs: <dst,netmask,ifp>
 :: :: 

got message of size 128 on Tue Jun  9 16:45:10 2015
RTM_CHANGE: Change Metrics or flags: len 128, pid: 76264, seq 1, errno 0, flags: <up,gateway,done,static>locks:  inits: 
sockaddrs: <dst,gateway,netmask>
 default yyy.yyy.yyy.105 default

got message of size 180 on Tue Jun  9 16:45:10 2015
RTM_GET: Report Metrics: len 180, pid: 76551, seq 1, errno 3, flags: <up,gateway,static>locks:  inits: 
sockaddrs: <dst,netmask,ifp>
 :: ::</dst,netmask,ifp></up,gateway,static></dst,gateway,netmask></up,gateway,done,static></dst,netmask,ifp></up,gateway,static></dst,gateway,netmask></up,gateway,done,static></dst,netmask,ifp></up,gateway,static></dst,gateway,netmask,ifp,ifa></up,gateway,done,static></dst,netmask,ifp></up,gateway,static></dst,gateway,netmask></up,gateway,done,static></dst,gateway,netmask,ifp,ifa></up,gateway,done,static>

Does anyone have any clue of what is going on? :-</if>

fernandopsilveira

I finally figured out what happened:

looks like some wise guy decided to play a little with his station and putted a valid external IP on it, using pfSense WAN IP for it >:(

KOM

Sounds like you've got a user who needs a reduction in his permissions.

doktornotor

@KOM:

Sounds like you've got a user who needs a reduction in his permissions.