Rancher node ipsec behind pfsense



  • Hello,
    I have a system using rancher 1.6 and pfsense routers.

    I have 3 datacenters, each one has one pfsense and they are connected each-other by an ipsec connection.

    Behind each pfsense I have a rancher node machine. Because Rancher create a ipsec connection between nodes, on each pfsense I have NAT rules in ports 500 and 4500 to forward traffic for the right node. Everything works great.

    Now a add a second node on one of the datacenters, behind the pfsense that were there before, I have created a second public ip in pfsense (virtual ip) and create the same NAT rules, from that node I can ping the other nodes on the other datacenters but not the node on the same datacenter, I've done lots of tests and it something on the pfsense that is blocking, anyone has a clue on what I'm missing or doing wrong here?
    rancherpfsense.png

    Note: The IP's are representative

    		Interface 	Protocol 	Source Address 	Source Ports 	Dest. Address 	Dest. Ports 	NAT IP 	NAT Ports 	Description 
    
    		WAN 	UDP 	* 	* 	1.1.1.1 	500 (ISAKMP) 	192.168.1.1 	500 (ISAKMP) 	rancher-node-1 	
    		WAN 	UDP 	* 	* 	1.1.1.2 	500 (ISAKMP) 	192.168.1.4 	500 (ISAKMP) 	rancher-node-4 	
    		WAN 	UDP 	* 	* 	1.1.1.1 	4500 (IPsec NAT-T) 	192.168.1.1 	4500 (IPsec NAT-T) 	rancher-node-1 	
    		WAN 	UDP 	* 	* 	1.1.1.2 	4500 (IPsec NAT-T) 	192.168.1.4 	4500 (IPsec NAT-T) 	rancher-node-4 	
    

    Outbound Mappings
    Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description Actions

    	WAN 	192.168.1.1/32 	* 	* 	* 	1.1.1.1 	* 		rancher-node-1 	
    	WAN 	192.168.1.4/32 	* 	* 	* 	1.1.1.2 	* 		rancher-node-4 	
    

    In pfsense1 ipsec log I see messages like this:
    Jun 5 12:35:35 charon 13[NET] received packet: from 192.168.1.1[1024] to 1.1.1.2[500] (500 bytes)
    Jun 5 12:35:35 charon 13[NET] <291> received packet: from 192.168.1.1[1024] to 1.1.1.2[500] (500 bytes)
    Jun 5 12:35:35 charon 13[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
    Jun 5 12:35:35 charon 13[ENC] <291> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
    Jun 5 12:35:35 charon 13[CFG] looking for an IKEv2 config for 1.1.1.2...192.168.1.1
    Jun 5 12:35:35 charon 13[CFG] <291> looking for an IKEv2 config for 1.1.1.2...192.168.1.1
    Jun 5 12:35:35 charon 13[IKE] no IKE config found for 1.1.1.2...192.168.1.1, sending NO_PROPOSAL_CHOSEN
    Jun 5 12:35:35 charon 13[IKE] <291> no IKE config found for 1.1.1.2...192.168.1.1, sending NO_PROPOSAL_CHOSEN
    Jun 5 12:35:35 charon 13[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
    Jun 5 12:35:35 charon 13[ENC] <291> generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
    Jun 5 12:35:35 charon 13[NET] sending packet: from 1.1.1.2[500] to 192.168.1.1[1024] (36 bytes)
    Jun 5 12:35:35 charon 13[NET] <291> sending packet: from 1.1.1.2[500] to 192.168.1.1[1024] (36 bytes)
    Jun 5 12:35:35 charon 13[IKE] IKE_SA (unnamed)[291] state change: CREATED => DESTROYING
    Jun 5 12:35:35 charon 13[IKE] <291> IKE_SA (unnamed)[291] state change: CREATED => DESTROYING
    Jun 5 12:36:34 charon 14[NET] received packet: from 192.168.1.4[1024] to 1.1.1.1[500] (500 bytes)
    Jun 5 12:36:34 charon 14[NET] <292> received packet: from 192.168.1.4[1024] to 1.1.1.1[500] (500 bytes)
    Jun 5 12:36:34 charon 14[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
    Jun 5 12:36:34 charon 14[ENC] <292> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
    Jun 5 12:36:34 charon 14[CFG] looking for an IKEv2 config for 1.1.1.1...192.168.1.4
    Jun 5 12:36:34 charon 14[CFG] <292> looking for an IKEv2 config for 1.1.1.1...192.168.1.4
    Jun 5 12:36:34 charon 14[IKE] no IKE config found for 1.1.1.1...192.168.1.4, sending NO_PROPOSAL_CHOSEN
    Jun 5 12:36:34 charon 14[IKE] <292> no IKE config found for 1.1.1.1...192.168.1.4, sending NO_PROPOSAL_CHOSEN
    Jun 5 12:36:34 charon 14[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
    Jun 5 12:36:34 charon 14[ENC] <292> generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
    Jun 5 12:36:34 charon 14[NET] sending packet: from 1.1.1.1[500] to 192.168.1.4[1024] (36 bytes)
    Jun 5 12:36:34 charon 14[NET] <292> sending packet: from 1.1.1.1[500] to 192.168.1.4[1024] (36 bytes)
    Jun 5 12:36:34 charon 14[IKE] IKE_SA (unnamed)[292] state change: CREATED => DESTROYING
    Jun 5 12:36:34 charon 14[IKE] <292> IKE_SA (unnamed)[292] state change: CREATED => DESTROYING


Log in to reply