Re: Issue with multi wan high availability setup - authenticating with radius -- WAY MORE INFO
-
Re: Issue with multi wan high availability setup - authenticating with radius
I decided to re-write the first post to include more detail.
I have a multi-WAN, High Availability setup with two Netgate 1537s running pfSense 22.01.
I am running into a few issues:
- I am unable to authenticate from pfSense to an external RADIUS server. When I try to authenticate via pfSense > Diagnostics > Authentication, or from openVPN, i don't see a response from the remote server on my WAN 2 interface. (WAN 2 is my only current active intrerface - way more detail below.)
** However, if I authenticate to the same remote RADIUS server from a device on the LAN (including a pfsense VM in multi-WAN, high availibility setup), i can authenticate just fine which leads me to believe the issue is something with NAT / Outbound NAT. - My ssh connection to pfSense B LAN IP disconnects after about a minute after connecting.
- Some sites are not fully loading all information (a couple examples below)
** When I try to load dashboard.envoy.com
** Amazon phone app does not load all information
Right now, we only have one ISP (our backup ISP / Secondary WAN interface) while we wait for our main ISP to finish their drop.
Although the secondary ISP is providing us with public IPs, they have us NATed and VLAN tagged.
- Public IP 1 forwarding all traffic to 10.99.80.0/24 (VLAN 80) | Gateway 10.99.80.1
- Public IP 2 forwarding all traffic to 10.99.81.0/24 (VLAN 81) | Gateway 10.99.81.1
- Public IP 3 forwarding all traffic to 10.99.82.0/24 (VLAN 82) | Gateway 10.99.82.1
I am only using Public IP 1 for this setup (Public IP 1) due to ISP NAT
Steps completed on both pfSense A & B
- Set up Sync interface for high availability (sync is working as intended)
Steps completed on pfSense A
- Set up WAN 1 interface (ix0) with 10.20.20.3/24 // Gateway 10.20.20.1
- Set up WAN 1 CARP IP 10.20.20.1
- Created VLAN 80 on WAN 2 interface (ix1)
- WAN 2 interface (ix1.80) with 10.99.80.3/24 // Gateway 10.99.80.1
- Set up WAN 2 CARP IP 10.99.80.2
- Set up Native LAN 10.10.3.2/24
- Set up Native LAN CARP IP 10.10.3.1
- Created VLAN 11 on LAN interface (ixl0)
- Set up LAN VLAN 11 (ixl0.11) IP 192.168.11.2/24
- Set up VLAN 11 CARP IP 192.168.11.1
- Set up gateway group
- Set up Firewall Rules
- Set up Outbound NAT
Steps completed on pfSense B
- Set up WAN 1 interface (ix0) with 10.20.20.4/24 // Gateway 10.20.20.1
- Set up VLAN 80 on WAN 2 interface (ix1)
- WAN 2 interface (ix1.80) with 10.99.80.4/24 // Gateway 10.99.80.1
- Set up Native LAN 10.10.3.2/24
- Set up VLAN 11 on LAN interface (ixl0)
- Set up LAN VLAN 11 (ixl0.11) IP 192.168.11.3/24
- NOTE: CARP IPs, Firewall rules and Outbound NAT all sync'd over from pfSense A
Here is what the physical setup looks like
Packet Captures
here is a packet capture of WAN 2 (ix1.80) when i try to authenticate from pfSense A > Diagnostics > Authentication. I don't get a response back from the remote server.
[22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp: tcpdump -r WAN_request_fail.pcap -vvvv reading from file WAN_request_fail.pcap, link-type EN10MB (Ethernet) 10:58:15.960167 IP (tos 0x0, ttl 64, id 51610, offset 0, flags [none], proto UDP (17), length 200, bad cksum 0 (->433a)!) 10.99.80.3.58597 > ec2-18-204-0-31.compute-1.amazonaws.com.radius: [bad udp cksum 0x6e16 -> 0x50d3!] RADIUS, length: 172 Access-Request (1), id: 0xe4, Authenticator: 1796ea4eb5da963713529a9d05e61077 Service-Type Attribute (6), length: 6, Value: Login 0x0000: 0000 0001 User-Name Attribute (1), length: 6, Value: marc 0x0000: 6d61 7263 Vendor-Specific Attribute (26), length: 58, Value: Vendor: Microsoft (311) Vendor Attribute: 25, Length: 50, Value: ......]Jk.{p..bd...........I.e0+....7.>...L..nS... 0x0000: 0000 0137 1934 0101 c807 1ef5 5d4a 6be9 0x0010: 7b70 9eee 6264 c785 0000 0000 0000 0000 0x0020: ab49 a265 302b 0da3 a4c3 37f8 3e12 f3c2 0x0030: 4cf1 956e 53d0 a58d Vendor-Specific Attribute (26), length: 24, Value: Vendor: Microsoft (311) Vendor Attribute: 11, Length: 16, Value: .%...Ss...Q..|.= 0x0000: 0000 0137 0b12 0125 bfab c553 73e0 11fe 0x0010: 5182 117c 913d NAS-IP-Address Attribute (4), length: 6, Value: 10.99.80.3 0x0000: 0a63 5003 NAS-Identifier Attribute (32), length: 17, Value: fw-1.sdhcap.lan 0x0000: 6677 2d31 2e73 6468 6361 702e 6c61 6e Called-Station-Id Attribute (30), length: 35, Value: 3c:ec:ef:32:91:d9:fw-1.sdhcap.lan 0x0000: 3363 3a65 633a 6566 3a33 323a 3931 3a64 0x0010: 393a 6677 2d31 2e73 6468 6361 702e 6c61 0x0020: 6e 10:58:21.150665 IP (tos 0x0, ttl 64, id 56483, offset 0, flags [none], proto UDP (17), length 200, bad cksum 0 (->3031)!) 10.99.80.3.58597 > ec2-18-204-0-31.compute-1.amazonaws.com.radius: [bad udp cksum 0x6e16 -> 0x50d3!] RADIUS, length: 172 Access-Request (1), id: 0xe4, Authenticator: 1796ea4eb5da963713529a9d05e61077 Service-Type Attribute (6), length: 6, Value: Login 0x0000: 0000 0001 User-Name Attribute (1), length: 6, Value: marc 0x0000: 6d61 7263 Vendor-Specific Attribute (26), length: 58, Value: Vendor: Microsoft (311) Vendor Attribute: 25, Length: 50, Value: ......]Jk.{p..bd...........I.e0+....7.>...L..nS... 0x0000: 0000 0137 1934 0101 c807 1ef5 5d4a 6be9 0x0010: 7b70 9eee 6264 c785 0000 0000 0000 0000 0x0020: ab49 a265 302b 0da3 a4c3 37f8 3e12 f3c2 0x0030: 4cf1 956e 53d0 a58d Vendor-Specific Attribute (26), length: 24, Value: Vendor: Microsoft (311) Vendor Attribute: 11, Length: 16, Value: .%...Ss...Q..|.= 0x0000: 0000 0137 0b12 0125 bfab c553 73e0 11fe 0x0010: 5182 117c 913d NAS-IP-Address Attribute (4), length: 6, Value: 10.99.80.3 0x0000: 0a63 5003 NAS-Identifier Attribute (32), length: 17, Value: fw-1.sdhcap.lan 0x0000: 6677 2d31 2e73 6468 6361 702e 6c61 6e Called-Station-Id Attribute (30), length: 35, Value: 3c:ec:ef:32:91:d9:fw-1.sdhcap.lan 0x0000: 3363 3a65 633a 6566 3a33 323a 3931 3a64 0x0010: 393a 6677 2d31 2e73 6468 6361 702e 6c61 0x0020: 6e [22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp:Here is a packet capture of VLAN 11 (ixl0.11) on pfSense A when i succesfully authenticate from a pfsense VM thats on VLAN 11.
[22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp: tcpdump -r WAN_request_from_lan_success.pcap -vvvv reading from file WAN_request_from_lan_success.pcap, link-type EN10MB (Ethernet) 10:57:11.517727 IP (tos 0x0, ttl 64, id 22073, offset 0, flags [none], proto UDP (17), length 210) 192.168.11.102.23310 > ec2-18-204-0-31.compute-1.amazonaws.com.radius: [udp sum ok] RADIUS, length: 182 Access-Request (1), id: 0xcc, Authenticator: c569f84d9b0bb4f4a5b5043fb4a17c1a Service-Type Attribute (6), length: 6, Value: Login 0x0000: 0000 0001 User-Name Attribute (1), length: 6, Value: marc 0x0000: 6d61 7263 Vendor-Specific Attribute (26), length: 58, Value: Vendor: Microsoft (311) Vendor Attribute: 25, Length: 50, Value: ...zs....J.!.....V........}.xC..G.`.3...99...%.... 0x0000: 0000 0137 1934 0101 fc7a 73c2 0eb1 1c4a 0x0010: 9221 941e f310 1856 0000 0000 0000 0000 0x0020: 7d0f 7843 9aad 47d0 60ea 33a1 c8ce 3939 0x0030: 8797 0925 e983 ccee Vendor-Specific Attribute (26), length: 24, Value: Vendor: Microsoft (311) Vendor Attribute: 11, Length: 16, Value: ..OY..u.3Gv}^n.. 0x0000: 0000 0137 0b12 17a8 4f59 1ee9 7587 3347 0x0010: 767d 5e6e 8ea9 NAS-IP-Address Attribute (4), length: 6, Value: 192.168.11.102 0x0000: c0a8 0b66 NAS-Identifier Attribute (32), length: 22, Value: pfSense-one.home.lan 0x0000: 7066 5365 6e73 652d 6f6e 652e 686f 6d65 0x0010: 2e6c 616e Called-Station-Id Attribute (30), length: 40, Value: b2:ed:80:01:16:bb:pfSense-one.home.lan 0x0000: 6232 3a65 643a 3830 3a30 313a 3136 3a62 0x0010: 623a 7066 5365 6e73 652d 6f6e 652e 686f 0x0020: 6d65 2e6c 616e 10:57:11.553134 IP (tos 0x0, ttl 42, id 26823, offset 0, flags [none], proto UDP (17), length 207, bad cksum 0 (->485e)!) ec2-18-204-0-31.compute-1.amazonaws.com.radius > 192.168.11.102.23310: [udp sum ok] RADIUS, length: 179 Access-Accept (2), id: 0xcc, Authenticator: 56a8f97d3af71a71d06015aec3408da3 Vendor-Specific Attribute (26), length: 42, Value: Vendor: Microsoft (311) Vendor Attribute: 17, Length: 34, Value: .w.....M.3..'K.4Z...Z...G.h./....e 0x0000: 0000 0137 1124 ec77 e285 aed3 fe4d e733 0x0010: ef0a 274b f534 5adc 8bf6 5a08 08e4 47ee 0x0020: 681b 2fa2 a2bc de65 Vendor-Specific Attribute (26), length: 51, Value: Vendor: Microsoft (311) Vendor Attribute: 26, Length: 43, Value: .S=7AFE09F54443B32898C5AF9400F8E23B933FF7BC 0x0000: 0000 0137 1a2d 0153 3d37 4146 4530 3946 0x0010: 3534 3434 3342 3332 3839 3843 3541 4639 0x0020: 3430 3046 3845 3233 4239 3333 4646 3742 0x0030: 43 Vendor-Specific Attribute (26), length: 42, Value: Vendor: Microsoft (311) Vendor Attribute: 16, Length: 34, Value: ...{YZ.Ky.y..eC..~..$..D...;.V>... 0x0000: 0000 0137 1024 f3f0 f87b 595a 0b4b 79c2 0x0010: 79d3 8765 43f9 b97e d9e2 248b f744 fde0 0x0020: 8c3b 0156 3ec0 e419 Vendor-Specific Attribute (26), length: 12, Value: Vendor: Microsoft (311) Vendor Attribute: 8, Length: 4, Value: .... 0x0000: 0000 0137 0806 0000 0006 Vendor-Specific Attribute (26), length: 12, Value: Vendor: Microsoft (311) Vendor Attribute: 7, Length: 4, Value: .... 0x0000: 0000 0137 0706 0000 0001 [22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp:Here is the same succesful request while monitoring WAN 2 (ix1.80)
[22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp: tcpdump -r lan_success.pcap -vvvv reading from file lan_success.pcap, link-type EN10MB (Ethernet) 10:57:11.517763 IP (tos 0x0, ttl 64, id 22073, offset 0, flags [none], proto UDP (17), length 210, bad cksum 0 (->b692)!) 10.99.80.2.40819 > ec2-18-204-0-31.compute-1.amazonaws.com.radius: [udp sum ok] RADIUS, length: 182 Access-Request (1), id: 0xcc, Authenticator: c569f84d9b0bb4f4a5b5043fb4a17c1a Service-Type Attribute (6), length: 6, Value: Login 0x0000: 0000 0001 User-Name Attribute (1), length: 6, Value: marc 0x0000: 6d61 7263 Vendor-Specific Attribute (26), length: 58, Value: Vendor: Microsoft (311) Vendor Attribute: 25, Length: 50, Value: ...zs....J.!.....V........}.xC..G.`.3...99...%.... 0x0000: 0000 0137 1934 0101 fc7a 73c2 0eb1 1c4a 0x0010: 9221 941e f310 1856 0000 0000 0000 0000 0x0020: 7d0f 7843 9aad 47d0 60ea 33a1 c8ce 3939 0x0030: 8797 0925 e983 ccee Vendor-Specific Attribute (26), length: 24, Value: Vendor: Microsoft (311) Vendor Attribute: 11, Length: 16, Value: ..OY..u.3Gv}^n.. 0x0000: 0000 0137 0b12 17a8 4f59 1ee9 7587 3347 0x0010: 767d 5e6e 8ea9 NAS-IP-Address Attribute (4), length: 6, Value: 192.168.11.102 0x0000: c0a8 0b66 NAS-Identifier Attribute (32), length: 22, Value: pfSense-one.home.lan 0x0000: 7066 5365 6e73 652d 6f6e 652e 686f 6d65 0x0010: 2e6c 616e Called-Station-Id Attribute (30), length: 40, Value: b2:ed:80:01:16:bb:pfSense-one.home.lan 0x0000: 6232 3a65 643a 3830 3a30 313a 3136 3a62 0x0010: 623a 7066 5365 6e73 652d 6f6e 652e 686f 0x0020: 6d65 2e6c 616e 10:57:11.553125 IP (tos 0x0, ttl 43, id 26823, offset 0, flags [none], proto UDP (17), length 207) ec2-18-204-0-31.compute-1.amazonaws.com.radius > 10.99.80.2.40819: [udp sum ok] RADIUS, length: 179 Access-Accept (2), id: 0xcc, Authenticator: 56a8f97d3af71a71d06015aec3408da3 Vendor-Specific Attribute (26), length: 42, Value: Vendor: Microsoft (311) Vendor Attribute: 17, Length: 34, Value: .w.....M.3..'K.4Z...Z...G.h./....e 0x0000: 0000 0137 1124 ec77 e285 aed3 fe4d e733 0x0010: ef0a 274b f534 5adc 8bf6 5a08 08e4 47ee 0x0020: 681b 2fa2 a2bc de65 Vendor-Specific Attribute (26), length: 51, Value: Vendor: Microsoft (311) Vendor Attribute: 26, Length: 43, Value: .S=7AFE09F54443B32898C5AF9400F8E23B933FF7BC 0x0000: 0000 0137 1a2d 0153 3d37 4146 4530 3946 0x0010: 3534 3434 3342 3332 3839 3843 3541 4639 0x0020: 3430 3046 3845 3233 4239 3333 4646 3742 0x0030: 43 Vendor-Specific Attribute (26), length: 42, Value: Vendor: Microsoft (311) Vendor Attribute: 16, Length: 34, Value: ...{YZ.Ky.y..eC..~..$..D...;.V>... 0x0000: 0000 0137 1024 f3f0 f87b 595a 0b4b 79c2 0x0010: 79d3 8765 43f9 b97e d9e2 248b f744 fde0 0x0020: 8c3b 0156 3ec0 e419 Vendor-Specific Attribute (26), length: 12, Value: Vendor: Microsoft (311) Vendor Attribute: 8, Length: 4, Value: .... 0x0000: 0000 0137 0806 0000 0006 Vendor-Specific Attribute (26), length: 12, Value: Vendor: Microsoft (311) Vendor Attribute: 7, Length: 4, Value: .... 0x0000: 0000 0137 0706 0000 0001 [22.01-RELEASE][admin@fw-1.sdhcap.lan]/tmp:Troubleshooting steps
Since I see that the failed authentication attempt is sent from ix1.80 10.99.80.3 (interface IP, not CARP IP) AND that the successful authentation attempt starts from VLAN 11 (192.168.11.102) and leaves WAN 2, ix1.80, as 10.99.80.2, i tried createing another outbound NAT rule:
Authentication works, however this seems to break routing on pfSense B. when i ssh into pfSense B, i am unable to ping 8.8.8.8 or google.com. When i enter CARP mode, i can ping to 8.8.8.8 and google.com. when i disable CARP mode and disable the outbound NAT rules, i am able to ping out to 8.8.8.8 and google.com from pfSense B again but radius auth no longer works. .
Another thought is that something is happening on the ISP side. I think this because when i set up 2 pfsense VMs in HA / multi-WAN on the LAN network, authentication works just fine from both VMs and I am able to ping out to google.com / 8.8.8.8 from both pfsense VMs. BUT, when im not in a HA / multi-WAN setup, auth works fine so i'm not certain its an ISP issue.
I hope this is enough information so that someone can assist. Thank you for your time!
pcap files:
WAN_request_fail.pcap
VLAN_11_request_from_lan_success.pcap
WAN2_during_lan_success.pcap - I am unable to authenticate from pfSense to an external RADIUS server. When I try to authenticate via pfSense > Diagnostics > Authentication, or from openVPN, i don't see a response from the remote server on my WAN 2 interface. (WAN 2 is my only current active intrerface - way more detail below.)
-
S se_marc referenced this topic on Mar 22, 2022, 6:16 PM
-
so i found that pfblockerNG-dev was causing dashboard.envoy.com and amazon app to not load properly.
i added outbound NAT from This Firewall out of 10.99.80.2 and that resolved the issue with RADIUS however it introduced a new issue.
my backup firewall is unable to ping anything on the internet and is showing thatthe gateway is down.
when i force HA to activate, I am able to ping out from the second router without any issues but i am unable to ping out from my primary router. is this expected behavior?
-
@se_marc said in Re: Issue with multi wan high availability setup - authenticating with radius -- WAY MORE INFO:
i added outbound NAT from This Firewall out of 10.99.80.2 and that resolved the issue with RADIUS however it introduced a new issue.
my backup firewall is unable to ping anything on the internet and is showing thatthe gateway is down.Presumably the NAT rule was synced to the secondary and now it uses the VIP for outbound connection, but this is occupied by the primary which is master.
Basically you should use the WAN address for outbound form pfSense itself at both nodes.
Doesn't it work with this?What I don't understand is the private WAN subnet. How gets this translated to the public IP?
-
@viragomann when i use the WAN addresses, radius doesn't authenticate. Sadly the public IPs are NATed by the ISP. I've been going back and forth with them to juts provide the publics directly to me but they don't have the proper equipment at this building and they don't want to pay to install the proper equipment.
-
@se_marc
Hmmm. Maybe it works, when you translate Radius packets only to the VIP.
I think of having two rules for pfSense itself. One for common upstream traffic, translating the packets source to WAN address and an additional one which translates Radius packets only to the VIP.
The later one has to placed to the top.So external Radius authentication will work on the master only naturally.
-
@viragomann i setup outbound NAT, source "This firewall", destination "IP of the radius server", NAT address "CARP WAN IP". when i go into diagnostic to test radius auth, it does not authenticate.