Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.3.2\. Multiwan: Cannot access some web-sites

    Scheduled Pinned Locked Moved Routing and Multi WAN
    2 Posts 1 Posters 849 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      hikmat
      last edited by

      Hi all,

      Pfsense 2.3.2 (updated from 2.2.4) on Dell PE1800 server with 4 Eth cards. One of them for Internet access.

      Intel em0 with 5 Vlans (3 uplink ISP - Uznet 213+Uznet27+SITA151 and 2 subnets Uznet9 behind Uznet213 + SITA155 behind SITA151)

      Have to use all 3 lines because Uznet27 has 30Mbps link, Uznet 213 only 2Mbps world + 10Mbps domestic, and SITA151 only 2Mbps, but without restrictions (Uznet ISP blocks some web-sites and services we need)

      For those blocked sites I use IP aliaces, then add alias in static routing with gateway SITA151_GW. Also point sites in DNS Resolver so every time users get same IP and last in LAN rules add special one (source LAN net, destination SITA_Web, gateway SITA151_GW) at the top.

      762/530.38 MiB 	IPv4 * 	DMZ net 	* 	SITA WEB 	* 	SITA151GW 	none 	  	ALLOW blocked sites to SITA

      So far this works like a charm, but Last 2-3 days cannot open 2 web-sites:
      1. ng.ru (188.40.89.58) full blocked by UzNET
      2. inosmi.ru (178.248.232.60) partially blocked by UzNET

      When try access to ng.ru get 403 forbidden
      When try access to inosmi.ru even over Uznet some time it works, but mostly nothing displayed

      traceroute from any user PC show that SITA151_GW is used.
      Diagnostics/States indicate strange behavior: some time  UzNET27_GW is used when accessing inosmi.ru.

      At the same time, if put SITA151 IP and SITA151_GW on laptop and connect right to CISCO router, can access both sites. Also sites opens freely with SITA155 subnet IP (routed by pfsense to SITA151).

      What is the problem?

      More info
      states for ng.ru (403 forbidden)

      
LAN1    tcp    192.168.1.164:30286 -> 188.40.89.58:80    ESTABLISHED:ESTABLISHED    20 / 13    6 KiB / 3 KiB    
SITA151    tcp    x.x.151.183:28775 (192.168.1.164:30286) -> 188.40.89.58:80    ESTABLISHED:ESTABLISHED    20 / 13    6 KiB / 3 KiB   
```   
**states for inosmi.ru**

      LAN1    tcp    192.168.1.110:54401 -> 178.248.232.60:80    CLOSED:SYN_SENT    3 / 0    152 B / 0 B   
      SITA151    tcp    x.x.151.183:45974 (192.168.1.110:54401) -> 178.248.232.60:80    SYN_SENT:CLOSED    3 / 0    152 B / 0 B   
      LAN1    tcp    192.168.1.110:54403 -> 178.248.232.60:80    CLOSED:SYN_SENT    3 / 0    152 B / 0 B   
      SITA151    tcp    x.x.151.183:58359 (192.168.1.110:54403) -> 178.248.232.60:80    SYN_SENT:CLOSED    3 / 0    152 B / 0 B   
      LAN1    tcp    192.168.1.110:54407 -> 178.248.232.60:80    CLOSED:SYN_SENT    3 / 0    152 B / 0 B   
      SITA151    tcp    x.x.151.183:22841 (192.168.1.110:54407) -> 178.248.232.60:80    SYN_SENT:CLOSED    3 / 0    152 B / 0 B

      
**packet capture on LAN for ng.ru**

      11:26:25.037718 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 5623, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30239 > 188.40.89.58.80: Flags , cksum 0x924c (correct), seq 4291653341, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:26:25.051331 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 5628, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30241 > 188.40.89.58.80: Flags , cksum 0x801e (correct), seq 1736776018, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:26:25.169843 00:0a:5e:08:71:06 > 98:90:96:ad:d0:ae, ethertype IPv4 (0x0800), length 66: (tos 0x40, ttl 54, id 9662, offset 0, flags , proto TCP (6), length 52)
          188.40.89.58.80 > 192.168.1.164.30239: Flags [S.], cksum 0xd98d (correct), seq 239702357, ack 4291653342, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0

      11:26:25.170286 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 128, id 5632, offset 0, flags , proto TCP (6), length 40)
          192.168.1.164.30239 > 188.40.89.58.80: Flags [.], cksum 0x1341 (correct), seq 1, ack 1, win 16425, length 0

      11:26:25.171017 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 5633, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30243 > 188.40.89.58.80: Flags , cksum 0x18ad (correct), seq 2030989112, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:26:25.183831 00:0a:5e:08:71:06 > 98:90:96:ad:d0:ae, ethertype IPv4 (0x0800), length 66: (tos 0x40, ttl 54, id 51180, offset 0, flags , proto TCP (6), length 52)
          188.40.89.58.80 > 192.168.1.164.30241: Flags [S.], cksum 0xfd3a (correct), seq 3001530075, ack 1736776019, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0

      11:26:25.184035 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 128, id 5634, offset 0, flags , proto TCP (6), length 40)
          192.168.1.164.30241 > 188.40.89.58.80: Flags [.], cksum 0x36ee (correct), seq 1, ack 1, win 16425, length 0

      11:26:25.185792 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 525: (tos 0x0, ttl 128, id 5635, offset 0, flags , proto TCP (6), length 511)
          192.168.1.164.30241 > 188.40.89.58.80: Flags [P.], cksum 0xd7e2 (correct), seq 1:472, ack 1, win 16425, length 471

      11:26:25.311653 00:0a:5e:08:71:06 > 98:90:96:ad:d0:ae, ethertype IPv4 (0x0800), length 66: (tos 0x40, ttl 54, id 13342, offset 0, flags , proto TCP (6), length 52)
          188.40.89.58.80 > 192.168.1.164.30243: Flags [S.], cksum 0x5fa0 (correct), seq 3700867925, ack 2030989113, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0

      11:26:25.311907 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 128, id 5636, offset 0, flags , proto TCP (6), length 40)
          192.168.1.164.30243 > 188.40.89.58.80: Flags [.], cksum 0x9953 (correct), seq 1, ack 1, win 16425, length 0

      11:26:25.325500 00:0a:5e:08:71:06 > 98:90:96:ad:d0:ae, ethertype IPv4 (0x0800), length 54: (tos 0x40, ttl 54, id 24132, offset 0, flags , proto TCP (6), length 40)
          188.40.89.58.80 > 192.168.1.164.30241: Flags [.], cksum 0x7521 (correct), seq 1, ack 472, win 31, length 0

      11:26:25.326994 00:0a:5e:08:71:06 > 98:90:96:ad:d0:ae, ethertype IPv4 (0x0800), length 380: (tos 0x40, ttl 54, id 12721, offset 0, flags , proto TCP (6), length 366)
          188.40.89.58.80 > 192.168.1.164.30241: Flags [P.], cksum 0x5418 (correct), seq 1:327, ack 472, win 31, length 326

      11:26:25.527244 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 128, id 5637, offset 0, flags , proto TCP (6), length 40)
          192.168.1.164.30241 > 188.40.89.58.80: Flags [.], cksum 0x3423 (correct), seq 472, ack 327, win 16343, length 0

      
**packet capture on SITA151 for ng.ru**

      11:27:57.056526 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 48868, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.18524 > 188.40.89.58.80: Flags , cksum 0x3acc (correct), seq 451899, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:27:57.188907 f8:72:ea:68:82:00 > 00:15:c5:88:1e:83, ethertype IPv4 (0x0800), length 66: (tos 0x40, ttl 55, id 0, offset 0, flags , proto TCP (6), length 52)
          188.40.89.58.80 > x.x.151.183.18524: Flags [S.], cksum 0xd976 (correct), seq 3924385355, ack 451900, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0

      11:27:57.189427 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 128, id 63634, offset 0, flags , proto TCP (6), length 40)
          x.x.151.183.18524 > 188.40.89.58.80: Flags [.], cksum 0x132a (correct), seq 1, ack 1, win 16425, length 0

      11:27:57.194542 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 551: (tos 0x0, ttl 128, id 53518, offset 0, flags , proto TCP (6), length 537)
          x.x.151.183.18524 > 188.40.89.58.80: Flags [P.], cksum 0x4907 (correct), seq 1:498, ack 1, win 16425, length 497

      11:27:57.331339 f8:72:ea:68:82:00 > 00:15:c5:88:1e:83, ethertype IPv4 (0x0800), length 60: (tos 0x40, ttl 55, id 12592, offset 0, flags , proto TCP (6), length 40)
          188.40.89.58.80 > x.x.151.183.18524: Flags [.], cksum 0x5143 (correct), seq 1, ack 498, win 31, length 0

      11:27:57.334447 f8:72:ea:68:82:00 > 00:15:c5:88:1e:83, ethertype IPv4 (0x0800), length 380: (tos 0x40, ttl 55, id 12593, offset 0, flags , proto TCP (6), length 366)
          188.40.89.58.80 > x.x.151.183.18524: Flags [P.], cksum 0x2e3b (correct), seq 1:327, ack 498, win 31, length 326

      11:27:57.535233 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 128, id 4416, offset 0, flags , proto TCP (6), length 40)
          x.x.151.183.18524 > 188.40.89.58.80: Flags [.], cksum 0x1045 (correct), seq 498, ack 327, win 16343, length 0

      
**packet capture on LAN for inosmi.ru**

      11:33:11.271975 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6261, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30311 > 178.248.232.60.80: Flags , cksum 0x8108 (correct), seq 1594464971, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:11.522529 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6279, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30313 > 178.248.232.60.80: Flags , cksum 0x49d0 (correct), seq 2309924700, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:14.272012 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6306, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30311 > 178.248.232.60.80: Flags , cksum 0x8108 (correct), seq 1594464971, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:14.516047 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6307, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30313 > 178.248.232.60.80: Flags , cksum 0x49d0 (correct), seq 2309924700, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:20.266430 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 6311,
      offset 0, flags , proto TCP (6), length 48)
          192.168.1.164.30311 > 178.248.232.60.80: Flags , cksum 0x9511 (correct), seq 1594464971, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:33:20.516339 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 6312, offset 0, flags , proto TCP (6), length 48)
          192.168.1.164.30313 > 178.248.232.60.80: Flags , cksum 0x5dd9 (correct), seq 2309924700, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:33:32.271890 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6322, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30316 > 178.248.232.60.80: Flags , cksum 0x40da (correct), seq 1316285321, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:32.523542 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6323, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30318 > 178.248.232.60.80: Flags , cksum 0x4f89 (correct), seq 2369101335, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:35.273376 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6327, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30316 > 178.248.232.60.80: Flags , cksum 0x40da (correct), seq 1316285321, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:35.523316 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6328, offset 0, flags , proto TCP (6), length 52)
          192.168.1.164.30318 > 178.248.232.60.80: Flags , cksum 0x4f89 (correct), seq 2369101335, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:33:41.276039 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 6332, offset 0, flags , proto TCP (6), length 48)
          192.168.1.164.30316 > 178.248.232.60.80: Flags , cksum 0x54e3 (correct), seq 1316285321, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:33:41.516843 98:90:96:ad:d0:ae > 00:0a:5e:08:71:06, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 6333, offset 0, flags , proto TCP (6), length 48)
          192.168.1.164.30318 > 178.248.232.60.80: Flags , cksum 0x6392 (correct), seq 2369101335, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      
**packet capture on SITA151 for ng.ru**

      11:36:09.201736 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21188, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.6805 > 178.248.232.60.80: Flags , cksum 0x0956 (correct), seq 4010397602, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:09.453032 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 20063, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.31524 > 178.248.232.60.80: Flags , cksum 0x2c8a (correct), seq 961413531, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:12.208786 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 39327, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.6805 > 178.248.232.60.80: Flags , cksum 0x0956 (correct), seq 4010397602, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:12.458782 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 8321, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.31524 > 178.248.232.60.80: Flags , cksum 0x2c8a (correct), seq 961413531, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:16.625279 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 48509, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.25498 > 178.248.232.60.80: Flags , cksum 0xba85 (correct), seq 2543856855, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:16.875155 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 36204, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.30899 > 178.248.232.60.80: Flags , cksum 0xa9a1 (correct), seq 4276470108, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:18.204186 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 52843, offset 0, flags , proto TCP (6), length 48)
          x.x.151.183.6805 > 178.248.232.60.80: Flags , cksum 0x1d5f (correct), seq 4010397602, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:36:18.453287 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 63430, offset 0, flags , proto TCP (6), length 48)
          x.x.151.183.31524 > 178.248.232.60.80: Flags , cksum 0x4093 (correct), seq 961413531, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:36:19.625366 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 26386, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.25498 > 178.248.232.60.80: Flags , cksum 0xba85 (correct), seq 2543856855, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:19.874360 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 59660, offset 0, flags , proto TCP (6), length 52)
          x.x.151.183.30899 > 178.248.232.60.80: Flags , cksum 0xa9a1 (correct), seq 4276470108, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

      11:36:25.627676 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 50339, offset 0, flags , proto TCP (6), length 48)
          x.x.151.183.25498 > 178.248.232.60.80: Flags , cksum 0xce8e (correct), seq 2543856855, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      11:36:25.874599 00:15:c5:88:1e:83 > f8:72:ea:68:82:00, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 10667, offset 0, flags , proto TCP (6), length 48)
          x.x.151.183.30899 > 178.248.232.60.80: Flags , cksum 0xbdaa (correct), seq 4276470108, win 8192, options [mss 1460,nop,nop,sackOK], length 0

      1 Reply Last reply Reply Quote 0
      • H
        hikmat
        last edited by

        More pics

        DMZ.png_thumb
        float.png
        DMZ.png
        float.png_thumb
        sita151.png
        sita151.png_thumb
        uznet27.png
        uznet27.png_thumb
        uznet213.png
        uznet213.png_thumb
        ng.png
        ng.png_thumb
        inosmi.png
        inosmi.png_thumb
        inosmi_ok_uznet9.png
        inosmi_ok_uznet9.png_thumb

        1 Reply Last reply Reply Quote 0
        • First post
          Last post
        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.