Send Traffic from 1 host to a specific GW

[johnpoz]

@smalldragoon you can policy route sure.. If you have multiple gateways you can route traffic via a firewall rule out a specific gateway.

Keep in mind if not working - that if the client already had a state created out the other gateway, traffic that matches that state would continue to use the old gateway.

Yes there is little different between policy route traffic out a vpn connection or via a normal gateway.

Other things that could cause it not to be working, is your alias is not resolving to the IP of the client you want to send out the specific gateway.. You can look in your tables to see that pfsense has resolved that alias to the correct IP.

Another issue is the order of the rules you have on the lan interface... Rules are evaluated top down, first rule to trigger wins, no other rules are evaluated.. So if your policy route rule is below say the default any any rule, then no your specific IP would not be routed via your policy route rule.

[smalldragoon]

@johnpoz Hi, Thanks for your answer .
First answering points you raised I already had :

Order of the rules : my routing rule is above the LAN - any - any , so I assume it is ok

Alias IP : I put an alias to make screenshots easier , but I made my tests with real IP address

State already created : as I was running out of clues, I even fully rebooted the machine to be sure that all connections / routes / states ... were reset

To be sure, I double checked again all these options ... no changes :(

Now, could you maybe give more insights on the differences between routing traffic through a gw and a VPN gw ? do you see anything I'm missing through my screenshot of my initial post ?

Thanks !

Rules :

[SteveITS]

@smalldragoon Your screen cap shows states implying it's working. Could the gateway be down? See this page.

[smalldragoon]

@steveits Hi,
Well, just rechecked as well, GW is up :
rules :

Gateway status :

maybe it is here than I'm missing something ?

[Edit] - add on to post : doing a tracert make me go through the right GW

But no other traffic seems to work : DNS, HTTP ... a basic nslookup leads to timeout

[johnpoz]

@smalldragoon said in Send Traffic from 1 host to a specific GW:

But no other traffic seems to work : DNS

where is your client pointing to for dns, if pfsense. How would it talk to say pfsense lan IP if your forcing its traffic out the orange3g gateway.

How are you trying to get it to open something http if it can not resolve that name, are you trying to access something http://ipaddress

edit: also is your pfsense lan also 192.168.1.1 ? What is your lan IP? 192.168.1.1 is default pfsense lan IP.

If your lan IP network is overlapping with the IP network your getting from this 3g connection via dhcp your going to have a bad day ;)

Also when you policy route out a gateway, you need to make sure that stuff you might not want to go out that gateway (like dns) is allowed via rules above it.. See the bypass policy route docs

here
https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html#bypassing-policy-routing

[smalldragoon]

@johnpoz I configured DNS for gogole one : 8.8.8.8.
I tried http using ip addresses and not hostnames, as the DNS is not working either.
There is no overlaping on networks if I'm not wrong and rules are set as highest possible , means it should work like that ( cf my previous post with the rules screenshot ? )
Again, I guess I'm missing something somewhere, so I'm open to any suggestion where I can have made something stupid

regarding my config , here is a more detailed schema :

[johnpoz]

@smalldragoon

Could you post up your outbound nat rules please. If your lan network is not overlapping and policy route source is resolving to your test machine.. Then it should be working.

You sure your dns/http traffic is not being blocked upstream from pfsense. When you try and go to http://ipaddress do you see state in the state table.

Do you see state for the dns?

Also simple sniff on orange3g interface while you try and do stuff from your client would show if traffic is actually being sent to your 192.168.1.1 device..

BTW when you say wifi net - that is not a wifi interface in pfsense is? You just get internet via wifi, and the connection from the orange box to pfsense is actually a wire?

edit: Do you have any rules on your floating tab?

[smalldragoon]

@johnpoz
Here are the answer
Outbound Rules NAT

I set all the rules to any any accept to avoid any blocking :

3G - Orange

Main WAN

sniffing could be done, but quite complicated. anyway, I have a "traffic counter" on the Orange 3G modem, which shows that there is nothing goign through really.

Last point on wifi : my hardware has a wifi card , that I have assigned as one interface of PFsense. This connection is a 2nd one for 1- Backup puposes in case of ,even with a low bandwith , 2 - When required, have a different public IP address for tests purposes. the connection between pfsense and the 3GModem is wifi only, no modem.

floating tab rules :

none defined

[smalldragoon]

@smalldragoon [Edit]
Reading again my post, I missed one answer :
If I configure DNS as being pfsense, I get resolution working ( of course )

[johnpoz]

@smalldragoon those outbound rules look ODD for hybrid..

Where are the auto rules?

example here is mine where I can policy route traffic out different gateway (vpn)

sniffing could be done, but quite complicated

How is that? Diagnostic menu, packet capture - pick the interface, hit start..

the connection between pfsense and the 3GModem is wifi only

Well that could be a problem.. Is there no way to wire this.. How do you not the issue is not the wifi connection..

[smalldragoon]

@johnpoz
Full screen then, with auto at the bottom

regarding packet capture, I was old school, was thinking to put a host on switch, copy traffic with tcpdump. Was not aware of the feature :( .. my bad sorry :

here is the logs, we can see only the regular "live check" on 8.8.4.4

14:54:30.989052 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51214, length 9
14:54:31.032082 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51214, length 9
14:54:31.114563 IP 192.168.1.21.40178 > 5.62.53.53.443: tcp 72
14:54:31.491202 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51215, length 9
14:54:31.526845 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51215, length 9
14:54:31.716827 IP 192.168.1.21.40178 > 5.62.53.53.443: tcp 72
14:54:31.870853 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 517
14:54:32.013028 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51216, length 9
14:54:32.042089 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51216, length 9
14:54:32.090728 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 0
14:54:32.122245 IP 5.62.40.127.443 > 192.168.1.21.38116: tcp 0
14:54:32.122787 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 0
14:54:32.123818 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:32.372577 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:32.525046 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51217, length 9
14:54:32.551868 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51217, length 9
14:54:32.673729 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:32.921461 IP 192.168.1.21.40178 > 5.62.53.53.443: tcp 72
14:54:33.037036 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51218, length 9
14:54:33.080653 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51218, length 9
14:54:33.276159 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:33.549048 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51219, length 9
14:54:33.590496 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51219, length 9
14:54:33.759219 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.792240 IP 34.107.221.82.80 > 192.168.1.21.14704: tcp 0
14:54:33.792787 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.794986 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 322
14:54:33.842543 IP 34.107.221.82.80 > 192.168.1.21.14704: tcp 243
14:54:33.843571 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.846300 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.847262 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:34.061075 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51220, length 9
14:54:34.100409 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51220, length 9
14:54:34.480645 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:34.573057 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51221, length 9
14:54:34.602147 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51221, length 9
14:54:34.750538 IP 192.168.1.21.11093 > 5.62.53.53.443: tcp 44
14:54:35.085061 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51222, length 9
14:54:35.111597 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51222, length 9
14:54:35.330553 IP 192.168.1.21.40178 > 5.62.53.53.443: tcp 72
14:54:35.597055 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51223, length 9
14:54:35.633887 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51223, length 9
14:54:36.109094 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51224, length 9
14:54:36.142105 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51224, length 9
14:54:36.616010 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51225, length 9
14:54:36.652550 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51225, length 9
14:54:36.673398 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 517
14:54:36.870766 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 0
14:54:36.889724 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:36.902138 IP 34.107.221.82.80 > 192.168.1.21.24556: tcp 0
14:54:36.902670 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 0
14:54:36.904829 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 322
14:54:36.935144 IP 34.107.221.82.80 > 192.168.1.21.24556: tcp 243
14:54:36.936140 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 0
14:54:36.938972 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 0
14:54:36.939811 IP 192.168.1.21.24556 > 34.107.221.82.80: tcp 0
14:54:37.074848 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:37.133052 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51226, length 9
14:54:37.162212 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51226, length 9
14:54:37.645096 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51227, length 9
14:54:37.672103 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51227, length 9
14:54:38.078553 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:38.157528 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51228, length 9
14:54:38.192096 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51228, length 9
14:54:38.669091 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51229, length 9
14:54:38.701893 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51229, length 9
14:54:39.082263 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:39.199200 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51230, length 9
14:54:39.242104 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51230, length 9
14:54:39.705194 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51231, length 9
14:54:39.742093 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51231, length 9
14:54:39.946235 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 0
14:54:39.972169 IP 34.107.221.82.80 > 192.168.1.21.28540: tcp 0
14:54:39.972626 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 0
14:54:39.973992 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 322
14:54:40.012199 IP 34.107.221.82.80 > 192.168.1.21.28540: tcp 243
14:54:40.012917 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 0
14:54:40.014800 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 0
14:54:40.015335 IP 192.168.1.21.28540 > 34.107.221.82.80: tcp 0
14:54:40.095186 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:40.142112 IP 192.168.1.21.40178 > 5.62.53.53.443: tcp 72
14:54:40.207717 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51232, length 9
14:54:40.242119 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51232, length 9
14:54:40.708015 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51233, length 9
14:54:40.718872 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 0
14:54:40.741284 IP 172.217.18.196.443 > 192.168.1.21.17873: tcp 0
14:54:40.741866 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 0
14:54:40.744162 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 517
14:54:40.752182 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51233, length 9
14:54:41.001739 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 517
14:54:41.095130 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:41.229094 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51234, length 9
14:54:41.262195 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51234, length 9
14:54:41.314075 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 517
14:54:41.704021 IP 192.168.1.21.38116 > 5.62.40.127.443: tcp 194
14:54:41.741109 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51235, length 9
14:54:41.772097 IP 8.8.4.4 > 192.168.1.21: ICMP echo reply, id 30718, seq 51235, length 9
14:54:41.923129 IP 192.168.1.21.17873 > 172.217.18.196.443: tcp 517
14:54:42.110939 IP 192.168.1.21.55915 > 99.84.5.21.443: tcp 1
14:54:42.253102 IP 192.168.1.21 > 8.8.4.4: ICMP echo request, id 30718, seq 51236, length 9

[johnpoz]

@smalldragoon said in Send Traffic from 1 host to a specific GW:

14:54:33.843571 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.846300 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0
14:54:33.847262 IP 192.168.1.21.14704 > 34.107.221.82.80: tcp 0

Looks like more than the live checks to me.. I just don't see any answers.. So pfsense sends traffic, if there is not response - there is nothing pfsense can do about that.

14:54:33.842543 IP 34.107.221.82.80 > 192.168.1.21.14704: tcp 243

There is some sort of reponse - was that a RST.. You can download the pcap into say wireshark and get more easy to view with details.

And that outbound nat is messed up... Delete all those "mappings" that already exist in auto.. Why would you put those in?? And since you have auto outbound for your 3g, there is no use of any hybrid mappings.

[smalldragoon]

@johnpoz
Cleaned all Outbound rules. I did not created anything, it was automatically generated.
I simplified my architecture , it now fully reflect the schema I posted before ( LAN is and only is 172.16.25.0/24 )
So, here is my outbound rules set now :

Lan Rules:

[johnpoz]

@smalldragoon well clearly there is listed at 14 different states being sent out that policy rule.. So pfsense is routing traffic like you told it too.

States are being created.. And from your sniff traffic is being sent out..

And again with your outbound nat being created auto, there is no need for hybrid.. Hybrid are used when doing say vpn.. And no gateway setup in the actual interface that tells pfsense is a "wan" connection and therefore create the outbound nats.

[smalldragoon]

@johnpoz said in Send Traffic from 1 host to a specific GW:

And again with your outbound nat being created auto, there is no need for hybrid.. Hybrid are used when doing say vpn.. And no gateway setup in the actual interface that tells pfsense is a "wan" connection and therefore create the outbound nats.

sorry, my bad english . you mean then I can stick back to auto and remove the 1st mapping I did , like this ?

?

[johnpoz]

@smalldragoon Yeah see in your auto, where it lists your 3g interface.. So there is no need for hybrid mappings.

So what does your state table show for that interface.. From your latest rules posting - it showed 14 states and and almost 4MB of traffic..

From your sniff - clearly traffic was being sent out that gateway from the .21 address.. And there were some responses.. But maybe they were RST, seems like maybe some retrans because of no answers going on, etc.

You can do your sniff, exclude the icmp stuff - set the count to more than 100... Do some testing of connections, dns etc. And then download the pcap and view with say wireshark to get a better understanding of what is going on via the sniff. But from what have shown traffic is flowing out the gateway..

[smalldragoon]

@johnpoz
Ok, thanks, will do , and keep you posted

[SteveITS]

@smalldragoon said in Send Traffic from 1 host to a specific GW:

I set all the rules to any any accept to avoid any blocking :
3G - Orange

Orange is an Internet connection? You probably don't want to allow all inbound traffic on it, which would allow the Internet to log in to pfSense.

[johnpoz]

@steveits While agree with your for sure - he is clearly behind a nat with that 192.168.1.21 address - so unless they are port forwarding to him, or he has something setup on the isp device I don't see how any inbound could come in unsolicited.

[SteveITS]

@johnpoz said in Send Traffic from 1 host to a specific GW:

behind a nat with that 192.168.1.21

I missed that, never mind. Unless it's set as a DMZ.
The bigger picture point is probably that an inbound rule on an interface won't affect outbound traffic.