Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway

chpalmer · Apr 7, 2020, 9:35 PM

Nope.. I sure don't believe ya to be dumb! It sometimes can take me a time or three reading to actually see what is printed there. So if I miss something I apologize.

That particular modem is as you mention a "Gateway" device. When you put it in bridge mode Im curious (and Im trying to research this) whether or not the last three interfaces are actually usable. You may be able to see the GUI of the modem (~~192.168.0.1~~) but nothing else. This is a guess right now. A quick call to your ISP may help with that question. edit- doc says now you access it via 192.168.100.1

Your modem is a Hitron. I despise Hitron.. But I digress. (The case is a bit flammable so don't keep any burning objects to close to it.)

I will keep looking but you might test my theory above. I will come back later whether or not I find anything.

chpalmer · Apr 7, 2020, 9:44 PM

You are not a Virgin Media customer are you? Im posting some of this for my own sanity as well. :)

https://community.virginmedia.com/t5/QuickStart-set-up-and/VMB-Hitron-CGNV4-Router-Configuration-Questions-Business/td-p/3899707

https://www.cableforum.uk/board/showpost.php?s=526647350ceff28c730808dc1e61b7c1&p=35860576&postcount=15

raviktiwari · Apr 7, 2020, 9:55 PM

@chpalmer Thanks again for your time and effort. Its good to know that I am not being considered a dumb in a forum at Netgate - that's a promotion for me. :-)

Yes it is Hitron and ISP is running it in Bridge mode ever since I asked for additional IP address. And yes, when I login to 192.168.0.1, I can only see the GUI of the modem but nothing else. There are few basic settings that I can change, but they are useless and not worth taking risk.

There is no point in calling my ISP because the tech support sitting there cannot even tell the full form of IP, let alone do any troubleshooting or support. At best they can restart my router - which most of the time, I have to do it myself.

Having said that, I can confirm you if I take out the same cable which is coming from 2nd port of Hitron model to my Test pfSense (converted Laptop) and connect it to any networking device including my own laptop and enter all the static IP details (IP, subnet mask, GW etc.) it works. I even tried it during my initial setup where I wanted the SOHO network to be different than my firewall network and so I used 2 static IP (right now everything is in one line - serial connection) and it worked like a charm.

Coming to useable IPs, I have been allocated 8 IPs, 1st is Network IP, 2nd if Gateway IP, then I have 5 useable IP and last one 8th IP is broadcast IP.

Out of 5 useable IP that has been assigned to me, I have reserved 2 IPs for different purpose, so I am left with 3 now - one of which is being used in Prod environmnt and 2nd I am trying to use for Test and 3rd will be lying around untill I find another interesting project.

Hope this helps.

And last but not the least, I was browsing thru your interesting website and I must acknowledge that my entire setup (modem, router firewall, laptop, desktop and everything else ) is on a a table which is just above one of the radiator used for central heating. :-(

Once all these connectivity issues are sorted, I have a plan to move everything downstairs - I am trying to reclaim a small section in a room with no central heating (generally used for hoarding kids toys and a small gym) as my personal baby datacenter. But at the moment wife and kids are blocking my plans and I am figting my case on humanitarisn grounds under EU law. Let's see who wins and when. :-)

Many Thx: Rav

raviktiwari · Apr 7, 2020, 10:02 PM

@chpalmer Yes I am Virgin Media customer... going thru both the links now. I have provided more details in my previous response.

WIll get back to you again - soon.

Many Thx: Rav

kiokoman · Apr 7, 2020, 10:14 PM

@raviktiwari said in Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway:

I can access WebGI on 192.168.1.1
I can go to package manager and download and install any package that I want to

so the wan is working if pfsense is able to download stuff from package manager, you can connect to the gui so lan to pfsense work also
did you try packet capture to see if pings are going out of the wan interface ? traceroute ? did you disable all the varius checksum offloading ?
can you ping from pfsense itself ? dignostic / ping 8.8.8.8?
are you using dns forwarder or resolver ?

raviktiwari · Apr 7, 2020, 10:25 PM

@chpalmer Refering to the 1st link, yes that is exactly my situation and like that guy, even I went through a very painful process of trying to understand how it works, configure it, reconfigure it and finally get it working. Tech Support including their corporate team is useless - actually useless is an understatement. I am using them only because they give 350 Mbps D/L line (on paper) and I get to use around 100 MB line and around 10-12 Mbps U/L (which is fastest in the UK).

Luckiliy and thankgot, I have gone past that hurdle and situation. And now my latest issue is: getting internet on LAN of my 2nd pfsense (converted laptop), which shows I have internet access, but I don't have it.

I went thru this post, which is very close to my situation:
https://forum.netgate.com/topic/90764/2x-pfsense-routers-1x-isp

But this guy has PPoE connection and as far as I understand, because PPoE uses login details, there can be one primary connection blocking the 2nd connection. But that should not be the case for me. What do you think?

Many Thanks: Rav

raviktiwari · Apr 7, 2020, 10:40 PM

@kiokoman Thanks for your time and effort.

Even I think the same - I think internet is available on both WAN and LAN and that is why I am more concerned and confused as to what is stopping me to use google or youtube from my laptop?

Just tried Diagnostic->Traceroutre (google) no changes made in the form and this is what I got

x.x.x.x (My Gateway IP) 1.330 ms 0.884 ms 1.038 ms
2 * * *
3 62.253.138.245 15.568 ms 13.780 ms 15.081 ms
4 * * *
5 62.252.192.246 23.822 ms 22.713 ms 23.017 ms
6 74.125.146.216 22.237 ms
212.250.14.162 24.249 ms
74.125.146.216 22.548 ms
7 * * *
8 216.239.57.120 24.950 ms
74.125.242.97 26.152 ms
172.253.71.188 22.858 ms
9 108.170.232.103 21.983 ms
108.170.232.105 21.803 ms
74.125.242.82 24.109 ms
10 216.58.210.46 19.902 ms 22.849 ms 22.047 ms

Hope this helps.

And as far as Packet capture is concernd, this comes down to me being a novice - but this is what I got when I used pfsense inbuilt tool:

23:37:56.924184 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 104
23:37:56.924214 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 104
23:37:56.924224 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 92
23:37:56.924235 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924237 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924239 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924979 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.924984 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925167 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925172 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925175 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1388
23:37:56.925414 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 970
23:37:56.925417 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1126
23:37:56.927044 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.927055 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.927060 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.967808 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:57.048923 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:57.048938 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:57.048948 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:57.048950 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:57.049121 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 42
23:37:57.056790 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 476
23:37:57.058675 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.049693 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 76
23:37:58.049714 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 116
23:37:58.049724 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:58.049732 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:58.049742 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049744 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049746 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049748 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.050028 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 55
23:37:58.058301 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 478
23:37:58.060702 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.139611 IP 192.168.1.12.53885 > 216.58.210.206.443: tcp 0
23:37:58.159598 IP 216.58.210.206.443 > 192.168.1.12.53885: tcp 0
23:37:58.178391 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 474
23:37:58.228330 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.240200 IP 192.168.1.12.51729 > 192.168.1.1.53: UDP, length 32
23:37:58.242571 IP 192.168.1.12.53889 > 216.58.204.35.443: tcp 0
23:37:58.268541 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:58.271821 IP 192.168.1.12.51729 > 192.168.1.1.53: UDP, length 32
23:37:58.284697 IP 192.168.1.12.53886 > 216.58.210.206.443: tcp 0
23:37:58.304749 IP 216.58.210.206.443 > 192.168.1.12.53886: tcp 0
23:37:58.308898 IP 192.168.1.1.53 > 192.168.1.12.51729: UDP, length 48
23:37:58.308938 IP 192.168.1.1.53 > 192.168.1.12.51729: UDP, length 48
23:37:58.311446 IP 192.168.1.12.53890 > 216.58.204.36.443: tcp 0
23:37:58.330730 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:58.515455 IP 192.168.1.12.53891 > 216.58.204.36.443: tcp 0
23:37:58.533651 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:58.577459 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:58.631615 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:58.838647 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:59.031587 IP 192.168.1.12.53892 > 172.217.169.74.443: tcp 0
23:37:59.052740 IP 172.217.169.74.443 > 192.168.1.12.53892: tcp 0
23:37:59.054462 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:59.054482 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:59.054493 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:59.054495 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:59.054734 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 42
23:37:59.062113 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 475
23:37:59.063967 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:59.243221 IP 192.168.1.12.53889 > 216.58.204.35.443: tcp 0
23:37:59.264690 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:59.312966 IP 192.168.1.12.53890 > 216.58.204.36.443: tcp 0
23:37:59.312978 IP 192.168.1.12.57407 > 192.168.1.1.53: UDP, length 29
23:37:59.314467 IP 192.168.1.12.49189 > 192.168.1.1.53: UDP, length 31
23:37:59.331652 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:59.342593 IP 192.168.1.12.61219 > 192.168.1.1.53: UDP, length 50
23:37:59.342652 IP 192.168.1.1.53 > 192.168.1.12.61219: UDP, length 95
23:37:59.344967 IP 192.168.1.12.53893 > 173.194.183.170.443: tcp 0
23:37:59.344989 IP 192.168.1.12.49189 > 192.168.1.1.53: UDP, length 31
23:37:59.344999 IP 192.168.1.12.57407 > 192.168.1.1.53: UDP, length 29
23:37:59.357706 IP 172.217.169.74.443 > 192.168.1.12.53892: tcp 0
23:37:59.365065 IP 173.194.183.170.443 > 192.168.1.12.53893: tcp 0
23:37:59.379039 IP 192.168.1.1.53 > 192.168.1.12.57407: UDP, length 45
23:37:59.381342 IP 192.168.1.12.53894 > 216.58.210.54.443: tcp 0
23:37:59.416154 IP 216.58.210.54.443 > 192.168.1.12.53894: tcp 0
23:37:59.422960 IP 192.168.1.1.53 > 192.168.1.12.49189: UDP, length 92
23:37:59.425720 IP 192.168.1.12.53895 > 216.58.205.33.443: tcp 0
23:37:59.459921 IP 216.58.205.33.443 > 192.168.1.12.53895: tcp 0
23:37:59.464776 IP 192.168.1.1.53 > 192.168.1.12.49189: UDP, length 92
23:37:59.465885 IP 192.168.1.1.53 > 192.168.1.12.57407: UDP, length 45
23:37:59.516224 IP 192.168.1.12.53891 > 216.58.204.36.443: tcp 0
23:37:59.535644 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:59.567976 IP 192.168.1.12.53896 > 216.58.210.54.443: tcp 0
23:37:59.569846 IP 192.168.1.12.53897 > 216.58.205.33.443: tcp 0
23:37:59.591774 IP 216.58.210.54.443 > 192.168.1.12.53896: tcp 0
23:37:59.593616 IP 216.58.205.33.443 > 192.168.1.12.53897: tcp 0
23:37:59.598347 IP 192.168.1.12.53898 > 173.194.183.170.443: tcp 0
23:37:59.618867 IP 173.194.183.170.443 > 192.168.1.12.53898: tcp 0
23:37:59.624472 IP 192.168.1.12.62206 > 192.168.1.1.53: UDP, length 33
23:37:59.655599 IP 192.168.1.12.62206 > 192.168.1.1.53: UDP, length 33
23:37:59.655721 IP 192.168.1.1.53 > 192.168.1.12.62206: UDP, length 259
23:37:59.658223 IP 192.168.1.12.53899 > 172.217.20.142.443: tcp 0
23:37:59.669700 IP 173.194.183.170.443 > 192.168.1.12.53893: tcp 0
23:37:59.677810 IP 172.217.20.142.443 > 192.168.1.12.53899: tcp 0
23:37:59.718731 IP 216.58.210.54.443 > 192.168.1.12.53894: tcp 0
23:37:59.759687 IP 216.58.205.33.443 > 192.168.1.12.53895: tcp 0

Hope this helps.

Many Thx: Rav

kiokoman · Apr 8, 2020, 8:01 AM

@raviktiwari said in Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway:

216.58.210.46 google.com

try to change from dns resolver to dns forwarder or set the option "Enable Forwarding Mode" under dns resolver,
go to system / general setup and add DNS server 8.8.8.8 , try to surf

raviktiwari · Apr 8, 2020, 11:53 AM

@kiokoman Thanks for the suggestion.

Tried both options and also played with few checkboxes, but still no cigar. Still unable to get to the outer world. :-(

Many Thx: Rav

kiokoman · Apr 8, 2020, 1:32 PM

from lan can you ping 8.8.8.8 ?
if you try to open
http://216.58.210.46
do you see google?

raviktiwari · Apr 8, 2020, 1:32 PM

@kiokoman Thanks again for gettig back to me.

Unable to ping 8.8.8.8 - Request Tine Out and unable to open the webiste - "The site can't be reached".

Just so you know, I have removed ISP provided 2 DNS servers and using only 1 DNS server 8.8.8.8 and DNS resolveer is being used with "Enable Forwarding Mode".

And lastly, I am suing latest pfSense release - 2.4.5

In case you need more details, please do not hesitate to ask for it.

Many Thanks for your time, support and patience - Rav

kiokoman · Apr 8, 2020, 1:39 PM

nothing in the firewall logs?

raviktiwari · Apr 8, 2020, 2:12 PM

@kiokoman Thanks again for getting back to me.

Earlier everything was failing in the firewall log, right now things are looking better but still internet not working.

My 2 pence guess is, it has got something to do with the rules, somethng is getting blocked somewhere -and we dont know why and how.

So PFA the screenshots of FW logs as well as my WAN and LAN rules. As per the good practice guide, I have obfuscated my public IP. So where you see something wiped off - that is my public IP, configured on the WAN port.

Hope this helps.

In case you need more info, pls do let me know.

Many Thx: Rav

login-to-view login-to-view login-to-view

raviktiwari · Apr 8, 2020, 2:38 PM

@kiokoman Just so you know, I have also disabled my AVG Antivirus and Windows firewall on thelaptop - just in case that is blocking me out.

However, my Production Setup does not need me to do that... so eventually I will have to enable my antivirus and firewall on laptop/desktop.

Many Thx: Rav

kiokoman · Apr 8, 2020, 2:25 PM

you have tcp:s going out, i suspect a routing problem or asymmetric somewhere. or the usb network card is not doing the job.
if you invert the wan cable between the production pfsense and testlab pfsense? production stop to go to internet and testlab start to work maybe?

raviktiwari · Apr 8, 2020, 2:50 PM

@kiokoman @chpalmer I was looking at the recent posts in the forum and it looks like many people are struggling with access issues specially with 2.4.5

VLANs work, but not LAN what am i doing wrong
Can't update packages on 2.4.5
list itempfSense v2.4.5 not able to resolve Domain Overrides against itself
list itemWAN Going Down and Some Errors

Not sure if there is a more widespred bug impacting connectivity in general. Any thoughts?

I am planning to uninstall and reinstall my Test pfSense and start all over again using 2.4.4 to see if the issue persists - may be that way we can narrow down something.

Many Thanks: Rav

raviktiwari · Apr 8, 2020, 2:56 PM

@kiokoman I can swap LAN and WAN within the TEST environment but not with Production because it is already in production and I wil have to change everything setting wise right from laptop to router to HAProxy, VPN and so on.

I have already tried swapping LAN and WAN on the laptop and I still had the same issue. Earlier LAN was WAN and vice-versa. Current configuration (USB as LAN and inbuilt port as WAN) is the new setup.

I also tried enabling wirelss card and configured it as OPT1, even on that I could not access internet.

I am therefore thinking to uninstall this and reinstall 2.4.4 and strat from the scratch.

Even I think there is some routing issue with latest 2.4.5 and I will be really scared if I am right as my Prod is running on 2.4.5 and so far - there is no issue.

I will report back shortly. Give me an hour or so.

Many Thx: Rav

chpalmer · Apr 8, 2020, 2:57 PM

All my 2.4.5 installs are working just fine.

Are you able to test out a 2.5 snapshot?

kiokoman · Apr 8, 2020, 2:59 PM

also mine, 2.4.5 with 8 ip is working fine.

raviktiwari · Apr 8, 2020, 3:55 PM

@chpalmer how do I do that?