Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway
-
@raviktiwari said in Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway:
My WAN is obviously WAN IP and my LAN is in 192.168.X.x range.
Does that help? Not sure if I wrote something contradictory in my post.
You did not. But I had to put my better glasses on.
How do you connect both routers to your modem? You state fiber optic line and then coax modem. Do you have an ONT device? fiber -- ONT -- router -- pfsense ?
-
@chpalmer Apologies for being dumb - I am a project manager with Sys Admin background and lost touch with technology long time ago and small projects like this helps me keep in touch. But when it comes to networking, I am still a novice - SO, I dont understand what is ONT (I googled it and I dont think I have ONT device.
Having said that, I dont have 2 routers.
For Prod:ISP Modem is connected to pfSense using a LAN cable and a static IP address.
Then from LAN port of pfSense (which is configured at 192.168.10.x running in DHCP mode) a cable is connected to TPLink and then TPLink has been configured to give internet and wifi to SOHO users an IP of 192.168.5.x - this working perfectly fine.
Then from OPT port of pfsense a cable comes out and goes to an unmanaged switch with an IP address of 192.168.11.x and with use of HAProxy, there are 3 servers connected there within same subnet and all my websites on different servers running on same port 80 and using SSL is accessible from outside world - so this is also working perfectly fine.
For Test:
Same ISP Modem is connected to "Laptop converted pfSense" using a LAN cable and different static IP address (from the pool of 3 IP that has been given to me). Both these IPs (Prod and Test) have same subnet mask, same Gateway IP and same Primary and Secondary DNS .Then from LAN port (which is USB ethernet cable) of "Laptop converted pfSense" (configured at 192.168.1.1 - running in DHCP mode) a cable is connected to my laptop where it shows I am connected and have access to internet - but it does not work.
Once I get internet on laptop using direct connection from "Laptop converted pfSense", I will connect this lan cable to another non managed switch from where another test server and a laptop will connected on same subnet (192.168.1.x). I will then use HAProxy to make my test server accessible from outside.
This will help me do all sort of testing on my server as well as pfsense (including IDS/IPS, GeoIP, DNS Blocker and so on) without impacting prod settings and once I am comfortable with the details and configuration, I might apply it on Prod environment.
I have attached a picture of my ISP model as well as my Network diagram. Hope this helps. I am new to this whole architecting thing, so if you see any issue/error please point it out as it will help me learn more.
In case you need more information, pls do let me know and Thanks once again for showing interest in my issue and trying to help me out.
Many Thx: Rav
-
Nope.. I sure don't believe ya to be dumb!
It sometimes can take me a time or three reading to actually see what is printed there. So if I miss something I apologize.That particular modem is as you mention a "Gateway" device. When you put it in bridge mode Im curious (and Im trying to research this) whether or not the last three interfaces are actually usable. You may be able to see the GUI of the modem (
192.168.0.1) but nothing else. This is a guess right now. A quick call to your ISP may help with that question. edit- doc says now you access it via 192.168.100.1Your modem is a Hitron. I despise Hitron.. But I digress. (The case is a bit flammable so don't keep any burning objects to close to it.)
I will keep looking but you might test my theory above. I will come back later whether or not I find anything.
-
You are not a Virgin Media customer are you? Im posting some of this for my own sanity as well. :)
https://community.virginmedia.com/t5/QuickStart-set-up-and/VMB-Hitron-CGNV4-Router-Configuration-Questions-Business/td-p/3899707
https://www.cableforum.uk/board/showpost.php?s=526647350ceff28c730808dc1e61b7c1&p=35860576&postcount=15
-
@chpalmer Thanks again for your time and effort. Its good to know that I am not being considered a dumb in a forum at Netgate - that's a promotion for me. :-)
Yes it is Hitron and ISP is running it in Bridge mode ever since I asked for additional IP address. And yes, when I login to 192.168.0.1, I can only see the GUI of the modem but nothing else. There are few basic settings that I can change, but they are useless and not worth taking risk.
There is no point in calling my ISP because the tech support sitting there cannot even tell the full form of IP, let alone do any troubleshooting or support. At best they can restart my router - which most of the time, I have to do it myself.
Having said that, I can confirm you if I take out the same cable which is coming from 2nd port of Hitron model to my Test pfSense (converted Laptop) and connect it to any networking device including my own laptop and enter all the static IP details (IP, subnet mask, GW etc.) it works. I even tried it during my initial setup where I wanted the SOHO network to be different than my firewall network and so I used 2 static IP (right now everything is in one line - serial connection) and it worked like a charm.
Coming to useable IPs, I have been allocated 8 IPs, 1st is Network IP, 2nd if Gateway IP, then I have 5 useable IP and last one 8th IP is broadcast IP.
Out of 5 useable IP that has been assigned to me, I have reserved 2 IPs for different purpose, so I am left with 3 now - one of which is being used in Prod environmnt and 2nd I am trying to use for Test and 3rd will be lying around untill I find another interesting project.
Hope this helps.
And last but not the least, I was browsing thru your interesting website and I must acknowledge that my entire setup (modem, router firewall, laptop, desktop and everything else ) is on a a table which is just above one of the radiator used for central heating. :-(
Once all these connectivity issues are sorted, I have a plan to move everything downstairs - I am trying to reclaim a small section in a room with no central heating (generally used for hoarding kids toys and a small gym) as my personal baby datacenter. But at the moment wife and kids are blocking my plans and I am figting my case on humanitarisn grounds under EU law. Let's see who wins and when. :-)
Many Thx: Rav
-
@chpalmer Yes I am Virgin Media customer... going thru both the links now. I have provided more details in my previous response.
WIll get back to you again - soon.
Many Thx: Rav
-
@raviktiwari said in Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway:
I can access WebGI on 192.168.1.1
I can go to package manager and download and install any package that I want toso the wan is working if pfsense is able to download stuff from package manager, you can connect to the gui so lan to pfsense work also
did you try packet capture to see if pings are going out of the wan interface ? traceroute ? did you disable all the varius checksum offloading ?
can you ping from pfsense itself ? dignostic / ping 8.8.8.8?
are you using dns forwarder or resolver ? -
@chpalmer Refering to the 1st link, yes that is exactly my situation and like that guy, even I went through a very painful process of trying to understand how it works, configure it, reconfigure it and finally get it working. Tech Support including their corporate team is useless - actually useless is an understatement. I am using them only because they give 350 Mbps D/L line (on paper) and I get to use around 100 MB line and around 10-12 Mbps U/L (which is fastest in the UK).
Luckiliy and thankgot, I have gone past that hurdle and situation. And now my latest issue is: getting internet on LAN of my 2nd pfsense (converted laptop), which shows I have internet access, but I don't have it.
I went thru this post, which is very close to my situation:
https://forum.netgate.com/topic/90764/2x-pfsense-routers-1x-ispBut this guy has PPoE connection and as far as I understand, because PPoE uses login details, there can be one primary connection blocking the 2nd connection. But that should not be the case for me. What do you think?
Many Thanks: Rav
-
@kiokoman Thanks for your time and effort.
Even I think the same - I think internet is available on both WAN and LAN and that is why I am more concerned and confused as to what is stopping me to use google or youtube from my laptop?
Just tried Diagnostic->Traceroutre (google) no changes made in the form and this is what I got
- x.x.x.x (My Gateway IP) 1.330 ms 0.884 ms 1.038 ms
2 * * *
3 62.253.138.245 15.568 ms 13.780 ms 15.081 ms
4 * * *
5 62.252.192.246 23.822 ms 22.713 ms 23.017 ms
6 74.125.146.216 22.237 ms
212.250.14.162 24.249 ms
74.125.146.216 22.548 ms
7 * * *
8 216.239.57.120 24.950 ms
74.125.242.97 26.152 ms
172.253.71.188 22.858 ms
9 108.170.232.103 21.983 ms
108.170.232.105 21.803 ms
74.125.242.82 24.109 ms
10 216.58.210.46 19.902 ms 22.849 ms 22.047 ms
Hope this helps.
And as far as Packet capture is concernd, this comes down to me being a novice - but this is what I got when I used pfsense inbuilt tool:
23:37:56.924184 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 104
23:37:56.924214 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 104
23:37:56.924224 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 92
23:37:56.924235 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924237 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924239 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:56.924979 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.924984 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925167 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925172 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1460
23:37:56.925175 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1388
23:37:56.925414 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 970
23:37:56.925417 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 1126
23:37:56.927044 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.927055 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.927060 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:56.967808 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:57.048923 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:57.048938 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:57.048948 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:57.048950 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:57.049121 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 42
23:37:57.056790 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 476
23:37:57.058675 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.049693 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 76
23:37:58.049714 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 116
23:37:58.049724 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:58.049732 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:58.049742 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049744 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049746 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.049748 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:58.050028 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 55
23:37:58.058301 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 478
23:37:58.060702 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.139611 IP 192.168.1.12.53885 > 216.58.210.206.443: tcp 0
23:37:58.159598 IP 216.58.210.206.443 > 192.168.1.12.53885: tcp 0
23:37:58.178391 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 474
23:37:58.228330 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:58.240200 IP 192.168.1.12.51729 > 192.168.1.1.53: UDP, length 32
23:37:58.242571 IP 192.168.1.12.53889 > 216.58.204.35.443: tcp 0
23:37:58.268541 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:58.271821 IP 192.168.1.12.51729 > 192.168.1.1.53: UDP, length 32
23:37:58.284697 IP 192.168.1.12.53886 > 216.58.210.206.443: tcp 0
23:37:58.304749 IP 216.58.210.206.443 > 192.168.1.12.53886: tcp 0
23:37:58.308898 IP 192.168.1.1.53 > 192.168.1.12.51729: UDP, length 48
23:37:58.308938 IP 192.168.1.1.53 > 192.168.1.12.51729: UDP, length 48
23:37:58.311446 IP 192.168.1.12.53890 > 216.58.204.36.443: tcp 0
23:37:58.330730 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:58.515455 IP 192.168.1.12.53891 > 216.58.204.36.443: tcp 0
23:37:58.533651 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:58.577459 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:58.631615 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:58.838647 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:59.031587 IP 192.168.1.12.53892 > 172.217.169.74.443: tcp 0
23:37:59.052740 IP 172.217.169.74.443 > 192.168.1.12.53892: tcp 0
23:37:59.054462 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 75
23:37:59.054482 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 64
23:37:59.054493 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:59.054495 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 0
23:37:59.054734 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 42
23:37:59.062113 IP 192.168.1.1.443 > 192.168.1.12.53723: tcp 475
23:37:59.063967 IP 192.168.1.12.53723 > 192.168.1.1.443: tcp 0
23:37:59.243221 IP 192.168.1.12.53889 > 216.58.204.35.443: tcp 0
23:37:59.264690 IP 216.58.204.35.443 > 192.168.1.12.53889: tcp 0
23:37:59.312966 IP 192.168.1.12.53890 > 216.58.204.36.443: tcp 0
23:37:59.312978 IP 192.168.1.12.57407 > 192.168.1.1.53: UDP, length 29
23:37:59.314467 IP 192.168.1.12.49189 > 192.168.1.1.53: UDP, length 31
23:37:59.331652 IP 216.58.204.36.443 > 192.168.1.12.53890: tcp 0
23:37:59.342593 IP 192.168.1.12.61219 > 192.168.1.1.53: UDP, length 50
23:37:59.342652 IP 192.168.1.1.53 > 192.168.1.12.61219: UDP, length 95
23:37:59.344967 IP 192.168.1.12.53893 > 173.194.183.170.443: tcp 0
23:37:59.344989 IP 192.168.1.12.49189 > 192.168.1.1.53: UDP, length 31
23:37:59.344999 IP 192.168.1.12.57407 > 192.168.1.1.53: UDP, length 29
23:37:59.357706 IP 172.217.169.74.443 > 192.168.1.12.53892: tcp 0
23:37:59.365065 IP 173.194.183.170.443 > 192.168.1.12.53893: tcp 0
23:37:59.379039 IP 192.168.1.1.53 > 192.168.1.12.57407: UDP, length 45
23:37:59.381342 IP 192.168.1.12.53894 > 216.58.210.54.443: tcp 0
23:37:59.416154 IP 216.58.210.54.443 > 192.168.1.12.53894: tcp 0
23:37:59.422960 IP 192.168.1.1.53 > 192.168.1.12.49189: UDP, length 92
23:37:59.425720 IP 192.168.1.12.53895 > 216.58.205.33.443: tcp 0
23:37:59.459921 IP 216.58.205.33.443 > 192.168.1.12.53895: tcp 0
23:37:59.464776 IP 192.168.1.1.53 > 192.168.1.12.49189: UDP, length 92
23:37:59.465885 IP 192.168.1.1.53 > 192.168.1.12.57407: UDP, length 45
23:37:59.516224 IP 192.168.1.12.53891 > 216.58.204.36.443: tcp 0
23:37:59.535644 IP 216.58.204.36.443 > 192.168.1.12.53891: tcp 0
23:37:59.567976 IP 192.168.1.12.53896 > 216.58.210.54.443: tcp 0
23:37:59.569846 IP 192.168.1.12.53897 > 216.58.205.33.443: tcp 0
23:37:59.591774 IP 216.58.210.54.443 > 192.168.1.12.53896: tcp 0
23:37:59.593616 IP 216.58.205.33.443 > 192.168.1.12.53897: tcp 0
23:37:59.598347 IP 192.168.1.12.53898 > 173.194.183.170.443: tcp 0
23:37:59.618867 IP 173.194.183.170.443 > 192.168.1.12.53898: tcp 0
23:37:59.624472 IP 192.168.1.12.62206 > 192.168.1.1.53: UDP, length 33
23:37:59.655599 IP 192.168.1.12.62206 > 192.168.1.1.53: UDP, length 33
23:37:59.655721 IP 192.168.1.1.53 > 192.168.1.12.62206: UDP, length 259
23:37:59.658223 IP 192.168.1.12.53899 > 172.217.20.142.443: tcp 0
23:37:59.669700 IP 173.194.183.170.443 > 192.168.1.12.53893: tcp 0
23:37:59.677810 IP 172.217.20.142.443 > 192.168.1.12.53899: tcp 0
23:37:59.718731 IP 216.58.210.54.443 > 192.168.1.12.53894: tcp 0
23:37:59.759687 IP 216.58.205.33.443 > 192.168.1.12.53895: tcp 0Hope this helps.
Many Thx: Rav
- x.x.x.x (My Gateway IP) 1.330 ms 0.884 ms 1.038 ms
-
@raviktiwari said in Parallel Test Environment using 2 pfSense, 2 Static IP, 1 ISP, 1 Gateway:
216.58.210.46 google.com
try to change from dns resolver to dns forwarder or set the option "Enable Forwarding Mode" under dns resolver,
go to system / general setup and add DNS server 8.8.8.8 , try to surf -
@kiokoman Thanks for the suggestion.
Tried both options and also played with few checkboxes, but still no cigar. Still unable to get to the outer world. :-(
Many Thx: Rav
-
from lan can you ping 8.8.8.8 ?
if you try to open
http://216.58.210.46
do you see google? -
@kiokoman Thanks again for gettig back to me.
Unable to ping 8.8.8.8 - Request Tine Out and unable to open the webiste - "The site can't be reached".
Just so you know, I have removed ISP provided 2 DNS servers and using only 1 DNS server 8.8.8.8 and DNS resolveer is being used with "Enable Forwarding Mode".
And lastly, I am suing latest pfSense release - 2.4.5
In case you need more details, please do not hesitate to ask for it.
Many Thanks for your time, support and patience - Rav
-
nothing in the firewall logs?
-
@kiokoman Thanks again for getting back to me.
Earlier everything was failing in the firewall log, right now things are looking better but still internet not working.
My 2 pence guess is, it has got something to do with the rules, somethng is getting blocked somewhere -and we dont know why and how.
So PFA the screenshots of FW logs as well as my WAN and LAN rules. As per the good practice guide, I have obfuscated my public IP. So where you see something wiped off - that is my public IP, configured on the WAN port.
Hope this helps.
In case you need more info, pls do let me know.
Many Thx: Rav
-
@kiokoman Just so you know, I have also disabled my AVG Antivirus and Windows firewall on thelaptop - just in case that is blocking me out.
However, my Production Setup does not need me to do that... so eventually I will have to enable my antivirus and firewall on laptop/desktop.
Many Thx: Rav
-
you have tcp:s going out, i suspect a routing problem or asymmetric somewhere. or the usb network card is not doing the job.
if you invert the wan cable between the production pfsense and testlab pfsense? production stop to go to internet and testlab start to work maybe? -
@kiokoman @chpalmer I was looking at the recent posts in the forum and it looks like many people are struggling with access issues specially with 2.4.5
- VLANs work, but not LAN what am i doing wrong
- Can't update packages on 2.4.5
- list itempfSense v2.4.5 not able to resolve Domain Overrides against itself
- list itemWAN Going Down and Some Errors
Not sure if there is a more widespred bug impacting connectivity in general. Any thoughts?
I am planning to uninstall and reinstall my Test pfSense and start all over again using 2.4.4 to see if the issue persists - may be that way we can narrow down something.
Many Thanks: Rav
-
@kiokoman I can swap LAN and WAN within the TEST environment but not with Production because it is already in production and I wil have to change everything setting wise right from laptop to router to HAProxy, VPN and so on.
I have already tried swapping LAN and WAN on the laptop and I still had the same issue. Earlier LAN was WAN and vice-versa. Current configuration (USB as LAN and inbuilt port as WAN) is the new setup.
I also tried enabling wirelss card and configured it as OPT1, even on that I could not access internet.
I am therefore thinking to uninstall this and reinstall 2.4.4 and strat from the scratch.
Even I think there is some routing issue with latest 2.4.5 and I will be really scared if I am right as my Prod is running on 2.4.5 and so far - there is no issue.
I will report back shortly. Give me an hour or so.
Many Thx: Rav
-
All my 2.4.5 installs are working just fine.
Are you able to test out a 2.5 snapshot?
