Bandwidth test= fine, browsing unusable (HYPER-V)



  • Hello! (I posted this somewhere incorrect)

    I have pfSense 2.1 in both physical and virtualized forms. Physical has worked flawlessly for years but I want more time on our UPS so moving it into the virtualized domain would be a good idea.

    I have hyper-v 2012 and it has been nightmares all around (understandably so) until I received a pre-built VHD(X) for it. Boots up and everything is perfect, the interface is full and snappy. I add the proper ISP and it automatically picks up the DHCP (simple cable modem, no PPPoE or MAC spoofing needed). I try to browse a web page and it is incredibly slow and only loads a portion before timing out. I run a bandwidth test and after it loads, I get the full 10mbps down and 0.5mbps up.

    So it seems like the bandwidth is not an issue, the LAN has no issues…..it seems like it is just the routing. I have defaulted everything with no firewall rules apart from allowing everything on the outbound interface.

    I have assigned it 2 cores of the Xeon L5520 and they aren't spiking. I have assigned 512mb RAM but will see about expanding it a bit later today even though an insignificant portion has been used. The VM does have the integration services added as well as it is using the 10gb adapters.

    I am also using a separate internal DHCP (not pfSense's) as well as a separate internal DNS (not pfSense's) and this works when I'm using the physical pfSense.

    Any help would be appreciated!
    Thanks



  • Can you post a snapshot of your rules page here?
    Also you interface assignments, DHCP server settings.
    And what IP are you using on the LAN?  How about the "Modem".
    Is that a modem modem or a modem+router?
    Is it routing?



  • @kejianshi:

    Can you post a snapshot of your rules page here?
    Also you interface assignments, DHCP server settings.
    And what IP are you using on the LAN?  How about the "Modem".
    Is that a modem modem or a modem+router?
    Is it routing?

    1. yep, give me a few minutes, I'm rebuilding from scratch. LAN rules are literally default anti-lockout and allow any from LAN to WAN
    2. LAN pfSense interface IP is 192.168.10.1, WAN is DHCP (valid). DHCP server is OFF because I have my own separate one
    3. 192.168.10.1,
    4. it is a modem modem. I specifically asked for it. I don't want my ISP to NAT for me at all. I want my 2 public IPs per WAN connection for DMZ and Lync
    5. it is routing. It does work, and total throughput on a single download works. But when I go browse, pages only load half way and artifact everywhere (tested on multiple machines). Most times they just time out and fail to load, though.


  • @stlnstln:

    @kejianshi:

    Can you post a snapshot of your rules page here?
    Also you interface assignments, DHCP server settings.
    And what IP are you using on the LAN?  How about the "Modem".
    Is that a modem modem or a modem+router?
    Is it routing?

    1. yep, give me a few minutes, I'm rebuilding from scratch. LAN rules are literally default anti-lockout and allow any from LAN to WAN
    2. LAN pfSense interface IP is 192.168.10.1, WAN is DHCP (valid). DHCP server is OFF because I have my own separate one
    3. 192.168.10.1,
    4. it is a modem modem. I specifically asked for it. I don't want my ISP to NAT for me at all. I want my 2 public IPs per WAN connection for DMZ and Lync
    5. it is routing. It does work, and total throughput on a single download works. But when I go browse, pages only load half way and artifact everywhere (tested on multiple machines). Most times they just time out and fail to load, though.

    http://imgur.com/wUha5bC is the rules page



  • I've moved the virtualized pfsense to 192.168.10.254 so that I can keep the physical pfsense active for the users.

    http://imgur.com/mDkJnrF is a screenshot of me trying to refresh this page when using the virtualized pfsense as my gateway



  • Just the one pic?  I see nothing broken on that.
    Describe your DHCP mechanism?  What is pfsense connecting to on the LAN side?  What is that connecting to?  what connects to that?



  • @kejianshi:

    Just the one pic?  I see nothing broken on that.
    Describe your DHCP mechanism?  What is pfsense connecting to on the LAN side?  What is that connecting to?  what connects to that?

    My LAN DHCP is a Microsoft DHCP. My WAN DHCP is provided by my ISP. pfSense internal LAN IP is static. pfSense connects to the virtual switch on the LAN side (same VLAN as the other devices on my internal network. This works as evidenced by me being able to log onto the pfSense configuration page with absolutely no issues at all.

    My network topology consists of 2 zones (WAN & LAN) linked by pfSense.

    LAN side has (all on same VLAN and subnet): Domain controller with DNS, Application server with AD CS, DHCP and a few other roles, Second Application Server with WDS, WSUS and PRTG, KMS server, SQL 2012 server and 6 laptops, 2 desktops, 3 access points and a few other phones.

    The internal LAN side is all working correctly because when I use the physical pfSense, the internet is normal. When I use the virtual pfSense, then things become unstable.

    I am able to get an IP from my ISP. I am able to ping google from internal machines. I am able to load some content but not everything loads before timeouts hit.



  • OK - Sounds like its going to get complicated now.
    So, you have a physical pfsense (with its own separate public IP and modem?) and switch and LAN and LAN clients and that is somehow connected to your virtual pfsense which has a /22 (not a /30?) and its on LAN (connecting to I'm not sure what or how) getting DHCP from the first pfsense?

    I think we are going to need a network diagram.  Even just a snapshot of something scribbled on paper is ok with me.



  • haha - Seems you keep posting answers to questions I'm about to ask.  Lets see if you beat me to the punch this time.

    OK.  So, I know there are at least 2 separate WAN IPs now.

    How is outbound routing of packets from the LAN being handled with 2 (or more) WAN IPs?

    Also, who is your ISP?

    I'm not saying definitively that no one hands out DHCPed multi-ip connections that are static I presume?
    Usually, they assign you a few IPs and you have to bridge the WAN interface to them like this:
    http://www.youtube.com/watch?v=zrBr0N0WrTY    (sorry if you cant get that)  Basically, its bridged and then Virtual IP assignment usually for me here)
    So, are you 1:1 NAT from virtual pfsense > physical pfsense?



  • http://imgur.com/KXeyuvc,xBi1s24

    Here are the 2 scenarios.

    Basically I can swap between the virtualized pfsense and the physical using VLANs and such to test. I can even keep them on the same network (with different LAN IPs.1 and .254) for testing purposes since each have their own separate IP from the ISP and they are also on separate WAN subnets, too.

    So basically changing my default gateway on my laptop to the physical pfSense leaves me with a solid internet connection and everything is great. Changing my gateway to the virtualized pfSense, things fall apart.

    I don't see anything out of the ordinary on the logs, either :(



  • @kejianshi:

    haha - Seems you keep posting answers to questions I'm about to ask.  Lets see if you beat me to the punch this time.

    OK.  So, I know there are at least 2 separate WAN IPs now.

    How is outbound routing of packets from the LAN being handled with 2 (or more) WAN IPs?

    Also, who is your ISP?

    I'm not saying definitively that no one hands out DHCPed multi-ip connections that are static I presume?
    Usually, they assign you a few IPs and you have to bridge the WAN interface to them like this:
    http://www.youtube.com/watch?v=zrBr0N0WrTY     (sorry if you cant get that)  Basically, its bridged and then Virtual IP assignment usually for me here)
    So, are you 1:1 NAT from virtual pfsense > physical pfsense?

    There are actually 4 public WAN IPs(to make your head hurt more, I actually have 2 WANs but we won't touch the second one until this works).
    The outbound routing is being handled ONLY by pfSense computers. One is a virtualized one residing in a Hyper-V host with LAN IP of 192.168.10.254. The other is a physical one residing in a shitty computer with 192.168.10.1.

    I can change in my TCP/IP settings and alter my default gateway between the two to test one or the other.

    The ISP is Shaw in Vancouver, Canada.

    There is ZERO packet flow between the virtual and the physical pfsense. Each are entirely separate gateways on the network. To make things easier, let's imagine I don't have 2 pfsense boxes.
    Lets say I am a user who has had a physical pfsense forever. I want to remove it and install a virtual pfsense appliance. The new virtual pfsense appliance fails to perform as well as the physical. What should I check?



  • Yeah - But I don't see where you have set up a virtual IP to use one of your 2 public IPs?

    How are the VIPs (or the VIP) being mapped out and assigned?



  • Also, my public IPs are DHCP'd dynamics. I do not need statics for what I do with them.

    PS: thanks for your help!



  • @kejianshi:

    Yeah - But I don't see where you have set up a virtual IP to use one of your 2 public IPs?

    How are the VIPs (or the VIP) being mapped out and assigned?

    I am not using any virtual IPs at all



  • "Lets say I am a user who has had a physical pfsense forever. I want to remove it and install a virtual pfsense appliance. The new virtual pfsense appliance fails to perform as well as the physical. What should I check?"

    OK - Easier.

    Your WAN IP assignments seem weird to me.  You can't call up your company and tell them "Give me two IPs.  Give me your gateway address".
    Connect to that by bridging to their network (not DHCP) and then hand out the public IPs to Virtual IP?

    This setup you have now with multi-IP dhcp at the WAN….  Did that work on a physical box there ever?



  • OK - Welllll….  If this setup worked for you on a physical box before but not on the VM, I'm stumped.
    It does seem complex bordering on unnecessarily so.  That must be one sweet Microsoft DHCP machine to keep it around with these kinds of headaches (-;



  • @kejianshi:

    OK - Welllll….   If this setup worked for you on a physical box before but not on the VM, I'm stumped.
    It does seem complex bordering on unnecessarily so.  That must be one sweet Microsoft DHCP machine to keep it around with these kinds of headaches (-;

    Hahah it works surprisingly well. The only non-Dell/Cisco/Microsoft item is the pfSense. I'm waiting for the ASA 1000v to hit hyper-V :-)

    For dualWAN I pick 2 IPs and use them as multiple gateways load balanced in the routing section on the far left tab.

    For the DMZ IPs I put them right on the edge.I'll move them in later when I have fully set up Lync.

    I basically use pfSense as a multiwan capable DD-WRT



  • @kejianshi:

    "Lets say I am a user who has had a physical pfsense forever. I want to remove it and install a virtual pfsense appliance. The new virtual pfsense appliance fails to perform as well as the physical. What should I check?"

    OK - Easier.

    Your WAN IP assignments seem weird to me.  You can't call up your company and tell them "Give me two IPs.  Give me your gateway address".
    Connect to that by bridging to their network (not DHCP) and then hand out the public IPs to Virtual IP?

    This setup you have now with multi-IP dhcp at the WAN….   Did that work on a physical box there ever?

    I can't get statics on a non-Business line. I have considered it but with the fantastic Dynamic DNS, I haven't needed to yet. The multidhcp WAN IPs currently are working. :-)



  • For my own education, can you post a snapshot of your WAN interface assignment?
    I'd actually like to see how you are doing that, the VIP assignment, the VLAN and your multi-wan handling also incase I ever need to work with something like yours.  There are lots of how-to pages for multiwan/load balance/fail-over etc.  I'd love to compare your settings to those for educational purposes.



  • It just baffles me. I give it a pair of Xeon cores, 1GB RAM, Dual 10GbE NICs, a decently fast RAID storage upgrade and it just says "LOL NOPE" :-(

    This works brilliantly in VMWare ESXi but I no longer have a RAID card :-(



  • I'm looking forward to studying those snapshots.



  • @kejianshi:

    For my own education, can you post a snapshot of your WAN interface assignment?
    I'd actually like to see how you are doing that, the VIP assignment, the VLAN and your multi-wan handling also incase I ever need to work with something like yours.  There are lots of how-to pages for multiwan/load balance/fail-over etc.  I'd love to compare your settings to those for educational purposes.

    OK the full set up (I've only had enough NICs when I've used ESXi, obviously) was pfSense with 5 vNICs.

    This setup had NO DMZ.

    Modem#1 provided 2 public IPs and was plugged into port 1 of the 24 port swith. This port 1 was untagged on the outbound but was tagged at the port to VLAN5 (I call it MODEM1 VLAN)

    Modem#2 was plugged into Port2 of the 24-port (core) switch. This was tagged as VLAN 10. This was to prevent the ISP DHCP broadcasts from overlapping.

    The ESXi host had a few ports trunked directly from the core switch containing all VLANS (5, 10 and the internal 15).

    pfSense VM has 2 NICs on VLAN5, 2 NICs on VLAN10, and 1 NIC on the internal VLAN15.

    From there, I added them as part of a gateway group and loadBalance based on latency. It works AMAZINGLY well.

    I can even use Dynamic DNS for things such as VPN.domain.com or ftp.domain.com etc. This way I haven't needed to use a DMZ as I would just forward what I needed. But soon I'll be expanding to a proper tiered topology.

    My ISP changes my IPs once every3-6 months so it is really nice.



  • I'm just at the gym right now but when I get home, I'll do what I've been delaying forever: make a proper visio diagram.



  • I'm just bummed that it works on a crappy physical computer, works great on a VMWare host, but fails catastrophically on Hyper-V



  • Well I will certainly study it, but have no idea if I'll understand how its working.  Should be fun.



  • http://imgur.com/hl2Xo77,VYt3Hts,a2W3uXi

    This is the current pfsense setup for the physical pfsense implementation and it worked very well. The ISP gives me all different IPs with all different subnets and all different gateways.



  • crap
    the first image is incorrect and is for the virtual and is a wrong screenshot. Disregard it.

    This is the correct one: http://imgur.com/XbBg4ii,PdsrAoT,2D4ps89



  • This is the one I was waiting for. 
    Question.  Was this modem you are using now, it was connected to the physical machines before?
    But your links in the main status page show correctly as up correct?



  • What are you DNS servers?
    Are they being served by numbers forwarded from your WAN connection?
    Could you try changing them to something like:
    127.0.0.1
    8.8.8.8
    8.8.4.4

    ?????



  • to note: in the second image under gateway, it says dynamic for one. This is the one I've disconnected for the virtual pfsense to use, so it is seen as down for the moment (I've removed the connection to VLAN 5 from the physical pfsense and applied it to the virtual to use).



  • http://imgur.com/GaydetJ,BRkUTaK is a simplified network diagram

    the first image is the functional view while the second image is the physical view



  • @kejianshi:

    What are you DNS servers?
    Are they being served by numbers forwarded from your WAN connection?
    Could you try changing them to something like:
    127.0.0.1
    8.8.8.8
    8.8.4.4

    ?????

    I use an internal DNS on the Domain Controller for internal name resolution. It forwards to the ISP's primary and secondary DNS as well as 8.8.8.8 and 8.8.4.4.



  • One second, trying out your suggestion



  • I like your suggestion because the internal DNS hadn't changed the gateway.

    I tried external DNS both 8.8.8.8/8.8.4.4 as well as my ISP's DNS servers and it was still slow (flushed DNS before the test as well as cleared the browser caches)



  • OK - Now….  Maybe its the return path thats a problem.

    Try unplugging WAN cable from one modem.
    Test.

    Then other modem...
    Test...

    Is it more reliable on one modem than two?



  • Crap.  You only have one. Right?

    OK.  Can you drop 1 IP for WAN and test?



  • Just dropped everything on Modem1 except for a single connection/IP to virtualize pfSense and it still has same issues

    Weird things though…...google loads quickly and has no issues, so does speedtest.net......youtube takes a while to load all the thumbs but the videos buffer at full speed



  • I can't load a single thing on forum.pfsense.org when I change to the virtualized pfsense

    Also I am seeing a ton of denials on the pfsense firewall from internal IPV6 addresses (probably broadcasts) even though I disabled IPv6….......how do I kill that off?

    Edit: Here are the firewall logs:  http://imgur.com/1cgaMr5



  • Unless you have some need of IPV6 today, why not:

    System > Advanced > networking

    Un-click all the IPV6 on both your PFsense boxes?

    Turn it all off.



  • Yeah I did just that. The logs are still getting filled with IPv6 denials…...weird.