URGENT HELP libvirt kvm



  • Hello Everyone,
    I just got paged and pfsense running on libvirt/kvm stop passing traffic between LAN and WAN(bridge0). No idea what happened, I checked logs and host him self attached bridge in promisc mode and never went down.
    Site is down right now, trying trouble shoot. Here are Information what I have.
    Firewall was in production for quite time never had one issue until not. Tried disable enable interfaces, tried change bridge settings,  did ifconfig bridge0 from terminal and it in promisc mode on pfsense. TX offload disable from Networking tab under Advance settings.
    I attached screen shot from vm console where virtual machine behind pfsense and ping hangs.

    From PFSense him self I can ping internet side or lan side. It only from LAN traffic is not forwarded through WAN(BRIDGE0) to outside.

    Any ideas welcome.

    Setup

    LAN em1 –---> WAN_BRIDGE0      -------------------------------> INTERNET
                                          |  ^
                                          |  ^
                                PUBLIC_DMZ (Servers with public IP's)
    ![PF_ISSUE01_Screenshotfrom2015-09-05 01-49-19.png](/public/imported_attachments/1/PF_ISSUE01_Screenshotfrom2015-09-05 01-49-19.png)
    ![PF_ISSUE01_Screenshotfrom2015-09-05 01-49-19.png_thumb](/public/imported_attachments/1/PF_ISSUE01_Screenshotfrom2015-09-05 01-49-19.png_thumb)



  • Still all down, I replaced virtual network drivers from e1000 to virtio, but no luck. Traffic still not passed from LAN to WAN(bridge0)


  • Rebel Alliance Global Moderator

    so nothing pfsense log??  I find that unlikely..  So your clients can ping pfsense lan interface.. Well sniff..  does pfsense pass on the traffic?



  • All What I see in logs is enter promisc mode for interfaces under the bridge. I tried packet  capture and all what I see see echo request to 8.8.8.8.

    See screenshot from console. I can't get to firewall just have to work through console.



  • Rebel Alliance Global Moderator

    I see all kinds of issues there - what is your notice?

    See that your clock is messed up with the kernel runtime backwards.  Looks like you can not bind to your ldap server - are you trying to auth users with a proxy that could break them from going anywhere..



  • The problem that traffic is not passing from LAN to WAN(bridge0).


  • Rebel Alliance Global Moderator

    Yeah dude are you running a proxy with user auth?  You clearly can not talk to your ldap server..  You clearly have an issue with your clock being off.. That could cause all kinds of problems..  Because it doesn't say error sending traffic your just going to not think it has anything to do with your problem??

    why would your wan be in a bridge in the first place??  Did you upgrade kvm?  Did you upgrade pfsense?  There are all kinds of issues with pfsense 2.2 and kvm.



  • Here are packet capture of ping from server which sit on lan interface. Send echo request and nothing.




  • We use ldap for remote access vpn. And this error indicate that missing entry in host file. NTP is service running. Need check why is not in sync.


  • Rebel Alliance Global Moderator

    Dude I see 198.50.221.177 pinging 198.50.221.190 every second and working.. That is your wan apinger I would gather..  What are you lan rules?  Did you sniff on your wan to see if those pings to 8.8.8.8 got forwarded?  Did they get a reply on your wan and just not seeing it back on your lan?

    KVM has known issues with pfsense 2.2 - what version of kvm are you using.  Are you using any packages?  Proxy for example - why don't you fix your errors.. Why did you clock go backwards?  why can you not bind with your ldap server??  etc. etc. etc…



  • Yes, I added to hosts file entry for ldap and restarted ntp. To make log clear.



  • Yes, packet capture is on WAN(bridge0) interface.

    Hypervisor: libvirt Fedora 22 server.

    libvirt-daemon-kvm-1.2.18-1.fc22.x86_64

    PFSense: 2.4 last release



  • Bridge on WAN because we have servers with public ip and  I want make sure IPS is will protect it.



  • Here correspond bridge on fedora server libvirt

    
    # ip a s wan_dmz
    7: wan_dmz@NONE: <broadcast,multicast,promisc,up,lower_up>mtu 1500 qdisc noqueue state UP group default 
        link/ether fe:54:00:4d:f0:64 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::fc5f:d0ff:fece:1501/64 scope link 
           valid_lft forever preferred_lft forever
    [root@dedi132 ~]# brctl show wan_dmz
    bridge name	bridge id		STP enabled	interfaces
    wan_dmz		8000.fe54004df064	yes		dummy1
    							vnet12
    							vnet2</broadcast,multicast,promisc,up,lower_up> 
    

  • Rebel Alliance Global Moderator

    "I want make sure IPS is will protect it."

    What???  So your running SNORT?  So maybe that is blocking your traffic.. When did 2.4 of pfsense come out??  2.2.4 is latest..

    Its a VM right - why don't you just load last snapshot or deploy base install again..  And again a KVM has known issues with pfsense 2.2 and above

    https://forum.pfsense.org/index.php?topic=88467.0



  • Sorry, yes you right version is 2.2.4



  • The interesting part setup was working until last night



  • Snort in not running.



  • Rebel Alliance Global Moderator

    Well you have it enabled and not running - so like blocking everything?  Why would you have to add a host file for ldap – wasn't it working before?

    So that sniff showing pings going out to 8.8.8.8 was on your wan??  Then why was it showing your 10 address then as source - that for sure would never freaking work.

    Here is me sniffing on wan for ping of client to 8.8.8.8 see how source is my public IP..

    That sniff made no sense to me - if on lan why was it showing what looks like your apinger every second.. But private IP pinging 8.8.8.8

    You had errors in your log about NAT as well.. Not installing nat reflection..  Seems to me you got ALL kinds of problems for something what was working yesterday..  Snort not running, time off, can not resolve your ldap server, rules not loading?  traffic going out wan with private addresses..




  • That sniff make perfect sense that point the issue is bridge0 interface, because from pfsense him self I can ping both ends LAN and WAN. Arp ping you see as I says, because pfsesne can reach ISP provider side no problem. That indicate that some issue on BRIDGE0 interface which cause traffic do not to be forwarded from LAN to WAN. I can share my screen and you can see.
    Host file entry need for ldap because encryption port 636 where ldap server validate the CN. And each time I am reboot pfsesnse it override /etc/hosts and connection to ldap is failing, I could't find how to preserve this entry in host file. If IPS down and it not blocking traffic.


  • Rebel Alliance Global Moderator

    How does that sniff make sense??  You can see pings from public ip to public IP working - sourced from a 10.x to public IP 8.8.8.8 not working.. As it shouldn't how exactly is 8.8.8.8 going to answer you from a 10 address?

    You stated you sniffed on pfsense WAN.. If that is the case it should be showing you WAN IP pinging 8.8.8.8 not some 10 address.

    How exactly do you think this would work??




  • Because it hit bridge0 interface and Outbound NAT is not applied and connection Outbound from LAN to WAN that why it show private ip. Secondly LAN subnet is directly connected to pfsense it will never show you in packet capture applied NAT rules.  If you sniff traffic on WAN interface and it will be inbound connection then you should see PUBLIC ip's.



  • How does that sniff make sense??  You can see pings from public ip to public IP working - sourced from a 10.x to public IP 8.8.8.8 not working..

    That exactly what I am says Bridge0 interface not working !!!!!!!!!!!!!!! PFSense not FORWARDING TRAFFIC FROM PRIVATE SUBNET LAN TO PUBLIC INTERNET



  • By The way Snort fail to start. I just updated to last package. Here the log for Snort start up.

    
    Sep 5 17:34:41	php-fpm[62471]: /snort/snort_interfaces.php: The command '/usr/pbi/snort-amd64/bin/snort -R 64956 -D -q --suppress-config-log -l /var/log/snort/snort_bridge064956 --pid-path /var/run --nolock-pidfile -G 64956 -c /usr/pbi/snort-amd64/etc/snort/snort_64956_bridge0/snort.conf -i bridge0' returned exit code '1', the output was ''
    Sep 5 13:34:41	snort[40294]: FATAL ERROR: pf.conf => Table snort2c does not exist in packet filter: Invalid argument
    Sep 5 17:34:41	php-fpm[62471]: /snort/snort_interfaces.php: [Snort] Snort START for WAN_PR01(bridge0)...
    Sep 5 17:34:41	php-fpm[62471]: /snort/snort_interfaces.php: [Snort] Building new sid-msg.map file for WAN_PR01...
    Sep 5 17:34:41	php-fpm[62471]: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN_PR01...
    Sep 5 17:34:38	php-fpm[62471]: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN_PR01 ...
    Sep 5 17:34:38	php-fpm[62471]: /snort/snort_interfaces.php: Toggle (snort starting) for WAN_PR01(bridge0)...
    Sep 5 17:34:38	
    
    

  • Banned

    Dude, when you don't have absolutely basic shit working, why are you mesing with Snort? (Totally OT here in addition.)



  • True Agree. I am focusing on issue of BRIDGE0



  • Here are TCP settings. I disabled offload as was recommended, but result the same.




  • I was thinking what else can be. I did tcpdump on virtual host him self Fedora Server and still see private IP, which I though on this stage I should see only public ip, because pfsense should apply outbound NAT rules. What else I can check.

    
    [root@dedi132 ~]# tcpdump -i any -vvv -nnnn -s 0 icmp
    16:32:24.378774 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.18.130.50 > 8.8.8.8: ICMP echo request, id 40991, seq 641, length 64
    16:32:24.378782 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.18.130.50 > 8.8.8.8: ICMP echo request, id 40991, seq 641, length 64
    16:32:24.378900 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.18.130.50 > 8.8.8.8: ICMP echo request, id 40991, seq 641, length 64
    16:32:24.378909 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.18.130.50 > 8.8.8.8: ICMP echo request, id 40991, seq 641, length 6
    
    

  • Rebel Alliance Global Moderator

    "Secondly LAN subnet is directly connected to pfsense it will never show you in packet capture applied NAT rules.  If you sniff traffic on WAN interface and it will be inbound connection then you should see PUBLIC ip's."

    What??

    I assume its wan is 198.50.221.177, pretty sure in fact that is your pfsense since when I hit it I get pfsense login!  That you have webgui allowed on your wan is just another F'd up thing with this setup that is for sure

    why don't you PM me a login and I will take a look directly to what is going on..  What kind of BS setup do you have that devices on the lan side of pfsense would not be natted through pfsense wan??  I can tell you for sure yes you would see natted traffic on the wan and on the lan you would see the non natted traffic.  So as you can see here is me pinging you on the lan interface of pfsense where it shows my private address, and there it is going out the wan with my public IP.  And you can also see pfsense pinging its gateway every 1 second as it does with default apinger running.






  • Not sure why it working from WAN, I don't see any rules which will allow the access.



  • Look like firewall is not running. Logs are from Sat 5 midnight. That why you can access it.



  • Check screenshot from PFSense where on tcpdump I am capture traffic on BRIDGE0. Server on LAN is send ping, but nothing coming back. look like firewall not running or something like that.




  • show the same capture but on pfsense WAN (not bridge0)



  • BRIDGE0 is the WAN



  • In first my post I have small schema design.



  • ok, just read the thread again. Your setup looks messed up.
    no other suggestion apart from restore from backup. :-)



  • I think it some thing to do with firewall. Logs in web UI is not writing still from midnight Sep 5.


  • Rebel Alliance Global Moderator

    well its not pinging now.



  • I shutdown old unit. And in progress of rebuild to new unit. Left restore firewall rules.  Because I need return site online.
    So far everything works interfaces same setup as before all the same. And don't have idea what happened, I suspect that some thing happened with firewall, because I tried restore form backup.

    I am still not back online 100% but,

    I want tell Everyone here HUGE THANK YOU FOR SUPPORT.


  • Rebel Alliance Global Moderator

    so now do your ping test with 8.8.8.8 on wan.. Does it show you public or private – there just is no freaking way it could work with private IP and no nat.