URGENT HELP libvirt kvm
-
Hello Everyone,
I just got paged and pfsense running on libvirt/kvm stop passing traffic between LAN and WAN(bridge0). No idea what happened, I checked logs and host him self attached bridge in promisc mode and never went down.
Site is down right now, trying trouble shoot. Here are Information what I have.
Firewall was in production for quite time never had one issue until not. Tried disable enable interfaces, tried change bridge settings, did ifconfig bridge0 from terminal and it in promisc mode on pfsense. TX offload disable from Networking tab under Advance settings.
I attached screen shot from vm console where virtual machine behind pfsense and ping hangs.From PFSense him self I can ping internet side or lan side. It only from LAN traffic is not forwarded through WAN(BRIDGE0) to outside.
Any ideas welcome.
Setup
LAN em1 –---> WAN_BRIDGE0 -------------------------------> INTERNET
| ^
| ^
PUBLIC_DMZ (Servers with public IP's)
 -
Still all down, I replaced virtual network drivers from e1000 to virtio, but no luck. Traffic still not passed from LAN to WAN(bridge0)
-
so nothing pfsense log?? I find that unlikely.. So your clients can ping pfsense lan interface.. Well sniff.. does pfsense pass on the traffic?
-
All What I see in logs is enter promisc mode for interfaces under the bridge. I tried packet capture and all what I see see echo request to 8.8.8.8.
See screenshot from console. I can't get to firewall just have to work through console.
-
I see all kinds of issues there - what is your notice?
See that your clock is messed up with the kernel runtime backwards. Looks like you can not bind to your ldap server - are you trying to auth users with a proxy that could break them from going anywhere..
-
The problem that traffic is not passing from LAN to WAN(bridge0).
-
Yeah dude are you running a proxy with user auth? You clearly can not talk to your ldap server.. You clearly have an issue with your clock being off.. That could cause all kinds of problems.. Because it doesn't say error sending traffic your just going to not think it has anything to do with your problem??
why would your wan be in a bridge in the first place?? Did you upgrade kvm? Did you upgrade pfsense? There are all kinds of issues with pfsense 2.2 and kvm.
-
Here are packet capture of ping from server which sit on lan interface. Send echo request and nothing.
-
We use ldap for remote access vpn. And this error indicate that missing entry in host file. NTP is service running. Need check why is not in sync.
-
Dude I see 198.50.221.177 pinging 198.50.221.190 every second and working.. That is your wan apinger I would gather.. What are you lan rules? Did you sniff on your wan to see if those pings to 8.8.8.8 got forwarded? Did they get a reply on your wan and just not seeing it back on your lan?
KVM has known issues with pfsense 2.2 - what version of kvm are you using. Are you using any packages? Proxy for example - why don't you fix your errors.. Why did you clock go backwards? why can you not bind with your ldap server?? etc. etc. etc…
-
Yes, I added to hosts file entry for ldap and restarted ntp. To make log clear.
-
Yes, packet capture is on WAN(bridge0) interface.
Hypervisor: libvirt Fedora 22 server.
libvirt-daemon-kvm-1.2.18-1.fc22.x86_64
PFSense: 2.4 last release
-
Bridge on WAN because we have servers with public ip and I want make sure IPS is will protect it.
-
Here correspond bridge on fedora server libvirt
# ip a s wan_dmz 7: wan_dmz@NONE: <broadcast,multicast,promisc,up,lower_up>mtu 1500 qdisc noqueue state UP group default link/ether fe:54:00:4d:f0:64 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc5f:d0ff:fece:1501/64 scope link valid_lft forever preferred_lft forever [root@dedi132 ~]# brctl show wan_dmz bridge name bridge id STP enabled interfaces wan_dmz 8000.fe54004df064 yes dummy1 vnet12 vnet2</broadcast,multicast,promisc,up,lower_up> -
"I want make sure IPS is will protect it."
What??? So your running SNORT? So maybe that is blocking your traffic.. When did 2.4 of pfsense come out?? 2.2.4 is latest..
Its a VM right - why don't you just load last snapshot or deploy base install again.. And again a KVM has known issues with pfsense 2.2 and above
https://forum.pfsense.org/index.php?topic=88467.0
-
Sorry, yes you right version is 2.2.4
-
The interesting part setup was working until last night
-
Snort in not running.
-
Well you have it enabled and not running - so like blocking everything? Why would you have to add a host file for ldap – wasn't it working before?
So that sniff showing pings going out to 8.8.8.8 was on your wan?? Then why was it showing your 10 address then as source - that for sure would never freaking work.
Here is me sniffing on wan for ping of client to 8.8.8.8 see how source is my public IP..
That sniff made no sense to me - if on lan why was it showing what looks like your apinger every second.. But private IP pinging 8.8.8.8
You had errors in your log about NAT as well.. Not installing nat reflection.. Seems to me you got ALL kinds of problems for something what was working yesterday.. Snort not running, time off, can not resolve your ldap server, rules not loading? traffic going out wan with private addresses..
-
That sniff make perfect sense that point the issue is bridge0 interface, because from pfsense him self I can ping both ends LAN and WAN. Arp ping you see as I says, because pfsesne can reach ISP provider side no problem. That indicate that some issue on BRIDGE0 interface which cause traffic do not to be forwarded from LAN to WAN. I can share my screen and you can see.
Host file entry need for ldap because encryption port 636 where ldap server validate the CN. And each time I am reboot pfsesnse it override /etc/hosts and connection to ldap is failing, I could't find how to preserve this entry in host file. If IPS down and it not blocking traffic.
