Unbelieveably bad performance

Douglas Haber

ok so you sniffed on wan of pfsense that 6.38 address and saw no answer. So from that its hard to tell if something just didn't answer or you don't have rules setup. So you forward that to something inside - did you sniff on the lan side of the pfsense box (xn0)? Did pfsense not send the traffic to where you were forwarding it?

can you post your wan and forwarding rules?

here you go.

i set up a brand new VM with just these rules (attached) with same slowness

it passed some of the telnet requests, after 45-90 seconds, and others timed out. see the above pastebin link in initial post

i ran the packet capture in the web ui which said it would check all interfaces for it.

![Screen Shot 2015-01-15 at 1.19.36 PM.png](/public/imported_attachments/1/Screen Shot 2015-01-15 at 1.19.36 PM.png)
![Screen Shot 2015-01-15 at 1.19.36 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-01-15 at 1.19.36 PM.png_thumb)
![Screen Shot 2015-01-15 at 1.19.27 PM.png](/public/imported_attachments/1/Screen Shot 2015-01-15 at 1.19.27 PM.png)
![Screen Shot 2015-01-15 at 1.19.27 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-01-15 at 1.19.27 PM.png_thumb)

johnpoz

Ok so your forwarding to 10.166.109.1

I see traffic to that.. But it never answers.. So pfsense sent the traffic to 10.166.109.1 - but it never answers.. so pfsense seems to be forwarding correctly.

Look at you image you posted.. for every syn and retrans of the syn that hits 65.98.6.38, you see traffic sent to 10.166.109.1 from 65.98.6.46, that was the sender to 65.98.6.38. That looks to be the forward to me.

So whatever is suppose to be listening on 80 is not, or its firewalled and doesn't allow from 65.98, etc.. Maybe 109.1 is wrong IP? Maybe there is something wrong with your vm setup that its not getting the traffic. Or maybe 109.1 doesn't have gateway? So it doesn't know how to send traffic back to 65.98 network? You could always sniff on the 65.98.109.1 box to check. But sure looks like pfsense is doing what you asked it to do.

Douglas Haber

@johnpoz:

Ok so your forwarding to 10.166.109.1

I see traffic to that.. But it never answers.. So pfsense sent the traffic to 10.166.109.1 - but it never answers.. so pfsense seems to be forwarding correctly.

Look at you image you posted.. for every syn and retrans of the syn that hits 65.98.6.38, you see traffic sent to 10.166.109.1 from 65.98.6.46, that was the sender to 65.98.6.38. That looks to be the forward to me.

So whatever is suppose to be listening on 80 is not, or its firewalled and doesn't allow from 65.98, etc.. Maybe 109.1 is wrong IP? Maybe there is something wrong with your vm setup that its not getting the traffic. Or maybe 109.1 doesn't have gateway? So it doesn't know how to send traffic back to 65.98 network? You could always sniff on the 65.98.109.1 box to check. But sure looks like pfsense is doing what you asked it to do.

I had this exact configuration on a machine running 2.1. A virtual machine. As a matter of fact, I created this configuration on 2.1, upgraded to 2.2, and it no longer works. So there is more to it than is meeting your eye I do believe

in addition, I have confirmed from one virtual machine to another behind the firewall, that the web server is listening and responding properly to request and has the correct gateway set.

Pardon me if there are typos in here, I am using voice dictation at the moment.

stephenw10

Was there not some issue with the Xen nic drivers? Was the 2.1.X vm using xn nics?

https://forum.pfsense.org/index.php?topic=84255.0

Steve

johnpoz

"upgraded to 2.2, and it no longer works."

All I can tell you from your sniff you posted is the traffic looks to have been sent on. Did it go out the right interface? I am not sure from that sniff.. But clearly the packets where forwarded to the IP. For example the top 2, you see the syn to 65.98.6.38, and then .000066 seconds later packet sent to 10.166.109.1

This tells me pfsense forwarded the packet - but I can not tell from the picture what interface that was captured on, if could see the mac address for example would know what interface it left on, etc.

From what I see in the sniff the problem with the 109.1 box getting the packet after it left pfsense or in the answer?. Lots of things could cause that - but then again can not be sure that the packet went out the correct interface from the image. What kind filter did you use for the sniff? I don't see any sort of broadcast traffic or other traffic that would validate that pfsense is seeing any traffic from 109.1 at all?

Douglas Haber

@johnpoz:

"upgraded to 2.2, and it no longer works."

All I can tell you from your sniff you posted is the traffic looks to have been sent on. Did it go out the right interface? I am not sure from that sniff.. But clearly the packets where forwarded to the IP. For example the top 2, you see the syn to 65.98.6.38, and then .000066 seconds later packet sent to 10.166.109.1

This tells me pfsense forwarded the packet - but I can not tell from the picture what interface that was captured on, if could see the mac address for example would know what interface it left on, etc.

From what I see in the sniff the problem with the 109.1 box getting the packet after it left pfsense or in the answer?. Lots of things could cause that - but then again can not be sure that the packet went out the correct interface from the image. What kind filter did you use for the sniff? I don't see any sort of broadcast traffic or other traffic that would validate that pfsense is seeing any traffic from 109.1 at all?

I told it to capture 80 only. I'll capture *.

johnpoz

Do 2 distinct captures.. Its easier to read that way.. Do one on the wan and one on the lan.. I just use tcpdump from ssh connection to do it.

Or post up the actual capture so can see the mac - so you can validate it forwarded it out the correct interface.

Douglas Haber

@johnpoz:

Do 2 distinct captures.. Its easier to read that way.. Do one on the wan and one on the lan.. I just use tcpdump from ssh connection to do it.

Or post up the actual capture so can see the mac - so you can validate it forwarded it out the correct interface.

Can't post the capture here. I'll upload them somewhere in a couple.

Douglas Haber

@johnpoz:

Do 2 distinct captures.. Its easier to read that way.. Do one on the wan and one on the lan.. I just use tcpdump from ssh connection to do it.

Or post up the actual capture so can see the mac - so you can validate it forwarded it out the correct interface.

http://douglashaber.com/dump/WANCapture.cap
http://douglashaber.com/dump/LANCapture.cap

stephenw10

Just to confirm, you've definitely not fallen foul of the driver change issue I linked to? I can't really see why it would affect you since you're not using VLANs or anything other than a standard config but it's worth checking.

Steve

Douglas Haber

@stephenw10:

Just to confirm, you've definitely not fallen foul of the driver change issue I linked to? I can't really see why it would affect you since you're not using VLANs or anything other than a standard config but it's worth checking.

Steve

I missed your question. Probably.

It was not xn in 2.1.5, it was re(4)

Hrmm.. found this on the ML:

http://lists.freebsd.org/pipermail/freebsd-xen/2014-April/002065.html

Maybe FreeBSD 10 just does not play nice on Xen.

Edit 2 - more quirks involving XS..

http://lists.freebsd.org/pipermail/freebsd-xen/2014-February/002010.html

stephenw10

Hmm, well that's interesting. You specified Realtek emulation in the Xen config then I assume? I'm unfamiliar with Xen.
I would try removing the paravirtualised NIC support in Xen so that pfSense goes back to using the re driver and see if that makes any difference. Additionally I would set it to emulate Intel NICs rather than Realtek.
As I say though I can't really see why the xn driver should be causing problems in your basic setup. Try removing all the hardware offloading options in System: Advanced: Networking:

Steve

Douglas Haber

@stephenw10:

Hmm, well that's interesting. You specified Realtek emulation in the Xen config then I assume? I'm unfamiliar with Xen.
I would try removing the paravirtualised NIC support in Xen so that pfSense goes back to using the re driver and see if that makes any difference. Additionally I would set it to emulate Intel NICs rather than Realtek.
As I say though I can't really see why the xn driver should be causing problems in your basic setup. Try removing all the hardware offloading options in System: Advanced: Networking:

Steve

Realtek is the default with XenServer. Switching to Intel emulation requires some hackery I am not ready to be doing yet. I don't want to change Xen necessarily.

EDIT: By hackery, I mean just a small change really (http://www.netservers.co.uk/articles/open-source-howtos/citrix_e1000_gigabit) but I also have other VM's running, and don't want to change too much.

I found this, which is interesting..

ssh from the Windows PV host to the FreeBSD PV DomU host appears to work
fine. Attempting to 'route' traffic from the Windows PV host 'through' the
FreeBSD PV DomU fails - pings go, DNS goes, initial TCP 'setups' go - but
stuff dies thereafter (i.e. may be packet size related or something).

Sounds pretty much like my issue (re: http not working) even though as another poster mentioned, requests are there.

http://lists.freebsd.org/pipermail/freebsd-xen/2014-February/002018.html

johnpoz

ok this looks different than before..

So looks like your getting back the syn,ack.. But then when you send a get, a 404 is sent back.. But then that is not working..

GET / HTTP/1.1
Host: 65.98.6.38
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

HTTP/1.1 404 Not Found
Date: Fri, 16 Jan 2015 13:45:34 GMT
Server: Apache/2.2.22 (Debian)

Then on the lan side you don't see the get?? Something really odd going on here..

From your wan sniff you can see that 404 was sent, but then you see retrans on the get and 404. But on the lan side not even seeing the get.. Were these sniffs taken at the same time?

edit: Ok looks like these were taken at different times.. wan goes from 7:45:31 to 7:47:14 But lan is from 7:47:31 to 7:49:16… You really need to take capture at the same time.. And wouldn't hurt to have sniff running over the same time period on the webserver.

wansniffinfo.png_thumb

Douglas Haber

@johnpoz:

ok this looks different than before..

So looks like your getting back the syn,ack.. But then when you send a get, a 404 is sent back.. But then that is not working..

GET / HTTP/1.1
Host: 65.98.6.38
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

HTTP/1.1 404 Not Found
Date: Fri, 16 Jan 2015 13:45:34 GMT
Server: Apache/2.2.22 (Debian)

Then on the lan side you don't see the get?? Something really odd going on here..

From your wan sniff you can see that 404 was sent, but then you see retrans on the get and 404. But on the lan side not even seeing the get.. Were these sniffs taken at the same time?

the 404 is to be expected. i wanted a simple thing to be spit back for testing purposes, rather than several MB webpage ,which is what would be on it in production. there is nothing to be served on the webserver now.
very close. couple of seconds apart max. i'll work on a set up exact same time ones.

johnpoz

no they are not a couple of seconds apart.. they are completely different time frames. See my edit.

Douglas Haber

@johnpoz:

no they are not a couple of seconds apart.. they are completely different time frames. See my edit.

I'll run a new set, same time. Hang on.

Douglas Haber

Same URL's. Same time. Literally within 1-2 seconds this time, as quick as I could move cursor and hit go.

No webserver capture in this group, though

EDIT: let me see if i can do it again and turn up verbosity on pfsense, it's capture is way way less verbose with the LAN interface than my tcpdump was for the WAN

johnpoz

well wan is going to see all the noise of a typical wan connection ;) I would expect to see lots of noise ;)

Douglas Haber

@johnpoz:

well wan is going to see all the noise of a typical wan connection ;) I would expect to see lots of noise ;)

I forgot to take of the default limit of 100 packets on the pf capture. :-X

Redoing now