Large Packet Captures and Xbox Live

xcrustwadx

I have been experiencing disconnections from Xbox Live when 3 or more Xbox 360s are signed in. I can't predict it at all, one Xbox will disconnect randomly during a game. It happens consistently on 1.2, and 1.2.1 RC2

I trying to pinpoint the issue by doing a packet capture. However this will be a large one since I'll need to capture for hours before I see the problem. I tried for 10 minutes and the file size was around 7MB. Is this sort of large capture going to break anything? Is there any other alternative?

My pfsense hardware is as follows:
10GB hdd
Athlon 1.5ghz
512mb ram

xcrustwadx

I tried to do a capture and it was over 200MB. So i narrowed it down a little and looked at the packets in ethereal. Here is the problem I think. What may be causing the disconnects and failures to log in is the following:

No. Time Source Destination Protocol Info
37 1.015793 65.55.42.132 192.168.1.238 KRB5 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED[Malformed Packet]

Frame 37 (162 bytes on wire, 162 bytes captured)
Ethernet II, Src: 3com_11:6c:f7 (00:50:04:11:6c:f7), Dst: Microsof_c4:1c:ce (00:12:5a:c4:1c:ce)
Internet Protocol, Src: 65.55.42.132 (65.55.42.132), Dst: 192.168.1.238 (192.168.1.238)
User Datagram Protocol, Src Port: kerberos (88), Dst Port: 1257 (1257)
Kerberos KRB-ERROR
Pvno: 5
MSG Type: KRB-ERROR (30)
stime: 2008-12-29 00:23:14 (Z)
susec: 374869
error_code: KRB5KDC_ERR_PREAUTH_REQUIRED (25)
Realm: PASSPORT.NET
Server Name (Service and Instance): krbtgt/XBOX.COM
e-text: \203\304\031J
e-data
[Malformed Packet: KRB5]

Does anyone have any idea? I get a few of these before a successful login.

Thank you in advance.

fastcon68

With multiple x-box's using logging in from the same external IP address, it could just be a timing issue.

Couple of network questions:
What type of switch are you using?
Any vLans or optional interfaces?
Is all the cabling at least cat5E?
Are all the cables hand made or manufactured cables?
Do you have traffic shapper run?

Just some basic questions?

RC

xcrustwadx

What type of switch are you using?

Netgear 24-port Gigabit switch unmanaged

Any vLans or optional interfaces?

None

Is all the cabling at least cat5E?

yes CAT5E and CAT6

Are all the cables hand made or manufactured cables?

manufactured - monoprice.com (25ft), hand crimped run from garage to house

Do you have traffic shaper run?

no shaper running now.

All Xbox 360s plugged direct to the switch with 25FT cable. However I have the 24-port switch uplinked to another netgear 5-port gigabit switch in the house which then connects to Pfsense. My run between the 24-port and the 5-port is sort of far (from detached garage to house basement then to second floor house). You have me thinking… maybe my run is too long although it can't be more than 250FT. I used solid CAT6 for the run by the way. If it were the run, wouldn't this affect other traffic too?

You mention a timing issue... that really sounds interesting because sometimes they will not be able to log in. Other times we are all playing together(4+ people, same or different games) and it will start to choke (aka lag out). I check the traffic graphs when this happens and the upstream is only half-saturated. After a few minutes it goes back to normal. I'd say this happens about once or twice per day (8 hours of gameplay). However overall, performance is good.

It's just so hard to do a useful packet capture when I'm not sure when it will happen. Will another capture help narrow the problem? My drive is big enough I suppose.

How can I test the cable without spending an arm and a leg for a industrial hardware tester? Is there a timing test I could run over the wire or something?

fastcon68

great responses,
It mightbe worth getting a inexpensive managed switch like the DELL 2708 or some thing in that catagory. That will give you some test information like cable length and some basic information. On a long un getting close to 250 FT you migh need to think about some different options to connect your build to your home.

I have 5 run's to my shop at home and that's close to 5 200' foot runs. I am running cat 5e for that with cat 5e jack's at each end. straight cables cause issues. I change to the jacks and got a good solid punch down and my issues went away. I did see about a 10 to 20 ms timining issue when my network was split between my huse and my shop.

I moved my pfsense box and 2003 sbs server back into the house. I running xen and have a small switch now, and single wireless switch. I cut out all most all of my long runs. I am down to 75 foot run that extends my DSL modem from the front of the house to the back of the house. All of my issues have gone away.

I wish I could have run 2 pair of 100 MB fiber and use media filters to go from fiber to cat5. That would have fixed my issue for sure.

I would try traffic shapping it really does help. I got it configured and it really helps out withhe online gamming we do and the VIOP traffic we have.

I ponder a few thoughts and seewhat else I can come up with that might help you.
RC

xcrustwadx

It might be worth getting a inexpensive managed switch like the DELL 2708 or some thing in that catagory. That will give you some test information like cable length and some basic information. On a long un getting close to 250 FT you migh need to think about some different options to connect your build to your home.

The 2708 and most other managed switches are just too expensive for me at this time. Likewise, fiber is completely out of my price range. Thanks for the suggestion though. I'm going to check my run… I left a TON of slack at the end. I bet I can shave off 30FT.

I am running cat 5e for that with cat 5e jack's at each end. straight cables cause issues. I change to the jacks and got a good solid punch down and my issues went away

Are you recommending crossover wiring instead of straight through? I actually have it terminated on one end straight through to a jack but the jack is the home-marketed version which requires no punch-down tool(from monoprice as well). On the garage end I have a patch panel(monoprice) which came with a plastic punch-down tool. Do you recommend I spend the $5 on a good jack and borrow a punch-down tool from work? I am also using 1FT CAT6 cables from the patch panel to the 24 port switch.

I moved my pfsense box and 2003 sbs server back into the house.

I can't do this unfortunately. However I may be able to move the modem and PFsense to the garage in the same room as the Xbox 360s. I only have a desktop and an access point in the house. No online gaming is done in the house anyway.

I would try traffic shapping it really does help. I got it configured and it really helps out withhe online gamming we do and the VIOP traffic we have.

I have no other traffic on the line so I don't think it will help much. Also there is that issue with anything using a uPnP port will be sent to default queue regardless of configuration. I've pieced together a semi-decent config to work around this but it's not perfect.

I did see about a 10 to 20 ms timining issue when my network was split between my huse and my shop

This just plain scares me. This sounds like it would affect my online gaming performance since my modem is in the house and my game systems are in the garage. I didn't think longer runs could cause that much latency. For good measure, I tried pinging a machine in the house from a machine the garage and then from the a machine in the garage to another machine in the garage and the ping times were all in the 0.5 to 0.8ms. I then tried ping from pfsense to machine in house and then garage and they ranged in the 0.2 to 0.3 range. Is this even accurate?

My game plan is this (thanks to your help):
1. Take off most of the slack at the end of the run
2. Go to home depot and get a new jack for each end of the long run. Then punch-down the line properly
3. If that doesn't help, I'll move the modem and PFsense to the garage.

fastcon68

xcrustwadx,
I would use a straight through and then a short crossover if necessary (See my comment above). From my shop to the house, I use a straight (2 x cable 200 ft (400 MB trunk)) through with a pair of crossover cables (6 ft) without issue.

I do agree with your game plan.

I got a pair of Nortel 350 managed switches that I am not using. I could get those to you if you would like them. Send me a private email at wcarterjr@embarqmail.com if you are interested.

At that point you could trunk 4 ports together to get a 800 MB trunk. I can pre-configure the pair if you would like them.

RC

xcrustwadx

First of all, thank you for your offer Fastcon68! I'm going to stick with my equipment for now.

Secondly I did some more packet capture and research. I have the issue where one Xbox will not be able to connect to Xbox Live (XBL), as I mentioned before. All packets were captured during about 5 "Xbox Live Connection Tests" (See attached image #2). I looked at a capture during that time and noticed a bunch of DNS queries to my ISP's DNS server with no responses back from the DNS server to the Xbox itself. Is it possible that the DNS server is using a DOS attack prevention mechanism which is preventing the DNS queries from being answered? I say this because we have 4 to 8 Xbox 360s / Xbox Live accounts logging in at the same time. I'm assuming EVERY time the Xbox turns on and tries to connect to XBL, it queries the DNS server. Multiply that by 8 and you get quite a few DNS requests from the same external IP(although this hardly seems like enough to trip a DOS prevention mechanism).
I can usually resolve the unable to connect to XBL issue by powering off the Xbox 360, waiting 10 minutes and then turning back on.

Anyway to see if I could fix this, I enabled DNS forwarder and looked up the Xbox Live servers IP addresses and DNS names and then entered them in the PFsense DNS forwarder override. (See the attached image #1)

It seems that there are also some requests for the dns name o.xbox.com and the IP address associated with that name changes often. After a successful login, there is constant communication with 65.55.42.130 during multiplayer games (in Gears of War 2 at least) although I was unable to get a DNS name for that host.

I'll report back to give the status but at the very least, it seems to sign on slightly faster now.

dns.jpg_thumb

dnsqueries.jpg_thumb

xcrustwadx

Update: We went through the entire weekend without any failures to sign in or disconnects from XBL. I don't want to say the problem is solved yet because it has only been 4 or 5 days. However things are looking darn good. We had all 8 Xbox 360s online all weekend so if it's solved, the DNS entiries for Xbox Live did it.

Thank you for your help.

xcrustwadx

Unfortunately adding the DNS entires did not fix the problem. I did experience some failures to log in and a disconnect or two this past weekend. However they seem to be far less frequent. It's a shame because I went a couple weeks without any issues. I'm going to keep investigating to see if I can alleviate this problem.

I noticed that the disconnects only seem to happen when another Xbox is signed on to live. For instance we'll have 3 online, which were all powered on about the same time, playing together great. Then the 4th is signed on and occasionally one will disconnect. This doesn't happen EVERY time but It seems like if it does happen, it's because an additional Xbox was signed on. I'm looking into the uPnP service because before I moved and upgraded to 1.2.1, I didn't have this issue and I had a script that restarted the uPnP service every morning at 7:30. This was to clear any entries because Xbox 360 neglects to remove mappings when it is powered down. So the mappings stay there indefinitely. I also noticed that when I upgraded to 1.2.1, the uPnP configuration page was different and had less choices. There was no more "clear old mappings" and some other settings that used to be there.

Maybe the one Xbox is seeing a seemingly old mapping, erasing it and remapping to itself when it is actually in use by another Xbox (causing the disconnect). The only thing that makes me hesitant is that uPnP lists the same Xbox 360s with the same ports mapped all the time. I don't think I've ever seen them request/map any ports but the one it was initially assigned. For example I've never seen Xbox 1 request or map any other port but 3074 and I've never seen Xbox 2 request or map any other port but 21934 etc.

I tried using the upnp_support script with the my current pfsense setup but it doesn't work. I'll type


cd /tmp
./upnp_support restart

Then it will say invalid argument. I also tried the stop, start and update arguments and all gave the same error. I noticed the upnp status page have a upnp_action function. Is there a way to make a cron job to restart the miniupnpd service every morning at 8am?

I bet you are all getting tired of Xbox stuff eh?

xcrustwadx

I recently made some headway on this issue. It appears that the uPnP port mappings are getting FUBAR'd (by other Xbox 360s on the network). I disabled static ports and it seems to solve the connection issue. However now I will occasionally get warnings about not having OPEN NAT on some Xbox 360s, but it's not such a big deal.

Anyway to investigate the problem I first upgraded to 1.2.3 RC1. Then I did some packet captures targeting the consoles that were unable to connect. The packets appear to be sent by the Xbox 360 to Xbox Live(XBL) but a reply is never received. All the while the other Xbox 360s are connected and playing fine. This leads me to believe it's a port forwarding issue. I think the replies were being sent to the wrong IP but there's no way I can see to distinguish that reply from all the other traffic in the packet capture since whoever the reply is being sent to is also receiving it's own packets from XBL. Once I turned off static ports, they have all been signing on OK not to mention sign on time is faster and no disconnects have occurred that I know of since then. Everyone is playing online and they're very happy… and if they're happy, I'm happy.

I checked and the uPnP port mappings are still being made, even without static ports. I'm sure uPnP wasn't developed with this in mind but I'm going to keep trying!

Thank you for your help.