Capturing traffic using tcpdump?
-
Well, thanks for confirming the chosen method as "not totally bonkers"
What I want/try to do is actually a bit tricky I think and certainly for me learning this... I have cut-outs in streamed music, and possibly other traffic too, so what I want to find is actually not what I can match, but what disappears when this happens, eg if something is possibly being temporarily blocked... Network is otherwise fiber, very fast, stable and from a well known ISP (who has no clue when calling them, hence the capture attempt). Challenge is up for anyone with suggestions... I first thought it would be a safe bet to assume it was caused in the service, or even the player but
I have had this
- over some time
- with different streaming services
- and several different (hardware) media players
So basically, what I (think) I would like to understand and/or learn how to do, is identify the traffic to and from my service, and how it changes when these outages occur, usually just a second or two.
-
@furom well just sniffing on the lan side interface might not give you the whole picture, you might want to sniff on both wan side and lan side.
While sure you might see retrans of acks on your lan side for data that doesn't show up, did it just not show up on the wan? You would want to rule on pfsense I would think, etc.
You may need to sniff on the client as well - is this connection your streaming with wireless?
Most of my media players (sticks) are wireless for example - while my tv is wired.. So if I was seeing problems on the sticks, but not the tv it might be wireless that is the problem, etc.
Are all your media players wired or wireless? Do you see it on both wired and wireless?
Either way your most likely going to want to filter it on something, grabbing all your traffic could be lots of traffic.. I take in this vlan 11 is only your media players? if not - you may want to atleast filter on IP of the media player your using for troubleshooting..
Are you using IPv6? First thing I would do if having such an issue is try and determine if only happens with IPv6 - disable of IPv6 might be a good first start without having to do any sniffing as of yet.
-
@johnpoz said in Capturing traffic using tcpdump?:
@furom well just sniffing on the lan side interface might not give you the whole picture, you might want to sniff on both wan side and lan side.
While sure you might see retrans of acks on your lan side for data that doesn't show up, did it just not show up on the wan? You would want to rule on pfsense I would think, etc.
You may need to sniff on the client as well - is this connection your streaming with wireless?
No it's wired, I don't use wireless unless I have to
Most of my media players (sticks) are wireless for example - while my tv is wired.. So if I was seeing problems on the sticks, but not the tv it might be wireless that is the problem, etc.
Are all your media players wired or wireless? Do you see it on both wired and wireless?
Also wired. I find those more reliable
Either way your most likely going to want to filter it on something, grabbing all your traffic could be lots of traffic.. I take in this vlan 11 is only your media players? if not - you may want to atleast filter on IP of the media player your using for troubleshooting..
True, this is the lan I keep streamers etc on. Right now just one, so fairly easy to single out... :O
Are you using IPv6? First thing I would do if having such an issue is try and determine if only happens with IPv6 - disable of IPv6 might be a good first start without having to do any sniffing as of yet.
No, that is an upcoming thing, to enable IPv6, but for now disabled
-
@furom said in Capturing traffic using tcpdump?:
No it's wired, I don't use wireless unless I have to
Same here.. Wireless is for stuff that moves ;) heheh
If you get some sniffs - happy to take a look see.. but intermittent issues like this can be difficult to track down sometimes.. What is the media player you are using? It happens on all services, netflix, amazon prime, hulu, etc. ? Do you have a local system like plex or emby, jellyfin, etc.
-
@johnpoz said in Capturing traffic using tcpdump?:
@furom said in Capturing traffic using tcpdump?:
No it's wired, I don't use wireless unless I have to
Same here.. Wireless is for stuff that moves ;) heheh
If you get some sniffs - happy to take a look see.. but intermittent issues like this can be difficult to track down sometimes.. What is the media player you are using? It happens on all services, netflix, amazon prime, hulu, etc. ? Do you have a local system like plex or emby, jellyfin, etc.
I use an Apple TV to play Spotify. And no, it is only noticeably for music. Netflix and such happily plays along.
One thing that comes to mind now, before going on a wild goose chase... Spotify is at times unhappy with the network settings and complains "Check your network to continue listening" etc I am not convinced the two are related though, but just to mention. I did have this happen during a sniff, but only for the internal network, and by briefly looking at it, it looked very similar to all the rest. Some strange stuff of a "googleuser" in there too, I don't use google where I can avoid it, and certainly has no users connected, so hopefully it is something quite normal...
It wouldn't pose a problem capturing on two interfaces simultaneously? Say lan and wan?
File had grown quite big, ~ 500MB, Perhaps 30 minutes was a little too much, 10 minutes should be plenty...This is the rules Spotify has to cope with, and complains about occasionally;
-
Mmm, I would be looking for errors or blocked traffic. It could be very hard to find the actual outage in a file that size. And it will be hidden by the fact that spotify will be buffering traffic so it likely won't align with when it actually stopped playing.
-
@stephenw10 said in Capturing traffic using tcpdump?:
Mmm, I would be looking for errors or blocked traffic. It could be very hard to find the actual outage in a file that size. And it will be hidden by the fact that spotify will be buffering traffic so it likely won't align with when it actually stopped playing.
Well, agreed. What I see is not easy to read... I do not see any errors on WAN though. What would "blocked traffic" look like? I see some sort of "frame" that repeats, showing a MAC to another MAC
-
If not as blocks in the firewall log then as a load of TCP retransmissions in the pcap.
-
@stephenw10 said in Capturing traffic using tcpdump?:
If not as blocks in the firewall log then as a load of TCP retransmissions in the pcap.
I don't see any of those either pror to or around the time of the outage. How big of a buffer are we talking about for spotify? Seconds, minutes? When unplugging WAN, it continues to play for a little while, but not very... Perhaps should just try again... :)
-
No idea I don't use Spotify. It's probably configurable somewhere. I'd guess a few minutes.
-
@stephenw10 Well, from what I could find it seems to differ with available storage, so will be even harder I suppose
-
@furom so what error is your client spewing exactly? Does it want you to be using some doh dns? Does it think your internet connection is down, ie doesn't have internet?
-
@johnpoz said in Capturing traffic using tcpdump?:
@furom so what error is your client spewing exactly? Does it want you to be using some doh dns? Does it think your internet connection is down, ie doesn't have internet?
That is the thing... I can't see any errors at all... Occasional packets has zero length, but that is by no means aligned or proportional to the sound-cuts...
Edit: I seem to have used the command wrong, or slightly. Using "-w" creates a binary file, unsure why one would want that. I did instead use ">" which produced a readable format. I suppose the binary could be used in wireshark etc perhaps...? But they seem pretty similar
-
But presumably the client itself throws an error when it stops playing?
-
@stephenw10 said in Capturing traffic using tcpdump?:
But presumably the client itself throws an error when it stops playing?
If I were to describe it, it's like the stream is paused, or volume lowered really. If I didn't know better (do I??) I could almost think someone had access to my control it... The client continues just fine after every occurrence. I know the Apple TV doesn't have a log, at least their support tells me that. Yesterday the frequency of their occurrence was a lot higher than before... Whatever that could mean.
My hope was to use the packet stream/capture as the "log", but yes, is immensely harder, if even possible.
Question is perhaps if there are any client that has a log that will show what is really going on between my client and the player - if that is what is relevant to look at...
-
Yes, I would approach this using a different client. Try using a desktop spotify client and make sure that still hits the issue first. That might at least have more debugging options asusming it does hit it. Of course if it doesn't that implies it's something with the appleTV or at least the way it runs spotify.
-
@stephenw10 I will, if nothing else to possibly rule something out. As previously said, I don't really think this is either the player, nor Spotify even, as I have changed all of it.. but still, can't explain it
Edit: Installed the Spotify client and it just happened again. Have not had time to figure out if there are any logs yet, but issue remains
-
@furom so you installed spotify on your windows pc and your still having the issue - while not a big spotify user, I have my own music collection on my plex that I use.. My dead collection has way more than spotify does ;) heheh
I count like only 162 on spotify ;) heheh
But be happy to fire it up and see if I can hear your issue..
-
@johnpoz said
But be happy to fire it up and see if I can hear your issue..
Thanks, I would be really surprised if it is a general issue, but a test is much appriciated :)
-
@johnpoz said in Capturing traffic using tcpdump?:
I count like only 162 on spotify ;) heheh
LOL! Well, that's 162 more than I knowingly have heard. You make me curious, have to switch to try it out... Anyone in particular I should start with? :)
