No, I didn't make any adjustments. Everything has been working fine regardless of my drop count. It seems like it doesn't matters unless there is actual contention for bandwidth. In that case, I would assume the higher priority queues such as VOIP will "win".
As you mentioned, the speed test may not be the best metric even if it appears to correlate to drops. I wish I could suggest a better test, but I really don't know how to properly test traffic shaping otherwise I would have done it to test my setup. I'm simply going by my end users phone call quality. If those drops were really impacting my setup I would have had people complaining about call quality.