So I thought perhaps it was a CPU issue with OpenVPN, I disabled encryption and I'm still having dropped packets.
The machine is pretty low-end (Atom D525) but processors are pretty free?
last pid: 54088; load averages: 0.76, 0.59, 0.61 up 10+19:46:55 13:28:42
482 processes: 5 running, 450 sleeping, 27 waiting
CPU 0: 7.4% user, 0.0% nice, 3.5% system, 3.1% interrupt, 85.9% idle
CPU 1: 11.3% user, 0.0% nice, 3.5% system, 2.0% interrupt, 83.2% idle
CPU 2: 12.9% user, 0.0% nice, 2.7% system, 5.1% interrupt, 79.3% idle
CPU 3: 4.7% user, 0.0% nice, 4.7% system, 6.3% interrupt, 84.4% idle
Mem: 29M Active, 148M Inact, 427M Wired, 531M Buf, 7317M Free
Swap: 16G Total, 16G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 64K CPU0 0 245.8H 86.47% idle{idle: cpu0}
11 root 155 ki31 0K 64K CPU3 3 253.4H 84.96% idle{idle: cpu3}
11 root 155 ki31 0K 64K RUN 2 246.6H 76.37% idle{idle: cpu2}
11 root 155 ki31 0K 64K CPU1 1 251.4H 75.20% idle{idle: cpu1}
23044 root 30 0 21624K 5684K select 2 8:53 18.36% openvpn
21063 root 52 0 262M 36096K accept 1 0:02 10.16% php-fpm
54088 root 40 0 262M 36124K accept 2 0:01 7.57% php-fpm
12 root -92 - 0K 432K WAIT 0 596:01 4.88% intr{irq258: em2:rx0}
12 root -72 - 0K 432K WAIT 3 22:44 4.79% intr{swi1: netisr 3}
12 root -92 - 0K 432K WAIT 2 534:21 2.88% intr{irq261: em3:rx0}
72603 root 20 0 21856K 3928K CPU1 1 0:00 0.39% top
12 root -92 - 0K 432K WAIT 1 60:03 0.20% intr{irq259: em2:tx0}
2475 nobody 20 0 16836K 4100K select 2 0:24 0.20% darkstat
0 root -92 - 0K 304K - 0 75:03 0.00% kernel{dummynet}
12 root -92 - 0K 432K WAIT 3 71:32 0.00% intr{irq262: em3:tx0}
12 root -60 - 0K 432K WAIT 2 16:23 0.00% intr{swi4: clock}
0 root -92 - 0K 304K - 3 14:40 0.00% kernel{em3 rxq (cpuid 2}
5 root -16 - 0K 16K pftm 0 7:41 0.00% pf purge
31241 root 20 0 16676K 2560K bpf 1 6:58 0.00% filterlog
15 root -16 - 0K 16K - 0 4:58 0.00% rand_harvestq
0 root -92 - 0K 304K - 2 4:32 0.00% kernel{em1 que}