Could it be that your hardware is too slow?
I am asking since another member here who has the same ISP as I have experienced slow downs in his VDSL after using a rather 'light weight' hardware configuration. I have a relatively powerful configuration (brand new stuff, Intel Celeron G1610, Intel mobo/NICS, fast WD hard disk) and I don't notice any speed losses at all.