@koenh
This indicates that somewhere between the pfSense WAN interface, and what ever monitoring ip you are using, has some ugly packetloss.
I would shift the monitoring ip back to "default" ... (the DHCP delivered default gateway) , and maybe change the network cable between the pfSense & the ISP Box.
Your pfSense Wan IF is connected directly to the ISP box, correct ??
Maybe i'd power off/on the ISP Box, just to make a "fresh start" ...
If it continues with packet loss between pfSense Wan and the ISP default gateway , i'd contact the ISP and explain the issue.
Edit: V'man beat me to it
/Bingo