Who ya with Peter?, I am with IINet and I get that as well during the day, as soon as it gets around 8 pm until around 10pm it all turns to sh1t big time. They do not have anywhere near enough backhaul to satisfy their customer base during peak periods.
See what it reads around 9.30pm
As you can see there is a monstrous difference between peak time at 9.50pm and 12.40am. as shown in my two images...