This hasn't happened to me when playing in the wee hours of the morning. Thus, it's server side lag and instability.
The servers aren't crashing for anyone else, and this isn't what happens to them when they get full. Also, the game's players come from every time zone, so it's rather unlikely that the busiest internet time for you personally happens to be the busiest internet time for the servers. Thus, no, it's not the servers.
What is enormously more likely, however, is that the busiest internet time in your time zone is also the busiest internet time for your local ISP, and that your ISP does something to deal with the increased bandwidth usage that is affecting how you connect with the game.
Talk to your ISP to find out what happens to make your connection different during your local peak usage times, and work from there. If I had to guess, I'd imagine you're running afoul of a measure designed to prevent giant video downloads from eating up all the limited bandwidth during busy times.