Resolved: backlog 5/19/2014

By | May 19, 2014

Starting at 6:30 AM on Monday, 5/19/2014, multiple servers were hit with a high-volume spam flood from a Russian botnet that had hijacked user email accounts.  The volume was so high that, which filters both incoming and outgoing mail, became backlogged and mail delivery became very slow.  We identified the problem shortly after 7 AM and blocked the botnet, but it took several hours to clear the unprocessed queue of this spam because we wanted to avoid accidentally deleting any legitimate mail.  This problem has been resolved and mail delivery was normal (about 3 seconds after receipt)  by 11 AM.

Our early-warning system caught this quickly, but the volume of spam from this large botnet was extraordinarily high, causing the backlog.  We’re looking at ways to detect hijacked email accounts more quickly.

Our servers do a good job of blocking the thousands of bots attempting to guess user passwords on each server every day, but if your password is a simple one, or is one that you’re using on some other system that has been compromised (Adobe, Twitter, Evernote, Dropbox, etc.), the bad guys can test and confirm your password on the first try and we won’t detect that as an attack.  If you’re using your password on any other system, consider changing it ASAP, and choose a strong password (mixture of upper/lower case letters, some numbers, some punctuation).