Ping.fm, one of the essential sites for my social networking services went down a few hours ago. I thought it was all just a minor hiccup, until I read their blog: they lost 5 days of data!
At around 11AM CDT, one of our database servers decided to take a leave of absence. What seemed like a routine "reboot and go" situation turned out to be much much worse than assumed.
We run daily snapshots of databases to ensure data integrity for situations like this. About 6 days ago, the backups started to crash out and not save properly.
So what does this mean for its registered subscribers and users? If you click on all my tweet links, they'll go to Ping.fm's page, not to the one that I intended for you to read. Therefore all the links in my tweets in the last five days are totally gone and it seems there's no way it'll return.
This reminded me of what happened to Ma.gnolia: a once Delicious bookmarking competitor that went open-source. They had it worse as their whole database got fried to hell and never came back. Almost all their users wanted to kick their heads in as their bookmarks are now all lost. The main problem was the same as Ping.fm's: backups. At least with their case, they still had the previous five days, but I hope they'll do it at the most every two days.
I'm disappointed but will still continue to use their service. Hopefully this won't happen again.
