BeTwittered Is Back, And here's the story… 9

Sorry for the down-time.  BeTwittered is back, again.  I notified the folks at Twitter as soon as I diagnosed the problem on Friday, January the 30th.  Being the second time this happened, it only took me a few minutes to diagnose and then notify Twitter.  There were some unfortunate delays, but I must say that today, @al3x apologized for the delay, got on top of the issue, and made sure that there was follow through.  Shortly after we chatted via email about the status this morning, there was obvious action.  It’s Alive!  @al3x really provided me (us!) with good customer service in the end.

The longer version of the story.

The reason for the down-time?  Twitter is pretty popular (you probably already know that!) and many people have written programs to do everything from act as a client (like BeTwittered) to attempt to read and process EVERY tweet that passes through Twitter. They then do all manner of data manipulations to provide info like who’s the most popular, the most chatty, who’s following who, where people chat from most often, etc.

Now, there are many ways to write a Twitter client. One, and the most common:  write a Windows/Mac/Linux application that runs on your PC.  We gleefully install these, and they run on our PC.  We send and receive Tweets, and they travel between our PC’s and the Twitter servers, directly.  That’s nice, but sometimes it’s nice to have another type of “no install” web based client.  This other type of Twitter client, like BeTwittered, is actually just a web page.  When we use BeTwittered, our tweets go directly between BeTwittered.com and Twitter.com.  The key thing to note here is that this is a SERIOUS flood of traffic, and it’s not spread out between all of our PC’s. There are tens of thousands of BeTwittered users, now (Much to my surprise, I must say.  I wrote BeTwittered because I thought it would be a nice thing to have, and I wanted to learn to write programs, especially web programs)

Now for a bit of conjecture mixed with a bit of my knowledge.  Twitter saw the huge amount of traffic being requested by BeTwittered.com and most certainly made an obvious assumption.  BeTwittered was being a rude Twitter citizen and just appeared to be trying to pull huge amounts of data for it’s own purposes.  If I were to design an app that was going to analyze large amounts of Twitter data, I would make an arrangement with the Twitter team and ask them how they’d like my server to “drink from the fire hose,” as it’s called.  My mistake was not realizing that the huge growth of BeTwittered had crossed a line at some point.  It started looking (and being?) a bit greedy and rude.   So they cut us off, not too surprising in hindsight.

I now know that they weren’t aware that BeTwittered was just aggregating the traffic of thousands of people.  I look back and understand that I didn’t actually make this clear after getting banned the first time.  This time, there was a bit of a “eureka!” moment when I communicated this properly.  Also complicating things a bit was a Twitter change of policy that happened between the 1st and 2nd “blacklisting”  They instituted a limit on hourly requests to the twitter servers, one that applies to high traffic generating servers like BeTwittered.com.  BeTwittered apparently burst through the gates like a track star and crossed that limit almost immediately.

When I get a chance, I’ll be looking much harder at caching more requests to Twitter.  I’m hoping to find ways of pruning any unnecessary requests that there might be. That’s a complicated business though, since I need to avoid meddling with legitimate requests… deceptively picky stuff to mess with.

I wonder now, about open vs closed systems.  Twitter is private and closed in the sense that it’s a private commercial business with openings only where they choose to allow them.  They are VERY open about sharing their data, though, as things stand today.  I must give them that.  The problem that looms large:  What happens, though, when lots of people depend on a closed service, and the terms of service suddenly change?  I’m not saying twitter would do this,  and to be fair, they didn’t even have to talk to me OR restore BeTwittered service.  But what if we depend on a communications system like Twitter and then the rug gets pulled out from beneath us?  Email is an example of an open system.  We don’t have to stick with our Internet provider to communicate with our friends.  You can be on AT&T and I can be on Comcast or even AOL (ick) and still send email back and forth.  There is, by the way,  identa.ca!  Like twitter, but open. We’d have to all move over there, though.  What’s twitter without our friends?  I’m digging a bit deep here, I admit.  We do need to keep an eye open for these things, though.

Oh, well, all better now.   Off with my tin-foil hat… and again, thanks Alex!  I appreciate the help again!

Robert
BTW – Sorry to anyone who was waiting on this explanation of the events.  I’d promised it after the first outage and just got into one of those places in life where I never seemed to get any time to do anything.