If you have used Twitter search before, you may notice that you can only go back a certain amount of time and/or number of tweets for a given search. In fact, if you read the Twitter search documentation, you'll note that the folks from Twitter say, "We also restrict the size of the search index by placing a date limit on the updates we allow you to search. This limit is currently around a month but is dynamic and subject to shrink as the number of tweets per day continues to grow."

Thus was born The Archivist, a new experiment from the Mix Online lab that Tim Aidlin and I cooked up recently.  Our motives for writing the app where to solve a problem that, from my research, hasn’t quite been solved before: How do you archive Twitter searches? I looked on the Twitter Apps Wiki but didn’t see anything that accomplished exactly what I was looking for. So, what can you do but write it yourself?

The Archivist is a Windows application that runs on your local system and allows you to archive tweets for later data-mining and analysis for a given search.

There are two screens. First the main screen, which is a big list of tweets:

The Archivist allows you to start a search and will get as many results as it can on the initial search.  If you leave The Archivist open, it will update with the latest results every 10 minutes.  You can also close The Archivist and open it later. The Archivist will save the tweets and get all the tweets it can since that search.

Then, there’s a graph that shows number of Tweets a day:

The Archivist will display a chart that shows the number of tweets per day for a given search, so that you can quickly assess traffic for a given search. For more comprehensive data analysis, The Archivist lets you export Tweets to Excel. It also natively saves tweets in an XML format, which could also be parsed  for deeper data analysis.

Install The Archivist today! (Requires .NET Framework 3.5 SP1.)

To learn more about The Archivist, read the documentation.

For details about the development of The Archivist, check out this post.

Oh, and follow Mix Online on Twitter!