In this post I’ll introduce some sketches* with results to explore and show you how the data was extracted.

*this is a term I’ve picked up from Tony Hirst along with explanatory and exploratory visualisations both presented in More Thoughts on a Content Strategy for Data. The other thing I have sitting heavily in my thoughts is Eric Berlow’s TEDTalk where he shows complex doesn’t always mean complicated (H/T @PaulHollins). My fear is I’m going to dump you with complicated exploratory sketches, when I should be giving you a simple explanatory answers.

Dump #1 Blooming great

For this first dump I’ve deliberately left it as low resolution as I only want to give you an overview and not analyse each node. In the graph you’ll spot dense patches of purple [A] these are made of the individual twitter screen names of people following one of the CETIS twitter accounts. So at the very top of the image there is a cluster of people following just me [B]. Other dense patches represents other groups of people following other CETIS Twitter accounts. In the centre of the main group [C] are Twitter users who follow 2 or more CETIS accounts. In Gephi by rolling over nodes it’s easy to explore who people follow. To the right of the graph [D] is the @ArchimateTool account. This cluster has fewer connections to the main CETIS following. Finally around the centre of the graph are loose groups [E] of users who follow 2 CETIS staff.

Update: Some other stats. The average out-degree in the network is 1.424 and 81% of the people in graph only follow one of the CETIS accounts. It would be interesting to see how this compares with other organisations. It’s important to also remember it’s not just about twitter (email probably still has the best reach and conversion)

[If you are desperate to explore an interactive version of this I’ve put a copy on my install of Gexf-JS viewer.

Dump #2 Many Eyes

Overall there are over 3,500 unique Twitter accounts that follow one or more CETIS staff accounts. 3,500 pairs of eyes looking at what CETIS or staff members are doing, with the potential to spread our message even further through their own networks. Here’s what a lot of those eyes look like (click for larger version on zoom.it):

I suppose the next question is do we have the right Twitter audience watching us.  A quick wordcloud of the profile description of the staff following us:

Getting the data

My regular top traffic generating blog post is Export Twitter Followers and Friends using a Google Spreadsheet which allows users to easily grab details of up to 5,000 (more if you don’t mind some code tinkering) Twitter account friend/followers. I don’t know how widely known it is but Twitter doesn’t just let you get your own friends/followers, you can get the data for any public Twitter account. So that’s what I did, snaffled details of who was following @jisccetis and JISC CETIS staff with public twitter accounts.

The way the spreadsheet is set up it generates a separate sheet for each persons follower details. To make it easy to import into Gephi/NodeXL I wrote this short script:

Here’s a copy of the modded spreadsheet. To use File > Make a copy, run through the authentication instructions, grab some follower details from different accounts then run Twitter > Combine follower sheets. If you’re going to be using Gephi last thing you should do before downloading as csv is change the column heading on the ‘combined’ sheet from screen_name to source.

Using Gephi

The best way I’ve found to get the data in Gephi is start a new project and then use the Import Spreadsheet option in the Data Laboratory pointing it to the csv file downloaded from Google Spreadsheet. I’ll let you play with manipulating the data. If you come up with any nice recipes please share ;)

Using NodeXL

Open a blank NodeXL template and then open the downloaded csv in Excel as a new workbook, then from the NodeXL ribbon Import > Open workbook. Its worth ticking the extra columns as vertex 1 properties. Again I’ll let you play, any recipes please share (the many eyes image was generated by switching the nodes to image and using the profile_image_url extracted using the Google Spreadsheet and using a grid layout. If anyone has worked out how to using images as nodes in Gephi I’d be very interested to hear).

So what

I avoided going into any deep analysis with this as there are probably internal discussions to be had, such as, should we be targeting college staff more? What I hope this posts illustrates is it’s relatively easy to extract this type of data and start to get the very beginnings of some answers (e.g. how many unique followers do we have). There still a lot to unpick in this area so I’m sure I’ll be revisiting. My question to you is if you were doing this type of study what answers would you be looking for?

Tony came up with this brilliant solution for plotting twitter hashtag community graphs (it was brilliant because it used Google Social Graph API to get friend/follower relationships, no authentication required). Then I came along and dumped his code into Google Spreadsheet and wrapped the output in an embeddable gadget. Google then launched Google Plus which appears to coincide with a deterioration in friend/follower information in Social Graph (one presumes my social network is better than yours type posturing).

So now here’s a new version of my spreadsheet which plays nice with Twitter and uses API authentication to get friend follower information:

*** D3 Twitter Community Visualizer ***

The big news probably isn’t the spreadsheet (yawn), but how the data,shown below if your RSS reader hasn’t stripped it, is rendered. It was achieved by wrapping a version of my EDGESExplorer tool in a Google Gadget. This means if you’ve got any Google Spreadsheet with two columns of edge data (target/source) you can insert this gadget, display the data and embed elsewhere (if your host allows <script> inserts).

To use the gadget in other Google Spreadsheets select Insert > Gadget > Custom then enter the url http://hosting.gmodules.com/ig/gadgets/file/108150762089462716664/edgeexplorer.xml

One thing to bear in mind in the gadget is unverified. The main impact of this is if you ‘Publish to the web’ or ‘File > Share’ viewers won’t see anything (but as can seen above publish embed works). I’ve tried to get the gadget verified but the send it to us link is broken (I’ve logged it as an issue so hopefully it will be fixed soon).

I have a job to go to in the morning so I’m going to let you ask/tell me it doesn’t work using the comments ;)

In this post I’m going to have another look at getting sentiment analysis data for a tweet stream and make a comparison between two free APIs (ViralHeat and Text-Processing) with some manually sentiment analysed data. In conclusion the Text-Processing Sentiment API data proved to be more accurate at calculating overall sentiment than ViralHeat, but both had a 18% probability of detecting the right sentiment within a tweet.    

Method

Having already looked at Using the Viralheat Sentiment API and a Google Spreadsheet of conference tweets to find out how that keynote went down I decided to reuse the data obtained from the tweet stream during Donald Clark’s ALT-C 2010 keynote.

ViralHeat

This data was generated my sending individual tweets (n. 561) to the ViralHeat Sentiment API, getting a positive or negative classification combined with a probability factor between 0 and 1 indicating confidence level. The positive/negative classifications were turned into numeric values by converting ‘positive’ to +1 and ‘negative’ to –1 and multiplying by probability. These values were accumulated and graphed to plot how tweet sentiment changed over time.

Text-Processing

Using a slight modification to the Google Apps Script used to collect ViralHeat data the same set of tweets were based to the Text-Processing Sentiment API. The data returned for each tweet consisted of the label ‘pos’ (positive), ‘neg’ (negative) or ‘neutral’ and probability for all of the labels. The label returned was determined by the API using a hierarchical classification whereby neutrality is calculated as a standalone value. If neutrality has a probability greater than 0.5 then the label returned is ‘neutral’. Positive and negative sentiment probability are balanced to add up to 1. To illustrate this below is an example of the data returned:

{
        "probability": {
                "neg": 0.39680315784838732,
                "neutral": 0.28207586364297021,
                "pos": 0.60319684215161262
        },
        "label": "pos"
} 

Similar to the ViralHeat data positive and negative labels were turned into numeric values by multiplying probability by +1 for positive and –1 for negative labels (neutral labels were ignored n.161). These values were also accumulated  

Manual Analysis

The 561 tweets were printed off as a table with no extra data. Each tweet was individually assessed (by me – not trained in social research in anyway, btw) and marked as having positive, negative or neutral sentiment. These values were entered into the spreadsheet and converted to 1, –1 or 0 for positive, negative and neutral. Like the Text-Processing data neutral tweets were ignored (n. 422) leaving 139 data points

Results

The graph below shows the accumulated sentiment scores for the manual analysis compared to the results from Text-Processing and ViralHeat plotted against the time the tweet was made (09/07/2010 09:20:12 to 09/07/2010 10:21:55) [click to see interactive version – takes a couple of seconds to appear].

The graph shows that the ViralHeat scores display an overall positive trend whilst the manual analysis and Text-Processing scores show an overall negative sentiment in the tweets, perhaps suggesting that the Text-Processing data is more accurate at calculating overall sentiment than ViralHeat with the given data. 

As the ViralHeat data doesn’t appear to have hierarchical classification, which indicates if text is neutral, I decided to filter the data and only use results which had a probability factor within the top 10%. This resulted in a subset of ViralHeat data (n.56) which had a probability of greater than 0.987 and Text-Processing (n.40*) with a probability greater than 0.867.

*less because 161 data points were already excluded as they were neutral   

The graph which includes only the top 10% of sentiment scores from ViralHeat and Text-Processing is shown below [click to see interactive version – takes a couple of seconds to appear]. The overall effect of this filter is the overall positive and negative sentiment scores for both ViralHeat and Text-Processing are reduced (this is expected as their are less data points). 

Whilst it looks like Text-Processing is more accurate at detecting sentiment within a tweet on a tweet-by-tweet basis it’s a different story. When I manual recorded sentiment data for each tweet within the dataset I was able to assign 139 with either positive (n.56) or negative (n.83) sentiment. Text-Processing returned positive (n.81) and negative (n.319) for 400 tweets and ViralHeat returned positive (n.296) and negative (n.265) for 561 of the tweets. When comparing the returned sentiment with the manually processed tweets Text-processing returned 72 (18%) results  with matching sentiment and ViralHeat 99 (18%).

Conclusion

Hmm. Sentiment analysis of individual tweets might only be 20% accurate using off the self tools with the data I looked at. This however might be enough if you are looking to filter large volumes of data. Both ViralHeat and Text-Processing have options that could potentially increase the accuracy by for example using a different text classification (it also worth noting that Text-Processing is built using the open source Python Natural Language Toolkit so even more tweaking is possible). 

I’m sure more conclusion can be made so why not have a look at the data I’ve captured and leave some notes in the comments ;)

It doesn’t have to be this way. Using a combination of Google Spreadsheets as a data source and a simple web interface to add interactivity it’s possible to let users explorer your entire event hashtag and replay any of conversations.

View example conversation replay

Try out a LIVE version

Update: If you are still struggling to understand the concept Radical Punch have done a overview of this tool

Here’s how to archive event hashtags and create an interactive visualization of the conversation (written instructions below):

Twitter: How to archive event hashtags and create an interactive visualization of the conversation

Capturing the tweets

Use this Google Spreadsheet template.

For more reliable data collection it’s recommended that you follow the steps to get authenticated API access to Twitter search results and setup a ‘script trigger’ to automate collection. Here are instructions on how to do it:

1. Open the TAGS Google Spreadsheet making a copy
2. Register for an API key with Twitter at http://dev.twitter.com/apps/new. In the form these are the important bits:
• Application Website = anything you like
• Application Type = Browser
• Callback URL = https://spreadsheets.google.com/macros
• Default Access type = Read-only
3. Once finished filling in the form and accepting Twitter’s terms and conditions you’ll see a summary page which includes a Consumer Key and Consumer Secret
4. Back in the Google Spreadsheet select Twitter > API Authentication (you’ll need to select this option twice, the first time to authorise read/write access to the spreadsheet). Paste in your Consumer Key and Secret from the previous step and click ‘Save’ (if the Twitter menu is not visible click on the blue button to show it)
5. From the spreadsheet select Tools > Script Editor … and then Run > authenticate and Authorize the script with Twitter using your Twitter account
6. While still in the Script Editor window select Triggers > Current script’s triggers… and Add a new trigger. Select to run ‘collectTweets’ as a ‘Time-driven’ choosing a time period that suits your search (I usually collect 1500 tweets once a day, but increase to hourly during busy periods eg during a conference). Click ‘Save’
7. Now close the Script Editor window. Back in the main spreadsheet on the Readme/Settings sheet enter the following settings (starting in cell B9):
• Who are you = any web address that identifies you or your event
• Search term = what you are looking for eg #jiscel11
• Period = default
• No. results = 1500 (this is the maximum Twitter allows)
• Continuous/paged = continuous
8. Click TAGS > Run Now! to check you are collecting results into a ‘Archive’ sheet
9. To allow the results to be visualised from the spreadsheet select File > Publish to the web… You can choose to Publish All sheets or just the  Archive sheet. Make sure Automatically republish when changes are made is ticked and click Start publishing

Creating a public interactive visualisation of the archived tweets

1. Copy the url of the spreadsheet you just created
2. Visit http://hawksey.info/tagsexplorer and paste your spreadsheet url in the box, then click ‘get sheet names’
3. When it loads the sheet names leave it on the default ‘Archive’ and click ‘go’
4. You now have a visualisation of your spreadsheet archive (click on nodes to delve deeper)
5. To share the visualisation at the top right-click ‘link for this’ which is a permanent link (as your archive grows and the spreadsheet is republished this visualisation will automatically grow)

Update: Some of these graphs may take a couple of seconds to appear [H/T @johnmclear]

Update 2: The interactive #fote community was freezing Internet Explorer so click through to see interactive version [H/T James Swansburg]

Update 3: Added a second snapshot of the Twitter conversations to cover whole day

Update 4: Coded the last snapshot with sentiment data from ViralHeat

Summary Data

Guages

Archive of Tweets

Here’s a archive of the tweets from the event captured in a Google Spreadsheet.

Sentiment

Open in new window

#fote11 community

[The interactive version was freezing Internet Explorer so click through to see interactive version H/T James Swansburg]

#fote11 conversations (made with NodeXL)

Snapshot below taken at midday
Node size = number of tweets with #fote11
Graphing 1622 tweets

Snapshot below taken post event
Node size = number of tweets with #fote11
Graphing 2649 tweets

#fote11 conversations coded with sentiment data from ViralHeat

Last graph for now. Using a combination of my Using Google Spreadsheets as a data source to analyse extended Twitter conversations in NodeXL (and Gephi) and Using the Viralheat Sentiment API and a Google Spreadsheet of conference tweets to find out how that keynote went down I was able to import sentiment data for each tweet (edge). I then accumulated the sentiment probability for each twitterer (vertex), -1 = 100% probability of negative sentiment +1 = 100% probability of positive sentiment, and averaged the overall sentiment by the number of tweets that person (vertex) made. Using the autofill the vertices were coloured by average sentiment probability (green = +ve, red = –ve) and grouped by overall positive sentiment, overall negative sentiment and no data.

The graph shows that over 80% (n.296) of #fote11 hash taggers (n.355) posted tweets with an overall positive sentiment detection.

Somethings worth noting with this data. Sentiment analysis is being analysed using machine detection (ie it might be wrong). Someone with overall negative sentiment doesn’t necessarily indicate that they had a bad event experience. If the person was reflecting on issues being presented or quoting others who had a negative experience this will be reflected in their sentiment score. The bottom line is the graph gives an overview of a more complex story. If you want to start unpicking that story yourself the GraphML data is available on the NodeXL GraphGallery.

RT @psychemedia: How do folk who send you twitter messages connect? http://bit.ly/dNoKGK < see address bar (this is depressingly good)

Was what I tweeted on the 13th April 2011. What Tony had managed to do was use the Protovis JavaScript library to let people make their own network visualizations for Twitter search terms (if you haven’t seen Tony’s other work on hashtag communities, mainly using Gephi, its well worth a read). The concept for the latest iteration was explained in more detail by Tony in  Using Protovis to Visualise Connections Between People Tweeting a Particular Term.

Two limitations of this solution are: it relies on a current Twitter search which will disappear after 7 days; and it’s difficult to embed elsewhere. Tony and I had discussed collecting the data in a Google Spreadsheet using one of my many Twitter/Spreadsheet mashups and then draw the network visualization from the data.

I thought I would go one step further not only collecting the data in the Spreadsheet but then also generate the Protovis visualization in the sheet by making a gadget for it. The reason for going down the gadget route is in theory they provide an easy way to embed the visualization in other webpages and web apps.

This wouldn’t be my first foray into gadgets having already done My first Google Gadget to embed a flickr photoset with Galleria navigation, and I already knew that to gadgetize Tony’s original code just needed some xml wrapping and a dash of the Google Visualization API. In fact because I used Google Apps Script to collect the data there was very little needed to do with Tony’s code as both used JavaScript syntax.

So here it is the:

*** Protovis Twitter Community Visualizer ***
[If the link above doesn't work open this version and File > Make a copy (make sure you are signed in)]

and here is some output from it (if you are reading this in an RSS aggregator you’ll need to visit this post) Update: Turns out the Protovis library doesn’t work with IE so you’ll just have to use a proper browser instead:

PS you can interact with the diagram by click-dragging nodes, using your mousescroll for zoom and click-drag empty parts to pan

Life is so much easier when you stand on the shoulders of giants ;)

I used to use my TwitterPad wordpress plugin to archive everything I said on Twitter. The main reason was often I find myself trying to remember links/resources I’ve previously tweeted. It worked well but every month I had to manually paginate my tweets to prevent the post getting too long.

Having got bored with this chore I wrote a Google Spreadsheet which interacts with the Twitter API to pull my timeline grabbing the last 3200 tweets. Part of this API allows you to just grab new updates since a tweet id (a similar parameter is available in the Twitter Search API used in Collect/backup tweets in a Google Spreadsheet on one sheet). Using this means I can setup a trigger in Google Apps Script to refresh my archive.

All was going well until I noticed it was pulling in duplicate entries. Part of the problem is Google Spreadsheet thinks 65078736491511804 is 65078736491511808. This meant the script was passing the wrong since id.

You can test this yourself by creating a new Google Spreadsheet and in the first cell [A1] enter 65078736491511804. With this cell highlighted if you select Format > Number and try to change it to nothing happens. This is because Google know there is a floating-point error and store the value as a text string.

If in the cell beneath the number (uh string ;) [A2] you enter the formula =A1+1 you get 65,078,736,491,511,808 (if you get 6.51E+16 with the cell highlighted change Format > Number to Normal)

So if you want to start archiving your or someone else’s tweets you can make a copy of my Export/Archive Twitter Status Updates Google Spreadsheet. The health warning is there maybe missing or duplicates tweets. I have posted on the Google Apps Script forum to see if there is a way around this.

I would like to say that this idea has gone down like a storm, but … well according to the Docs Template Gallery its got 17 users – 5 star rating though ;) that might be my vote :( I wonder if this is so low because people can make a copy of the spreadsheet from the link in the original post?

Ignoring this overwhelming success there was one tweak I was meaning to do for a while. Currently the Spreadsheet dumps the results from it’s search into a new sheet each time it runs. This means you can end up with lots of duplicate tweets across different sheets. So when Tony Hirst recently asked:

@mhawksey do you have a twitterlytics archiver that appends tweets to one sheet rather than form a new sheet for each run? 28 Apr 2011

the time felt right to do something about this. To get the ball rolling Tony also kindly contributed a coding snippet which finds new tweets from the search results. This solution is a bit greedy running a search collecting up to 1500 items then comparing with what is already there. Meanwhile I was looking at the Twitter Search API documentation to see if I could filter the results to remove previous results (the answer was yes). The disadvantage of this solution is the results can’t be paged so you are limited to 100 results. 

The latest version of the spreadsheet has been included in the Collect/backup tweets in a Google Spreadsheet post (it defaults to using the twitter API to work out the last tweet, but the greedy version is included in the source code). Below is a copy of the main bits of the code. Thanks to Tony for the prod and code suggestion!

function collectTweets() {
  // if continuous sheetname = archive else make a name
  if (RESULT_TYPE == "continuous"){
    var sheetName = "Archive";
  } else {
    var sheetName = Utilities.formatDate(new Date(), "GMT", "dd-MMM-yy hh:mm"); //make a new sheet name based on todays date
  }
  // if sheetname doesn't exisit make it
  if (!ss.getSheetByName(sheetName)){
    var temp = ss.getSheetByName("TMP");
    var sheet = ss.insertSheet(sheetName, {template:temp});
  } else {
    var sheet = ss.getSheetByName(sheetName);
  }
  // else if already have archive get since id
  var stored = getRowsData(sheet);
  if (stored.length >0 ){
    var sinceid = stored[0]["id"];
  } 
  // prepare search term
  var myUntil=new Date();
  myUntil.setDate(myUntil.getDate()-1);
  var mySince = new Date(myUntil);
  if (SEARCH_DURATION == " weekly"){
    myUntil.setDate(myUntil.getDate()+1);
    mySince.setDate(mySince.getDate()-6);
  }
  //if no since id grab search results
  if (sinceid){
    var data = getTweets(SEARCH_TERM, mySince, myUntil, NUMBER_OF_TWEETS, sinceid); // get results from twitter sinceid
  } else {
    var data = getTweets(SEARCH_TERM, mySince, myUntil, NUMBER_OF_TWEETS); // get results from twitter
  }
  // if some data insert rows
  if (data.length > 0){
    sheet.insertRowsAfter(1, data.length);
    setRowsData(sheet, data);
  }

  /*
  // This is a very greedy grab (the limit of since id is it has max of 100 results, this method gives you 1500)
  var data = [];  
  if (sinceid){  
    for (var i = 0; i < objects.length; i++){
      var row = objects[i];
      if (row.id > sinceid){
        data.push(row);
      }
    }
    if (data.length > 0){
      sheet.insertRowsAfter(1, data.length);
    }
  } else {
    data = objects;
  }
  */
}

function getTweets(searchTerm, since, until, maxResults, sinceid, languageCode) {
    //Based on Mikael Thuneberg getTweets - mod by mhawksey to convert to json
    // if you include setRowsData this can be used to output chosen entries
    try {
        var pagenum = 1;
      var data =[];
      var idx = 0;
      var sinceurl ="";
        if (typeof maxResults == "undefined") {
            maxResults = 100;
        }
        if (maxResults > 1500) {
            maxResults = 1500;
        }
        if (maxResults > 100) {
            resultsPerPage = 100;
            maxPageNum = maxResults / 100;
        } else {
            resultsPerPage = maxResults;
            maxPageNum = 1;
        }
        if (typeof sinceid != "undefined") {
            if (sinceid.length > 0) {
                resultsPerPage = 100;
                maxPageNum = 1;
                sinceurl = "&since_id=" + sinceid;
            }
        }
        Logger.log(twDate(since)+" "+twDate(until));
        searchTerm = encodeURIComponent(searchTerm);
        for (pagenum = 1; pagenum <= maxPageNum; pagenum++) {
            var URL = "http://search.twitter.com/search.json"
            URL = URL + "?q=" + searchTerm;
            URL = URL + "&since=" + twDate(since);
            URL = URL + "&until=" + twDate(until);
            URL = URL + "&rpp=" + resultsPerPage;
            URL = URL + "&page=" + pagenum;
            URL = URL + "&result_type=recent";
            URL = URL + sinceurl;
            if (typeof languageCode != "undefined") {
                if (languageCode.length > 0) {
                    URL = URL + "&lang=" + languageCode;
                }
            }
            
            var response = UrlFetchApp.fetch(URL, {method:'get', headers: { "User-Agent": REFERER}});
     
            if (response.getResponseCode() == 200) {
              var objects = Utilities.jsonParse(response.getContentText()).results;
              
              for (i in objects){ // not pretty but I wanted to extract geo data
                if (objects[i].geo != null){
                  objects[i]["geo_coordinates"] = "loc: "+objects[i].geo.coordinates[0]+","+objects[i].geo.coordinates[1];
                }
                objects[i]["status_url"] = "http://twitter.com/"+objects[i].from_user+"/statuses/"+objects[i].id_str;
                data[idx]=objects[i];
                idx ++;
              }
            }      
          }         
    return data;
    } catch (e) {
        return e.message;
    }
} 

Note: Twitter recently wrote to me reminding me that

methods of obtaining and exporting Twitter content obtained via the Twitter API is prohibited by section I.4.A of our API Terms of Service (https://dev.twitter.com/terms/api-terms ), specifically:

You will not attempt or encourage others to sell, rent, lease, sublicense, redistribute, or syndicate access to the Twitter API or Twitter Content to any third party without prior written approval from Twitter. If you provide an API that returns Twitter data, you may only return IDs (including tweet IDs and user IDs). You may export or extract non-programmatic, GUI-driven Twitter Content as a PDF or spreadsheet by using “save as” or similar functionality. Exporting Twitter Content to a datastore as a service or other cloud based service, however, is not permitted.

Following clarification from Twitter I making this template available again but please note.

By using this template you agree it is for personal use only and the data is not made publicly available.

Recently I’ve noticed a growing number of people arrive at this blog having searched for ‘export twitter followers’. Rather than them leaving disappointed here’s a Google Spreadsheet I threw together which allows you to grab a copy of your friends/followers:

*** Google Spreadsheet to Export Twitter Friends and Followers ***

Benefits of using Google Spreadsheet

• Control – You register for your own API key with Twitter so you have full control of the account
• Playing with the data – as you are importing straight into a spreadsheet you can do all of your own data manipulation like sorting, filtering and creating your own formula for things like follow/follower ratios
• Backup – Google Spreadsheets allow you to download copies of spreadsheets in different formats
• Share – You can make your lists of friends/followers easily viewable

Where’s this all going?

Having already done other things with the Twitter API and Google Spreadsheets (See Populating a Twitter List via Google Spreadsheet … Automatically!, Collect/backup tweets in a Google Spreadsheet, Google Apps Script, Spreadsheets, Twitter and Gadgets) the Twitter/Google Spreadsheet back is well and truly broken. You’ll probably see fewer posts one this area with new stuff instead I’ll probably start properly documenting the little code snippets I use (but if you have any interesting ideas you want help with get in touch).

This doesn’t mean I’ll be walking away from Google Spreadsheets. As recent posts like Turning Google Spreadsheets into a personal or group bookmarking service, show there is huge scope in using Spreadsheets as a very flexible rapid development platform.

Below are some bits of the code used in my new spreadsheet (all the code is viewable via the Script Editor in the Spreadsheet):

function tw_request(method, api_request){
  // general purpose function to interact with twitter API
  // for method and api_request doc see http://dev.twitter.com/doc/
  // retuns object
  var oauthConfig = UrlFetchApp.addOAuthService("twitter");
  oauthConfig.setAccessTokenUrl(
      "https://api.twitter.com/oauth/access_token");
  oauthConfig.setRequestTokenUrl(
      "https://api.twitter.com/oauth/request_token");
  oauthConfig.setAuthorizationUrl(
      "https://api.twitter.com/oauth/authorize");
  oauthConfig.setConsumerKey(getConsumerKey());
  oauthConfig.setConsumerSecret(getConsumerSecret());
  var requestData = {
        "method": method,
        "oAuthServiceName": "twitter",
        "oAuthUseToken": "always"
      };
   try {
      var result = UrlFetchApp.fetch(
          "https://api.twitter.com/1/"+api_request,
          requestData);
      var o  = Utilities.jsonParse(result.getContentText());
    } catch (e) {
      Logger.log(e);
    }
   return o;
}

function getFriendAndFo(sheetName){
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = ss.getSheetByName(sheetName);
  sheet.getRange(2, 1, sheet.getLastRow(), sheet.getMaxColumns()).clear({contentsOnly:true}); //clear sheet
  var cursor = "-1";
  while(cursor != "none"){ // while twitter returns data loop
    try {
      var o = tw_request("GET", "statuses/"+sheetName+".json?cursor="+cursor); // note using sheetname to build api request
      var data = o.users;
      for (i in data){ // extracting some subobjects to top level (makes it easier to setRowsData)
        if (data[i].status){
          for (j in data[i].status){
            data[i]["status_"+j] = data[i].status[j];
          }
        }
        if (data[i].screen_name){ // also build url to jump to profile page
          data[i]["profile_link"] = "http://twitter.com/"+data[i].screen_name;
        }
      }
      var headRange = sheet.getRange(1, 1, 1, sheet.getMaxColumns());
      var rowIndex = sheet.getLastRow()+1;
      setRowsData(sheet, data, headRange, rowIndex); // dump data for this loop to sheet
      if (o.next_cursor!="0"){
        cursor = o.next_cursor; // get next cursor
      } else {
        cursor = "none"; // break
      }
    }  catch (e) {
      Logger.log(e);
    }

  }
}

Recently Twitter has been paying close attention to 3rd party developer use of their API. First they stopped whitelisting new applications, then UberSocial and Twitdroid were suspended for terms of service violations, now the popular tweet archiving service TwapperKeeper announced that they will be removing export and API access in response to a request from Twitter.

Fortunately TwapperKeeper has been given until the 20th March to comply, but this is still a blow for 4th party developers, like myself, who build applications around the TwapperKeeper archive/API and will no longer have an easy way to grab historic tweets. Part of the problem is Twitter provides no easy access to collections of tweets over 5 days old necessitating the need for services like TwapperKeeper, so it’s discontinuation is going to cause headaches for some.

Personally, when developing my Twitter Subtitling service iTitle I was keen to make sure data could be used from various sources starting with the official Twitter Search results and TwapperKeeper integration, later adding support for .CSV file upload. Most recently I added support for my Google Spreadsheet/Twitter Search collection mashup [BTW after feedback from @AJCann there is now a setup video for this on YouTube]. So hopefully iTitle will live for a little longer.

So what about other services like my other Twitter mashup uTitle and Andy Powell’s Summarizr, which rely on the TwapperKeeper service. There are obviously other Twitter archive services out there, TwapperKeeper even has the JISC funded open source version yourTwapperKeeper, but all the online services potentially face the same problem of being prevented from making archived tweets reusable. One solution might be to develop a distributed cloud based archive. For example, initially I thought of using Google Spreadsheet hack for capturing tweets, and then sharing these archives with the community as a searchable index. Anyone fancy building an interface for this?