Twitter Archiving Google Spreadsheet TAGS v3

Important: Changes to Twitter API

Because Twitter are making changes to their API this version of TAGS will stop working in March 2013. TAGSv5.0 is compatible with the new API and you should get it here! 

For existing TAGS users:

What will happen to my existing TAGS sheets?

When Twitter turn off the old API (test outages this March) all authenticated and unauthenticated search requests will stop working.

How do I upgrade my existing version of TAGS?

As I can’t push an update to existing copies of TAGS you’ll have to manually update by opening your spreadsheet, then opening Tools > Script editor… and replacing the section of code that starts function getTweets() { and finishes 134 lines later (possiblly with the line function twDate(aDate){ ) with the code here. [And yes I know that’s a pain in the ass but best I could do] … or you can just start a new archive using TAGSv5.0

To support my research in Twitter community visualisation I’ve updated my Twitter Archiving Google Spreadsheet (TAGS) [formerly called twitteralytics – I pre-emptively changed the name to play nice with Twitter ToS].

This new version has some coding improvements and new features including a dashboard summary and advanced tools for getting user profile information and  friend/follower relationships for social network analysis.

You can get a copy by selecting one of the links below [make sure you are looked in to your Google account first]:

*** Twitter Archive Google Spreadsheet – TAGS v3.0 ***
*** Twitter Archive Google Spreadsheet – TAGS v5.0 ***
[Please rate it in the Template Gallery]
[If the first link doesn't work try making a copy from the Template Gallery
or Opening this Spreadsheet and File > Make a copy]

Below are the setup instructions for TAGS v3.x and may not work with TAGS v5.0

Basic setup/use

  1. Open the TAGS Google Spreadsheet and copy
  2. On the Readme/Settings sheetenter the following settings (starting in cell B9):
    • Who are you = any web address that identifies you or your event
    • Search term = what you are looking for eg #cetis12
    • Period = default
    • No. results = 1500 (this is the maximum twitter allows but without authenticated access you might get less. See the Advanced setup for info on configuration)
    • Continuous/paged = continuous
  3. To configure the spreadsheet to automatically update select Tools > Script Editor … and then in the Script Editor window select Triggers > Current script’s triggers… and Add a new trigger. Select to run ‘collectTweets’ as a ‘Time-driven’ choosing a time period that suits your search (For unauthenticated access I collect 1500 tweets every hour). Click ‘Save’
  4. The collection can manually be trigger by TAGS > Run Now! (Results appear on the 'Archive' Sheet).

Advanced setup/use

  1. Open the TAGS Google Spreadsheet and make a copy
  2. Register for an API key with Twitter at http://dev.twitter.com/apps/new. In the form these are the important bits:
    • Application Website = anything you like
    • Application Type = Browser
    • Callback URL = https://spreadsheets.google.com/macros
    • Default Access type = Read-only
  3. Once finished filling in the form and accepting Twitter’s terms and conditions you’ll see a summary page which includes a Consumer Key and Consumer Secret
  4. Back in the Google Spreadsheet select Twitter > API Authentication (you’ll need to select this option twice, the first time to authorise read/write access to the spreadsheet). Paste in your Consumer Key and Secret from the previous step and click ‘Save’ (if the Twitter menu is not visible click on the blue button to show it)
  5. From the spreadsheet select Tools > Script Editor … and then Run > authenticate and Authorize the script with Twitter using your Twitter account
  6. While still in the Script Editor window select Triggers > Current script’s triggers… and Add a new trigger. Select to run ‘collectTweets’ as a ‘Time-driven’ choosing a time period that suits your search (I usually collect 1500 tweets once a day, but increase to hourly during busy periods eg during a conference). Click ‘Save’
  7. Now close the Script Editor window. Back in the main spreadsheet on the Readme/Settings sheetenter the following settings (starting in cell B9):
    • Who are you = any web address that identifies you or your event
    • Search term = what you are looking for eg #cetis12
    • Period = default
    • No. results = 1500 (this is the maximum twitter allows)
    • Continuous/paged = continuous
  8. Click TAGS > Run Now! to check you are collecting results into a ‘Archive’ sheet
  9. To allow the results to be visualised from the spreadsheet select File > Publish to the web…You can choose to Publish All sheets or just the  Archive sheet. Make sure Automatically republish when changes are made is ticked and click Start publishing

Creating a public interactive visualisation of the archived conversation

  1. Copy the url of the published spreadsheet
  2. Visit http://hawksey.info/tagsexplorer and paste your spreadsheet url in the box, then click‘get sheet names’
  3. When it loads the sheet names leave it on the default ‘Archive’ and click ‘go’
  4. You now have a visualisation of your spreadsheet archive (click on nodes to delve deeper)
  5. To share the visualisation at the top right-click ‘link for this’ which is a permanent link (as your archive grows and the spreadsheet is republished this visualisation will automatically grow)

Quick way to display archive community (Links) data

    1. Run TAGS Advanced menu options (1-3)
    2. Copy the url of the published spreadsheet
    3. Visit http://hawksey.info/edgesexplorer and paste your spreadsheet url in the box, then click‘get sheet names’
    4. When it loads the sheet names leave it on the default ‘Links’ and click ‘go’

 

221 thoughts on “Twitter Archiving Google Spreadsheet TAGS v3

  1. Pingback:

    Collect/backup tweets in a Google Spreadsheet [Twitteralytics v2] – MASHe

    1. Martin Hawksey

      Post author

      @Tom ah the Twitter Search API reads location using a geoencode parameter rather than the search term [ref] A quick fix would be to add the geoencode parameter in the getTweets function

      Reply
  2. David Boyle

    This is absolutely amazing! Thanks so much. I’m going build daily reporting and analysis based on this for some projects I’m working on. THNAKS!!!!

    Reply
  3. Pingback:

    A way to archive and display Twitter hashtag chats – MASHe

  4. AT

    Is there a way to extract geocoordinates with the tweets? My copy of TAGS has a geo_coordinates column that has come out largely empty.

    Reply
  5. Ben

    This was working GREAT until today whereby the test runs and will return data, but when it actually runs it returns a blank sheet. What is going on and how can I fix it?

    thanks for this AMAZING template!

    Ben…

    Reply
    1. Ben

      Ok after reading more It seems i might be getting rate limited by twitter. So i did the API authentication and now when I run it i get an error of:
      Oops
      Authorisation is required to perform that action.

      Ben…

      PS. Authorisation should be Authorization

      Reply
      1. Martin Hawksey

        Post author

        Hi Ben – the oops authorisation (British spelling ;) message can mean a couple of things: callback url was entered incorrectly when registering for an API on Twitter, customer secret/key have a typo, and very rarely Google doesn’t record the returned authentication token from twitter (in this scenario try running the authenticate script again, if that doesn’t work you need a fresh copy of the spreadsheet, but you can render the existing API details)
        Hope that helps,
        Martin

        Reply
  6. Ben

    I had no idea there was a British spelling of that word. how funny.

    Ok I got a new copy of the spreadsheet and followed all the instructions. I no longer get the error but I still get a blank page as the results. the test results show info, but no data is returned on the page when I run it.

    any ideas? Thanks

    Ben

    Reply
  7. Pingback:

    Follow #lis_dream3 online « LIS Research Coalition

  8. Stas

    Martin,I have a problem with advanced functions. When I run “Get Followers and Relationships” and “Replace users IDs with Names” everything seems OK except that there’s no header in Nodes spreadsheet, but there’s data. But I get blank Nodes sheet when I run “Fill Twitter user information”.

    Of course I autorize script with my Consumer and Consumer secret key.Also in Dashboard there’s no Top Twitters chart.

    Am I do something wrong?
    Stas

    Reply
  9. charlton

    HI Martin – I’m Using Tags 3.1. and having two issues. First, I’m trying to add a column (To_User) in the TMP and while I can add it, when I run and return results, the new column/data does not appear. Second (to try to get around that) I tried to use the advance tags tab option to “Replace User Ids with names” and I get the following error message: The coordinates or dimensions of the range are invalid. (line 527). Any ideas?

    Reply
    1. Martin Hawksey

      Post author

      Hi Charlton – the adding column to the TMP sheet only works if you are using paged results (this is a historic feature) Most people just use continuous which writes the tweets to the Archive sheet. So if you are using continuous collections just add ‘to_user’ to the Archive sheet and it should work. The replace id error message probably appeared if step 1 didn’t finish successfully, Edges should have a pair listing of ids and Vertices should have a list of id and usernames before trying step 2.

      Hope this helps (as you see from my blog i’m interested in Twitter analysis and it’s always nice to know what other people are getting up to ;)

      Martin

      Reply
  10. Ben

    Ok I got 3.1 and its now collecting tweets in the archive, but not when its set to paged.

    Also I have the period set for yesterday, but it pulled today’s tweets. We shall see what it pulls tonight when its set to collect them.

    Any more helpful advice?

    Ben…

    Love this spreadsheet…

    Reply
    1. Martin Hawksey

      Post author

      Hi Ben – sounds like paged results is broken. It’s a feature from v1 that I don’t use anymore which is why it’s neglected – will have a look if I get a chance but low priority for me. Re yesterday tweets. Twitter API works off GMT so my yesterday might not be the same as your yesterday. Also can be effected by Script Editor project properties (there’s a timezone setting)

      Reply
    2. Lindsey

      Hello-
      I am having the same problem that @Ben had. I was successfully using TAGSV3.1 yesterday, now it is giving me the Oops Authorization error. I have opened a new spreadsheet, API authenticated it, and tried a new search, but still the same error. Any thoughts? Thank you!

      Reply
  11. Jason Seitz

    Hi,
    Thanks! This spreadsheet is going to be a lifesaver! It is working quite well, however, I am getting this error:

    “The coordinates or dimensions of the range are invalid. (line 527)”

    when I try to run the script for replaceIDsWithNames

    What am I doing wrong? Also, is it as simple as adding a column for that variable under the Archive page?

    Reply
  12. Su Butcher

    Hi Martin,
    I do live blogging and have been using your google docs spreadsheet TAGSV3 spreadsheet to pick up hashtagged tweets – it works very well. For the first time I’m actively using the TAGSExplorer visualiser as well to visualise the conversations that happen at an event on Wednesday.
    I just wanted to thank you for sharing these tools which are really helping communicate the value of twitter to non-twitter users in the construction industry (which is my area of operation). Of course I’ve been telling loads of people about how great they are! Anything I can help you with, just give me a shout.
    Su

    Reply
    1. Martin Hawksey

      Post author

      Hi Su – many thanks for the kind words. Whilst I gave up working in the construction industry over a decade ago it still has a fond place in my heart so I’m glad you find the tool useful with what you do. Once you have the data in a spreadsheet there are many more things you can do with it. Stay tuned for updates ;)
      Thanks again
      Martin

      Reply
  13. Pingback:

    Using TAGS at LIS DREaM 3 « The Event Amplifier

  14. Pingback:

    #vircomm12: a Twitter archive at Danegeld

  15. Pingback:

    Roma, analisi di una nevicata. « Geek ma non solo!

  16. Pingback:

    5 Web Tools for #research — Cameron King

  17. Pingback:

    Visualizing #envhist | Wilko von Hardenberg

  18. Brian E. Bennett (@bennettscience)

    Martin,
    I hosted a chat last night on Twitter and I was panicking because I couldn’t find a good archive tool to handle the volume of tweets I was expecting on a Mac (Archivist works only on Windows). This template has saved my life and gives so much more information than I was looking for.
    Thank you so much for putting this together. It is a fantastic tool and I’m already excited about using it more often in the future.

    -Brian

    Reply
  19. Kevin Blocksom

    Martin,

    Let me start by saying this is an outstanding template to use for Twitter archiving.

    I have a question that has been brought to my attention from our social media group. After we gather tweets during a Twitter chat, we use this information to disseminate to various groups, but the only problem that we are having with any Twitter tool for analytics is the timezone issue. Because alot of the chats that we do are worldwide, this is an important aspect when it comes to graphing the results to know that it is differentiating and taking the timezones into account.

    Is there an easier way to do this in order to allow us to graph those results without having to go through thousands of tweets and doing this conversion by hand.

    Thanks

    Kevin

    Reply
    1. Martin Hawksey

      Post author

      Hi Kevin, not entirely sure what your use case is. Do you need to change all the dates in the archive, or does it need to change based on who is looking at the spreadsheet?
      Martin

      Reply
      1. Kevin Blocksom

        When it graph’s the results, what about someone that tweeted in the chat at 10am EST and someone tweets at 7am PST. How does that graph? Does it put both of those tweets at the 10am EST or separate and then splits them up based on the time 0900 and 1000 and does not take the time zones into consideration.

        Reply
        1. Martin Hawksey

          Post author

          Hi Kevin – results from the Twitter API are all stamped with GMT date/times which is used in the dashboard charts so in essence there is no timezone consideration
          Hope that clarifies things,
          Martin

          Reply
  20. Pingback:

    Twitter-Archive : cherchez et sauvegardez vos hashtags. « Johnny Ramos

  21. Robin T

    Hi Martin
    Thanks for the work with this – I’ve been running a sheet for a while, but the one I’ve just set up keeps giving me Apps script: Line 329 Unexpected error: Exception
    I’ve been through and checked all the settings.
    Thanks in advance

    Reply
    1. Martin Hawksey

      Post author

      Might be temporary server problems with Google or Twitter. I would try a fresh copy and see if it’s still an issue (also check that if you are using Twitter API authentication that your app hasn’t been suspended)
      Martin

      Reply
  22. Steven

    Hi Martin-

    This sheet is awesome. Is there a way to collect tweets using a combined feed instead outside of the twitter search? Ideally, I’d like to combine search feeds from yahoo pipes all in one, set a trigger for every 1 hour, and archive as many results from there as possible on a continuous basis. I mention Yahoo pipes because I’m trying to mash all the search feeds together and filter out keywords I don’t want before its outputted in the spreadsheet and archived.

    Reply
    1. Martin Hawksey

      Post author

      I find twitter feeds from yahoo pipes very unreliable. If your search isn’t too complex you can use operators in the search term field like ‘keyword1 OR keyword2 OR keyword3 -notthisone -notthisone2′. If you still want to use Yahoo pipes not aware of anything out of the box but I’d look at getting the pipe as JSON and handling with Apps Script
      Martin

      Reply
  23. Arno

    Hi Martin,

    Is it possible to conduct several queries with the same template, or should I use 1 query per template?

    I tried one query and several minutes later another one. It seemed that the archive page does not refresh with the new results, but keeps on generating results from the previous query.
    Great tool though.

    Reply
    1. Martin Hawksey

      Post author

      your query can include OR operators (e.g. searchforA OR searchforB) which should work. Manually switching between different search terms won’t work (or you can just have multiple spreadsheets – I think I have over 50 running)

      Reply
  24. AT

    Hi Martin,

    I am trying to use the TAGS advanced options and I get this error: “TypeError: Cannot call method “getRange” of undefined. (line 486)”

    line 486 on the script is: var REFERER = doc.getSheetByName(“Readme/Settings”).sumSheet.getRange(B80).getValue();

    Could you tell what is going wrong?

    Reply
  25. Raf

    Thanks for this script!
    It worked nice the first day, but now the timed execution returns an error:
    “TypeError: Cannot call method “getSheetByName” of null. (línia 177)”
    If I launch it by hand it works fine. At first I tought it was something related with the table name, but it isn’t. Don’t know what’s wrong.

    Reply
    1. Martin Hawksey

      Post author

      Hi Raf – I occasionally get timeouts but usually work fine on the next trigger. I’d try removing the script trigger and then adding it back again (making sure your using collectTweets)

      Reply
  26. Pingback:

    Automatic translation of TAGS Twitter archives using Google Apps Script ‘Language’ services – MASHe

  27. Pingback:

    #CiC12 tweets | cradledincaricature

  28. Pingback:

    #envhist: Social media and community-building in environmental history | Ant, Spider, Bee

  29. Pingback:

    MicrobiologyBytes » Blog Archive » #SGMDub Under The Microscope

  30. Pingback:

    Daily Links 03/30/2012 | EduEyeView

  31. Pingback:

    Moodle Meitheal | eLearning Island

  32. Pingback:

    AERA 2012 Summary and “Overview” | CConley.ca

  33. Pingback:

    使用Google Docs备份你的Tweets | 何必呢

  34. Tim

    Impressive piece of code – looks like it will be very useful. I tried to setup the API authentication – got my key and entered it, but when I try to run I get:

    Opps – should get a new ref before getting an old one (line 1012)

    Also doesn’t seem any way to get back to the API authentication menu config now to remove the key

    Any ideas? Running TAGS v3.1

    Best wishes,

    Tim

    Reply
    1. Sam G.

      Hi Martin,

      I am also having the same issue with API authentification “should get a new ref before getting an old one”.

      Can you help me?

      Thanks a lot for this impressive work.

      Best

      Sam

      Reply
    2. Martin Hawksey

      Post author

      For those getting ‘get a new ref before getting an old one’ errors Google changed the way the code is handled. I’m updated the spreadsheet to fix this. Old copies should continue working if already authenticated. For new archives you should take a fresh copy of the spreadsheet from the link in the most.

      Thanks for bringing this to my attention – sorry for the delay in fixing
      Martin

      Reply
  35. Pingback:

    Visualizing Tweets from the 2012 OAH/NCPH Annual Meeting — 2012 OAH/NCPH Annual Meeting

  36. Pingback:

    Twitter Archiving Google Spreadsheet TAGS v3 – JISC CETIS MASHe | Engineering Better Teachers | Scoop.it

  37. Pingback:

    Everything You Have Ever Wanted To Know About Twitter Search | Človíčci.cz

  38. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide « Buy Facebook Fans,Twitter Followers,YouTube Views,Google Plus Votes

  39. Pingback:

    #nlsled: amplifying and curating an event at Danegeld

  40. Pingback:

    User feedback to improve Conversions | distilled

  41. Pingback:

    Twitter Archiving Google Spreadsheet | CPD, by George

  42. Petter Östergren

    Really great plugging/addon API..

    But i have a small problem… Can’t see my things att the “tagsexplorer” : 0AtnDDai279KadDBoQlNTcWNIdURubUVLTkxzcS15cUE

    Reply
  43. Jonas K.

    Hi Martin,
    I really love the app you created!
    I have just a small question: Is it possible to block specific users (bots) and retweets from getting into your archive?

    Reply
    1. Martin Hawksey

      Post author

      if it’s only a couple of users you should be able to add it to the search term using the ‘-’ operator e.g. #aTag -from:botA -from:botB

      -RT might also remove retweets but the danger is it removes anything with ‘RT’ in it

      Martin

      Reply
  44. Pingback:

    Archive Your Tweets with IFTTT - ProfHacker - The Chronicle of Higher Education

  45. Pingback:

    #heasocsci at Danegeld

  46. Pingback:

    Isle Lights | markbraggins

  47. Alberto Cottica

    Small question: I am trying use the “until:” operator at the end of the research string to retrieve older (but still less old than two weeks) tweets. It does not seem to be working, though. Should I perhaps alter the “period” cell in the settings page? Not sure how to use that…

    Sorry for taking up your time, and thanks again for making this great tool

    Reply
  48. Jakes

    Quick question, how do I cope with Drive Spreadsheet size limits. I recently tried to use your amazing spreadsheet to monitor a launch we were have but hit the size limit before to end of the day. Is there anyway around this or what would be your suggestion?

    Reply
    1. Jakes

      Yeah, it might be the name because the emails don’t actually say which sheet it is, on the other hand, from the amount of emails it might be all of them and therefore just the server.

      It started on Sunday and I’ve been getting them all week, but for the sake of argument, if I did change a name, how do I fix it, do I reauthenticate or something?

      Reply
  49. Pingback:

    177-366: Not At ISTE 2012 - Interlocked Pieces

  50. Pingback:

    178-366: Hacking Google Spreadsheets - Interlocked Pieces

  51. meg

    How do I add multiple search terms? Ex. I want to search for an @username and a #hashtag. Can I put them both as terms with “OR” inbetween?

    Reply
      1. Victoria

        Hi Martin,

        Thanks very much for your tool. It is very useful!

        I am trying to collect data of three hashtags and I’m joining them by an OR between each of them but it does not seem to be working.

        Am I doing something wrong?

        Thanks in advance.

        Reply
          1. Victoria

            Now it is working properly!

            It seems that I was doing something wrong related with the API Authentication.

            Thanks for the support.

  52. Pingback:

    HEA Annual Conference 2012 – Archive of Tweets & Top Tweeters! | PlanetChemistry's Blog

  53. RV

    I have the same question as Jakes – what is the suggested procedure when the Spreadsheet reaches its size limit?

    The error I receive via email is:

    Your script, Copy of TAGS 3.1, has recently failed to finish successfully. A summary of the failure(s) is shown below. To configure the triggers for this script, or change your setting for receiving future failure notifications, click here.

    The script is used by the document Twitter TAGS – v3.1.

    Details:
    Start Function Error Message Trigger End
    7/17/12 10:46 AM collectTweets This document has reached the cell per spreadsheet limit. Create a new spreadsheet to continue working, as you may not be able to add any more rows or columns. For more information on size and complexity limits in Google spreadsheets, visit the Google Docs Help Center. (line 206)

    Many thanks

    Reply
    1. Martin Hawksey

      Post author

      You can free up some more room by deleting unnecessary sheets and columns from the archive. If you still need more room Make a copy of you original and then clear all of the archive sheet except the header and 1st row after that. Then just keep on archiving. (I find it best to remerge the archives in Excel)

      Reply
      1. RV

        Good day Marin,

        Awesome!! I appreciate your reply – thank you!

        Two additional questions:
        1. What is the citation or reference for your TwitterTAGS spreadsheet? Would it be: Hawksey, M. (2012). Twitter Archive Google Spreadsheet – TAGS v3.1. http://mashe.hawksey.info/2012/01/twitter-archive-tagsv3/

        2. Could point me in the direction of a way to search, collect, and archive (i.e., something like TwitterTAGS) that can be used for Google Scholar. If my question is not clear – please let me know.

        Many thanks,
        RV

        Reply
          1. RV

            I usually use APA 6th Ed. – however, as long as I have the basic information (i.e., name, year, title, URL, etc.) then I can cite your work and give you credit for TwitterTAGS in a manuscript.

            I’ve conducted several searches for a application that can archive/scrape Google Scholar data based on a search term and so far, no luck. Thank you for link to ReaderMeter.org!

            Your work is AWESOME – thank you for assisting researchers!

            Best
            RV

  54. Pingback:

    Free Tools to Archive Twitter Search results : Best Free Tools and Resources

  55. Pingback:

    Archive of #dh2012 Tweets « SOCIAL NETWORK ANALYSIS & SEMANTIC NETWORK ANALYSIS

  56. Danilo Giambra

    Hi, Martin.
    Thank you so much for this amazing research tool. I will definitely use it for my research. I’d like to ask you if there is anything similar that could retrieve for example Facebook public posts, as well. Also, advanced functions don’t seem to work with all searches and I don’t understand why. Although I can collect the tweets, when I run GetFriendFollowerRelationships command I get the following message:

    Oops
    The coordinates or dimensions of the range are invalid. (line 511).

    And then, of course, also the other functions won’t work either. What I don’t understand is why it works for some searches and not for others. Does it have anything to do with the fact I am searching for Japanese keywords?
    Thank you again for your work, that’s great!

    Reply
    1. Martin Hawksey

      Post author

      Currently Twitter only make the last 7 days of data available. If you know what you are looking for you can set up the script to continuously collect results. If you are looking for more/older data then check out datasift.com (they are one of Twitter ‘s official data resellers)
      Thanks
      Martin

      Reply
      1. Rajagopalan C

        Hi Martin,

        Awesome stuff you are doing. keep up the good work.

        Quick question on archiving Facebook Page posts + comments. Plus facebook public posts and comments. Is there any utility that can help us do this?

        Reply
  57. Pingback:

    Twitter in der Wissenschaft: Ein Leitfaden für Historiker/innen | Digital Humanities am DHIP

  58. Pingback:

    Sheila’s work blog » Analytics and #moocmooc

  59. Dani

    This is precisely what I was looking for! I have a pie-in-the-sky plan to feed the links from my tweets into the bit.ly API to automatically collect click data, and step one is getting a spreadsheet of our tweets.

    I am, however, getting an error in line 1085 when I try to run the authenticate function. I thought it might be the missing s in the http part of the link to the Twitter API, but that doesn’t seem to be it. (Looks like it has all of the necessary }s, )s, etc.) Any ideas?

    Thank you again for all of your hard work, and for making this available!

    Reply
      1. Dani

        Thanks for the speedy response!

        I tried that, and I’m not coming up with anything useful. I’m getting an error in v 4.0, as well, on line 376.

        My programming teacher told me I had code gremlins back in college, and I think she might have been on to something.

        UPDATE: It did, however, run perfectly when I removed the // in front of the commented-out code in line 105 (in the authenticate function). I can’t explain why, exactly, but I’ll take it!

        Reply
  60. Jakes

    Hi, quick question, I suddenly get these errors on all my TAGS spreadsheets every time the triggers run:

    collectTweets TypeError: Cannot call method “getSheetByName” of null.

    Any suggestions?

    Reply
      1. Stephen

        Having problem too, line 1012

        Start Function Error Message Trigger End
        8/20/12 8:54 PM collectTweets Data storage error (line 1012) time-based 8/20/12 8:56 PM
        8/20/12 9:00 PM collectTweets Data storage error (line 1012) time-based 8/20/12 9:02 PM

        Reply
        1. Martin Hawksey

          Post author

          Okay something has changed in apps script I’ve updated the template but to get you own copies of this working again open the spreadsheet, open Tools > Script editor and change lines 321-325 from

          if (typeof languageCode != "undefined") {
          if (languageCode.length > 0) {
          URL = URL + "&lang=" + languageCode;
          }
          }

          to

          /*
          if (typeof languageCode != "undefined") {
          if (languageCode.length > 0) {
          URL = URL + "&lang=" + languageCode;
          }
          }
          */

          Reply
  61. Jonas

    Hi Martin,

    I occasionally get this error from Google: “collectTweets Service timed out: Spreadsheets time-based”
    Usually there was nothing wrong with the archive itself, i.e. until recently where it at first seems to having stopped archiving but it rather seems that it adds blank rows in the archive. I found out about this yesterday and today I’ve got already over 1100 empty rows. But the tweets, it seems, were archived correctly. So my question is: Can I just delete those empty rows or do you have an idea why I get them in the first place?

    Once again: Great job, you’ve done there!

    Reply
      1. Jonas

        Thanks so much! I just did that. Weirdly enough, after I deleted the rows I found out that I had several duplicate tweets. Huh. That’s not really a problem though, just wanted to let you know :)

        Reply
      2. Jonas

        Hi Martin,
        just had a follow-up question: Is there a way to only archive tweets from a special region, seeing that (i.e. language) the archive automatically saves the languages?

        Reply
      3. Jonas

        Hi Martin,
        just had a follow-up question: Is there a way to only archive tweets from a special region, seeing that (i.e. language) the archive automatically saves the languages?
        (Sorry if this post is double: just got an error page when I tried to submit my comment)

        Reply
  62. Erwin

    Dear Martin,

    I’ve used your spreadsheet before with great success, and you make me look very smart for using it, all of which I owe you much gratitude for. Today, however, I don’t feel so smart, as I’m trying to set up the doc with a new search term and I get the following error:
    “Line 329 Address unavailable: http://search.twitter.com/search.json?q=pluk%20de%20nacht&rpp=100&page=1&result_type=recent&include_entities=true&with_twitter_user_id=trueException

    After authenticating the API it repeats the same error but in brief: “Line 329 Unexpected error: Exception”

    My gratitude expands infinitely for any hints or ideas how this could be solved.

    Also, very much wondering if there would be a possible expansion whereby images that are referred to in a tweet could be somehow stored with the original message.

    Kind regards,
    erwin

    Reply
  63. Pingback:

    First look at analysing threaded Twitter discussions from large archives using NodeXL #moocmooc JISC CETIS MASHe

  64. Dora

    Hi Martin,

    Thank you so much for this, I work for a broadcaster and it’s a great way for us to the catch the tweets (which we later use for promos/audience satisfaction).

    However there is one show that is extremely popular, both at its premiere and its late-night repeat. So in the morning I’ll only have the tweets from the late-night transmission.

    Is there a way to schedule when the script runs? Or to collect the tweets from 9pm-10pm?

    Thanks again!

    Reply
    1. Martin Hawksey

      Post author

      Hi Dora, I’ve struggled in the mast to collect popular TV tags like #bbcqt but it might be possible on less popular shows. Unfortunately there isn’t a straight forward way of doing this, but here is one solution:

      1. Open Tools > Script editor and insert the code below after line 21.
      2. Enter as many start and end dates by replicating the line schedule.push({start:"08/29/2012 20:00", end:"08/29/2012 22:00"}); using your own date/time. Note date/times match the timezone setting in the Script Editor settings (File > Project properties)
      3. In the script editor open Resources > Current script’s triggers… and remove any triggers for collectTweets and add scheduledCollection to run every 5 minutes.

      That should do it (if not let me know)

      Martin

      function scheduledCollection(){
      var schedule = [];
      // dates/times in mm/dd/yyyy hh:mm - timezone matches settings in File > Project properties
      schedule.push({start:"08/29/2012 15:00", end:"08/29/2012 16:00"});
      schedule.push({start:"08/29/2012 20:00", end:"08/29/2012 22:00"});
      checkCollect(schedule);
      }

      function checkCollect(schedule){
      var now = new Date();
      for (i in schedule){
      var start = new Date(schedule[i].start);
      var end = new Date(schedule[i].end);
      if (now > start && now < end){
      collectTweets();
      }
      }
      }

      Reply
  65. Pingback:

    Thing 4: Current awareness – Twitter, RSS and Storify | The Library of Babel

  66. Pingback:

    Google App Script: Scheduling timed triggers JISC CETIS MASHe

  67. Pingback:

    Can the Museum Collect Tweets? |

  68. Pingback:

    Domestic Violence Tweets, v2.0 | Domestic Violence and the Internet -

  69. Pingback:

    #libroelectrónico: Visualising an Archive of the eBook Symposium in Mexico « Ernesto Priego

  70. Pingback:

    #libroelectrónico: What Mexico’s eBook Symposium Backchannel Tells Us | 4Humanities

  71. Pingback:

    #libroelectrónico: Lo que nos dicen los tweets sobre el simposio | 4Humanities

  72. Pingback:

    Tentative steps in the Twitterverse | Every Day is a School Day

  73. Pingback:

    Joeyanne Libraryanne » Five free methods of archiving tweets from events – which are best?

  74. Casey

    Hi Martin,

    I am wondering what the historical reach of TAGS is. I am working on a dissertation on television finales, and I’d like to be able, for instance, to go back to the Lost finale and collect tweets from that night. Is this possible?

    Thanks for creating this excellent and user-friendly tool.

    Reply
    1. Martin Hawksey

      Post author

      Limitation imposed by Twitter API is last 7 days up to 1500 tweets. So if there was a lot of traffic on your search there’s no easy way to capture data. Configuring the spreadsheet to run during the broadcast means you would have been able to capture a lot more eg capturing up to the last 1500 tweets every 15 minutes

      Reply
  75. Pingback:

    ds106: Not Fractal nor MOOC, Just ds106 - CogDogBlog

  76. Pingback:

    Tweets as Texts | LLCU-606: Reading Digital Texts

  77. egilaslak

    Wow! This is fantastic! And even as a coding novice I was able to steer this juggernaut of code!

    One question:
    The hashtag I’m tracking was launched this friday, and I’ve been able to collect 1 500 tweets from both friday and saturday. But I believe there were more than 1 500 tweets that were tweeted that day. As Twitter has a cap of 1 500 for each search – is there any way to narrow the search down to say – every hour of the day for this friday and saturday?

    Reply
  78. Pingback:

    Any Questions? Filtering a Twitter hashtag community for questions and responses [situational awareness] #CFHE12 JISC CETIS MASHe

  79. Viola

    Hi Martin,
    thanks so much for this tool – I am using it for my linguistics Bachelor thesis.
    However, I am having difficulties with the API Authentication (and therefore the daily automatic collection) – it says “Unexpected error: (line 1108″ and there it says
    ;var result = UrlFetchApp.fetch(
    “http://api.twitter.com/1/account/verify_credentials.json”,
    requestData);

    do you have any tip for me? Thanks again!!

    Reply
    1. Viola

      Oh I am sorry it seems that I did use the wrong information for authentication (access token instead of consumer key)…

      Reply
  80. Viola

    Hi Martin,
    I’m sorry that I have to post again – may I know how reliable the language categorization is and what it is based on? I found some English tweets in the category “de”. Thanks!

    Reply
      1. Viola

        oh I see! that’s important to know, so I guess I can’t assume that everyone sticks to that. Thanks a lot! I’m doing a linguistic analysis (looking for appraisal in English tweets about China)

        Reply
  81. Pingback:

    ds106 is a Complex Universe Full of Stars - CogDogBlog

  82. Pingback:

    #UPRLKA: Complete tweet archive and related visualisation around Sri Lanka’s UPR review - Groundviews

  83. Pingback:

    #UPRLKA: ශ‍්‍රී ලංකාව පිළිබඳ UPR සමාලෝචනය ආශ‍්‍රිත පූර්ණ tweet ලේඛනාගාරය සහ අදාළ දෘශ්‍ය රූප : Vikalpa

  84. Pingback:

    The most comprehensive aggregation and visualisation of #jiscel12 tweets JISC CETIS MASHe

  85. John Mattos

    Hi Martin

    Thanks so much for this! Totally amazing! I have a couple of questions.

    1. if Im tracking a high volume live twitter event, how often can I safely set the script to run to ensure that I get all of the occurances of a given hashtag?

    2. My concern with (1) was that if I ran TOO often I might get duplicate results, for instance if someone’s tweet came back in two consecutive queries would they both be added to the spreadsheet?

    Thanks!

    Reply
    1. Martin Hawksey

      Post author

      The issue is usually how quickly the spreadsheet can write the data. Setting up to run every 10 min would be my recommendation. You’ll might also want to delete the summary and dashboard sheets (the formulas can take to long to calculate with a lot of data). Overall this solution isn’t ideal for high volume/velocity searches

      Reply
      1. John Mattos

        I’m not too worried about the time to calculate, honestly, I could stand to update it once every half hour. I was really wondering if its going to append the newest 1500 results with each new run, or replace the previous 1500 results.

        Seems like its going to tack them on, which is awesome, I’d definitely want that.

        would it enter duplicate tweets though?

        Thank you again, this thing is really amazing!

        Reply
        1. Martin Hawksey

          Post author

          Hi John – by default it’s designed to avoid duplicates, so if it’s set to get 1500 tweets and there are only 100 new ones since the last collection it’ll only insert the new tweets

          Reply
  86. toni

    Hi Martin,

    you’ve made a great job!
    I’m trying to get the coordinates from the tweets that include some political slogan as a hashtag in order to compare the use of twitter to the real results in an election.

    I added the line URL = URL + “&geocode=41,2,500km”; to define the geographical area but the main problem is that most of the tweets aren’t geocoded, so I’d like to get the location as well.

    How can I do that?

    thank you.

    Reply
    1. Martin Hawksey

      Post author

      Hi Toni, I haven’t done much work with geocoding, but know the proportion of geo tweets is very low. One way I do it is post process the archive extracting all the profiles of users tweeting the search then get coordinates for where they say they are located and then ads this data to the archive

      Reply
  87. Nika

    Hello!

    In the “Vertices” sheet I noticed some columns that remain empty (i.e. betweeness, size etc). Are they placeholders that you intend on filling up in a future version of the tool or is there a way to fill them up currently?

    Thanks again for this great tool!

    Reply
  88. Pingback:

    Archive and visualisation of tweets from #NHSAlliance2012 | Knowledge 4 Commissioning

  89. Greg McVerry

    Martin,

    I am showing your tool to literacy researchers at a national conference. I am having trouble with the SNA scripts. I am getting an error message each time they load. Should I delete and restart it. I did delete some unecessary rows in the archive that were from another conference using the same hashtag. Could this have messed me up?

    Reply
  90. Pingback:

    Twitter provides proof of real-time engagement with the public: How to record it JISC CETIS MASHe

  91. Danielle

    “The coordinates or dimensions of the range are invalid” I keep getting this response when I try to run the advanced version step #1

    Reply
    1. Danielle

      Also, thank you in advance for you help :) Is there anyway that I can pick out a piece of time to analyze. For instance, I want to analyze an hour long twitter chat on 12/12/12 – is there a way to do that? thanks again!

      Reply
  92. Pingback:

    ¡Hemos perdido pero nos hemos divertido! | Territorio creativo

  93. Pingback:

    1 of 52 Google: Google Script - Interlocked Pieces

  94. Pingback:

    Building ETMOOC Twitter Syndication/Archive - CogDogBlog

  95. Pingback:

    Analysing Wordpress post velocity and momentum stats with Google Sheets (Spreadsheet) JISC CETIS MASHe

  96. Pingback:

    Play is Not The Opposite of Work: Stanford mediaX 2013 Conference | Tim McCormick

  97. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide | Blogging Enterprises

  98. Jessica

    Hi Martin,

    Awesome tool! I have a quick question: I’m using this TAGS to archive tweets related to a recent book release, so I’m grabbing tweets with the most relevant hashtag and tweets at the author, but I also want to grab tweets BY the author and I haven’t been able to figure out what language to use to do that in the search term field. Can you help?

    Reply
  99. Clare Mackay

    Hi Martin, I’m getting an error message ‘script invoked too many times for one day’ when trying to set up a new archive. Why is this/what can I do to overcome this?

    Reply
      1. Clare Mackay

        Hi, thanks for getting back to me so quickly. I’m archiving tweets pertaining to the hashtag Glasgow2014 for my PhD research so I’ve been setting up a new spreadsheet whenever one reaches its cell limit and was therefore trying to set up a new one. I had also started archives relating to #CWG, Commonwealth Games, Gc2018, and Gold Coast 2018 for future research projects. If I delete some of the latter, should I then be able to set up the new archive for my current research? Or what else can I do to overcome this? I’m not very technologically minded! :-)

        Reply
  100. Pingback:

    TWITTER RESOURCES | Four Go Further

  101. Pingback:

    #elysée50 – Twitter als Erinnerungsraum | Digital Humanities am DHIP

  102. Kristof

    Hi Martin,

    thanks! A great tool!
    Got it set up good for continuous tracking.
    I am tracking a hashtag which is supposed to have generated 25,000 or so tweets within the last 3 days. I guess I am a bit late starting the tracking.
    I am having trouble getting a hold of the earlier ones though. I always retrieve the same tweets. Is there a possibility to adjust the time frame in a more granular way?

    All the best!

    Reply
  103. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide « Blogging Enterprise

  104. Pingback:

    Keep Your Official Twitter Archive Fresh - ProfHacker - The Chronicle of Higher Education

  105. Pingback:

    Herramientas para la elaboración de la tesis « Alternando TIC, educación y software libre

  106. Erica

    Hey Martin!

    Thanks for this application, it is a life saver. I do have one question–not sure if you answered it already (I didn’t see it while I was skimming)–for “search term” I had 1 hashtag, changed it to 2 hashtags and then back to the original 1 hashtag. But the results of the 2nd hashtag still appear in the archive–is there a way that I can reset it back to the original search term to show the just results of the 1 hashtag or somehow delete the tweets pulled for the 2nd search term?

    Thanks!

    Reply
    1. Martin Hawksey

      Post author

      Hi Erica – I had a similar problem recently. What I did was scroll through the archive sheet looking for first occurrence of hashtag 2 and then selected all the rows until it went back to hashtag 1 and deleted them

      Reply
  107. Pingback:

    Twitter Archiving Google Spreadsheet TAGS v3 Jisc CETIS MASHe | Writing New Media

  108. Pingback:

    5 pasos para desarrollar con éxito un Twitter Chat: La guía definitiva | SocialMediando

  109. Pingback:

    Twitter and micro-blogging notes on day 2 » PJ Evans

  110. Pingback:

    Prize to create and expand Brighton & Hove Twitty Council idea | BlueLightCamp

  111. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide | Empower Network + BIG IDEA MASTERMIND Team

  112. Pingback:

    Twitterarchiv und Reaktionen auf #dhiha5 | Digital Humanities am DHIP

  113. FredericGSole

    Is there any way to insert usernames in a blank sheet and then get all the information about these users? Name, description, location, and so on?

    Many thanks,
    Frederic

    Reply
    1. Martin Hawksey

      Post author

      If you fill the column that usually has usernames in the Summary sheet and run the TAGS Advanced menu options you should get the info (you’ll need to unhide the vertices sheet

      Reply
  114. FredericGSole

    Hi, Martin,
    Many thanks again for your TAGS! I can’t live without it!
    A question: how can I add the name of the user in the sheet? And the location?

    Many thanks,
    Frederic

    Reply
  115. Jeffery Bonner

    Martin,

    I am interested in returning the name of a user (note the username) from a twitter archive search. For example, I make a twitter post and hashtag BIOL3210 then using the twitter archiving tool I do a search for keyword #biol3210. I want my name “Jeffery Bonner” (associated with my twitter account) to display in the spreadsheet rather than my twitter username. The purpose is to properly identify students in my class when they make posts related to class. \

    Thanks!

    Reply
  116. Pingback:

    Using Viewshare to Visualize Conference Tweets | The Signal: Digital Preservation

  117. Pingback:

    Google Analytics API made easy - Google docs Magic Tutorial | distilled

  118. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide | Social Media Examiner

  119. Deepak Raikwar

    Hi Martin,

    Thanks for this tutorial. Can you please do me a small favor, i am getting error when i try to Run Nwo() function, my error is:

    TypeError: Cannot read property “length” from undefined.

    What i have to do?

    Regards,
    Deepak

    Reply
  120. Mia Constantine

    Hi Martin – Thank you so much for this tool. I was very humbled to find it. I know the max/day is 1500 but how can I download OLD tweets more than 1500 tweets ago?

    Reply
  121. Pingback:

    Archive and visualisation of tweets from #NHSAlliance2012 | Knowledge 4 Commissioning

  122. Pingback:

    5 Steps to Hosting Successful Twitter Chats: Your Ultimate Guide « MindCorp | Newsfeed

  123. Pingback:

    Twitter Archiving Google Spreadsheet TAGS v3 | MASHe | settlement

  124. Pingback:

    Twitter Archiving Google Spreadsheet TAGS v3 - ...

  125. carla2198

    Silly question – but will the archive continue to add to it, or do I need to export the data and continue to save them every few days? I’m SUPER new to all this and just learning how to use TAGS and understand how it works. Thanks for your help and for creating this!

    Reply

Leave a Reply