Like a growing number of other people I’ve requested and got a complete archive of my tweets from Twitter … well almost complete. The issue is that while Twitter have done a great job of packaging the archives even going as far as creating a search interface powered by HTML and JavaScript as soon as you’ve requested the data it is stale. The other issue is unless you have some webhosting where can you share your archive to give other people access.
Fortunately as Google recently announced site publishing on Google Drive by uploading your Twitter archive to a folder and then sharing the folder so that it’s ‘Public on the web’ you can let other people explore your archive (here’s mine). Note: Mark Sample (@samplereality) has discovered that if you have file conversion on during upload this will break your archive. [You can also use the Public folder in Dropbox if you don’t want to use a Google account]
The documentation wasn’t entirely clear on how to do this. Basically it seems that as long as there’s a index.html file in the folder root and links to subdirectories are relative all you need to do is open the folder in Google Drive and swap the first part of the url with https://googledrive.com/host/ e.g. https://drive.google.com/#folders/0B6GkLMU9sHmLRFk3VGh5Tjc5RzQ becomes https://googledrive.com/host/0B6GkLMU9sHmLRFk3VGh5Tjc5RzQ/
So next we need to keep the data fresh. Looking at how Twitter have put the archive together we can see tweets are stored in /data/js/tweets/ with a file for each months tweets and some metadata about the archive in /data/js/, the most important being tweet_index.js.
Fortunately not only does Google Apps Script provides an easy way to interface Drive and other Google Apps/3rd party services but the syntax is based on JavaScript making it easy to handle the existing data files. Given all of this it’s possible to read the existing data, fetch new status updates and write new data files keeping the archive fresh.
To do all of this I’ve come up with this Google Spreadsheet template:
*** Update Twitter Archive with Google Drive ***
[Once open File > Make a copy for your own copy]
The video below hopefully explains how to setup and use:
A nice feature of this solution is that even if you don’t publically share your archive, if you are using the Google Drive app to syncs files with your computer the archive stays fresh on your local machine.
The model this solution uses is also quite interesting. There are a number of ways to create interfaces and apps using Google Apps Script. Writing data files to Google Drive and having a static html coded based interface is ideal for scenarios like this one where you don’t rely on heavy write processes or dynamic content (aware of course that there will be some sanitisation of code).
It would be easy to hook some extra code to push the refreshed files to another webserver or sync my local Google Drive with my webhost but for now I’m happy for Google to host my data ;s
Last updated by at .
Loading...
I tried making a copy of the spreadsheet, but that option was not available to me.
Hi Jedidiah, first thing to check is you are logged in to your google account when you open the spreadsheet
Hi Martin – I was logged into my google account. I have tried it since and been able to make a copy. Thanks.
I’m not getting any action, have triple checked settings. U’ve entered my folder ID twice but the status says “Folder ID not entered yet” When I runt he script, it just stays at Running script updateArchive – it looks like its not getting my folder ID?
Think it was my dodgy coding not updating the location on the readme. If you open the Log sheet has it recorded any updates?
Nope, updates are blank. Should I share the doc with you?
[closed - for some reason authentication with Twitter didn't work 1st time. Added some extra UI to help]
Looking for a link to archiving twitter on a website (other than on Google Drive)…
Hi,Martin it is great what you’ve done. thanks for this beautiful script.
I tried to follow your instruction. Everything loooks good till at last what appears is yours, not mine. Check this. https://googledrive.com/host/0B1yKt8K-OtLOUDFvY1laMjNMS1E/
So, how to change the name account?
thanks
Hi Lilla – did you start with an upload of your own archive from Twitter?
hi
am getting Unexpected error: (line 35) when i try to authorize function?
var result = UrlFetchApp.fetch(“http://api.twitter.com/1.1/account/verify_credentials.json”, requestData);
thanks in advance
Sounds like you API key/secret are not set properly. Make sure there’s so spaces before/after them when you enter them into the configuration dialog (I’ve added a trim to prevent this in the future)
hi again
used your new spreadsheet template and now getting a different error message says:
—
Apps Script
Unexpected error:
—
have tripled checked the api key/secret.
and ideas?
thanks
ak ignore last message working now, i forgot to add the callback url, doh!
thanks again great script!
Holy smokes, this is great! I had trouble getting it to work at first, and I wanted to share the solution, in case your readers encounter the same problem. Depending upon one’s Google Drive settings, your Twitter archive folder might be converted to “Google Docs” format during the uploading process. This breaks your files and makes them unreadable (as I found out!). So make sure that in your Google Drive upload settings, you have all the file conversion rules turned off.
Here’s my own now-working archive: https://googledrive.com/host/0By7OircJ9labZktFR0xkT1ExUXc/
Thanks Mark! I’ve added a couple of notes in the template/post to highlight this.
That good to knoe Mark- I used the Google Drive desktop app which takes care of that. I know point my subdomain to GDrive http://tweets.cogdogblog.com/ The storage and capability here has me reeling with new ideas.
Wow, great. That even worked for a coding/web-admin dummy like me. Thank you so much. Because I was too stupid to set up Tweetnest.
Great sheet Martin,
I have a problem though when I want to start the sync, it keeps telling me “application has not been authorised” or something like that…I repeated the process again and again and no way to have it work.
For the moment I just managed to get my tweets online.
https://googledrive.com/host/0B_RajZialboVTFNhYi1HcmpvVkE/
if you have any hint…
In Tools > Script Editor when you then clicked Run > authorize did any error msgs come up?
Hi Francisco George,
I had the same problem – if you do as Martin suggested and Run > authorize part of the script, you’ll get a dialog box (can’t remember what it said now) with a button. If you click the button, it takes you off to Twitter, and you’ll be able to authorize the script there.
It fixed the problem for me at least – hope it works for you too.
Nice work Martin!
Hi Martin,
Thanks a lot, it worked!
I somehow din’t quite understood this step…and now with your hint everything worked fine.
You should maybe submit your solution to Twitter, it would ease their load if by having this solution implemented as they would not have every one downloading their archive too frequently.
Very nice work!
Francisco
PS: is there anyway to change the frequency of the automatic update and set it for a for a every 8 hours update for example?
UX isn’t my strong point. I’m not sure what Twitter would make of this ;s
[You can update frequency by opening Tools > Script Editor and then Resources > Current script triggers. In the dialog box you should see a function set to fun daily. Just adjust the frequency from one of the drop downs and save. I wouldn't recommend going any lower than every 15 min to avoid api/usuage limits]
Hi Martin, sorry if I’m being dense but what is the current update frequency set at – daily? I tried the route you mention above Tools > Script Editor > Resources – I don’t have ‘current script triggers’, instead I get the option ‘current project triggers’. The dialog box this brings up shows one trigger but it isn’t editable at all and I can’t see anywhere that mentions update frequency
It’s not the end of the world – I would be happy with once a day if this is what it is
Btw, love it. Thanks very much :)
Hi Patrick – in the spreadsheet there should be a Sync Twitter Setup menu where you can toggle the auto-refresh
Hi Martin, thanks I found that in the end but thanks for getting back to me. Very chuffed with this – great bit of work, well done :)
OMG, this is really interesting.
Is the formatting of the added (updated) tweets exactly the same as the original Twitter Archive? I had a script written to parse the original archive JSON by stripping the first line (per the README in the archive) but that seems to fail with the updated archive files. It seems that in the original archives, the first line reads as “Grailbird.data.tweets_2013_01 = ” whereas yours outputs “Grailbird.data.tweets_2013_01 = [” with that opening bracket appearing on the first line rather than the second. It’s probably a simple fix to your script but it does break JSON parsing if the first line is stripped.
ah hadn’t spotted that. I’ve updated the template. The changes are line 109 which is now
tweet_file.replace(“Grailbird.data.”+var_name+” = \n”+ JSON.stringify(Grailbird.data[var_name], null, ‘\t’)); // replace old file
and line 116 which is now
new_tweet_file.replace(“Grailbird.data.”+i+” = \n”+ JSON.stringify(newData[i], null, ‘\t’)); // replace content with new data
Does this update is meant for every one of us? Or is it only relevant for the script Shiruken is using?
Yes only relevant if you are doing your own post processing
So I think something may be broken now that a new month has rolled around. The script generated a “2013_02.js” file in the root of my Google Drive rather in the appropriate directory which resulted in future updates failing.
Well for the moment I can’t really test it, DRIVE and Youtube are not responding.
Ok…My “2013_02.js” has been created in the correct directory. under data>js>tweets
Very weird. I moved the file back into the appropriate location and everything seems to be working fine. Guess I’ll find out next month if it’s actually a problem.
Maybe it is due to the short downtime of drive.google.com and youtube.com that prevented a correct sync. My guess
but you are right too, the file “2013_02.js” appears in my root too but along it, it says it is located in the “Tweets” folder(the public folder I use)
Appeared to work okay for me. Like @paco229 the new file appeared in the root directory and data>js>tweets (which is surprising as I thought Drive had done away with files having multiple labels/locations). I’ve added an extra line in the template to prevent this. For existing copies of the script insert the following line after line 115:
new_tweet_file.removeFromFolder(DocsList.getRootFolder());
so it should look like:
new_tweet_file.addToFolder(tweetsFolder); // move new file to the data/js/tweets/ folder
new_tweet_file.removeFromFolder(DocsList.getRootFolder());
It is back now
hi
my archive is giving me a 404 not found error.
when i try to manually update archive using your script i get this error message:
TypeError: Cannot call method “getContentAsString” of undefined
thanks in advance!
Sounds like your folder is not shared/missing?
hi
had to eventually upload new Twitter folder, and set share permissions to “Public on the web – Anyone on the Internet can find and view”
thanks
Thanks for sharing this routine!
Defenitely going to try this out.
Nice to see our conversation back in your archive.
I’ve changed some backgrounds in mine:
http://www.karssenberg.nl/twitter/
Hi Willem, how did you manage to change your background?
Thank you so much Martin! I’ve got it all up and running, the way you described.
Only the search function does not work yet. May be it needs some time indexing?
You can view my archive at: https://googledrive.com/host/0By2d5B0yP7tLbmdsVkRyS2pQQk0/
This is so nice!
Hi Willem, search is done via the browser (no clever indexing). When I tried doing a search in your archive the console said the file data/js/tweets/2008_05.js was missing
Thanks,
Martin
Thanks so much, you found the bug! Actually, the file was not missing, but existed two times! Ni idea how that happened. May be a hickup whith ftp.
Anyways, search works now thanks to your attention.
Lots of people here in the Netherlands tried your script already. It’s been retweeted quite a lot!
Hi Martin
I’m getting this message from apps-scripts-notifications@google.com on intervals of one hour(my refresh rate)
26/02/13 18:51 updateArchive We’re sorry, a server error occurred. Please wait a bit and try again. (línea 72, archivo “Code”) time-based 26/02/13 18:51
Any idea of the problem?
Thanks
Hi – There is an open issue related to this. I hoping Google will have it resolved soon!
Martin
When starting “update Archive Now” I’m getting to see (in Dutch) an error message telling me that function getFolders cannot be found in object false. There the script stops. What’s going wrong ?
How to fix this Twitter Api not configured???
did you get a twitter api key and secret? If you did, did you add them via the Twitter sync menu? If you did, did you open Tools > script editor and run authenticate?
Yes, I took all these steps without getting any error message, everything went smooth as far as these steps are concerned.
Sorry, reaction on the wrong thread …
what happens if you run the authenticate function again? (In particular does a dialog box appear taking you to Twitter to authorise the connection or nothing)
Harmen – I’ve confused myself my responses were supposed to be for Mike. Your problem sounds like all the folders/files aren’t uploaded to Drive properly
I’ll try to fix that Mike, thanks.
Sorry, I don’t know how to get the key and secret, I stuck there. I followed the instructions on video, by clicking authorize and run, but it appears a box – Twitter Api not configured.
In the spreadsheet menu bar there should be a Sync Twitter Setup menu option. From it click API Authentication for instructions on setting up the API connection
Thanks a lot! I just realized I haven’t create that things, that’s the reason I can’t authorize the app. Everything is fine and my twitter archive is able to be updated now. Thank you again. =)
Between, it only updated the latest tweets, how about some old tweets that I have deleted, but it still appear on my archive?
Hi, Martin. Great script and easy to set up. Thank you for sharing it. I credit you on my website.
Is it possible to edit the title tags within the index.html & how would I go about using a bespoke address instead of the long googledrive address.
Great script and so easy to set up.
Thanks
Adam
Hi Adam, I’ve done some minor editing of the index.html for my own archive but haven’t dug that deep. I also use a redirect on my webserver http://tweets.hawksey.info (doesn’t cloak the address, I’m sure there are ways of doing it)
Hi Martin,
I’ve tried this a few times, followed step by step and to a T, but every time I get an “oAuth Error” after running the “authorize” script. Any ideas?
Running this since weeks and I have to say I’m very satisfied with it. The search is great and really works. But there’s one question I worry about: Will this still works, when Twitter will change to its 1.1 API completely?
Fear not – when I wrote the script I was aware of the impending changes so it entirely uses version 1.1 of the API
HurraY! And many thanks.
Double hurray!
I hope this isn’t a cheeky question, but does anyone know how I would go about putting my own Twitter background on this for when people are searching tweets? At the moment it’s pretty uninspiring and blank. Thanks
Easiest way is to change the img/bg.png image. For more tweaking most of the styling is done in the css/application.min.css file. It’s minified so not easy to navigate (sites like http://procssor.com/process can decompress it if you are doing lots of edits)
Phew, you were right about it being minified! Thanks Martin I’ll have a bash and thanks too for keeping up with everyone’s comments, it’s great style. Much appreciated
Hi, it’s me again! I would like to ask you whether that’s a way to remove DELETED tweets on twitter archive from google drive?
Easiest way is to re-request archive from Twitter and replace the files. The script will just continue updating the data files it has available
Hi Martin,
Since this morning I’m getting this error report. Any advice?
Details:
Inicio Función Mensaje de error Activación Fin
16/03/13 19:51 updateArchive TypeError: No se puede leer la propiedad “id_str” de undefined. (línea 95, archivo “Code”) time-based 16/03/13 19:51
The error code from Appscript has changed 3 hours ago now it states
17/03/13 6:51 updateArchive ReferenceError: “tweet_index” no está definido. (línea 80, archivo “Code”) time-based 17/03/13 6:51
Hmm looks like your archive files have become corrupted (I’m guessing a write process wasn’t completed properly). Easiest solution is to request your archive from Twitter again and replace the files on drive. If you use the same folders setting up the script again is not required
Thanks! This works like a charm. Here’s an archive of the Texas Tribune, where I work: https://googledrive.com/host/0B0ATO9kgH3hJdjVTWnpER1QzYlE/
Hi Rodney – I know a number of graduate researchers swing by this blog and I’m sure they find it a valuable resource :)
Martin
Hi, Thanks for this wonderful guide. I have some issues though. When I get to the API Configuration step, I am not sure what to put in for the Application Details Page. I see: Name, Description, Website, and Callback URL. What is the website we should set to?
Also will this work with a private twitter account? I don’t want to share my tweets and want my own personal archive.
Thank you!
Hey, I have downloaded twitter archive several times, it was complete. However, my recent download was incomplete, it started at my latest 3000 tweets. Why it happened to me?
Is this your download from Twitter?
Yes, it is.
This sounds like a question for Twitter. I’ve no control over what is in your sorehead from them
I’ve just figured out how to create a twitter archive page hosted by Google Drive thanks to your great article and video. Just a quick question- is it possible to insert/include the actual web page- the one that the twitter archive is hosted on- in a new Google Document or Spreadsheet on Drive? Thanks for any advice!
When starting “update Archive Now” I’m getting to see (in Dutch) an error message telling me that function getFolders cannot be found in object false. There the script stops. What’s going wrong?