This one Data Scraping Wikipedia with Google Spreadsheets

…

If you want to know how I arrived at this conclusion it’s mostly the same process as And the most engaging JISC Project is…, but as well as getting social bookmark/share counts I’m also counting the number of post comments (an important part of engagement).

Here’s a copy of the full spreadsheet with 771 analysed links (if you make a copy of this you can see the code in Tools > Script editor) and a summary below:

Counting comments

It actually took two methods. Knowing that WordPress blogs have a comment rss feed for each post I initially grabbed this and snaffled the number of comment items (at the same time I grabbed the post title, as the sitemap.xml doesn’t have this info).  Here’s the code for getting comment counts from comment rss feeds.

What I forgot was that the comment feed is limited to the number of rss items set in the blog administration, usually 10, so if a post has more these are missed. To solve this I wrote this additional code which goes through each post with 10 comments and confirms this count by screen scraping the number from the original HTML page (its ironic that Tony’s top post is also screen scraping using Google Spreadsheets using an alternative method).

Bonus – Topsy tweet data

In my previous post I mentioned how I used the Topsy service to get a secondary tweet count. As well as the link share count Topsy also keep a record of the tweet which made. This data is also accessible via their Topsy (Otter) API as part of trackbacks. As Tony has done a lot of work with community visualisation (some posts here) I thought that as I was querying Topsy I should grab the tweets and dump them in a separate sheet in their JSON format (I should say that unless you start paging the results the maximum you’ll get is 50 tweets). 

Engagement £value£

Something else I mentioned in my my previous post was can a monetary value be attached to a Tweet or a Like. In Sheila MacNeill’s related post I commented that services like PostRank put a suggested value on these forms of engagement. I’m not convinced that a comment is worth $2.50, there are too many factors like who made the comment, who sees the comment and many others to put a flat rate on it.

For example, using the PostRank formula I calculated that OUseful.info has generated $20,393.25 in ‘engagement’ which I think is a gross underestimate. Considering the number of things I’ve learned and followed up as a direct result of Tony’s work he’s due a sizeable cut from any ‘engagement value’ generated from this site.

Last updated by Martin Hawksey at July 12, 2012.