Posterous Blogger Sidebar Widget Thumbnail Feed Script

It's been a while since this blog actually lived up to its name and I posted something to do with actual hacking on my Linux box.

You may recall a post a while back where I used the 'sed' command to create a modified copy of my TwitPic feed so that a thumbnail would show up when I imported the feed into a Blog List gadget in Blogger.

Well, I recently switched from using TwitPic for uploading pictures from my phone to using Posterous for uploading pictures and video from my phone.  There were many reasons in the "pros" column, but in the "cons" was the fact that, when I imported my feed into that same Blogger widget, no thumbnail appeared.

So, just like with the TwitPic feed, I set out to modify my Posterous feed in order to get the thumbnail to appear.  One problem I encountered is that the feeds were totally different formats.  I based my TwitPic feed modification on a feed I knew to be working (from Digg), but performing that same transformation on the Posterous feed proved to be problematic.

What I ended up doing was simply extracting the information I needed from the Posterous feed, and then creating a one-item feed in the known-good format.  The feed looks nothing like the original Posterous feed, but that's just fine, since all it will be used for is pulling the latest post into my blog sidebar.

One improvement I'm considering working on is providing a useful thumbnail when I upload a video.  Currently (at least with the 3gp format), the Posterous feed just sticks a generic blank file icon in the thumbnail field.  What I would like is a still frame from the movie.  In order to do this myself, I would need to download the enclosure link, process the video into a still image, post the image on the web, and then put the image URL into the feed.  All very doable given the right tools.

I'll have to test out what happens when I use the MP4 format for video, which my phone is also capable of creating.

Here's my script (so far).  Feel free to use it under the terms of the license listed below.  If you have any questions or suggestions, please feel free to leave a comment.

posterous.sh (run as an hourly cron job):

#!/bin/sh

# Copyright 2010 Tim "burndive" of http://burndive.blogspot.com/
# This software is licensed under the Creative Commons GNU GPL version 2.0 or later.
# License informattion: http://creativecommons.org/licenses/GPL/2.0/

# This script was obtained from here:
# http://tuxbox.blogspot.com/2010/03/posterous-blogger-sidebar-widget.html

DOMAIN=$1
FEED_DIR=$2
FEED_TITLE=Posterous
FEED_DESC="The purpose of this feed is to provide a thumbnail of the latest item in a Blogger sidebar widget."


if [ -z $DOMAIN ]; then
  echo "You must enter a Posterous DOMAIN."
  exit
fi

if [ -z $FEED_DIR ]; then
  echo "You must supply a directory."
  exit
fi

if [ ! -d $FEED_DIR ]; then
  echo "You must supply a valid directory."
  exit
fi

FEED_URL="http://$DOMAIN/rss.xml"
TMP_FILE="/tmp/posterous-$DOMAIN.xml"
FEED_FILE="$FEED_DIR/posterous-$DOMAIN.xml"

# Fetch the RSS feed
wget -q $FEED_URL -O $TMP_FILE

if [ ! -f $TMP_FILE ]; then
  echo "Failed to download $FEED_URL to $TMP_FILE"
  exit
fi

NEW_LATEST=`grep guid $TMP_FILE | head -n1`

if [ ! -f $FEED_FILE ]; then
  FEED_LATEST="" 
else 
  FEED_LATEST=`grep guid $FEED_FILE | head -n1`
fi

# Comment these out
#echo "FEED_LATEST: $FEED_LATEST"
#echo "NEW_LATEST : $NEW_LATEST"

if [ "$FEED_LATEST" = "$NEW_LATEST" ]; then
#  echo "There is no change in the feed."
#  echo "FEED_LATEST: $FEED_LATEST"
  exit
fi

IMG_HTML=`grep -i "img src" $TMP_FILE | head -n1 | grep -Eo "<img src='[^']*'[^>]*>" | sed -e 's/\"/\&quot;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g'`
#echo "IMG_HTML: $IMG_HTML"

IMG_URL=`grep -i "img src" $TMP_FILE | head -n1 | grep -Eo "http:[^']*" | tail -n1`
#echo "IMG_URL: $IMG_URL"

# Create a minimalist RSS feed
echo "<?xml version='1.0'?> " > $FEED_FILE
echo "<rss version='2.0' xmlns:media='http://search.yahoo.com/mrss/'>" >> $FEED_FILE
echo "<channel>" >> $FEED_FILE
echo "<title>$FEED_TITLE</title>" >> $FEED_FILE
echo "<description>$FEED_DESC</description>" >> $FEED_FILE
echo "<link>http://$DOMAIN/</link>" >> $FEED_FILE

echo "<item>" >> $FEED_FILE
grep "<title>" $TMP_FILE | head -n2 | tail -n1 >> $FEED_FILE
grep "<pubDate>" $TMP_FILE | head -n1 >> $FEED_FILE
echo "<description>$IMG_HTML</description>" >> $FEED_FILE
grep "<link" $TMP_FILE | head -n3 | tail -n1 >> $FEED_FILE
echo "$NEW_LATEST" >> $FEED_FILE
echo "<media:thumbnail url=\"$IMG_URL\" height=\"56\" width=\"75\" />" >> $FEED_FILE
echo "</item>" >> $FEED_FILE

echo "</channel>" >> $FEED_FILE
echo "</rss>" >> $FEED_FILE

# Cean up
rm $TMP_FILE

This software is licensed under the CC-GNU GPL version 2.0 or later.

Google Buzz Kill

For the past few days the Internet has been all aflutter about Google Buzz, some saying it's a Twitter killer.  Google Buzz is not like Twitter.  Rather, it is like FriendFeed.  I have used FriendFeed for quite some time to aggregate all of my online content into a single stream, and Google Buzz is designed to do exactly the same thing.

Like FriendFeed, Google Buzz consumes Twitter and other content-generators, that is, you can have your Twitter posts show up on either service, as well as you blog posts, your online photos, forum comments, and so forth. Content originates in multiple places, but these services enable it to all come together in one place specific to the person who created it.

Now, I have no problem with Google creating their own FriendFeed and then foisting it on all Gmail users.  I think it's a great idea.  My mother, for example, will never set up an account with FriendFeed, and she hasn't quite figured out Google Reader, but she might just try out this Buzz thing in her Gmail inbox, there she might see my latest tweet, or blog post, or photo album, or shared article. 

Now, some of this stuff she already sees, since I import my Twitter updates and blog posts into Facebook, and so in a way, Google Buzz is competing with Facebook.

Online Identity

When you activate Buzz in your Google account, they let you know that they are making your Google Profile public, and that that includes your first and last name. 

As it happens, I have gone through the trouble of NOT directly associating any of my public online content with my real name.  That way if you "Google" my name, you don't find all of my content (insert horror story here about prospective employers finding something they don't like or disagree with). 

My real name is of course associated with my content inside of Facebook, but that content-name link is only available to my friends.  Most of the same content is available outside of Facebook, but it is tied to "burndive", not my name.

Google's corporate mission is "to organize the world's information and make it universally accessible and useful".  This is no doubt why they are pushing for people to publicly link their full real name with all of their online content.  

Well, I'm not going to be pushed.

For many people who do not maintain a barrier online between their friends and the public, this will not be an issue, but for me it is.

My Google account is used as my primary e-mail address.  I want my name to be associated with my e-mail address to my contacts, so I can't simply change my name to a pseudonym on my Google Profile, and it would be extremely disruptive for me to switch to another Google account. 

Google is obviously aware of people in my situation, because they already have a feature in the Google Profile called a "nickname".  Anyone on my contacts list will see my real name on my profile, everyone else will see my nickname.  This is how it works with Google Reader shared items, and it's a very good system.  They just don't want to use it with Google Buzz.

The Problem

I went ahead and added my blogs, Google Reader, my Twitter account, and my Picasa Web account to Google Buzz, but nothing was being imported except Google Reader. 

I looked further into the matter, and it turns out that it wasn't importing my content because after signing up for Buzz, I realized what had happened, and had restored my profile privacy settings.  "That's logical", I thought, "They won't let me post publicly because my name isn't public.  I'll just change the import settings so only my friends and family see the posts.  They can already see my name."  No dice.

So what was up with Google Reader?  Google Reader has separate privacy settings, as it turns out, but, as I discovered, it will STILL share your full name on the posts and make it visible to the world.

So, after a brief stint, I have turned off Google Buzz.  I never really intended to consume content there, but I had hoped it would be a venue for others to consume my content who would otherwise not occasion to see it, and a user-friendly comment forum.

Software To Install On A Windows Box

With the Windows 7 release candidate out, I recently wiped my laptop's hard drive clean and installed it (I was previously running beta 2). Whenever I flatten a computer and re-install everything fresh, I like to have a list of things to install that covers the bases.

I put together a bookmarks folder of software to install on a new Windows box, and I decided to share it with the world, so here it is:

• Software to install on a new Windows box

The nice thing about sharing it in this way is that I can keep it up-to-date in my browser if there is anything to add or remove, or if any of the links change.

Enjoy.

That's What She sed

Lately, I've been uploading pictures to Twitter from my phone using TwitPic. Basically, you send them to a TwitPic e-mail address via multimedia messaging, and they are automatically posted to your Twitter account, along with the text from the subject. This all works quite well, and they even supply an RSS feed of your pictures, which you can take and (among other thing) put on your blog's sidebar. The problem was, when I put it in my blog sidebar, there was no thumbnail image. Other feeds that had images in them would have thumbnails, but not this one. This one just had a text link to the picture page. I found that disappointing. So I examined feeds that showed thumbnails and the TwitPic feed to see what the difference was. Feeds that contained images within the feed content showed up in the Blogger widget with a thumbnail. But the TwitPic feed showed images. What was the difference? The difference turned out to be CDATA. CDATA is a way to tell a feed reader, "Don't try to decipher my contents, just pass them along and leave the rendering to the end user application." It so happens that TwitPic's thumbnail images are within a CDATA block, and Blogger obediently ignores the CDATA contents when looking for images to display as a thumbnail. So, how do I fix that? I need to read the feed, and for each item, locate the line that contains the thumbnail URL, and create a new attribute containing the thumbnail in a format that is decipherable to Blogger's widget. Using my digg feed as a model, I figured out what the end result should look like, but how to achieve it? First, I tried Yahoo Pipes. Yahoo has a tool for processing feeds with a number of tools, controlled by a graphical pipe-looking interface. The problem is, none of the tools that I could find would add an attribute based on the transformed contents of another attribute. There were widgets that came close, but I couldn't get it to work, so I decided to host the feed myself and modify it using sed. I had never used sed before, except when the exact command was given, so I didn't know how to use it, but I knew that it was a powerful enough tool to get the job done. So I created a shell script on my Linux box, and a cron job to run it. The script basically downloaded the RSS feed from TwitPic to a local file, and then called sed on it with a particular set of parameters designed to extract the necessary information, and add the appropriate information in a format that is decipherable to Blogger. In order to understand sed, I searched the Internet for a tutorial, and found this page from the Gentoo Linux Documentation to be the most helpful. My sed command does two things, which are piped together:

1. It adds an xmlns:media declaration, which allows me to use the media tag later on.
2. It examines each CDATA line with the thumbnail URL, and below it, it adds a line with the media:thumbnail tag and the URL extracted from above.

sed -e 's/<rss version="[^"]*"/& xmlns:media="http:\/\/search.yahoo.com\/mrss\/"/g' $TMP_FILE | sed -e 's/\(http:\/\/twitpic.com\/show\/thumb\/[^"]*\).*/&\n <media:thumbnail url="\1" height="150" width="150" \/>/g' > $FEED_FILE

I know it's possible to consilidate the two sed commands into one and do it in one pass, but this works. I may tweak it in further revisions. It is not necessary to use a yahoo-defined media tag, so I might modify the script later on to simply transform the CDATA portion into parseable encoded HTML. I might also add that I'm using Feedburner to host the feed. Basically, I change the file on my server, and Feedburner goes there to get it, and offers it to the rest of the world. That way if my server is offline, the feed is still active and available, and I don't have to deal with the traffic, just the Feedburner hits. If anyone else wants their TwitPic feed to have thumbnails available, let me know, and I can set one up for you on my server through Feedburner. (It's pretty easy, since the TwitPic username is passed in to the script as a parameter). I can't guarantee anything, but since it's in my interest to keep the script working and up-to-date, you don't have much to worry about. All I need to know is your TwitPic (Twitter) username.

• Update (2009-04-16): I have modified the code to accept all image formats, and be shorter.

The Google Reader Commenting Problem

Google Reader is an excellent feed reader. Let me just go right out and say that. It does an awesome job of gathering together the content from all my feeds, organizing it, and presenting it to me. When you have all that content together in one place, sometimes you want to share your thoughts on what you read, or point out an article to your friends so that they can read it as well. Google Reader has a feature both to "Share" and "Share with note" which perform their stated purposes. There is also a bookmarklet available so that if you come across something on the web that you want to share, you can do so even if you are not subscribed to a feed of it. The problem arises when you have a group of friends who all like to share things back and forth: sometimes an article starts a discussion, and two or more friends wish to have a forum for that discussion. The discussion was started over Google Reader, so the natural thing would be for Google Reader to provide a seamless arena for that discussion to occur. The way that Google Reader implements sharing "under the covers" (as we in Computer Science like to say) is that each user has what essentially amounts to a blog, where all of our shared items and our comments about them are stored. This blog page is accessible under "Shared items" and it has a web page, and a feed of its own. When a friend shares their shared items with you, you are simply granted access to their feed. There is even a web page at which your blog can be accessed, which, if you wish, you can share with your friends and/or the world. (The URL for this blog contains a unique identifier, which would be nearly impossible to guess, in order to protect your privacy--but only if you wish to keep it a secret.) Now, herein lies the problem: when I share something and comment on it, all my friends see that comment and the shared article. When Josh, who is my friend, sees this article and my comment, he can also share the article with his own comment: which will be seen by his friends, including me. But the set of my friends is disjoint from the set of his friends. The problem compounds as more people wish to participate in the discussion, since there will be a growing cloud of people on the edges of the friend network who (1) are being repeatedly shared the same article, and (2) do not have access to the whole discussion or are uninterested in it. The request to have an integrated, more fully functional system for story commenting has been brought before Google, and is currently being ignored. I think this is because in order to fix the problem, they would need to change the basic architecture of Google Reader, and they're not prepared to do that. Google Shared Stuff showed some promise, but it never offered a comment feature, and is being discontinued. Friend Connect might someday fill this niche. The ideal use case would be that every time someone shares an article, a new forum would be created for that article. If one of your friends has shared an article before you, you have the option of starting your own forum on that article, or joining the existing one, thereby making it available to all of your friends who weren't friends with the original forum creator. The privacy side-effect of this is that when you participate in one of these forums, your comments can potentially be passed along to anyone, if they're a friend of a friend of a friend. I don't see that as bad, but Google Reader is built on the assumption that if I want my comments to be exclusive to only my friends, Google Reader isn't going to pass them along to anyone else. In the use case I'm thinking of, articles that had been shared would be annotated like this:

• Josh: "I think this is hilarious."
  • 3 replies | Reply
• Brian: "I think these people should be locked up and the key thrown away!"
  • 0 replies | Reply
• Share | Start discussion | Ignore

Design goals in setting up the above example:

• Maintain current functionality by allowing a user to see his friends' notes on shared items when they come up in his reader.
• Co-locate friends' comments, so that a user can see what each of his friends said about an item in one place.
• Ensure that the item appears when a friend starts a new discussion, but allow users to prevent a popular discussion they are not interested in from becoming annoying by repeatedly popping up (hence the "Ignore" option).
• Allow a user to create a forum in order to share a comment that can be replied to by any of his friends, and any of their friends, who will now be able to see the discussion.
• Notify other users that a forum has been created (or joined) by one of their friends when they see the article, so that they will not create redundant discussions unless that is there intent.

Frankly, I don't see this happening, especially since Google Reader users have been promised the privacy of their comments, and this system allows comments to propagate along with items over the friend network. Come to think of it, I'm not sure I want Google Reader to turn into something like the above, since it would inevitably be abused. After all, MySpace wasn't that bad of an idea in concept, it's just that the users were given more control than they could handle responsibly, and the result is quite ugly. Most of what I want to read is what I'm subscribed to, and I'm willing to take a look at something that my friends think is especially great that they've read. So, the options are as follows:

1. The status quo: continue to complain, hoping that Google will fix it someday. Lower expectations in the meantime and refrain from attempting to have 'conversations' about Google Reader shared content, being content with mere comments.
2. Integrate a 3rd party solution: look for another service to which Google Reader conversations can be redirected in a relatively seamless manner.
   1. If none are satisfactory, create our own. IPO and retire as millionaires within a decade.
3. Ditch Google Reader: stop using Google Reader as the primary feed reader, and switch to another feed reader that provides a conversation thread for each shared item.
   1. This would involve getting the entire group of friends to switch to a new network for sharing (not easy).
   2. I don't know that there are any that actually fit this bill, since this isn't so much a reader feature as a social feature.
4. Split the difference: Use Google Reader for what it's good for--reading feeds and commenting on why you're sharing that particular feed, not what someone else said about it. If you want to start a discussion, start it elsewhere: e-mail, FriendFeed, Facebook, etc.

• Right off the bat, I know FriendFeed can be a good supplement to Google Reader, and I'm trying to work out exactly how the two can fit together for discussions originating on Google Reader.
• Facebook has this sort of comment system down pat, but they don't have a reader, and I don't want to spam all of my Facebook friends with every story I share unless they specifically want it. Mostly though, Facebook doesn't prioritize maintaining users' privacy and ownership of data.
• There's a Firefox extension that turns any page into a chat box: Socialbrowse. However, it looks like it's simply a public comment system like Digg and reddit. I'm looking more for a system where comments and articles are primarily shared with friends.

If you can't tell, I'm a verbal processor, and in the process of organizing the information laid out in this blog post, I have convinced myself that the best course of action is to encourage those who wish to have back-and-forth discussion about an online article to take those discussions to a service that specializes in that sort of thing.

• E-mail is a perfectly legitimate system for a discussion thread, and Google Reader seems to have anticipated this by providing an "Email" feature, located right next to "Share with note".
• For more casual discussions, where the attention demanded by an e-mail would be too much, I think FriendFeed is the way to go. I have all my Google Reader shared items automatically show up on my FriendFeed, and if you have get an account, you can start a comment thread on any item.

Google Reader vs. FriendFeed

It seems that I've been introducing some of my friends to the features of various online tools a lot lately. Not that I've been going out of my way, but it seems that I'm "that guy" who is both "in the know" about tech stuff and willing to answer questions. A friend of mine recently asked me about Gmail. Her frustration was with the fact that the inbox was perpetually cluttered, and there was no way to clear out the old e-mails she had already dealt with without deleting them. So she thought, that is, until I told her what that "Archive" button does. Without it, there is no difference between the Inbox and "All Mail". Labels (aka "Tags") are also an indespensible feature of Gmail when used properly, but that's not my topic. Recently, I started using FriendFeed, and I thought I would explain what how I got there, what it is, and how its niche differs from that of Google Reader. Recently, I started using Twitter. It began with just subscribing to a friend's Twitter feed in Google Reader (my feed (i.e., RSS, Atom) reader of choice), but it would only refresh the Twitter content about once every couple of days, so I would get nothing for a while, and then about ten posts all at once. When things come in little bits, they're time-sensitive, and so I decided to get a Twitter account of my own in order to keep track of my friends' feeds in a timely manner, and be able to reply if desired. Thus, I became a Twitter lurker. However, it didn't take me long to figure out how to hook up Twitter to my Facebook status, making my "tweets" (another 'word' I dislike) a replacement for updating my Facebook status, and transforming me into a 'real' Twitter user. That wasn't enough, however. Apparently, my pallet for up-to-the-minute online content would not be satisfied until I found FriendFeed. When I first encountered FriendFeed, it looked redundant to Twitter's funcitonality, with the added detriment that none of my friends (and only one of my acquaintances) actually used it (at the time). This turned out not to be the case upon further inspection. FriendFeed is to Twitter what Google Reader is to Blogs. Yes, it will keep you informed of your friends' latest updates to Twitter, but it will also keep you informed of their activities on Digg, Reddit, Flickr, Picasa Web, their shared articles (on any number of services), and absolutely anything else that can be accessed through a feed. The problem was, since none of my friends were using FriendFeed, that none of their content and activity was available there. Enter the "imaginary friend" feature. Say I have a friend who uses Blogger, Google Reader, Facebook, Twitter, and Digg, but this Friend does not use FriendFeed. I can create an "imaginary friend", complete with nickname and profile picture, that aggregates this friend's content from all the various sites and makes it visible in myfeed exactly as if that friend had signed up and entered all those services as their own. All I need to provide, depending on the service, is their account's display name or feed URL. But why, you might ask, would I need yet another feed aggregator, when I am already using Google Reader? The difference in the way that these two services are used is that while Google Reader provides a hub for consuming content, FriendFeed provides a hub for observing events. Google Reader is all about pulling in the meaty content: reading the whole article. Google Reader, therefore, displays all of the text and multimedia that it can, and keeps track for you of what you have and haven't read. FriendFeed, on the other hand, is a stream of things that "happened": so-and-so posted on their blog, this person updated their status, that person posted a photo or video. The substance of the content is not displayed on FriendFeed: only that it happened and a title, short snippet, or small thumbnail. When an event occurs, it appears as quicly as possible at the top of the page. Once something drops off the page, it's irrelevant. There are items that appear both in Google Reader and FriendFeed, but there are also items that only appear on one or the other. FriendFeed is all about, well, friends, therefore news sites, webcomics, and blogs by people I don't know are much better suited exclusively to Google Reader, where nothing is skipped, and information is handled exhaustively. By contrast, items with very little content, or of a transient nature, are better suited to FriendFeed, where they can be noticed and optionally interacted with should the occasion arise to do so.

Web Feeds and Aggregators: Thoughts

I just "discovered" Google Reader. Oh, no, I knew it was there all along. It's even one of Firefox's default feed subscription options. I had simply been ignoring its existence this whole time, content to use Firefox's Live Bookmarks feature for all my RSS/Atom needs. For the uninitiated, a "web feed" is a way to "subscribe" to the content of a website, such as a blog, news outlet, podcast, or just about anything these days. After subscribing to a web feed, a visitor is automatically notified of new content on that website by their feed reader of choice. There are quite a few out there, including Firefox's Web Feeds feature, Google Reader, the Opera Browser, and Thunderbird. Basically, in stead of having to go to every website to see if there is new content available, the reader can subscribe to the websites' feeds, and will be automatically notified of any new content on each site. The problem is that with some feed aggregators, they simply pull all of your content off the site, and allow the readers to get the content without visiting the site. This becomes problematic for ad-supported websites, which typically either draw the readers to the site by providing unique participatory content, such as a discussion forum or comments, by only providing a summary of the actual content in the feed, or by injecting ads into the feed. I am not an ad-supported website, but I do like my readers to interact with me and each other through comments. If none of the users are drawn to my actual website, then none of them will see each other's comments. The other thing I like to do is keep track of roughly how many people are reading my blog, and blogger doesn't provide tools to track users on the site itself, much less the feed. They do provide a mechanism to insert something into the feed at the bottom, which could be used to tally readers. What I have been doing is having the feed only contain the first paragraph or so of the post, and then the readers are directed to the post's actual page. This may prove inconvenient for some readers, although I was trying out Thunderbird, and what it did with my blog was to simply load the post's page directly into the reading frame, which is actually ideal from my perspective. Other readers, particularly aggregators such as Google Reader only display the text and image content, and use their own formatting. I have been considering switching the feed to contain the entire post, but I'm not yet sure. What do you think? Is anyone actually reading this? Do you use web feeds? What reader(s) do you use? Do you prefer to have blogger format the post, or do you prefer your reader's formatting? I have decided that for the time being I will try out Google Reader for all of my friends' blogs, and for newsletters that I read every time, but for news sites where I tend to cherry-pick the articles, I'm sticking with Firefox's live bookmarks: it gives you a menu of the latest posts, with the ones you've read already grayed out. I wouldn't want my feed list to get clogged with every article on Ars Technica, Slashdot, Technocrat, and certainly not Digg.