Gentoo is not ready for systemd

This is mostly just a rant into the aetherwebs, and it won't make a lick of sense to someone who doesn't manage a Gentoo Linux system.

You have been warned.  Move along.

Yeah, I know systemd is optional (for now) on Gentoo.  The thing is, unless you take specific steps to avoid it, it just happens to your system.  This is a bad way to introduce such a fundamental change.

Several months (a year?) ago, my emerges simply stopped working.  There were package conflicts all over the place, and I didn't have time to untangle everything and get to the bottom of it.  So, I just stopped updating.  Obviously, as a Gentoo user, I hated this.

Now, months later, I decided to build up a new system on my newly-obsoleted laptop.  It looks to me like systemd is the future, so I go that route.

Here's what doesn't work, out of the proverbial box:

• Service configurations usually adjusted in /etc/conf.d/
• DHCP
• crons
• webalizer
• Dropbox (at least, the way I use it)

I was able to figure out how to get dhcp working via NetworkManager, and that got me to the point of a functional-ish system at boot.  That was the easy part.  Next, I transitioned what system cron jobs would fit into cron.hourly and such.

I still haven't figured out how to set up user-specific timer jobs that run on a schedule (say, every 15 minutes).  The file formats are simple enough for services and timers, but I can't get ssytemctl to recognize the files I place in ~/.local/share/systemd/user/.  In theory, though, that should all work, and it would be worth figuring out and moving forward.

It's also worth noting that the Gentoo Handbook and other documentation has thus far not been updated to support users building with systemd by default.

Here's what is flat-out broken for me, that I had working in my previous system:

• Dropbox running in the background, launched the first time my user logs in.  Since Gentoo completely disables running anything out of /etc/init.d, I would need to write my own service file, which I'm not prepared to do.
• Webalizer running just before the apache log gets zipped up by systemd.  This is probably as simple as specifying "run this task just before you rotate the apache log", but I don't know enough about the guts of systemd's log rotation for that.

For now, I think I'm going to revert my new system back to OpenRC.  It's the path of least pain, now that I know how to exorcise systemd from a box.  During this whole process, I got my old system's portage tree untangled.  I basically had to move the world file to a temporary location, untangle the system dependencies, depclean everything, restore the world file, and re-emerge it all. 

See, most of what my Linux system does is act as a web server, and a Dropbox client.  I have several cron jobs that rename files in Dropbox which get uploaded via IFTTT and other automated means, and sometimes these files have characters in their names that aren't valid on Windows.

Suffice to say that right now a systemd-based Gentoo system significantly fails to meet my specific needs.  If it was just a desktop workstation, and I wanted the latest version of Gnome I would probably stick with that.

Hopefully these issues will get resolved in the future.  Maybe I'll even help that to happen.  In the meantime, my main Gentoo system is dropping systemd. I have other priorities.

Trying VirtualBox

In my last post, I compared Windows VirtualPC to VMWare Player.  I thought I would also try out Oracle's VirtualBox to see how it stacks up against the other two.

VirtualBox is in many ways very similar to VirtualPC.  When you launch VirtualBox, you get a management window with your virtual machines in a column on the left, and configuration options on the right.  From there, you can launch them in a separate window.

The integration tools install in the same manner as the other two (virtual CD-ROM with auto-run installer).

One problem I ran into was that at first I couldn't take screenshots of just the VirtualBox VM window using Alt+Print Screen.  This is because the VM was capturing my keystrokes when the window was selected.  To fix this, I needed to disable the "Auto Capture Keyboard" setting under File -> Preferences | Input.  Once that was configured I was able to take screenshots without problem.  Interaction isn't seamless with this setting, but I actually prefer this for my purpose, which is to run old games, because then there's no danger of scrolling off of the screen.

I like the default escape key of right Ctrl, since it's easy to find with my finger whether my fingers are using the keyboard or the touchpad of my laptop.

In order to get a cool screenshot, I attempted to load all three of the VMs: VirtualPC, VMWare, and VirtualBox.  I was able to launch the first two, but then VirtualBox wouldn't load the VM because it couldn't get full control of the CPU the way it wanted, and my PC slowed to a crawl.  I think this was because of the amount of RAM I had allocated to the clients, but also running two different kinds of VM is probably a bad idea.  I know I can run two VMWare VMs without problem, so I think it's just the host programs that conflict, and not the fact that there was more than one of them.

Anyway, I think VirtualBox is an excellent choice.  Mouse performance was just as good as VirtualPC, and since it's an open source product, they have no reason to hold back full features like snapshots that aren't supported anymore in VirtualPC and that VMWare reserves for its Workstation product.

Because of these extra features, I would recommend VirtualBox over VirtualPC and VMWare Player.

My Take on Windows Live Essientials

Microsoft just released their Live Essentials suite of software downloads for Vista and Windows 7 machines.  I've been using them since January 2009.  Here are my thoughts.

You shouldn't install them all, first of all.  When you download and run the installer, you get the choice to install everything, or pick and choose.  Make the latter selection.  If you have a previous version of something, they won't give you the choice not to upgrade, so if there's something you don't want to install, quit the installer and uninstall it first.

Which programs in particular to install will be an individual choice.  I already had Mail, Writer, Photo Gallery, and Windows Live Mesh installed from the beta.  I had installed Microsoft Office since last updating the software, and so the installer offered me the "Outlook Connecter Pack".  I'm not sure what it is, but it probably won't hurt.

Speaking of hurt, though, unless you really, really want it, don't install the Bing Bar.  It's just a bad idea.  It will try to take over all of your browsers, and seriously, who needs a toolbar in their browser? 
 
I've never tried the updated Messenger, Messenger Companion, or Family Safety.  I hardly ever use my hotmail account to chat, and I use Pidgin when I do, so I don't really have a use for the Messenger enhancements.

Writer is apparently a very good blogging tool that works with a lot of popular blogging sites (like Blogger, which hosts this blog), but so far, I've stuck with the web interface for composition.
By far the most useful tool is Windows Live Mesh.  If you're like me, you have a bunch of pictures, music, files, and other documents on various computers.  The file sets are simply too large to fit into a free Dropbox account, and you don't really need access to them over the web, at least not most of them, you just want them on your various computers.  It's a hassle to keep all of photos or music organized in more than one place, so you don't.  You keep them organized in one place, and (hopefully) make periodic backups to another computer just in case. Well, Live Mesh allows you to keep it organized the way you want it, everywhere you want it, and it doesn't matter how big the files are, because Microsoft isn't going to store any of them (except for a special 5GB folder, which it will store in the cloud and allow you to access from anywhere on the web.)

Microsoft doesn't upload your files to its servers, but it does keep track of them for you.  Any change you make to your shared folders gets copied to the other computers where that folder is synced, and the copying is peer-to-peer, so if you're at home, it happens at the speed of your home network.  It will also keep your files in sync even if you're not at home, directly from your other computer, not through their servers.

The management interface is pretty simple, though it's easy to miss the "Remote" settings, which allow you to connect to your computer over the Internet if you have enabled it on that device.  Connect is a lot like Remote Desktop, if you're familiar with that.  Basically, it's just like you're sitting at the other computer.  You have to be running MSIE on the computer you're connecting from.

The web interface is a lot like the desktop interface, except in addition to your shared folders, you also have access to all of your devices as well, and you can see which devices sync to each of your folders.

Update:  After installing  Windows Live Mesh on my wife's new netbook, she experienced extremely slow performance.  Her netbook has a 2GHz x64 processor and 2GB of ram, so it wasn't simply the fact that it was a netbook that was making it slow.  I opened Task Manager, and found that the MOE process ("Mesh Operating Environment") was consistently taking up 40 - 60% of the CPU.  I shut down the process, and deleted the "Run" entry from the registry to disable it starting up automatically.  Any syncing that happens will need Live Mesh to be started manually.  I also observed similar behavior on my laptop, but the media PC (which is on all the time) has the MOE process taking only 3 - 5% of the CPU.  It's probably checking the synced files for updates every time it starts up.

Anyway, be warned: Windows Live Mesh is a resource hog on machines that need to turn on and off all the time.

The Cure For Vista Media Center's Insomnia

At home we have a computer hooked up to our TV.  It has two TV tuners, and is set up with Windows 7 Media Center to record our favorite television shows (and automatically detect and skip commercials).  It's wonderful.  We can also watch Blu-ray movies, access Netflix, Hulu, Amazon Unbox, and any number of other streaming services right from our living room TV. 

This post is not about that computer.  This post is about my laptop.  Occasionally, when traveling, or when simply in another room of the house, I like to use Windows Media Center on my laptop.  Currently, it's running Windows Vista, so in order to be able to view shows in the .wtv format used by Windows 7's Media Center, I have installed the "TV Pack" unofficially leaked by Microsoft.

It works great.  Basically, I browse to the file I want to watch on a shared drive (I have a shortcut to the Recorded TV folder on the desktop), double-click it, and it plays on the laptop.  The commercial scan files are automatically synced. When traveling, I usually copy what I want to watch to my Laptop's hard drive, but there's also this.

So, what's the problem?  My laptop won't sleep.  Or hibernate.  At least not all night.  It wakes up at 3:30 AM to download the latest TV listings, even though I never configured it to work with a tuner, so it has no listings to download.  Needless to say, this is annoying.  It drains my battery unnecessarily, and if it's in its case, there's a danger that it will overheat.

After living with this problem, usually dealing with it by shutting the laptop down every time I stop using it--which means a cold boot every time I start using it, and it takes a while to load everything up--I finally found the solution to my problem.  Step 9 on this page points you in the right direction, but here's how you do it:

Launch the Task Scheduler.  You can do this by opening the Start Menu, typing "Task Scheduler", and pressing Enter.  You will get a UAC prompt, which you should authorize. 

In the left pane, click on the arrows left of the text to expand down to the following item: Task Scheduler (Local) -> Task Scheduler Library -> Microsoft -> Windows -> Media Center

Once you have selected Media Center, look on the top middle pane for a task named mcupdate_scheduled.  Double-click this task to load the Properties window.

In the Properties window, click the Conditions tab, and uncheck the box next to "Wake the computer to run this task".

Click OK, and close Task Scheduler.  That's it.  No more waking up from hibernate or sleep in the middle of the night!

TwitPic to Posterous Export Script

Note: This post (and the script it contains) has been updated as of December 14, 2010.  (v1.4.0) The script can also be downloaded from my server here.

Also, Posterous has done a lot of work on solving this problem since I wrote my script.   You can see their latest solutions here.

Recently, I switched from TwitPic to Posterous as my method of posting phone pictures (and now video) to the Internet.  But since I switched, I didn't want to have my data history split in two, so I decided to write a script to download each of my TwitPic images with their associated text and date, and upload them to Posterous with the same information.

Initially, I wanted to make one long post with all of the images, and their text below.  However, with the Posterous API, it isn't possible to refer to a specific image in your body text, so individual posts is the way I went.

Along the way, I became familiar with yet another Linux command: curl.

I love that Posterous has an API that (once you figure out curl) is pretty easy to use.  TwitPic, on the other hand, has absolutely zero support for exporting anything.  The fact that they're so non-user-centric and out-dated was a driving force in my switching.  The only reason I hadn't switched to img.ly already was because img.ly has a bug that prevents images sent from my phone from being posted, since my phone sends them without a file extension.  I worked with their tech support for a while, but they didn't fix it.  I got a new phone, but it was also a Samsung, and it did the same thing with images.  Oh, well.  Posterous is better.

Anyway, here is the script:

First run it with just the first two arguments, and it will download all of your TwitPic data, including thumbnail images.  Once you're satisfied, supply your Posterous User ID, Password, and Site ID.  (If you don't know your Site ID, run the script with your Posterous User ID, Password, and no Site ID, and it will query your Posterous site info as long as your Posterous credentials are valid.)

Note: if you want to run this from Windows, you should install Cygwin (with, at a mimum, curl and sed) and run it from there.

./twitpic-to-posterous.sh [twitpic-id] [working-dir] [postrous-id] [posterous-password] [posterous-site-id] [skip-number]

#!/bin/sh

# Copyright 2010 Tim "burndive" of http://burndive.blogspot.com/ and http://tuxbox.blogspot.com/
# This software is licensed under the Creative Commons GNU GPL version 2.0 or later.
# License informattion: http://creativecommons.org/licenses/GPL/2.0/

# This script was obtained from here:
# http://tuxbox.blogspot.com/2010/03/twitpic-to-posterous-export-script.html

RUN_DATE=`date +%F--%H-%m-%S`
SCRIPT_VERSION_STRING="v1.4.0"

TP_NAME=$1
WORKING_DIR=$2
P_ID=$3
P_PW=$4
P_SITE_ID=$5
UPLOAD_SKIP=$6

# Comma separated list of tags to apply to your posts
P_TAGS="twitpic"
# Whether or not to auto-post from Posterous
P_AUTOPOST=0
# Whether or not the Posterous posts are marked private
P_PRIVATE=0

# This is the default limit of the number of posts that can be uploaded per day
P_API_LIMIT=50

DOWNLOAD_FULL=1
DOWNLOAD_SCALED=0
DOWNLOAD_THUMB=0
PREFIX=twitpic-$TP_NAME
HTML_OUT=$PREFIX-all-$RUN_DATE.html
UPLOAD_OUT=posterous-upload-$P_SITE_ID-$RUN_DATE.xml

if [ -z "$TP_NAME" ]; then
  echo "You must supply a TP_NAME."
  exit
fi
if [ ! -d "$WORKING_DIR" ]; then
  echo "You must supply a WORKING_DIR."
  exit
fi
if [ -z "$UPLOAD_SKIP" ]; then
  UPLOAD_SKIP=0
fi
UPLOAD_SKIP_DIGITS=`echo $UPLOAD_SKIP | sed -e 's/[^0-9]//g'`
if [ "$UPLOAD_SKIP" != "$UPLOAD_SKIP_DIGITS" ]; then
  echo "Invalid UPLOAD_SKIP: $UPLOAD_SKIP"
  exit
fi

cd $WORKING_DIR

if [ -f "$HTML_OUT" ]; then
  rm -v $HTML_OUT
fi

# If Posterous username and password were supplied, but not site ID, query the server and exit.
P_SITE_INFO_FILE=posterous-$P_SITE_ID.out
if [ ! -z "$P_ID" ] && [ ! -z "$P_PW" ] && [ -z "$P_SITE_ID" ]; then
  echo "Getting Posterous account info..."
  curl -u "$P_ID:$P_PW" "http://posterous.com/api/getsites" -o $P_SITE_INFO_FILE
  SITE_ID_RET=`grep "<id>$P_SITE_ID</id>" $P_SITE_INFO_FILE`
  if [ -z "$SITE_ID_RET" ]; then
    echo "Please supply your Posterous Site ID as the fifth argument."
    echo "Here is the response from the Posterous server.  If you entered correct credentials, you should see your Site ID(s):"
    cat $P_SITE_INFO_FILE | tee -a $UPLOAD_OUT
    exit
  fi
fi

# Confirm that we have a valid Posterous Site ID
if [ ! -z "$P_SITE_ID" ]; then
  echo "Getting Posterous account info..."
  curl -u "$P_ID:$P_PW" "http://posterous.com/api/getsites" -o $P_SITE_INFO_FILE
  SITE_ID_RET=`grep "<id>$P_SITE_ID</id>" $P_SITE_INFO_FILE`
  if [ -z "$SITE_ID_RET" ]; then
    echo "Make sure that you have supplied a valid Posterous Site ID as the fifth parameter.  If you don't know your Site ID, leave it out, and this script will query the server."
    echo "Here is the response from the Posterous server.  If you entered correct credentials, you should see your site ID(s):"
    cat $P_SITE_INFO_FILE | tee -a $UPLOAD_OUT
    exit
  fi
fi

MORE=1
PAGE=1
while [ $MORE -ne 0 ]; do
  echo PAGE: $PAGE
  FILENAME=$PREFIX-page-$PAGE.html
  if [ ! -s $FILENAME ]; then
    wget http://twitpic.com/photos/${TP_NAME}?page=$PAGE -O $FILENAME
    if [ ! -s "$FILENAME" ]; then
      echo "ERROR: could not get $FILENAME" | tee -a $LOG_FILE
      sleep 5
    fi
  fi
  if [ -z "`grep "More photos &gt;" $FILENAME`" ]; then
    MORE=0
  else
    PAGE=`expr $PAGE + 1`
  fi
done

ALL_IDS=`cat $PREFIX-page-* | grep -Eo "<a href=\"/[a-zA-Z0-9]+\">" | grep -Eo "/[a-zA-Z0-9]+" | grep -Eo "[a-zA-Z0-9]+" | sort -r | xargs`

# For Testing
#ALL_IDS="1kdjc"

COUNT=0
LOG_FILE=$PREFIX-log-$RUN_DATE.txt

echo $ALL_IDS | tee -a $LOG_FILE

for ID in $ALL_IDS; do
  COUNT=`expr $COUNT + 1`
  echo $ID: $COUNT | tee -a $LOG_FILE

  echo "Processing $ID..."
  FULL_HTML=$PREFIX-$ID-full.html
  if [ ! -s "$FULL_HTML" ]; then
    wget http://twitpic.com/$ID/full -O $FULL_HTML
    if [ ! -s "$FULL_HTML" ]; then
      echo "ERROR: could not get FULL_HTML for $ID" | tee -a $LOG_FILE
      sleep 5
    fi
  fi
  TEXT=`grep "<img src=" $FULL_HTML | tail -n1 | grep -oE "alt=\"[^\"]*\"" | sed \
        -e 's/^alt="//'\
        -e 's/"$//'\
        -e "s/&#039;/'/g"\
        -e 's/&quot;/"/g'\
        `
  if [ "$TEXT" = "" ]; then
    TEXT="Untitled"
  fi
  echo "TEXT: $TEXT" | tee -a $LOG_FILE
  # Recognize hashtags and username references in the tweet
  TEXT_RICH=`echo "$TEXT" | sed \
        -e 's/\B\@\([0-9A-Za-z_]\+\)/\@<a href="http:\/\/twitter.com\/\1">\1<\/a>/g' \
        -e 's/\#\([0-9A-Za-z_-]*[A-Za-z_-]\+[0-9A-Za-z_-]*\)/<a href="http:\/\/twitter.com\/search\?q\=%23\1">\#\1<\/a>/g' \
        `
  echo "TEXT_RICH: $TEXT_RICH" | tee -a $LOG_FILE

  # Convert hashtags into post tags
  P_TAGS_POST=$P_TAGS`echo "$TEXT" | sed \
        -e 's/\#\([^A-Za-z_-]\)*\B//g' \
        -e 's/^[^\#]*$//g' \
        -e 's/[^\#]*\(\#\([0-9A-Za-z_-]*[A-Za-z_-]\+[0-9A-Za-z_-]*\)\)[^\#]*\(\#[0-9]*\B\)*/,\2/g' \
        `
  # Uncomment if you don't want hashtags converted into post tags
  #P_TAGS_POST=$P_TAGS

  # Add custom tags from a file (optional).  The file is formatted like this:
  # ,tag1,tag2,tag3
  TAGS_FILE=$PREFIX-$ID-tags-extra.txt
  if [ -s "$TAGS_FILE" ]; then
    P_TAGS_POST=$P_TAGS_POST`cat $TAGS_FILE`
  fi
  echo "P_TAGS_POST: $P_TAGS_POST" | tee -a $LOG_FILE

  TEXT_FILE=$PREFIX-$ID-text.txt
  if [ ! -s $TEXT_FILE ]; then
    echo "$TEXT" > $TEXT_FILE
  fi
  FULL_URL=`grep "<img src=" $FULL_HTML | grep -Eo "src=\"[^\"]*\"" | grep -Eo "http://[^\"]*"`
  echo "FULL_URL: $FULL_URL" | tee -a $LOG_FILE

  SCALED_HTML=$PREFIX-$ID-scaled.html
  if [ ! -s "$SCALED_HTML" ]; then
    wget http://twitpic.com/$ID -O $SCALED_HTML
    if [ ! -s "$SCALED_HTML" ]; then
      echo "ERROR: could not get SCALED_HTML for $ID" | tee -a $LOG_FILE
      sleep 5
    fi
  fi
  SCALED_URL=`grep "id=\"photo-display\"" $SCALED_HTML | grep -Eo "http://[^\"]*" | head -n1`
  echo "SCALED_URL: $SCALED_URL" | tee -a $LOG_FILE
  POST_DATE=`grep -Eo "Posted on [a-zA-Z0-9 ,]*" $SCALED_HTML | sed -e 's/Posted on //'`
  echo "POST_DATE: $POST_DATE" | tee -a $LOG_FILE

  THUMB_URL=`cat $PREFIX-page-* | grep -E "<a href=\"/$ID\">" | grep -Eo "src=\"[^\"]*\"" | head -n1 | sed -e 's/src=\"//' -e 's/\"$//'`
  echo "THUMB_URL: $THUMB_URL" | tee -a $LOG_FILE

  EXT=`echo "$FULL_URL" | grep -Eo "[a-zA-Z0-9]+\.[a-zA-Z0-9]+\?" | head -n1 | grep -Eo "\.[a-zA-Z0-9]+"`
  if [ -z "$EXT" ]; then
    EXT=`echo "$FULL_URL" | grep -Eo "\.[a-zA-Z0-9]+$"`
  fi
  echo "EXT: $EXT"
  if [ "$DOWNLOAD_FULL" -eq 1 ]; then
    FULL_FILE="$PREFIX-$ID-full$EXT"
    if [ ! -s $FULL_FILE ]; then
      wget "$FULL_URL" -O $FULL_FILE
      if [ ! -s "$FULL_FILE" ]; then
        echo "ERROR: could not get FULL_URL for $ID: $FULL_URL" | tee -a $LOG_FILE
        sleep 5
      fi
    fi
  fi
  if [ "$DOWNLOAD_SCALED" -eq 1 ]; then
    SCALED_FILE=$PREFIX-$ID-scaled$EXT
    if [ ! -s $SCALED_FILE ]; then
      wget "$SCALED_URL" -O $SCALED_FILE
      if [ ! -s "$SCALED_FILE" ]; then
        echo "ERROR: could not get SCALED_URL for $ID: $SCALED_URL" | tee -a $LOG_FILE
        sleep 5
      fi
    fi
  fi
  if [ "$DOWNLOAD_THUMB" -eq 1 ]; then
    THUMB_FILE=$PREFIX-$ID-thumb$EXT
    if [ ! -s $THUMB_FILE ]; then
      wget "$THUMB_URL" -O $THUMB_FILE
      if [ ! -s "$THUMB_FILE" ]; then
        echo "ERROR: could not get THUMB_URL for $ID: $THUMB_URL" | tee -a $LOG_FILE
        sleep 5
      fi
    fi
  fi

  BODY_TEXT="$TEXT_RICH <p>[<a href=http://twitpic.com/$ID>Twitpic</a>]</p>"

  # Format the post date correctly
  YEAR=`echo "$POST_DATE" | sed -e 's/[A-Z][a-z]* [0-9]*, //'`
  DAY=`echo "$POST_DATE" | sed -e 's/[A-Z][a-z]* //' -e 's/, [0-9]*//'`
  MONTH=`echo "$POST_DATE" | sed -e 's/ [0-9]*, [0-9]*//' | sed \
    -e 's/January/01/' \
    -e 's/February/02/' \
    -e 's/March/03/' \
    -e 's/April/04/' \
    -e 's/May/05/' \
    -e 's/June/06/' \
    -e 's/July/07/' \
    -e 's/August/08/' \
    -e 's/September/09/' \
    -e 's/October/10/' \
    -e 's/November/11/' \
    -e 's/December/12/' \
    `
  # Adjust the time to local midnight when west of GMT
  HOURS_LOC=`date | grep -Eo " [0-9]{2}:" | sed -e 's/://' -e 's/ //'`
  HOURS_UTC=`date -u | grep -Eo " [0-9]{2}:" | sed -e 's/://' -e 's/ //'`
  HOURS_OFF=`expr $HOURS_UTC - $HOURS_LOC + 7`
  echo "HOURS_LOC: $HOURS_LOC"
  echo "HOURS_UTC: $HOURS_UTC"
  echo "HOURS_OFF: $HOURS_OFF"
  if [ "$HOURS_OFF" -lt 0 ]; then
    # We're east of GMT, do not adjust
    HOURS_OFF=0
  fi
  if [ "$HOURS_OFF" -lt 10 ]; then
    HOURS_OFF=0$HOURS_OFF
  fi
  if [ "$DAY" != "" ] && [ "$DAY" -lt 10 ]; then
    DAY=0$DAY
  fi
  DATE_FORMATTED="$YEAR-$MONTH-$DAY-$HOURS_OFF:00"
  echo "DATE_FORMATTED: $DATE_FORMATTED" | tee -a $LOG_FILE

  echo "<p><img src='$FULL_FILE' alt='$TEXT' title='$TEXT' /></p>" >> $HTML_OUT
  echo "$BODY_TEXT" >> $HTML_OUT
  echo "  Post date: $DATE_FORMATTED; Count: $COUNT" >> $HTML_OUT

  # Upload this Twitpic data to Posterous
  if [ ! -z "$P_SITE_ID" ]; then

    # First make sure we're under the API upload limit
    if [ "$COUNT" -le "$UPLOAD_SKIP" ]; then
      echo Skipping upload...
      continue
    fi
    if [ "$COUNT" -gt "`expr $UPLOAD_SKIP + $P_API_LIMIT`" ]; then
      echo "Skipping upload due to daily Posterous API upload limit of $P_API_LIMIT."
      echo "To resume uploading where we left off today, supply UPLOAD_SKIP parameter of `expr $UPLOAD_SKIP + $P_API_LIMIT`."
      continue
    fi

    P_OUT_FILE="posterous-$P_SITE_ID-$ID.out"
    if [ -s "$P_OUT_FILE" ]; then
      rm "$P_OUT_FILE"
    fi
    echo "Uploading Twitpic image..."
    curl -u "$P_ID:$P_PW" "http://posterous.com/api/newpost" -o "$P_OUT_FILE" \
      -F "site_id=$P_SITE_ID" \
      -F "title=$TEXT" \
      -F "autopost=$P_AUTOPOST" \
      -F "private=$P_PRIVATE" \
      -F "date=$DATE_FORMATTED" \
      -F "tags=$P_TAGS_POST" \
      -F "source=burndive's Twitpic-to-Posterous script $SCRIPT_VERSION_STRING" \
      -F "sourceLink=http://tuxbox.blogspot.com/2010/03/twitpic-to-posterous-export-script.html" \
      -F "body=$BODY_TEXT" \
      -F "media=@$FULL_FILE"
    cat $P_OUT_FILE  | tee -a $UPLOAD_OUT
  fi
done
echo Done.

This software is licensed under the CC-GNU GPL version 2.0 or later.

PS: If you use my code, I appreciate comments to let me know, and any feedback you may have, especially if it's not working right for you, but also just to say thanks.

For convenience, you can download this script from my server.

Posterous Blogger Sidebar Widget Thumbnail Feed Script

It's been a while since this blog actually lived up to its name and I posted something to do with actual hacking on my Linux box.

You may recall a post a while back where I used the 'sed' command to create a modified copy of my TwitPic feed so that a thumbnail would show up when I imported the feed into a Blog List gadget in Blogger.

Well, I recently switched from using TwitPic for uploading pictures from my phone to using Posterous for uploading pictures and video from my phone.  There were many reasons in the "pros" column, but in the "cons" was the fact that, when I imported my feed into that same Blogger widget, no thumbnail appeared.

So, just like with the TwitPic feed, I set out to modify my Posterous feed in order to get the thumbnail to appear.  One problem I encountered is that the feeds were totally different formats.  I based my TwitPic feed modification on a feed I knew to be working (from Digg), but performing that same transformation on the Posterous feed proved to be problematic.

What I ended up doing was simply extracting the information I needed from the Posterous feed, and then creating a one-item feed in the known-good format.  The feed looks nothing like the original Posterous feed, but that's just fine, since all it will be used for is pulling the latest post into my blog sidebar.

One improvement I'm considering working on is providing a useful thumbnail when I upload a video.  Currently (at least with the 3gp format), the Posterous feed just sticks a generic blank file icon in the thumbnail field.  What I would like is a still frame from the movie.  In order to do this myself, I would need to download the enclosure link, process the video into a still image, post the image on the web, and then put the image URL into the feed.  All very doable given the right tools.

I'll have to test out what happens when I use the MP4 format for video, which my phone is also capable of creating.

Here's my script (so far).  Feel free to use it under the terms of the license listed below.  If you have any questions or suggestions, please feel free to leave a comment.

posterous.sh (run as an hourly cron job):

#!/bin/sh

# Copyright 2010 Tim "burndive" of http://burndive.blogspot.com/
# This software is licensed under the Creative Commons GNU GPL version 2.0 or later.
# License informattion: http://creativecommons.org/licenses/GPL/2.0/

# This script was obtained from here:
# http://tuxbox.blogspot.com/2010/03/posterous-blogger-sidebar-widget.html

DOMAIN=$1
FEED_DIR=$2
FEED_TITLE=Posterous
FEED_DESC="The purpose of this feed is to provide a thumbnail of the latest item in a Blogger sidebar widget."


if [ -z $DOMAIN ]; then
  echo "You must enter a Posterous DOMAIN."
  exit
fi

if [ -z $FEED_DIR ]; then
  echo "You must supply a directory."
  exit
fi

if [ ! -d $FEED_DIR ]; then
  echo "You must supply a valid directory."
  exit
fi

FEED_URL="http://$DOMAIN/rss.xml"
TMP_FILE="/tmp/posterous-$DOMAIN.xml"
FEED_FILE="$FEED_DIR/posterous-$DOMAIN.xml"

# Fetch the RSS feed
wget -q $FEED_URL -O $TMP_FILE

if [ ! -f $TMP_FILE ]; then
  echo "Failed to download $FEED_URL to $TMP_FILE"
  exit
fi

NEW_LATEST=`grep guid $TMP_FILE | head -n1`

if [ ! -f $FEED_FILE ]; then
  FEED_LATEST="" 
else 
  FEED_LATEST=`grep guid $FEED_FILE | head -n1`
fi

# Comment these out
#echo "FEED_LATEST: $FEED_LATEST"
#echo "NEW_LATEST : $NEW_LATEST"

if [ "$FEED_LATEST" = "$NEW_LATEST" ]; then
#  echo "There is no change in the feed."
#  echo "FEED_LATEST: $FEED_LATEST"
  exit
fi

IMG_HTML=`grep -i "img src" $TMP_FILE | head -n1 | grep -Eo "<img src='[^']*'[^>]*>" | sed -e 's/\"/\&quot;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g'`
#echo "IMG_HTML: $IMG_HTML"

IMG_URL=`grep -i "img src" $TMP_FILE | head -n1 | grep -Eo "http:[^']*" | tail -n1`
#echo "IMG_URL: $IMG_URL"

# Create a minimalist RSS feed
echo "<?xml version='1.0'?> " > $FEED_FILE
echo "<rss version='2.0' xmlns:media='http://search.yahoo.com/mrss/'>" >> $FEED_FILE
echo "<channel>" >> $FEED_FILE
echo "<title>$FEED_TITLE</title>" >> $FEED_FILE
echo "<description>$FEED_DESC</description>" >> $FEED_FILE
echo "<link>http://$DOMAIN/</link>" >> $FEED_FILE

echo "<item>" >> $FEED_FILE
grep "<title>" $TMP_FILE | head -n2 | tail -n1 >> $FEED_FILE
grep "<pubDate>" $TMP_FILE | head -n1 >> $FEED_FILE
echo "<description>$IMG_HTML</description>" >> $FEED_FILE
grep "<link" $TMP_FILE | head -n3 | tail -n1 >> $FEED_FILE
echo "$NEW_LATEST" >> $FEED_FILE
echo "<media:thumbnail url=\"$IMG_URL\" height=\"56\" width=\"75\" />" >> $FEED_FILE
echo "</item>" >> $FEED_FILE

echo "</channel>" >> $FEED_FILE
echo "</rss>" >> $FEED_FILE

# Cean up
rm $TMP_FILE

This software is licensed under the CC-GNU GPL version 2.0 or later.

Firefox Extensions Collection

A while ago, I wrote a post in which I create a list of software to install on a new Windows box.  I did this mostly for my own reference, but it might be useful to others.

The first item on my list is Firefox, but (until now) I didn't include any extensions.  Firefox is all about customizations, and extensions are the most powerful way to customize it.  But who wants to go through the trouble of sorting through the thousands of extensions to find the useful ones?  Well, it's more a matter of keeping your ear to the ground and trying out the ones that sound good and/or come recommended.

After years of research, I've created a collection of extensions!  I didn't write any of them, I just bunched them together because they were all useful to me.

I plan on updating the collection as time goes on, so here's the link:

• burndive's essential extensions

You can choose to install them individually, or as a group.  I hope you find them as useful as I have.

Dropbox: File Synchronization

Ever since we got our second computer (back in 1998, I believe), I have been dealing with the problem of how to keep my files in sync between multiple computers.  Initially, I simply didn't, or I used floppy disks to move files back and forth.  Then I bought an Ethernet hub, and used windows shares to pass the data back and forth.  When I bought a CD burner, I would periodically create snapshots of the family's files.

Over the years, I have had many hard drives crash, and many more clean installs.  Solving the file sync problem is often best accomplished in conjunction with backing up those files.

Until recently, I still basically used the LAN solution:  Keep two copies of my files on different computers, and periodically (or sporadically) copy one set of files over the other.  Of course, if you do it this way, you can never change your directory structure, or you have duplicates, and when you try to clean up the duplicates, you lose files.

I also back up important files weekly to an external drive, and I keep four weekly backups, plus six monthly backups.  This process is automated thanks to a customized version of a backup script and some cron jobs on my Linux box.  This part hasn't changed.

Recently, however, I discovered a handy little service called Dropbox.  Dropbox will back up your files, keep them in sync on all of your computers (2 GB for free, pay for more), and enable you to share them with other users if you choose.  I've tried Windows Live Mesh, and I still might use that for remote login, but Dropbox gives you more free storage space, and it is able to sync files from one computer to another over a local LAN (which saves ISP bandwidth).  Also, Dropbox supports Linux, which is a must for me.

[Note: if you want to sign up, use my Dropbox referral link an we'll both get an extra 250MB of free space.]

Dropbox enables some pretty cool syncing tricks if you're willing to roll up your sleeves at the command line.  Here are a few things I'm doing:

• The Linux client for Dropbox treats symlinks to folders as if they were just folders.  Initially, I didn't like this, because it meant I couldn't just plop my existing file structure in place (because it contained symlinks to large data sets in other locations).  Also, I didn't want certain directories synced.  My solution was to simply link to the things I want to sync from my Dropbox folder.  That way, I can structure my directories any way I want, and cherry-pick the things I want to sync from that structure.
  
  
  
  
• I use the Pidgin client for all my Instant Messaging accounts on Windows and Linux.  Pidgin logs all of my conversations, and saves them to a local folder.  Whenever someone IMs me or I open a chat window to IM someone else, the chat window is automatically populated with the latest conversation with that person from the chat log history.  In order to synchronize these logs between computers I created a symlink in the Dropbox folder to the logs folder on my Linux box.  In order to get my Windows pidgin accounts to use this folder, I created a folder "Junction" within the Pidgin AppData folder (.purple) using the command: "mklink /J".
  
  
  
  
• I use DVRMSToolbox along with ShowAnalyzer to automatically find and skip commercials in Windows Media Center.  ShowAnalyzer is run on our living room media PC, and that is where the files are stored that tell the DTBAddin component where the commercials are within a given recorded TV file.  (If you're interested in setting this up yourself, see this guide.)  Normally, I would have to periodically copy new files in the CommercialsXml folder from C:\Public\Users\DvrmsToolbox on the media PC to my laptop in order for my laptop to know when to skip a commercial.  Now, the files are synced automatically, and I don't have to think about them.  I just open my laptop, fire up Media Center, and select the show I want to watch.  It's a pretty sweet setup.
  
  
  
  

Google Wave (Preview)

I just got in on the Google Wave preview.

I immediately had 8 invites, of which I have used 4, so far.  The first one went to my wife, of course.

I like it.  I think it has awesome potential.

It's a "preview" (a.k.a. "beta"--or, since we're talking Google's definition of "beta", it's an "alpha"), so there are lots of rough edges.

Right off the bat, here's what bugs me.  Hopefully the bugs will get fixed.

First of all, here's Google Wave:

So here's the first thing I don't like.  Say I want to narrow down my contacts, to find the one I'm looking for.  I go to the Contacts widget, and type some text:

Okay, so now I want to see the rest of my contacts, in the default view.  What do I press?  There's no little [X] button.  I have to select the text in the box and hit Delete or Backspace.  That is just bad UI design.  There needs to be an [X] right next to the magnifying glass that clears the field.  Google, please fix that.  (Yes, I've already submitted feedback about this.)

This next one is probably just something I'll have to get used to.  When you go to create a new wave, or add someone to the wave, be careful when you click!  Creating a wave or adding someone to it is irrevocable, and the UI doesn't ask you for confirmation, it just shows up in their inbox, and there's nothing you can do about it.

I accidentally added my friend Victor to my first wave, because I clicked his name:

I was expecting there to be a confirmation dialogue, in stead, he was added.  I hope he enjoys my work.

Applications are not easy to find.  There's a link to an Extensions Gallery wave in the introductory "Welcome to Google \/\/ave" wave that starts in your inbox, but all it has is Sodoku and half a dozen other apps.  I'm not knocking the apps.  They're good, but there are a whole bunch more out there that were shown off in the video.

After some poking around, I found this list (and this list) of Bots, Apps and Gadgets.  The one I was most interested in, Bloggy, doesn't seem like it's working.  I added it to a wave, but it didn't do anything.  It's supposed to make the wave public and post it to your blog so that everyone (even those not logged in to Wave) can see it, and interact with it if you allow public editing.  Hopefully they brought it offline so that they could fix it, and that it will be fixed soon. 

One last thing, and then I need to go to bed.  There's an "Options..." menu item that doesn't do anything for me.  I've only tried it in Firefox (and I'm using 3.6b2), but clicking on it doesn't do anything at all.

Just so no one thinks I'm being negative, let me state this plainly:   I'm VERY excited about Google Wave.  I know it's a pre-released product, and I don't expect perfection by any means.  I'm documenting these things here because I want to share my experience, and I would like to see the shortcomings addressed so that the final product will be awesome.

Yay for Google Wave!

Google Wave

Google Wave looks amazing! (For those with 10-minute attention spans, here's the abridged version. The longer version is worth the watch if you have the time.) It's a new Internet communication protocol/framework being developed by Google. It's not a walled garden: anyone can make their own implementation (or even base it on Google's code), and it will be interoperable with everyone else, just like e-mail. But this isn't your grandfather's e-mail. Wave is basically a conversation/collaboration tree, with full version control, history, and really cool tools for mashups, transformations, translations, and anything else anyone on earth can think of and write a plugin for. This is really that revolutionary. Imagine if every e-mail, chat, wiki, invitation, blog post, tweet, photo album, forum, and whatever else on the Internet was as simple to interact with as an item on Facebook or a wiki, only with way more powerful tools, and it wasn't confined to anyone's walled garden. Tomorrow's kids will laugh at us for using Facebook for the same reason that today's kids laugh at their elders for using Juno. I don't expect all of those other technologies to disappear, but to some extent they will be eclipsed, and they will have to keep up or be left behind.

Software To Install On A Windows Box

With the Windows 7 release candidate out, I recently wiped my laptop's hard drive clean and installed it (I was previously running beta 2). Whenever I flatten a computer and re-install everything fresh, I like to have a list of things to install that covers the bases.

I put together a bookmarks folder of software to install on a new Windows box, and I decided to share it with the world, so here it is:

• Software to install on a new Windows box

The nice thing about sharing it in this way is that I can keep it up-to-date in my browser if there is anything to add or remove, or if any of the links change.

Enjoy.

If You're Not A Gentoo User, Move Along...

Whenever I spend time sifting through Internet forums looking for the solution to a technical problem, I like to post a description of the problem along with the solution in a straightforward manner, so that the next person who has the same problem will be able to find my post when they search. This is one such posts. I ran into a frustrating error when updating the software on a Gentoo Linux system that hadn't been updated in a while. I started with:

emerge --sync && emerge -avuDN world

And when that was (finally) done, I ran:

emerge -av --depclean

Which uninstalled a bunch of packages. The problem came when I ran:

revdep-rebuild -- -av

Nothing would emerge. Nothing. Every package I tried to emerge would end in the error:

configure: error: C compiler cannot create executables

The root problem turned out to be a bug with the way that the gcc ebuild updates from certain versions. Portage was configured to use a compiler version that no longer existed on the system. The solution was to run:

binutils-config -l binutils-config num gcc-config -l gcc-config num

Where num above is the numerical selection of one of the items on the list output by the previous command. My gcc-config selection told me when I selected it that it had a bug with GCC_SPECS and to re-emerge it, so I did that. I probably didn't have to, but it's good to be cautious and I had a few hours to kill (this is a 1GHz Pentium III system). After that, I was good to go.

Windows 7 Beta & Windows Live - First Impressions

I just installed the beta version of Windows 7 on my laptop. Thus far, the experience has been generally good. I like what they've done with the Start Menu and Taskbar (though I did tweak the settings slightly), and the new User folder makes organizing your files a seamless experience. I downloaded and installed the Windows Live programs, and I've given a couple of them a try. These programs also work on Vista and XP, so they're not unique to Win7, except that Microsoft developed these programs along with Windows 7 as an answer to what other companies have to offer.

• Windows Live Writer appears to be a blog publishing program. I've messed with it a little bit, but this post is being authored using the Blogger website (running on Firefox, no less), so that ought to give you an indication of how much I'm plugged in to the Microsoft universe.
• Windows Live Messenger is the same old chat program, where you sign in with your Windows Live ID (a.k.a Hotmail/Passport/MSN account). I don't use my Live ID for much, and I don't connect with others using it much either. I do chat over MSN, but my Hotmail account is just one of three IDs on three different services that I use all the time (and the least active of the three), so there is very little chance that I will switch to using Messenger over Pidgin.
• Windows Live Photo Gallery is basically Microsoft's answer to Picasa. It's a relatively slick interface, and from what I hear it integrates with Facebook (and Facebook has a patent on user-initiated facial tagging, so let's hope that's in there somewhere).
• Windows Live Mail is an e-mail program, along the lines of Outlook Express, but hopefully not as cumbersome and annoying. I haven't tried it yet, but it looks like it will work with multiple e-mail providers, not just Microsoft-owned ones.
• Windows Live Call is a VoIP client that works with Messenger (I think).
• Windows Live Family Safety is a console for parents to monitor and control their kids' online activities (though I'm sure it only works with other Live services).
• Windows Live Movie Maker Beta is a video editing and publishing tool. I haven't messed much with video editing, so it's nice to have a free program that does that.

Some of these programs interest me and some don't. As I already mentioned, I'm not likely to switch to Messenger from Pidgin because Pidgin works with Google Talk and AIM as well as Live. The same can be said of MSIE, which lacks indispensible extensions that Firefox has (though I hear Foxmarks is testing a version of their bookmarks manager to work on MSIE as well as Chrome). Writier is worth a try, as is Photo Gallery, and Movie Maker looks interesting if I ever have a need for it. With the release of Windows 7, Microsoft is pushing for modularization, since anyone can develop their own programs to fill these roles and have them seamlessly integrated into the operating system, and from the Application end. Microsoft is providing their Live applications as a download so that they can escape the criticism they've gotten in the past of forcing their services on the users. Some OEMs will install them by default, some will install competing application suites (such as from Google). In the end, I hope that this will result in innovation and competition that benefits all users. If and when I have a chance to explore any of these in the future, I'll try to put a post up about my experience. I want to give the Microsoft versions of software a fair chance, but I'm not prepared to simply do without the familiar programs that I use all the time. To that end, immediately upon installation, I installed Firefox, Pidgin, Chrome, Picasa, OpenOffice, and PyTTY. I was pleased to find that Windows 7 has PowerShell already installed. PowerShell is a much better command line interface than the old DOS-like cmd. I still haven't used the command line much in Windows (at least not at home), and when I do, I'm usually running Cygwin commands or logged in to a Linux box via SSH.

Vista Media Center TV Pack: Pros and Cons

Not too long ago, Microsoft leaked an upgrade to Media Center which they released to OEMs. This upgrade contained some "well duh" features: features that should have been in Media Center all along. It also contained a number of "WTF" featuers: features that were not added in the interest of end users; features designed to please television stations and content producers. Features that annoy me. Nevertheless, I have installed the TV Pack upgrade (codenamed Fiji), and continue to run with it. The good:

• Tuners with different feature sets can be used (and it won't reduce them to the lowest common denominator): analog, digital, HDTV.
• Multiple tuners can be hooked up to different sources (cable, over-the-air).
• Clear QAM support: Media Center now supports HDTV over cable. Unfortunately, I can't seem to be able to get it to recognize my Hauppauge 1250 tuner as a digital tuner when connected to cable, so I can't receive the QAM channels.

The bad:

• DRM: every recording is "protected" (from me accessing it) by DRM. There is no law or FTC policy that requires this to be done.
• WTV container format (replacing the DVR-MS format). This format is currently incompatible with commercial skipping and/or removal programs such as Lifextender or DVRMSToolbox.
• DVD burning of recorded content no longer works. At all.
• If I go back to the pre-TV Pack configuration (which requires re-installing the operating system), I won't be able to access any of the content that I've recorded since the upgrade.
• There is no WTV playing support on any non-[Vista-with-TV Pack] system, or Linux. The format is a complete black box that only Media Center knows how to decipher.

It went like this: I have two tuners: an analog USB tuner (Dell branded, Angel manufactured), and a digital HDTV tuner (Hauppuage 1250). Before the TV pack, I had both tuners hooked up to the cable. The other option was to have just the digital tuner hooked up to the over-the-air antenna, which would then receive HDTV channels, but no cable. Media Center allows you to browse through the next two weeks of movies that will be on TV and pick which ones to record. Cable (aside from having shows like Battlestar Galactica) has orders of magnitude more in terms of movie selection, so without the TV Pack, I would have to choose between quality (HDTV for broadcast, but only getting broadcast channels) and quantity (all the shows and movies on cable, but no HDTV channels). The Hauppauge customer support folks claim that since Microsoft hasn't officially released the TV Pack, and since the OEMs haven't yet released PCs with the TV Pack update yet, they aren't supporting Clear QAM in media center on my hybrid card. Clearly, it's a driver issue, and they haven't released one that deals with the problem, at least not officially. So, for now, I'm waiting: waiting for OEMs to start releasing TV Pack systems, which will in turn (hopefully) lead to driver support by the manufacturers; waiting for the Media Center user community to produce software that will decrypt, de-commercialize, transcode, and otherwise access the WTV format. This is already starting to happen, but more progress needs to be made for there to be a true solution. I don't like having my recorded TV locked in a proprietary DRM-restricted format, especially one that doesn't use much compression (I would prefer to be able to transcode to H.264), but for now, it 'works.'

Shell access to Vista from Linux

Yep, you read that right, I'm going to ssh into a Windows command prompt from Linux box. The first question to answer is, "why on earth would anyone want to do that?" to which my reply is that I wanted to be able to be able to remotely access Elrond, my media box, since it's an always-on machine. I might want to download something directly from it, check what files are there, move them around, etc. My initial thought was to do all of this from Linux, and then just use the default shares (e.g., \\elrond\c$, etc.) to move files, but I ran into trouble accessing default shares from Linux, and it's much more powerful to have command-line access, especially since I've been learning some VBScript lately. Not content with (and not too familiar with) using only the native Windows Shell command-line utilities, I of course installed Cygwin first, and added "c:\cygwin\bin;" to the Path environment variable (put it right after "%systemroot%\system32;"--not before!). Presto! Instant access to all of the Linux utilities and programs that I'm used to. See? This is much better than just Samba/CIFS mounts. The next task was to get an SSH server. My requirements were:

1. Free
2. Runs on Vista as a daemon (I don't have to be logged in).
3. Easy to configure

(Open source would be nice, but it's not a must on Windows.) This article was helpful in rounding up the canidates. My choice was freeSSHd. I tested the server by logging in from my laptop using PuTTY. It worked great, but when I tried to ssh from a Linux box, I got:

$ ssh elrond ssh: elrond: Name or service not known

Oh, yeah, that's right. I use DHCP for my windows boxes, so they're not in the hosts file, and DNS doesn't resolve Windows names not in a domain. Sure, it works if I use the IP address, but using that over time would work havoc on my known_hosts file, and given the nature of DHCP, it's bound to bite me in the butt sooner or later. But Samba can find them, so there must be a way. It took me about a minute (and some bash-completion) to remember the proper command:

$ nmblookup elrond querying elrond on 192.168.5.255 192.168.5.143 elrond<00>

I thought about setting up a script that would parse out the IP address with egrep and then pass it along to the ssh command, but I like a nice clean known_hosts file. There had to be a way to log on to the host by name, so I turned to the Internet. It turns out that the solution is called WINS. Using it requires that "wins" be in the hosts line of nsswitch.conf, like this:

hosts: files dns wins

It also (at least on Gentoo) requires that Samba be compiled with WINBIND support (which is not WINS). This was a rather non-intuitive step, and I have my ignorance to thank for trying WINBIND support in the first place. Okay, at this point I have it all up-and-running!

tim@athena ~ $ ssh elrond tim@elrond's password: Permission denied, please try again. tim@elrond's password: Permission denied, please try again. tim@elrond's password: Received disconnect from 192.168.5.143: 2: Too many attempts.

Wait, what? This worked before! "ssh Tim@elrond" didn't work, and neither did "ssh 192.168.5.142". The server is up and responding. I was using the right password, but something else was wrong. After a little bit of poking around on freeSSHd's website (the forums are, thankfully, rather sparse) I was able to find this, which it turns out is also linked directly from their homepage. Apparently it's a common problem. When running freeSSHd as a service, you have to configure it in "elevated mode"--even if your account is an administrator, that is, you have to explicitly launch the configuration program with elevated privileges, or else you won't be altering the settings for the system, just for the logged-on user. That's why it worked when I had tested it before (my first clue should have been that when I launched the configuration program, I had to stop the service to keep it from saying it was already running. Another problem I ran into was this:

$ ssh elrond Tim@elrond's password: No console window found! Unable to emulate VT100 terminal. Unable to execute command or shell on remote system: Failed to Execute process. Connection to elrond closed.

Basically, I can log in, but it won't give me a terminal. It fails, and kicks me out. The solution was on the forums:

Problem already reported. Disable "use new console engine" until we find a solution.

So I did that, and now it finally works. Update: see comment(s).

Elrond: Blu-ray, Windows Media Center, Etc.

So I bought another computer: Elrond. Elrond will be the living room media PC.

I believe that makes a total of 6 computers. Obviously not all of them are this nice. Some of them are scratch boxes. I'm thinking of installing Ubuntu 8.04 (Hardy Heron) on one of them in the near future. When I have time (HA!)

Elrond is a Dell Inspiron 530 with Blu-ray and an HDTV tuner. It has Windows Vista Home Premium, which comes with Windows Media Center. Media Center This is the first time I've messed with Media Center, and I'm mostly impressed. It was very easy to set up. The only really annoying thing is that if you have two tuners, they can't be hooked up to two different sources: I wanted to have the HD tuner record over-the-air broadcasts (which it can do in HD) and have a second USB tuner (that I had bought with my laptop) handle the SD signal from cable. No dice. It only supports managing one guide, and assumes that all tuners have access to it that guide and nothing else. Lame. Currently, I have it set up with just the HDTV tuner hooked up to cable. It can't record any of the cable HDTV channels, unfortunately, because it doesn't have a QAM tuner (HDTV signals are in a different format on cable than over the air). I think Athena's tuner supports QAM (and it ignores the broadcast flag too), but I haven't set up MythTV yet (which is ironic, since MythTV was one of my primary reasons behind putting Athena together in the first place). The reason I decided to go with SD cable is because it has the most content by far. If I want to record some over-the-air HDTV, I can switch it over to that temporarily. If I get MythTV running on Athena, I should be able to record anything I want. [Edit: According to the manufacturer's page, my tuner can indeed receive QAM HDTV signals, but they apparently have to be clear QAM (i.e., not encrypted or "scrambled"). My TV is able to tune to the HD stations without the cable box, which means it might be simply a matter of configuration: downloading the right listings that tell my computer it can tune to HDTV channels. Whether Comcast provides listings including HD channels is another matter.] [Edit: Apparently, it's not Comcast, it's Microsoft. Windows Media Center doesn't support QAM, unless you buy a special computer designed specifically for it. This seems stupid to me, because these channels are being sent over the wire in the clear. Back to square one. Time to get MythTV up and running.] [Edit: There appear to be ways around WMC's artificial restriction (by getting tuner hardware and drivers that lie about where the video comes from). Interesting. I'm still going to explore my MythTV solution. Currently, I have MythTV up and running, but nothing shows up on a channel scan.] Blu-ray So, I've finally bought myself a Blu-ray drive. I was hoping to wait until full-featured stand-alone players went down in price to the $200 mark. It doesn't look like this is going to happen anytime soon. Blu-ray players actually went up in price after the end of the format war with HD DVD (surprise, surprise), and they don't show any signs of going down any time soon. The best "value" on a Blu-ray player was still the PS3, and I'm not that interested in the PS3 for gaming (and I already have as many Linux boxes than I need) so that wasn't looking like a good deal either. I was looking at the Dell website for a PC for the living room, and I saw that they had a special deal on a Blu-ray player, in addition to a big chunk of money off the computer, and my employee discount program percentage off, so I bit. I was hoping that I could have the option of hooking up my HD DVD drive (Xbox 360 attachment) to the PC and have it play both, but it only came with PowerDVD DX version 7, which supports Blu-ray and not HD DVD. Not a big deal, of course, since I have a stand-alone player (HD-A3). The only problem I ran into as far as Blu-ray playback is concerned is that I wanted to output Surround Sound to my receiver digitally. For this, I had to buy a sound card, and since the two digital optical input ports were already occupied (by the HD DVD player and the TV), I needed a coaxial digital S/PDIF out. I found one at Fry's (Diamond XtremeSound 7.1 for $30 with a $10 rebate), hooked it up, and told Windows to use that card's digital out. In order to get it to output Dolby Digital 5.1 during movie playback, I had to configure the settings of the PowerDVD program to output to "SPDIF"; before that, it was still sending stereo.

My FreeBSD Adventure With Thebeast

Warning: not for the faint of heart. This is a pretty long chronicle, full of technical jargon, but it might prove useful/entertaining to Linux/BSD users.

A while ago I bought a computer from the surplus store. There were a couple of things that were noticeable about this computer: the first was that when I turned it on, the hard drives inside were exceptionally loud, and the second was its identification:

Behold, the mark of thebeast!

The hard drive noise turned out to be an old SCSI hard drive that had been hooked up to the power supply, but as there is no SCSI support, not to the motherboard. I took it out, obviously, but because of its roar, and the markings on the case, I decided that this computer's name would be known on the network as thebeast (not to be eschatologically confused with the Antichrist).

Another beast-like quality of this computer is its chimera-like nature: there are four different operating systems installed in a quad-boot configuration. An interesting thing about the hard drive that came with this computer is that it's a 20GB hard drive, but Windows thinks it's a 7.8GB hard drive. There apparently are some contradictory drive geometry settings programmed onto the drive. I suspect that this was done in order to keep the same standard part number and specs, while upgrading to newer, more often-produced hardware: it's been a while since they've made 8GB desktop hard drives in quantity.

This of course means that if ever Windows tries to install on the drive, it totally screws up the partition table. I initially tried to install it on a 4 GB partition at the beginning of the disk, but it insisted on writing its own. "Fine," I said, and I let it write its own table. This worked, but as soon as I tried to partition the rest of the disk on the same table, Windows wouldn't boot anymore. What I ended up doing was taking a 3GB hard drive that I had around and installing Windows 2000 Pro on that. Of course, to do this, I had to set it as the primary drive, install it, and then switch it back to the secondary drive (so that I could use Grub, a boot loader, to pick an operating system at boot time). Windows doesn't play nice with other operating systems. It thinks it owns the disk. To get it to boot off of the second disk, you even have to trick it into thinking that its disk is really first. Here's my grub.conf file:

default 0
fallback 1
timeout 15
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title Gentoo Linux 2.6.20-r8
root (hd0,0)
kernel /boot/gentoo/kernel-linux-2.6.20-gentoo-r6 root=/dev/hda6 video=i810fb:accel
savedefault

title Gentoo Linux 2.6.20-r8 (rescue)
root (hd0,0)
kernel /boot/gentoo/kernel-linux-2.6.20-gentoo-r8 root=/dev/hda6 init=/bin/bb

title Gentoo/FreeBSD 6.2
root (hd0,1,d)
kernel /boot/loader
savedefault

title FreeBSD 6.2
root (hd0,2,d)
kernel /boot/loader
savedefault

title Windows 2000 Professional
# Make Windows think its disk is first
map (hd1) (hd0)
map (hd0) (hd1)
rootnoverify (hd1,0)
makeactive
chainloader +1
savedefault

As you may be able to tell, when the system boots from the BIOS, it gets a menu screen with the following choices:

• Gentoo Linux 2.6.20-r8
• Gentoo Linux 2.6.20-r8 (rescue)
• Gentoo/FreeBSD 6.2
• FreeBSD 6.2
• Windows 2000 Professional

Whichever option is chosen is the operating system that will boot. Each of the operating systems sits on its own partition, but they can all see each other's files, with the exception of Windows, which can't properly read the partition table/disk geometry of the first hard drive. If not for that limitation, I would be able to see the Ext2 and Ext3 partitions (in this case Linux's /boot and /home) after installing a driver in Windows. I typically use ReiserFS for the other Linux partitions (/, /usr, and /var), for which there might be a read-only Windows driver, and FreeBSD uses the UFS file system, which as far as I know has no Windows driver.

Getting FreeBSD and Linux to see Windows' NTFS partition was pretty easy. In Linux the line in /etc/fstab looks like this:

/dev/hdb1       /mnt/win2k      ntfs            ro,nosuid,nls=utf8      0 0 # ~3.0GiB

and in FreeBSD it looks like this:

/dev/ad1s1      /mnt/win2k      ntfs            ro                      0 0 # ~3.0GiB

They're essentially the same. Linux uses "hd" (for an IDE hard drive), "b" meaning the second drive (the first drive is "hda"), and "1" denoting the first (and in this case, only) partition on that drive. FreeBSD identifies devices by the name of the driver used to access them in stead of their function as in Linux. "ad" refers to the disk driver, "1" refers to the second disk (ad0 is the first disk), "s" is for 'slice' which is the BSD word for a what Linux and Windows call primary partitions, and "1" refers to the first slice. This is, by the way, distinct from the way that Grub refers to hard drives. Grub calls your first hard drive (hd0), and its first partition is (hd0,0). If that partition has sub-partitions, as BSD or Solaris UFS slices are wont to have, then your root slice on a second partition on a third disk would be: (hd2,1,a).

The first partition of the first disk:
Linux: /dev/hda1
Grub: (hd0,0)
BSD: /dev/ad0s0

The first sub-partition in the first slice of the first disk:
Grub: (hd0,0,a)
BSD: /dev/ad0s0a

Simple, right? It took a fair amount of searching and sifting to find all that out.

Getting FreeBSD to recognize Linux partitions is as easy as identifying the partitions and filesystem:

# Gentoo Linux
/dev/ad0s6      /mnt/linux      reiserfs        rw,noatime,nodev        0 0
/dev/ad0s1      /mnt/linux/boot ext2fs          rw,noatime,nodev        0 0
/dev/ad0s7      /mnt/linux/usr  reiserfs        ro,noatime,nodev        0 0
/dev/ad0s8      /mnt/linux/var  reiserfs        ro,noatime,nodev        0 0
/dev/ad0s9      /mnt/linux/home ext3            rw,noatime,nodev        0 0

Getting Linux to see my Gentoo/FreeBSD partition was equally straight-forward, since there were no sub-partitions, though I did have to specify the ufstype attribute:

/dev/hda2       /mnt/gfbsd      ufs             ro,ufstype=ufs2,nosuid,nodev    0 0 # 3586MiB

Getting Linux to see UFS sub-partitions was a bit trickier. Doing an "fdisk -l" only showed the main slice (/dev/hda3), not the sub-partitions, but entering "cat /proc/partitions" revealed the device names of the missing sub-partitions. (I suppose ls /dev/hd*) would have revealed the names, too). It was a bit tricky to figure out which device went with which. The extended partition with the Linux partitions on it began at hda5, and where those left off was where the UFS sub-partitions began, but they weren't in the order in which they were created. UFS apparently assigned the first sub-partition I had created within this slice (which was a swap partition, shared with the Gentoo/FreeBSD install) as ad0s3b, reserving the "a" slot for my / partition, and making the /var and /usr partitions "d" and "e" respectively. Therefore, this is how I mapped my drives from Linux:

# First Hard Drive
/dev/hda1       /boot           ext2            noauto,noatime                  1 2 #   16MiB
/dev/hda2       /mnt/gfbsd      ufs             ro,ufstype=ufs2,nosuid,nodev    0 0 # 3586MiB
#/dev/hda3 is the freebsd slice: hda10,11,12,13                                     # 7680MiB
#/dev/hda4 is the extended partition: hda5,6,7,8,9
/dev/hda5       none            swap            sw                              0 0 # 1024MiB
/dev/hda6       /               reiserfs        noatime,user_xattr              0 1 #  256MiB
/dev/hda7       /usr            reiserfs        noatime,user_xattr              0 1 # 3586MiB
/dev/hda8       /var            reiserfs        noatime,user_xattr              0 1 #  256MiB
/dev/hda9       /home           ext3            noatime,user_xattr              0 1 #  ~3.8GB
/dev/hda10      /mnt/freebsd    ufs             ro,ufstype=ufs2,nosuid,nodev    0 0 # 7680MiB
#/dev/hda11     /mnt/freebsd    is the FreeBSD swap partition                       #  512MiB
/dev/hda12      /mnt/freebsd/var ufs            ro,ufstype=ufs2,nosuid,nodev    0 0 # 1024MiB
/dev/hda13      /mnt/freebsd/usr ufs            ro,ufstype=ufs2,nosuid,nodev    0 0 #  ~5.3GB

# Second Hard Drive
/dev/hdb1       /mnt/win2k      ntfs            ro,nosuid,nls=utf8              0 0 #  ~3.0GB

After I had Windows and Linux up and running (Gentoo is my favorite flavor of Linux, so that's what I used), I set my sights on FreeBSD. This was done for a number of reasons:

1. With a chimera machine called thebeast, one must, must have a BSD install. I mean, have you seen their 'Beastie the BSD daemon' mascot? BSD is the essence of geeky hotness, and with that "E280666" sticker on the box, who could resist? It was a sign.
   
2. Operating systems are a hobby of mine, and I've never worked with a BSD before. Caleb had mentioned that I should look at BSD, because Mac OS X is based on it.
   
3. I hear Gentoo is a very BSD-like Linux, so it wouldn't be totally foreign, and there's even a project to get Gentoo's Portage on BSD.
   

I had once before tried to install FreeBSD on this computer, but I kept running into problems during the install. Once I finally had it installed, I had no idea how to get X working, let alone any actual user software. This time, I decided I would try the Gentoo/FreeBSD approach. I chose FreeBSD over NetBSD and OpenBSD because there seems to be a better community and support around FreeBSD. Along with that, (and probably having to do with that) it was the most supported (read 'least experimental') in the Gentoo-Alt *BSD project, and that was my stepping stone of choice from Linux.

Installing Gentoo/FreeBSD was as simple as following the provided online instructions. They were written by Gentoo/Linux developers who mess with FreeBSD, not FreeBSD developers, which means I could follow them more naturally: they basically gloss over the similarities, while slowing down to explain the differences.

I went with the LiveCD option for installing, since I had no existing installation: I downloaded and burned a FreeSBIE CD, and followed the guide, deviating where necessary. My biggest BSD-related installation issue was getting the right Ethernet driver configured in /etc/conf.d/net. In Linux, you don't care what the driver is called, you just configure it as net.eth0, but in BSD, I had to know that I was using the xl driver, so my init script was called net.xl0.

Also, I ran into trouble with the /etc/conf.d/net config file. In Linux, I had this:

dhcpcd_eth0="-t 15 -N"
dhcp_eth0=( "nontp nonis" )
config_eth0=( "192.168.5.16/24 brd 192.168.5.255" )
routes_eth0=( "default gateway 192.168.5.1" )
fallback_eth0=( "dhcp" )

but in FreeBSD, I needed to remove the parentheses, and I also discovered that I should use "via" in stead of "gw" or "gateway" to specify the default route to the Internet.

dhcpcd_xl0="-t 15 -N"
dhcp_xl0=" 'nontp nonis' "
config_xl0=" '192.168.5.16/24 broadcast 192.168.5.255' "
routes_xl0=" 'default via 192.168.5.1' "
fallback_xl0=" 'dhcp' "

One thing that was nice about doing Gentoo/FreeBSD before the actual FreeBSD install was that in Gentoo/FreeBSD, the init script layout is the same, and you install all of your userland applications through portage, which with I'm very familiar, and so I was tweaking use flags, configuring daemons and services, and using package.keywords without so much as batting an eye.

Configuring and compiling the kernel was relatively straight-forward, FreeBSD's handbook is an excellent resource, and I used it extensively later on in the actual installation.

When I got to /etc/fstab, I was accustomed to having the /tmp directory (as well as some others) on its own virtual memory disk, so I looked up how to do that. Here's the line from /etc/fstab:
In Linux:

tmpfs   /tmp            tmpfs   size=512M,nr_inodes=1M,mode=1777,noexec 0 0

In BSD:

md      /tmp            mfs     rw,-s100m               0 0

Unfortunatley, the Gentoo/FreeBSD project doesn't enable you to easily install just anything. I was able to install X with gnome-light, and firefox, as well as a few other things, but the majority of ebuilds in the portage tree aren't ready for installation on FreeBSD out of the box. To get a real FreeBSD installation, I was going to have to do it the right way. One program that I annoyingly could not install was fortune-mod. An earlier installation of Gentoo/FreeBSD had successfully installed with it, but then the ebuild had been updated to a new version (and all older versions removed), and I was left without a viable fortune program. A tragedy, I know. Anyway, onward.

Installing FreeBSD
I burned the FreeBSD CD ISOs that I had downloaded via Bittorrent, pulled up the handbook, and dove in. This time I was able to follow the handbook much better than the last time I had done the installation, and thanks to the copious amount of poking around the system I had done during my Gentoo/FreeBSD install, I was up and running in no time. I customized the kernel to exclude hardware I didn't have, and also to build my sound driver. To build it, I added the following to my kernel config file:

# Sound card support customized to this machine
device          sound
device          snd_ich

and the following line to the /boot/loader.conf file to have the module automatically loaded:

snd_ich_load="YES"              # Intel ICH

Then I ran into a bit of a problem. In my previous installations, my mouse had worked just fine. I had been successfully using a 2-port Iogear KVM switch between a small Linux box (minix) and thebeast, but I had since switched to using a 4-port LinksKey KVM (with audio support) between four different boxes, and the mouse wouldn't work in BSD. It worked in Windows and Linux just fine, but in FreeBSD, no matter what kernel I used, and even if I used the FreeSBIE LiveCD, the mouse pointer would be there on the screen in X, but it wouldn't move or click. What was more, I discovered that the mouse device was absent from /dev. It was quite frustrating to finally have a fancy graphical environment up and running on BSD, and not be able to use it, except by keyboard shortcuts and tabbing.

I searched the forums online to see if others had had similar problems with the FreeBSD psm driver. I found quite a few people who had issues with touchpads, and a few who had trouble with KVMs. I found solutions to the touchpad issues by adjusting the flags passed into the driver from the kernel configuration device.hints file, but those flags were specific to touchpad initialization and waking up from a laptop's suspended state. No one that I could find had fixed the KVM issue in a way that was helpful for me. What worked for most people was recompiling the kernel with

options       KBD_RESETDELAY=200
options       KBD_MAXWAIT=5

adjusted to higher values, but this didn't help me. What I found to be the most help was the psm man page and the dmesg output. Adjusting the KBD_RESETDELAY didn't help at all, even at ridiculously high values, because I found that the mouse was failing to reset, or at least not properly responding, no matter how long the driver waited for a response. Increasing the logging threshold using the kernel following kernel config options:

options       PSM_DEBUG=2
options       KBDIO_DEBUG=2

...yielded the following in dmesg:

psm0: current command byte:0065
psm0: failed to reset the aux device.

This combined with the psm man page led me to a flag in /boot/device.hints that disables the initialization attempt at boot time:

hint.psm.0.flags="0x400"

This gave me the following in dmesg (and /var/log/messages) with PSM_DEBUG still at 2:

psm0: current command byte:0065
psm0: found IntelliMouse Explorer
psm0:  flags 0x400 irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4-00, 5 buttons
psm0: config:00000400, flags:00000008, packet size:4
psm0: syncmask:08, syncbits:00

...but I still had no mouse movement in X, whereas before, I had been able to use this mouse just fine. A few hardware options presented themselves: I could use a serial mouse that I have stuffed in a drawer in my room; I could use an adapter to make the PS/2 keyboard and mouse plug into a USB port in stead, and thus use a different driver, or at least a different initialization routine; I could get an extra mouse, or a different mouse, but that defeats the purpose. This should work. It occurred to me finally to specify the mouse device directly in xorg.conf; in stead of /dev/sysmouse, to use /dev/psm0:

Section "InputDevice"
     Identifier  "Mouse0"
     Driver      "mouse"
     Option      "Protocol" "auto"
#       Option      "Device" "/dev/sysmouse"
     Option      "Device" "/dev/psm0"
     Option      "ZAxisMapping" "4 5 6 7"
     Option      "Buttons" "6"
EndSection

And it worked. Yay!

I installed ports (FreeBSD's package management system), and thus far I've only installed packages using the "pkg_add -r name" method, although I do hear you can install directly from source if you so desire. In my experience, it's been pretty slick with just binary packages, and I'm content for now. I'm not nearly as adept at ports' intricacies as I am with Gentoo's portage.

Here's what I ended up doing with the graphical interfaces:

• Since Gentoo/FreeBSD doesn't work with KDE yet, and Gnome is minimal, I went with Xfce4, which is an excellent desktop environment (although I'm using GDM as the login manager: Gentoo has an excellent gdm-themes package, which I unmasked with the x86 keyword in order to install). I'm also using Xfce4 on a 200MHz Pentium-MMX machine that I'm tinkering with (it's called crunchy, and I paid a whopping $5 for it). The Gentoo/FreeBSD partition is also relatively small, so I don't think I'm ever going to install a full Gnome on it, and certainly not KDE.
  
• My usual desktop environment for Linux has been Gnome. I also use KDE occasionally, but I prefer Gnome, so in order to be consistent, the Linux install uses GDM and Gnome by default.
  
• I installed GDM, Gnome, Fluxbox, Xfce, and some others, but I decided that my FreeBSD system would use KDM for graphical login and KDE as the default desktop.