One of the biggest things I regret is not utilizing Xen more. I’ve finally been admitted to Amazon’s EC2 Limited Beta, just two days before I leave, so not enough time to actually do anything fun. However, I think Xen is an ideal infrastructure aid for SMEs in particular. The cost of technology is continuing to decrease, which means bigger servers cost less. This is great for the small/branch office. Let me explain.

One of the themes I noticed while studying and taking the MCSE was that the solution to the majority of the problems was to just buy more servers. Even for simple things like DHCP, buy another server. I’ve always operated on a limited budget, and anyways, I don’t think money should be wasted on resources when it isn’t needed. With a VT chipset, you aren’t tied to any OS in particular.

My friend Ian and I were talking and he illustrated a great usage of Xen through his work. What he’s ended up doing is installing the Small Business Edition of Server 2003 in a Xen node. The reasoning is that SBE is, apparently extremely difficult to create backups of – mainly due to odd file locking behavior. I’ve had similar thoughts, but mainly taking advantage of Xen’s migration feature. The idea of taking a small branch office and putting everything on a Xen server is quite appealing to me, especially considering a second server could be used to create virtual hot spare.

As you can see, I like Xen. I’ve found it relatively easy to install, and the fact that it is starting to come bundled with recent distributions is pretty darn, sweet.

This will be my last post from Taiwan, and I’m placing it in my tech section. Shortly I will be flying to Hong Kong, and then traveling into China. I’m not bringing my laptop with me. I’m always a little wary of using public computers, especially in many of the poorly run internet cafes. Often the logged in user is the administrator, and we all know the computers are obviously crawling with worms and keyloggers. What can I do?

I had a 128MB flash drive, which is the perfect size fo this. I’ve installed the following applications to run directly from it:

• Portable Clamwin - I plan to fire it up and do a memory scan before I start typing any passwords.
• Portable Putty - This is useful for two reasons. Firstly, in case my server (or any server with SSH) needs help, I’m on it. Secondly, and more importantly, for security. Putty can easily be used as a SOCKS5 proxy over SSH, so I can tunnel Firefox and IM securely. Password sniffers, be gone! A side benefit is the ability to bypass the “Great Firewall”, if needed (e.g. the block my Google account).
• Portable Miranda - In case I’m feeling home sick, or have some crazy desire to talk on IRC. Don’t count on it.
• Firefox - I tried the Portable Apps package, yet it didn’t work.

I noticed in the “known issues” that it doesn’t work if loaded on a drive with a non-asci path, which this machine (and those in China) usually have. The “resolution” is to run it in Win98 compat mode, but this didn’t work for me. To get around this, I downloaded the normal Firefox, installed it, copied the contents of “Mozilla Firefox” and dumped it in /Firefox. Then I created a profile directory called /FFProfile, and created a bat file called “firefox.bat”:

start \Firefox\firefox.exe -profile \FFProfile

Double click the bat file, and you have FF running on your usb drive.
I’m in a search for a better keylogger detector, as I don’t know how complete ClamAV will be. If you know of one, let me know. Until then, I’m going to pretend I have the perfect traveling USB companion.

Two weeks ago I had the privilege of meeting up with some of the OpenMoko guys. You’ve seen me write a little bit about this project, but meeting them and actually getting to play with a device was quite enlightening.

Most foreigners in Taipei take taxis, but not I. I already had an idea how to go to FIC headquarters, yet it was – rough. I jumped on the right bus, going the right direction, yet didn’t really know where to get off. In my broken Chinese I started talking to a group of high school kids, who somewhat pointed me in the right direction. At one point we were coming up to a light and a lady just said: “get off, go down that road.” I grabbed my bag and jumped off the bus. The street signs started to resemble the characters I had written down, but I didn’t know which direction to go in. I asked a travel agency, and they pointed me in the right direction – and then it started to rain. I luckily found FIC without getting too wet, and quickly met Sean, the project leader for OpenMoko.

Sean, myself, and several other OpenMoko developers went out to lunch and talked shop. Mickey pulled out a Neo, the hardware platform OpenMoko is going to be based off of. I’ll admit, when I saw photos I wasn’t really impressed – but in person, it is pretty sexy. Not too big, but big enough to be functional. In one of my previous posts I mentioned learning C++ back in university, and having basically not touched it since. In a way, I haven’t had to, and I’ve recently found Python, which is just fun. However, I’m so excited about the OpenMoko, I’ve picked up the K & R bible to read while we are traveling. Not the ideal way to learn a language, but the concepts are somewhat a refresher. I can’t wait to play with SIP/IAX on this thing, it will then be my dream phone.

Well, I have one more post to make to get some thoughts off my chest. If you want a postcard, send me an email.

You can easily see that I am a big Linux fan, I’ve mentioned this over and over. I’ve also mentioned that I’m finishing up my MCSE, which started as a dare with my buddy, Ian. In the process of studying for my MCSE I’ve developed a certain amount of respect for Windows Server 2003, yet I’ll comment more on this later.

What I really wanted to write two paragraphs about was an interesting discovery I made today. I happened to stumble over to Microsoft’s hardware site, and noticed in my little Server Spy monitor that it was running IIS/7 – the first time I have seen this anywhere. Upon further investigation I perused their GoLive program, which details several ways to test drive IIS7. There are some details about Microsoft’s setup, which one would expect to be impressive. And it is. So, if you are a Microsoft person, you may want to grab a test drive of Beta3 while you still can.

The buzz of Computex is finally wearing off, so I’ll finally scribe my account of the event. YS was kind enough to go with me, we actually called it a date. The scene was quite similar to the trade shows we saw last year, however there seemed to be fewer people. This might be because we went on the final day, although we suspect there weren’t that many people because there aren’t “sweet” deals to be had. Regardless, there were several main “halls” filled full of stuff. The first three (two were in one building) weren’t particularly interesting. I mean, there were some cool displays and lots of technology, but nothing that left me thinking “oh, that’s cool” or “oh, me wanna.” Until we hit the last hall.

The first place we stumbled upon was the SunComm booth, which seemed to have more “traffic” than any of the other VoIP booths. Let me say this: I think VoIP stuff is awesome, that it is going to be the future, and I dig looking at demos. Even though the SunComm website pretty much looks quick ugly, their products are way cool. I was lucky enough to purchase a phone directly from them, and have it mailed directly to my house in Taipei (which was just around the corner from their office). They don’t normally do this.

Next up was one of the main reasons I even wanted to go to Computex: to see the OpenMoko phone. To be honest with you, if I had a million dollars, this would be one of the first things I would buy. The other would be a new ThinkPad X61s. So why is this phone so cool? First off, it is based on Linux, which if you haven’t gathered, I like. The entire stack is open, or nearly open. I’ve always purchased pretty average phones, or been given them for free. The reason I do this is because my requirements have never been met: I want a SIP/IAX client on a wifi tri (or preferably quad) band phone. GPS is a plus, but not necessarily required. So far, I haven’t been able to fine anything that has suited my needs – the HTC Universal almost would work, apparently. The slogan is “Free Your Phone”, but I think of this project more to “make your phone never become obsolete.” Indeed, the hardware may become obsolete, but the software never will. I keep being reminded that this project could really put a dent in the future iPhone sales. I can’t wait to get one with WiFi and start hacking on it – in the meantime, I’ll need to refresh my C.

Up next, directly across the aisle, was Via’s new line of motherboards. At Computex there were quite a few small sets of motherboards, but nothing quite like this. Smaller than mini-itx. Smaller than nano-itx. We have pico.

This thing is freaking small. Seeing this little board made me realize my desire to minimize how much (and the size) of the technology I own. Currently I’m selling my AMD Athlong64 3000 (it has two gigs of RAM), and will at some point attempt to switch to a single laptop solution (with Xen, of course). At the same time, I’m considering how useful one of these little boards could be. Many have two NICs, which would make perfect routers. I’ve seen some boards for as inexpensive as $65usd.

Today on Linux.com, Liferea was mentioned I find this particularly because I’ve been using Liferea for quite a few months, and haven’t been able to find anything that better suits my needs. It sits quietly in the corner until a new RSS is available, and slightly changes color. Plus, it integrates wonderfully into GNOME.

Well, the Linux.com article does a great job summarizing the reader, so if you are in the market, I’d take a look. For posterities sake, I took a screenshot of the Linux.com feed of Liferea, in Liferea.

As a rule, whenever I’m online I’m logged into my server back in the States. I’m also usually wireless, which we all know is beyond insecure – I’ve found it especially useful to tunnel firefox over SSH. I try my best to tunnel stuff over SSH back, and if you want to also, this is how.

Setup the SSH/SOCKS tunnel

I’m on Linux, so this is pretty darn easy.

ssh [email protected] -D 1080

If the SSH daemon runs on a different port, you’d do something like this:

ssh -oPort=1234 [email protected] -D 1080

Remember ports below 1024 are reserved, and you would need root access. Now it is time to configure the different programs to use the newly created tunnel.

Setting up Gnome (optional)

Tunneling Pidgen

Tunneling XChat

Tunneling Firefox
Note: I’m going to list two examples, one is with FoxyProxy and the other is with the ordinary proxy settings.

FoxyProxy

Normal Proxy

Make sure the other fields or empty, or you won’t connect.

So, there you have it. There are quite a few unix shell providers out there, I’m sure it wouldn’t be too hard to spot a link for one. I’ve seen QuadSpeedInternet having SSH access for $3/month, and JVDS or Lonestar offering possible free shells. Alternatively, you could just get a really inexpensive VPS at VPSLink ($6-$8/month, but they often have 25% off discounts).

I generally like Firefox, and generally like Google. But having Google suggest enabled when I search for things is annoying, at least on my rather small laptop screen. Mostly it is because my internet right now is pretty average, so I try to cut down on any extra traffic. Anyhow, I didn’t even both Googling how to do this:

Type in ‘about:config’ into your address bar and type ‘search’ in the top filter bar. Look about half-way down for this:

browser.search.suggest.enabled

Double click it to turn it to false. All done.

So, you’ve seen how to look up thumbnails via python, but wonder how to integrate this with Django? I created a sample app to demonstrate. One thing to note about this app is it is slightly more complex than just using the previously mentioned ThumbnailUtility. For starters, the thumbnail is downloaded from Alexa onto the server. Another part is first searching if the thumbnail exists already, and if it does, serving that instead of querying Alexa. Let’s just start with some code.

getAST.py

  
  
#!/usr/bin/python
import base64
import datetime
import hmac
import sha
import sys
import re
import urllib
import xml.dom.minidom
import os
from urlparse import urlsplit

AWS_ACCESS_KEY_ID = 'your-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-secret-key'
STORELOC = "/path/to/store/webthumbs/"

def create_thumbnail(site_url):
    image_name = re.sub("\.|\/", "_", '.'.join(urlsplit(site_url)[1].rsplit('.', 2)[-2:])) + ".jpg"
    if not os.path.isfile(STORELOC+image_name):
        def generate_timestamp(dtime):
            return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
        def generate_signature(operation, timestamp, secret_access_key):
            my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
            my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
            return my_b64_hmac_digest
        timestamp_datetime = datetime.datetime.utcnow()
        timestamp_list = list(timestamp_datetime.timetuple())
        timestamp_list[6] = 0
        timestamp_tuple = tuple(timestamp_list)
        timestamp = generate_timestamp(timestamp_datetime)
        signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
        parameters = {
            'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
            'Timestamp': timestamp,
            'Signature': signature,
            'Url': site_url,
            'Action': 'Thumbnail',
            }
        url = 'http://ast.amazonaws.com/?'
        result_xmlstr = urllib.urlopen(url, urllib.urlencode(parameters)).read()
        result_xml = xml.dom.minidom.parseString(result_xmlstr)
        image_urls = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[0].firstChild.data
        store_name = STORELOC + image_name
        urllib.urlretrieve(image_urls, store_name)
    return image_name
      
  
      

Not too much to mention here, basically just an extended version of the ThumbnailUtility. The only difference is the test at the beginning, and actually downloading the thumbnail.

views.py

  
# Create your views here.
from django.shortcuts import render_to_response, get_object_or_404
from django.http import HttpResponseRedirect, HttpResponse
from webthumbs.models import *
from django import newforms as forms
from getAST import create_thumbnail

attrs_dict = { 'class': 'form-highlight' }

class imageForm(forms.Form):
    url = forms.URLField(max_length=100, verify_exists=True, widget=forms.TextInput(attrs=attrs_dict), initial='http://', label='Site URL')
    
def index(request):
    disp_img = ''
    # generate default form
    f = imageForm()
    # handle add events
    if request.method == 'POST':
        if request.POST['submit_action'] == 'Submit':
            # attempt to do add
            add_f = imageForm(request.POST)
            if add_f.is_valid():
                site_url = request.POST['url']
                disp_img = create_thumbnail(site_url)
        else:
            f = add_f
    return render_to_response(
        'webthumbs/index.html', 
        {'printform': f, 
        'disp_img': disp_img
        }
    )
  
  

The key thing to look at here is how getAST is called:

  
  
site_url = request.POST['url']  
disp_img = create_thumbnail(site_url)  
  
  

index.html

  
{% extends "base.html" %}  
  
{% block title %}Kelvinism.com - Blog{% endblock %}  
  
{% block content %}  

  
What thumbnail do you want?  
  

  
    {{ printform }}  
  
  
  
{% endblock %}  
  

So, there you have it, the code to take a url via form and display it right away.

For one of my sites I needed to get thumbnails, yet Alexa Site Thumbnail didn’t have any code snippets for Python. Well, no they/you do.

ThumbnailUtility.py

import base64
import datetime
import hmac
import sha
import sys
import re
import urllib
import xml.dom.minidom

AWS_ACCESS_KEY_ID = 'your-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-super-secret-key'

# This one is for an individual thumbnail...
def create_thumbnail(site_url, img_size):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
    parameters = {
        'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
        'Timestamp': timestamp,
        'Signature': signature,
        'Url': site_url,
        'Action': 'Thumbnail',
        'Size': img_size,
        }
    url = 'http://ast.amazonaws.com/?'
    result_xmlstr = urllib.urlopen(url, urllib.urlencode(parameters)).read()
    result_xml = xml.dom.minidom.parseString(result_xmlstr)
    image_url = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[0].firstChild.data
    return image_url
  
# And this one is for a list
def create_thumbnail_list(all_sites, img_size):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    
    signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
    
    image_loc = {}
    image_num = []
    image_size = {}
    
    count = 1   
    for s in all_sites:
        image_num = 'Thumbnail.%s.Url' % count
        image_loc[image_num] = s
        count += 1
        
    parameters = {
        'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
        'Timestamp': timestamp,
        'Signature': signature,
        'Action': 'Thumbnail',
        'Thumbnail.Shared.Size': img_size,
        }
        
    parameters.update(image_loc)
    
    ast_url = 'http://ast.amazonaws.com/?'
        
    result_xmlstr = urllib.urlopen(ast_url, urllib.urlencode(parameters)).read()
    result_xml = xml.dom.minidom.parseString(result_xmlstr)
    
    image_urls = []
    count = 0
    for s in all_sites:
        image_urls.append(result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[count].firstChild.data)
        count += 1
    return image_urls

This is how you interact with this code for a single thumbnail:

>>> from ThumbnailUtility import *
>>> create_thumbnail('kelvinism.com', 'Large')
u'http://s3-external-1.amazonaws.com/alexa-thumbnails/A46FF6A30BECB0730455F2AB306EDC28605BC19Cl?Signature=XpsxgPey4b0JgreZA46XnvHVVLo%3D&Expires=1181110547&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2'

And for a list:

>>> from ThumbnailUtility import *
>>> all_sites = ['kelvinism.com', 'alexa.com', 'vpslink.com']
>>> create_thumbnail_list(all_sites, 'Small')
[u'http://s3-external-1.amazonaws.com/alexa-thumbnails/A46FF6A30BECB0730455F2AB306EDC28605BC19Cs?Signature=%2BfcOUKwH4xD9IH9o1vfto%2FMoALU%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2', u'http://s3-external-1.amazonaws.com/alexa-thumbnails/D798D8CE8F821FCC63159C92C85B70319E44D0EFs?Signature=6jriChrGM%2F8DoejN9dn9Dv3Lc5w%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2', u'http://s3-external-1.amazonaws.com/alexa-thumbnails/23529C34E0518AA9C2577653AC237D3647BA8D2Ds?Signature=5ksuwZx0I5TqXWL3Kt%2BWP6r2LQk%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2']

This is just a simple example to get your feet wet, maybe you’ll find it useful. If you are wondering how to integrate this with Django, don’t worry, I’ve got you covered.