Back in the day the barrier to entry for the Internet was quite high. The technology used required a steep learning curve, the equipment extremely expensive, and sometimes even hard to acquire. Fast forward to 2007 and things have certainly changed. If you know any tech people you can likely get free hosting for a small website, and even more demanding websites can be hosted for not much. The cost of dedicated servers has dropped even more. And the final kicker: web services. I’ve started to think of some web services not as a service, but more like outsourcing requirements.

This very dependency adds risk for a multitude of reasons, and when your entire web application platform revolves around a third party, such as is the case with mashups, you incur great risk.

One of the nice things when requirements are outsourced is the fact that risk is mitigated. I’ll use SmugMug as an example. In summary, they moved their storage to Amason’s S3 network, which is something I will be utilizing as well. Amazon’s S3 (and other web services) continue to drive down the barrier of entry – now you don’t even need to purchase hugely expensive servers for the sole purchase of storage! If you don’t need to purchase them, you also don’t need to manage them. Risk mitigated.

However, continuing the slight allusion from The Other Blog’s article on mashups, I see a slight problem with the outsourcing of requirements. While the following thought isn’t particularly innovative: mitigating risk and outsourcing requirements creates a dependency on the third-party. This very dependency adds risk for a multitude of reasons, and when your entire web application platform revolves around a third party, such as is the case with mashups, you incur great risk.

But, as is evident by the fact that I’ve had stitches nine different times, I’m still going to do some cool mashups anyways, so stay tuned.

For one of my projects I need to generate thumbnails for a page. And lots and lots and lots of them. Even though I can generate them via a python script and a very light “gtk browser”, I would prefer to mitigate the server load. To do this I’ve decided to tap into the Alexa Thumbnail Service. They allow two methods: REST and SOAP. After several hours of testing things out, I’ve decided to toss in the towel and settle on REST. If you can spot the error with my SOAP setup, I owe you a beer.
I’m using the ZSI module for python.

1. wsdl2py

I pull in the needed classes by using wsdl2py.

wsdl2py -b http://ast.amazonaws.com/doc/2006-05-15/AlexaSiteThumbnail.wsdl

2. Look at the code generated.

See AlexaSiteThumbnail_types.py and AlexaSiteThumbnail_client.py.

3. Write python code to access AST over SOAP.

#!/usr/bin/env python
import sys
import datetime
import hmac
import sha
import base64
from AlexaSiteThumbnail_client import *

print 'Starting...'

AWS_ACCESS_KEY_ID = 'super-duper-access-key'
AWS_SECRET_ACCESS_KEY = 'super-secret-key'

print 'Generating signature...'

def generate_timestamp(dtime):
    return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")

def generate_signature(operation, timestamp, secret_access_key):
    my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
    my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
    return my_b64_hmac_digest

timestamp_datetime = datetime.datetime.utcnow()
timestamp_list = list(timestamp_datetime.timetuple())
timestamp_list[6] = 0
timestamp_tuple = tuple(timestamp_list)
timestamp_str = generate_timestamp(timestamp_datetime)

signature = generate_signature('Thumbnail', timestamp_str, AWS_SECRET_ACCESS_KEY)

print 'Initializing Locator...'

locator = AlexaSiteThumbnailLocator()
port = locator.getAlexaSiteThumbnailPort(tracefile=sys.stdout)

print 'Requesting thumbnails...'

request = ThumbnailRequestMsg()
request.Url = "alexa.com"
request.Signature = signature
request.Timestamp = timestamp_tuple
request.AWSAccessKeyId = AWS_ACCESS_KEY_ID
request.Request = [request.new_Request()]

resp = port.Thumbnail(request)

4. Run, and see error.

ZSI.EvaluateException: Got None for nillable(False), minOccurs(1) element 
(http://ast.amazonaws.com/doc/2006-05-15/,Url), 



 xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" 
xmlns:ns1="http://ast.amazonaws.com/doc/2006-05-15/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

[Element trace: /SOAP-ENV:Body/ns1:ThumbnailRequest]

55. Conclusion

I’m not entirely certain what I’m doing wrong. I’ve also written another version but actually with NPBinding connecting to the wsdl file. It seems to work much better, as it fully connects, and I get a 200, but it doesn’t return the thumbnail location in the response, and I get a:

TypeError: Response is "text/plain", not "text/xml"

So, while I have things working fine with REST, I would like to get the SOAP calls working. One beer reward.

As some of you may know, I have some projects cooked up. I don’t expect to make a million bucks (wish me luck!), but a few extra bills in the pocket wouldn’t hurt. Plus, I’m highly considering further education, which will set me back a few-thirty grand. That said, one of my projects will rely heavily on Amazon Web Services. Amazon has, for quite some time now, opened up their information via REST and SOAP. I’ve been trying (virtually the entire day) to get SOAP to work, but seem to get snagged on a few issues. Stay tuned.
However, in my quest to read every RTFM I stumbled upon a post regarding Python+REST to access Alexa Web Search. After staring at Python code, especially trying to grapple why SOAP isn’t working, updating the outdated REST code was a 5 minute hack. So, if you are interested in using Alexa Web Search with Python via Rest, look below:

websearch.py

#!/usr/bin/python

"""
Test script to run a WebSearch query on AWS via the REST interface.  Written
 originally by Walter Korman ([email protected]), based on urlinfo.pl script from 
  AWIS-provided sample code, updated to the new API by  
Kelvin Nicholson ([email protected]). Assumes Python 2.4 or greater.
"""

import base64
import datetime
import hmac
import sha
import sys
import urllib
import urllib2

AWS_ACCESS_KEY_ID = 'your-access-key'
AWS_SECRET_ACCESS_KEY = 'your-super-secret-key'

def get_websearch(searchterm):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    
    signature = generate_signature('WebSearch', timestamp, AWS_SECRET_ACCESS_KEY)
    
    def generate_rest_url (access_key, secret_key, query):
        """Returns the AWS REST URL to run a web search query on the specified
        query string."""
    
        params = urllib.urlencode(
            { 'AWSAccessKeyId':access_key,
              'Timestamp':timestamp,
              'Signature':signature,
              'Action':'WebSearch',
              'ResponseGroup':'Results',
              'Query':searchterm, })
        return "http://websearch.amazonaws.com/?%s" % (params)
    
    # print "Querying '%s'..." % (query)
    url = generate_rest_url(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, searchterm)
    # print "url => %s" % (url)
    print urllib2.urlopen(url).read()

You run it like this:

>>> from websearch import get_websearch
>>> get_websearch('python')

I’ve been playing around more with Hamachi, and have decided that it officially rules. Since I’m a big Linux guy I don’t have access to some features, but the program seems to be a gem. It is brainlessly easy to install (even when doing 20 things at once), and works quite well. Thanks to Ben and Sean for helping me test it out.

You know you are too nerd like when your conversations are like this. Preface: I posted some packages I needed to upgrade into a Skype window (much better than a clipboard)….

[11:56:03] Kelvin Nicholson: sorry, needed to post that somewhere

[11:56:04] Ian FItzpatrick: i am not apt!

[11:56:15] … you can’t pass packages to me ;)

[11:56:34] Kelvin Nicholson: blah blah blah

[11:56:43] … apt-get upgrade ian

[11:57:02] Ian FItzpatrick: apt-get error: unmet dependency, “beer 1.0-4 not found”

[11:57:14] Kelvin Nicholson: yea, that got a good laugh

Well, I’m basically all done upgrading to Version 3.0, I deserve a cake or something. Here’s the 411:

For the past few years I have been using Mambo, then Joomla, to manage the content on my site. It worked quite well, and was in PHP, so I could add or remove any code. Indeed, I’ve written a decent amount of PHP apps. In early 2004 I wrote a PHP platform to track adventures people had gone on, and networked people seeking to go on adventures with each other. I never marketed it, and mainly created it to learn PHP, but it was a CMS (Content Management System), and a little more. Late in 2004 I wrote another blog-esque platform for my second trip to Europe. It was pretty cool, I’ll admit: Casey and I each had a blog, and people could leave us “dares” and/or messages – and we could easily update our status. Overall, it worked great. You can also see the projects section of my site for some of the other things I’ve done in PHP.

Fast forward a few years, and here it is in early 2007. I’ve never really liked PHP all that much, but I couldn’t put my thumb on it. Deciding to switch to something else, I picked up and read the book, Beginning Python, from Novice to Professional. If anybody is looking for a well written book, I would highly recommend this one. Anyways, with my goal to drop PHP in mind, I held the debate of Django and TurboGears. I went through the demos for each, and felt like I really played around with them. Ultimately it came down to 1) Django has obvious crazy cool caching, 2) Django has pretty darn good documentation, and a freaking online book, and 3) the “powered by” sites are quite impressive – both the length of the list and the large amount of traffic some of these sites entertain.

So I went with Django. My friend in New Zealand, Ben Ford, has been ragging me for two months to get my ass in gear and learn it, saying I would love it. And he is right, the framework is simply beautiful. For the last week I’ve been reading through the documentation, going through the online book (both are incomplete, in my opinion, but compliment each other nicely). I think it is important to write your own code instead of just repeating examples, so my goal: transform my blog/site by using just Django.

So, while some of the kinks still need to be worked out, everything is no transfered over. I’ll mention my experiences shortly, but overall: I’m very impressed.

Over the years I have been shrinking the number of computers I own.  At one point my dorm was littered with old P100s, running whatever OS I felt like playing with at the time.  

VMWare comes to help.  In this recent oops, an image I made (for Windows XP), was slightly too small.  I didn’t feel like reinstalling XP + cruft, so just resized the image.  This is the process:

1. Make Clone or just backup your VMWare image.
2. Note: if your disk is a Dynamic Disk, you won’t be able use GParted.  There is a chance you can use Disk Management inside Computer Managemen inside XP.
3. Turn off VMWare image.
4. Grow the image.  

 vmware-vdiskmanager -x sizeGB yourimagename.vmdk 

5. Download the GParted LiveCD
6. Change the CD-ROM drive of your  VMWare image to boot from the ISO you just downloaded.
7. Boot VMWare image.  Make sure to press ESC right when it starts.
8. Follow the instructions for GParted. I had to select the Xvesa option, then Done.  Choose your language and keyboard and resolution.
9. GParted will come up.  First delete the partition (the empty one!), and make sure it says unallocated.  Then go up to Edit and hit Apply.  Then select the partition and say Resize.  Hit apply again.
10. Reboot image.  Windows XP will come up, and go through checking the disk.  It will reboot again, and you should then be able to log in.

To all my loyal (but most likely few, and mainly with the same last name as me):

I think I’m going to redo the structure of my site, yes, it is about time.  I’ve been using a CMS (Content Management System) to categorize everything, but it is time to ditch that and write something.  Time to stop being lazy.  Time to stop procrastinating.  Time to learn something new.

Let me make this clear: version 3.0 surely won’t be based off PHP.

As you may know by now, I’m in Taiwan.  Live is pretty good here, especially the internet – I’m sitting on a pretty decent 12M/1M connection.  But the company I have my internet through seems pretty laid back about network usage
-- which for me isn’t good.  Their entire IP subnet appears banned from IRC, which means I have to be a little sneaky when I want to talk to my I.T. friends in New Zealand.  Worse, some websites even banned the entire subnet:

Your access to this site has been denied because of the large amount of abuse produced by the users from your country. The access will be enabled once we perform the investigation of the abuse issues. Thank you for your patience and understanding.

Luckily, Tor+Privoxy+FoxyProxy provides a quick way around any blocks that might be setup.  So, to the developers of these solutions, my kindest thanks.

I love my Snort, I really do.  But sometimes, I just don’t need all the extra overhead – sometimes the resources on a server are somewhat, limited.  Looking for a solution I stumbled upon PSAD , a way to detect port scans.  Since port scans are often one of the first tactics used to find vulnerabilities on a server, it is a pretty good idea to detect them.   Depending on the attack, I receive a nice little email telling me what is going on.  To test it out I first fired up nmap, and received a few emails.  Next I fired up nessus with updated plugins – you can be the judge.  I now have 50 emails from myself telling me somebody is doing something naughty:

\=-=-=-=-=-=-=-=-=-=-=-= Tue Jan 23 10:30:04 2007 =-=-=-=-=-=-=-=-=-=-=-=


         Danger level: [5] (out of 5) Multi-Protocol

    Scanned tcp ports: [11-41111: 337 packets]
            tcp flags: [SYN: 337 packets, Nmap: -sT or -sS]
       iptables chain: INPUT, 337 packets

               Source: 218.167.75.27
                  DNS: 218-167-75-27.dynamic.hinet.net

          Destination: 64.79.194.165
                  DNS: kelvinism.com

      Syslog hostname: kelvinism

     Current interval: Tue Jan 23 10:29:54 2007 (start)
                       Tue Jan 23 10:30:04 2007 (end)