Amazon offers some pretty cool services: S3, EC2, Alexa Site Thumbnail, and others. A while back I wanted to use AST with Django, so ended up writing the Python bindings to the REST API (they didn’t previously exist. I even wrote up a quick tutorial.
Update: Amazon no longer maintains AST. I’ve decided to archive a few of the old sites, so no longer need to take thumbnails. However, a few other thumbnail services seem to have crept up, including SnapCasa", and WebSnapr.
A normal internet connection gets attacked, a lot. The majority of attacks are of the form “hello, anybody there?” – where most people just don’t answer. But sometimes, just sometimes, the question gets an answer. Depending on the answer, the attacker will start to explore.
A few weeks back I was a little bored and started fiddling. I wanted to play with my Cisco, but also wanted to play with OSSEC, but also has a GIS craving. In the end I decided to create a map of the people who ask, “hello”.
Take a look at the map and explanation if that sort of thing is your cup of tea.
I’ve been side-tracked on another little project, and keep coming back to NetFlow. For this project I’ll need to access NetFlow data with Django, but this is a bit tricky. First, I’m sort of lazy when it comes to my own project; maybe not lazy, I just like taking the most direct route. The most up-to-date NetFlow collector I noticed was flow-tools, and there is even a switch to export the information into MySQL. Sweet! However, I wanted to insert the flows into MySQL automatically, or at least on a regular basis. I first started writing a python script that would do the job, but after a few minutes noticed flow-capture had a rotate_program switch, and started investigating. Since I somehow couldn’t find anywhere instructions how to insert the data automatically, here’s what I came up with:
CREATE TABLE `flows` (
`FLOW_ID` int(32) NOT NULL AUTO_INCREMENT,
`UNIX_SECS` int(32) unsigned NOT NULL default '0',
`UNIX_NSECS` int(32) unsigned NOT NULL default '0',
`SYSUPTIME` int(20) NOT NULL,
`EXADDR` varchar(16) NOT NULL,
`DPKTS` int(32) unsigned NOT NULL default '0',
`DOCTETS` int(32) unsigned NOT NULL default '0',
`FIRST` int(32) unsigned NOT NULL default '0',
`LAST` int(32) unsigned NOT NULL default '0',
`ENGINE_TYPE` int(10) NOT NULL,
`ENGINE_ID` int(15) NOT NULL,
`SRCADDR` varchar(16) NOT NULL default '0',
`DSTADDR` varchar(16) NOT NULL default '0',
`NEXTHOP` varchar(16) NOT NULL default '0',
`INPUT` int(16) unsigned NOT NULL default '0',
`OUTPUT` int(16) unsigned NOT NULL default '0',
`SRCPORT` int(16) unsigned NOT NULL default '0',
`DSTPORT` int(16) unsigned NOT NULL default '0',
`PROT` int(8) unsigned NOT NULL default '0',
`TOS` int(2) NOT NULL,
`TCP_FLAGS` int(8) unsigned NOT NULL default '0',
`SRC_MASK` int(8) unsigned NOT NULL default '0',
`DST_MASK` int(8) unsigned NOT NULL default '0',
`SRC_AS` int(16) unsigned NOT NULL default '0',
`DST_AS` int(16) unsigned NOT NULL default '0',
PRIMARY KEY (FLOW_ID)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
kelvin@monitor:/usr/bin$ cat flow-mysql-export
#!/bin/bash
flow-export -f3 -u "username:password:localhost:3306:netflow:flows" < /flows/router/$1
flow-capture -w /flows/router -E5G 0/0/9801 -R /usr/bin/flow-mysql-export
I’ve evaluated Zenoss before, but forgot the default password, and searching for it didn’t come up with anything quickly. I tried everything under the sun: password, 1234, admin, God, Sex, but alas, grep to the rescue:
kelvin@monitor:/usr/local/zenoss/zenoss/etc$ grep admin *
hubpasswd:admin:zenoss
Update: it is listed on page 4 of the Admin PDF :)
My little server doesn’t have a cdrom, but I didn’t want to actually run ESX from a USB (i.e. esx-on-a-stick). Here are my notes of configuring a flash disk to boot the ESX installer (so you can install it onto a local disk). For this demo, my USB is /dev/sdb
sudo install-mbr /dev/sdb
sudo syslinux /dev/sdb1
default menu.c32
menu title ESXi Boot
timeout 100
label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkernel.gz --- binmod.tgz --- environ.tgz --- cim.tgz
ipappend 2
I recently had the need to move a rather large (450GB) VMDK file from an external hard drive into ESXi. Since ESXi doesn’t support external hard drives, this makes things quite a bit more difficult. At first I tried using SCP to copy the file over (after enabling SSH access for ESXi). However, when I tried to do this the time left was almost 20 hours – a tad too long!
I rethought my idea and decided to use this process:
Creating the NFS share on Linux is extermily easy. After install nfs via whatever package management tool you choose, put this entry into your /etc/exports file:
/media/disk-1 192.168.1.0/24(ro,no_root_squash,async)
This assumes your USB disk is mounted as /media/disk-1, and your local subnet is 192.168.1.0/24. In OpenFiler, add a new storage with type NFS and use your laptops IP as the hose, and /media/disk-1 as the mount point. For safey, tick read-only.
Next, unlock SSH if you haven’t already. Once you are in, browse to /vmfs/volumes and you can see your nfs share and your other datastores. Let’s say you USB virtual disk is located at /vmfs/volumes/nfs/bigdisk.vmdk, and you want to import it into your normal datastore, under a folder called ‘NAS’. Using vmware specific tools, you can import the file as so:
# vmkfstools -i /vmfs/volumes/nfs/bigdisk.vmdk /vmfs/volumes/datastore1/NAS/bigdisk.vmdk
I needed to update the hardware version of my imported disk. To do this, open up the .vmdk file (you should also have a -flat.vmdk file), and update the virtualHWVersion entry from 7 to 4. With that, join your disk to an image, and you should be good to go.
An addition result I noticed was the speed at which it came over. By using SCP, the entire file was going to take 20hr. By using NFS and vmkfstools, the files was migrated in under 10 hours.
I’ve had issues before with OpenFiler where doesn’t update the permissions, although they appear correct in the UI. To rectify that, I stumbled upon a one liner that fixed it. Let’s say you have a group called “Trusted” that you want to have full access to your music folder. Here’s the one-liner:
[root@files data]# pwd
/mnt/openfiler/data
[root@files data]# setfacl --recursive -m u:nobody:rwx,g:Trusted:rwx music
I found VMWare Server to have very slow I/O, and sought to improve it. Below are a list of tests I performed, the change, and the results.
### Host OS ###
/dev/sdb1:
Timing buffered disk reads: 220 MB in 3.05 seconds = 72.17 MB/sec
kelvin@gorilla:~$ sudo hdparm -t /dev/sdb1
/dev/sdb1:
Timing buffered disk reads: 266 MB in 3.01 seconds = 88.33 MB/sec
kelvin@gorilla:~$ sudo hdparm -t /dev/sdb1
/dev/sdb1:
Timing buffered disk reads: 310 MB in 3.01 seconds = 102.99 MB/sec
### Before Changes ###
/dev/mapper/openfiler-data:
Timing buffered disk reads: 8 MB in 3.36 seconds = 2.38 MB/sec
[root@files etc]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 24 MB in 3.63 seconds = 6.61 MB/sec
[root@files etc]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 28 MB in 4.54 seconds = 6.16 MB/sec
I made several changes, but the changes that seemed to have the most impact are below:
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.swappiness = 0
Pop this into the virtual machine’s .vmx file, reboot, and off you go. One unfortunate side effect is that you can no longer overload the memory (e.g. allocate more memory with the VMs than you actually have available).
### After Changes ###
/dev/mapper/openfiler-data:
Timing buffered disk reads: 52 MB in 3.13 seconds = 16.61 MB/sec
[root@files ~]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 82 MB in 3.31 seconds = 24.75 MB/sec
[root@files ~]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 118 MB in 3.19 seconds = 36.97 MB/sec
[root@files ~]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 144 MB in 3.32 seconds = 43.37 MB/sec
[root@files ~]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 160 MB in 3.10 seconds = 51.57 MB/sec
UPDATE: Those wanting all the speed and still want to use memory overloading, I’d suggested you give ESXi a try. So far, so good.
## With ESXi, same hardware ##
[root@files ~]# hdparm -t /dev/mapper/openfiler-data
/dev/mapper/openfiler-data:
Timing buffered disk reads: 200 MB in 3.18 seconds = 62.92 MB/sec
I rank OSSEC as one of my favorite pieces of open source software, and finally decided to play around with it more in my own free time. (Yup, I do this sort of stuff for fun). My goal was quite simple: send syslog packets from my Cisco to my “proxy” server, running OSSEC. I found that, although OSSEC supports Cisco IOS logging, it didn’t really work. In fact, I couldn’t find any examples or articles of anybody actually getting it to work.
I initially tried to get it to work “correctly,” and soon settled to “just getting it to work.” I implemented some rules in the local_rules.xml file, which worked, but I’m pretty stubborn, and wanted to do it “correctly.” With a couple pots of tea I became much, much more familiar with OSSEC. The key (and a lot of credit) goes to Jeremy Melanson for hinting at some of the updates to the decoder.xml file that need to take place.
The first step is to read the OSSEC + Cisco IOS wiki page. Everything on that page is pretty straight forward. I then added three explicit drop rules at the end of my Cisco’s ACL:
...
access-list 101 deny tcp any host 220.244.xxx.xxx log
access-list 101 deny ip any host 220.244.xxx.xxx log
access-list 101 deny udp any host 220.244.xxx.xxx log
(220.244.xxx.xxx is my WAN IP, and I’m sure you could figure out xxx.xxx pretty darn easily, but I’ll x them out anyways).
To reiterate, OSSEC needs to be told to listen for syslog traffic, which you should be set on the Cisco. If you haven’t done this, go re-read the wiki above.
<remote>
<connection>syslog</connection>
<allowed-ips>192.168.0.1</allowed-ips>
</remote>
On or around line 1550 in /var/ossec/etc/decoder.xml I needed to update the regex that was used to detect the syslog stream.
...
<decoder name="cisco-ios">
<!--<prematch>^%\w+-\d-\w+: </prematch>-->
<prematch>^%\w+-\d-\w+: |^: %\w+-\d-\w+:</prematch>
</decoder>
<decoder name="cisco-ios">
<program_name>
<!--<prematch>^%\w+-\d-\w+: </prematch>-->
<prematch>^%\w+-\d-\w+: |^: %\w+-\d-\w+: </prematch>
</program_name></decoder>
<decoder name="cisco-ios-acl">
<parent>cisco-ios</parent>
<type>firewall</type>
<prematch>^%SEC-6-IPACCESSLOGP: |^: %SEC-6-IPACCESSLOGP: </prematch>
<regex offset="after_prematch">^list \d+ (\w+) (\w+) </regex>
<regex>(\S+)\((\d+)\) -> (\S+)\((\d+)\),</regex>
<order>action, protocol, srcip, srcport, dstip, dstport</order>
</decoder>
...
In the general OSSEC configuration file, re-order the list of rules. I had to do this because syslog_rules.xml includes a search for “denied”, and that triggers an alarm.
...
<include>telnetd_rules.xml</include>
<include>cisco-ios_rules.xml</include>
<include>syslog_rules.xml</include>
<include>arpwatch_rules.xml</include>
...
Remember that these dropped events will go into /var/ossec/logs/firewall/firewall.log. Because this is my home connection, and I don’t have any active_responses configured (yet!), I tightened the firewall_rules.xml file (lowering the frequency, raising the timeframe).
And in the end, I get a pretty email when somebody tries to port scan me.
OSSEC HIDS Notification.
2008 Nov 15 23:19:36
Received From: proxy->xxx.xxx.xxx.xxx
Rule: 4151 fired (level 10) -> "Multiple Firewall drop events from same source."
Portion of the log(s):
: %SEC-6-IPACCESSLOGP: list 101 denied tcp 4.79.142.206(36183) -> 220.244.xxx.xxx(244), 1 packet
: %SEC-6-IPACCESSLOGP: list 101 denied tcp 4.79.142.206(36183) -> 220.244.xxx.xxx(253), 1 packet
: %SEC-6-IPACCESSLOGP: list 101 denied tcp 4.79.142.206(36183) -> 220.244.xxx.xxx(243), 1 packet
: %SEC-6-IPACCESSLOGP: list 101 denied tcp 4.79.142.206(36183) -> 220.244.xxx.xxx(254), 1 packet
--END OF NOTIFICATION
As you can tell from reading some of the other pages, I like Linux and open source. But I also like to answer the question “what if…” This post is my [brief] run down of answering “what if I could run Django on Server 2003 with SQL Server and IIS.” Why, you may ask? To be honest with you, at this point, I don’t really know. One of the deciding factors was seeing that the django-mssql project maintains support for inspectdb, which means I could take a stock 2003 server running SQL Server, inspect the DB, and build a web app on top of it. The Django docs offer a lengthy howto for using Django with IIS and SQL Server, but the website for PyISAPIe seems to have been down for the last month or so. Without further delay, below are my notes on installing Django with SQL Server and IIS.
1a) Install python-2.x.x.msi from python.org
1b) Consider adding C:\Python25\ to your Path (right click My Computer, Advanced, Environment Variables. Enter in blahblahblah;C:\Python25\)
3a) Extract the contents of the Django. From inside Django-1.0, execute:
C:\Python25\python.exe setup.py install
3b) Consider adding C:\Python25\Script to your path.
4) Look in C:\Python25\Lib\site-packages – confirm there is a Django package.
5) Checkout django-mssql (http://code.google.com/p/django-mssql/), copy sqlserver_ado from inside source to the site-packages directory
6) Download and install PyWin32 from sf.net
7) Start a test project in C:\Inetpub\ called ’test'
c:\Python25\scripts\django-admin.py startproject test
8a) Create a database using SQL Management Studio, create a user. (First, go to the Security dropdown. Right click Logins, add a new user. Next, right click Databases, New Database. Enter in the name, and change the owner to the user you just created).
8b) Edit the settings.py and add ‘sqlserver_ado’ and add database credentials. Use the below example if your database comes up in the Studio as COMPUTERNAME\SQLEXPRESS (you are using SQLExpress).
import os
DATABASE_ENGINE = 'sqlserver_ado' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = 'crmtest' # Or path to database file if using sqlite3.
DATABASE_USER = 'crmtest' # Not used with sqlite3.
DATABASE_PASSWORD = 'password' # Not used with sqlite3.
DATABASE_MSSQL_REGEX = True
DATABASE_HOST = os.environ['COMPUTERNAME'] + r'\SQLEXPRESS' # I use SQLEXPRESS
DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3.
python setup.py install
10a) Download pyisapi-scgi from http://code.google.com/p/pyisapi-scgi/
10b) Extract the files to somewhere you can remember on your computer, like, c:\scgi
11) Double click pyisapi_scgi.py
12a) Follow the directions here: http://code.google.com/p/pyisapi-scgi/wiki/howtoen – I set a temporary different port since I’m just testing this out.
12b) The last few parts might be better served with an image or two:
(No resource/photo)
(No resource/photo)
(No resource/photo)
13) Start the scgi process from the Django folder directory
python manage.py runfcgi method=threaded protocol=scgi port=3033 host=127.0.0.1
(No resource/photo)