Status: ✅

One of the greatest benefits, in my opinion, of Cisco routers is the ability to generate netflows. In a lot of ways, I would prefer to do this than implement some appliance (say, using ntop). The ability to analyse the amount of traffic becomes extremely valuable. Not only can one measure the amount of traffic, but the type of traffic that is being generated through the network.

Using a similar configuration, I setup all four Ciscos to export netflows that stream back to a server in the States. I decided to use nfdump as a collector. After the dumps are collected, it is simple to setup nfsen to parse and analyse the received flows. It even allows you to generate really pretty graphs.

So, why do this? For starters, collecting netflows allows the basic analysis of data, which can tell you several things. You can know instantly how saturated your connection is, if there are any anomalies, if there is any file sharing going on or when heavy traffic usage is. For instance, if the connection becomes slow during the end of the day, you can analyse what protocol is used the most during that time. Or, as was my case, hunting down virus infected computers that were fully saturating a 10mbit pipe.

A week in the life of NFSEN:

Status: ✅

Users don’t like to remember passwords, heck, I don’t like to remember to use passwords. I decided to upgrade all the webapps to authenticate off the domain, welcome a start to SSO. To do this I implemented the adldap php class to control authentication to each webapp. Thus, a simple GPO can control who has access to the app or not. A simple solution to a rather simple problem.

Status: ✅

Backing up across the states has worked decently well, but due to several changes a more dedicated backup solution is in order. Desiring something quick, simple and inexpensive, research revealed a company that would perfectly fit the requirements. iBackup was a perfect substitute - instead of SSH+rsync to another office, iBackup provides rsync over ssl to their data center. A few simple changes to the cron job, and backup location is thus changed.

Status: ✅

Time has come to bring another network on the VPN, and perform some more upgrades. The usual by now, I guess.

• Get China on VPN
• Limit access to other locations
• Update all systems
• Perform security audits
• Upgrade wifi
• Setup video conferencing

Ian and I set off for our China office out of Hong Kong, and the next day started working. Total preparation was around a month, maybe a little large, mainly due to red tape. We first acquired assistance of IBM China, who were of a great help aiding us in finding our desired Cisco. One of the most important factors, which we couldn’t resolve by purchasing the Cisco in the States, is support/warranty contracts (if the Cisco totally dies, what then). Through our contact we were also able to find some local vendors that would support Wifi and the Cisco, in case of an emergency.

Before leaving I prepared the necessary configurations for the Cisco, or at least a good guide to start from. The technician who came out tried to get things going through the built in GUI, however wasn’t have so much luck. I took over using my pre-built configuration and soon (we swapped out the old router with the Cisco during lunch) everything, including overloaded NAT, was working fine. By the time employees came back from lunch, they couldn’t notice any difference.

While the Cisco tech (who I believe is a good guy, even though I did the Cisco install) was waiting for some paper work went through and upgraded the way obsolete wifi point from WEP (which wasn’t even turned on anyways) to WPA. The reasons for this, especially connected to the VPN, are very obvious. Technically the AP wasn’t supposed to support WPA, but he found the correct Chinese firmware and it worked. This is good, as the new AP wouldn’t be coming for a little while.

Next on the list was video conferencing. The solution was the path of least resistance: Skype on a laptop. Ian took this one, setup the laptop, and tested conferencing back to the States.

On the agenda for that night was VPN. The problem with bringing the China office on the VPN is one of security. Virus’ were quite prevalent (e.g. my shared drive on my Linux laptop, to use as a sandbox, had a couple .exe files dropped into it. All with rather odd names…) – so we first ran some security audits. Nessus was a great help, as always, and we tracked down over [an UNFATHOMABLE amount of] critical holes. Picking the biggest culprits we started patching computers, removing spyware and running anti-virus. Slowly (a few days) we got the number knocked down significantly.

Lastly I hooked China up to the VPN. In order to do this safely I created some very strict access lists, to only allow outgoing communication over ports 80 and 443 (since that is all they needed at that point). Previously setup we had a webshare website (auth linked to the PDC), so no need to open any other ports.

Overall we completed what we set out to do. We made a few good contacts, achieved our goals, and once again learned more about doing I.T. overseas.

Status: ✅

Before we grow any further, a new DNS scheme is in order. Following the pattern of: citycode.domain.com - shouldn’t be too hard. A slightly stressful rename of the PDC (just one so far, still small) was in order. After that (and client computers re-associated), the routers were updated, the DNS server updated, and everything worked peachie. Not bad for a weekends worth of work.

Status: ✅

All is well for some disasters, but what happens if our entire office burns down? SSH+rsync to the rescue, again.

I first setup the PDC and webapp server to backup to the file server on a regular basis (PDC: incremental every day, full on Saturday). Then the file server takes those backups (including the files stored on the file server) every night and syncs them with another server across the States. In case something drastic happens, these off-site backups should be a savior.

Status: ✅

The webapp server is running fine, but backups are important. Better yet, a hot computer is a great idea. To do this, I setup an older spare rackmount as a ’live’ webapp server, just in case. A duplicate LAMP was setup, web apps copied over SSH via rsync on a regular basis, and the icing on the cake: mysql replication.

So, if the dedicated webapp server dies a painful death, a quick change of IP for the webapp server in the internal DNS to the backup rackmount, and nobody will know anything happened.

Status: ✅

Time has come to upgrade a few servers in the office. An older P4 2.8 was being used as a webapp server, and that needs to go. The resource utilization wasn’t too much of an issue, however the computer was aging. Plus, it wasn’t strictly built to host critical services, but since we grew so quickly, it is what was available. Additionally, the PDC was hosting user files and with these mounting in size, a dedicated file server is in order.

Oh, and Ian and I are on a strict budget, as usual.

Our trusty CDW shipped over two IBM rackmounts. Plenty of CPU and RAM to grow, the key feature that we were needing was hardware RAID1. With those shipped out, Ian screwed them into the rackmount and we started working on them. Both servers had Debian slapped on, and one then into a true LAMP server. On the LAMP server we also loaded up our ticketing system, and several IMAP based email accounts (good ol’ Dovecot).

On the other server was setup as a dedicated file server. For several reasons, including the strict budget, we synced Samba up to the 2003 PDC. Thus, all profiles (through file redirection) are mapped to the Samba box, which does auth via kerberos back to the PDC. Besides user profiles, several shared folders exist, and access is based on GPO. I must admit, Samba+Windows2003 is a very handy combo.

Status: ✅

I realized at the end of the 2005 school year that I was going to be two business credits short of graduating. Two! Understandable I think, since I had been working full time or over full time (one to four jobs) for the last three years. Plus, I changed my degree after the first year; not to mention I somehow crammed five years worth of credits in four years. Well, I arranged with one of my teachers (who had a great class, I must add) to guide me through the last two credits. The credits involved writing a paper regarding China and the Internet. A broad topic, I must admit.

The paper was a joy to research, write and edit. I explored many faucets I didn’t think I would, and realized that I mostly enjoy semi-technical writing. Through this paper I realized that technology in emerging countries is an interest to me, and I look forward to what this interest has in store for my future. You can download my paper (PDF) here.

Status: ✅

A network upgrade is in order, since we are depending more and more on our internet connectivity. Historically we have been using D-Link “Business Grade” equipment over DSL lines (decent bandwidth, but not 100% reliable, plus latency that is a little high). Time for an upgrade.

Since FiOS isn’t offered yet where our office is located, we had to settle on a T1. However, since both locations in the States will have T1s through the same company, the quality should be decent. Since I’m telecommuting now, my colleague organized what company to order the T1 through, and had the line installed. Since our offices are both quite small, there isn’t a need for any huge routers, we aren’t moving a tremendous amount of traffic. Then again, we do need a certain amount of features. Initially the T1 company almost required us to use their equipment (which was luckily discounted highly), and after we gave them our set of requirements, they gave us a pair of Cisco 1723s, which I was a little skeptical about. A Cisco technician came out and sort of set them up (enough for me to gain remote access at least). However, a slew of issues surrounded the 1723s. The routing wasn’t setup up quite properly, and the IOS was a little outdated.

Ultimately it turns out the routers weren’t right, and wouldn’t support our requirements (which was my guess in the first place). Oh well. A quick call to CDW and we had a pair of Cisco 1841s sent to Portland. After some widgetry magic (my knowledge of the Cisco CLI, to some degree at least) I got them both configured for their respective networks to run over the T1, then quickly setup NAT and then IPSec. Overall they run very smooth, and after installation they just have kept working.

So, there you have it. A quick network upgrade in two sites. Go Cisco.