I like most standards, I can’t deny it. One thing that I find too humorous is the amount of sites that are not valid, even sites that claim they are. One tool in my box is HTML Validator, an extensioin for Firefox. A nice little icon appears in the lower right, indicating whether a page is valid, not-so-valid, or really-not-so-valid. Makes it easy to check your own sites, and others.
Information is power, or so many of us think. As an I.T. administrator, no matter what the level, it is of my opinion that knowing what your network is doing is important. This includes regular operation, what it could do in the event of a disaster, or when it is just slightly not functioning.
Jeez, I’m personifying a computer network.
As I wrote in another post, I setup network monitoring for several nodes. But, what happens when an anomaly occurs? This is the story of hunting down a worm/virus, from far, far away.
While I can’t go into too many details, for obvious reasons, I’ll try and tell the story as-it-The overall process took for cleaning took several weeks to resolve, although minimizing the effect occurred the second the anomaly was discovered. The lengthy time to resolve is mainly due to the time to request computers to be patched/updated/scanned (like I said, this anomaly was with a large branch office in another country).
I’ve blurred out any relevant information for obvious reasons.
Almost immediately after setting up monitoring I noticed something strange occurring. As you can notice from the below graph (from nfsen), something is obviously wrong. If you can’t notice it, that big spike, in what we’ll call Network Green, should give you a clue. Now, I can expect a spike during lunch when people watch movies or send their 50mb picture email attachments, but this spike isn’t always at noon, and as we’ll see, it sure ain’t emails.
Find the anomaly:
So, let’s try to find out what is going on. Indeed, there are some analyzers of flow traffic that can display the types of traffic, and are useful if the issue isn’t apparent, but NFSEN’s filtering capabilities haven’t failed me yet (although seeing a graph with types of traffic is useful).
We can now take a look at what is going on. Let’s next just take a look at the statistics offered:
I’ve left the dates in there for you to see something interesting: Network Green transferred over 15 GIGS of UDP traffic. If we think back to networking basics, we can remember that UDP is a connectionless protocol. So, what uses UDP? DNS/TFTP, some streaming media, VoIP, and several types of encryption. That said, there isn’t any legitimate reason for any of the employees to use that much bandwidth, of this type. Only one thing seemed apparent: somebody was massively downloading from some p2p source, or we had a worm/virus.
Now that we know it is UDP related, let’s find out more. I’ve filtered by UDP in the graph just so you can see how much traffic that really is. Let’s select the anomaly. We first select the left side…
Then we select the right side, and we have a pretty green highlighted section.
Now it is time to hunt the person down. Using NFSEN’s built-in filtering capabilities, I was able to find which computer was being naughty. I first created the filter rule for the appropriate network (IP obscured for confidentiality):
Then could see the obvious place to look next:
As is highlighted, you can see that one IP is continually transferring UDP traffic over port 14857. Well, it certainly isn’t DNS or TFTP! (But we knew that when 15+ gigs was transferred:) When we look at the Top 10 Src IP Addr, ordered by bytes, we can see that one IP transferred a whole lot of traffic.
What now? I pulled out the nmap/nessus combo and tried to see what was running. Telneting to port 14857 didn’t return any form of hello message, and nmap didn’t return any known services for whatever was on port 14857. At this point I sent out the emails/documentation to managers in the remote office requesting for anti-virus to be checked and loaded onto any computers. Next, I blocked the port, from that computer, from sending outgoing traffic. Occasionally, as you can see in the graph, another computer or two would show the same symptoms, but within two weeks the oddity had disappeared. Thanks you NFSEN and Cisco.
The following sites might be helpful to configure IPSec again, or on a more complex basis:
http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_60/ipsec/conipsec.htm
Logs spiraling crazy, we run out of disk space all the time. A nifty trick to find where the disk went is to issue:
du -cks * |sort -rn |head -11
This returns where the disk usage is, and makes finding the bloated log a lot easier.
Here’s a tip for Firefox. An extension called BugMeNot enables you to right click on “general” forms and say “Login with BugMeNot.” Firefox will call the BugMeNot extension which looks up in an external database login information. For instance, www.nytimes.com requires a login, which is a great place to test it out.
I have a feeling we will shortly be deploying many Linux servers to perform certain actions. Maybe we will implement Asterisk to be used as a VoIP interchange between locations, maybe the backup servers will be Linux based, maybe the BDCs.
One thing that could speed up implementation at remote sites is to build live cds for certain purposes. For instance, on the file server in PDX to keep updated live cds for certain projects. Like, a BDC live cd or a backup live cd. Already setup with the most current packages (or scripts to fetch+install them). So when we get to the site we just put the CD in, click or type “load” and poof, the server is installed and configured.
These links (haven’t read all of the process) may be helpful:
A developer just showed me an interesting framework to produce python-backed sites VERY quickly. This is mainly for you Ian, it natively supports AJAX as well. Here’s the link:
I watched the demo, pretty interesting.
We don’t want to have a billion servers each doing their own task – so what can we use as a solution? Server virtualization (or semi-virtualization or para-virtualization). This involves cutting down a server into mini servers that each have full customization. Our VPS at hostmysite is like this. So why would you want to do this? A few reasons actually.
-Localize exploits. Let’s say DNS gets exploited – the access gained would only be for DNS, and not for mail and web and everything else.
-Easy “upgrades,” backups and redundancy. Let’s say we start to use MySQL more and more, but the server can’t handle it. To upgrade (ignoring replication for this example) we could just turn off the virtual server (in essense lock files), move it to other server, drop it into another server that is setup to do virtualization, and turn it it on. Nearly no downtime, and you know it will work.
Anyhow, worth looking at. Here are some of the most mature linux virtualization packages out there:
http://openvz.org/ – This is the open source version of hostmysites VPS. The main difference is it isn’t setup for doing mass hosting (like, 1000 VPSs on a huge mainframe).
http://linux-vserver.org/ – Very plain website, but there is news that the authors are pushing for this code to be included in the Linux kernal natively.
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ – I’ve heard rumors also about this being one of the most advanced.
http://www.vmware.com/ – The one and only. This is full virtualization so will contain the most overhead (some of the previous packages have almost no overhead, not even 1%). Oh yea, and this “costs” money.
My friend Ian told me about this originally, but my pen-testing cousin just send me the link as well. p2p VPN, w00t. Hamachi is a VPN alternative that does not have the normal router problems associated with IPSEC and PPTP vpns. That is good because of firewalls and nat and things like that.