Wyatt Roersma Blog

This year was the 2^nd time GrrCON decided to do a DFIR challenge during the conference and the winner this year received $500.

2013 Winners

1st place – @5ck

2nd place – @kjake

3rd place – @Patories

Jack Crook did an amazing job with the challenge the first time around in 2012. If you haven’t checked out the one from 2012, here is a review from the Volatility Labs blog.

http://volatility-labs.blogspot.com/2012/10/solving-grrcon-network-forensics.html

We only had about 2 weeks to actually design and build out the 2013 challenge this year so the limited time was a huge constraint on everything we really wanted to do. It actually took 10 different virtual machines, and 4 separate networks to create this challenge.

I would like to give a special thanks out to NVINT for hosting the servers and providing dedicated firewalls and IP’s for use. The Hacker Academy and Mad Security provided a huge support for the grading system and the website part of my challenge. I would also like to personally thank Rob Marmo and Nick Deneweth for their help making the challenge. Without their hard long nights working on the challenge before the conference the GrrCON 2013 DFIR challenge wouldn’t have happened this year.

Download the files for the 2013 DFIR Challenge here:

https://drive.google.com/folderview?id=0Bz3L4ZnVlUY8Q0VJbmJCV3JzR28&usp=sharing

2013 DFIR Challenge PDF Walkthrough here – GrrCON-Challenge-walkthrough

Another long gap between since my last post, what else is new.  I have no idea where the time has gone between working on development projects and testing too many things to list. So to kick off this post here is some interesting personal perspective on security in virtualization.

Now I am more than aware that 99% of the virtualization technology in production today is VMware and that Hyper-V is not at the top of most organizations’ consideration. I hope to change that perspective through this small review of some approaches that NVINT is taking as a small company. I am not here to try and sell Microsoft products (I am more of a Google Fan than anything) but when something is done right for more than one reason I always will like that product.

This will be a multi-part blog post that covers a few topics across Server 2012’s Hyper-V 3.0 to the 2012 System Center product family as they relate to the DFIR world.

Hyper-V

I will start with Hyper-V 3.0 inside server 2012 because there are a lot of great new features that really put it above other hypervisors. The first thing that really stands out as a major feature in 3.0 is that you can fully encrypt your storage clusters and allow access to that Shared SMB cluster over the network. This allows you to separate the Hyper-V boxes hosting the Virtual Machines from the machines hosting the SMB Storage cluster. This allows you to lock down access from only the Hyper-V servers to the storage cluster and monitor what accesses that data directly.

At NVINT I currently work with a Hyper-V 2008 R2 cluster that hosts over one-hundred virtual machines across six HP blade servers that make up the Hyper-V host machines. This Hyper-V cluster hosts the production virtual servers for our clients. I have been testing Hyper-V 3.0 for the past ten months, and I still feel like I have only opened one of the presents Santa left me under the tree. In my experience the stability and redundancy is really now on par with other hypervisors as where our current production 2008 R2 Hyper-V cluster lacks.

Windows Server 2012 features a lot of new capabilities, such as the new cloud networking features that scale a higher level network across WAN connections. It also contains the same powerful security features from the past like ASLR and DEP that are critical for security. I am not going to go into the infrastructure aspects because that has been done by the Microsoft engineers and I have provided the links below.

Check out these links if you want to know more about Server 2012 Hyper-V features.

Windows Server 2012 Hyper-V & Network Card (NIC) Teaming – http://www.aidanfinn.com/?p=12924

Hyper-V Network Virtualization Gateway Architectural Guide – http://technet.microsoft.com/en-us/library/jj618319.aspx

Hyper-V Network Virtualization Overview – http://technet.microsoft.com/library/jj134230.aspx

Network Virtualization technical details – http://technet.microsoft.com/en-US/library/jj134174.aspx

Windows Server 2012 Hyper-V Replica http://www.aidanfinn.com/?p=12147

The major upside to Hyper-V for NVINT is the fact that the Datacenter edition of windows allows you to host unlimited Standard editions of Windows in virtual machines without any extra license fees. (Note: this has limitations but that’s outside the scope of this post)

There is one really big feature of Hyper-V that I want to point out, as I feel there is a huge benefit from an Incident Response/Forensics point of view in terms of what you can do with the system that other hypervisors lack. For starters there are a large number of tools that you can use to analyze the Windows OS in many different ways. One of the best features is the ability to take a memory dump of the host os itself.

I have taken memory captures up to 80GB in the system and used Volatility to analyze those dumps successfully. This allows me to peer into the virtualization platform with more in depth detail that can be critical to understanding what is going on in the environment. Most people use VMware which makes it the most targeted Platform. Advanced techniques can be used to detect if the system is a virtual machine and even exploit out of the system to the host causing problems they were intended to prevent.  This may change in the future, but currently Hyper-V is not the largest target on the market in its virtual machines. I am well aware the Windows OS is targeted by everything because it’s used by almost any originations today. I also feel that VMware currently has risks with USB support, currently however that isn’t a drastic problem yet. Someday however I would guess that it will become a more targeted exploit.

Alongside Server 2012 and the features it brings to the table the system center products allow a great deal of management and monitoring across all typed of devices outside the Microsoft Family.

DPM 2012

I currently work with Data Protection Manager almost every single day, and I have come to enjoy this product for a few good reasons. A major benefit for DPM is that it allows you to actively back up the memory as a scheduled process that Microsoft calls a “System State” backup. It also has standard features with Full-Disk and Incremental backups based on a scheduled format. It does anything from attached SAN storage to Tape backups which allows you to scale it out fairly easily.

At NVINT we maintain 15 days of full disk backups and the “System State” included. This allows me to restore a Virtual Machine to a saved state inside Hyper-V or to a network location much like the features inside VMware consolidated backup. The huge benefit from an analysis standpoint is that Hyper-V stores the system RAM for a virtual machine in two separate files the .bin and the .vsv which can be used with vm2dpm  http://archive.msdn.microsoft.com/vm2dmp to convert them into a crash dump format that is compatible with Volatility or Microsoft Debugger for analysis. In an incident I can go into my Hyper-V server and use FTK Imager to create a copy of the live system’s disc and memory without installing anything in the Virtual Machine.

A key benefit of this process with FTK is to gather evidence without direct system interaction thus reducing the footprint on the compromised system. As a hosting provider this is critical because we can’t actively monitor with tools inside the OS itself. The tool Vm2dmp can also convert the DPM backup files because the .bin and .vsv file format is used to store the “System State” for protected systems. I feel that if something happens, that this type of insight can be critical with huge amounts of detail in what happened in the past for a better investigation. These backup files could reduce the data lost by an attacker using anti-forensics techniques. In most cases today the bad guys are getting really good at this process of covering their tracks so the DFIR community needs to get ahead of the problem before it occurs. Outside of virtual machines DPM can still perform “System State” backups of physical machines from Windows 7, Windows 8, Server 2008 R2 and Server 2012.

Here is some quick information on how to use vm2dmp

Create a dump file using virtual machine state files:

vm2dmp.exe -bin C:\VM\example.bin -vsv C:\VM\example.vsv -dmp C:\VM\memory.dmp

Create a dump file from virtual machine and snapshot name:

vm2dmp.exe –vm SERVER2008R2SP1-ENT-64 -dmp C:\VM\memory.dmp

vm2dmp.exe –vm vm SERVER2008R2SP1-ENT-64 –snap “vm SERVER2008R2-ENT-64 -snapshot-SP1” -dmp C:\VM\memory.dmp

Create a dump file using local symbols:

vm2dmp.exe -bin C:\VM\example.bin -vsv C:\VM\example.vsv -dmp C:\VM\memory.dmp –sym C:\symbols

For more information download the vm2dmp guide here: http://archive.msdn.microsoft.com/vm2dmp/Release/ProjectReleases.aspx?ReleaseId=3866

SCOM 2012

System Center Operations manager allows you to manage your network and give some detailed information into your environment in a central location. It supports Windows/Linux and SNMP devices alike. It allows options from an installed agent or agentless monitoring for information.

It has plenty of powerful rules that integrate into active directory and auto discovery rules to find new devices in the environment. This management server allows a centralized event log collection instead of taking up critical space on production systems. It can alert you via text or email if an event occurs based on custom rules. It even provides infrastructure information for System Administrators on descriptions, possible resolutions, and common causes.

I feel like it’s a more advanced Splunk that allows you target information with custom dashboards. I will not go into all the details because it can almost do anything and I provided some links to check out if you are interested.

For more information on SCOM check out these websites

http://blogs.technet.com/b/kevinholman/archive/2011/07/26/deploying-opsmgr-2012-a-quick-start-guide.aspx

http://www.scom2k7.com/installing-the-scom-2012-web-console-prerequisites-the-easy-way/

http://blogs.technet.com/b/kevinholman/archive/2011/07/26/deploying-opsmgr-2012-a-quick-start-guide.aspx

http://www.scom2k7.com/installing-the-scom-2012-web-console-prerequisites-the-easy-way/

http://technet.microsoft.com/library/hh278852.aspx

http://blogs.technet.com/b/momteam/

http://www.windowsnetworking.com/articles_tutorials/Introduction-System-Center-Operations-Manager-2012-Part1.html

http://technet.microsoft.com/library/hh278852.aspx

http://blogs.technet.com/b/momteam/

http://www.windowsnetworking.com/articles_tutorials/Introduction-System-Center-Operations-Manager-2012-Part1.html

 

VMM 2012

Virtual Machine Manger simply allows you to deploy fully updated virtual machines to your Infrastructure that follow a “Best Practice” standard with repeatable processes to avoid human mistakes. Even the simplest detail can be catastrophic for security protections in a complex environment. This also allows you to save time for the infrastructure guys allowing them to focus on much more important tasks.

I would love to go into more detail but I feel this post has covered enough for the day. Please feel free to leave any questions below or email me and I will respond as quickly as possible. Constructive criticism is always desired so please don’t hesitate.

I would to thank Andrew Case and Michael Hale Ligh for some proof reading and great suggestions. They did more than I expected as my first version was a little rough.

I want to review some notes from another previous Digital Forensics challenge. I will not present anything in here that no one hasn’t seen somewhere else and this is NOT  A REAL FORENSICS investigation and nor am I a real forensics expert or professional. I’m a student learning providing some of the very little I know.

I am only putting up notes if you actually want the reports and evidence files I used, please just ask and I would be glad to put them up. Honestly I have never done anything with volatility up until this point and this was the first rabbit hole I really went down in the field of forensics. I have played with a lot of forensics tools by I am by no means an expert. This is also stuff from last November.

These commands are for some people who want to get started with some really awesome memory forensics.

Using Volatility 2.0 inside a Backtrack 5 Virtual machine Run by VMWare Workstation 8  I ran the following commands  to obtain software information like running processes, dll, connections, and sid information and exported the information to report text files.

root@bt:~/Desktop/volatility-2.0#python vol.py psscan -f /root/Windows-XP-Professional.vmem –output-file=Report_Psscan.txt

root@bt:~/Desktop/volatility-2.0# python vol.py pslist -f /root/Windows-XP-Professional.vmem –output-file=Report_Pslist.txt

root@bt:~/Desktop/volatility-2.0# python vol.py connections -f /root/Windows-XP-Professional.vmem –output-file=Report_Connections.txt

root@bt:~/Desktop/volatility-2.0# python vol.py dlllist -f /root/Windows-XP-Professional.vmem –output-file=Report_Dlllist.txt

root@bt:~/Desktop/volatility-2.0# python vol.py getsids -f /root/Windows-XP-Professional.vmem –output-file=Report_getsids.txt

 

 

Then I used Volatility 1.3 with updated plugins and the following software in Volatility

Imaging-1.1.7

Inline-0.48_01

libdasm-1.5

Inline-Python-0.39

Pycrypo-2.0.x

Yara-1.4

Yara-python-1.4a

Pefile-1.2.10-63

volreg-0.6.tar.gz

volrip-0.1.tar.gz

by running the get_plugins.bsh script Author: Jamie Levy (gleeda)

Distorm3

Registry Ripper

 

root@bt:/Volatility# python volatility psscan2 -d -f /root/Windows-XP-Professional.vmem > /root/psscan2.dot

-psscan2.dot provides a processtree image in doty format to help understand running processes.

root@bt:/Volatility# python volatility screenshot -f /root/Windows-XP-Professional.vmem

Saving screenshot to 600.858ecda8.png

-screenshot plugin allows the creation of a screenshot for what the computer screen looked like at the time of the capture

 

python volatility hivelist -o 107563872 -f /root/Windows-XP-Professional.vmem > /203Report/Hivelist-Report.txt

–          The hivelist command locates where the registry hive files reside in memory so we can run registry ripper against them.

 

The following commands where used to export system hive file reports with reg ripper against the provided system hive files because his program can examine registry hive files and export evidence reports into easy to read text format documents.

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe1cc2008 -f ntuser > /203Report/NTUSER-dat-01-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe10ea820 -f ntuser > /203Report/NTUSER-dat-02-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe10aab60 -f ntuser > /203Report/NTUSER-dat-03-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe15a3a80 -f software > /203Report/software-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe1580448 -f sam > /203Report/Sam-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe15834b8 -f security > /203Report/Security-Report.txt

perl rip.pl -r /root/Windows-XP-Professional.vmem@0xe1035b60 -f system > /203Report/System-Report.txt

 

root@bt:/Volatility# python volatility window_list -f /root/Windows-XP-Professional.vmem > /203Report/Windows_List-Report.txt

–          The windw_list command allows us to export everything that was on the users screen at the point and time of capture of the memory image that answers the windows list question in the challenge.

I ran a lot of other stuff just looking around but this is some of the cool parts I wanted to highlight. Maybe I will put some more stuff out after I do a list of other things I want to do.

 

 

 

Tool Information

 

Cool Links to check out with more details and the real goods.

Type      Name    Publisher

Open Source      Volatility 2.0 + Volatility 1.3          Google Code

Site: http://code.google.com/p/volatility/

 

Type      Name    Publisher

Open Source      Backtrack 5         Backtrack Linux

Site: http://www.backtrack-linux.org/backtrack/backtrack-5-release/

 

Type      Name    Publisher

Commercial       VMWare Workstation 8                VMware

Site: http://www.vmware.com/products/workstation/overview.html

 

 

Type      Name    Publisher

Open Source      Imaging-1.1.7

Fredrik Lundh

Site: http://effbot.org/downloads/

 

Type      Name    Publisher

Open Source     Inline-0.48_01

Brian Ingerson

 

Site: http://search.cpan.org/~sisyphus/Inline-0.48_01/

 

 

Type      Name    Publisher

Open Source      Inline-Python-0.39

Ange Albertini – Google Code

Site: http://code.google.com/p/libdasm/updates/list

 

Type      Name    Publisher

Open Source     Pycrypo-2.0.1

A.M. Kuchling

Site: http://www.amk.ca/python/code/crypto.html

 

Type      Name    Publisher

Commercial       Yara-1.4

Google Code – Victor Manuel Alvarez

Site: http://code.google.com/p/yara-project/downloads/list

 

 

Type      Name    Publisher

Open Source     Yara-python-1.4a

Google Code – Victor Manuel Alvarez

Site: http://code.google.com/p/yara-project/downloads/list

 

Type      Name    Publisher

Open Source     Pefile-1.2.10-63

Ero Carrera

Site: http://code.google.com/p/pefile/

 

 

Type      Name    Publisher

Open Source     volreg-0.6.tar.gz

PUSH THE RED BUTTON – MOYIX

Site: http://www.cc.gatech.edu/~brendan/volatility/

 

 

Type      Name    Publisher

Open Source     volrip-0.1.tar.gz

PUSH THE RED BUTTON – MOYIX

Site: http://www.cc.gatech.edu/~brendan/volatility/

 

 

 

Type      Name    Publisher

Open Source     Distorm3

Google Code – Gil Dabah

Site: http://code.google.com/p/distorm/

 

Type      Name    Publisher

Open Source      Registry Ripper v2.02      Harlan Carvey

Site: http://regripper.wordpress.com/program-files/

 

 

Screenshot Plugin output of the challenge