Security Incident: Be Prepared – Memory Dumps

25/11/2018
Posted in AWS Blog
25/11/2018 Koenraad de Boevé

Memory Dumps

You just finished setting up your super-duper AWS environment..

Highly available & Fault Tolerant: check!
Backups in place: check!
MFA enforced: check!
Security Groups and NACLs: check!
CloudTrail enabled: check!

You even deserve bonus points for activating Amazon GuardDuty and putting AWS WAF & Shield in front of your CloudFront distribution and loadbalancers.
Time to lean back with a bit of smugness, while you take a sip of your well-deserved cup of coffee.

Seems like you have covered all your bases, or have you?

One often overlooked topic is security incident response.
While a lot of security incidents can (and should) be mitigated through automation,
some incidents will require manual intervention, such as information and evidence gathering during and after a successful malicious attack on one of your instances.

Effective incident response and forensics require preparation, well ahead of time.
It is critical to have your forensics and remediation tools readily available for whenever the proverbial shit hits the fan.
Documenting your investigative steps and being able to execute them swiftly, will contribute to a well-controlled, thorough and effective investigation.

In this blog post, I’d like to focus on some of the first steps you might take in your investigation:

  • Building a forensics workstation and  taking a memory dump of a compromised instance.
  • Preparation steps and tools, both for windows and linux.
  • The forensics investigation process.
  • An investigation of a real memory dump.

Although some great tools are already available from Threadresponse (such as AWS_IR and Margarita Shotgun), we will build a solution ourselves, in order to gain a thorough knowledge on how the process works behind the scenes.
Creating a memory dump is  something that  should be done immediately, as it provides a snapshot of the memory at the time of the attack.
The dump can then be analyzed and used to build a timeline, improve security after the fact and optionally provide evidence for any follow up with law enforcement.

Tools

  • Linux: LiME ( Linux Memory Extractor )
  • Windows: FTK or Belkasoft Live  RamCapturer
    ( both are free, but require registration, you get the download link in a mail )
  • Volatility

Preparation Steps

Create Quarantine and Forensic Security Groups

  1. Forensics Security Group.
    This SG will be attached to your Forensics Workstation later on.

    This will output something like this:

    Now we use this GroupId to add an inbound rule that allows  SSH  connection towards our Forensics Workstation

    By default , outbound everything is allowed, and in this case, that is ok, so we leave it at that.
  2. Quarantine Security Group
    This SG will be attached to the compromised instance, to isolate it from any network, except the forensics network.

    Output will look similar to the following:

    Remove all rules from outbound ( egress )

    Add rules to allow access from the Forensics Security Group
  3. Create Isolation Functionality
    Create lambda-execution-forensics-trust-policy.json file with following content:

    Create the Lambda Execution role, forensics-lambda-exec-role, using the policy file you just created:

    The output should look like this:

    Create lambda-forensics-policy.json with following content:

    Create the policy:

    This will output something like this:

    Attach the policy to the forensics-lambda-exec-role, using the ARN found in the output from the previous comand

    Create index.py with following content:

    This function will replace all Security Groups on all attached network interfaces of an instance with the Quarantine Security Group we created earlier. When you discover that an instance is compromised, you can invoke this function, and it will isolate the instance.  It fetches the Security Group from the environment, and takes the instance id of the compromised instance as input as json in the following format:

    The function will output the removed Security Groups so you have a reference for later on.

    zip the file to index.zip

    Create the Lambda function, using the IAM Policy ARN from the output of the role creation (forensics-lambda-exec-role)

    OK, Done, now let s move on and create a forensics workstation.

Build a Forensics Workstation

  1. Create EC2 Instance Role for the forensics Workstation
    Create ec2-forensics-trust-policy.json file with following content:

    Create the EC2 Instance role, ec2-forensics-role, using the Policy file you just created:

    Output:

    Create ec2-forensics-policy.json with following content:

    Create the ec2-forensics-policy IAM Policy:

    Attach the policy to the ec2-forensics-role, using the ARN found in the output from the previous command.
    The Forensics Workstation also needs access to S3 and AWS Systems Manager, so we also include some AWS Managed Policies

    Create an Instance Profile, called ec2-forensics-profile

    Attach the ec2-forensics-role to the ec2-forensics-profile

  2. Provision the Forensics Workstation
    Create user-data.txt script with following content:

    Create the EC2 Instance

    Tip: You can now fetch the latest Amazon AMI by querying the SSM Parameter Store:

    We now have a basic workstation.

  3. Create a forensic volume
    We will need a volume we can attach/detach from any instance to temporarily store memory dumps and provide tools to the compromised instance.
    Since we need to support both Windows and Linux, we create an exfat filesystem on it, as it is supported on all platforms.

    This creates a 100 GB volume. You can lower the size , but make sure that it is comfortably bigger then the largest ram size on any of your instances.

    Connect to the Forensics Workstation and attach the new volume ( you probably need to run aws configure to set your default region first )

    Partition and format the disk

    Mounting the volume

    Copy your local LiME and Volatility folders to the volume, to be able to compile a kernel module or create a Volatility profile on the fly, just in case.
    For Windows, we copy  RamCapturer or/and FTK Imager to the volume as well.

    Once you have all tools on the volume, unmount it, detach it and make a snapshot.

    Next step, we will gather LiME kernel modules and create Volatility Profiles for instances, running in our environment.
    The LiME kernel module needs to be loaded on the compromised instance

Create LiME kernel modules and Volatility Profiles

Note that you need to compile the kernel module for the EXACT kernel version , running on your instances. If you don’t patch your systems and/or have a lot of different Linux flavors, you will have a hard time maintaining the lime kernel modules and volatility profiles.

You have 2 options in obtaining a lime kernel module matching your kernel:

  1. threadresponse lime module repository
    https://threatresponse-lime-modules.s3.amazonaws.com/
    This opens an xmlfile in which the available lime modules can be found
    Here is an excerpt of that xml file.

    The reference to the kernel modules  ( in the above example on line 8 ) is  the s3 object key to the object.
    So if your kernel is 2.6.32-131.0.15.el6, you can download it from https://threatresponse-lime-modules.s3.amazonaws.com/modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko
  2. build your own repository
    If you cannot find the lime modules for kernels running in your environment, you can build your own.
    Install volatility and LiME on either an existing instance, or launch a new instance of  the Linux distribution for which you want to create a module and volatility profile.For Redhat/CentOS/ Amazon Linux, you can use the userdata from the Forensics Workstation (2 .Provision the Forensics Workstation)
    For Debian/Ubuntu you can use this user-data content:

    1. Compile the LiME module
      First, Make sure you have kernel headers, source and images installed on this instance.

      This should have created a file called lime-$(uname -r).ko
      If you need to create a module for a different kernel version ( for example for an older unpatched instance ),
      install the required version and change the above command to:

      Note that this does not require the targeted kernel version to be active
    2. Create Volatility Profiles
    3. Copy the lime module (.ko file) and volatility profile  (.zip file) to an S3 Bucket that acts as a central repository.

Incident Workflow

  1. Logon onto the Forensics WorkStation
    ssh into your Forensics Workstation in 2 terminal windows. ( Use ssh-agent, ssh-add and ssh -A)
  2. Isolate the compromised instance
  3. Create a snapshot of the compromised instance
  4. Attach the Forensics Volume to the compromised EC2 Instance and mount it.
    On The Forensics Workstation:

    On the compromised EC2 Instance:

    Linux:
    ssh into the compromised instance, fetch the kernel version and mount the volume.
    We need the kernel version for the next steps

    Windows:
    RDP into the compromised instance ( you need to setup an SSH Tunnel via the Forensics Workstation )
    Mount the volume via Computer Management -> Disk Management

    The new Volume is attached and identified as Disk 1 , but it is in an offline state.
    Right-click on Disk 1 and select ‘Online’

    The Volume is now online and mapped to D:

    On Windows, you can now skip to Step 7

  5. In the other terminal ( Forensics WS ), fetch the lime kernel module from your S3 Bucket or the Threadresponse repository
  6. scp the LiME module to the compromised instance
  7. Run the memory dump on the compromised instance
    On Linux:

    This will create the memory dump file ram.lime and the digest file ram.sha1 on the forensics volume.

    On Windows:
    Open File Explorer, and go to D:\
    If  RamCapturer is not yet unzipped, unzip RamCapturer.zip first.
    Then run D:\RamCapturer\x64\RamCapturer.exe as Administrator


    Save the dump to D:\ and run ‘Capture!’
    The dump will be saved as YYYYMMDD.mem where YYYYMMDD is the current date.

  8. Fetch the memory dump onto the Forensics WorkStation
    unmount the Forensics Volume on Linux, or , on Windows, put it offline again using the Disk Management.
    Detach the volume from the compromised EC2 Instance and attach it back to the Forensics Workstation.
  9. Stop the Compromised instance.

Sample Results from Memory Dumps

Volatility requires a profile matching your kernel. For Windows this is already included , but for Linux , you might need the volatility profile to be imported into your volatility setup.
let’s first test if the profile for your kernel is already configured:

If your kernel is not listed, you can add it by copying the volatility profile ( created in section 2.2 Create Volatility Profile )
to the volatility/plugins/overlays/linux/ directory.
Rerunning the above command should show your added profile.

OK, Pièce de résistance: some results:

Fetching lsof for the linux memory dump:

List all established connections on Linux

List all open ports on Linux

List all processes on Linux

There are other interesting possibilities , to get an idea what you can query run volatility with the -h option

This will list all comands, available for linux memory dumps.
Note that some commands might not work, because they are not supported for a specific profile.

Process list on Windows

Open ports on Windows

Established connections on Windows

A whole range of other commands are supported, but remember, not all commands work under all circumstances, YMMV.

Now where is that coffee?

Share this AWSome post
, , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *