How do you know if your AMIs are secure?

Do you want your Development teams to use a Linux AMI (Amazon Machine Image) offered in the Amazon Market Place published by Joe Smith that could have malware embedded in it?

Most of the “big” firms do not use AMIs (Amazon Machine Images) available in the Amazon Market Place. The “big” firms do not allow use of publicly available AMIs and build their own AMIs, with their own security stack embedded in the AMIs.

But, how do you know the AMIs are being built with the right security stack and the OS at the right patch level? You need to test the AMIs as best as you can before you make them available for your Development teams to use.

Couple of simple options could be to:

  1. Write server spec test cases to test the AMIs meet your specifications
  2. Implement some sort of static analysis on the AMIs. But, currently I am not aware of any products that can do this.

But, these are not effective and there is no way to know if the test cases are written and the AMIs are safe to use.

How about you deploy the AMIs  in a lower SysLevel and run scanning tools against a EC2 instance that uses the AMI? So, before you promote the AMIs for general use, have a private test VPC where you deploy an EC2 instance with the AMI and run your scanning tools such as Amazon Inspector, Qualys etc., to make sure that the OS is at the right patch level. This gives your better results and the confidence that the AMI is safe to use.

Once you are confident that AMIs are safe to use, promote the AMIs for use by Development teams. You can rely upon tagging to indicate what AMIs are available for production use.

Now that you have made a safe AMI available for Development teams to use, how do you know they continue to be safe, given that the threat environment changes and new attacks emerge constantly.

You need to create a farm of EC2 instance with AMIs that are currently in use in production and constantly scan them on a schedule basis. This allows you to be on top of new threats. If a vulnerability is detected, you need to have line of sight in to all the EC2 instance that are using that vulnerable image. After you identify all the EC2 instances, you have couple of options:

  1. Patch the EC2 instance in place
  2. Re-hydrate with EC2 instances that have the patched AMIs

The options you pick depends on your environment.

Now that we considered new AMI development and ongoing scanning, the EC2 server farm that you setup for scheduled scans also allows you to verify zero day vulnerabilities.

Building this automation for scanning your AMIs and promoting safe AMIs to use, will allow your vulnerability scanning to be effective for cloud native infrastructure. Without this option, you scan all available instances in the cloud and the instances that you scan might be part of auto scale and you will never have an accurate way to map the vulnerability back to a specific instance that need to be patched.

Having a server farm with all the AMIs that are currently promoted to be used by the development teams, you make the scanning pretty effective and you can take the results and map it back to all the EC2 instances currently using the vulnerable AMIs and limits the number of servers you need to scan and use your scanning licences in a cost effective manner.

Please share your approach to making AMIs safe for your development teams to use.


Do you know what your attacks are?

Do you know what your attacks are so that you can validate that the defensive controls that you are building are really helping keep your applications safe?

In most of the organizations, the attacks detected by SOC (Security Operations Center) never make it to the Development teams or even their counterparts in SSG (Software Security Group) that work with development teams to fortify their application defensive controls. In my opinion this leaves a big gap between the actual attacks and the defenses that SSG and Application teams are building. This leaves organizations susceptible as they may not be fortifying defenses where they need to. Does this sound like what happens in your organization?

In the absence of real information on the attacks seen by the organizations, the SSG teams rely upon OWASP Top 10 Categories to prioritize application defenses to build out. Although this better than nothing, the OWASP Top 10 are based on the vulnerabilities found in the applications based on the penetration testing done by security firms that participate in the OWASP Top 10 projects. For the most part, the Top 10 vulns seems like they align with High Value rewards the attackers go after. But, there is no validation or data supporting this that I could put my hands on.

So, why does the SOC team hesitant to share the attack patterns they detect and respond with the SSG and Development teams. If you work in SOC, I would like to hear from you. Please drop in a note.

In the meantime, OWASP Top 10 is the best we have to prioritize defensive controls to fortify applications. So, long !