Penny Pincher – Cloud Resource Optimization recommendation based on historical data to save cost.

With growing businesses in Amazon Web Services (AWS), finding idle resources, auditing your costs and managing your inventory are some of the most complex use-cases. This is particularly true for businesses or organizations that utilize a wide range of AWS services and resources, as keeping track of all these or idle resources can quickly become overwhelming.

Challenges:

1. Finding idle resources: Idle resources are the resources that are deployed but are not being used at a significant level hence are being underutilized. These idle resources can include EC2 instances, RDS databases, load balancers, elasticate, OpenSearch, and other resources. This results in unnecessary costs and contributes to overspending on AWS.
2. Cost Auditing: Another challenge is auditing cost usage. AWS offers a wide range of services with different pricing models, making tracking costs across different services and regions challenging. Organizations must regularly audit their AWS usage to identify cost inefficiencies and optimize their spending.
3. Extracting inventory: There is often a need to look at the AWS inventory and go through the complete list of resources. This can be needed for various reasons – identifying resources that are not needed, categorizing different types of workloads, forecasting, optimization, or further processing and analysis by feeding into a different system.
4. Lack of DevOps expertise: All these problems become even more challenging when you don’t have DevOps expertise in the team. To fulfill the need, you have to invest some time and effort in writing and testing the solutions to make them work to a satisfactory level.

Why existing solutions to the above problems don’t suffice the needs:

Let’s identify why existing solutions fail to solve the above problems completely.

1. Identification of idle resources

• One of the existing solutions AWS provides is Trusted Advisor, an AWS service offering recommendations to optimize resources, increase security, cut costs, and improve the performance of AWS resources. However, there are still some challenges with this system. Trusted Advisor requires an active AWS Support subscription, which may not be feasible for all organizations. Additionally, while Trusted Advisor can help identify idle resources, it has standard and fixed criteria around why the resource is idle. Moreover, implementing a custom rule on any custom condition is beyond a Trusted advisor’s scope.
• Sometimes you have your definition of idle resources against which you want to traverse your resources to check which meets your criteria. For example, a resource that is idle for one team or department may not be idle for another team. E.g., 100 hits per week on the load balancer might be idle of department A but not B. Therefore, it is essential to have a tool that allows for customization of the criteria used to identify idle resources. This tool should enable organizations to define their own thresholds for idle resources based on their unique business requirements.

2. Extracting inventory of resources

• Similarly, extracting the complete inventory of an AWS account is not a very straightforward solution in AWS. You have to either use Systems Manager, AWS Glue, and Crawl S3 bucket to configure some operations on raw files to reach the subset of the final inventory. There are some third-party tools also available e.g. on site24x7 which might not be allowed to configure in the AWS account for this purpose.

To fulfill the above-explained requirements or use-cases, we have developed a python solution ‘PennyPincher’, that is fully configurable.

Lets understand ‘Penny Pincher’ in brief

Penny Pincher is an open-source, cost-saving tool that can scan your whole AWS account, helps to identify all the resources that are provisioned but not being used. It not only generates reports of idle resources but all resources. It notifies you by sending generated reports via email or Slack and also stores it in s3 as per the user’s choice.

Note – By default, it updates the final report after every two weeks. 

Penny Pincher currently supports the following services:

The tool crawls all the regions to get the findings for the following supported services-

1. EBS
2. EC2
3. EIP
4. Elasticache
5. Elasticsearch
6. Loadbalancer
7. Redshift
8. RDS

Features of Penny Pincher

• Customizable

By default, the solution is intelligent enough to identify whether a resource is idle. But you can customize the conditions against which it reports the idle resources and their potential cost saving.

• Reports Potential cost saving 

When we say that something has the potential for cost savings, it means that it has the ability to help reduce expenses or expenditures.PennyPIncher gives you recommendations and suggestions for the same.

• Inventory Extraction

PennyPincher can provide the inventory scanned in your AWS account and generate a CSV file. In simpler terms, an inventory of an AWS account is essentially a comprehensive list of all the resources within that account, it enables you to have complete visibility over your resources, making it easier to manage and optimize them.

• Multiregion support

Penny Pincher can scan resources across all the regions in your AWS account and reports findings on the basis of that. Manually scanning your AWS environment can be time-consuming and error-prone.

• Reporting over Email, Slack, and S3 bucket

It can notify the summary of idle resources, potential cost savings, and inventory on Slack, over email, or s3 as per the user’s choice.

• Install once and forget

You can configure this solution as a corn, and it will timely report the findings over your chosen communication channel. This removes the overhead of running the tool again and again.

We have tested the solution for up to 300 resources in the AWS account.

Go through the technical documentation and installation steps on our Pennypincher Github page

A few considerations while using this tool:

1. Currently, the solution needs to be deployed separately in every account.
2. The solution needs to be deployed in the us-east-1 region
3. We have tested the solution for 300 resources.
4. In an account with more than 300 resources, you can run it locally, or we will roll out fargate support in the next version.

I want to thank my colleagues, Yash Khandelwal and Akshay Shrivastava, for providing constant support and help throughout building this solution.

Welcoming suggestions and enhancements.