Mastering AWS Cost Management with QuickSight: A Comprehensive Workflow from Data Export to Dashboard Insights!

Challenges with Existing Cost Explorer

AWS Cost Explorer provides resource-based cost insights, but it has a limitation: it only allows viewing cost data for individual resources for the last 14 days. If cost trends need to be analyzed over a longer period and specific resources contributing to cost increases beyond two weeks need to be identified, a more advanced solution is required. AWS QuickSight, in combination with AWS Cost and Usage Report (CUR) and Athena, enables the creation of powerful dashboards for in-depth cost analysis.

Objective

The goal is to create cost and usage dashboards in QuickSight to obtain a detailed overview of the services used in all AWS accounts within the AWS Organization. This will enable analysis of cost increases in specific services based on resource ID and usage type.

Solution

Description

1. AWS Data Export is configured to push AWS Cost and Usage Report (CUR) to an S3 bucket.
2. AWS Glue Crawler is set up to crawl the CUR from the S3 bucket and store it in its database and tables, which Athena then uses for querying.
3. AWS QuickSight uses Athena as a data source to execute queries, analyze data, and publish insights on dashboards.

Setup

1. Data Export
AWS enables the extraction of CUR data from an AWS account and stores it in an S3 bucket for processing and analysis.

Configuration Details

a. Granularity: Data is exported with hourly granularity to allow detailed cost analysis and spike detection.
b. Automatic Refresh: AWS updates the data daily into the S3 bucket at the specified export path.

Folder Structure

a. AWS creates a data folder within the specified S3 path.
b. Inside the data folder, separate sub-folders are created for each month.
c. The data for each month is refreshed daily in its respective sub-folder, ensuring up-to-date information is available.

2. Crawler Configuration
AWS Glue Crawlers are used to scan the S3 bucket, identify Parquet or Gzip files, and update the AWS Glue Data Catalog for efficient querying.

Configuration Details

a. The crawler runs daily at 4:30 AM UTC(Can be run anytime), ensuring the latest data from S3 is available in Athena.
b. After execution, the updated data is accessible in Athena for querying and analysis.

b

3. QuickSight Data Source Setup
AWS QuickSight connects to Athena, enabling direct querying and visualization of cost data for actionable insights.

Configuration Details

a. Athena is set up as the primary data source in QuickSight.
b. A custom query is created in QuickSight’s dataset to fetch required data.
c. The dataset query is scheduled to run daily at 6:00 AM UTC, ensuring the latest data is available for dashboard updates.

4. Analysis and Dashboard Creation
QuickSight Analyses transform raw data into interactive dashboards and reports, allowing users to explore cost trends based on usage type, resource, and linked accounts.

Setup Details

1. A QuickSight Analysis is created to calculate service costs based on linked accounts, usage type, and resource ID.
2. Interactive dashboards are built based on use cases and shared with end users.

Usage Based Dashboards:

Resource Based Dashboards:

What and Why to Use Amazon Q in QuickSight?

Amazon Q is an advanced Business Intelligence (BI) tool in QuickSight that enhances data analysis with Natural Language Processing (NLP) and generative AI capabilities. It allows users to interact with data intuitively by asking questions in plain English, generating insights, and creating data-driven stories effortlessly.

Capabilities of Amazon Q in QuickSight

1. Build dashboards using natural language – Users can create and modify dashboards by simply describing what they need.
2. Create Amazon Q Topics – Define specific topics for in-depth data analysis.
3. View executive dashboard summaries – Quickly generate summaries for high-level insights.
4. Build and share generative data stories – Automatically create narratives based on data trends.

Steps to Enable Amazon Q in QuickSight

1. The Admin plan in QuickSight includes only the Q&A capability of Amazon Q.
2. To access advanced features such as AI-driven dashboard creation and executive summaries, users must be added to the Admin Pro group.
3. Once added to the Admin Pro group, all users in that group gain full access to Amazon Q features, ensuring that only those who need advanced BI capabilities have access, optimizing cost management.

Conclusion

By integrating AWS CUR, Glue, Athena, and QuickSight, a scalable and automated cost analysis solution is established, overcoming the limitations of AWS Cost Explorer. The implementation of Amazon Q further enhances the analytical experience, enabling deeper insights and AI-driven storytelling. This setup empowers cost trend monitoring effectively and facilitates proactive AWS spending optimization.