About cloud-scanner
Boavizta Cloud-scanner returns environmental impacts of your AWS Instances (EC2) and Storage usage.
graph LR;
inventory[ Account inventory]
api[Impacts from BoaviztaAPI]
metrics[Prometheus metrics]
json[JSON output]
inventory --> api
api --> metrics
api --> json
It combines real time inventory and usage data from your AWS account with Boavizta API to offer a view of your impacts on a given region.
- multi criteria: Primary Energy consumption (PE), Abiotic resource depletion potential (ADP), and Global Warming Potential (GWP)
- multi stage: separate impacts of ressources Usage and Manufacture
Estimations can be filtered by tags. It eases attribution to a specific server, environment, application or service.
Cloud-scanner can be used:
- from command line, to get an immediate view of your impacts š»
- as a metric server . You can use it to monitor and display real time impacts in a dashboard š
Cloud-scanner is an Open Source application maintained here: https://github.com/Boavizta/cloud-scanner.
How it works
Cloud scanner relies on cloud providers APIs to perform an inventory of your cloud resources. It collects information about usage (instance types, tags, CPU load or volume size).
This inventory is used to query Boavizta API which returns impact data.
Results are exposed as JSON or metrics.
Contributing
Cloud scanner is stable, but with limited functionality.
Development versions may already provide new functionalities that have not yet be included to a release. So have a look at the changelog and Issues on this repository.
And most importantly, do not hesitate to ask questions or share your feedback by opening an Issue or discussion on GitHub.
Quick start run CLI using docker š³
No installation needed, you will run a public docker image of cloud-scanner CLI.
Pre-requisites
- Docker.
- A working AWS account (and your AWS CLI profile already configured)
Run cloud-scanner cli
docker pull ghcr.io/boavizta/cloud-scanner-cli:latest
docker run -it ghcr.io/boavizta/cloud-scanner-cli:latest --help
# Ensure you have a working AWS profile setup locally (like you would do for AWS CLI)
# Note
# - we map local credentials on the container (-v)
# - we force a using 'myprofile' profile by setting the AWS_PROFILE environment variable with -e flag
# - the -it flag is optional, only purpose is to get colored output if any
# Just list instances
docker run -it -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e AWS_PROFILE='myprofile' ghcr.io/boavizta/cloud-scanner-cli:latest inventory
# List instances and standard impacts (for 10 hours of use)
docker run -it -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e AWS_PROFILE='myprofile' ghcr.io/boavizta/cloud-scanner-cli:latest estimate --use-duration-hours 10
# Serve metrics
# /!\ Note that we need to provide CA certificates and bind listen address to 0.0.0.0.
docker run -it -p 8000:8000 -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e ROCKET_ADDRESS=0.0.0.0 -e ROCKET_PORT=8000 -e AWS_PROFILE='myprofile' ghcr.io/boavizta/cloud-scanner-cli:latest serve
ā This method of passing credentials is not secure nor very practical. In a production setup on AWS, you should rather rely on the role of the instance that execute the container to manage authentication of the cli.
ā Running metric server in container require setting extra variables:
- to map AWS credentials
- to map SSL ca certificates
- and more importantly to configure Rocket to listen to 0.0.0.0 instead of default 127.0.0.1 (which is internal to the container. This is done with the extra variable:
ROCKET_ADDRESS=0.0.0.0
Alternatively you may build and use a local docker image, See build a local docker image
Quick start : display dashboard using docker-compose š³
Visualize the live impacts of you account in a dashboard.
No installation needed, you will run a public docker image of cloud-scanner CLI, Boavizta API, Prometheus and Grafana to get access to a demo dashboard..
All data remain local to your environment (this docker-compose stack uses a private instance of Boavizta API).
Pre-requisites
- Docker and docker-compose
- A working AWS account (and your AWS CLI profile already configured)
Run the demo dashboard
# Map your AWS credentials
export AWS_PROFILE=name-of-your-profile
# Start the stack (from the root of the repository)
docker-compose up
- Demo dashboard is exposed on http://localhost:3001
- Log in with user admin/admin
- Select the dashboard in the left menu.
ā The demo docker-compose is preconfigured to generate metrics only for the following regions:
- eu-west-1
- eu-west-3
- eu-central-1
- us-east-1
Refer to Generating metrics for additional regions to retrieve metrics for additional regions.
Additional info
- ā This docker-compose example is not intended for production deployment, but rather for quick testing.
- ports of all services are exposed.
- Grafana is served on http with default login.
- You may have to update the line mapping your AWS profile (Replace
AWS_PROFILE=${AWS_PROFILE}byAWS_PROFILE=the-real-name-of-your-profile) when using Podman. It seems that Podman compose does not map the variables of the environment to the containers. - In corporate environments, you may need to provide your certificates authorities certificates (
ca-certificates) to the cloud-scanner container (uncomment the mapping line in the docker-compose file). - For the demo, we deliberately set a short metrics scrapping interval (30 seconds). In a production environment, you may want to increase this metric scraping interval to reduce the API calls and volume of data. This scrapping period is set in the Prometheus configuration file.
Quick start - Run CLI using Cargo š¦
This quick start will show the impacts of your EC2 instances for 10 hours of use, by building a local version of the cloud-scanner CLI from source.
Prerequisites
- a Rust toolchain installed locally rustup.rs - The Rust toolchain installer
- A working AWS account (and you AWS CLI profile already configured)
- optional: jq to format json results
Compile an run CLI š»
- Clone the cloud-scanner repository
- build and run CLI
# Ensure you have a working AWS profile setup locally (like you would do for AWS CLI)
export AWS_PROFILE='<YOUR_PROFILE_NAME>'
# Get impacts of 10 hours of use (on your default account region)
cargo run estimate --use-duration-hours 10 | jq
# Get impacts of 10 hours of use, for a explicit region. Results as JSON
cargo run -- --aws-region eu-west-1 estimate --use-duration-hours 10 | jq
# Get impacts of 10 hours of use, for a explicit region. Results as Prometheus metrics
cargo run -- --aws-region eu-west-1 metrics --use-duration-hours 10
Use cloud-scanner as a serverless applicationā”
This will deploy the cloud-scanner inside your AWS account. You can use it to:
- scan the account to get json impacts (as you would do with the CLI)
- or
- get metrics (that you can scrape with prometheus or the monitoring system of your choice)
The application is build and deployed using the serverless framework (see serverless-design).
Prerequisites
The deployment creates an aws role configured to get sufficient permission to scan your resources without requesting end-user authentication.
- Nodejs installed locally
- An AWS account/profile with sufficient permissions to deploy lambda, configure API gateway, and create a role.
ā The deployment process was only tested on Linux.
Deploy the application ā”
# Install node the serverless framework and it's dependencies
npm i
export aws_profile = <my profile>
# Deploy the application to your AWS account
serverless deploy
Usage
Scan the account / region š»
This returns results in json format (see below, same as CLI)
https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/scan?hours_use_time=5&aws_region=eu-west-1
Use hours_use_time and aws_region parameters in the query to pass the values to the lambda.
Get Metrics š
https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/metrics?aws_region=eu-central-1
Returns metrics for 1 hour of use in prometheus format.
The metrics represent the costs / impacts of one hour of use of the resources present in your account at the time of the query.
Use aws_region parameters in the query.
# HELP boavizta_number_of_instances_total Number of instances detected during the scan.
# TYPE boavizta_number_of_instances_total gauge
boavizta_number_of_instances_total{awsregion="eu-central-1",country="DEU"} 7
# HELP boavizta_number_of_instances_assessed Number of instances that were considered in the measure.
# TYPE boavizta_number_of_instances_assessed gauge
boavizta_number_of_instances_assessed{awsregion="eu-central-1",country="DEU"} 5
# HELP boavizta_duration_of_use_hours Number of instances detected during the scan.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-central-1",country="DEU"} 1.0
# HELP boavizta_pe_manufacture_megajoules Power consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-central-1",country="DEU"} 1760.0
# HELP boavizta_pe_use_megajoules Power consumed during usage.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-central-1",country="DEU"} 0.86
# EOF
Methodology and source of data
Cloud scanner relies on the Boavizta methodology and data to estimate the impacts of cloud resources.
It performs an inventory of the resources used in a cloud account. It then returns impacts of the use phase but also impacts related to the manufacture of the identified resources (embodied impacts).
The source of impact data
Impact data is retrieved from BOAVIZTA reference data API v1.2.x.
General approach and limits
Cloud scanner estimates impacts according to the duration of use of cloud resources. It also considers the āintensityā of use (like the type of instance and CPU load for a VM or size of a disk for storage). The region of use is also considered to take into account the electricity mix of the region (carbon intensity).
Thibault Simon, David Ekchajzer, Adrien Berthelot, Eric Fourboul, Samuel Rince, et al.. BoaviztAPI: a bottom-up model to assess the environmental impacts of cloud services. HotCarbonā24. Workshop on Sustainable Computer Systems, Jul 2024, Santa Cruz, United States. āØhal-04621947v1ā©
The general approach is to map cloud vendor specific ressources, either to Boavizta cloud resources (like for VM instances) or to more generic components (Block storage volumes are approximated as Boavizta HDD or SSD).
But this approach means that a large portion of the cloud provider infrastructure is excluded from the estimation.
ā Cloud scanner underestimates the impacts of a cloud account. Because it only considers the instances and block storage, many additional sources of impacts (network, potential redundancy, cloud control plan, buildingsā¦) are not included in the estimation.
See also other limits.
Methodology
The Boavizta methodology is described in Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta
The impacts (use and embedded) are attributed according to the principles described in Cloud instances - Boavizta API documentation.
The results of cloud scanner are similar to what you can visualize in Datavizta, but with automated inventory.
- Boavizta website: https://www.boavizta.org/en
How we estimate instance workload
Workload (or intensity of use) of instances is estimated using CPU load level as a proxy.
Estimating instances workload for AWS
The CPU load of AWS instances is retrieved using AWS Cloudwatch CPU metrics summary.
Cloud scanner uses a sampling period of 15 minutes, but impacts metrics are returned as impacts equivalent to one hour of use.
This means that instance impacts metrics data returned can be understood as: impact for one hour of use (considering the CPU workload of 15 last minutes).
Why this default sampling period of 15 minutes ?
- It is sufficient for our current monitoring needs (but maybe we may make this setting configurable in the future).
- It is hard to go below 10 minutes. Default (and free) period of AWS instance metrics is 5 minutes. You need to activate
detailed monitoring(extra feature) for 1 minute granularity: List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud).
Estimating instances workload for Azure
Not implemented.
Estimating impacts of block storage
Block storage volumes are approximated either as SSD or HDD depending on cloud provider specific storage type.
This is a basic estimation. It takes into account the size of the block storage volume.
ā Only the manufacture impacts of the Block storage are returned. The impacts of the use phase are returned as zero (mainly because we lack the related data in current version of Boavizta API).
Block storage (EBS) estimation (AWS)
What is considered in estimation:
- storage type (HDD vs SSD)
- storage size
- duration of use
| AWS EBS volume type | Boavizta storage used for estimation | comments |
|---|---|---|
| st1 | HDD | |
| sc1 | HDD | |
| magnetic (standard) | HDD | |
| gp2 | SSD | |
| gp3 | SSD | |
| io1 | SSD | |
| io2 | SSD | |
| unknown type | SSD | If it cannot identify the type of storage (like when a new type is introduced, cloud scanner uses SSD as an approximation) |
What is not considered for the estimation
- If the storage offers provisioned IOPS or not.
- Implicit redundancy related to storage type.
- Any other infrastructure (network, backup, supervision or redundancy) that may allow the cloud provider to offer EBS service.
Block storage estimations (Azure)
Not yet implemented.
Get impacts of an existing inventory file
You can use cloud scanner to take a snapshot of an existing inventory and run estimation of impacts later.
Get an inventory file
From command line
# Save an inventory to file
cloud-scanner inventory > my_inventory.json
Using http server / API mode
# start server
cloud-scanner serve
# get the inventory
curl -X 'GET' \
'http://localhost:8000/inventory?aws_region=eu-west-3&include_block_storage=true' \
-H 'accept: application/json' > my_inventory.json
Get estimate for a existing inventory
From command line
You have to explicitly pass the duration of use for which you would like to retrieve the estimation.
# Get impacts of an existing inventory
cloud-scanner estimate --use-duration-hours 1 --inventory-file my_inventory.json
Using http server / API mode
You have to explicitly pass the duration of use for which you would like to retrieve the estimation.
# start server
cloud-scanner serve
# Post the previously saved inventory (my_inventory.json) for estimate
curl -X POST -H "Content-Type: application/json" -d @my_inventory.json http://localhost:8000/impacts-from-arbitrary-inventory\?use_duration_hours\=10\&verbose_output\=false
Get impacts of an arbitrary inventory
This is useful to simulate the impacts of a non-existing infrastructure or a variant of existing infrastructure.
This involves building an inventory file and passing it to cloud scanner for evaluation.
- describe an arbitrary inventory (the to-be infrastructure) as a json file.
- request cloud scanner to estimate impacts of this inventory.
š” It may be easier to adapt an existing inventory file, rather than creating it from scratch. See Estimate the impacts of an existing inventory.
The JSON schema of the inventory file is in the git repository: cloud-scanner/cloud-scanner-cli/test-data/INVENTORY_JSON_SCHEMA.json. This schema can also be retrieved with the command cargo run inventory --print-json-schema.
Building cloud-scanner CLI with Cargo š¦
On Linux
Install Rust and linux-musl dependencies.
# Install rust (see https://www.rust-lang.org/tools/install), validate when prompted
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Add Linux musl target (needed for cross compilation for aws lambda)
rustup target add x86_64-unknown-linux-musl
sudo apt update && sudo apt install -y musl-tools musl-dev
# Test a build
cargo build
# build a release
cargo build --release
On Windows with WSL2
Tested method to build Rust on Windows is to use Windows Subsystem For Linux (WSL2)
- Prerequisite: install WSL2 and the latest Ubuntu LTS image (22.04). If you do not WSL yet, you may inspire from Set up Node.js on WSL 2.
- After WSL is setup, follow Linux instructions above inside the Linux VM.
Using docker
See Build a local docker image
Usage with Docker š³
Build a local docker image
# Local build of docker image
docker build . --tag cloud-scanner-cli
# Test run
docker run -it cloud-scanner-cli --help
Run cloud-scanner as a docker image
Deploy cloud scanner as a serverless application
You can deploy cloud scanner as a serverless application (AWS lambda).
The application is build and deployed using the serverless framework (see serverless-design).
Using Linux
- Install Rust
- Install nvm, nodejs
- test serverless package
First, donāt forget to clone the repo using
git clone https://github.com/Boavizta/cloud-scanner.git
cd cloud-scanner
Install Rust
# Install rust (see https://www.rust-lang.org/tools/install), validate when prompted
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Add Linux musl target (needed for cross compilation for lambda)
rustup target add x86_64-unknown-linux-musl
sudo apt update && sudo apt install -y musl-tools musl-dev
Install node (for serverless deployment)
nvm, node.js and npm
See https://docs.microsoft.com/en-us/windows/dev-environment/javascript/nodejs-on-wsl
sudo apt-get install curl
# Install nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
# Install node
nvm install --lts
Install serverless framework and deps
npm install -g serverless
npm i
Configure environment
Everything you need should now be installed, but there is a last step you have to do before deploying. Since the serverless framework automatically deploys the lambda onto your AWS instance without you having to upload a zip file or anything, you have to give it access to it by giving it an access key (see this guide for more informations on how to generate an access keys).
serverless config credentials \
--provider aws \
--key YOURACCESSKEY \
--secret YOURSECRETKEY
Optionally, you can config a private instance of Boaviztapi by setting the environment variable BOAVIZTA_API_URL so :
export BOAVIZTA_API_URL="boaviztapi.example.com"
If the environment variable is not set, cloud scanner will use the public instance (https://api.boavizta.org) by default.
Deploy
You should be good to go by now, simply run
serverless package
serverless deploy
and wait for it to be done. You should by now see two lamda functions appear on your AWS instance, Congratulations !
If any error happen, redo those steps carefully and make sure you didnāt miss anything before posting a GitHub Issue.
Using Windows
Tested method to deploy this serverless app from on windows is to use Windows Subsystem For Linux (WSL2).
- Prerequisite: install WSL2 and the latest Ubuntu LTS image (22.04). If you do not have it yet, you may inspire from Set up Node.js on WSL 2.
- After WSL is setup, follow Linux instructions above inside the Linux VM.
AWS authentication
Cloud scanner uses AWS Rust SDK to query AWS account. By default Rust SDK picks up AWS credentials from environment variables.
AWS permissions required by Cloud Scanner
š„ An Important pre-requisite is to have a user with sufficient permissions to list resources of the account (or an instance profile / role pre-configured when running cloud-scanner from EC2).
The minimal set of permissions to perform inventory of resources (and query CPU load of instances) is:
- ec2:DescribeInstances
- cloudwatch:GetMetricStatistics
- cloudwatch:DescribeAlarm
You could also restricts permissions to a specific set of instances or resources.
Pass credentials as environment variables
Option 1: Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Set environment variables with your account detail.
# Example for Linux / macOS
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=eu-west-1
See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html for equivalent Windows command prompt or Powershell syntax example.
Option 2: Use a an existing AWS_PROFILE
If you have configured an AWS CLI profile, the easiest way to pass aws credential to cloud scanner is use an environment variable that points to this profile.
Pre-requisite to use a profile:
- AWS CLI installed: Installing or updating the latest version of the AWS CLI - AWS Command Line Interface
- AWS CLI configured: Configure the AWS CLI - AWS Command Line Interface.
# Example for Linux / macOS
# cloud-scanner can use the AWS_PROFILE set as en environment variable.
# You have to reference one of the profiles previously configured in ~/.aws/credentials
export AWS_PROFILE=YOUR_PROFILE_NAME_EXAMPLE
export AWS_DEFAULT_REGION=eu-west-3
See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html for equivalent Windows command prompt or Powershell syntax example.
Reference
See Specify your credentials and default Region - AWS SDK for Rust for more options to pass credentials to the SDK.
Integrate metrics to a monitoring dashboard
Cloud-scanner can be used to export metrics related to the impact of your cloud account.
The metrics are exposed in Prometheus/OpenMetrics format. You can use it to feed a monitoring dashboard and keep an eye on the evolution of impacts of your usage.
Overview
- Start cloud-scanner in metrics mode (using the
--serveoption or by deploying it as a serverless application). - Setup Prometheus to scrape the metrics. Metrics are generated separately for each region.
- Configure a dashboard to display the results.
Detailed steps
Demo for some EU regions.
You can refer to the provided docker compose example for a quick start.
It comes preconfigured to generate metrics for the following regions:
- eu-west-1
- eu-west-3
- eu-central-1
- us-east-1
Generating metrics for additional regions.
Examples of Prometheus and Grafana configuration files are located in the dashboard-config directories.
To generate (and display) metrics for an additional region:
- Edit the prometheus configuration (
prometheus.yml) - Add a new job and set its region parameter like
aws_region: ['us-east-2']. - Restart Prometheus container and verify that metrics are now collected for this region.
In the exemple compose stack, prometheus admin GUI is exposed. You can check status of the scrape targets at: http://localhost:9090/targets.
# Extract from prometheus.yml
# Example extra job configuration to cover us-east-2
- job_name: 'cloud-scanner-metrics-us-east-2'
scheme: http
static_configs:
- targets: ['cloud_scanner_boa:8000']
labels:
group: 'production'
metrics_path: '/metrics'
params:
aws_region: ['us-east-2']
include_block_storage: ['true']
Adapting configuration for production use:
In production environment, you may want to increase the metrics scraping interval.
This It is set to 30 seconds in the sample prometheus configuration. In production you shoud increase this intervall (e.g. 10 minutes).
# Prometheus scrapping interval / extract from prometheus.yml configuration
global:
scrape_interval: 30s # By default, scrape targets every 30 seconds.
evaluation_interval: 30s # Evaluate rules every 30 seconds.
Filter by tags
You can limit cloud-scanner results (metriccs or json) to the cloud resources that match specific tags.
You can use multiple tags.
Works with inventory or estimates.
Filter in cli
# Retrieve only resources with the following tags:
# CustomTagNameForDebug=olivierdemeringoadm
# and
# Name=test-boavizta
cargo run -- -t Name=test-boavizta-2 -t CreatorName=olivierdemeringoadm inventory| jq
Filter (with serverless/lambda)
Use one (or multiple) filter_tag parameter in the query string to restrict results.
# Retrieve only resources with the following tags:
# CustomTagNameForDebug=olivierdemeringoadm
# and
# Name=test-boavizta
https://<EDITED>.execute-api.eu-west-1.amazonaws.com/dev/scan?use_duration_hours=5&aws_region=eu-west-1&filter_tag=CustomTagNameForDebug=olivierdemeringoadm&filter_tag=Name=test-boavizta
Filter metrics
Use the filter_tag query parameter in the URL.
It can be used multiple time (only instances matching all tags will be returned).
Example queries
- http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value
- http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest
- http://localhost:8000/metrics?aws_region=eu-west-1&filter_tag=Name=test-boavizta
Important limitation
Suppose the following instances (and tags)
-
instance1 (
- ENV=prod
- PURPOSE=CI
-
instance2
- ENV=prod
- PURPOSE=service
-
Filtering on āENV=prodā will return instance1 and instance2
-
Filtering on āENV=prodā and PURPOSE=CI will return only instance2
-
No filter will return all instances
Using a private instance of Boavizta API
To avoid that the scanner shares your inventory data to the public version of Boavizta API, It is recommended to deploy your own instance of Boavizta API.
This is also useful if you want to stick to a specific version of the API.
The Boavizta API can be deployed in several ways (docker, lambda a.s.o.). Refer to the API documentation: Deploy - Boavizta API documentation or refer to the provided docker compose example for an example of integration cloud-scanner and a private instance of the API.
Once you have deployed your instance of API, use either the command line flags (CLI) or environment variable (docker, lambda) to configure the scanner to use your API instance.
Frequently Asked Questions
What are differences between Datavizta and Cloud scanner ?
Both tools rely on the same source of impact data (i.e. the BoaviztAPI. The API is used to estimation the impacts of aws instances.
-
Datavizta is positioned as a pedagogical front end to the API. It helps users understand which kind of data the API, quickly test hypothesis. It offers visualizations of impacts of multiple indicators for different lifecycle steps (use vs manufacture).
-
Cloud scanner is more production oriented. It returns same data as Datavizta but in different form and with different inputs. The main objective of cloud scanner is to help quickly retrieve the impacts of an entire AWS account by automating the inventory of resource used and the retrieval of associated impacts.
In addition cloud scanner can be used as a monitoring system that runs regularly. In this later case, it generates results as metrics (in standard Prometheus / Open metrics format). Theses metrics can be consumed by external systems, to ease further analysis (e.g. breakdown by service / tags or countries) or use custom visualizations (like an external dashboard).
Which cloud provider or service is supported ?
AWS (instances and block storage).
- Only AWS is supported for the time being. The reason is that historically BoaviztAPI could only provide reference data for AWS instances.
- Azure instances are on the roadmap. After data becomes available in Boavizta API, cloud scanner will be adapted to support Azure https://github.com/Boavizta/cloud-scanner/issues/15.
- In addition BoaviztAPI is working on providing impacts of Storage as a service. It will be integrated to cloud scanner once available. https://github.com/Boavizta/boaviztapi/issues/143
Common issues
The demo dashboard does not show any metricā¦
Several reasons may explain why metrics do not appear in the dashboard.
- Verify in the docker-compose logs that there is no error related to AWS authentication.
- Check that prometheus is configured to retrieve metrics for the region you use.
ā The demo docker-compose is preconfigured to generate metrics only for the following regions:
- eu-west-1
- eu-west-3
- eu-central-1
- us-east-1
Refer to Generating metrics for additional regions to retrieve metrics for additional regions.
The docker compose demo may not display the latest version of the Grafana dashboard
/!\ The existing Grafana volume is preserved even after deploying a newer version of the docker-compose demo stack.
As a consequence the Grafana provisioning does not occur. Any update of the demo dashboard provided with the new stack is not reflected in the instance, even if Grafana container itself is recreated.
Workaround
We have 2 possibilities, depending if we want to preserve the existing Grafana configuration (volume) or not.
- Preserve the existing volume if it was previously customized to contain additional Grafana configuration (outside cloud scanner) or if you add custom dashboards.
- Delete the volume if you use a vanilla instance of cloud-scanner demo.
Updating the dashboard while keeping existing volume
Wee need to upload the new dashboard definition manually using the Grafana web UI.
- The dashboard definition can be found under
dashboard-config/provisioning/dashboards/: grafana-dashboard-cloud-impacts.template.json - By default Grafana UI is exposed at
http://<host-ip>:3001
Updating the dashboard by recreating Grafana configuration from scratch
/!\ Use this method only if Grafana was not customized after deployment of the demo or you are ok to loose these changes.
The easiest is to delete the Grafana volume and Grafana container. They will be recreated, and Grafana provisioned with latest dashboard, when we restart the docker compose stack.
# stop docker compose stack
docker compose down
# delete Grafana container (necessary because it may be using the volume even if stopped)
docker rm grafana_boa
# now delete volume
docker volume rm cloud-scanner_grafana-data
# Recreate compose stack
docker compose up
CLI options
List aws instances and their environmental impact (from Boavizta API)
Usage: cloud-scanner-cli [OPTIONS] <COMMAND>
Commands:
estimate Get estimation of impacts for a given usage duration as json
metrics Get estimation of impacts for a given usage duration as OpenMetrics (Prometheus) instead of json
inventory List resources (and instances average cpu load for the last 5 minutes) without returning impacts
serve Run as a standalone server. Access metrics (e.g. http://localhost:8000/metrics?aws_region=eu-west-3), inventory or impacts (see http://localhost:8000/swagger-ui)
help Print this message or the help of the given subcommand(s)
Options:
-a, --aws-region <AWS_REGION>
AWS region (The default aws profile region is used if not provided)
-b, --boavizta-api-url <BOAVIZTA_API_URL>
Optional Boavizta API URL if you want to use your own instance (URL without the trailing slash, e.g. https://api.boavizta.org)
-t, --filter-tags <FILTER_TAGS>
Filter instances on tags (like tag-key-1=val_1 tag-key_2=val2)
-v, --verbosity...
Enable logging and show execution duration, use multiple `v`s to increase logging level warning to debug
-h, --help
Print help
-V, --version
Print version
Experimental feature: estimate block storage
Use the --include-block-storage command line flag or parameter to consider block storage (either in inventory or when requesting an estimation of impacts.). This parameter defaults to false . This means that by default block storage (volumes) are not counted in the inventory nor in the results.
ā In any case, for storage, the impacts of the use phase are not counted. Boavizta API returns only the impacts of the manufacturing phase for HDD and SSD. Furthermore the impacts of storage are likely are likely overestimated. They are calculated by taking into account the size of the logical volume. In reality, a volume that is not full may not claim the full space on a physical device, which would result in smaller impacts.
# Experimental: get impacts of instances and attached storage
cargo run estimate --use-duration-hours 1 --include-block-storage --output-verbose-json
Print the JSON schema of the inventory file
# Print the JSON schema of an inventory file (without producing the inventory)
cargo run inventory --print-json-schema
Display statistics
Use -v will display statistics on std error.
- First output line shows the time spend specifically gathering CPU load of instances.
- Second line shows global statistics:
- total inventory duration: total time taken to retrieve resource lists and CPU statistics from AWS.
- impact estimation duration: time taken to query Boavizta API
- total execution duration.
cloud-scanner-cli -v estimate -u 1
# use -- -v with cargo
# cargo run -- -v estimate -u 1
[...]
cloud_scanner_cli::aws_inventory: Total time spend querying CPU load of instances: 372.153481ms
cloud_scanner_cli: ExecutionStatistics { inventory_duration: 911.526773ms, impact_duration: 398.993816ms, total_duration: 1.310520822s }
[...]
Environment variables
Cloud scanner uses environment variables to configure connection to your cloud account. See AWS authentication.
Format of results in server mode
When run in server mode (cloud-scanner-cli serve), Cloud-scanner exposes 3 endpoints:
/metrics: returns Prometheus metrics (plain text)/inventory: returns an inventory (json format, see schema below)/impacts: returns impacts (json format, see schema below)
Open API specification (Swagger)
To access the OpenAPI specification:
- start a server (
cloud-scanner-cli serve) - access the OpenAPI specification at http://127.0.0.1:8000/openapi.json and a swagger-ui at http://127.0.0.1:8000/swagger-ui/
The latest (up-to-date) version of OpenAPI specification is exposed under <BaseURL>/openapi.json path and displayed using swagger-ui at <BaseURL>/swagger-ui/index.html.
{
"openapi": "3.0.0",
"info": {
"title": "cloud-scanner-cli",
"version": "2.0.5"
},
"paths": {
"/metrics": {
"get": {
"tags": [
"metrics"
],
"summary": "Returns Prometheus metrics.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nResults are estimated for one hour of use by default.\n\nExample query: http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value&use_duration_hours=1.0&include_storage=true",
"operationId": "metrics",
"parameters": [
{
"name": "aws_region",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "filter_tags",
"in": "query",
"schema": {
"type": "array",
"items": {
"type": "string"
},
"nullable": true
}
},
{
"name": "use_duration_hours",
"in": "query",
"schema": {
"type": "number",
"format": "float",
"nullable": true
}
},
{
"name": "include_block_storage",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
}
],
"responses": {
"200": {
"description": "",
"content": {
"text/plain": {
"schema": {
"type": "string"
}
}
}
}
}
}
},
"/inventory": {
"get": {
"tags": [
"inventory"
],
"summary": "Returns current inventory.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/inventory?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value",
"operationId": "inventory",
"parameters": [
{
"name": "aws_region",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "filter_tags",
"in": "query",
"schema": {
"type": "array",
"items": {
"type": "string"
},
"nullable": true
}
},
{
"name": "include_block_storage",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
}
],
"responses": {
"200": {
"description": "",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Inventory"
}
}
}
}
}
}
},
"/impacts": {
"get": {
"tags": [
"impacts"
],
"summary": "Returns the impacts of current inventory.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/impacts?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value&use_duration_hours=1.0",
"operationId": "impacts",
"parameters": [
{
"name": "aws_region",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "filter_tags",
"in": "query",
"schema": {
"type": "array",
"items": {
"type": "string"
},
"nullable": true
}
},
{
"name": "use_duration_hours",
"in": "query",
"schema": {
"type": "number",
"format": "float",
"nullable": true
}
},
{
"name": "verbose_output",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
},
{
"name": "include_block_storage",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
}
],
"responses": {
"200": {
"description": "",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/EstimatedInventory"
}
}
}
}
}
}
},
"/impacts-from-arbitrary-inventory": {
"post": {
"tags": [
"impacts"
],
"summary": "Retrieve the impacts of arbitrary inventory.",
"description": "This can be used to evaluate impacts of a not yet implemented architecture.\n\nThe inventory is passed as json data in the request body.",
"operationId": "impacts_from_arbitrary_inventory",
"parameters": [
{
"name": "use_duration_hours",
"in": "query",
"schema": {
"type": "number",
"format": "float",
"nullable": true
}
},
{
"name": "verbose_output",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
}
],
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Inventory"
}
}
},
"required": true
},
"responses": {
"200": {
"description": "",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/EstimatedInventory"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"Inventory": {
"description": "Inventory: a list of resources",
"type": "object",
"required": [
"metadata",
"resources"
],
"properties": {
"metadata": {
"$ref": "#/components/schemas/InventoryMetadata"
},
"resources": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CloudResource"
}
},
"execution_statistics": {
"allOf": [
{
"$ref": "#/components/schemas/ExecutionStatistics"
}
],
"nullable": true
}
}
},
"InventoryMetadata": {
"description": "Details about the inventory",
"type": "object",
"properties": {
"inventory_date": {
"type": "string",
"format": "date-time",
"nullable": true
},
"description": {
"type": "string",
"nullable": true
}
}
},
"CloudResource": {
"description": "A cloud resource (could be an instance, function or any other resource)",
"type": "object",
"required": [
"id",
"location",
"provider",
"resource_details",
"tags"
],
"properties": {
"provider": {
"$ref": "#/components/schemas/CloudProvider"
},
"id": {
"type": "string"
},
"location": {
"$ref": "#/components/schemas/UsageLocation"
},
"resource_details": {
"$ref": "#/components/schemas/ResourceDetails"
},
"tags": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CloudResourceTag"
}
}
}
},
"CloudProvider": {
"type": "string",
"enum": [
"AWS",
"OVH"
]
},
"UsageLocation": {
"description": "The location where cloud resources are running.\n\nTODO! the usage location should be abstracted and vendor specific implementation should be part of the cloud_provider model (region names are tied to a specific cloud provider)",
"type": "object",
"required": [
"aws_region",
"iso_country_code"
],
"properties": {
"aws_region": {
"description": "The AWS region (like eu-west-1)",
"type": "string"
},
"iso_country_code": {
"description": "The 3-letters ISO country code corresponding to the country of the aws_region",
"type": "string"
}
}
},
"ResourceDetails": {
"oneOf": [
{
"type": "string",
"enum": [
"object_storage"
]
},
{
"type": "object",
"required": [
"instance"
],
"properties": {
"instance": {
"type": "object",
"required": [
"instance_type"
],
"properties": {
"instance_type": {
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/components/schemas/InstanceUsage"
}
],
"nullable": true
}
}
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"block_storage"
],
"properties": {
"block_storage": {
"type": "object",
"required": [
"storage_type"
],
"properties": {
"storage_type": {
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/components/schemas/StorageUsage"
}
],
"nullable": true
},
"attached_instances": {
"type": "array",
"items": {
"$ref": "#/components/schemas/StorageAttachment"
},
"nullable": true
}
}
}
},
"additionalProperties": false
}
]
},
"InstanceUsage": {
"type": "object",
"required": [
"average_cpu_load",
"state"
],
"properties": {
"average_cpu_load": {
"type": "number",
"format": "double"
},
"state": {
"$ref": "#/components/schemas/InstanceState"
}
}
},
"InstanceState": {
"type": "string",
"enum": [
"running",
"stopped"
]
},
"StorageUsage": {
"type": "object",
"required": [
"size_gb"
],
"properties": {
"size_gb": {
"type": "integer",
"format": "int32"
}
}
},
"StorageAttachment": {
"type": "object",
"required": [
"instance_id"
],
"properties": {
"instance_id": {
"type": "string"
}
}
},
"CloudResourceTag": {
"description": "A tag (just a mandatory key + optional value)",
"type": "object",
"required": [
"key"
],
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string",
"nullable": true
}
}
},
"ExecutionStatistics": {
"description": "Statistics about program execution",
"type": "object",
"required": [
"impact_estimation_duration",
"inventory_duration",
"total_duration"
],
"properties": {
"inventory_duration": {
"$ref": "#/components/schemas/Duration"
},
"impact_estimation_duration": {
"$ref": "#/components/schemas/Duration"
},
"total_duration": {
"$ref": "#/components/schemas/Duration"
}
}
},
"Duration": {
"type": "object",
"required": [
"nanos",
"secs"
],
"properties": {
"secs": {
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"nanos": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
"EstimatedInventory": {
"description": "An estimated inventory: impacting resources with their estimated impacts",
"type": "object",
"required": [
"impacting_resources"
],
"properties": {
"impacting_resources": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CloudResourceWithImpacts"
}
},
"execution_statistics": {
"allOf": [
{
"$ref": "#/components/schemas/ExecutionStatistics"
}
],
"nullable": true
}
}
},
"CloudResourceWithImpacts": {
"type": "object",
"required": [
"cloud_resource",
"impacts_duration_hours"
],
"properties": {
"cloud_resource": {
"$ref": "#/components/schemas/CloudResource"
},
"impacts_values": {
"description": "The impacts",
"allOf": [
{
"$ref": "#/components/schemas/ImpactsValues"
}
],
"nullable": true
},
"impacts_duration_hours": {
"description": "The duration for which impacts are calculated",
"type": "number",
"format": "float"
}
}
},
"ImpactsValues": {
"description": "Impacts of an individual resource",
"type": "object",
"required": [
"adp_manufacture_kgsbeq",
"adp_use_kgsbeq",
"gwp_manufacture_kgco2eq",
"gwp_use_kgco2eq",
"pe_manufacture_megajoules",
"pe_use_megajoules"
],
"properties": {
"adp_manufacture_kgsbeq": {
"type": "number",
"format": "double"
},
"adp_use_kgsbeq": {
"type": "number",
"format": "double"
},
"pe_manufacture_megajoules": {
"type": "number",
"format": "double"
},
"pe_use_megajoules": {
"type": "number",
"format": "double"
},
"gwp_manufacture_kgco2eq": {
"type": "number",
"format": "double"
},
"gwp_use_kgco2eq": {
"type": "number",
"format": "double"
},
"raw_data": {
"nullable": true
}
}
}
}
}
}
Output data
The inventory is always returned as JSON.
The estimated inventory (i.e. inventory with impacts data ) is either returned as json or as a set of Open Metrics (Prometheus).
Format of the estimated inventory as JSON
Cloud scanner returns a json array of impacting_resources. Each resource contains resource_details as well as impact_values.
ā Cloud scanner returns empty impacts_values when the instance type is not known in Boavizta database
{
"impacting_resources": [
{
"cloud_resource": {
"provider": "AWS",
"id": "instance-F",
"location": {
"aws_region": "eu-west-1",
"iso_country_code": "IRL"
},
"resource_details": {
"instance": {
"instance_type": "FICTIVE-INSTANCE-TYPE",
"usage": {
"average_cpu_load": 100,
"state": "running"
}
}
},
"tags": []
},
"impacts_values": null,
"impacts_duration_hours": 5
},
{
"cloud_resource": {
"provider": "AWS",
"id": "instance-1",
"location": {
"aws_region": "eu-west-1",
"iso_country_code": "IRL"
},
"resource_details": {
"instance": {
"instance_type": "a1.medium",
"usage": {
"average_cpu_load": 0.3,
"state": "running"
}
}
},
"tags": []
},
"impacts_values": {
"adp_manufacture_kgsbeq": 8.8e-07,
"adp_use_kgsbeq": 1.29e-10,
"pe_manufacture_megajoules": 0.057,
"pe_use_megajoules": 0.063,
"gwp_manufacture_kgco2eq": 0.0041,
"gwp_use_kgco2eq": 0.00226,
"raw_data": {
"impacts": {
"adp": {
"description": "Use of minerals and fossil ressources",
"embedded": {
"max": 1.261e-06,
"min": 5.808e-07,
"value": 8.8e-07,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "kgSbeq",
"use": {
"max": 1.552e-10,
"min": 1.164e-10,
"value": 1.29e-10
}
},
"gwp": {
"description": "Total climate change",
"embedded": {
"max": 0.005669,
"min": 0.002324,
"value": 0.0041,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "kgCO2eq",
"use": {
"max": 0.002718,
"min": 0.002038,
"value": 0.00226
}
},
"pe": {
"description": "Consumption of primary energy",
"embedded": {
"max": 0.07877,
"min": 0.03178,
"value": 0.057,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "MJ",
"use": {
"max": 0.07577,
"min": 0.05683,
"value": 0.063
}
}
}
}
},
"impacts_duration_hours": 5
}
]
}
Schema of the json output
A schema describe the format of the JSON estimated inventory (as part of theOpenAPI specification).
To access it:
- start a server (
cloud-scanner-cli serve) - access the OpenAPI specification at http://127.0.0.1:8000/openapi.json and a swagger-ui at http://127.0.0.1:8000/swagger-ui/
See OpenAPI specification in server mode
OpenMetrics/Prometheus output
As CLI application, when using the metrics or serve command, cloud-scanner returns consolidated results as OpenMetric/Prometheus format instead of json.
This is also the default format of the serverless application metrics route.
When using the metric output format, you get 2 sets of metrics
- Metrics named boavizta_xxxxx are summary metrics (total number of resources, summed impacts, a.s.o)
- Metrics named boavizta_resource_yyy are specific to individual resources. The metric labels can be filtered to identify resource.
cargo run metrics -u 1
Returns:
# HELP boavizta_number_of_resources_total Number of resources detected during the inventory.
# TYPE boavizta_number_of_resources_total gauge
boavizta_number_of_resources_total{awsregion="eu-west-1",country="IRL"} 4
# HELP boavizta_number_of_resources_assessed Number of resources that were considered in the estimation of impacts.
# TYPE boavizta_number_of_resources_assessed gauge
boavizta_number_of_resources_assessed{awsregion="eu-west-1",country="IRL"} 4
# HELP boavizta_duration_of_use_hours Use duration considered to estimate impacts.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-west-1",country="IRL"} 1.0
# HELP boavizta_pe_manufacture_megajoules Energy consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-west-1",country="IRL"} 0.0704
# HELP boavizta_pe_use_megajoules Energy consumed during use.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-west-1",country="IRL"} 0.2636
# HELP boavizta_adp_manufacture_kgsbeq Abiotic resources depletion potential of manufacture.
# TYPE boavizta_adp_manufacture_kgsbeq gauge
boavizta_adp_manufacture_kgsbeq{awsregion="eu-west-1",country="IRL"} 8.3e-7
# HELP boavizta_adp_use_kgsbeq Abiotic resources depletion potential of use.
# TYPE boavizta_adp_use_kgsbeq gauge
boavizta_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL"} 5.387e-10
# HELP boavizta_gwp_manufacture_kgco2eq Global Warming Potential of manufacture.
# TYPE boavizta_gwp_manufacture_kgco2eq gauge
boavizta_gwp_manufacture_kgco2eq{awsregion="eu-west-1",country="IRL"} 0.00532
# HELP boavizta_gwp_use_kgco2eq Global Warming Potential of use.
# TYPE boavizta_gwp_use_kgco2eq gauge
boavizta_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL"} 0.009430000000000001
# HELP boavizta_resource_duration_of_use_hours Use duration considered to estimate impacts.
# TYPE boavizta_resource_duration_of_use_hours gauge
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 1.0
# HELP boavizta_resource_pe_embedded_megajoules Energy consumed for manufacture.
# TYPE boavizta_resource_pe_embedded_megajoules gauge
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.021
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.017
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.0114
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.021
# HELP boavizta_resource_pe_use_megajoules Energy consumed during use.
# TYPE boavizta_resource_pe_use_megajoules gauge
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.082
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.0126
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.081
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.088
# HELP boavizta_resource_adp_embedded_kgsbeq Abiotic resources depletion potential of embedded impacts.
# TYPE boavizta_resource_adp_embedded_kgsbeq gauge
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 2.1e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 2.1e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 1.8e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 2.3e-7
# HELP boavizta_resource_adp_use_kgsbeq Abiotic resources depletion potential of use.
# TYPE boavizta_resource_adp_use_kgsbeq gauge
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 1.81e-10
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 1.65e-10
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 2.57e-11
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 1.67e-10
# HELP boavizta_resource_gwp_embedded_kgco2eq Global Warming Potential of embedded impacts.
# TYPE boavizta_resource_gwp_embedded_kgco2eq gauge
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.0016
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.0016
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.0013
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.00082
# HELP boavizta_resource_gwp_use_kgco2eq Global Warming Potential of use.
# TYPE boavizta_resource_gwp_use_kgco2eq gauge
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.00289
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.00317
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.00045
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.00292
# EOF
Serverless design
We use the serverless framework and the softprops/serverless-rust plugin to ease packaging and deployment as lambda.
The cloud-scanner-cli is wrapped into a set of lambdas functions exposed behind an AWS API gateway.
This is certainly not the only way to deploy the application. If you want more control, you could compile, package and deploy the application with Terraform or CDK, but this is not documented yet.
Serverless deployment
When deploying as serverless, you have to provide and configure Grafana and Prometheus seprately. This repository only wrapps the cloud-scanner in a lambda.
The recommended serveless deployment is the following.
The scanner is deployed as a lambda inside a first region of your account. It performs inventory of other regions.
- You may restrict to specifc regions by tuning the role of the lambda defined in
serverless.yml - You can (and should) also deploy Boaviztapi as a lambda or other in your account. This allow to perform scans and evaluate impacts without any inventory data leaking outside your account. Once you have deployed your own instance of BoaviztAPI, tune the
BOAVIZTA_API_URLin theserverless.ymlfile to ensure that cloud-scanner points to your private instance of API. See using private Boavizta API.
ā Open issue prevents Boavizta API deployment as Lambda (https://github.com/Boavizta/boaviztapi/issues/153). As a workaround we recommend deploying Boavizta API as a docker container (for example using AWS ECS) . You can still use cloud-scanner itself as serverless application deployed with Lambda.
Serverless routes
Instance impacts as JSON
The scan route returns individual instances metrics in json format (see below, same as CLI)
Use hours_use_time and aws_region parameters in the query
E.g.: https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/scan?hours_use_time=5&aws_region=eu-west-1
[
{
"instance_id": "i-03c8f84a6318a8186",
"instance_type": "a1.medium",
"usage_data": {
"hours_use_time": 5,
"time_workload": 0,
"usage_location": "IRL"
},
"impacts": {
"adp": {
"manufacture": 0.0063,
"unit": "kgSbeq",
"use": 8e-12
},
"gwp": {
"manufacture": 31,
"unit": "kgCO2eq",
"use": 0.0001
},
"pe": {
"manufacture": 430,
"unit": "MJ",
"use": 0.004
}
}
},
{
"instance_id": "i-033df52f12f30ca66",
"instance_type": "m6g.xlarge",
"usage_data": {
"hours_use_time": 5,
"time_workload": 0,
"usage_location": "IRL"
},
"impacts": {
"adp": {
"manufacture": 0.0083,
"unit": "kgSbeq",
"use": 8e-10
},
"gwp": {
"manufacture": 83,
"unit": "kgCO2eq",
"use": 0.01
},
"pe": {
"manufacture": 1100,
"unit": "MJ",
"use": 0.4
}
}
},
{
"instance_id": "i-0a3e6b8cdb50c49b8",
"instance_type": "c5n.xlarge",
"usage_data": {
"hours_use_time": 5,
"time_workload": 0,
"usage_location": "IRL"
},
"impacts": {
"adp": {
"manufacture": 0.0086,
"unit": "kgSbeq",
"use": 1e-9
},
"gwp": {
"manufacture": 64,
"unit": "kgCO2eq",
"use": 0.02
},
"pe": {
"manufacture": 840,
"unit": "MJ",
"use": 0.5
}
}
},
{
"instance_id": "i-003ea8da7bb9bfff9",
"instance_type": "m6g.xlarge",
"usage_data": {
"hours_use_time": 5,
"time_workload": 0,
"usage_location": "IRL"
},
"impacts": {
"adp": {
"manufacture": 0.0083,
"unit": "kgSbeq",
"use": 8e-10
},
"gwp": {
"manufacture": 83,
"unit": "kgCO2eq",
"use": 0.01
},
"pe": {
"manufacture": 1100,
"unit": "MJ",
"use": 0.4
}
}
}
]
Account impacts as metrics
The metrics route returns aggregated account metrics for one hour of use in Prometheus format.
Use aws_region parameters in the query.
E.g.: https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/metrics?aws_region=eu-central-1
# HELP boavizta_number_of_instances_total Number of instances detected during the scan.
# TYPE boavizta_number_of_instances_total gauge
boavizta_number_of_instances_total{awsregion="eu-central-1",country="DEU"} 7
# HELP boavizta_number_of_instances_assessed Number of instances that were considered in the measure.
# TYPE boavizta_number_of_instances_assessed gauge
boavizta_number_of_instances_assessed{awsregion="eu-central-1",country="DEU"} 5
# HELP boavizta_duration_of_use_hours Number of instances detected during the scan.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-central-1",country="DEU"} 1.0
# HELP boavizta_pe_manufacture_megajoules Power consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-central-1",country="DEU"} 1760.0
# HELP boavizta_pe_use_megajoules Power consumed during usage.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-central-1",country="DEU"} 0.86
# EOF
See also Set up monitoring dashboard for an example of scrapping and displaying these metrics.
Limitations and perimeter
We do our best to offer accurate estimations of impacts. But these figures should still be considered with a grain of salt.
All models are wrong, but some are useful.
The Boavizta impact data result from a modelling effort. It is important to consider:
- the limited Perimeter of the resources that cloud-scanner takes into account.
- the limits of estimations methodology.
Perimeter / scope
Cloud scanner only estimates the impacts of your AWS EC2 instances, and optionally block storage.
Several significant aspects of the global impacts of cloud usage are excluded:
- It only estimate impacts of compute instances (EC2 Virtual Machines) and optionally block storage (HDD, SSD volumes).
- Cloud scanner does not take into account the PUE of the data center. See Add an option to override PUE in the queries or cli option Ā· Issue #422 Ā· Boavizta/cloud-scanner
- Cloud scanner does not take into account the surrounding cloud infrastructure (network, control plan).
- For storage (experimental feature), only the impacts of manufacture are counted. The impacts of the use phase are not counted. At the moment, Boavizta API returns only the impacts of the manufacturing phase for HDD and SSD. Furthermore the impacts of storage are likely overestimated. They are calculated by taking into account the size of the logical volume. In reality, a volume that is not full may not claim the full space on a physical device, which would result in smaller impacts.
- do not take into account the over-commit (mutualization) or over-provisioning(redundancy) that cloud provider may apply to provide the service.
- do not account managed services (like DB as a service or Containers as a service).
- do not account serverless (lambda) compute.
- unsupported instance types returns zero for their impacts.
- Cloud scanner does not provide error margins https://github.com/Boavizta/boaviztapi/issues/147.
Other limits
Instances supported
Cloud scanner supports all instances types of Boavizta API. But when a specific instance type cannot be matched with the Boavizta data set, Cloud-scanner returns empty impacts (i.e. zeroed impact values).
Carbon intensity of electricity is not real time
Carbon intensity of electricity is not real time. It uses and yearly average corresponding to the country where the data center is located. This is more related to the API itself, see https://github.com/Boavizta/boaviztapi/issues/117 and general Boavizta API methodology.
We do not provide margins of error
Cloud scanner is not yet able to provide error margin related to itās estimation. Note that there is ongoing work on Boavizta API to provide more details about the error margins: https://github.com/Boavizta/boaviztapi/issues/147
Open issues / bugs
There may be additional identified issues or bugs, see Issues.
Testing
Unit tests
Unit tests are launched with cargo test command.
End to end tests
When launched with cargo test -- --include-ignored additional integration tests need:
- specific instance to be running
- aws credentials to be setup
These integration tests require specific instances to be up and running to pass. This means they are tied to a specific cloud account.
Memo: commands to start or stop instances:
# List instance state
aws ec2 describe-instance-status --include-all-instances --filters Name=instance-state-name,Values='*' --query 'InstanceStatuses[*].{InstanceId: InstanceId, State: InstanceState.Name}' --ouptut table
# start instance
aws ec2 start-instances --instance-id i-03c8f84a6318a8186
# stop instance
aws ec2 stop-instances --instance-id i-03c8f84a6318a8186