About cloud-scanner

Boavizta Cloud-scanner returns environmental impacts of your AWS Instances (EC2) and Storage usage.

  graph LR;
      inventory[ Account inventory] 
      api[Impacts from BoaviztaAPI]
      metrics[Prometheus metrics]
      json[JSON output]
      inventory --> api
      api --> metrics
      api --> json

It combines real time inventory and usage data from your AWS account with Boavizta API to offer a view of your impacts on a given region.

• multi criteria: Primary Energy consumption (PE), Abiotic resource depletion potential (ADP), and Global Warming Potential (GWP)
• multi stage: separate impacts of ressources Usage and Manufacture

Estimations can be filtered by tags. It eases attribution to a specific server, environment, application or service.

Cloud-scanner can be used:

• from command line, to get an immediate view of your impacts 💻
• as a metric server . You can use it to monitor and display real time impacts in a dashboard 📊

Cloud-scanner is an Open Source application maintained here: https://github.com/Boavizta/cloud-scanner.

How it works

Cloud scanner relies on cloud providers APIs to perform an inventory of your cloud resources. It collects information about usage (instance types, tags, CPU load or volume size).

This inventory is used to query Boavizta API which returns impact data.

Results are exposed as JSON or metrics.

Contributing

Cloud scanner is stable, but with limited functionality.

Development versions may already provide new functionalities that have not yet be included to a release. So have a look at the changelog and Issues on this repository.

And most importantly, do not hesitate to ask questions or share your feedback by opening an Issue or discussion on GitHub.

Quick start run CLI using docker 🐳

No installation needed, you will run a public docker image of cloud-scanner CLI.

Pre-requisites

• Docker.
• A working AWS account (and your AWS CLI profile already configured)

Run cloud-scanner cli

docker pull ghcr.io/boavizta/cloud-scanner-cli:latest
docker run -it ghcr.io/boavizta/cloud-scanner-cli:latest --help

# Ensure you have a working AWS profile setup locally (like you would do for AWS CLI)
# Note
# - we map local credentials on the container (-v)
# - we force a using 'myprofile' profile by setting the AWS_PROFILE environment variable with -e flag
# - the -it flag is optional, only purpose is to get colored output if any

# Just list instances
docker run -it -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e AWS_PROFILE='myprofile' ghcr.io/boavizta/cloud-scanner-cli:latest inventory

# List instances and standard impacts (for 10 hours of use)
docker run -it -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e AWS_PROFILE='myprofile' ghcr.io/boavizta/cloud-scanner-cli:latest estimate --use-duration-hours 10

# Serve metrics
``` sh
# Note that we need to provide ca certificates and bind  listen address to 0.0.0.0.
docker run -it -p 8000:8000 -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt -v $HOME/.aws/credentials:/root/.aws/credentials:ro -e ROCKET_ADDRESS=0.0.0.0 -e ROCKET_PORT=8000 -e AWS_PROFILE='myprofile'  ghcr.io/boavizta/cloud-scanner-cli:latest serve

⚠ This method of passing credentials is not secure nor very practical. In a production setup on AWS, you should rather rely on the role of the instance that execute the container to manage authentication of the cli.

⚠ Running metric server in container require setting extra variables:

• to map AWS credentials
• to map SSL ca certificates
• and more importantly to configure Rocket to listen to 0.0.0.0 instead of default 127.0.0.1 (which is internal to the container. This is done with the extra variable: ROCKET_ADDRESS=0.0.0.0

Alternatively you may build and use a local docker image, See build a local docker image

Quick start : display dashboard using docker-compose 🐳

Visualize the live impacts of you account in a dashboard.

No installation needed, you will run a public docker image of cloud-scanner CLI, Boavizta API, Prometheus and Grafana to get access to a demo dashboard..

All data remain local to your environment (this docker-compose stack uses a private instance of Boavizta API).

Pre-requisites

• Docker and docker-compose
• A working AWS account (and your AWS CLI profile already configured)

Run the demo dashboard

# Map your AWS credentials
export AWS_PROFILE=name-of-your-profile
# Start the stack (from the root of the repository)
docker-compose up

• Demo dashboard is exposed on http://localhost:3001
• Log in with user admin/admin
• Select the dashboard in the left menu.

⚠ The demo docker-compose is preconfigured to generate metrics only for the following regions:

• eu-west-1
• eu-west-3
• eu-central-1
• us-east-1

Refer to Generating metrics for additional regions to retrieve metrics for additional regions.

Additional info

• ⚠ This docker-compose example is not intended for production deployment, but rather for quick testing.
  • ports of all services are exposed.
  • Grafana is served on http with default login.
• You may have to update the line mapping your AWS profile (Replace AWS_PROFILE=${AWS_PROFILE} by AWS_PROFILE=the-real-name-of-your-profile) when using Podman. It seems that Podman compose does not map the variables of the environment to the containers.
• In corporate environments, you may need to provide your certificates authorities certificates (ca-certificates) to the cloud-scanner container (uncomment the mapping line in the docker-compose file).
• For the demo, we deliberately set a short metrics scrapping interval (30 seconds). In a production environment, you may want to increase this metric scraping interval to reduce the API calls and volume of data. This scrapping period is set in the Prometheus configuration file.

Quick start - Run CLI using Cargo 🦀

This quick start will show the impacts of your EC2 instances for 10 hours of use, by building a local version of the cloud-scanner CLI from source.

Prerequisites

• a Rust toolchain installed locally rustup.rs - The Rust toolchain installer
• A working AWS account (and you AWS CLI profile already configured)
• optional: jq to format json results

Compile an run CLI 💻

1. Clone the cloud-scanner repository
2. build and run CLI

# Ensure you have a working AWS profile setup locally (like you would do for AWS CLI)
export AWS_PROFILE='<YOUR_PROFILE_NAME>'

# Get impacts of 10 hours of use (on your default account region)
cargo run estimate --use-duration-hours 10 | jq

# Same thing but as metrics
cargo run -- --as-metrics estimate --use-duration-hours 10

# Same query for explicit region
cargo run -- --aws-region eu-west-3 estimate --use-duration-hours 10 | jq

Use cloud-scanner as a serverless application⚡

This will deploy the cloud-scanner inside your AWS account. You can use it to:

• scan the account to get json impacts (as you would do with the CLI)
• or
• get metrics (that you can scrape with prometheus or the monitoring system of your choice)

The application is build and deployed using the serverless framework (see serverless-design).

Prerequisites

The deployment creates an aws role configured to get sufficient permission to scan your resources without requesting end-user authentication.

1. Nodejs installed locally
2. An AWS account/profile with sufficient permissions to deploy lambda, configure API gateway, and create a role.

⚠ The deployment process was only tested on Linux.

Deploy the application ⚡

# Install node the serverless framework and it's dependencies
npm i
export aws_profile = <my profile>
# Deploy the application to your AWS account
serverless deploy

Usage

Scan the account / region 💻

This returns results in json format (see below, same as CLI)

https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/scan?hours_use_time=5&aws_region=eu-west-1

Use hours_use_time and aws_region parameters in the query to pass the values to the lambda.

Get Metrics 📊

https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/metrics?aws_region=eu-central-1

Returns metrics for 1 hour of use in prometheus format.

The metrics represent the costs / impacts of one hour of use of the resources present in your account at the time of the query.

Use aws_region parameters in the query.

# HELP boavizta_number_of_instances_total Number of instances detected during the scan.
# TYPE boavizta_number_of_instances_total gauge
boavizta_number_of_instances_total{awsregion="eu-central-1",country="DEU"} 7
# HELP boavizta_number_of_instances_assessed Number of instances that were considered in the measure.
# TYPE boavizta_number_of_instances_assessed gauge
boavizta_number_of_instances_assessed{awsregion="eu-central-1",country="DEU"} 5
# HELP boavizta_duration_of_use_hours Number of instances detected during the scan.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-central-1",country="DEU"} 1.0
# HELP boavizta_pe_manufacture_megajoules Power consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-central-1",country="DEU"} 1760.0
# HELP boavizta_pe_use_megajoules Power consumed during usage.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-central-1",country="DEU"} 0.86
# EOF

Methodology and source of data

Cloud scanner uses the Boavizta methodology to estimate the impacts of cloud resources.

Source of impact data

Impact data is retrieved from BOAVIZTA reference data API v1.2.x.

Methodology

The general approach of Boavizta is described in Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta

The impacts (use and embedded) are attributed according to the principles described in Cloud instances - Boavizta API documentation.

The results are similar to what you can visualize in Datavizta, but with automated inventory.

⚠ Cloud scanner underestimates the impacts of the cloud resources. Because it only considers the instances and block storage a several sources of impacts (network, potential redundancy, cloud control plan) are not included in the estimation.

See also other limits.

• https://www.boavizta.org/en

How we process workload

Workload of instances are estimated using AWS cloudwatch CPU metrics summary.

Cloud scanner uses a sampling period of 15 minutes, but impacts metrics are returned as impacts equivalent to one hour of use.

This means that instance impacts metrics data returned can be understood as: impact for one hour of use (considering the CPU workload of 15 last minutes).

Why this default sampling period of 15 minutes ?

• It seems sufficient for our current monitoring needs (but maybe we can make it configurable in the future).
• It seems hard to go below 10 minutes (because default period of AWS instance metrics is 5 minutes. You need to activate detailed monitoring (extra feature) for 1 minute granularity: List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud).

Building cloud-scanner CLI with Cargo 🦀

On Linux

Install Rust and linux-musl dependencies.

# Install rust (see https://www.rust-lang.org/tools/install), validate when prompted
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Add Linux musl target (needed for cross compilation for aws lambda)
rustup target add x86_64-unknown-linux-musl

sudo apt update && sudo apt install -y musl-tools musl-dev

# Test a build
cargo build

#  build a release
cargo build --release

On Windows with WSL2

Tested method to build Rust on Windows is to use Windows Subsystem For Linux (WSL2)

1. Prerequisite: install WSL2 and the latest Ubuntu LTS image (22.04). If you do not WSL yet, you may inspire from Set up Node.js on WSL 2.
2. After WSL is setup, follow Linux instructions above inside the Linux VM.

Using docker

See Build a local docker image

Usage with Docker 🐳

Build a local docker image

# Local build of docker image
docker build . --tag cloud-scanner-cli
# Test run
docker run -it cloud-scanner-cli --help

Run cloud-scanner as a docker image

See Quickstart using docker

Deploy cloud scanner as a serverless application

You can deploy cloud scanner as a serverless application (AWS lambda).

The application is build and deployed using the serverless framework (see serverless-design).

Using Linux

1. Install Rust
2. Install nvm, nodejs
3. test serverless package

First, don’t forget to clone the repo using

git clone https://github.com/Boavizta/cloud-scanner.git
cd cloud-scanner

Install Rust

# Install rust (see https://www.rust-lang.org/tools/install), validate when prompted
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Add Linux musl target (needed for cross compilation for lambda)
rustup target add x86_64-unknown-linux-musl

sudo apt update && sudo apt install -y musl-tools musl-dev

Install node (for serverless deployment)

nvm, node.js and npm

See https://docs.microsoft.com/en-us/windows/dev-environment/javascript/nodejs-on-wsl

sudo apt-get install curl

# Install nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
# Install node
nvm install --lts

Install serverless framework and deps

npm install -g serverless
npm i

Configure environment

Everything you need should now be installed, but there is a last step you have to do before deploying. Since the serverless framework automatically deploys the lambda onto your AWS instance without you having to upload a zip file or anything, you have to give it access to it by giving it an access key (see this guide for more informations on how to generate an access keys).

serverless config credentials \
  --provider aws \
  --key YOURACCESSKEY \
  --secret YOURSECRETKEY

Optionally, you can config a private instance of Boaviztapi by setting the environment variable BOAVIZTA_API_URL so :

export BOAVIZTA_API_URL="boaviztapi.example.com"

If the environment variable is not set, cloud scanner will use the public instance (https://api.boavizta.org) by default.

Deploy

You should be good to go by now, simply run

serverless package
serverless deploy

and wait for it to be done. You should by now see two lamda functions appear on your AWS instance, Congratulations !

If any error happen, redo those steps carefully and make sure you didn’t miss anything before posting a GitHub Issue.

Using Windows

Tested method to deploy this serverless app from on windows is to use Windows Subsystem For Linux (WSL2).

1. Prerequisite: install WSL2 and the latest Ubuntu LTS image (22.04). If you do not have it yet, you may inspire from Set up Node.js on WSL 2.
2. After WSL is setup, follow Linux instructions above inside the Linux VM.

AWS authentication

Cloud scanner uses AWS Rust SDK to query AWS account. By default Rust SDK picks up AWS credentials from environment variables.

AWS permissions required by Cloud Scanner

🔥 An Important pre-requisite is to have a user with sufficient permissions to list resources of the account (or an instance profile / role pre-configured when running cloud-scanner from EC2).

The minimal set of permissions to perform inventory of resources (and query CPU load of instances) is:

• ec2:DescribeInstances
• cloudwatch:GetMetricStatistics
• cloudwatch:DescribeAlarm

You could also restricts permissions to a specific set of instances or resources.

Pass credentials as environment variables

Option 1: Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Set environment variables with your account detail.

# Example for Linux / macOS
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=eu-west-1

See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html for equivalent Windows command prompt or Powershell syntax example.

Option 2: Use a an existing AWS_PROFILE

If you have configured an AWS CLI profile, the easiest way to pass aws credential to cloud scanner is use an environment variable that points to this profile.

Pre-requisite to use a profile:

1. AWS CLI installed: Installing or updating the latest version of the AWS CLI - AWS Command Line Interface
2. AWS CLI configured: Configure the AWS CLI - AWS Command Line Interface.

# Example for Linux / macOS
# cloud-scanner can use the AWS_PROFILE set as en environment variable.
# You have to reference one of the profiles previously configured in ~/.aws/credentials
export AWS_PROFILE=YOUR_PROFILE_NAME_EXAMPLE
export AWS_DEFAULT_REGION=eu-west-3

See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html for equivalent Windows command prompt or Powershell syntax example.

Reference

See Specify your credentials and default Region - AWS SDK for Rust for more options to pass credentials to the SDK.

Integrate metrics to a monitoring dashboard

Cloud-scanner can be used to export metrics related to the impact of your cloud account.

The metrics are exposed in Prometheus/OpenMetrics format. You can use it to feed a monitoring dashboard and keep an eye on the evolution of impacts of your usage.

Overview

1. Start cloud-scanner in metrics mode (using the --serve option or by deploying it as a serverless application).
2. Setup Prometheus to scrape the metrics. Metrics are generated separately for each region.
3. Configure a dashboard to display the results.

Detailed steps

Demo for some EU regions.

You can refer to the provided docker compose example for a quick start.

It comes preconfigured to generate metrics for the following regions:

• eu-west-1
• eu-west-3
• eu-central-1
• us-east-1

Generating metrics for additional regions.

Examples of Prometheus and Grafana configuration files are located in the dashboard-config directories.

To generate (and display) metrics for an additional region:

1. Edit the prometheus configuration (prometheus.yml)
2. Add a new job and set its region parameter like aws_region: ['us-east-2'].
3. Restart Prometheus container and verify that metrics are now collected for this region.

In the exemple compose stack, prometheus admin GUI is exposed. You can check status of the scrape targets at: http://localhost:9090/targets.

# Extract from prometheus.yml
# Example extra job configuration to cover us-east-2
  - job_name: 'cloud-scanner-metrics-us-east-2'
    scheme: http
    static_configs:
      - targets: ['cloud_scanner_boa:8000']
        labels:
          group: 'production'
    metrics_path: '/metrics'
    params:
      aws_region: ['us-east-2']
      include_block_storage: ['true']

Adapting configuration for production use:

In production environment, you may want to increase the metrics scraping interval.

This It is set to 30 seconds in the sample prometheus configuration. In production you shoud increase this intervall (e.g. 10 minutes).

# Prometheus scrapping interval / extract from prometheus.yml configuration
global:
  scrape_interval:     30s # By default, scrape targets every 30 seconds.
  evaluation_interval: 30s # Evaluate rules every 30 seconds.

Filter by tags

You can limit cloud-scanner results (metriccs or json) to the cloud resources that match specific tags.

You can use multiple tags.

Works with inventory or estimates.

Filter in cli

# Retrieve only resources with the following tags:
#  CustomTagNameForDebug=olivierdemeringoadm
#     and 
#  Name=test-boavizta
cargo run -- -t Name=test-boavizta-2 -t CreatorName=olivierdemeringoadm  inventory| jq

Filter (with serverless/lambda)

Use one (or multiple) filter_tag parameter in the query string to restrict results.

# Retrieve only resources with the following tags:
#  CustomTagNameForDebug=olivierdemeringoadm
#     and 
#  Name=test-boavizta
https://<EDITED>.execute-api.eu-west-1.amazonaws.com/dev/scan?use_duration_hours=5&aws_region=eu-west-1&filter_tag=CustomTagNameForDebug=olivierdemeringoadm&filter_tag=Name=test-boavizta

Filter metrics

Use the filter_tag query parameter in the URL.

It can be used multiple time (only instances matching all tags will be returned).

Example queries

• http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value
• http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest
• http://localhost:8000/metrics?aws_region=eu-west-1&filter_tag=Name=test-boavizta

Important limitation

Suppose the following instances (and tags)

• instance1 (

  • ENV=prod
  • PURPOSE=CI

• instance2

  • ENV=prod
  • PURPOSE=service

• Filtering on “ENV=prod” will return instance1 and instance2

• Filtering on “ENV=prod” and PURPOSE=CI will return only instance2

• No filter will return all instances

Using a private instance of Boavizta API

To avoid that the scanner shares your inventory data to the public version of Boavizta API, It is recommended to deploy your own instance of Boavizta API.

This is also useful if you want to stick to a specific version of the API.

The Boavizta API can be deployed in several ways (docker, lambda a.s.o.). Refer to the API documentation: Deploy - Boavizta API documentation or refer to the provided docker compose example for an example of integration cloud-scanner and a private instance of the API.

Once you have deployed your instance of API, use either the command line flags (CLI) or environment variable (docker, lambda) to configure the scanner to use your API instance.

Common issues and FAQ

The demo dashboard does not show any metric…

Several reasons may explain why metrics do not appear in the dashboard.

1. Verify in the docker-compose logs that there is no error related to AWS authentication.
2. Check that prometheus is configured to retrieve metrics for the region you use.

⚠ The demo docker-compose is preconfigured to generate metrics only for the following regions:

• eu-west-1
• eu-west-3
• eu-central-1
• us-east-1

Refer to Generating metrics for additional regions to retrieve metrics for additional regions.

The docker compose demo may not display the latest version of the Grafana dashboard

/!\ The existing Grafana volume is preserved even after deploying a newer version of the docker-compose demo stack.

As a consequence the Grafana provisioning does not occur. Any update of the demo dashboard provided with the new stack is not reflected in the instance, even if Grafana container itself is recreated.

Workaround

We have 2 possibilities, depending if we want to preserve the existing Grafana configuration (volume) or not.

• Preserve the existing volume if it was previously customized to contain additional Grafana configuration (outside cloud scanner) or if you add custom dashboards.
• Delete the volume if you use a vanilla instance of cloud-scanner demo.

Updating the dashboard while keeping existing volume

Wee need to upload the new dashboard definition manually using the Grafana web UI.

• The dashboard definition can be found under dashboard-config/provisioning/dashboards/: grafana-dashboard-cloud-impacts.template.json
• By default Grafana UI is exposed at http://<host-ip>:3001

Updating the dashboard by recreating Grafana configuration from scratch

/!\ Use this method only if Grafana was not customized after deployment of the demo or you are ok to loose these changes.

The easiest is to delete the Grafana volume and Grafana container. They will be recreated, and Grafana provisioned with latest dashboard, when we restart the docker compose stack.

# stop docker compose stack
docker compose down
# delete Grafana container (necessary because it may be using the volume even if stopped)
docker rm grafana_boa
# now delete volume
docker volume rm cloud-scanner_grafana-data
# Recreate compose stack
docker compose up

CLI options

List aws instances and their environmental impact (from Boavizta API)

Usage: cloud-scanner-cli [OPTIONS] <COMMAND>

Commands:
  estimate   Get estimation of impacts for a given usage duration
  inventory  List instances and  their average cpu load for the last 5 minutes (without returning impacts)
  serve      Run as a standalone server. Access metrics (e.g. http://localhost:8000/metrics?aws_region=eu-west-3), inventory or impacts (see http://localhost:8000/swagger-ui)
  help       Print this message or the help of the given subcommand(s)

Options:
  -a, --aws-region <AWS_REGION>
          AWS region (The default aws profile region is used if not provided)
  -b, --boavizta-api-url <BOAVIZTA_API_URL>
          Optional Boavizta API URL if you want to use your own instance (URL without the trailing slash, e.g. https://api.boavizta.org)
  -t, --filter-tags <FILTER_TAGS>
          Filter instances on tags (like tag-key-1=val_1 tag-key_2=val2)
  -v, --verbosity...
          Enable logging and show execution duration, use multiple `v`s to increase logging level warning to debug
  -h, --help
          Print help
  -V, --version

Experimental feature: estimate block storage

Use the --include-block-storage command line flag or parameter to consider block storage (either in inventory or when requesting an estimation of impacts.). This parameter defaults to false . This means that by default block storage (volumes) are not counted in the inventory nor in the results.

⚠ In any case, for storage, the impacts of the use phase are not counted. Boavizta API returns only the impacts of the manufacturing phase for HDD and SSD. Furthermore the impacts of storage are likely are likely overestimated. They are calculated by taking into account the size of the logical volume. In reality, a volume that is not full may not claim the full space on a physical device, which would result in smaller impacts.

# Experimental: get impacts of instances and attached storage
cargo run estimate --use-duration-hours 1 --include-block-storage --output-verbose-json

Display statistics

Use -v will display statistics on std error.

• First output line shows the time spend specifically gathering CPU load of instances.
• Second line shows global statistics:
  • total inventory duration: total time taken to retrieve resource lists and CPU statistics from AWS.
  • impact estimation duration: time taken to query Boavizta API
  • total execution duration.

cloud-scanner-cli -v estimate -u 1
# use -- -v with cargo
# cargo  run -- -v estimate -u 1
[...]
cloud_scanner_cli::aws_inventory: Total time spend querying CPU load of instances: 372.153481ms
cloud_scanner_cli: ExecutionStatistics { inventory_duration: 911.526773ms, impact_duration: 398.993816ms, total_duration: 1.310520822s }
[...]

Environment variables

Cloud scanner uses environment variables to configure connection to your cloud account. See AWS authentication.

Format of results in server mode

When run in server mode (cloud-scanner-cli serve), Cloud-scanner exposes 3 endpoints:

• /metrics: returns Prometheus metrics (plain text)
• /inventory: returns an inventory (json format, see schema below)
• /impacts: returns impacts (json format, see schema below)

Open API specification (Swagger)

The latest (up-to-date) version of OpenAPI specification is exposed under <BaseURL>/openapi.json path and displayed using swagger-ui at <BaseURL>/swagger-ui/index.html.

{
  "openapi": "3.0.0",
  "info": {
    "title": "cloud-scanner-cli",
    "version": "2.0.4"
  },
  "paths": {
    "/metrics": {
      "get": {
        "tags": [
          "metrics"
        ],
        "summary": "Returns Prometheus metrics.",
        "description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nResults are estimated for one hour of use by default.\n\nExample query: http://localhost:8000/metrics?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value&use_duration_hours=1.0&include_storage=true",
        "operationId": "metrics",
        "parameters": [
          {
            "name": "aws_region",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "filter_tags",
            "in": "query",
            "schema": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "nullable": true
            }
          },
          {
            "name": "use_duration_hours",
            "in": "query",
            "schema": {
              "type": "number",
              "format": "float",
              "nullable": true
            }
          },
          {
            "name": "include_block_storage",
            "in": "query",
            "schema": {
              "type": "boolean",
              "nullable": true
            }
          }
        ],
        "responses": {
          "200": {
            "description": "",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    },
    "/inventory": {
      "get": {
        "tags": [
          "inventory"
        ],
        "summary": "Returns the inventory as json.",
        "description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/inventorynew?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value",
        "operationId": "inventory",
        "parameters": [
          {
            "name": "aws_region",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "filter_tags",
            "in": "query",
            "schema": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "nullable": true
            }
          },
          {
            "name": "include_block_storage",
            "in": "query",
            "schema": {
              "type": "boolean",
              "nullable": true
            }
          }
        ],
        "responses": {
          "200": {
            "description": "",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/Inventory"
                }
              }
            }
          }
        }
      }
    },
    "/impacts": {
      "get": {
        "tags": [
          "impacts"
        ],
        "summary": "Returns the impacts (use and embedded) as json.",
        "description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/impacts?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value&use_duration_hours=1.0",
        "operationId": "impacts",
        "parameters": [
          {
            "name": "aws_region",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "filter_tags",
            "in": "query",
            "schema": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "nullable": true
            }
          },
          {
            "name": "use_duration_hours",
            "in": "query",
            "schema": {
              "type": "number",
              "format": "float",
              "nullable": true
            }
          },
          {
            "name": "verbose_output",
            "in": "query",
            "schema": {
              "type": "boolean",
              "nullable": true
            }
          },
          {
            "name": "include_block_storage",
            "in": "query",
            "schema": {
              "type": "boolean",
              "nullable": true
            }
          }
        ],
        "responses": {
          "200": {
            "description": "",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/EstimatedInventory"
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "Inventory": {
        "description": "Inventory: a list of resources",
        "type": "object",
        "required": [
          "resources"
        ],
        "properties": {
          "resources": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/CloudResource"
            }
          },
          "executionStatistics": {
            "allOf": [
              {
                "$ref": "#/components/schemas/ExecutionStatistics"
              }
            ],
            "nullable": true
          }
        }
      },
      "CloudResource": {
        "description": "A cloud resource (could be an instance, function or any other resource)",
        "type": "object",
        "required": [
          "id",
          "location",
          "provider",
          "resource_details",
          "tags"
        ],
        "properties": {
          "provider": {
            "$ref": "#/components/schemas/CloudProvider"
          },
          "id": {
            "type": "string"
          },
          "location": {
            "$ref": "#/components/schemas/UsageLocation"
          },
          "resource_details": {
            "$ref": "#/components/schemas/ResourceDetails"
          },
          "tags": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/CloudResourceTag"
            }
          }
        }
      },
      "CloudProvider": {
        "type": "string",
        "enum": [
          "AWS",
          "OVH"
        ]
      },
      "UsageLocation": {
        "description": "The location where cloud resources are running.\n\nTODO! the usage location should be abstracted and vendor specific implementation should be part of the cloud_provider model (region names are tied to a specific cloud provider)",
        "type": "object",
        "required": [
          "aws_region",
          "iso_country_code"
        ],
        "properties": {
          "aws_region": {
            "description": "The AWS region (like eu-west-1)",
            "type": "string"
          },
          "iso_country_code": {
            "description": "The 3-letters ISO country code corresponding to the country of the aws_region",
            "type": "string"
          }
        }
      },
      "ResourceDetails": {
        "oneOf": [
          {
            "type": "string",
            "enum": [
              "ObjectStorage"
            ]
          },
          {
            "type": "object",
            "required": [
              "Instance"
            ],
            "properties": {
              "Instance": {
                "type": "object",
                "required": [
                  "instance_type"
                ],
                "properties": {
                  "instance_type": {
                    "type": "string"
                  },
                  "usage": {
                    "allOf": [
                      {
                        "$ref": "#/components/schemas/InstanceUsage"
                      }
                    ],
                    "nullable": true
                  }
                }
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "BlockStorage"
            ],
            "properties": {
              "BlockStorage": {
                "type": "object",
                "required": [
                  "storage_type"
                ],
                "properties": {
                  "storage_type": {
                    "type": "string"
                  },
                  "usage": {
                    "allOf": [
                      {
                        "$ref": "#/components/schemas/StorageUsage"
                      }
                    ],
                    "nullable": true
                  },
                  "attached_instances": {
                    "type": "array",
                    "items": {
                      "$ref": "#/components/schemas/StorageAttachment"
                    },
                    "nullable": true
                  }
                }
              }
            },
            "additionalProperties": false
          }
        ]
      },
      "InstanceUsage": {
        "type": "object",
        "required": [
          "average_cpu_load",
          "state",
          "usage_duration_seconds"
        ],
        "properties": {
          "average_cpu_load": {
            "type": "number",
            "format": "double"
          },
          "usage_duration_seconds": {
            "type": "integer",
            "format": "uint32",
            "minimum": 0.0
          },
          "state": {
            "$ref": "#/components/schemas/InstanceState"
          }
        }
      },
      "InstanceState": {
        "type": "string",
        "enum": [
          "Running",
          "Stopped"
        ]
      },
      "StorageUsage": {
        "type": "object",
        "required": [
          "size_gb",
          "usage_duration_seconds"
        ],
        "properties": {
          "size_gb": {
            "type": "integer",
            "format": "int32"
          },
          "usage_duration_seconds": {
            "type": "integer",
            "format": "uint32",
            "minimum": 0.0
          }
        }
      },
      "StorageAttachment": {
        "type": "object",
        "required": [
          "instance_id"
        ],
        "properties": {
          "instance_id": {
            "type": "string"
          }
        }
      },
      "CloudResourceTag": {
        "description": "A tag (just a mandatory key + optional value)",
        "type": "object",
        "required": [
          "key"
        ],
        "properties": {
          "key": {
            "type": "string"
          },
          "value": {
            "type": "string",
            "nullable": true
          }
        }
      },
      "ExecutionStatistics": {
        "description": "Statistics about program execution",
        "type": "object",
        "required": [
          "impact_estimation_duration",
          "inventory_duration",
          "total_duration"
        ],
        "properties": {
          "inventory_duration": {
            "$ref": "#/components/schemas/Duration"
          },
          "impact_estimation_duration": {
            "$ref": "#/components/schemas/Duration"
          },
          "total_duration": {
            "$ref": "#/components/schemas/Duration"
          }
        }
      },
      "Duration": {
        "type": "object",
        "required": [
          "nanos",
          "secs"
        ],
        "properties": {
          "secs": {
            "type": "integer",
            "format": "uint64",
            "minimum": 0.0
          },
          "nanos": {
            "type": "integer",
            "format": "uint32",
            "minimum": 0.0
          }
        }
      },
      "EstimatedInventory": {
        "description": "An estimated inventory: impacting resources with their estimated impacts",
        "type": "object",
        "required": [
          "impactingResources"
        ],
        "properties": {
          "impactingResources": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/CloudResourceWithImpacts"
            }
          },
          "executionStatistics": {
            "allOf": [
              {
                "$ref": "#/components/schemas/ExecutionStatistics"
              }
            ],
            "nullable": true
          }
        }
      },
      "CloudResourceWithImpacts": {
        "type": "object",
        "required": [
          "cloud_resource",
          "impacts_duration_hours"
        ],
        "properties": {
          "cloud_resource": {
            "$ref": "#/components/schemas/CloudResource"
          },
          "impacts_values": {
            "description": "The impacts",
            "allOf": [
              {
                "$ref": "#/components/schemas/ImpactsValues"
              }
            ],
            "nullable": true
          },
          "impacts_duration_hours": {
            "description": "The duration for which impacts are calculated",
            "type": "number",
            "format": "float"
          }
        }
      },
      "ImpactsValues": {
        "description": "Impacts of an individual resource",
        "type": "object",
        "required": [
          "adp_manufacture_kgsbeq",
          "adp_use_kgsbeq",
          "gwp_manufacture_kgco2eq",
          "gwp_use_kgco2eq",
          "pe_manufacture_megajoules",
          "pe_use_megajoules"
        ],
        "properties": {
          "adp_manufacture_kgsbeq": {
            "type": "number",
            "format": "double"
          },
          "adp_use_kgsbeq": {
            "type": "number",
            "format": "double"
          },
          "pe_manufacture_megajoules": {
            "type": "number",
            "format": "double"
          },
          "pe_use_megajoules": {
            "type": "number",
            "format": "double"
          },
          "gwp_manufacture_kgco2eq": {
            "type": "number",
            "format": "double"
          },
          "gwp_use_kgco2eq": {
            "type": "number",
            "format": "double"
          },
          "raw_data": {
            "nullable": true
          }
        }
      }
    }
  }
}

Output data

Cloud scanner CLI and serverless application return data as json or Open Metrics (Prometheus) format.

JSON CLI output (the default)

Cloud scanner returns a json array of instances metadata.

⚠ Returns empty impacts when the instance type is not known in Boavizta database

[
  {
    "instance_id": "i-001dc0ebbf9cb25c0",
    "instance_type": "t2.micro",
    "usage_data": {
      "use_duration_hours": 5,
      "usage_location": "IRL"
    },
    "impacts": {}
  },
  {
    "instance_id": "i-004599844f7c24814",
    "instance_type": "t2.small",
    "usage_data": {
      "use_duration_hours": 5,
      "usage_location": "IRL"
    },
    "impacts": {}
  },
  {
    "instance_id": "i-075444d7293d8bd76",
    "instance_type": "t2.micro",
    "usage_data": {
      "use_duration_hours": 5,
      "usage_location": "IRL"
    },
    "impacts": {}
  },
  {
    "instance_id": "i-033df52f12f30ca66",
    "instance_type": "m6g.xlarge",
    "usage_data": {
      "use_duration_hours": 5,
      "usage_location": "IRL"
    },
    "impacts": {
      "adp": {
        "manufacture": 0.0084,
        "unit": "kgSbeq",
        "use": 1.7e-9
      },
      "gwp": {
        "manufacture": 87,
        "unit": "kgCO2eq",
        "use": 0.029
      },
      "pe": {
        "manufacture": 1100,
        "unit": "MJ",
        "use": 0.82
      }
    }
  }
]

Server mode json results

The format of the json results is slightly more complex in server mode.

When run in server mode, the server exposes an OpenAPI specification at http://127.0.0.1:8000/openapi.json and a swagger-ui: http://127.0.0.1:8000/swagger-ui/

See OpenAPI specification in server mode

OpenMetrics/Prometheus output

As CLI application, If using --as-metrics or -m option or the serve command, cloud-scanner returns consolidated results as OpenMetric/Prometheus format instead of json details. This is also the default format of the serverless app metrics route.

When using the metric output format, you get 2 sets of metrics

• Metrics named: boavizta_xxxxx are summary metrics (total number of resources, summed impacts, a.s.o)
• Metrics named boavizta_resource_yyy are specific to individual resources. The metric label can be filtered to identify resource.

cargo run -- --as-metrics estimate -u 1 

Returns:

# HELP boavizta_number_of_resources_total Number of resources detected during the inventory.
# TYPE boavizta_number_of_resources_total gauge
boavizta_number_of_resources_total{awsregion="eu-west-1",country="IRL"} 4
# HELP boavizta_number_of_resources_assessed Number of resources that were considered in the estimation of impacts.
# TYPE boavizta_number_of_resources_assessed gauge
boavizta_number_of_resources_assessed{awsregion="eu-west-1",country="IRL"} 4
# HELP boavizta_duration_of_use_hours Use duration considered to estimate impacts.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-west-1",country="IRL"} 1.0
# HELP boavizta_pe_manufacture_megajoules Energy consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-west-1",country="IRL"} 0.0704
# HELP boavizta_pe_use_megajoules Energy consumed during use.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-west-1",country="IRL"} 0.2636
# HELP boavizta_adp_manufacture_kgsbeq Abiotic resources depletion potential of manufacture.
# TYPE boavizta_adp_manufacture_kgsbeq gauge
boavizta_adp_manufacture_kgsbeq{awsregion="eu-west-1",country="IRL"} 8.3e-7
# HELP boavizta_adp_use_kgsbeq Abiotic resources depletion potential of use.
# TYPE boavizta_adp_use_kgsbeq gauge
boavizta_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL"} 5.387e-10
# HELP boavizta_gwp_manufacture_kgco2eq Global Warming Potential of manufacture.
# TYPE boavizta_gwp_manufacture_kgco2eq gauge
boavizta_gwp_manufacture_kgco2eq{awsregion="eu-west-1",country="IRL"} 0.00532
# HELP boavizta_gwp_use_kgco2eq Global Warming Potential of use.
# TYPE boavizta_gwp_use_kgco2eq gauge
boavizta_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL"} 0.009430000000000001
# HELP boavizta_resource_duration_of_use_hours Use duration considered to estimate impacts.
# TYPE boavizta_resource_duration_of_use_hours gauge
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 1.0
boavizta_resource_duration_of_use_hours{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 1.0
# HELP boavizta_resource_pe_embedded_megajoules Energy consumed for manufacture.
# TYPE boavizta_resource_pe_embedded_megajoules gauge
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.021
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.017
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.0114
boavizta_resource_pe_embedded_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.021
# HELP boavizta_resource_pe_use_megajoules Energy consumed during use.
# TYPE boavizta_resource_pe_use_megajoules gauge
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.082
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.0126
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.081
boavizta_resource_pe_use_megajoules{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.088
# HELP boavizta_resource_adp_embedded_kgsbeq Abiotic resources depletion potential of embedded impacts.
# TYPE boavizta_resource_adp_embedded_kgsbeq gauge
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 2.1e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 2.1e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 1.8e-7
boavizta_resource_adp_embedded_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 2.3e-7
# HELP boavizta_resource_adp_use_kgsbeq Abiotic resources depletion potential of use.
# TYPE boavizta_resource_adp_use_kgsbeq gauge
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 1.81e-10
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 1.65e-10
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 2.57e-11
boavizta_resource_adp_use_kgsbeq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 1.67e-10
# HELP boavizta_resource_gwp_embedded_kgco2eq Global Warming Potential of embedded impacts.
# TYPE boavizta_resource_gwp_embedded_kgco2eq gauge
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.0016
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.0016
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.0013
boavizta_resource_gwp_embedded_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.00082
# HELP boavizta_resource_gwp_use_kgco2eq Global Warming Potential of use.
# TYPE boavizta_resource_gwp_use_kgco2eq gauge
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-033df52f12f30ca66",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boavizta") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }]",resource_state="Stopped"} 0.00289
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-0a3e6b8cdb50c49b8",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "appname", value: Some("app1") }, CloudResourceTag { key: "created_by", value: Some("demeringo") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("boavizta-c5n.xlarge") }]",resource_state="Stopped"} 0.00317
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-03c8f84a6318a8186",resource_tags="[CloudResourceTag { key: "Name", value: Some("test-boapi") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }]",resource_state="Running"} 0.00045
boavizta_resource_gwp_use_kgco2eq{awsregion="eu-west-1",country="IRL",resource_type="Instance",resource_id="i-003ea8da7bb9bfff9",resource_tags="[CloudResourceTag { key: "CustomTagNameForDebug", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "CreatorName", value: Some("olivierdemeringoadm") }, CloudResourceTag { key: "Name", value: Some("test-boavizta-2") }]",resource_state="Running"} 0.00292
# EOF

Serverless design

We use the serverless framework and the softprops/serverless-rust plugin to ease packaging and deployment as lambda.

The cloud-scanner-cli is wrapped into a set of lambdas functions exposed behind an AWS API gateway.

This is certainly not the only way to deploy the application. If you want more control, you could compile, package and deploy the application with Terraform or CDK, but this is not documented yet.

Serverless deployment

When deploying as serverless, you have to provide and configure Grafana and Prometheus seprately. This repository only wrapps the cloud-scanner in a lambda.

The recommended serveless deployment is the following.

The scanner is deployed as a lambda inside a first region of your account. It performs inventory of other regions.

• You may restrict to specifc regions by tuning the role of the lambda defined in serverless.yml
• You can (and should) also deploy Boaviztapi as a lambda or other in your account. This allow to perform scans and evaluate impacts without any inventory data leaking outside your account. Once you have deployed your own instance of BoaviztAPI, tune the BOAVIZTA_API_URL in the serverless.yml file to ensure that cloud-scanner points to your private instance of API. See using private Boavizta API.

⚠ Open issue prevents Boavizta API deployment as Lambda (https://github.com/Boavizta/boaviztapi/issues/153). As a workaround we recommend deploying Boavizta API as a docker container (for example using AWS ECS) . You can still use cloud-scanner itself as serverless application deployed with Lambda.

Serverless routes

Instance impacts as JSON

The scan route returns individual instances metrics in json format (see below, same as CLI)

Use hours_use_time and aws_region parameters in the query

E.g.: https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/scan?hours_use_time=5&aws_region=eu-west-1

[
  {
    "instance_id": "i-03c8f84a6318a8186",
    "instance_type": "a1.medium",
    "usage_data": {
      "hours_use_time": 5,
      "time_workload": 0,
      "usage_location": "IRL"
    },
    "impacts": {
      "adp": {
        "manufacture": 0.0063,
        "unit": "kgSbeq",
        "use": 8e-12
      },
      "gwp": {
        "manufacture": 31,
        "unit": "kgCO2eq",
        "use": 0.0001
      },
      "pe": {
        "manufacture": 430,
        "unit": "MJ",
        "use": 0.004
      }
    }
  },
  {
    "instance_id": "i-033df52f12f30ca66",
    "instance_type": "m6g.xlarge",
    "usage_data": {
      "hours_use_time": 5,
      "time_workload": 0,
      "usage_location": "IRL"
    },
    "impacts": {
      "adp": {
        "manufacture": 0.0083,
        "unit": "kgSbeq",
        "use": 8e-10
      },
      "gwp": {
        "manufacture": 83,
        "unit": "kgCO2eq",
        "use": 0.01
      },
      "pe": {
        "manufacture": 1100,
        "unit": "MJ",
        "use": 0.4
      }
    }
  },
  {
    "instance_id": "i-0a3e6b8cdb50c49b8",
    "instance_type": "c5n.xlarge",
    "usage_data": {
      "hours_use_time": 5,
      "time_workload": 0,
      "usage_location": "IRL"
    },
    "impacts": {
      "adp": {
        "manufacture": 0.0086,
        "unit": "kgSbeq",
        "use": 1e-9
      },
      "gwp": {
        "manufacture": 64,
        "unit": "kgCO2eq",
        "use": 0.02
      },
      "pe": {
        "manufacture": 840,
        "unit": "MJ",
        "use": 0.5
      }
    }
  },
  {
    "instance_id": "i-003ea8da7bb9bfff9",
    "instance_type": "m6g.xlarge",
    "usage_data": {
      "hours_use_time": 5,
      "time_workload": 0,
      "usage_location": "IRL"
    },
    "impacts": {
      "adp": {
        "manufacture": 0.0083,
        "unit": "kgSbeq",
        "use": 8e-10
      },
      "gwp": {
        "manufacture": 83,
        "unit": "kgCO2eq",
        "use": 0.01
      },
      "pe": {
        "manufacture": 1100,
        "unit": "MJ",
        "use": 0.4
      }
    }
  }
]

Account impacts as metrics

The metrics route returns aggregated account metrics for one hour of use in Prometheus format.

Use aws_region parameters in the query.

E.g.: https://xxxxx.execute-api.eu-west-1.amazonaws.com/dev/metrics?aws_region=eu-central-1

# HELP boavizta_number_of_instances_total Number of instances detected during the scan.
# TYPE boavizta_number_of_instances_total gauge
boavizta_number_of_instances_total{awsregion="eu-central-1",country="DEU"} 7
# HELP boavizta_number_of_instances_assessed Number of instances that were considered in the measure.
# TYPE boavizta_number_of_instances_assessed gauge
boavizta_number_of_instances_assessed{awsregion="eu-central-1",country="DEU"} 5
# HELP boavizta_duration_of_use_hours Number of instances detected during the scan.
# TYPE boavizta_duration_of_use_hours gauge
boavizta_duration_of_use_hours{awsregion="eu-central-1",country="DEU"} 1.0
# HELP boavizta_pe_manufacture_megajoules Power consumed for manufacture.
# TYPE boavizta_pe_manufacture_megajoules gauge
boavizta_pe_manufacture_megajoules{awsregion="eu-central-1",country="DEU"} 1760.0
# HELP boavizta_pe_use_megajoules Power consumed during usage.
# TYPE boavizta_pe_use_megajoules gauge
boavizta_pe_use_megajoules{awsregion="eu-central-1",country="DEU"} 0.86
# EOF

See also Set up monitoring dashboard for an example of scrapping and displaying these metrics.

Limitations and perimeter

We do our best to offer accurate estimations of impacts. But these figures should still be considered with a grain of salt.

All models are wrong, but some are useful.

The Boavizta impact data result from a modelling effort. It is important to consider:

1. the limited Perimeter of the resources that cloud-scanner takes into account.
2. the limits of estimations methodology.

Perimeter / scope

Cloud scanner only estimates the impacts of your AWS EC2 instances, and optionally block storage.

Several significant aspects of the global impacts of cloud usage are excluded:

• It only estimate impacts of compute instances (EC2 Virtual Machines) and optionally block storage (HDD, SSD volumes).
• Cloud scanner does not take into account the PUE of the data center. See Add an option to override PUE in the queries or cli option · Issue #422 · Boavizta/cloud-scanner
• Cloud scanner does not take into account the surrounding cloud infrastructure (network, control plan).
• For storage (experimental feature), only the impacts of manufacture are counted. The impacts of the use phase are not counted. At the moment, Boavizta API returns only the impacts of the manufacturing phase for HDD and SSD. Furthermore the impacts of storage are likely overestimated. They are calculated by taking into account the size of the logical volume. In reality, a volume that is not full may not claim the full space on a physical device, which would result in smaller impacts.
• do not take into account the over-commit (mutualization) or over-provisioning(redundancy) that cloud provider may apply to provide the service.
• do not account managed services (like DB as a service or Containers as a service).
• do not account serverless (lambda) compute.
• unsupported instance types returns zero for their impacts.
• Cloud scanner does not provide error margins https://github.com/Boavizta/boaviztapi/issues/147.

Other limits

Instances supported

Cloud scanner supports all instances types of Boavizta API. But when a specific instance type cannot be matched with the Boavizta data set, Cloud-scanner returns empty impacts (i.e. zeroed impact values).

Carbon intensity of electricity is not real time

Carbon intensity of electricity is not real time. It uses and yearly average corresponding to the country where the data center is located. This is more related to the API itself, see https://github.com/Boavizta/boaviztapi/issues/117 and general Boavizta API methodology.

We do not provide margins of error

Cloud scanner is not yet able to provide error margin related to it’s estimation. Note that there is ongoing work on Boavizta API to provide more details about the error margins: https://github.com/Boavizta/boaviztapi/issues/147

Open issues / bugs

There may be additional identified issues or bugs, see Issues.

Testing

Unit tests

Unit tests are launched with cargo test command.

End to end tests

When launched with cargo test -- --include-ignored additional integration tests need:

• specific instance to be running
• aws credentials to be setup

These integration tests require specific instances to be up and running to pass. This means they are tied to a specific cloud account.

Memo: commands to start or stop instances:

# List instance state
aws ec2 describe-instance-status --include-all-instances --filters Name=instance-state-name,Values='*' --query 'InstanceStatuses[*].{InstanceId: InstanceId, State: InstanceState.Name}' --ouptut table

# start instance 
aws ec2 start-instances --instance-id i-03c8f84a6318a8186

# stop instance
aws ec2 stop-instances --instance-id i-03c8f84a6318a8186