Skip to content

grafana/Grafana-Alloy-for-Beginners

BranchesTags

Repository files navigation

image

Resources

Series overview

⬇️ Click below to watch the Grafana Alloy for Beginners series overview. Watch the video

image image image image image

What is Alloy? When does it make sense to use it?

💡 Watch the episode below to see what this section is all about! Watch the video image image image image image image

Alloy configuration language 101

🧩 Watch the episode below to learn the basics of Alloy’s configuration language and how to start building your own pipelines!

Watch the video

Think of Alloy as our trusty pal who can collect, process, and export our telemetry data.

Alt Text

To instruct Alloy on how we want that done, we must write these instructions in a language (Alloy syntax) that Alloy understands.

image image image image image image image image image image image image

The usage section gives you an example of how this particular component could be configured.

image

The arguments and blocks sections list what you could do with the data. Pay close attention to the name, type, description, default, and required columns so Alloy could understand what you want it to do!

image image

Focusing on these 3 things will point us in the right direction as we configure our pipeline. image

Learning environment setup

✨ Watch the episode below to learn how to set up your Grafana Alloy learning environment and get ready to build your first pipelines!

Watch the video image

Before getting started, make sure you:

git clone https://github.com/grafana/Grafana-Alloy-for-Beginners.git

To start the environment, run the following command from within the project's root directory:

make run

To stop the environment, run the following command from within the project's root directory:

make stop

Open the project using a text editor of your choice.

  • Expand the alloy folder and open the config.alloy file.
  • We will be using this file to build pipelines for Infrastructure Observability and Applications Observability.

Infrastructure Observability

Collect, process, and export infrastructure logs and metrics

Section 1: Build a pipeline for infrastructure logs with Alloy

🪵 Watch the episode below to learn how to build a pipeline for infrastructure logs using Grafana Alloy! Watch the video

Objectives

Instructions

Open config.alloy in your editor and copy the following starter code into it:

//Section 1

logging {
  format = "//TODO: Fill this in"
  level  = "//TODO: Fill this in"
  write_to = [//TODO: Fill this in]
}

loki.relabel "alloy_logs" {
   forward_to = [//TODO: Fill this in]

    rule {
        target_label = "//TODO: Fill this in"
        replacement = "//TODO: Fill this in"
    }

    rule {
        target_label = "//TODO: Fill this in"
        replacement = "//TODO: Fill this in" 
    }
}

loki.write "mythical" {
    endpoint {
       url = "//TODO: Fill this in"
    } 
}

Tasks

logging block:

  • set the log format to "logfmt"
  • set the log level to "debug"
  • send the logs to the receiver of theloki.relabel.alloy_logs component

loki.relabel component:

  • Use the rule block to
    • set the group label to "infrastructure"
    • set the service label to "alloy"
    • Note: These rules are applied in the order they are written!
  • forward the logs to the receiver of the loki.write.mythical component

loki.write component:

image

Expand to see the solution

 
//Section 1
logging {
    format = "logfmt"
    level  = "debug"

    write_to = [loki.relabel.alloy_logs.receiver]
}

loki.relabel "alloy_logs" {
    forward_to = [loki.write.mythical.receiver]

    rule {
        target_label = "group"
        replacement  = "infrastructure"
    }

    rule {
        target_label = "service"
        replacement  = "alloy" 
    }
}

loki.write "mythical" {
    endpoint {
        url = "http://loki:3100/loki/api/v1/push"
    } 
}

Reloading the config

Whenever we make changes to the file, we must reload the config.

To reload Alloy's config, hit the following endpoint in a browser or with a tool like curl:

curl -X POST http://localhost:12347/-/reload

If the config is valid, we should see a response like the following:

config reloaded

Verification

Navigate to the Dashboards page and select the Section 1 Verification dashboard.

You should see the panels populated with data, showing the number of logs being sent by Alloy as well as the logs themselves.

image

Expand a log line to view its labels.

You should see labels "group = infrastructure" and "service = alloy".

Section 2: Build a pipeline for infrastructure metrics with Alloy - Part I

📊 Watch the episode below to learn how to discover, collect, process, and export infrastructure metrics using Grafana Alloy!

Watch the video

Objectives

In this section, we will learn how to perform service discovery using Alloy. (discovery.http).

When we’re monitoring infrastructure or applications, we’re often working in dynamic environments where things are constantly changing. image

There could be 1000 servers or containers starting and stopping whose names and addresses are constantly changing. image

We want to avoid keeping up with ever changing list of sources that we need to collect telemetry from.

Instead of hard coding all the names and addresses of all the telemetry sources, what if there was a system that automatically tracked all the telemetry sources in our environment and Alloy could query it to discover what to collect from?

That’s exactly what service discovery components do. They are configured to query systems like the Kubernetes API or AWS EC2, which already track your infrastructure. Then they retrieve and format the list of discovered targets, exposing them to other Alloy components to collect telemetry from.

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 2
discovery.http "service_discovery" {
    url = "//TODO: Fill this in"
    refresh_interval = "2s"
}

prometheus.scrape "infrastructure" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets    = //TODO: Fill this in
    forward_to = [//TODO: Fill this in]
}

prometheus.remote_write "mimir" {
   endpoint {
    url = "//TODO: Fill this in"
   }
}

Tasks

discovery.http component:

  • Ping an HTTP within our lab environment in charge of finding targets("http://service-discovery/targets.json")
    • this http endpoint is aware of all instances of Loki, Tempo, Mimir, and Pyroscope databases that are currently running within our environment
  • Set the refresh_interval argument to 2 seconds for demo purposes

prometheus.scrape component:

  • Set the scrape interval and scrape timeout to 2 seconds
  • Scrape discovery.http.service_discovery component's target
  • Forward the metrics to prometheus.remote_write.mimir component's receiver

prometheus.remote_write component:

image

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 2
discovery.http "service_discovery" {
    url              = "http://service-discovery/targets.json" 
    refresh_interval = "2s"
}

prometheus.scrape "infrastructure" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets    = discovery.http.service_discovery.targets
    forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
    endpoint {
        url = "http://mimir:9009/api/v1/push"
    }
}

Verification

Navigate to the Dashboards page and select the Section 2 Verification dashboard.

You should see an up value of 1 for the Loki, Mimir, Tempo, and Pyroscope services.

If we see a 0, that indicates there has been an error somewhere.

image

Section 3: Build a pipeline for infrastructure metrics with Alloy - Part II

📈 Watch the episode below to learn how to expose, scrape, process, and export metrics from a Postgres database using Grafana Alloy!

Watch the video

Objectives

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 3
prometheus.exporter.postgres "mythical" {
    data_source_names = ["postgresql://postgres:mythical@mythical-database:5432/postgres?sslmode=disable"]
}

prometheus.scrape "postgres" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets    =  //TODO: Fill this in
    forward_to =  [//TODO: Fill this in]
}

prometheus.relabel "postgres" {
    forward_to =  [//TODO: Fill this in]

    rule {
        target_label = "//TODO: Fill this in"
        replacement  = "//TODO: Fill this in"
    }
    
    rule {
        target_label = "//TODO: Fill this in"
        replacement  = "//TODO: Fill this in"
    }

 //What we have: {instance="postgresql://mythical-database:5432/postgres"}
 //What we want: {instance="mythical-database:5432/postgres"}
    
    rule {
        // Replace a label's value.
        action        = "//TODO: Fill this in"

        // The label we want to replace is 'instance'.
        target_label  = "//TODO: Fill this in"

        // Look in the existing 'instance' label for anything starting with postgresql:// and capture what follows.
        source_labels = ["//TODO: Fill this in"]
        regex         = "^postgresql://(.+)"
        
        // Replace the value with just the captured part, leaving out the prefix.
        replacement   = "$1"
    }
}

Tasks

prometheus.exporter.postgres component:

  • Specify the url of a local Postgres database to connect to and expose metrics for a Postgres database ("postgresql://postgres:mythical@mythical-database:5432/postgres?sslmode=disable")

prometheus.scrape component:

  • Scrape the prometheus.exporter.postgres.mythical component's targets
  • Forward the metrics to the prometheus.relabel.postgres component's receiver

prometheus.relabel component:

  • Add the group="infrastructure" and service="postgres" labels to the metrics
  • Modify the instance label to clean it up
    • Before: postgresql://mythical-database:5432/postgres
    • After: mythical-database:5432/postgres
image

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 3
prometheus.exporter.postgres "mythical" {
    data_source_names = ["postgresql://postgres:mythical@mythical-database:5432/postgres?sslmode=disable"]
}

prometheus.scrape "postgres" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets    =  prometheus.exporter.postgres.mythical.targets
    forward_to =  [prometheus.relabel.postgres.receiver]
}

prometheus.relabel "postgres" {
    forward_to =  [prometheus.remote_write.mimir.receiver]

    rule {
        target_label = "group"
        replacement  = "infrastructure"
    }

    rule {
        target_label = "service"
        replacement  = "postgres"
    }

     rule {
        // Replace a label's value.
        action        = "replace"

        // The label we want to replace is 'instance'.
        target_label  = "instance"

        // Look in the existing 'instance' label for anything starting with postgresql:// and capture what follows.
         source_labels = ["instance"]
        regex         = "^postgresql://(.+)"
        
        // Replace the value with just the captured part, leaving out the prefix.
        replacement   = "$1"
    }
}

Verification

Navigate to Dashboards and select Section 3 Verification. We should see a dashboard populating with Postgres metrics.

We should also see an instance value of mythical-database:5432/postgres instead of postgresql://mythical-database:5432/postgres.

image

How to use the Alloy UI to debug pipelines

🐞 Watch the episode below to learn how to debug pipelines visually with the Alloy UI!

Watch the video

Alloy UI is a useful tool that helps you visualize how Alloy is configured and what it is doing so you are able to debug efficiently.

Navigate to localhost:12347 to see the list of components (orange box) that alloy is currently configured with. Click on the blue ‘view’ button on the right side (red arrow). image

This page shows us the health of the component, the arguments it's using, and its current exports (green box).

This page also gives us quick access to the component’s documentation (orange arrow) and a Live Debugging view (yellow arrow). image

When we click on the Live Debugging view, we will be able to see a real-time stream of telemetry flowing through a component. image

Navigate to the ‘Graph’ tab (blue arrow) to access the graph of components and how they are connected. image

The number (pink box) shown on the dotted lines shows the rate of transfer between components. The window at the top (orange box) configures the interval over which alloy should calculate the per-second rate, so a window of ‘10’ means that alloy should look over the last 10 seconds to compute the rate.

The color of the dotted line signifies what type of data are being transferred between components. See the color key (purple box) for clarification.

Navigate to the Clustering tab (blue arrow) to view the instances of Alloy.

image

A cluster node is an instance of Alloy that participates in workload distribution and ensures high availability.

The Clustering page in the Alloy UI shows the status and role of each node, so you can easily monitor which nodes are active, their addresses, and its current state.

To debug the piplines using the Alloy UI

  • Ensure that no component is reported as unhealthy.
  • Ensure that the arguments and exports for misbehaving components appear correct.
  • Ensure that the live debugging data meets your expectations.

Application Observability

Collect, transform, and export application metrics, traces, and logs

Section 4: Build a pipeline for application metrics with Alloy

🔧 Watch the video below to learn how to build your appplication metrics pipeline with Alloy!

Watch the video

Objectives

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 4
prometheus.scrape "mythical" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets    =  [
        {"__address__"= "//TODO: Fill this in", group = "//TODO: Fill this in", service = "//TODO: Fill this in"},
        {"//TODO: Fill this in"}, 
        ]
    forward_to =  [//TODO: Fill this in]
}

prometheus.write.queue "experimental" {
    endpoint "mimir" {
        url = "//TODO: Fill this in"
    }
}

Tasks

prometheus.scrape component:

  • Define scrape targets for mythical services directly by creating a scrape object.
    • Scrape targets are defined as a list of maps, where each map contains a __address__ key with the address of the target to scrape.
    • Any non-double-underscore keys are used as labels for the target.
      • For example, the following scrape object will scrape Mimir's metrics endpoint and add env="demo" and service="mimir" labels to the target:
targets = [{"__address__" = "mimir:9009",  env = "demo", service = "mimir"}]
  • create two targets using the following addresses.
    • "mythical-server:4000"
    • "mythical-requester:4001"
  • Add the following labels for each target.
    • mythical-server:
    • group = "mythical", service = "mythical-server"
    • mythical-requester:
      • group = "mythical", service = "mythical-requester"
  • Forward the metrics to the prometheus.write.queue component we will define next.

prometheus.write.queue component:

image

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 4
prometheus.scrape "mythical" {
    scrape_interval = "2s"
    scrape_timeout  = "2s"

    targets = [
        {"__address__" = "mythical-server:4000", group = "mythical", service = "mythical-server"},
        {"__address__" = "mythical-requester:4001", group = "mythical", service = "mythical-requester"}, 
    ]

    forward_to = [prometheus.write.queue.experimental.receiver]
}

prometheus.write.queue "experimental" {
    endpoint "mimir" {
        url = "http://mimir:9009/api/v1/push"
    }
}

Verification

Navigate to Dashboards > Section 4 Verification and we should see a panel with the request rate per beast flowing!

image

Section 5: Build a pipeline for application traces with Alloy

🚀 Watch the video below to learn how to build your appplication traces pipeline with Alloy! Watch the video

Objectives

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 5
otelcol.receiver.otlp "otlp_receiver" {
    grpc {
        endpoint = "//TODO: Fill in the default value shown in the grpc block section of the otelcol.receiver.otlp doc"
    }
    http {
        endpoint = "//TODO: Fill in the default value shown in the http block section of the otelcol.receiver.otlp doc"
    }
    output {
        traces = [
            //TODO: Fill this in,
        ]
    }
}

otelcol.processor.batch "default" {
    output {
        traces = [
            //TODO: Fill this in,
            ]
    }

    send_batch_size = //TODO: Fill this in 
	  send_batch_max_size = //TODO: Fill this in

	  timeout = "//TODO: Fill this in"
}

otelcol.exporter.otlp "tempo" {
    client {
        endpoint = "//TODO: Fill this in"

        // This is a local instance of Tempo, so we can skip TLS verification
        tls {
            insecure             = true
            insecure_skip_verify = true
        }
    }
}

otelcol.receiver.otlp component:

  • Open the doc for the otelcol.receiver.otlp component
  • Find the default port for grpc and set its endpoint equal to it
  • Find the default port for http and set its endpoint equal to it
  • Using the output block, send the traces to the input of the otelcol.processor.batch component we will define next

otecol.processor.batch component:

  • The batch processor will batch spans until a batch size or a timeout is met, before sending those batches on to another component
  • Configure it to batch minimum 1000 spans, up to 2000 spans, or until 2 seconds have elapsed
  • Using the output block, send the batched traces to the input of the otelcol.exporter.otlp component we will define next

otelcol.exporter.otlp component:

  • Using the client block, export batches of spans to a local instance of Tempo
  • The Tempo url is "http://tempo:4317"
image

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 5
otelcol.receiver.otlp "otlp_receiver" {
    grpc {
        endpoint = "0.0.0.0:4317"
    }
    http {
        endpoint = "0.0.0.0:4318"
    }
    output {
        traces = [
            otelcol.processor.batch.default.input,
            otelcol.connector.spanlogs.autologging.input,
        ]
    }
}

otelcol.processor.batch "default" {
    output {
        traces = [
            otelcol.exporter.otlp.tempo.input,
        ]
    }

    send_batch_size     = 1000
    send_batch_max_size = 2000

    timeout = "2s"
}

otelcol.exporter.otlp "tempo" {
    client {
        endpoint = "http://tempo:4317"

        // This is a local instance of Tempo, so we can skip TLS verification
        tls {
            insecure             = true
            insecure_skip_verify = true
        }
    }
}

Verification

Navigate to Dashboards > Section 5 Verification and you should see a dashboard with a populated service graph, table of traces coming from the mythical-requester, and the rate of span ingestion by Tempo

image

You can also navigate to Dashboards > MLT Dashboard. These dashboards are configured to use the metrics from Spanmetrics, so you should see data for the spans we're ingesting.

image

Section 6: Build a pipeline for application logs with Alloy

💡 Watch the video below to learn how to build your appplication logs pipeline with Alloy! Watch the video

Objectives

  • Ingest application logs using the loki.source.api component
  • Add labels to logs using the loki.process component
  • Use stage.regex and stage.timestamp to extract the timestamp from the log lines and set the log’s timestamp
image image

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 6
loki.source.api "mythical" {
     http {
        listen_address = "0.0.0.0"
        listen_port    = "3100"
    }
    forward_to = [//TODO: Fill this in]
}

loki.process "mythical" {
    stage.static_labels {
        values = {
           //TODO: Fill this in = "//TODO: Fill this in",        
        }
    }
   stage.regex {
        expression=`^.*?loggedtime=(?P<loggedtime>\S+)`
   }

   stage.timestamp {
        source = "//TODO: Fill this in"
        format = "2006-01-02T15:04:05.000Z07:00"
    }

    forward_to = [loki.write.mythical.receiver]
}

Tasks

loki.source.api component:

  • Ingest application logs sent from the mythical services

loki.process component:

  • add a static `service="mythical" label
  • extract the timestamp from the log line using stage.regex with this regex: ^.*?loggedtime=(?P<loggedtime>\S+)
  • set the timestamp of the log to the extracted timestamp
  • Forward the processed logs to Loki

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 6
loki.source.api "mythical" {
    http {
        listen_address = "0.0.0.0"
        listen_port    = "3100"
    }
    forward_to = [loki.process.mythical.receiver]
}

loki.process "mythical" {
    stage.static_labels {
        values = {
           service = "mythical",
        }
    }
    stage.regex {
            expression=`^.*?loggedtime=(?P<loggedtime>\S+)`
    }

    stage.timestamp {
        source = "loggedtime"
        format = "2006-01-02T15:04:05.000Z07:00"
    }

    forward_to = [loki.write.mythical.receiver]
}

Verification

Navigate to Dashboards > Section 6 Verification and you should see a dashboard with the rate of logs coming from the mythical apps as well as panels showing the logs themselves for the server and requester

image

Section 7: Generate logs from application traces with Alloy

🌟 Watch the video below to learn how to generate logs from application traces with Alloy! Watch the video

Objectives

Instructions

Open config.alloy in your editor and copy the following code into it:

//Section 7
otelcol.connector.spanlogs "autologging" {
    roots = // TODO: Fill this in
    spans = // TODO: Fill this in
    processes = // TODO: Fill this in

    span_attributes = ["// TODO: Fill this in", "// TODO: Fill this in", "// TODO: Fill this in"]
    //these are the span attributes that I would like to include in the logs

    output {
    logs = [// TODO: Fill this in]
  }
}

otelcol.exporter.loki "autologging" {
    forward_to = [// TODO: Fill this in]
}

// The Loki processor allows us to accept a Loki-formatted log entry and mutate it into
// a set of fields for output.
loki.process "autologging" {
    stage.json {
       expressions = {"body" = ""}
    }

    stage.output {
       source = "body"
    }

    stage.logfmt {
        mapping = {
            http_method_extracted = "// TODO: Fill this in",
            http_status_code_extracted = "// TODO: Fill this in", 
            http_target_extracted = "// TODO: Fill this in", 

        }
    }

    stage.labels {
        values = {
            method = "// TODO: Fill this in", 
            status = "// TODO: Fill this in",
            target = "// TODO: Fill this in", 
        }
    }

    forward_to = [// TODO: Fill this in]
}

Tasks

otelcol.connector.spanlogs component:

  • Forward the spans from the otelcol.receiver.otlp's output > traces we have defined in section 5 to the otelcol.connector.spanlogs's input.
  • Generate a log for each full trace(root), not for each span or process
  • Include the http.method,http.status_code, http.target attributes in the logs.
  • Send the generated logs to the otelcol.exporter.loki's input.

otelcol.exporter.loki component:

  • This component accepts OTLP-formatted logs from other otelcol components and converts them to Loki-formatted log entries without further configuration.
  • Forward the Loki-formatted logs to the loki.process "autologging"'s receiver for further processing.

loki.process component:

  • Convert the body from JSON to logfmt using the stage.json and stage.logfmt stages
  • Add the method, status, and target labels from the http.method, http.status_code, and http.target attributes
image

Don't forget to reload the config after finishing.

Expand to see the solution

 
//Section 7
otelcol.connector.spanlogs "autologging" {
    roots = true
    spans = false
    processes = false

    span_attributes = ["http.method", "http.target", "http.status_code"]

    output {
        logs = [otelcol.exporter.loki.autologging.input]
    }
}

otelcol.exporter.loki "autologging" {
    forward_to = [loki.process.autologging.receiver]
}

loki.process "autologging" {
    stage.json {
       expressions = {"body" = ""}
    }

    stage.output {
       source = "body"
    }

    stage.logfmt {
        mapping = {
            http_method_extracted      = "http.method",
            http_status_code_extracted = "http.target",
            http_target_extracted      = "http.status_code",

        }
    }

    stage.labels {
        values = {
            method = "http_method_extracted",
            status = "http_status_code_extracted",
            target = "http_target_extracted",
        }
    }

    forward_to = [loki.write.mythical.receiver]
}

Verification

Navigate to Dashboards > Section 7 Verification and you should see a dashboard with panels containing the rate of spanlog ingestion as well as the spanlogs themselves.

image image

Hands-on Exercises

🕵️‍♂️ Take a look at these videos to uncover the mission and how the solutions come together.

Watch the video Watch the video

Mission 1: The Hidden Key

Description

One of our trusted informants has stashed an encrypted file—secret_message.txt.enc—on a remote dead-drop. This file can be found within the series repo.

The decryption key? Hidden in plain sight, embedded in an internal label on the service discovery targets. Since internal labels are stripped before metrics make it to Mimir, this covert tactic kept the key out of enemy hands.

Your mission: use Alloy to uncover the hidden key, decrypt the message, and reveal the intel within.

Objectives

  • Use the Alloy UI to find the key hidden in the internal label on the service discovery targets
  • Decode the key and decrypt the secret message

Instructions

Access the Alloy UI and look for the hidden key on one of the service discovery targets.

To decrypt and print the AES-256-CBC encrypted secret message, run the following command in the terminal at the root of the project directory, using the key you just found: openssl enc -aes-256-cbc -d -salt -pbkdf2 -in secret_message.txt.enc -k '<key>'

Verification

You should see the secret message in the console!

Mission 2: The Cardinality Crisis

Description

A rogue actor has tampered with IMF's monitoring systems, slipping a high-cardinality instance_id label into a metric that counts database calls.

This unexpected spike in cardinality is putting Mimir under serious pressure -- and it's up to us to defuse the situation before it blows.

You can see the dashboard that informed the IMF that this was happening by navigating to Dashboards > Mission 2.

But it's not all bad news. Hidden within the instance_id is valuable intel: the name of the cloud provider. IMF wants us to extract that information and promote it to a dedicated cloud_provider label—transforming this mess into a mission success.

IMF has equipped you with the following regex to help you complete this mission: ^(aws|gcp|azure)-.+

Objectives

  • Using prometheus.relabel, use the provided regex to replace the cloud_provider label with the extracted value from the instance_id label.
  • Drop the instance_id label.

Instructions

For this exercise, you may find the following components useful:

Go back to the portion of config from Section 4, where we started scraping metrics from the mythical services. Paste the following above the prometheus.write.queue component (note: the order of components does not matter, this is just for organization and readability):

prometheus.relabel "mission_2" {
    forward_to = [prometheus.write.queue.experimental.receiver]

  //define a relabel rule to extract the cloud provider from the instance_id label and add it as a new label called cloud_provider
    rule {
        action        = "// TODO: Fill this in"
        target_label  = "// TODO: Fill this in"
        source_labels = ["// TODO: Fill this in"]
        regex         = "^(aws|gcp|azure)-.+"
        replacement   = "$1"
    }

    // drop the instance_id label from metrics
    rule {
        action  = "// TODO: Fill this in"
        regex   = "// TODO: Fill this in"
    }
}

Verification

Navigate to the Explore page and look at the metrics. Query for count by (cloud_provider) (rate(mythical_db_request_count_total [$__rate_interval])) and you should see a non-zero value.

image

Mission 3: Attribute Alignment

After much debate, the various departments within IMF have reached a rare consensus: it's time to standardize the attribute name for service tiers.

Until now, teams have been using conflicting keys like servicetier and tier, creating chaos in spanmetrics and cross-department dashboards.

Headquarters has spoken: service.tier is the new standard.

Your mission: use Alloy to bring order to the data. Standardize the attribute across the board so that spanmetrics flow smoothly and dashboards speak a common language.

Objectives

  • Use the otelcol.processor.attributes component to set the service.tier attribute to the value of the servicetier or tier attributes.
  • Drop the servicetier and tier attributes.

Instructions

The otelcol.processor.attributes component allows you to add, set, or drop attributes.

Go back to the portion of config from Section 5, where we received traces from the mythical services. Paste the following above the otelcol.processor.batch.default component (note: the order of components does not matter, this is just for organization and readability):

otelcol.processor.attributes "mission_3" {
    // These two actions are used to add the service.tier attribute to spans from
    // either the servicetier or tier attributes.
    action {
        action         = "//TODO: Fill this in"
        key            = "//TODO: Fill this in"
        from_attribute = "//TODO: Fill this in"
    }
    action {
        action         = "//TODO: Fill this in"
        key            = "//TODO: Fill this in"
        from_attribute = "//TODO: Fill this in"
    }

    // This isn't required, but shows how to exclude the attributes we just copied.
    exclude {
        match_type = "strict"

        attribute {
            key = "//TODO: Fill this in"
        }

        attribute {
            key = "//TODO: Fill this in"
        }
    }

    output {
        traces = [otelcol.processor.batch.default.input]
    }
}

Verification

Navigate to Dashboards > Mission 3 and you should see a dashboard with data including the new service_tier label, which came from spanmetrics generation using the service.tier attribute we just consolidated.

image

Mission 4: Redact and Protect

Description

The IMF needs your expertise for one final mission.

An opposing state actor exploited a Zero-Day vulnerability in one of our servers, causing sensitive tokens to be logged by the mythical-requester.

The security team is standing by, but before they can act, we need to make sure no tokens are being written to Loki.

Your task: use Alloy to identify and redact any sensitive tokens from the mythical-service logs—effectively, clean up the trail and keep things secure.

Navigate to Dashboards > Mission 4. You will see logs coming in with sensitive token information.

Objectives

  • Redact any tokens found in the logs from the mythical services

Instructions

Take a look at the loki components. Are there any that seem like they could be useful for this mission?

Which section would you add this component to and how would you have to change the previous configuration?

Verification

Navigate to Dashboards > Mission 4 and you should see a dashboard with a panel showing the rate of logs with tokens coming from the mythical services as well as the logs themselves with the secret token redacted.

image

About

No description, website, or topics provided.

Resources

Readme

License

AGPL-3.0 license

Code of conduct

Code of conduct

Security policy

Security policy
Activity
Custom properties

Stars

43 stars

Watchers

92 watching

Forks

8 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages