Integrating Workflows with CI/CD
Last updated
Was this helpful?
Last updated
Was this helpful?
Related Guides and Examples:
: High-level concepts and options.
: How to set up managed workflows.
: How to define and enforce policies and best practices.
: Real-world pipeline YAMLs and integration templates for Jenkins, GitHub Actions, and more.
Firefly Workflows can be integrated into your existing CI/CD pipelines for Terraform and OpenTofu. This integration allows you to gain enhanced visualization and monitoring of your current deployment processes directly within Firefly, improving efficiency and visibility without requiring a disruptive overhaul of your established workflow setup. This document provides a comprehensive guide on how to achieve this integration.
Integrating your existing CI/CD pipeline with Firefly Workflows offers several benefits:
Enhanced Visibility: Gain a centralized view of your Terraform/OpenTofu operations (plans and applies) across all your environments.
Monitoring & Insights: Track deployment history, understand changes over time, and leverage Firefly's analytical capabilities.
Non-Disruptive: Continue using your existing CI/CD tools and processes. Firefly augments your setup by collecting and displaying relevant data.
Streamlined Collaboration: Share deployment information easily across teams.
This integration primarily uses the fireflyci
tool, which can be run as a Docker container or a standalone CLI binary. It sends data from your terraform plan
and terraform apply
steps to Firefly.
Supported IaC Tools:
Terraform
OpenTofu
Before you begin, ensure you have the following:
An active Firefly account.
An existing CI/CD pipeline that executes terraform
or tofu
commands.
Docker installed and available in your CI/CD environment if you choose the Docker-based integration for fireflyci
.
Access to modify your CI/CD pipeline configuration to add steps for fireflyci
.
To send data to Firefly, the fireflyci
tool must be authenticated using an Access Key and a Secret Key.
Log in to your Firefly account.
Navigate to the appropriate section to generate a new API key pair (Access Key and Secret Key). Refer to Firefly's UI documentation for the exact steps.
Securely store these keys. They will be used to configure authentication in your CI/CD pipeline.
fireflyci
There are a few ways to provide the keys to fireflyci
:
Environment Variables (Recommended for CI/CD): Set the following environment variables in your CI/CD pipeline's execution environment:
FIREFLY_ACCESS_KEY
: Your Firefly Access Key.
FIREFLY_SECRET_KEY
: Your Firefly Secret Key.
This is the most common and secure method for CI/CD systems.
CLI Arguments: You can pass the keys directly as command-line arguments:
--access-key YOUR_ACCESS_KEY
--secret-key YOUR_SECRET_KEY
This method is generally less secure for CI/CD pipelines as keys might be logged.
Configuration Profile:
The fireflyci
CLI supports a configure
command to set up named profiles:
Then use --profile myprofile
with post-plan
or post-apply
commands. This is more suitable for local development rather than CI/CD.
It is crucial to store your Firefly Access Key and Secret Key securely. Most CI/CD systems provide a mechanism for managing secrets:
GitHub Actions:
Go to your GitHub repository > Settings
> Secrets and variables
> Actions
.
Select New repository secret
.
For the Name, enter FIREFLY_ACCESS_KEY
and paste the Access Key value into the Secret field.
Select Add secret
.
Repeat steps 2-4 for FIREFLY_SECRET_KEY
.
Azure Pipelines:
Go to your Azure DevOps project > Pipelines
> Library
.
Select + Variable group
.
Enter a name for the variable group (e.g., FireflySecrets
).
Select + Add
to add variables.
Add FIREFLY_ACCESS_KEY
with its value, ensuring you mark it as a secret (by selecting the lock icon).
Add FIREFLY_SECRET_KEY
with its value, marking it as a secret.
Select Save
.
To use this variable group in your pipeline, you'll need to link it in your YAML definition or pipeline settings.
After adding the pipeline, grant it access to your new variable group: Select Library
> Pipeline permissions
> +
icon > Select the pipeline from the dropdown menu.
Consult your CI/CD tool's documentation for best practices on managing secrets.
fireflyci
You can run fireflyci
either as a Docker image or by installing the CLI directly. Using the Docker image is generally recommended for CI/CD environments for consistency and ease of setup.
The fireflyci
CLI is available as a Docker image from Amazon ECR Public Gallery:
public.ecr.aws/firefly/fireflyci:latest
You will use docker run
commands in your CI/CD pipeline to execute fireflyci post-plan
and fireflyci post-apply
.
fireflyci
CLIIf you prefer to install the CLI directly on your CI runners:
Download the relevant CLI for your operating system:
MacOS:
Linux:
Microsoft Windows:
Using cURL (Example for Linux x86): You can download and extract the CLI using cURL and tar:
Move to PATH (Optional but Recommended):
To make the fireflyci
command accessible from anywhere, move the extracted binary to a directory in your system's $PATH
:
Adjust the destination directory based on your environment and permissions.
Follow these steps to integrate Firefly into your existing Terraform/OpenTofu CI/CD pipeline:
terraform plan
CommandYour terraform plan
(or tofu plan
) command needs to be modified to output the plan and its logs in JSON format that fireflyci
can process.
Modify your plan command to generate two files:
plan_log.json
: The JSON-formatted log output from terraform plan -json
.
plan.json
: The JSON representation of the plan file itself, obtained using terraform show -json
.
Example:
tf.plan
: This is a binary file containing the execution plan.
plan_log.json
: Contains the structured JSON log output from the plan
command. This will be sent to Firefly using the -l
flag in fireflyci post-plan
.
plan.json
: Contains the JSON representation of the plan, which details the changes to be made. This will be sent to Firefly using the -f
flag in fireflyci post-plan
.
post-plan
StepAfter your terraform plan
command successfully executes and generates plan_log.json
and plan.json
, add a new step to run fireflyci post-plan
. This step should run before terraform apply
.
Purpose: This command sends the plan data, logs, and associated metadata to Firefly for visualization and analysis.
Workspace Name (-w <WORKSPACE_NAME>
):
You must specify a workspace name. This name is how Firefly identifies and groups data for this specific Terraform configuration and environment.
It should be unique for each distinct Terraform workspace or deployment target (e.g., my-app-dev
, my-app-prod
, shared-vpc-staging
).
Naming workspaces appropriately helps manage and distinguish between different deployment stages within Firefly.
Using Docker (Recommended):
--rm
: Removes the container after it exits.
-e FIREFLY_ACCESS_KEY -e FIREFLY_SECRET_KEY
: Passes your Firefly credentials as environment variables to the container.
The $(printenv ...)
part is an example of how you might pass through relevant CI/CD environment variables. Adjust this based on your CI system (see "Passing CI/CD Environment Variables" section).
-v $(pwd):/app/cicd
: Mounts the current working directory (where plan_log.json
and plan.json
are located) into the container at /app/cicd
.
-l /app/cicd/plan_log.json
: Specifies the path inside the container to the plan log JSON file.
-f /app/cicd/plan.json
: Specifies the path inside the container to the plan JSON file.
-w "<YOUR_UNIQUE_WORKSPACE_NAME>"
: Your chosen workspace name.
Using Installed CLI:
If you installed fireflyci
directly:
terraform apply
CommandYour terraform apply
(or tofu apply
) command needs to be modified to output its logs in JSON format.
Example:
apply_log.json
: Contains the structured JSON log output from the apply
command. This will be sent to Firefly using the -f
flag in fireflyci post-apply
.
We use the tf.plan
file generated in Step 1 to ensure the apply operation matches the plan sent to Firefly.
If you are not using a plan file with apply (e.g., terraform apply -auto-approve -json > apply_log.json
), ensure consistency.
post-apply
StepAfter your terraform apply
command successfully executes and generates apply_log.json
, add a new step to run fireflyci post-apply
.
Purpose: This command sends the apply logs and results to Firefly, associating them with the previously sent plan for the same workspace.
Workspace Name (-w <WORKSPACE_NAME>
):Crucially, use the exact same <WORKSPACE_NAME>
that you used in the fireflyci post-plan
step. This ensures Firefly links the apply operation to the correct plan.
Using Docker (Recommended):
Using Installed CLI:
When running fireflyci
inside a Docker container, it's often necessary to pass through certain environment variables from your CI/CD system. Firefly uses these variables to gather contextual information about the run (e.g., repository URL, branch name, commit SHA, PR details).
The fireflyci
agent automatically tries to detect the CI/CD provider and collect relevant variables. However, when running in Docker, these variables need to be explicitly passed into the container's environment using the -e VAR_NAME
flag with docker run
.
Here are common environment variables used by Firefly, categorized by CI/CD tool. You should include these in your docker run
command for fireflyci post-plan
and fireflyci post-apply
:
GitLab CI:
GitHub Actions:
Atlantis:
Semaphore:
Azure Pipelines:
Bitbucket Pipelines:
Env0:
Jenkins:
Alternatively, Jenkins often allows passing all environment variables.
Example Jenkins Jenkinsfile
(scripted pipeline) snippet for post-plan
:
And for post-apply
:
The generic $(printenv | grep ...)
command provided in the Docker examples earlier attempts to capture many common CI variables. You may need to customize this based on your specific CI/CD system and the variables it exposes.
fireflyci
Command ReferenceBelow are details on the post-plan
and post-apply
commands and their flags.
fireflyci post-plan
Sends Terraform/OpenTofu plan data to Firefly.
Usage:fireflyci post-plan [flags]
Key Flags:
--config-file string
: Path to a config file (YAML).
-h, --help
: Help for post-plan
.
-i, --init-log-file string
: Path to the terraform init
log file (e.g., init.log
).
-f, --plan-file string
: Path to the JSON plan file to scan (e.g., plan.json
or plan_output.json
). Required.
-l, --plan-output-json-log-file string
: Path to the plan output log file in JSON lines format (e.g., plan_log.json
or plan_log.jsonl
). Required.
--plan-output-raw-log-file string
: Path to the raw plan output log file (e.g., plan_output_raw.log
).
-r, --redact
: Redact plan file data (default true
).
-n, --redaction-workers int
: Number of concurrent workers for redaction (default 10
).
--repository string
: Full repository name (e.g., your-org/your-repo
). Often filled automatically if running from a CI/CD pipeline with environment variables passed correctly.
--context string
: Terraform Working directory. If both context
and work-dir
are provided, context
takes precedence.
--work-dir string
: Terraform Working directory.
--silent-iac-logs
: Skip printing IaC logs to stdout/stderr.
-t, --timeout int
: Timeout in seconds for the scan to complete (only relevant when --wait
is set) (default 180
).
--wait
: Waits for Firefly scans (e.g., policy checks if configured) to finish.
-w, --workspace string
: Workspace identifier. Required.
Global Flags (apply to all fireflyci
commands):
-a, --access-key string
: Your Firefly Access Key.
-s, --secret-key string
: Your Firefly Secret Key.
-p, --profile string
: Profile to use from your Firefly CLI configuration (default "default"
).
--server-url string
: FireflyCI API URL (default "https://prodapi.gofirefly.io/api"
).
--fail-on-internal-errors
: Exit with an error code if an internal error is encountered by fireflyci
.
--fail-on-rate-limit
: Exit with an error code if a rate limit is encountered.
--ignore-errors
: Exit with a success code even if errors are encountered during fireflyci
execution (e.g., communication issues with Firefly API). This does not affect the exit code of your Terraform commands themselves.
--pretty
: Pretty-print JSON output from fireflyci
.
fireflyci post-apply
Sends Terraform/OpenTofu apply data to Firefly.
Usage:fireflyci post-apply [flags]
Key Flags:
-f, --apply-log-file string
: Path to the apply log file in JSON lines format (e.g., apply_log.json
or apply_log.jsonl
). Required.
-h, --help
: Help for post-apply
.
-r, --redact
: Redact the apply log file data (default true
).
-n, --redaction-workers int
: Number of concurrent workers for redaction (default 10
).
--repository string
: Full repository name. Often filled automatically.
--silent-iac-logs
: Skip printing the IaC log file to stdout/stderr.
-t, --timeout int
: Timeout in seconds for the post-apply task to complete (only relevant when --wait
is set) (default 180
).
--wait
: Waits for the post-apply tasks on Firefly's side to finish.
-w, --workspace string
: Workspace name. Must match the one used in post-plan
. Required.
Global Flags: (Same as for post-plan
)
When Firefly is integrated with your Version Control System (VCS) provider (e.g., GitHub, GitLab), it can automatically add comments to your pull requests (PRs) or merge requests (MRs). These comments summarize the terraform plan
output, including:
Cost estimation changes.
Policy violations (if Guardrails are configured).
Resource changes (creations, updates, deletions).
A link to the full plan details within the Firefly platform.
To enable this feature, ensure your VCS is properly integrated with Firefly through the Firefly application settings.
Firefly provides a GitHub Custom Action that wraps the fireflyci
Docker tool, simplifying its use in GitHub Actions workflows. This action can handle setting up fireflyci
and calling post-plan
or post-apply
with appropriate inputs.
For detailed instructions on how to use the Firefly GitHub Custom Action, please refer to the official Firefly documentation or search for "Firefly CI" on the GitHub Marketplace.
Q: What happens if Firefly's endpoint is unavailable during my CI/CD run?
A: If Firefly's endpoint is unavailable, your CI/CD pipeline will operate normally (i.e., terraform plan
and apply
will still execute as defined in your pipeline). The fireflyci
tool might fail to send data to Firefly, or it might retry for a period. Deployment updates, logs, and statuses will not appear in the Firefly application or dashboard until the endpoint is accessible again and data can be successfully transmitted. Pipeline execution itself (the actual infrastructure changes) should remain unaffected by Firefly's availability, assuming fireflyci
is configured not to block the pipeline on its own errors (e.g., by not using --fail-on-internal-errors
or by handling its exit code).
Q: Are there any limitations on the size or complexity of Terraform projects that Firefly Workflows can handle when integrated this way?
A: Firefly Workflows are designed to accommodate Terraform projects of varying sizes and complexity levels. When integrating an existing pipeline, fireflyci
primarily sends log and plan data. The performance impact on your CI/CD pipeline is generally minimal, related to generating the JSON outputs and the fireflyci
execution time (which involves data transmission and redaction). For extremely large plans, ensure your CI runner has adequate resources. There are no strict limitations imposed by Firefly Workflows on project size through this integration method.
Q: Do Firefly Workflows support multi-cloud deployments with this integration?
A: Yes, Firefly Workflows support multi-cloud deployments. If your Terraform code manages resources across multiple cloud providers (e.g., AWS, Azure, GCP), Firefly will visualize and monitor these deployments as reported by your terraform plan
and apply
logs.
Q: How do Firefly Workflows handle state management and synchronization across multiple environments when integrated?
A: When integrating an existing CI/CD pipeline, Firefly Workflows do not manage your Terraform state. Your existing state management configuration (e.g., S3 backend, Terraform Cloud backend) remains in place and is managed by your Terraform setup. Firefly leverages Terraform's state management features by observing the outcomes of plan
and apply
operations. Each Firefly workspace you define (via the -w
flag) corresponds to a specific Terraform configuration and its associated state.
Q: My fireflyci
step is failing. What should I check?
A:
Authentication: Ensure FIREFLY_ACCESS_KEY
and FIREFLY_SECRET_KEY
are correctly set and available to the fireflyci
process.
File Paths: Verify that the paths to your plan_log.json
, plan.json
, and apply_log.json
files are correct, especially when using Docker (paths are relative to the container's file system).
Terraform Output: Confirm that your terraform plan
and terraform apply
commands are correctly generating the JSON output files in the expected format.
Network Connectivity: Ensure your CI runner can reach the Firefly API endpoint (https://prodapi.gofirefly.io/api
by default). Check for firewall rules or proxy configurations.
fireflyci
Version: Ensure you are using a recent version of fireflyci
.
Verbose Logs: Check the logs from the fireflyci
command itself for any specific error messages.
CI Environment Variables: If using Docker, double-check that all necessary CI/CD environment variables are being passed into the container.
Q: Can I use this integration with OpenTofu?
A: Yes, the integration process and fireflyci
tool are compatible with OpenTofu. Simply replace terraform
commands with tofu
in your CI/CD scripts (e.g., tofu plan -json ...
, tofu show -json ...
, tofu apply -json ...
).
ARM:
x86:
ARM:
x86:
x86:
Tip: For real-world pipeline YAMLs and integration templates for Jenkins, GitHub Actions, Azure Pipelines, Bitbucket, and more, see the .
Note: The Jenkins examples from the gofireflyio/workflows-examples
repo use slightly different file names (plan_log.jsonl
, plan_output.json
, apply_log.jsonl
) than the simplified integration example (plan_log.json
, plan.json
, apply_log.json
). Adapt your Terraform output commands and fireflyci
arguments accordingly if you use these more detailed Jenkins examples. For more Jenkins, GitHub Actions, and other CI/CD templates, visit .
See also: The contains ready-to-use YAMLs and scripts for many CI/CD systems, which can help troubleshoot and accelerate your integration.