Workflows

Streamline Terraform deployments in your CI/CD pipelines.

Firefly Workflows is a powerful tool used to deploy Terraform within your CI/CD pipelines. Our documentation provides comprehensive guidance on creating and customizing workflows, in addition to integrating with existing pipelines.

Determining which type of workflow to integrate

The decision between creating a new workflow or integrating into an existing one depends on the specific requirements and structure of your project and CI/CD environment.

Managed workflows

Create a new workflow in Firefly when you need to establish a dedicated deployment process for a specific project or task. Ideal for:

• Simplifying the setup and management of CI/CD pipelines

• Automating Terraform deployments

Integrate into existing CI pipelines

This method is only for visualization and monitoring of your current deployment processes, improving efficiency and visibility without causing disruption to your existing workflow setup.

Creating Firefly workflows

To automate the deployment of your Terraform code, use our wizard to create a workflow within your CI/CD pipeline.

• Each workflow corresponds to a workspace and facilitates the deployment of new resources or modifications to existing ones within your IaC stack.

• For each workspace, we run terraform plan and apply when making changes to your resources.

• Each row under Workspace represents a pull request in Git that contains your changes to the Terraform pipeline.

• Creating a pull request automatically runs terraform plan.

• Merging a pull request automatically runs terraform apply.

• To edit attributes and/or modify variables, select the edit icon.

Procedure

1. Select Workflows > + Add New Workflow.

2. Select your IaC provisioning engine > Next.

3. Select Generate new IaC pipeline or Integrate into an existing pipeline > Next.

4. Generate Firefly key pair and store it in your CI variables.

5. In the Workspace Name field, enter the name of the workspace you want to create. Naming workspaces appropriately helps manage and distinguish between different deployment stages (like development, testing, or production) within the same Terraform configuration.

6. Under VCS integration, select the VCS (Git) integration where you want to create the workflow.

7. Select the Code repository where the workspace Terraform code is located.

8. Select the Default branch of the repository. Merging into this branch will apply the changed workspace (e.g., main).

   • (Optional) Enter the Working directory of the workspace. If it's empty, the working directory will run from the root folder.

9. Select Next.

10. Enter the Terraform version used to deploy your Workspace.

    • (Optional) In the TFvars file path field, enter the workspace tfvars file path relative to the working directory path.

    • (Optional) In the Terraform environment name field, enter the Terraform workspace name of your workspace. If using the same backend for multiple environments, you can set a Terraform workspace that changes the backend path to the environment name.

    • (Optional) Enter non-sensitive Terraform variables.

    • (Optional) Enter sensitive Terraform variables.

11. Select Next.

    • (Optional) In the Self-hosted runner labels field, enter the relevant labels to run the CI/CD pipeline on self-hosted runners.

    • (Optional) When using the AWS profile as the AWS provider authentication, enter the AWS profile configuration .

    • (Optional) When using environment variables as the provider authentication, enter the Provider credentials.

    • (Recommended) Enter the workspace labels for the workspace in Firefly. Naming workspace labels simplifies the management and filtering of deployments, domains, responsible teams, and other categorizations.

12. Select Next > Done. Your workflow has been created.

Integrating into an existing CI pipeline

To integrate your existing Terraform deployment pipelines, use Firefly's docker or CLI.

To create a custom workflow, you must be authenticated with Firefly. To authenticate your account, procure an access key and secret key from Firefly.

Use one of the methods below to send your data to Firefly:

• Pass the keys as arguments (--access-key and --secret-key)

• Pass the keys as environment variables (FIREFLY_ACESS_KEY and FIREFLY_SECRET_KEY)

Downloading the CLI

To download, install, and use the Firefly CI CLI, follow the instructions below. If you encounter any issues, refer to the FAQ section and/or seek support from our support team.

Download the relevant CLI for your operating system using the links below:

MacOS

MacOS ARM: https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Darwin_arm64.tar.gz

MacOS x86: https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Darwin_x86_64.tar.gz

Linux

Linux ARM: https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Linux_arm64.tar.gz

Linux x86: https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Linux_x86_64.tar.gz

Microsoft Windows

Windows x86: https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Windows_x86_64.zip

Using cURL

Download the CLI using the cURL commands below:

curl -O https://gofirefly-prod-iac-ci-cli-binaries.s3.amazonaws.com/fireflyci/latest/fireflyci_Linux_x86_64.tar.gz
tar -xf fireflyci_Linux_x86_64.tar.gz
chmod a+x fireflyci

Move the extracted binary to one of the directories in your $PATH to make it executable from anywhere:

mv fireflyci /usr/local/bin

Docker image

The CLI is available as a Docker image:

docker run public.ecr.aws/firefly/fireflyci:latest

Using the CLI

Post Plan

Flags:
      --config-file string                 Path to the config file (YAML)
  -h, --help                               help for post-plan
  -i, --init-log-file string               Path to the init log file
  -f, --plan-file string                   Path to the plan file to scan
  -l, --plan-output-json-log-file string   Path to the plan output log file in json lines format
      --plan-output-raw-log-file string    Path to the raw plan output log file
  -r, --redact                             Redact plan file data (default true)
      --repository string                  Full repository name, will be filled automatically if running from a CI/CD pipeline
  -o, --show-only-relevant                 Will show only relevant misconfigurations
  -t, --timeout int                        Timeout in seconds for the scan to complete (only relevant when --wait is set) (default 180)
      --wait                               Waits for the scans to finish and checks if there are any misconfigurations (same as --config-file with empty file)
  -w, --workspace string                   Workspace identifier

Global Flags:
  -a, --access-key string   Access key (will be prompted for if not provided)
      --ignore-errors       Exit with a success code even though errors are encountered
      --pretty              Pretty-print JSON output
  -p, --profile string      Profile to use (default "default")
  -s, --secret-key string   Secret key (will be prompted for if not provided)
      --server-url string   FireflyCI API URL (default "https://prodapi.gofirefly.io/api")

Post Apply

Flags:
  -f, --apply-log-file string   Path to the apply log file
  -h, --help                    help for post-apply
  -r, --redact                  Redact the apply log file data (default true)
      --repository string       Full repository name, will be filled automatically if running from a CI/CD pipeline
  -t, --timeout int             Timeout in seconds for the post apply task to complete (only relevant when --wait is set) (default 180)
      --wait                    Waits for the scans to finish
  -w, --workspace string        Workspace name

Global Flags:
  -a, --access-key string   Access key (will be prompted for if not provided)
      --ignore-errors       Exit with a success code even though errors are encountered
      --pretty              Pretty-print JSON output
  -p, --profile string      Profile to use (default "default")
  -s, --secret-key string   Secret key (will be prompted for if not provided)
      --server-url string   FireflyCI API URL (default "https://prodapi.gofirefly.io/api")

Procedure

1. Select Create key pair.

2. Store the access key and secret key in your secret manager that is accessible to your CI/CD tool.

3. In your pipeline, add the Firefly access key and secret key as Environment Variables as FIREFLY_ACCESS_KEY and FIREFLY_SECRET_KEY.

4. Modify your terraform plan to include file output to plan_log.json and plan.json. Example: terraform plan -json -out=tf.plan > plan_log.json && terraform show -json tf.plan > plan.json

5. Add a new pipeline step: Firefly CI Post Plan, which runs between terraform plan and terraform apply. The step should execute the docker run command below: docker run --rm -e FIREFLY_ACCESS_KEY -e FIREFLY_ACCESS_KEY -v $(pwd):/app/cicd public.ecr.aws/firefly/fireflyci:latest post-plan -l /app/cicd/plan_log.json -f /app/cicd/plan.json -w <WORKSPACE_NAME>

   • -w Workspace name displayed in Firefly, unique to this workflow for each Terraform workspace. Naming workspaces appropriately helps manage and distinguish between different deployment stages (like development, testing, or production) within the same Terraform configuration.

6. Modify your terraform apply to include file output to apply_log.json. Example: terraform apply -auto-approve -json > apply_log.json

7. Add a new pipeline step: Firefly CI Post Apply, which runs after terraform apply.

   This should execute the docker run command below: docker run --rm -e FIREFLY_ACCESS_KEY -e FIREFLY_ACCESS_KEY -v $(pwd):/app/cicd public.ecr.aws/firefly/fireflyci:latest post-apply -f /app/cicd/apply_log.json -w <WORKSPACE_NAME>

   • -w Workspace name displayed in Firefly, unique to this workflow for each Terraform workspace. Use the same value you provided in step #5

Pull-request comments with plan summaries

Firefly automatically adds a pull-request comment that summarizes your terraform plan. This summary includes a cost estimation, policy violations, tags coverage information, basic information on the changes made, and a link to the full plan from our platform. To enable this feature, integrate your VCS with Firefly.

Terraform Plan CLI

Using only Firefly credentials, the Terraform plan CLI allows you to execute Terraform plans from your local machine without committing changes to your VCS or storing credentials locally. The CLI works by sending the git-diffs from your default branch to Firefly. Firefly then creates a temporary branch, executes the necessary workflow, and returns the plan output to the CLI, which is displayed on your terminal.

To download and use the CLI to obtain Terraform outputs directly in your command line, refer to the instructions below:

Support Matrix

IaC type	CI tool	VCS integration with Firefly	Managed workflow
Terraform	GitHub actions	Necessary	Necessary

Downloading the CLI

To download, install, and use the Firefly Plan CLI, follow the instructions below. If you encounter any issues, refer to the FAQ section and/or seek support from our support team.

Download the relevant CLI for your operating system using the links below:

MacOS

MacOS ARM: https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Darwin_arm64.tar.gz

MacOS x86: https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Darwin_x86_64.tar.gz

Linux

Linux ARM: https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Linux_arm64.tar.gz

Linux x86: https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Linux_x86_64.tar.gz

Microsoft Windows

Windows x86: https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Windows_x86_64.zip

Using cURL

Download the CLI using the cURL commands below:

curl -O https://gofirefly-prod-plan-cli-binaries.s3.amazonaws.com/fireflyplan/latest/fireflyplan_Linux_x86_64.tar.gz
tar -xf fireflyplan_Linux_x86_64.tar.gz 
chmod a+x fireflyplan

Move the extracted binary to one of the directories in your $PATH to make it executable from anywhere:

mv fireflyplan /usr/local/bin

Docker image

The CLI is available as a Docker image:

docker run public.ecr.aws/firefly/fireflyplan:latest

Using the CLI

Procedure

1. Authenticate using your Firefly keys: fireflyplan configure -p profilename

2. Run the Firefly plan:

   1. Open the Git repository where your workspace is located.

   2. Update your IaC files with the necessary changes.

   3. To execute the plan, run the command below: fireflyplan plan <WORKSPACE_NAME>

   The output of the plan is displayed in your terminal.

Firefly Migration CLI

As a Terraform Enterprise (TFE) user, you can use the CLI tool to migrate to a generic CI/CD system and leverage Firefly's workflows. We map each workspace in TFE to a Firefly workflow and create a new, corresponding backend file. The CLI uses configured backend provider credentials to transfer the state file from TFE to the backend. TFE secrets and variables are mapped to corresponding concepts in the CI/CD system, with a Terraform module generated to set up these configurations. Secret values are used a place holder and must be replaced manually.

To download, install, and use the Firefly Migration CLI effectively, follow the instructions below. If you encounter any issues, refer to the FAQ section and/or seek support from our support team.

Support Matrix

IaC type	CI tool	Backend	VCS integration with Firefly
Terraform	GitHub actions	S3	Necessary

Downloading the CLI

Download the appropriate CLI for your operating system using the links below:

MacOS

MacOS ARM: https://firefly-prod-iac-ci-migrate-cli-binaries.s3.amazonaws.com/fireflymigration/latest/fireflymigration_Darwin_arm64.tar.gz

MacOS x86: https://firefly-prod-iac-ci-migrate-cli-binaries.s3.amazonaws.com/fireflymigration/latest/fireflymigration_Darwin_x86_64.tar.gz

Linux

Linux ARM: https://firefly-prod-iac-ci-migrate-cli-binaries.s3.amazonaws.com/fireflymigration/latest/fireflymigration_Linux_x86_64.tar.gz

Microsoft Windows

Windows x86: https://firefly-prod-iac-ci-migrate-cli-binaries.s3.amazonaws.com/fireflymigration/latest/fireflymigration_Windows_x86_64.zip

Using cURL

Download the CLI using the cURL commands below:

curl -O https://firefly-prod-iac-ci-migrate-cli-binaries.s3.amazonaws.com/fireflymigration/latest/fireflymigration_Linux_arm64.tar.gz 
tar -xf fireflymigration_Linux_arm64.tar.gz 
chmod a+x fireflymigration

Move the extracted binary to one of the directories in your $PATH to make it executable from anywhere:

mv fireflymigration /usr/local/bin

Using the CLI

Procedure

1. Authenticate using your Firefly keys: fireflymigration configure -p profilename

2. Run the Firefly migrate: fireflymigration migrate

Configuration Prompts

After running the migrate command, you are prompted to provide the configuration information below:

• Firefly access key and secret key

• Name of S3 bucket to move the state file to

• Name of DynamoDB table to use for locking Terraform states

• (Optional) For OIDC to connect to AWS, an IAM role with a policy allowing GitHub to assume the role

• (Optional) A comma-separated list of workspaces to migrate in the format `<organization>:<workspaceId>`

Output

The script produces a directory containing Terraform code to create the GitHub Actions environments, along with the variables and secrets. Non-sensitive values are be pre-filled, but sensitive values need to be input manually.

Table

• Each Workspace represents one IaC stack and one Firefly Workflow.

• Each Workspace deploys a few assets connected to each other. They can be in multiple providers or in one provider.

• To view Workspace details, select the caret.

Column	Description
Started at	When the Terraform plan or apply was started
Build ID	CI/CD pipeline run ID with link
Title	CI/CD run title
Commit ID	Commit used to run the CI/CD pipeline
Branch	Branch used to run the CI/CD pipeline
Summary	Resource actions (e.g. create, destroy, change, import)
Policy Violations	Summary of the violations in your Terraform plan
Cost Est	Estimated cost adjustment in your Terraform plan or apply
Tags Coverage	Overview of the tagging status, indicating the percentage of resources that are tagged.
Owner	Owner of the Terraform plan or apply
Status	Whether the Terraform plan or apply failed

Using AI-Generated Error Remediation Suggestions

Use this tool when an error is detected during the plan or apply stages of your deployment. When an error occurs, a Thinkerbell icon appears next to the diagnostic message. To view AI-generated suggestions and/or remediation steps that include detailed explanations and proposed code changes, select the Thinkerbell icon.

Viewing Run Logs

You can view the run details, which incorporate the logs from all stages of a run, including init, plan, and apply.

Support Matrix

Type of Workflow	CI tools	IaC type	Integration method
Managed by Firefly	Azure Pipelines, GitHub actions	OpenTofu, Terraform	Firefly workflow wizard
Integrated with an existing Workflow	Any	OpenTofu, Terraform	Self-service

FAQ

What is Firefly Workflows?

Firefly Workflows is a powerful tool for deploying Terraform within CI/CD pipelines. It streamlines deployment processes, offering an intuitive wizard interface to automate Terraform code deployment. Each workflow corresponds to a workspace, simplifying the deployment of resources. Additionally, Firefly Workflows provides visualization and monitoring features, enhancing deployment efficiency and visibility.

When should I create a new workflow in Firefly?

Creating a new workflow is suitable when you need to establish a dedicated deployment process for a specific project or task, simplifying CI/CD pipeline setup and management.

How do I create a Firefly workflow?

Use the Firefly wizard to automate Terraform deployment by creating a workflow within your CI/CD pipeline, with each workflow corresponding to a workspace.

How do I integrate Firefly into an existing CI pipeline?

Firefly provides Docker-based integration for existing Terraform deployment pipelines. Authentication is required, and deployment triggers are set up between terraform plan and terraform apply. To integrate Firefly with your existing CI pipeline, follow the procedure outlined in the documentation. You can also view the workflows examples template on wide variety of CI tools.

What are the system requirements for using Firefly Workflows?

Firefly Workflows require a compatible CI/CD environment with support for Docker and Terraform/OpenTofu.

Can Firefly Workflows integrate with other CI/CD tools apart from the ones mentioned in the documentation?

Yes, Firefly Workflows can integrate with all CI/CD tools. Firefly provides a Docker-based solution that can work on any CI/CD tool.

Are there any limitations on the size or complexity of Terraform projects that Firefly Workflows can handle?

Firefly Workflows are designed to accommodate Terraform projects of varying sizes and complexity levels. There are no strict limitations imposed by Firefly Workflows.

Do Firefly Workflows support multi-cloud deployments?

Yes, Firefly Workflows support multi-cloud deployments, allowing you to manage infrastructure across multiple cloud providers within a unified workflow.

How do Firefly Workflows handle state management and synchronization across multiple environments?

Firefly Workflows leverage Terraform's state management features to track and synchronize infrastructure state across multiple environments. Each workspace within Firefly corresponds to a Terraform workspace, enabling isolation and management of state files for different environments.

Can Firefly Managed Workflows automatically trigger deployments based on specific events or conditions?

Yes, Firefly managed Workflows can automatically trigger deployments based on specific events or conditions. When you create a pull request (PR) is created, Firefly initiates a Terraform plan, which provides insights into the changes to be applied to the infrastructure. After merging the PR into the default branch, Firefly triggers the terraform apply process, which applies the planned changes to the infrastructure.

How do Firefly Workflows ensure the security of sensitive data?

Firefly Workflows prioritize the security of sensitive data. Before transmitting any data to Firefly's servers, sensitive information is meticulously redacted to prevent unauthorized access or exposure. This ensures that sensitive data remains protected throughout the deployment process.

Do Firefly Workflows provide any built-in monitoring or alerting capabilities for deployments?

Firefly is currently in the process of developing monitoring and alerting functionalities for deployments and will be incorporated into Firefly Workflows in the near future. Stay tuned for further updates on the integration of monitoring and alerting capabilities within the platform.

Is there a limit to the number of workspaces or workflows that can be created within Firefly Workflows?

Firefly Workflows do not impose strict limits on the number of workspaces or workflows that can be created within the platform. You can create and manage multiple workspaces and workflows to accommodate diverse projects, environments, and deployment scenarios effectively.

Is there a way to track and visualize the progress of deployments across multiple environments in Firefly Workflows?

Yes, Firefly Workflows provide tools and dashboards for tracking and visualizing the progress of deployments across multiple environments in real-time. You can view deployment status, execution logs, resource changes, and pipeline metrics using integrated monitoring and visualization features within the Firefly Workflows dashboard.

Do I need to provide Firefly Workflows with my cloud credentials?

No, Firefly Workflows do not require you to provide your cloud credentials. Firefly prioritizes security and does not handle or store your cloud credentials.

Does Firefly host the CI/CD runners to execute the Terraform runs?

No, Firefly does not host the CI/CD runners. You are responsible for managing your CI/CD infrastructure, including runners, to execute Terraform runs.

What happens to my run if Firefly's endpoint is unavailable?

If Firefly's endpoint is unavailable, your CI/CD pipeline will operate normally. However, deployment updates won't appear in the Firefly app or dashboard until the endpoint is accessible again. Please note that pipeline execution remains unaffected, but visibility into deployment status within Firefly will be temporarily interrupted.

Does Firefly have GitHub custom action?

Yes, Firefly has GitHub custom action that wraps the Firefly CI docker tool. View it here.

How can I securely grant my GitHub workflow access to AWS services?

GitHub AWS OpenID Connect (OIDC) integration allows GitHub Actions to authenticate and interact with AWS services securely. To grant the necessary permissions to your GitHub workflow for accessing your AWS account, follow the guide.

How do I authenticate Firefly in GitHub Actions?

1. Issue a new Firefly key pair.

2. Store the access key and secret key values as repository secrets in your GitHub account:

   1. Go to your Github account > Settings > Secrets and variables > Actions > New repository secret.

   2. In the Name field, enter FIREFLY_ACCESS_KEY.

   3. In the Secret field, enter the value of the access key.

   4. Select Add secret.

How do I authenticate Firefly in Azure Pipelines?

1. Issue a new Firefly key pair.

2. Store the access key and secret key values in the fireflySecrets variable group in your Pipeline Library:

   1. Go to your Azure DevOps project > Pipelines > Library > + Variable group > + Add.

   2. Under Name, enter FIREFLY_ACCESS_KEY.

   3. Under Value, enter the access key value.

   4. Select Save.

   5. To download the Terraform extension for Azure DevOps, go here.

   6. After adding the pipeline, grant the Firefly pipeline access to your new variable group:

      1. Select Library > Pipeline permissions > + icon > Select the pipeline from the dropdown menu.

How do I create the Azure Pipeline after merging the Firefly workflow pull-request?

1. Go to your Azure DevOps project > Pipelines > New pipeline > Azure Repos Git.

2. Select your repository > Existing Azure Pipelines YAML file > Select the default branch of your repository.

   1. Provide the path to the workflow YAML created by Firefly (/firefly-pipelines/firefly-deploy-<YOUR-WORKSPACE-NAME>.yaml).

   2. Verify the YAML you selected is NOT the template for the Firefly CI runner (/firefly-pipelines/templates/firefly.yaml).

3. Select Continue > Run. You’re good to go!

How do I trigger the Firefly workflow on pull-requests in Azure Pipelines?

If your code is stored in Bitbucket/Github, the pipeline is automatically triggered for PRs.

If your code is stored in Azure Repos, you can add the pipeline as a Build Validation policy to make it run for PRs:

1. Go to your Azure DevOps project > Settings > Repositories > Select your repository > Policies > Branch Policies.

2. To add a build policy, under Build Validation, select the + icon.

3. Select the Build pipeline you created after merging the PR from Firefly.

4. Under Trigger, select Automatic (whenever the source branch is updated).

5. Select Save. You’re good to go!

How do I add a notification for a Workflow event?

1. Select Notifications > + Add new.

2. Under Event Type, select Workspace Run.

3. Select the specific workspaces, repositories, and labels you want to monitor. By default, all options are selected, but the fields are customizable.

4. Under Workplace Status, select the events you want to be notified about:

   • Plan/Apply completed

   • Plan/Apply/Init failed

   • Plan/Apply empty

   • Blocked (by violated Guardrail rule)

5. Under Destination, select the notification destination.

6. Select Create.