Creating Workflows

Related Guides and Examples:

Firefly Workflows offer a streamlined and powerful way to automate your Terraform, OpenTofu, and Terragrunt deployments directly within the Firefly platform. When you choose to create a new workflow managed by Firefly, you leverage Firefly's infrastructure to execute your IaC operations (plan and apply). This approach simplifies the setup and management of your CI/CD pipelines for IaC, as Firefly handles the execution environment.

This document provides a comprehensive guide to creating and configuring a new Firefly-managed workflow.

Introduction to Firefly-Managed Workflows

Firefly-managed workflows are ideal for:

  • Simplifying CI/CD Setup: Eliminates the need to configure and maintain your own runners or complex pipeline scripts for Terraform/OpenTofu/Terragrunt execution.

  • Automating Deployments: Fully automates terraform plan on pull requests and terraform apply on merges to your designated default branch.

  • Centralized Management: Manage your IaC deployments, variables, and configurations directly within the Firefly UI.

  • Secure Execution: Firefly provides a secure environment for running your Terraform/OpenTofu/Terragrunt operations.

  • VCS Integration: Seamlessly integrates with your Version Control System (VCS) like GitHub, GitLab, etc.

Each workflow in Firefly corresponds to a Workspace. A Workspace represents a specific IaC stack (a collection of Terraform/OpenTofu/Terragrunt files defining a set of resources) and its deployment configuration.

Key Concepts for Firefly-Managed Workflows:

  • Workspace: The core entity representing your Terraform/OpenTofu/Terragrunt configuration, its associated VCS repository, variables, and run history.

  • Firefly Runners: Firefly's managed infrastructure that executes your terraform plan and terraform apply commands.

  • VCS Integration: Connects Firefly to your Git provider (e.g., GitHub) to trigger workflows based on code changes (pull requests, merges).

  • Pull Request (PR) Trigger: Creating a PR in your connected repository automatically triggers a terraform plan for the corresponding Workspace.

  • Merge Trigger: Merging a PR into the Workspace's default branch automatically triggers a terraform apply (or waits for manual approval, depending on Workspace settings).

Prerequisites

Before creating a Firefly-managed workflow, ensure you have:

  • An active Firefly account with appropriate permissions to create Workflows and Workspaces.

  • Your Terraform, OpenTofu, or Terragrunt code hosted in a supported Version Control System (e.g., GitHub, GitLab, Azure DevOps, etc.).

  • Your VCS provider integrated with Firefly. This is done at an organizational level within Firefly settings.

  • Understanding of your Terraform/OpenTofu/Terragrunt project structure (working directory, tfvars files, etc.).

Creating a New Firefly-Managed Workflow: Step-by-Step

Follow the wizard in the Firefly UI to create a new workflow.

Procedure:

  1. Navigate to Workspaces:

    • In the Firefly UI, click on the Workflows > Workspaces from the main navigation menu.

  2. Add New Workspace:

    • Click on the + Add New Workspace button.

  3. Select IaC Provisioning Engine:

    • Choose the Infrastructure as Code engine your project uses.

      • Terraform

      • OpenTofu

      • Terragrunt

    • Click Next.

  4. Choose Workflow Type:

    • Select Generate new IaC pipeline (Firefly Runners). This option indicates you want Firefly to manage and run your Terraform/OpenTofu/Terragrunt operations.

  5. Configure Workspace Details:

    • Workspace Name:

      • Enter a unique and descriptive name for your Workspace.

      • Example: production-database, staging-eks-cluster, shared-network-dev.

      • Naming workspaces appropriately is crucial for managing and distinguishing between different deployment environments (e.g., development, staging, production) or components within your infrastructure.

      • This name will be displayed throughout Firefly.

    • VCS Integration:

      • VCS Provider: Select the Version Control System where your IaC code is hosted (e.g., GitHub, GitLab). This list is populated based on your configured integrations in Firefly.

      • Code Repository: Choose the specific repository that contains the Terraform/OpenTofu/Terragrunt code for this Workspace.

      • Default Branch: Select the primary branch of your repository (e.g., main, master). Merging changes into this branch will (if configured) trigger the apply phase of your workflow.

    • (Optional) Working Directory:

      • Specify the path to the directory within your repository where the Terraform/OpenTofu/Terragrunt root module for this Workspace is located.

      • If your Terraform files are in a subdirectory like terraform/my-project/ or infra/production/vpc/, enter that path here (e.g., terraform/my-project/).

      • If left empty, Firefly will run Terraform/OpenTofu/Terragrunt commands from the root of the repository.

    • Click Next.

  6. Configure Terraform/OpenTofu/Terragrunt Settings:

    • Terraform/OpenTofu/Terragrunt Version:

      • Enter the specific version of Terraform, OpenTofu, or Terragrunt that your code requires.

      • Example for Terraform: 1.5.7

      • Example for OpenTofu: latest or a specific version like 1.6.0

      • Example for Terragrunt: latest or a specific version like 0.54.0

      • Firefly will use this version when executing plan and apply operations.

    • (Optional) TFvars File Path:

      • If you use a .tfvars file to specify input variables for this Workspace, enter the path to that file relative to the Working Directory defined in the previous step.

      • Example: If your working directory is infra/production/ and your variables file is infra/production/prod.tfvars, enter prod.tfvars.

      • If this Workspace uses multiple .tfvars files, Firefly might have specific conventions or you might manage complex variable scenarios using Variable Sets (discussed later).

    • (Optional) Terraform Environment Name (Terraform Workspaces):

      • This refers to Terraform's built-in CLI workspaces feature (not to be confused with Firefly Workspaces).

      • If you use Terraform CLI workspaces to manage multiple environments using the same configuration and backend (e.g., by running terraform workspace select <env_name>), enter the Terraform workspace name here (e.g., dev, staging, prod).

      • Firefly will then execute commands within the context of this Terraform CLI workspace, which often influences the backend state file path.

      • If you don't use Terraform CLI workspaces, leave this blank.

    • (Optional) Non-Sensitive Terraform Variables:

      • Enter any non-sensitive input variables for your Terraform/OpenTofu/Terragrunt configuration directly in the UI.

      • Provide the variable key and value.

      • These are variables that are not secret and can be stored in plain text.

      • You can add multiple variables.

    • (Optional) Sensitive Terraform Variables:

      • Enter any sensitive input variables (e.g., API keys, passwords, database credentials).

      • Provide the variable key and value. These values will be stored securely by Firefly and redacted in logs.

      • It's highly recommended to manage sensitive data through this mechanism or dedicated secrets management integrations if available, rather than committing them to .tfvars files.

    • Click Next.

  7. Configure Runner and Provider Settings:

    • (Optional) Self-hosted Runner Labels:

      • This field is for workflows using self-hosted runners. Since you are creating a Firefly-managed workflow, you would leave this field empty.

      • Firefly-managed workflows use Firefly's shared runner infrastructure.

    • Cloud Provider Authentication: Firefly needs credentials to allow Terraform/OpenTofu to authenticate with your cloud provider(s) (e.g., AWS, Azure, Google Cloud) to manage resources. The exact options available here will depend on Firefly's capabilities and integrations. Common methods include:

      • (Optional) AWS Profile Configuration (if using AWS):

        • If your Terraform AWS provider is configured to use a named profile, you might be asked to provide details for Firefly to assume a role or use specific credentials associated with that profile. This often involves setting up an OIDC trust relationship between Firefly/your VCS and AWS IAM.

        • Follow the on-screen instructions or link to Firefly documentation for setting up AWS authentication (e.g., using IAM Roles for Service Accounts - IRSA, or OIDC).

      • (Optional) Provider Credentials (Environment Variables):

        • For many providers, Terraform/OpenTofu relies on specific environment variables for authentication (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY for AWS; ARM_CLIENT_ID, ARM_CLIENT_SECRET for Azure; GOOGLE_CREDENTIALS for Google Cloud).

        • You can enter these as sensitive variables here. Firefly will set them in the execution environment when running plan and apply.

        • Example for AWS (less common if OIDC is used):

          • Key: AWS_ACCESS_KEY_ID, Value: YOUR_AWS_ACCESS_KEY_ID (mark as sensitive)

          • Key: AWS_SECRET_ACCESS_KEY, Value: YOUR_AWS_SECRET_ACCESS_KEY (mark as sensitive)

          • Key: AWS_DEFAULT_REGION, Value: your-aws-region (non-sensitive)

      Refer to Firefly's specific documentation or UI prompts for the recommended and most secure way to configure cloud provider authentication for Firefly-managed runners. OIDC-based approaches are generally preferred over static credentials.

    • (Recommended) Workspace Labels:

      • Enter labels to categorize and organize your Workspace within Firefly.

      • Labels are key-value pairs (e.g., env:production, team:backend, project:phoenix).

      • Naming Workspace labels appropriately simplifies management, filtering of deployments, identifying domains, responsible teams, and other categorizations within Firefly.

      • You can add multiple labels.

    • Click Next.

  8. Review and Finalize:

    • Review all the configuration details you've entered for your new Workspace.

    • Go back and make any necessary corrections.

    • Once you are satisfied, click Done (or a similar finalization button).

Post-Creation: Your Workflow is Ready

Once created, your Firefly-managed Workspace is active. Here's what happens:

  • Initial Plan (Optional): Firefly might run an initial terraform plan to establish a baseline for the Workspace.

  • Pull Request (PR) Integration:

    • When a new PR is opened in the connected repository that modifies files within the Workspace's configured Working Directory and targets the Default Branch, Firefly will automatically trigger a terraform plan.

    • The plan results (including changes, cost estimations, policy violations if Guardrails are configured) will be displayed in Firefly and often commented back on the PR itself.

  • Merge Integration:

    • When a PR with planned changes is merged into the Workspace's Default Branch, Firefly will automatically trigger a terraform apply.

    • The apply operation will execute the changes defined in the last successful plan for that PR.

  • Workspace Dashboard: You can view the status, history of runs (plans and applies), logs, and other details for your Workspace within the Firefly UI.

Managing Your Firefly-Managed Workspace

After creation, you can manage and modify your Workspace settings:

  • Editing Workspace Attributes:

    • Navigate to your Workspace in Firefly.

    • Look for an Edit icon or button (often a pencil icon).

    • You can modify most settings defined during creation, such as:

      • VCS settings (repository, branch, working directory)

      • Terraform/OpenTofu version

      • Variables (add, remove, update non-sensitive and sensitive variables)

      • Provider authentication details

      • Workspace labels

  • Viewing Run History:

    • The Workspace dashboard will show a table or list of all plan and apply runs.

    • Each row represents a run triggered by a PR or a merge.

    • Details often include:

      • Status: Succeeded, Failed, In Progress, Pending Apply, etc.

      • Started At: Timestamp of when the run began.

      • Build ID/Run ID: Link to the CI/CD run (if applicable, though for fully Firefly-managed, this might be an internal Firefly ID).

      • Title/Commit Message: Information from the triggering commit or PR.

      • Commit ID: The Git commit SHA.

      • Branch: The branch that triggered the run.

      • Policy Violations: Summary from Guardrails.

      • Cost Estimation: Changes in cost.

      • Owner: User associated with the triggering event.

      • Summary: Overview of resource actions (create, update, delete).

  • Accessing Logs:

    • You can view detailed logs for each init, plan, and apply stage of a run.

    • Firefly redacts sensitive information from these logs.

Advanced Configurations (May Vary)

Depending on Firefly's features, you might encounter more advanced configurations for your managed workflows:

  • Apply Rules: Some Workspaces might allow configuration of apply behavior, such as:

    • Auto-apply: Automatically apply on merge to the default branch (common).

    • Manual Apply: Require manual approval within Firefly to proceed with an apply after a plan is generated from a merge.

  • Scheduled Drift Detection: Configure the Workspace to run periodic terraform plan operations to detect drift between your IaC and the actual state of your infrastructure.

  • Notifications: Set up notifications for workflow events (e.g., plan failure, apply success) to be sent to Slack, email, PagerDuty, etc.

  • Variable Sets: Utilize Firefly's Variable Sets feature to group and reuse common variables across multiple Workspaces.

Troubleshooting Common Issues

  • Plan/Apply Failures:

    • Check Logs: The detailed logs in Firefly are the first place to look. They will contain output from terraform init, plan, or apply which usually indicates the error.

    • Terraform Code Issues: Syntax errors, provider issues, or logical errors in your IaC.

    • Provider Authentication: Ensure cloud provider credentials are correctly configured and have necessary permissions.

    • Variable Issues: Incorrect or missing variables.

    • Terraform Version Mismatch: Ensure the version specified in Firefly matches your code's requirements.

  • VCS Integration Problems:

    • Ensure Firefly's VCS App (e.g., GitHub App) has access to the repository.

    • Check webhooks if used by the integration.

    • Verify branch names and working directory paths are correct.

  • Resource Conflicts or Limits: Cloud provider errors due to resource limits, naming conflicts, or dependencies.

Refer to Firefly's specific error messages and documentation for detailed troubleshooting steps.

For hybrid or CI/CD integration, see the Integrating Existing CI/CD Pipelines with Firefly Workflows guide. For real-world pipeline YAMLs, visit the workflows-examples GitHub repository.

By following this guide, you can effectively create and manage Firefly-managed workflows, automating your Terraform and OpenTofu deployments with ease and confidence.

Last updated

Was this helpful?