Creating Workflows

Related Guides and Examples:

• Overview: Firefly Workflows and Guardrails: High-level concepts and options.

• Integrating Existing CI/CD Pipelines with Firefly Workflows: For hybrid or external CI/CD integration.

• Creating Guardrail Rules in Firefly: How to define and enforce policies and best practices.

• Firefly Workflows Example Pipelines (GitHub): Real-world pipeline YAMLs and integration templates.

Firefly Workflows offer a streamlined and powerful way to automate your Terraform and OpenTofu deployments directly within the Firefly platform. When you choose to create a new workflow managed by Firefly, you leverage Firefly's infrastructure to execute your IaC operations (plan and apply). This approach simplifies the setup and management of your CI/CD pipelines for IaC, as Firefly handles the execution environment.

This document provides a comprehensive guide to creating and configuring a new Firefly-managed workflow.

Introduction to Firefly-Managed Workflows

Firefly-managed workflows are ideal for:

• Simplifying CI/CD Setup: Eliminates the need to configure and maintain your own runners or complex pipeline scripts for Terraform/OpenTofu execution.

• Automating Deployments: Fully automates terraform plan on pull requests and terraform apply on merges to your designated default branch.

• Centralized Management: Manage your IaC deployments, variables, and configurations directly within the Firefly UI.

• Secure Execution: Firefly provides a secure environment for running your Terraform/OpenTofu operations.

• VCS Integration: Seamlessly integrates with your Version Control System (VCS) like GitHub, GitLab, etc.

Each workflow in Firefly corresponds to a Workspace. A Workspace represents a specific IaC stack (a collection of Terraform/OpenTofu files defining a set of resources) and its deployment configuration.

Key Concepts for Firefly-Managed Workflows:

• Workspace: The core entity representing your Terraform/OpenTofu configuration, its associated VCS repository, variables, and run history.

• Firefly Runners: Firefly's managed infrastructure that executes your terraform plan and terraform apply commands.

• VCS Integration: Connects Firefly to your Git provider (e.g., GitHub) to trigger workflows based on code changes (pull requests, merges).

• Pull Request (PR) Trigger: Creating a PR in your connected repository automatically triggers a terraform plan for the corresponding Workspace.

• Merge Trigger: Merging a PR into the Workspace's default branch automatically triggers a terraform apply (or waits for manual approval, depending on Workspace settings).

Prerequisites

Before creating a Firefly-managed workflow, ensure you have:

• An active Firefly account with appropriate permissions to create Workflows and Workspaces.

• Your Terraform or OpenTofu code hosted in a supported Version Control System (e.g., GitHub, GitLab).

• Your VCS provider integrated with Firefly. This is done at an organizational level within Firefly settings.

• Understanding of your Terraform/OpenTofu project structure (working directory, tfvars files, etc.).

Creating a New Firefly-Managed Workflow: Step-by-Step

Follow the wizard in the Firefly UI to create a new workflow.

Procedure:

1. Navigate to Workflows:

   • In the Firefly UI, select Workflows from the main navigation menu.

2. Add New Workflow:

   • Click on the + Add New Workflow button.

3. Select IaC Provisioning Engine:

   • Choose the Infrastructure as Code engine your project uses.

     • Terraform

     • OpenTofu

   • Click Next.

4. Choose Workflow Type:

   • Select Generate new IaC pipeline (Managed by Firefly). This option indicates you want Firefly to manage and run your Terraform/OpenTofu operations.

   • Click Next.

5. Generate Firefly Key Pair (if applicable for specific setups - often handled by VCS app integration):

   • The wizard might prompt you to generate a Firefly key pair (Access Key and Secret Key) and store it in your CI variables if you were integrating an existing CI. However, for Firefly-managed workflows, authentication with your VCS is handled via a Firefly VCS App installation (e.g., GitHub App) or a token, which grants Firefly access to repositories. Ensure your VCS integration is correctly set up in Firefly.

   • For most Firefly-managed workflows, direct key pair generation at this step for the CI itself is less common, as Firefly directly interacts with the VCS via its app integration. Follow UI prompts carefully.

6. Configure Workspace Details:

   • Workspace Name:

     • Enter a unique and descriptive name for your Workspace.

     • Example: production-database, staging-eks-cluster, shared-network-dev.

     • Naming workspaces appropriately is crucial for managing and distinguishing between different deployment environments (e.g., development, staging, production) or components within your infrastructure.

     • This name will be displayed throughout Firefly.

   • VCS Integration:

     • VCS Provider: Select the Version Control System where your IaC code is hosted (e.g., GitHub, GitLab). This list is populated based on your configured integrations in Firefly.

     • Code Repository: Choose the specific repository that contains the Terraform/OpenTofu code for this Workspace.

     • Default Branch: Select the primary branch of your repository (e.g., main, master). Merging changes into this branch will (if configured) trigger the apply phase of your workflow.

   • (Optional) Working Directory:

     • Specify the path to the directory within your repository where the Terraform/OpenTofu root module for this Workspace is located.

     • If your Terraform files are in a subdirectory like terraform/my-project/ or infra/production/vpc/, enter that path here (e.g., terraform/my-project/).

     • If left empty, Firefly will run Terraform/OpenTofu commands from the root of the repository.

   • Click Next.

7. Configure Terraform/OpenTofu Settings:

   • Terraform/OpenTofu Version:

     • Enter the specific version of Terraform or OpenTofu that your code requires.

     • Example for Terraform: 1.5.7

     • Example for OpenTofu: latest or a specific version like 1.6.0

     • Firefly will use this version when executing plan and apply operations.

   • (Optional) TFvars File Path:

     • If you use a .tfvars file to specify input variables for this Workspace, enter the path to that file relative to the Working Directory defined in the previous step.

     • Example: If your working directory is infra/production/ and your variables file is infra/production/prod.tfvars, enter prod.tfvars.

     • If this Workspace uses multiple .tfvars files, Firefly might have specific conventions or you might manage complex variable scenarios using Variable Sets (discussed later).

   • (Optional) Terraform Environment Name (Terraform Workspaces):

     • This refers to Terraform's built-in CLI workspaces feature (not to be confused with Firefly Workspaces).

     • If you use Terraform CLI workspaces to manage multiple environments using the same configuration and backend (e.g., by running terraform workspace select <env_name>), enter the Terraform workspace name here (e.g., dev, staging, prod).

     • Firefly will then execute commands within the context of this Terraform CLI workspace, which often influences the backend state file path.

     • If you don't use Terraform CLI workspaces, leave this blank.

   • (Optional) Non-Sensitive Terraform Variables:

     • Enter any non-sensitive input variables for your Terraform/OpenTofu configuration directly in the UI.

     • Provide the variable key and value.

     • These are variables that are not secret and can be stored in plain text.

     • You can add multiple variables.

   • (Optional) Sensitive Terraform Variables:

     • Enter any sensitive input variables (e.g., API keys, passwords, database credentials).

     • Provide the variable key and value. These values will be stored securely by Firefly and redacted in logs.

     • It's highly recommended to manage sensitive data through this mechanism or dedicated secrets management integrations if available, rather than committing them to .tfvars files.

   • Click Next.

8. Configure Runner and Provider Settings:

   • (Optional) Self-hosted Runner Labels:

     • This field is for workflows using self-hosted runners. Since you are creating a Firefly-managed workflow, you would leave this field empty.

     • Firefly-managed workflows use Firefly's shared runner infrastructure.

   • Cloud Provider Authentication: Firefly needs credentials to allow Terraform/OpenTofu to authenticate with your cloud provider(s) (e.g., AWS, Azure, GCP) to manage resources. The exact options available here will depend on Firefly's capabilities and integrations. Common methods include:

     • (Optional) AWS Profile Configuration (if using AWS):

       • If your Terraform AWS provider is configured to use a named profile, you might be asked to provide details for Firefly to assume a role or use specific credentials associated with that profile. This often involves setting up an OIDC trust relationship between Firefly/your VCS and AWS IAM.

       • Follow the on-screen instructions or link to Firefly documentation for setting up AWS authentication (e.g., using IAM Roles for Service Accounts - IRSA, or OIDC).

     • (Optional) Provider Credentials (Environment Variables):

       • For many providers, Terraform/OpenTofu relies on specific environment variables for authentication (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY for AWS; ARM_CLIENT_ID, ARM_CLIENT_SECRET for Azure; GOOGLE_CREDENTIALS for GCP).

       • You can enter these as sensitive variables here. Firefly will set them in the execution environment when running plan and apply.

       • Example for AWS (less common if OIDC is used):

         • Key: AWS_ACCESS_KEY_ID, Value: YOUR_AWS_ACCESS_KEY_ID (mark as sensitive)

         • Key: AWS_SECRET_ACCESS_KEY, Value: YOUR_AWS_SECRET_ACCESS_KEY (mark as sensitive)

         • Key: AWS_DEFAULT_REGION, Value: your-aws-region (non-sensitive)

     Refer to Firefly's specific documentation or UI prompts for the recommended and most secure way to configure cloud provider authentication for Firefly-managed runners. OIDC-based approaches are generally preferred over static credentials.

   • (Recommended) Workspace Labels:

     • Enter labels to categorize and organize your Workspace within Firefly.

     • Labels are key-value pairs (e.g., env:production, team:backend, project:phoenix).

     • Naming Workspace labels appropriately simplifies management, filtering of deployments, identifying domains, responsible teams, and other categorizations within Firefly.

     • You can add multiple labels.

   • Click Next.

9. Review and Finalize:

   • Review all the configuration details you've entered for your new Workspace.

   • Go back and make any necessary corrections.

   • Once you are satisfied, click Done (or a similar finalization button).

Post-Creation: Your Workflow is Ready

Once created, your Firefly-managed Workspace is active. Here's what happens:

• Initial Plan (Optional): Firefly might run an initial terraform plan to establish a baseline for the Workspace.

• Pull Request (PR) Integration:

  • When a new PR is opened in the connected repository that modifies files within the Workspace's configured Working Directory and targets the Default Branch, Firefly will automatically trigger a terraform plan.

  • The plan results (including changes, cost estimations, policy violations if Guardrails are configured) will be displayed in Firefly and often commented back on the PR itself.

• Merge Integration:

  • When a PR with planned changes is merged into the Workspace's Default Branch, Firefly will automatically trigger a terraform apply.

  • The apply operation will execute the changes defined in the last successful plan for that PR.

• Workspace Dashboard: You can view the status, history of runs (plans and applies), logs, and other details for your Workspace within the Firefly UI.

Managing Your Firefly-Managed Workspace

After creation, you can manage and modify your Workspace settings:

• Editing Workspace Attributes:

  • Navigate to your Workspace in Firefly.

  • Look for an Edit icon or button (often a pencil icon).

  • You can modify most settings defined during creation, such as:

    • VCS settings (repository, branch, working directory)

    • Terraform/OpenTofu version

    • Variables (add, remove, update non-sensitive and sensitive variables)

    • Provider authentication details

    • Workspace labels

• Viewing Run History:

  • The Workspace dashboard will show a table or list of all plan and apply runs.

  • Each row represents a run triggered by a PR or a merge.

  • Details often include:

    • Status: Succeeded, Failed, In Progress, Pending Apply, etc.

    • Started At: Timestamp of when the run began.

    • Build ID/Run ID: Link to the CI/CD run (if applicable, though for fully Firefly-managed, this might be an internal Firefly ID).

    • Title/Commit Message: Information from the triggering commit or PR.

    • Commit ID: The Git commit SHA.

    • Branch: The branch that triggered the run.

    • Policy Violations: Summary from Guardrails.

    • Cost Estimation: Changes in cost.

    • Owner: User associated with the triggering event.

    • Summary: Overview of resource actions (create, update, delete).

• Accessing Logs:

  • You can view detailed logs for each init, plan, and apply stage of a run.

  • Firefly redacts sensitive information from these logs.

Advanced Configurations (May Vary)

Depending on Firefly's features, you might encounter more advanced configurations for your managed workflows:

• Apply Rules: Some Workspaces might allow configuration of apply behavior, such as:

  • Auto-apply: Automatically apply on merge to the default branch (common).

  • Manual Apply: Require manual approval within Firefly to proceed with an apply after a plan is generated from a merge.

• Scheduled Drift Detection: Configure the Workspace to run periodic terraform plan operations to detect drift between your IaC and the actual state of your infrastructure.

• Notifications: Set up notifications for workflow events (e.g., plan failure, apply success) to be sent to Slack, email, PagerDuty, etc.

• Variable Sets: Utilize Firefly's Variable Sets feature to group and reuse common variables across multiple Workspaces.

Troubleshooting Common Issues

• Plan/Apply Failures:

  • Check Logs: The detailed logs in Firefly are the first place to look. They will contain output from terraform init, plan, or apply which usually indicates the error.

  • Terraform Code Issues: Syntax errors, provider issues, or logical errors in your IaC.

  • Provider Authentication: Ensure cloud provider credentials are correctly configured and have necessary permissions.

  • Variable Issues: Incorrect or missing variables.

  • Terraform Version Mismatch: Ensure the version specified in Firefly matches your code's requirements.

• VCS Integration Problems:

  • Ensure Firefly's VCS App (e.g., GitHub App) has access to the repository.

  • Check webhooks if used by the integration.

  • Verify branch names and working directory paths are correct.

• Resource Conflicts or Limits: Cloud provider errors due to resource limits, naming conflicts, or dependencies.

Refer to Firefly's specific error messages and documentation for detailed troubleshooting steps.

For hybrid or CI/CD integration, see the Integrating Existing CI/CD Pipelines with Firefly Workflows guide. For real-world pipeline YAMLs, visit the workflows-examples GitHub repository.

By following this guide, you can effectively create and manage Firefly-managed workflows, automating your Terraform and OpenTofu deployments with ease and confidence.