Creating a Self-Hosted Runner Pool

Related Guides and Examples:

• Overview: Firefly Workflows and Guardrails: High-level concepts and options.

• Creating New Firefly Workflows: How to set up managed workflows.

• Creating Guardrail Rules in Firefly: How to define and enforce policies and best practices.

• Creating Projects: How to create projects to group IaC orchestration resources and control access.

• Creating Variable Sets: Managing reusable variable collections.

• Triggering Workspace Deployment: How to trigger a workspace deployment from the Firefly UI.

• Integrating Existing CI/CD Pipelines with Firefly Workflows: For hybrid or external CI/CD integration.

• Firefly Workflows Example Pipelines: Real-world pipeline and integration templates for Jenkins, GitHub Actions, and more.

Self-hosted runners allow you to execute Terraform, OpenTofu, and Terragrunt operations within your own infrastructure while maintaining full control over network access, data residency, and compliance requirements. By deploying runner pools in your environment, you can ensure that all IaC operations run within your network boundaries while still benefiting from Firefly's orchestration, monitoring, and governance features.

This document provides a comprehensive guide on how to create and configure self-hosted runner pools in Firefly.

Creating a New Self-Hosted Runner Pool: Step-by-Step

Follow the wizard in the Firefly UI to create a new self-hosted runner pool.

Procedure:

1. Navigate to Self-Hosted Runners:

   • In the Firefly UI, click on Workflows from the main navigation menu.

   • Then, click on the Self Runner Runners sub-section.

2. Add New Runner Pool:

   • Click on the + New runner pool button.

3. Enter Runner Pool Details:

   • Name:

     • Provide a unique name for your self-hosted runner pool (3-64 characters).

     • Use alphanumeric characters, hyphens, and underscores only.

     • The name must be unique within your organization.

     • Example: production-runners, us-east-1-pool, secure-network-runners.

   • Labels (Optional):

     • Add labels for workspace targeting and organization. Workspaces can target specific pools using these labels.

   • Runner Agent Replicas:

     • Number of runner agents to deploy. Can be scaled up or down later.

   • Project Assignment (Optional):

     • Select the project this runner pool will be assigned to. Assigning to the global project means that this runner pool will be available to all projects.

   • Release Name:

     • Provide a unique name for the release name (3-64 characters).

     • Use alphanumeric characters, hyphens, and underscores only.

   • Namespace Name:

     • Provide a unique name for the namespace name (3-64 characters).

     • Use alphanumeric characters, hyphens, and underscores only.

   • Description (Optional):

     • Add a free-text explanation to help identify the runner pool's purpose.

     • Example: "Production runner pool for secure network deployments in us-east-1".

4. Press Create to View Installation Instructions:

   • Click Create to create the runner pool and view the installation instructions.

5. Installation Instructions:

   • Deploy runners in your Kubernetes cluster using Helm to automatically manage multiple runner agents as a single pool. Run the Helm command below. The pool status will change from Inactive to Active once the agents successfully connect and send their first heartbeat to Firefly.

   • Copy the generated Helm command from the installation instructions.

   • Deploy the command in your Kubernetes cluster.

   • Important: Firefly doesn't save the JWT token, meaning that once this wizard is closed, the instructions cannot be replicated.

6. Press Next to go to the Completion step:

   • Click Next to proceed to the completion step.

7. Press Done to close the wizard:

   • Click Done to close the wizard.

Self-Hosted Runner Pool Status

The self-hosted runner pool status will change from Inactive to Healthy once the self-hosted runner is deployed in your Kubernetes cluster and Firefly receives a heartbeat from the runner agents. This indicates that the runner pool is successfully connected and ready to execute IaC operations.

Assigning a Self-Hosted Runner Pool: Firefly Managed Workspace and Projects

Once you have created a self-hosted runner pool, you can assign it to workspaces or projects to control where your IaC operations execute.

Connecting Self-Hosted Runner to a Project or Workspace

1. Navigate to Workflows > Self Hosted Runners.

2. Select the runner pool you want to assign to a project or workspace and select the Assign action from the action list.

3. Choose the relevant projects and workspace you want this runner pool to be assigned to.

4. Click Assign to save the changes.

Assigning to a Project

When you assign a runner pool to a project, all workspaces within that project and its sub-projects automatically inherit the runner pool assignment. This provides a centralized way to manage runner assignments across multiple workspaces.

To assign a runner pool to a project:

1. Navigate to Workflows > Projects & Variables.

2. Select the project you want to configure and select the Edit action from the action list.

3. In the edit project wizard locate the Runner Pool field.

4. Select the runner pool from the dropdown menu.

5. Save the changes.

All workspaces created within this project will automatically use the assigned runner pool for execution.

Assigning to a Workspace

You can also assign a runner pool directly to a specific workspace. Workspace-level assignments override project-level assignments, allowing you to customize runner selection for individual workspaces when needed.

To assign a runner pool to a workspace:

1. Navigate to Workflows > Workspaces.

2. Select the workspace you want to configure and select the Edit action from the action list.

3. In the edit workspace wizard click next until you are at the Runner Configuration step.

4. Select the "self-hosted" in the Runner Type section.

5. Click next and Done to save the changes.

The workspace will use the selected runner pool for all Terraform/OpenTofu/Terragrunt operations.

Runner Pool Inheritance

Runner pool assignments follow a hierarchical inheritance model:

• Project Level: When assigned to a project, all child projects and workspaces inherit the runner pool.

• Workspace Level: Workspace-level assignments override project-level assignments.

• Default Behavior: If no runner pool is assigned at either level, workspaces use Firefly's managed runners.

This inheritance model allows you to set default runner pools at the project level while maintaining the flexibility to override at the workspace level when specific requirements exist.