Creating Variable Sets

Related Guides and Examples:

• Overview: Firefly Workflows and Guardrails: High-level concepts and options.

• Creating New Firefly Workflows: How to set up managed workflows.

• Creating Guardrail Rules in Firefly: How to define and enforce policies and best practices.

• Creating Projects: How to create projects to group IaC orchestration resources and control access.

• Triggering Workspace Deployment: How to trigger a workspace deployment from the Firefly UI.

Variable Sets in Firefly Workflows are centralized, reusable collections of configuration variables that can be shared across workspaces and projects. They provide a powerful way to manage execution variables at scale, and enforce consistency across your Infrastructure as Code (IaC) deployments. Variable Sets eliminate the need to duplicate configuration values across multiple workspaces while maintaining the flexibility to override values at specific scopes when needed. This document provides a comprehensive guide on how to create, manage, and utilize Variable Sets effectively in Firefly Workflows.

Key Benefits of Using Variable Sets:

• Centralized Management: Define variables once and reuse them across multiple workspaces and projects.

• Consistency Enforcement: Ensure uniform configuration values across your infrastructure deployments.

• Reduced Duplication: Eliminate the need to manually configure the same variables across multiple workspaces.

• Hierarchical Inheritance: Leverage variable precedence rules and inheritance patterns to maintain flexibility.

• Security Management: Centrally manage sensitive values like API keys, passwords, and certificates.

• Conflict Detection: Automatically detect and prevent variable conflicts across inheritance chains.

Variable Sets Structure and Inheritance

Variable Sets in Firefly follow a sophisticated inheritance model that provides both flexibility and consistency:

Variable Inheritance Hierarchy

Variables in Firefly follow a hierarchical scope structure with specific inheritance and precedence rules:

1. Organization Level (Global):

   • Organization-wide variables that serve as defaults for all projects and workspaces.

   • Lowest precedence in the inheritance chain.

   • Can be overridden at any lower scope.

2. Project Level:

   • Project-specific variables that apply to all workspaces within the project.

   • Override organization-level variables.

   • Inherited by all sub-projects and workspaces.

3. Sub-Project Level:

   • Sub-project-specific variables that apply to workspaces within the sub-project.

   • Override project-level variables.

   • Inherited by workspaces within the sub-project.

4. Workspace Level:

   • Workspace-specific variables with the highest precedence.

   • Override all inherited variables.

   • Final values used during execution.

Variable Precedence Rules

Precedence Order (highest to lowest):

1. Workspace-level variables.

2. Variable set variables consumed by workspace.

3. Sub-project variables.

4. Project variables.

5. Organization-level variables.

Variable Set Scopes

Variable Sets can be created with different scopes:

• Global Variable Sets: Available across all projects and workspaces in the organization. All users can view these variable sets and Admin users can edit them.

• Project-Level Variable Sets: Available only within specific projects and their sub-projects. Only users assigned to those projects can access and view these variable sets.

Variable Set Inheritance Rules

• Association vs. Consumption: Variable sets can be associated with projects (making them available) and consumed by workspaces, projects, sub-projects, and organization-level (actively using their values). Only users assigned to the project can access and view these variable sets.

• Inheritance Chain: Variable sets can inherit from other variable sets, creating a chain of configuration inheritance.

• Conflict Resolution: When multiple variable sets contain the same variable name at the same hierarchy level, conflicts must be resolved before consumption.

• Override Capability: Variables from consumed variable sets can be overridden by workspace-specific, project-specific, sub-project-specific, and organization-level variables.

Creating a New Variable Set: Step-by-Step

Follow the wizard in the Firefly UI to create a new Variable Set.

Procedure:

1. Navigate to Variable Sets:

   • In the Firefly UI, click on Workflows from the main navigation menu.

   • Then, click on the Variable Sets sub-section.

2. Create New Variable Set:

   • Click on the + Create new variable set button.

3. Enter Variable Set Details:

   • Name:

     • Provide a unique name for your variable set (3-64 characters).

     • Use alphanumeric characters, hyphens, and underscores only.

     • The name must be unique across the organization.

     • Example: aws-production-secrets, common-terraform-vars, dev-environment-config.

   • Description (Optional):

     • Add a free-text explanation to describe the variable set's purpose.

     • Example: "Production AWS credentials and configuration for customer-facing applications".

4. Configure Project Assignment:

   • Project Assignment (Optional):

     • Select one or more projects to assign this variable set to.

     • If no projects are selected, the variable set becomes globally available.

     • Only workspaces and sub-projects under assigned projects can consume this variable set.

     • Only users assigned to the project can access and view these variable sets.

     • Example: Assign to /aws project to make it available only to AWS-related workspaces and sub-projects.

5. Set Up Variable Inheritance:

   • Inherited Variable Sets (Optional):

     • Select one or more existing variable sets to inherit configuration from.

     • Variables from inherited sets can be overridden in the current set.

     • Conflict Detection: If inherited sets contain duplicate variable names, conflicts will be highlighted.

     • Example: Inherit from base-terraform-config and add environment-specific overrides.

6. Define Variables:

   • Variable Configuration:

     • Add individual variables to the set.

     • Each variable requires a name and value.

     • Variable Properties:

       • Name: The variable key (must be unique within the set).

       • Value: The variable value (can be overridden at lower scopes).

       • Sensitive: Mark as sensitive to hide the value in logs and UI.

       • Environment Variable: Export as environment variable during execution.

   • Variable Management:

     • Add Variables: Use Add new for single variables or Multiple at once for bulk addition.

     • Search and Filter: Use the search input to find specific variables.

     • Override Indicators: Variables with override warnings show the original source.

     • Conflict Resolution: Address any conflicts between inherited variable sets.

7. Review and Create:

   • Carefully review all configured variables and inheritance settings.

   • Ensure no conflicts exist between inherited variable sets.

   • Verify that sensitive variables are properly marked.

   • Click Create to create the variable set.

Managing Variable Sets

Once created, Variable Sets can be managed through various operations:

Viewing and Searching Variable Sets

• Variable Sets List View: The Variable Sets page displays all sets you have access to.

• Search and Filter:

  • Text Search: Search by variable set name or description.

  • Column Sorting: Sort by name, projects assigned, workspace usage, or variable count.

Editing Variable Sets

• Basic Information: Update name, description, and project assignments.

• Variable Management: Add, modify, or remove variables from the set.

• Inheritance Changes: Modify inherited variable sets or resolve conflicts.

• Conflict Resolution: Address conflicts that arise from inheritance changes.

• Edit Validation: Before saving, the system will validate the variable set to ensure it is valid and does not contain conflicts in the inheritance chain.

Variable Set Assignment and Consumption

• Project Assignment: Associate variable sets with specific projects to control access.

• Consumption: Organization, projects, sub-projects, and workspaces can consume variable sets to inherit their variables.

• Bulk Operations: Assign variable sets to multiple projects simultaneously.

Deleting Variable Sets

• Impact Analysis: View all affected consumers (projects, sub-projects, and workspaces) before deletion.

• Confirmation Required: Type the exact variable set name to confirm deletion.

• Irreversible Action: Deletion permanently removes the set from all consuming resources.

Variable Inheritance Scenarios

Understanding how variables inherit and override each other is crucial for effective variable set management:

Scenario 1: Basic Inheritance

Given:

• Organization has variable env = "global".

• Project /aws has variable env = "aws".

• Variable set prod-config (consumed by workspace) has env = "production".

• Workspace /aws/app has variable env = "app".

Result: Workspace uses env = "app" (workspace value takes highest precedence).

Scenario 2: Variable Set Inheritance

Given:

• Variable set A has region = "us-east-1".

• Variable set B inherits from A and overrides region = "us-west-2".

• Workspace consumes variable set B.

Result: Workspace uses region = "us-west-2" (override value from inherited set).

Scenario 3: Conflict Resolution

Given:

• Variable set A has api_key = "key-from-a".

• Variable set B has api_key = "key-from-b".

• Workspace attempts to consume both sets.

Result: Conflict error - "Variable sets A and B have conflicts on variables: api_key".

Scenario 4: Multiple Inheritance Levels

Given:

• Organization variable: stage = "dev".

• Project variable set: stage = "test".

• Workspace variable: stage = "prod".

Result: Workspace uses stage = "prod" (workspace variable overrides all inherited values).

Setting Organization Variables

Organization variables serve as the foundation of your variable inheritance hierarchy:

Procedure:

1. Access Organization Variables:

   • From the Variable Sets page, click Set organization variables.

2. Configure Global Defaults:

   • Variable Properties:

     • Name: Organization-wide variable name.

     • Value: Default value for all projects and workspaces.

     • Sensitive: Mark sensitive values appropriately.

     • Environment Variable: Export as environment variable.

3. Save Configuration:

   • Click Save to apply organization-wide defaults.

Examples of Variable Set Structures

Here are practical examples of how to organize variable sets:

Environment-Based Variable Sets

Organization Variables:
├── terraform_version = "1.5.0"
├── company_name = "acme-corp"
└── default_region = "us-east-1"

Variable Sets:
├── base-terraform-config
│   ├── terraform_version = "1.5.0"
│   └── provider_version_constraints = "~> 5.0"
├── aws-production-secrets
│   ├── aws_access_key_id = "***" (sensitive)
│   ├── aws_secret_access_key = "***" (sensitive)
│   └── aws_region = "us-east-1"
└── development-overrides
    ├── aws_region = "us-west-2"
    └── instance_type = "t3.micro"

Project-Specific Variable Sets

Project-Level Variables:
├── owner = "platform-team"
├── managed_by = "firefly"
└── environment = "unspecified"

Project-Assigned Variable Sets:
├── aws-credentials (assigned to /aws project)
│   ├── aws_access_key_id = "***" (sensitive)
│   └── aws_secret_access_key = "***" (sensitive)
├── azure-credentials (assigned to /azure project)
│   ├── azure_client_id = "***" (sensitive)
│   └── azure_client_secret = "***" (sensitive)
└── production-config (assigned to /production project)
    ├── environment = "production"
    └── monitoring_enabled = "true"

Inherited Variable Sets

Base Variable Set: network-config
├── vpc_cidr = "10.0.0.0/16"
├── availability_zones = ["us-east-1a", "us-east-1b"]
└── enable_dns_hostnames = "true"

Inherited Variable Set: production-network-config
├── inherits: network-config
├── vpc_cidr = "10.1.0.0/16" (override)
├── availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"] (override)
└── backup_retention_days = "30" (new variable)

Best Practices for Variable Sets

• Plan Your Inheritance Strategy: Design your variable set hierarchy before creating sets to avoid conflicts and ensure efficient inheritance.

• Use Meaningful Names: Choose descriptive names that clearly indicate the variable set's purpose and scope.

• Leverage Inheritance: Use variable set inheritance to create base configurations that can be extended for specific use cases.

• Manage Sensitive Values: Always mark sensitive variables appropriately and avoid exposing secrets in logs or UI.

• Document Your Sets: Use descriptions to explain the purpose and intended usage of each variable set.

• Regular Review: Periodically review variable sets to remove unused variables and optimize inheritance chains.

• Test Inheritance: Verify that variable inheritance works as expected by checking effective variables at the workspace level.

• Avoid Deep Inheritance: Keep inheritance chains shallow to maintain clarity and avoid complex conflict resolution.

• Use Project Assignment: Assign variable sets to specific projects to implement proper access control.

• Monitor Usage: Track which workspaces consume each variable set to understand impact before making changes.

Variable Set Security and Governance

Implementing proper security and governance practices for variable sets is essential:

Security Best Practices

• Sensitive Variable Management:

  • Always mark credentials, API keys, and passwords as sensitive.

  • Use environment variables for secrets that should not appear in Terraform/OpenTofu/Terragrunt configurations.

  • Regularly rotate sensitive values and update variable sets accordingly.

• Access Control:

  • Assign variable sets to specific projects to limit access.

  • Use global variable sets sparingly and only for truly universal values.

  • Regularly review project assignments and user access.

Governance Practices

• Change Management:

  • Implement approval processes for changes to production variable sets.

  • Document changes and their impact on consuming workspaces.

  • Test changes in non-production environments first.

• Audit and Compliance:

  • Track variable set usage across workspaces and projects.

  • Monitor for unused or duplicated variable sets.

  • Maintain documentation of variable set purposes and owners.

Troubleshooting Common Issues

Conflict Resolution

Problem: Variable sets contain conflicting variables. Solution:

• Review inherited variable sets and remove conflicts.

• Use variable precedence rules to determine which value should take priority.

• Consider splitting conflicting sets into separate, more specific sets.

Missing Variables

Problem: Workspace cannot access expected variables from a variable set. Solution:

• Verify the variable set is properly assigned to the workspace's project.

• Check if the workspace is consuming the variable set.

• Ensure the user has proper permissions to access the variable set.

Inheritance Issues

Problem: Variable inheritance is not working as expected. Solution:

• Review the inheritance hierarchy and precedence rules.

• Check for conflicts between inherited variable sets.

• Verify that variable sets are properly configured with inheritance relationships.

By following this comprehensive guide, you can effectively create, manage, and utilize Variable Sets in Firefly Workflows to streamline your infrastructure management, enforce consistency, and implement proper governance across your organization's IaC deployments.