Backup and Disaster Recovery

Firefly's Backup and Disaster Recovery (DR) capabilities provide robust tools to safeguard your cloud infrastructure, enabling you to recover quickly from failures and prevent misconfigurations that could lead to outages. This guide details how to use Firefly to mitigate, diagnose, and recover from infrastructure failures, as well as how to proactively prevent them.

Overview

Disaster recovery (DR) is the process of restoring your cloud environment to a healthy state after an incident such as accidental deletion, misconfiguration, or infrastructure failure. Firefly offers:

• Rapid recovery tools for deleted or misconfigured assets.

• Comprehensive mutation logs and clickops events for root cause analysis.

• Proactive notifications and insights to prevent disasters.

• Automated backups and configuration history (coming soon).

DR Readiness: Building a Resilient Foundation

Before diving into recovery procedures, establishing proper disaster recovery readiness is crucial. Firefly provides three foundational capabilities that significantly enhance your DR posture: Codification, Tagging, and Drift Management. These practices ensure your infrastructure is well-documented, organized, and consistent—making recovery faster and more reliable when disasters strike.

Codification: Your Infrastructure's Blueprint

Why it matters for DR: Infrastructure-as-Code (IaC) serves as the definitive blueprint of your environment. When disasters occur, having your entire infrastructure codified means you can recreate resources exactly as they were, with all configurations, dependencies, and relationships intact. Without codification, recovery often involves manual recreation, leading to inconsistencies, missing configurations, and extended downtime.

How Firefly helps: Firefly's Codification feature seamlessly converts existing cloud resources into Infrastructure-as-Code definitions across multiple formats (Terraform, Pulumi, CloudFormation, and more). This ensures that even resources created manually or through ClickOps are captured in code, providing complete infrastructure documentation for recovery scenarios.

Key benefits for DR:

• Complete asset recreation: Regenerate exact resource configurations during recovery.

• Dependency mapping: Automatically capture resource relationships and dependencies.

• Version control: Track infrastructure changes and revert to known-good states.

• Consistency: Ensure recovered infrastructure matches original specifications.

For detailed instructions on codifying your infrastructure, see Codification.

Tagging: Organizing for Rapid Recovery

Why it matters for DR: Proper tagging strategies enable rapid identification and prioritization of critical resources during disaster scenarios. Tags help you quickly locate business-critical assets, understand resource ownership, and implement recovery procedures in the correct order based on business impact and dependencies.

How Firefly helps: Firefly's governance engine can enforce tagging policies across your entire infrastructure, ensuring consistent labeling practices. You can create policies that require specific tags (environment, criticality, owner, backup-schedule) and automatically remediate missing or incorrect tags.

Key benefits for DR:

• Asset prioritization: Quickly identify critical resources that need immediate recovery.

• Ownership clarity: Know who to contact for specific resources during incidents.

• Environment segregation: Separate production, staging, and development resources.

• Backup scheduling: Organize resources by backup requirements and retention policies.

Use Firefly's Policy & Governance features to implement and enforce tagging standards. For more information, see Policy & Governance.

Drift Management: Maintaining Configuration Integrity

Why it matters for DR: Configuration drift occurs when live infrastructure diverges from its IaC definition. During disaster recovery, drifted resources may not behave as expected, leading to failed recoveries or inconsistent environments. Maintaining alignment between code and cloud ensures predictable recovery outcomes.

How Firefly helps: Firefly continuously monitors your infrastructure for drift, comparing live resource configurations against their IaC definitions. When drift is detected, Firefly provides clear remediation options to either update the code to match the current state or reconcile the cloud resources to match the desired IaC configuration.

Key benefits for DR:

• Predictable recovery: Ensure recovered resources behave exactly as designed.

• Configuration accuracy: Maintain consistency between documentation and reality.

• Reduced recovery time: Eliminate surprises during critical recovery operations.

• Compliance maintenance: Keep security and compliance configurations intact.

For step-by-step drift remediation procedures, see Remediating Drifted Assets.

Implementing DR Readiness

To establish strong DR readiness using Firefly:

1. Start with Codification: Use Firefly to codify all unmanaged resources, prioritizing critical systems first.

2. Implement Tagging Policies: Create and enforce consistent tagging standards across your infrastructure

3. Monitor and Remediate Drift: Set up drift alerts and establish regular remediation cycles.

4. Test Recovery Procedures: Regularly validate that your codified infrastructure can be successfully deployed in recovery scenarios.

Recovering from Infrastructure Failure

When an infrastructure failure occurs, Firefly provides tools to help you diagnose, resolve, and recover. The recovery process depends on whether you know which asset caused the failure.

Recovering Deleted Assets: When the Responsible Asset is Known

If you know which asset was deleted or misconfigured (e.g., a team member accidentally deleted a resource), you can restore it using Firefly's codification and GitOps integration.

Procedure:

1. Click on Inventory > Deleted.

   • This view lists all assets that have been deleted from your environment.

2. Filter by Time Range.

   • Use the filter to narrow down the list to the relevant time period.

3. Select the Deleted Asset and Codify.

   • Click on the asset that was deleted. Use the Codify action to generate the Infrastructure-as-Code (IaC) template for the asset.

4. Create a Pull Request.

   • Firefly will prompt you to select the appropriate repository and branch for your GitOps workflow. Submit a pull request to restore the asset via code.

5. Review and Merge.

   • Once reviewed and merged, your CI/CD pipeline will recreate the asset in your cloud environment.

Tip: This process ensures that the restored asset is managed by code, reducing the risk of future drift or manual misconfiguration.

Viewing Mutations & ClickOps Events: When the Responsible Asset is Unknown

If you do not know which asset caused the failure, use Firefly's mutation tracking to investigate recent changes and identify the root cause.

Procedure:

1. Click on Event Center

   • View all mutations and clickops events in your environment.

2. Apply Filters

   • Filter assets by data source, environment, account, and location to narrow your search.

3. Review Mutation Log or ClickOps Event

   • Click on an asset name to transfer to the asset page and open its mutation log to see a timeline of configuration changes.

4. Codify Revision

   • For any suspicious or recent change, select the revision date and use Codify Revision to generate the IaC template for that point in time.

5. Revert via Pull Request

   • Restore the asset to a previous configuration by submitting a pull request.

Tip: Mutation logs and clickops events provide a detailed audit trail, including who made each change and what was modified, making root cause analysis straightforward.

Preventing Misconfiguration and Reliability Risks

Proactive prevention is key to avoiding disasters. Firefly enables you to set up notifications and subscribe to insights that alert you to risky configurations or changes.

Receiving Notifications on Asset Changes

Stay informed about changes in your infrastructure by subscribing to notifications. These alerts help you:

• Detect new drifts and clickops events.

• Monitor IaC deployment failures.

• Get alerts on policy and guardrail violations.

How to Subscribe:

• Go to Settings > Notifications in Firefly.

• Choose your preferred notification channels (Slack, Teams, email, etc.).

• Select which events or asset changes should trigger notifications (e.g., deletions, drifts, policy violations).

Tip: Fine-tune your notification settings to avoid alert fatigue and focus on critical events.

For more information, check the Notifications guide.

Subscribing to Policy Checks for Reliability and Misconfiguration Prevention

Firefly Governance contains policy-driven checks that highlight risky configurations. Subscribing to these policies helps you proactively address issues before they lead to outages.

Example of Top 5 Policies to Reduce Disaster Risk:

1. Reliability: K8s Deployments running containers without a configured CPU limit

   • K8s Deployments running containers without a configured CPU limit are vulnerable to resource exhaustion and can lead to outages.

2. Reliability: AWS Database Instances in a Single Availability Zone

   • Databases in one zone are vulnerable to zone failures. Multi-AZ deployment is recommended for resilience.

3. Reliability: AWS RDS Instance Without Deletion Protection

   • Without deletion protection, accidental or automated deletions can cause permanent data loss.

4. Reliability: AWS DynamoDB Tables Without Point-in-Time Recovery

   • Enable point-in-time recovery to restore tables to any previous state and protect against data loss.

5. Misconfiguration: AWS ELB/LB Without Access Logs Enabled

   • Access logs are essential for troubleshooting, monitoring, and security analysis. Enable logging to maintain visibility.

How to Subscribe:

• Go to Settings > Governance in Firefly.

• Subscribe to the above policies and configure notification preferences.

Tip: Regularly review policy recommendations and remediate flagged issues to maintain a resilient infrastructure.

Summary

Firefly's backup and disaster recovery features empower you to:

• Rapidly recover from accidental deletions or misconfigurations.

• Investigate and revert problematic changes.

• Proactively prevent outages with real-time notifications and policy-driven insights.

By integrating these tools and practices into your operations, you can ensure your cloud environment remains resilient, auditable, and secure.