Custom Infrastructure with Pulumi Dynamic Resource Providers in Python

When automating cloud infrastructure with traditional Infrastructure as Code (IaC) tools like Terraform or CloudFormation, you may encounter limitations when dealing with complex logic, external APIs, or unsupported resources. This is where Pulumi enables you to embed custom logic directly into your infrastructure definitions. In this article, we’ll explore what dynamic resource providers are and how to implement them in Pulumi.

Photo by Emile Perron on Unsplash

Understanding Dynamic Resource Providers

A dynamic resource provider is a custom implementation that you define within your Pulumi project. Unlike standard resources, dynamic resource providers give you complete control over the resource’s lifecycle. You define how the resource is created, updated, and deleted, allowing you to inject any logic necessary to meet your specific needs.

For example, if you need to create a resource that interacts with an external API, like registering a domain name with a third-party provider, a dynamic resource provider would be the perfect fit. You could write the logic to make the necessary API calls during the resource creation phase and handle any responses or errors in a way that fits seamlessly into your infrastructure code.

How Do Dynamic Resource Providers Work?

Dynamic resource providers in Pulumi are characterized by a few main methods that give you full control over the lifecycle of the resource, allowing you to tailor its behavior to your exact requirements:

1. check: This method validates the input properties before any resource actions are taken. It ensures that all necessary properties are provided and that they meet any specific requirements.
2. create: The method handles the actual creation of the resource. This could involve making API calls, executing scripts, or interacting with external systems. The create method returns the resource’s ID and any outputs you want to expose to the rest of your Pulumi stack.
3. update: This method is responsible for updating the resource if any of its properties change. It compares the old and new values and performs the necessary operations to bring the resource up to date.
4. delete: When a resource is deleted, the delete method defines how to clean it up. This might involve making a final API call to remove the resource from an external service or to perform other necessary cleanup tasks.

Implementing a Dynamic Resource Provider

The first step in creating a dynamic resource provider is to define the schema for the inputs and outputs of the resource. This involves creating classes to represent the input properties and the outputs that will be generated once the resource is created.

from item.item_provider import ItemProvider
from pulumi import Input, Output, ResourceOptions
from pulumi.dynamic import *
from typing import Optional

class ItemInputs(object):
    item_id: Input[str]
    name: Input[str]

    def __init__(self, item_id, name):
        self.item_id = item_id
        self.name = name

Here, ItemInputs defines the necessary properties item_id and name that will be passed when creating the resource.

Next, you define the custom resource class that represents the resource being managed. This class inherits from pulumi.dynamic.ResourceProvider and utilizes the dynamic resource provider to handle the resource’s lifecycle.

class Item(Resource):
    item_id: Output[str]
    name: Output[str]

    def __init__(self, name: str, props: ItemInputs, opts: Optional[ResourceOptions] = None):
        """
        Pulumi dynamic custom resource
        :param name: pulumi's resource name
        :param props: resource-specific arguments
        :param opts: optional options
        """

        super().__init__(ItemProvider(), name, {**vars(props)}, opts)

In this example, the Item class represents a custom resource and the ItemProvider is the dynamic provider that manages its lifecycle.

The core of the dynamic resource provider lies in its ability to manage the creation, updating, and deletion of resources through custom logic. This is achieved by implementing a class that inherits from pulumi.dynamic.ResourceProvider and overrides the key methods that define the resource’s behavior.

from pulumi.dynamic import *


class _ItemProviderInputs(object):
    """
    It is the unwrapped version of the same inputs from the ItemInputs class.
    """

    item_id: str
    name: str

    def __init__(self, item_id: str, name: str):
        self.item_id = item_id
        self.name = name


class ItemProvider(ResourceProvider):
    """
    Dynamic resource provider to CRUD a new kind of custom resource
    """

    def check(self, _olds: _ItemProviderInputs, news: _ItemProviderInputs) -> CheckResult:
        """
        It is invoked before any other methods. It has two jobs and it verifies that the inputs (particularly the news)
        are valid or return useful error messages if they are not.
        :param _olds: old input properties that were stored in the state file after the previous update to the resource
        :param news: new inputs from the current deployment
        :return: set of checked inputs
        """

        failures = []

        for key in {**news}:
            value = {**news}[key]
            if not value:
                failures.append(CheckFailure(property_=key, reason="Property's value cannot be emtpy"))

        return CheckResult(inputs={**news}, failures=failures)

    def create(self, inputs: _ItemProviderInputs) -> CreateResult:
        """
        It is invoked when the URN of the resource is not found in the existing state of the deployment.
        :param inputs: inputs to be created
        :return: a set of outputs from the backing provider
        """

        # Call your external provider (e.g. HTTP POST to create entity)
        return CreateResult(id_=inputs["item_id"], outs={**inputs})

    def update(self, _id: str, _olds: _ItemProviderInputs, _news: _ItemProviderInputs) -> UpdateResult:
        """
        It is invoked if the call to diff indicates that a replacement is necessary.
        :param _id: id of the resource as returned by create method
        :param _olds: old outputs from the checkpoint file
        :param _news: new checked inputs from the current deployment
        :return: new set of outputs
        """

        # Call your external provider (e.g. HTTP PUT/PATCH to update entity)
        return UpdateResult({**_news})

    def delete(self, _id: str, _props: _ItemProviderInputs) -> None:
        """
        It is invoked if the URN exists in the previous state but not in the new desired state, or if a replacement
        is needed. The method deletes the corresponding resource from the cloud provider.
        :param _id: id of the resource as returned by create method
        :param _props: the old outputs from the checkpoint file
        """

        # Call your external provider (e.g. HTTP DELETE to delete entity)
        pass

Once the dynamic resource provider is implemented, you can use the custom resource just like any other resource in your Pulumi stack. You instantiate the resource, pass the necessary inputs, and Pulumi handles the rest.

from item.item import Item, ItemInputs
from pulumi import export

def main():
    item = Item("foo", ItemInputs("7", "J4NN0"))
    export("item_id", item.item_id)
    export("name", item.name)

if __name__ == '__main__':
    main()

Dynamic resource providers in Pulumi offer a powerful way to extend your infrastructure as code capabilities, allowing you to manage custom resources and complex scenarios with ease. By leveraging Pulumi’s support for dynamic providers, you can handle unique requirements that go beyond the built-in capabilities of traditional IaC tools, integrating custom logic and external systems seamlessly into your infrastructure management.

For those interested in exploring a practical implementation of dynamic resource providers, you can find the example code on my GitHub repository.

GitHub - J4NN0/pulumi-dynamic-iac: Pulumi dynamic resource provider for custom Infrastructure as…

This example demonstrates the concepts discussed and provides a concrete starting point for your projects.