Pulumi ESC Config: Simplify Your Local Configuration Management

This is a submission for the Pulumi Deploy and Document Challenge: Shhh, It's a Secret! What I Built I created Pulumi ESC Config, a command-line tool that bridges the gap between Pulumi's Environment Secrets and Configuration (ESC) service and application configuration files. This tool securely fetches secrets from Pulumi ESC and transforms them into various configuration formats (.env, JSON, YAML) that any application can easily consume. The tool addresses a common challenge in modern development: securely managing secrets across different environments while maintaining a consistent interface for applications to access them. Live Demo Link This project doesn't have a live Link. Project Repo The project is available on GitHub: pulumi-esc-config Key features: Secure fetching of secrets from Pulumi ESC Support for multiple output formats (.env, JSON, YAML) Environment-specific configuration generation Simple CLI interface with comprehensive documentation My Journey The Problem As a developer working across multiple projects and environments, we often found ourselve constantly copying secrets between different configuration files and worrying about accidentally committing sensitive information to version control. We therefore need a solution that would: Keep secrets secure and centralized Support multiple application types (Node.js, Python, etc.) Make environment-specific configuration easy to manage The Solution Pulumi ESC offered the perfect foundation for secure secret management, but I needed a way to seamlessly integrate it with different application frameworks. That's when I decided to build the ESC Config Generator. Development Process I started by exploring the Pulumi ESC SDK and understanding how to authenticate and retrieve secrets. The JavaScript SDK made this relatively straightforward: const client = esc.DefaultClient(); const openEnv = await client.openAndReadEnvironment(orgName, projName, envName); Next, I built the formatting layer to transform the retrieved secrets into different configuration file formats. This involved creating custom serializers for .env, JSON, and YAML formats: // For .env files content = Object.entries(secrets) .map(([key, value]) => `${key}=${value}`) .join("\n"); // For YAML files content = yaml.dump(secrets); // For JSON files content = JSON.stringify(secrets, null, 2); The most challenging part was handling the different ways applications consume configuration files. I researched various frameworks and added examples to the documentation showing how to load the generated configurations in Node.js, Python. Lessons Learned Through this project, I gained valuable insights into: Secret Management Best Practices: The importance of secret rotation, least privilege access, and environment isolation SDK Integration: How to work with third-party SDKs and handle authentication flows CLI Design: Creating an intuitive command-line interface with clear documentation Cross-Language Support: Supporting multiple programming languages and frameworks Using Pulumi ESC Pulumi ESC provided the ideal foundation for this project for several reasons: Centralized Secret Management: ESC offers a secure, centralized place to store secrets with proper access controls Environment Segmentation: The project/environment structure aligns perfectly with development workflows (dev/staging/prod) SDK Availability: The JavaScript SDK made integration straightforward Strong Security Model: Pulumi's security practices gave me confidence in building on their platform The tool interacts with Pulumi ESC through these key steps: Authentication via the ESC CLI: esc login. Actually this was the easiest and secured way of authentication I found to integrate in this project Defining the organization, project, and environment context Retrieving secrets using the SDK: client.openAndReadEnvironment() Transforming and saving the secrets to the appropriate format Here's a practical example of generating a .env file for a development environment: npm run generate -- --proj myproject --env dev --type env --output .env Benefits Over Alternative Approaches Compared to other secret management approaches, using Pulumi ESC provided: Better Security: Secrets never need to be manually copied or stored in insecure locations Simplified Workflows: Developers can focus on code instead of configuration management Consistent Experience: The same approach works across all environments and languages Audit Capabilities: All secret access is logged and auditable Next Steps I'm planning to enhance the tool with: Secret rotation reminders and automation Integration of OIDC generation for differents providers Support for more configuration formats A web interface for non-technical users If you're struggling with configuration management acro

Apr 6, 2025 - 20:23

Pulumi ESC Config: Simplify Your Local Configuration Management

This is a submission for the Pulumi Deploy and Document Challenge: Shhh, It's a Secret!

What I Built

I created Pulumi ESC Config, a command-line tool that bridges the gap between Pulumi's Environment Secrets and Configuration (ESC) service and application configuration files. This tool securely fetches secrets from Pulumi ESC and transforms them into various configuration formats (.env, JSON, YAML) that any application can easily consume.

The tool addresses a common challenge in modern development: securely managing secrets across different environments while maintaining a consistent interface for applications to access them.

Live Demo Link

This project doesn't have a live Link.

Project Repo

The project is available on GitHub: pulumi-esc-config

Key features:

Secure fetching of secrets from Pulumi ESC
Support for multiple output formats (.env, JSON, YAML)
Environment-specific configuration generation
Simple CLI interface with comprehensive documentation

My Journey

The Problem

As a developer working across multiple projects and environments, we often found ourselve constantly copying secrets between different configuration files and worrying about accidentally committing sensitive information to version control. We therefore need a solution that would:

Keep secrets secure and centralized
Support multiple application types (Node.js, Python, etc.)
Make environment-specific configuration easy to manage

The Solution

Pulumi ESC offered the perfect foundation for secure secret management, but I needed a way to seamlessly integrate it with different application frameworks. That's when I decided to build the ESC Config Generator.

Development Process

I started by exploring the Pulumi ESC SDK and understanding how to authenticate and retrieve secrets. The JavaScript SDK made this relatively straightforward:

const client = esc.DefaultClient();
const openEnv = await client.openAndReadEnvironment(orgName, projName, envName);

Next, I built the formatting layer to transform the retrieved secrets into different configuration file formats. This involved creating custom serializers for .env, JSON, and YAML formats:

// For .env files
content = Object.entries(secrets)
  .map(([key, value]) => `${key}=${value}`)
  .join("\n");

// For YAML files
content = yaml.dump(secrets);

// For JSON files
content = JSON.stringify(secrets, null, 2);

The most challenging part was handling the different ways applications consume configuration files. I researched various frameworks and added examples to the documentation showing how to load the generated configurations in Node.js, Python.

Lessons Learned

Through this project, I gained valuable insights into:

Secret Management Best Practices: The importance of secret rotation, least privilege access, and environment isolation
SDK Integration: How to work with third-party SDKs and handle authentication flows
CLI Design: Creating an intuitive command-line interface with clear documentation
Cross-Language Support: Supporting multiple programming languages and frameworks

Using Pulumi ESC

Pulumi ESC provided the ideal foundation for this project for several reasons:

Centralized Secret Management: ESC offers a secure, centralized place to store secrets with proper access controls
Environment Segmentation: The project/environment structure aligns perfectly with development workflows (dev/staging/prod)
SDK Availability: The JavaScript SDK made integration straightforward
Strong Security Model: Pulumi's security practices gave me confidence in building on their platform

The tool interacts with Pulumi ESC through these key steps:

Authentication via the ESC CLI: esc login. Actually this was the easiest and secured way of authentication I found to integrate in this project
Defining the organization, project, and environment context
Retrieving secrets using the SDK: client.openAndReadEnvironment()
Transforming and saving the secrets to the appropriate format

Here's a practical example of generating a .env file for a development environment:

npm run generate -- --proj myproject --env dev --type env --output .env

Benefits Over Alternative Approaches

Compared to other secret management approaches, using Pulumi ESC provided:

Better Security: Secrets never need to be manually copied or stored in insecure locations
Simplified Workflows: Developers can focus on code instead of configuration management
Consistent Experience: The same approach works across all environments and languages
Audit Capabilities: All secret access is logged and auditable

Next Steps

I'm planning to enhance the tool with:

Secret rotation reminders and automation
Integration of OIDC generation for differents providers
Support for more configuration formats
A web interface for non-technical users

If you're struggling with configuration management across environments, give Pulumi ESC Config Generator a try!