At Buildo, we consistently begin every project with an understanding of the needs implied, including internal projects. As part of the DevOps team, we decided to take action to improve our developer team's workflow and autonomy (you can read how we started the development of our Clustero Developer Platform in our previous blog post, "The Missing Link in Simplifying Database Complexity”). To quickly sum up, we identified the key features that a developer would require through interviews and previous experience:

• Expose the app to the internet via an HTTPS URL
• Authentication for access segregation utilizing Buildo’s Google User
• The capability of injecting secrets from 1Password, which we use commonly at Buildo

Currently, developers wanting to explore new technology need the DevOps team to start. Our objective is to remove barriers that inhibit independent experimentation. Unchecked, this leads to an overabundance of unique, snowflake environments that are difficult to manage and share knowledge.

In this post, we will discuss our approach to enabling independent experimentation without DevOps support, and the implementation details of our solutions. We'll also cover how developers can interface with our system using simple YAML files, the role of Argo CD and the App Operator, and our plans for future developments to integrate databases seamlessly.

Interface with Developers

So, how can a developer experiment without generating snowflakes or requiring DevOps support?

The answer is as simple as adding a rudimentary YAML file to a repository, allowing ArgoCD to seamlessly install your app into the cluster.

Argo CD, a declarative, GitOps continuous delivery tool for Kubernetes, will automatically examine Buildo’s repositories for the presence of an App CR (the Custom Resource defined by the Operator) and subsequently install it into the cluster, triggering the App operator for the primary installation.

apiVersion: applications.buildo.io/v1alpha1
kind: App
metadata:
  name: my-app
  namespace: my-app
spec:
  image: my-app-image:v1.2.3

Simple, right? But how about acquiring an internet address without divulging your work to the world? A few extra lines and the app operator will manage this on your behalf.

apiVersion: applications.buildo.io/v1alpha1
kind: App
metadata:
  name: my-app
  namespace: my-app
spec:
  image: my-app-image:v1.2.3
  env:
    MY_ENV_VAR: wow
  route:
    domain: my-app.buildo.io
    port: 3000
    path: /custom-path
  auth:
    enabled: true
  secretsFrom: 1password-item
  storage:
    - /cache
    - /data

The App CR in Detail

You may have noticed that the App kind isn't a native K8s resource. We've extended the Kubernetes API with a custom resource definition to allow an operator, Buildo’s App operator, to convert this simple YAML into multiple, complex resources.

The App Operator is a crucial part of our architecture. It was created using the Operator SDK with Helm. We defined the Custom Resource Definition as our agreement with developers. We chose Helm as a templating language because it is widely used and easy to understand. However, one downside of Helm is that the client controls the package content and version. With the Helm Operator approach, the DevOps team is responsible for maintaining the operator. This means that developers don't have to change the package version if an infrastructure component is updated. You can also check out our previous blog post on this topic for more information.

Ingress, Service, and Certificate

In the Ingress, we link the user-specified domain to the Service. The domain links to the load balancer's IP via ExternalDNS. If the developer opts for authentication, the operator includes the required annotations to incorporate Ory Oathkeeper. An affiliated Certificate - issued by the cert-manager operator using Let’s Encrypt - is generated automatically.

...
spec:  
  route:
    domain: my-app.buildo.io
    port: 3000
    path: /custom-path
  auth:
    enabled: true
 ...

Ory Oathkeeper authorizes incoming HTTP requests. It can function as a Policy Enforcement Point within your cloud architecture, serving as a reverse proxy for your upstream API or web server, rejecting unauthorized requests and forwarding authorized ones.

ExternalSecret

ExternalSecret is essential to generate a Secret from a 1Password item, which is then bound as an environment variable in the Deployment. ExternalSecret's operator and 1Password Secrets Automation facilitate this.

...
spec:  
	secretsFrom: 1password-item
...

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur, and more. The operator fetches information from external APIs and automatically injects the data into a Kubernetes Secret.

PersistenceVolumeClaim

The PersistenceVolumeClaim (PVC) procures a volume that is automatically mounted to our Deployment. The PVC creates a PersistenceVolume using the default storage class of clustero: EBS through the aws-ebs-csi-driver.

...
spec:  
	storage:
    - /var/lib/cache
    - /data
...

Our First Case Study: Conto and the Evolution of our Developer Platform

We didn't have to wait long for the first opportunity to test our work: Conto, one of our internal tools, needed something like our platform to improve its workflow. Since it was an internal tool without a specific budget, it was quite simple: a Python web app with Streamlit. After some time, together with the Conto’s team, we realized they were using a volume to cache some data, and they wanted not to lose it every time the app was restarted for an update. We saw an opportunity to enhance our Clustero Developer Platform by addressing Conto's problem, so we decided to augment the platform's features, adding the possibility of persisting volumes and ensuring that data would not be lost during app restarts.

This success story quickly resonated with other internal teams, leading to the adoption of Clustero new feature across various small projects. What started as a simple intuition evolved into a feature that empowered multiple teams and reduced dependencies on DevOps support.

Future Developments

We've succeeded in eliminating snowflake environments in favor of rapidly deployed applications. Our journey does not end here, though. We aim to integrate our solution into preexisting projects. To achieve this, we need a crucial new feature: databases. As you may have read in Benedetto’s post, in Clustero, we are already providing a straightforward, declarative method to operate a database and will incorporate this feature into the App CR.