Coralization (import and extract templates)

Background

In large organizations, differences in practices and how software delivery pipelines are created along side how permissions are managed, tend to either cause fragmentation or very rigid non-flexible setups.

The idea of Coral is to strike a balance between alignment and flexibility. Few individuals should have standing permissions, but developers should be able to easily request and create the resources needed for software delivery. The way Coral does this is by using templates.

For organizations that have allowed developers to do as they please, there will be a massive number of software delivery pipelines and practices. Such pipelines are sometimes referred to as "free radicals", as it is hard to say if they comply to the organization's requirements for how to deliver software.

Problem statement

The problem that Coral needs to solve is how to get these "free radical" pipelines into a governed state where it is easier to assess and make them comply to requirements for how to deliver software or to simply get more alignment in practices.

Suggested solution

  1. Coral needs to be able to import pipelines that are not yet in its index, by crawling from a known entry point. This would typically be the repository, but one could imagine other entry points to a pipeline as well. It would then traverse from the repository to all know resources it can get to and associate them with their mirror in Coral.
  2. Next, the ability to extract similarities between the pipelines needs to be built. Coral will have a template for each of the imported pipelines, so it can analyse them all, extract parameters, namespaces and resources that look similar and different. Then one or more composable templates can be created. This needs to include tooling to help reason over the differences.
  3. Then, to get the pipelines under governance, the templates will have to be applied back to the pipeline.
  4. Finally, a person or a team can reason over the templates whether they are compliant or not, make the necessary changes and utilize drift management to deploy the changes back to the pipelines.

Limitations

This is a massive problem to solve so this list is unlikely complete.

  • Coral can only do this for resources it knows about, so if the build pipeline for a repository is say, Jenkins. Then Coral needs to know how to crawl Jenkins, i.e., jenkins will have to be built as a vendor into Coral.
  • If there is no link from the entry point of the import to the resource, there is no way to crawl it.
  • Big data analysis will have an error margin. Making the extracted template more or less accurate.
  • Swapping out template references will likely have to involve an heuristic, potentially making it unreliable.

Value proposition

This solution, whether it is accurate or not, would be a great way to reduce the surface of what compliance experts will have to reason over. Reducing the 1000s of pipelines into a small set of template would be highly valuable.