Writing Coral Templates

A Coral template is the blueprint for automated resource creation and management. One can author a Coral template as an Azure DevOps git repo. We are planning to support other template authoring options in the future.

We aim to make template authoring a seamless experience. This is why we're working on a Coral template for bootstrapping new Coral templates. Until this is available, please follow the instructions below to get started.

Creating Your First Template

  1. Create a new git repository in Azure DevOps.
    • It can be in any project, but should be in the same account you plan to use the template in. Using templates from other Azure DevOps accounts is not currently supported.
  2. Protect your template with branch policies.
    • Add policies that make sense for your organization.
    • You can also register a status policy that will validate the template against Coral. For now, please contact us if you need help setting this up.
  3. Add an README.md to the root of the repo.
    • In this file, describe your template using the markdown syntax.
    • This content will be surfaced in Coral UI when users are presented with the template selection.
  4. Create a "namespaces" (and optionally "artifacts") folder in the root of the repo.
    • In "namespaces", there needs to be 1 or more namespace manifests in yml format.
    • In "artifacts", one can optionally put any arbitrary artifact files. These can be referenced from namespace manifests.
  5. Start declaring resources in the namespace manifests.

Creating Your First Namespace Manifest

  1. Select a unique namespace id.
    • This is used to uniquely reference resources across namespaces within a single template.
  2. Select your namespace provider/schema.
    • Namespace providers are vendorized services that facilitate automated resource orchestration through Coral.
    • Each namespace provider has a unique vendorized URI and a corresponding namespace manifest schema.
    • You can find a list of available namespace providers (with documentation and samples) in the Reference section.
    • If there is something that you need and is not available yet, we welcome contrubitions!
  3. Declare the resources using the selected provider/schema.
    • A single namespace manifest can declare many resources.
      • Each one needs to be assigned an id. This id needs to be unique within the namespace. Namespace id and resource id together form a resource address which can be used to uniquely reference any resource in the template.
      • Each one can optionally be assigned a stage.
  4. Parameterize the namespace manifest.
    • These are used to facilitate variations between different groups of resources managed by the same template.
    • Read this for more information about namespace parameters.
  5. Reference artifacts.
    • In some cases, it is hard to fully declare a resource using only yml without increasing the manifest complexity significantly. In these cases, artifacts can be referenced from yml.
  6. Declare dependencies between resources.
    • Resources often depend on each other. When a resource depends on another resource that is declared in the same template, we need to declare this dependency using something called a Dynamic Variable.

Namespace Parameters

Namespace parameters declare required user input for a particular namespace of a template. They are used to dynamically generate the wizard when a user selects a particular template in the UI. Parameters can only be consumed in the manifest file that declared them.

Each parameter declaration needs to have the following defined:

  • id: A string that can be used to uniquely identify a parameter within a namespace manifest.
  • display: A string that will be presented to the user explaining the input field in the wizard.
  • placeHolder (optional): Used to prepopulate the input giving more information about requirements.
  • valueProvider (optional): Used for complex input types. Read more about them here.
  • type (optional): only applicable if valueProvider is specified. Possible values are:
    • validator: input is free text, but that text is validated against a provider.
    • single: input can only have a single input (given by provider). It will be rendered in the form, but will not be editable
    • hidden: same as single, only will not be visible.
    • if not specified, "dropdown" will be assumed

If no other properties are defined, a simple text input type with no input validation will be assumed.

Here's an example namespace manifest using the "local://vsts/namespaces/repos/v1" namespace provider with a basic parameter declaration:

namespace: repos
schema: local://vsts/namespaces/repos/v1
parameters:
- id: RepositoryName
  display: Repository name
payload:
  repositories:
  - id: repository
    stage: Incubation
    name: '{RepositoryName}'

This manifest declares a single git repository and nothing else. It is expecting the user to supply the name for it using the "RepositoryName" parameter.

The "name" property of the repository resource is defined by the "RepositoryName" parameter. String interpolation is also supported.

The schema URI defines the schema for the payload part of the manifest. Parameters can only be referenced from within the payload.

Value Providers

In most cases, simple text input is not enough. This is where value providers can help.

Here's an example of how a value provider can be used to validate that the user input is a valid AAD email enabled security group:

parameters:
- id: RequiredReviewers
  display: Required reviewers email
  valueProvider: local://aad/parameters/email?type=group&mailenabled=true&security=true
  type: validator
  • "valueProvider" points to a vendorized service that provides AAD email input validation. This particular provider accepts "mailenabled" and "security" query parameters for validation behavior customization.
  • "type" declares that this input is free text that will be validated against the provider.

Here's an example of how a value provider can be used to help user select one of available Service Endpoints for ESRP signing:

parameters:
- id: EsrpEndpointId
  display: Esrp Service Endpoint
  placeHolder: Select the endpoint to use for signing
  valueProvider: local://vsts/parameters/serviceendpoints?type=PRSS

As type was not specified, it will default to dropdown.

All value providers support the "validator" type, but the same is not true for other types. You can find all available value providers with documentation and samples in the Reference section.

Artifacts

Files placed in the "artifacts" folder can be referenced from namespace manifest. This is useful when a resource requires a lot of complex configuration that should not be surfaced directly in yml.

Here's an example of an artifact reference to an Azure DevOps build definition json:

namespace: pipelines
schema: local://vsts/namespaces/pipelines/v1
parameters:
- id: QueueId
  display: Agent Queue
  valueProvider: local://vsts/parameters/agentqueues
- id: RepositoryName
  display: Repository Name
- id: RepositoryId
  display: Repository Id
artifacts:
- id: BuildDefinitionJson
  path: BuildDefinition.json
  encoding: base64
payload:
  build:
  - id: buildDefinition
    stage: Incubation
    jsonBase64: '{BuildDefinitionJson}'
    jsonModel:
      queueId: '{QueueId}'
      repositoryName: '{RepositoryName}'
      repositoryId: '{RepositoryId}'

As you can see from above, the buildDefinition resource declaration contains the "jsonBase64" field which expects a base64 encoded templatized json string. It also contains the jsonModel which will be used to generate the final json. Both base64 decoding and json template expansion are responsibility of the vendorized namespace provider.

In this case, the "BuildDefinitionJson" artifact is configured to be base64 encoded. This is useful because the artifact is treated by the template engine the same way that regular parameters are. The file is loaded, contents encoded, and the resulting base64 string stored in the "BuildDefinitionJson" artifact. This artifact can be referenced the same way one would reference a regular parameter, and the value will be expanded in place in yaml. It is therefore important to encode the contents to avoid breaking yml syntax.

Dynamic Variables

If you look closely at the build definition declaration example from above, you can probably notice a problem. The namespace manifest expects the user to pass in repository name and repository id. This is problematic for several reasons:

  • The id needs to match the name. Asking the user for both is error prone.
  • There is no validation on input parameters.
  • If the user needs to input all these values, this means that the corresponding repository that the id and name belong to needs to bre created before the user can ask for the build definition to be created. This means repository and build definition couldn't be created by the same template.

Luckily, Coral supports declaring dependencies between any two resources in a single template. Cross namespace dependencies are also supported. In fact, inter-resource dependencies are one of the most important facets of Coral and is what drives the entire resource orchestrator. A dependency can be declared using something called a dynamic variable.

A dynamic variable is similar to a parameter, but instead of referencing a user input value, it is referencing a property of a resource in the same template. The referenceable properties are defined by the model of the referenced resource.

Resource Models

Every resource in a namespace has a corresponding configuration which is determined by a portion of a namespace manifest, parameters, attributes and dynamic variables. This configuration needs to be available at the moment the resource is being created. Once created, we will learn additional information about the new resource. For example, before creating a git repository in Azure DevOps, we need to supply its name, but only after it is created will we learn its Azure DevOps id.

The resource model is an object that contains the entire resource configuration, and additionally some information that is unique to this instance of a resource. These properties are often required when creating dependent resources.

Here's the same example from above, but with above described issues resolved using dynamic variables:

namespace: pipelines
schema: local://vsts/namespaces/pipelines/v1
parameters:
- id: QueueId
  display: Agent Queue
  valueProvider: local://vsts/parameters/agentqueues
artifacts:
- id: BuildDefinitionJson
  path: BuildDefinition.json
  encoding: base64
payload:
  build:
  - id: buildDefinition
    stage: Incubation
    jsonBase64: '{BuildDefinitionJson}'
    jsonModel:
      queueId: '{QueueId}'
      repositoryName: ''
      repositoryId: ''

A dynamic variable consists of 3 parts separated with a dot:

  1. namespace id: the value of the "namespace" property at the top of the yml declaring the referenced resource
  2. resource id: the value of the "id" property of the referenced resource
  3. model property name: name of the property of the referenced resource model. All referenceable properties are documented in the Reference section.

The syntax for dynamic variable references is similar to parameters and artifacts. The only difference is that instead of ‘{}', we must enclose the reference in ‘'. It is important to note for all parameters, artifacts and dynamic variables, that ‘' are required to avoid breaking yml syntax. Since yml is an extension of json, {} is reserved for declaring objects.

Dynamic Variables Restrictions

When using dynamic variables, one must be careful not to create circular dependencies. If this does happen, Coral will detect it and will invalidate the template and report the issue. These dependencies should also be considered when tagging resources with stages. When the resource dependency graph is collapsed into a stage graph, there shouldn't be any circular dependencies between stages. If there are, Coral will again invalidate the template and report the issue.