Kafka Support

From version 4.3.0, Spring Cloud Azure for Kafka supports various types of credentials to authenticate and connect to Azure Event Hubs.

Supported Kafka version

The current version of the starter should be compatible with Apache Kafka Clients 2.0.0 using Java 8 or above.

Supported authentication types

The following authentication types are supported:

  • Plain connection string authentication

    • Direct connection string authentication

    • ARM-based connection string authentication

  • OAuth credential authentication

    • Managed identity authentication

    • Username/password authentication

    • Service principal authentication

    • DefautlAzureCredential authentication

How it works

OAuth credential authentication

This section describes the overall workflow of Spring Cloud Azure OAuth authentication.

Spring Cloud Azure will first build one of the following types of credentials depending on the application authentication configuration:

  • ClientSecretCredential

  • ClientCertificateCredential

  • UsernamePasswordCredential

  • ManagedIdentityCredential

If none of these types of credentials are found, the credential chain via DefaultAzureTokenCredential will be used to obtain credentials from application properties, environment variables, managed identity, or IDEs. For detailed information, see the Authentication section.

Plain connection string authentication

For the connection string authentication mode, you can use connection string authentication directly or use the Azure Resource Manager to retrieve the connection string. For more information about the usage, see the Basic usage for connection string authentication section.

Since version of 4.3.0, connection string authentication is deprecated in favor of OAuth authentications.

Configuration

Configurable properties when using Kafka support with OAuth authentication

Spring Cloud Azure for Kafka supports the following two levels of configuration options:

  1. Spring Cloud Azure for Event Hubs Kafka properties.

  2. The global authentication configuration options of credential and profile with prefixes of spring.cloud.azure.

  3. Kafka-specific level configurations. The Kafka-level configurations are also available for Spring Boot and Spring Cloud Stream binders for common, consumer, producer, or admin scopes, which have different prefixes.

The global properties are exposed via com.azure.spring.cloud.autoconfigure.context.AzureGlobalProperties. The Kafka-specific properties are exposed via org.springframework.boot.autoconfigure.kafka.KafkaProperties (Spring Boot) and org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties (Spring Cloud Stream binder).

The following list shows all supported configuration options.

  • The Spring Cloud Azure global authentication configuration options

    • Prefix: spring.cloud.azure

    • Supported options: spring.cloud.azure.credential., spring.cloud.azure.profile.

For the full list of global configuration options, see the Global properties section.

  • Spring Boot Kafka common configuration

    • Prefix: spring.kafka.properties.azure

    • Example: spring.kafka.properties.azure.credential.*

  • Spring Kafka consumer configuration options

    • Prefix: spring.kafka.consumer.properties.azure

    • Example: spring.kafka.consumer.properties.azure.credential.*

  • Spring Kafka producer configuration options

    • Prefix: spring.kafka.producer.properties.azure

    • Example: spring.kafka.producer.properties.azure.credential.*

  • Spring Kafka admin configuration options

    • Prefix: spring.kafka.admin.properties.azure

    • Example: spring.kafka.admin.properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder common configuration

    • Prefix: spring.cloud.stream.kafka.binder.configuration.azure

    • Example: spring.cloud.stream.kafka.binder.configuration.azure.credential.*

  • Spring Cloud Stream Kafka Binder consumer configuration

    • Prefix: spring.cloud.stream.kafka.binder.consumer-properties.azure

    • Example: spring.cloud.stream.kafka.binder.consumer-properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder producer configuration

    • Prefix: spring.cloud.stream.kafka.binder.producer-properties.azure

    • Example: spring.cloud.stream.kafka.binder.producer-properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder admin configuration

    • Prefix: Not supported, should use Spring Boot Kafka common or admin configuration.

Table 1. Spring Boot Kafka common configuration options
Property Description

spring.kafka.properties.azure.credential.client-certificate-password

Password of the certificate file.

spring.kafka.properties.azure.credential.client-certificate-path

Path of a PEM certificate file to use when performing service principal authentication with Azure.

spring.kafka.properties.azure.credential.client-id

Client ID to use when performing service principal authentication with Azure. This is a legacy property.

spring.kafka.properties.azure.credential.client-secret

Client secret to use when performing service principal authentication with Azure. This is a legacy property.

spring.kafka.properties.azure.credential.managed-identity-enabled

Whether to enable managed identity to authenticate with Azure. If true and the client-id is set, will use the client ID as user assigned managed identity client ID. The default value is false.

spring.kafka.properties.azure.credential.password

Password to use when performing username/password authentication with Azure.

spring.kafka.properties.azure.credential.username

Username to use when performing username/password authentication with Azure.

spring.kafka.properties.azure.profile.environment.active-directory-endpoint

The Azure Active Directory endpoint to connect to.

spring.kafka.properties.azure.profile.tenant-id

Tenant ID for Azure resources.

The configuration options in different levels apply the following rules. The more specific configuration options have higher priority than the common ones. For example:

  • Spring Kafka common configuration options supersede the global options.

  • Spring Kafka consumer configuration options supersede the common options.

  • Spring Kafka producer configuration options supersede the common options.

  • Spring Kafka admin configuration options supersede the common options.

  • The Spring Cloud Stream Kafka Binder options are just like the above.

Configurable properties when using Kafka support with plain connection string authentication

Table 2. Spring Boot Event Hubs for Kafka common configuration options
Property Description

spring.cloud.azure.eventhubs.kafka.enabled

Whether to enable the Azure Event Hubs Kafka support. The default value is true.

spring.cloud.azure.eventhubs.connection-string

Azure Event Hubs connection string. Provide this value when you want to provide the connection string directly.

spring.cloud.azure.eventhubs.namespace

Azure Event Hubs namespace. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

spring.cloud.azure.eventhubs.resource.resource-group

The resource group of Azure Event Hubs namespace. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

spring.cloud.azure.profile.subscription-id

The subscription ID. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

Dependency Setup

Add the following dependency to your project. This will automatically include the spring-boot-starter dependency in your project transitively.

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
Remember to add the BOM spring-cloud-azure-dependencies along with the above dependency. For details, see the Getting started section.

Basic usage

The following sections show the classic Spring Boot application usage scenarios.

Use OAuth authentication

When you use the OAuth authentication provided by Spring Cloud Azure for Kafka, you can configure the specific credentials using the above configurations. Alternatively, you can choose to configure nothing about credentials, in which case Spring Cloud Azure will load the credentials from the environment. This section describes the usages that load the credentials from the Azure CLI environment or the Azure Spring Apps hosting environment.

If you choose to use a security principal to authenticate and authorize with Azure Active Directory for accessing an Azure resource, see the Authorize access with Azure Active Directory section to make sure the security principal has been granted the sufficient permission to access the Azure resource.

The following section describes the scenarios using different Spring ecosystem libraries with OAuth authentication.

Spring Kafka application support

This section describes the usage scenario for Spring Boot application using Spring Kafka or Spring Integration Kafka library.

Dependency setup
<dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
<!-- Using Spring Kafka library only-->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:2.8.6-->
</dependency>
<!-- Using Spring Integration library only -->
<dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:5.5.12-->
</dependency>
Configuration update

To use the OAuth authentication, just specify the Event Hubs endpoint, as shown in the following example:

spring.kafka.bootstrap-servers=<NAMESPACENAME>.servicebus.windows.net:9093
Spring Cloud Stream binder Kafka application support

This section describes the usage scenario for Spring Boot applications using the Spring Cloud Stream binder Kafka library.

Dependency setup
<dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:3.2.3-->
</dependency>
Configuration

To use the OAuth authentication, just specify the Event Hubs endpoint and com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration, as shown in the following example:

spring.cloud.stream.kafka.binder.brokers=<NAMESPACENAME>.servicebus.windows.net:9093
spring.cloud.stream.binders.kafka.environment.spring.main.sources=com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration
If you’re using version 4.3.0, don’t forget to set the spring.cloud.stream.binders.<kafka-binder-name>.environment.spring.main.sources=com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration property to enable the whole OAuth authentication workflow, where kafka-binder-name is kafka by default in a single Kafka binder application. The configuration AzureKafkaSpringCloudStreamConfiguration specifies the OAuth security parameters for KafkaBinderConfigurationProperties, which is used in KafkaOAuth2AuthenticateCallbackHandler to enable Azure Identity. For version after 4.4.0, this property will be added automatically for each Kafka binder environment, so there’s no need for you to add it manually.
Samples

See the azure-spring-boot-samples repository on GitHub.

Use connection string authentication

You can use connection string authentication directly or use the Azure Resource Manager to retrieve the connection string.

Since version of 4.3.0, connection string authentication is deprecated in favor of OAuth authentications.
Dependency setup

Add the following dependencies if you want to migrate your Apache Kafka application to use Azure Event Hubs for Kafka.

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>

If you want to retrieve the connection string using Azure Resource Manager, add the following dependency:

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-resourcemanager</artifactId>
</dependency>
Configuration
Use Event Hubs connection string directly

The simplest way to connect to Event Hubs for Kafka is with the connection string. Just add the following property.

spring.cloud.azure.eventhubs.connection-string=${AZURE_EVENTHUBS_CONNECTION_STRING}
Use Azure Resource Manager to retrieve connection string

If you don’t want to configure the connection string in your application, you can use Azure Resource Manager to retrieve the connection string. To authenticate with Azure Resource Manager, you can also use credentials stored in Azure CLI or another local development tool such as Visual Studio Code or Intellij IDEA. Alternately, you can use Managed Identity if your application is deployed to Azure Cloud. Just be sure the principal has sufficient permission to read resource metadata.

If you choose to use a security principal to authenticate and authorize with Azure Active Directory for accessing an Azure resource, see the Authorize access with Azure Active Directory section to be sure the security principal has been granted the sufficient permission to access the Azure resource.

To use Azure Resource Manager to retrieve the connection string, just add the following property.

spring:
  cloud:
    azure:
      profile:
        subscription-id: ${AZURE_SUBSCRIPTION_ID}
      eventhubs:
        namespace: ${AZURE_EVENTHUBS_NAMESPACE}
        resource:
          resource-group: ${AZURE_EVENTHUBS_RESOURCE_GROUP}

Samples

See the azure-spring-boot-samples repository on GitHub.