Kafka Support

From version 4.3.0, Spring Cloud Azure for Kafka supports various types of credentials to authenticate and connect to Azure Event Hubs.

Supported Kafka version

The current version of the starter should be compatible with Apache Kafka Clients 2.0.0 using Java 8 or above.

Supported authentication types

The following authentication types are supported:

  • Plain connection string authentication

    • Direct connection string authentication

    • ARM-based connection string authentication

  • OAuth credential authentication

    • Managed identity authentication

    • Username/password authentication

    • Service principal authentication

    • DefautlAzureCredential authentication

How it works

OAuth credential authentication

This section describes the overall workflow of Spring Cloud Azure OAuth authentication.

Spring Cloud Azure will first build one of the following types of credentials depending on the application authentication configuration:

  • ClientSecretCredential

  • ClientCertificateCredential

  • UsernamePasswordCredential

  • ManagedIdentityCredential

If none of these types of credentials are found, the credential chain via DefaultAzureTokenCredential will be used to obtain credentials from application properties, environment variables, managed identity, or IDEs. For detailed information, see the Authentication section.

Plain connection string authentication

For the connection string authentication mode, you can use connection string authentication directly or use the Azure Resource Manager to retrieve the connection string. For more information about the usage, see the Basic usage for connection string authentication section.

Since version of 4.3.0, connection string authentication is deprecated in favor of OAuth authentications.

Configuration

Configurable properties when using Kafka support with OAuth authentication

Spring Cloud Azure for Kafka supports the following two levels of configuration options:

  1. The global authentication configuration options of credential and profile with prefixes of spring.cloud.azure.

  2. Kafka-specific level configurations. The Kafka-level configurations are also available for Spring Boot and Spring Cloud Stream binders for common, consumer, producer, or admin scopes, which have different prefixes.

The global properties are exposed via com.azure.spring.cloud.autoconfigure.context.AzureGlobalProperties. The Kafka-specific properties are exposed via org.springframework.boot.autoconfigure.kafka.KafkaProperties (Spring Boot) and org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties (Spring Cloud Stream binder).

The following list shows all supported configuration options.

  • The Spring Cloud Azure global authentication configuration options

    • Prefix: spring.cloud.azure

    • Supported options: spring.cloud.azure.credential., spring.cloud.azure.profile.

For the full list of global configuration options, see the Global properties section.

  • Spring Boot Kafka common configuration

    • Prefix: spring.kafka.properties.azure

    • Example: spring.kafka.properties.azure.credential.*

  • Spring Kafka consumer configuration options

    • Prefix: spring.kafka.consumer.properties.azure

    • Example: spring.kafka.consumer.properties.azure.credential.*

  • Spring Kafka producer configuration options

    • Prefix: spring.kafka.producer.properties.azure

    • Example: spring.kafka.producer.properties.azure.credential.*

  • Spring Kafka admin configuration options

    • Prefix: spring.kafka.admin.properties.azure

    • Example: spring.kafka.admin.properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder common configuration

    • Prefix: spring.cloud.stream.kafka.binder.configuration.azure

    • Example: spring.cloud.stream.kafka.binder.configuration.azure.credential.*

  • Spring Cloud Stream Kafka Binder consumer configuration

    • Prefix: spring.cloud.stream.kafka.binder.consumer-properties.azure

    • Example: spring.cloud.stream.kafka.binder.consumer-properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder producer configuration

    • Prefix: spring.cloud.stream.kafka.binder.producer-properties.azure

    • Example: spring.cloud.stream.kafka.binder.producer-properties.azure.credential.*

  • Spring Cloud Stream Kafka Binder admin configuration

    • Prefix: Not supported, should use Spring Boot Kafka common or admin configuration.

Table 1. Spring Boot Kafka common configuration options
Property Description

spring.kafka.properties.azure.credential.client-certificate-password

Password of the certificate file.

spring.kafka.properties.azure.credential.client-certificate-path

Path of a PEM certificate file to use when performing service principal authentication with Azure.

spring.kafka.properties.azure.credential.client-id

Client ID to use when performing service principal authentication with Azure. This is a legacy property.

spring.kafka.properties.azure.credential.client-secret

Client secret to use when performing service principal authentication with Azure. This is a legacy property.

spring.kafka.properties.azure.credential.managed-identity-enabled

Whether to enable managed identity to authenticate with Azure. If true and the client-id is set, will use the client ID as user assigned managed identity client ID. The default value is false.

spring.kafka.properties.azure.credential.password

Password to use when performing username/password authentication with Azure.

spring.kafka.properties.azure.credential.username

Username to use when performing username/password authentication with Azure.

spring.kafka.properties.azure.profile.environment.active-directory-endpoint

The Azure Active Directory endpoint to connect to.

spring.kafka.properties.azure.profile.tenant-id

Tenant ID for Azure resources.

The configuration options in different levels apply the following rules. The more specific configuration options have higher priority than the common ones. For example:

  • Spring Kafka common configuration options supersede the global options.

  • Spring Kafka consumer configuration options supersede the common options.

  • Spring Kafka producer configuration options supersede the common options.

  • Spring Kafka admin configuration options supersede the common options.

  • The Spring Cloud Stream Kafka Binder options are just like the above.

Configurable properties when using Kafka support with plain connection string authentication

Table 2. Spring Boot Event Hubs for Kafka common configuration options
Property Description

spring.cloud.azure.eventhubs.kafka.enabled

Whether to enable the Azure Event Hubs Kafka support. The default value is true.

spring.cloud.azure.eventhubs.connection-string

Azure Event Hubs connection string. Provide this value when you want to provide the connection string directly.

spring.cloud.azure.eventhubs.namespace

Azure Event Hubs namespace. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

spring.cloud.azure.eventhubs.resource.resource-group

The resource group of Azure Event Hubs namespace. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

spring.cloud.azure.profile.subscription-id

The subscription ID. Provide this value when you want to retrieve the connection information through Azure Resource Manager.

Dependency Setup

Add the following dependency to your project. This will automatically include the spring-boot-starter dependency in your project transitively.

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
Remember to add the BOM spring-cloud-azure-dependencies along with the above dependency. For details, see the Getting started section.

Basic usage

The following sections show the classic Spring Boot application usage scenarios.

Use OAuth authentication

When you use the OAuth authentication provided by Spring Cloud Azure for Kafka, you can configure the specific credentials using the above configurations. Alternatively, you can choose to configure nothing about credentials, in which case Spring Cloud Azure will load the credentials from the environment. This section describes the usages that load the credentials from the Azure CLI environment or the Azure Spring Apps hosting environment.

If you choose to use a security principal to authenticate and authorize with Azure Active Directory for accessing an Azure resource, see the Authorize access with Azure Active Directory section to make sure the security principal has been granted the sufficient permission to access the Azure resource.

The following section describes the scenarios using different Spring ecosystem libraries with OAuth authentication.

Spring Kafka application support

This section describes the usage scenario for Spring Boot application using Spring Kafka or Spring Integration Kafka library.

Dependency setup
<dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
<!-- Using Spring Kafka library only-->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:2.8.6-->
</dependency>
<!-- Using Spring Integration library only -->
<dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:5.5.12-->
</dependency>
Configuration update

To use the OAuth authentication, just specify the Event Hubs endpoint, as shown in the following example:

spring.kafka.bootstrap-servers=<NAMESPACENAME>.servicebus.windows.net:9093
Spring Cloud Stream binder Kafka application support

This section describes the usage scenario for Spring Boot applications using the Spring Cloud Stream binder Kafka library.

Dependency setup
<dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kafka</artifactId>
    <version>{version}</version><!--Need to be set, for example:3.2.3-->
</dependency>
Configuration

To use the OAuth authentication, just specify the Event Hubs endpoint and com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration, as shown in the following example:

spring.cloud.stream.kafka.binder.brokers=<NAMESPACENAME>.servicebus.windows.net:9093
spring.cloud.stream.binders.kafka.environment.spring.main.sources=com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration
Set com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration to enable the whole OAuth authentication workflow.
Samples

See the azure-spring-boot-samples repository on GitHub.

Use connection string authentication

You can use connection string authentication directly or use the Azure Resource Manager to retrieve the connection string.

Since version of 4.3.0, connection string authentication is deprecated in favor of OAuth authentications.
Dependency setup

Add the following dependencies if you want to migrate your Apache Kafka application to use Azure Event Hubs for Kafka.

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-starter</artifactId>
</dependency>

If you want to retrieve the connection string using Azure Resource Manager, add the following dependency:

<dependency>
  <groupId>com.azure.spring</groupId>
  <artifactId>spring-cloud-azure-resourcemanager</artifactId>
</dependency>
Configuration
Use Event Hubs connection string directly

The simplest way to connect to Event Hubs for Kafka is with the connection string. Just add the following property.

spring.cloud.azure.eventhubs.connection-string=${AZURE_EVENTHUBS_CONNECTION_STRING}
Use Azure Resource Manager to retrieve connection string

If you don’t want to configure the connection string in your application, you can use Azure Resource Manager to retrieve the connection string. To authenticate with Azure Resource Manager, you can also use credentials stored in Azure CLI or another local development tool such as Visual Studio Code or Intellij IDEA. Alternately, you can use Managed Identity if your application is deployed to Azure Cloud. Just be sure the principal has sufficient permission to read resource metadata.

If you choose to use a security principal to authenticate and authorize with Azure Active Directory for accessing an Azure resource, see the Authorize access with Azure Active Directory section to be sure the security principal has been granted the sufficient permission to access the Azure resource.

To use Azure Resource Manager to retrieve the connection string, just add the following property.

spring:
  cloud:
    azure:
      profile:
        subscription-id: ${AZURE_SUBSCRIPTION_ID}
      eventhubs:
        namespace: ${AZURE_EVENTHUBS_NAMESPACE}
        resource:
          resource-group: ${AZURE_EVENTHUBS_RESOURCE_GROUP}

Samples

See the azure-spring-boot-samples repository on GitHub.

Remove credentials from Spring Kafka applications

Overview

You can use the Event Hubs Kafka endpoint in your Spring Kafka application. From Spring Cloud Azure 4.3.0, you can configure and run your application without credentials. This article is a migration guide for removing credentials from Spring Kafka applications.

Update dependencies

First, add the spring-cloud-azure-dependencies BOM, as shown in the following example:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.azure.spring</groupId>
      <artifactId>spring-cloud-azure-dependencies</artifactId>
      <version>${version.spring.cloud.azure}</version><!--	The version for spring-cloud-azure-dependencies is 4.3.0+.-->
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Then, add the Spring Cloud Azure starter, as shown in the following example:

<dependencies>
  <dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter</artifactId>
  </dependency>
</dependencies>

Update configuration

If you’re using Spring Kafka, remove the following options if you have customized values:

  • spring.kafka.security.protocol

  • spring.kafka.security.properties.sasl.mechanism

  • spring.kafka.security.properties.sasl.jaas.config

The final configuration should look like the following example:

spring.kafka.bootstrap-servers=<NAMESPACENAME>.servicebus.windows.net:9093

If you’re using Spring Cloud Stream Binder Kafka, remove the following options if you have customized values:

  • spring.kafka.security.protocol

  • spring.kafka.security.properties.sasl.mechanism

  • spring.kafka.security.properties.sasl.jaas.config

  • spring.cloud.stream.kafka.configuration.security.protocol

  • spring.cloud.stream.kafka.configuration.sasl.mechanism

  • spring.cloud.stream.kafka.configuration.sasl.jaas.config

Then, add the following option:

  • spring.cloud.stream.binders.kafka.environment.spring.main.sources

The final configuration should look like the following example:

spring.cloud.stream.kafka.binder.brokers=<NAMESPACENAME>.servicebus.windows.net:9093
spring.cloud.stream.binders.kafka.environment.spring.main.sources=com.azure.spring.cloud.autoconfigure.kafka.AzureKafkaSpringCloudStreamConfiguration
The spring.cloud.stream.binders.kafka.environment.spring.main.sources option is used to specify the additional configuration of KafkaBinderConfigurationPropertiesBeanPostProcessor specifying the OAuth security parameters for the particular binder.

Run locally

Grant permissions

With Azure AD, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user or an application service principal.

Because Azure Event Hubs supports Azure role-based access control, you need to assign the corresponding data plane roles to the security principal you use when you want to read or write data to it. In this article, you’ll use an Azure CLI credential to connect to Azure Event Hubs, so you need to assign roles to an Azure CLI account. For more information about assigning access roles, see Authorize access to Event Hubs resources using Azure Active Directory.

For data access, set the data plane access role: Azure Event Hubs Data Sender and Azure Event Hubs Data Receiver.
Sign in to your Azure account

To use the Azure CLI credential, first use the Azure CLI command az login to sign in. Then, build and test your application.

If you want to use other local environment credentials, for example with IntelliJ, see Authentication for details.

Deploy to Azure Spring Apps

This section describes how to run the application locally. In production, you can deploy the application to Azure hosting environments like Azure Spring Apps.

Create and configure managed identity

To connect with managed identities, enable the managed identity on Azure Spring Apps and grant the access permissions. For more information, see Create and configure a managed identity on Azure hosting services.

For information on how to assign roles to the managed identity, see Assign Azure roles using the Azure portal.

For data access, set the data plane access role: Azure Event Hubs Data Sender and Azure Event Hubs Data Receiver.
Deploy to Azure Spring Apps

For more information, see Deploy application to Azure hosting services.