Real Time Intelligence

Data Sources

Data coming into Fabric Real-Time Intelligence can come from anywhere. Anything that emits telemetry can be a data source for Fabric Real-Time Intelligence. But there are proven patterns that you can follow that should drive how you get data from you’re data source to Fabric Real-Time Intelligence.

Option 1: Output to Eventstream

The best pattern for data coming into Fabric RTI is to ingest via Evenstream. If any of these options are available you can output data from your Data Source to Evenstreams

Can it write to an Azure Eventhub endpoint If the data source has built-in capabilities to write to Azure Eventhub then it can write to Fabric Eventstream via the Custom Endpoint
Can it write to a Kafka Endpoint If the data source has built-in capabilities to write to a Kafka endpoint then it can write to Fabric Evenstream via the Custom Endpoint
Can it write to a AMQP Endpoint If the data source has built-in capabilities to write to a Kafka endpoint then it can write to Fabric Evenstream via the Custom Endpoint
Utilize CDC on Database Platform Eventstream supports CDS feeds from – Azure SQL Database – PostgresSQL Database – MySQL Database – Azure Cosmos DB – SQL Server on VM – Azure SQL Managed Instance
Also Supported – Google Cloud Pub/Sub – Amazon Kinesis Data Streams – Confluent Cloud Kafka – Apache Kafka – Amazon Managed Streaming for Apache Kafka – Fabric Workspace Item Events – Fabric OneLake Events – Fabric Job Events – Azure Blob Storage Events
Is it already landing in Azure Event Hub, Azure Service Bus, or Azure IoT Hub? If data is already landing in any of these product in Azure then Eventstream can bring that data in by connect directly to the exising data.

Option 2: Output to Storage (Fabric OneLake or Azure Blob Storage)

We are introducing continous ingestion from both Azure Storage and also from Fabric OneLake in the coming months. Currently the approach to do continous ingestion is to utilize Data Pipeline in fabric and the trigger for storag.

Fabric Data Pipeline Storage Trigger Data Pipeline Eventhouse Connector

Option 3: Connectors/Agents

We’ve worked with many of the most popular agents to support an output to kusto. Any of these agents that can output to ADX in Azure can also output to Eventhouse in Fabric. The most popular agents are Logstash, Fluent Bit, Cribl, Telegraf, and Apache Flink. But for a full list and documentation go here.

Option 4: KQL API/SDKs

Eventhouse has a full set of APIs and SDKs that allow you to write data directly to a KQL DB on an Eventhouse. This allows you to write directly to the platform from code and adds flexibility.

We have SDKs for .NET, Python, Node, Go, and Java. You can also utiize the Rest API

Option 5: Amazon S3 Bucket

Eventhouse has the capability of ingesting data from S3 Buckets. Combining AWS Lambda functions with our native capabilities allows you to continously ingest from S3. More details and implementation guide can be found here