Batch Inference Toolkit

Overview

Batch Inference Toolkit

Batching Modes

Local Mode (ModelHost)
Remote Mode (RemoteModelHost)

Batching Strategies

What is Batcher?
Concatenate Batcher
Sequence Batcher
Bucket Sequence Batcher

Examples

Bert Embedding
GPT Completion

Batch Inference Toolkit

Index

Index

© Copyright 2023, AI Platform Team, STCA, Microsoft.

Built with Sphinx using a theme provided by Read the Docs.