Batch Inference Toolkit

Overview

  • Batch Inference Toolkit

Batching Modes

  • Local Mode (ModelHost)
  • Remote Mode (RemoteModelHost)

Batching Strategies

  • What is Batcher?
  • Concatenate Batcher
  • Sequence Batcher
  • Bucket Sequence Batcher

Examples

  • Bert Embedding
  • GPT Completion
Batch Inference Toolkit
  • Search

© Copyright 2023, AI Platform Team, STCA, Microsoft.

Built with Sphinx using a theme provided by Read the Docs.