Configuration

You will need to configure the LLM connection and authorization secrets. You can use remote (like OpenAI, Azure, etc.) and local models (like Ollama, Jan, LMStudio, etc.) with GenAIScript.

Model selection

The model used by the script is configured through the model field in the script function. The model name is formatted as provider:model-name, where provider is the LLM provider and the model-name is provider specific.

script({
    model: "openai:gpt-4o",
})

Large, small, vision models

You can also use the small, large, vision model aliases to use the default configured small, large and vision-enabled models. Large models are typically in the OpenAI gpt-4 reasoning range and can be used for more complex tasks. Small models are in the OpenAI gpt-4o-mini range, and are useful for quick and simple tasks.

script({ model: "small" })

script({ model: "large" })

The model aliases can also be overridden from the cli run command, or environment variables or configuration file. Learn more about model aliases.

genaiscript run ... --model largemodelid --small-model smallmodelid

or by adding the GENAISCRIPT_MODEL_LARGE and GENAISCRIPT_MODEL_SMALL environment variables.

.env

GENAISCRIPT_MODEL_LARGE="azure_serverless:..."
GENAISCRIPT_MODEL_SMALL="azure_serverless:..."
GENAISCRIPT_MODEL_VISION="azure_serverless:..."

You can also configure the default aliases for a given LLM provider by using the provider argument. The default are documented in this page and printed to the console output.

script({ provider: "openai" })

genaiscript run ... --provider openai

Model aliases

In fact, you can define any alias for your model (only alphanumeric characters are allowed) through environment variables of the name GENAISCRIPT_MODEL_ALIAS where ALIAS is the alias you want to use.

.env

GENAISCRIPT_MODEL_TINY=...

Model aliases are always lowercased when used in the script.

script({ model: "tiny" })

.env file

GenAIScript uses a .env file to load secrets and configuration information into the process environment variables.

1. Create or update a .gitignore file in the root of your project and make it sure it includes .env. This ensures that you do not accidentally commit your secrets to your source control.

   .gitignore

   ...
   .env

2. Create a .env file in the root of your project.

   • .gitignore
   • .env

3. Update the .env file with the configuration information (see below).

Do Not Commit Secrets

The .env file should never be commited to your source control! If the .gitignore file is properly configured, the .env file will appear grayed out in Visual Studio Code.

.gitignore

...
.env

Custom .env file location

You can specify a custom .env file location through the CLI or an environment variable.

• by adding the --env <file> argument to the CLI.

npx genaiscript ... --env .env.local

• by setting the GENAISCRIPT_ENV_FILE environment variable.

GENAISCRIPT_ENV_FILE=".env.local" npx genaiscript ...

• by specifying the .env file location in a configuration file.

~/genaiscript.config.yaml

envFile: ~/.env.genaiscript

No .env file

If you do not want to use a .env file, make sure to populate the environment variables of the genaiscript process with the configuration values.

Here are some common examples:

• Using bash syntax

OPENAI_API_KEY="value" npx --yes genaiscript run ...

• GitHub Action configuration

.github/workflows/genaiscript.yml

run: npx --yes genaiscript run ...
env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

configure command

The configure command is an interactive command to configure and validate the LLM connections.

npx genaiscript configure

OpenAI

openai is the OpenAI chat model provider. It uses the OPENAI_API_... environment variables.

1. Upgrade your account to get access to the models. You will get 404s if you do not have a paying account.

2. Create a new secret key from the OpenAI API Keys portal.

3. Update the .env file with the secret key.

   .env

   OPENAI_API_KEY=sk_...

4. Find the model you want to use from the OpenAI API Reference or the OpenAI Chat Playground.

   

5. Set the model field in script to the model you want to use.

   script({
       model: "openai:gpt-4o",
       ...
   })

Default Model Configuration

Use GENAISCRIPT_MODEL_LARGE and GENAISCRIPT_MODEL_SMALL in your .env file to set the default model and small model.

GENAISCRIPT_MODEL_LARGE=openai:gpt-4o
GENAISCRIPT_MODEL_SMALL=openai:gpt-4o-mini

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	gpt-4o
small	gpt-4o-mini
vision	gpt-4o
vision_small	gpt-4o-mini
embeddings	text-embedding-3-small
reasoning	o1
reasoning_small	o1-mini
transcription	whisper-1
speech	tts-1

GitHub Models

The GitHub Models provider, github, allows running models through the GitHub Marketplace. This provider is useful for prototyping and subject to rate limits depending on your subscription.

script({ model: "github:gpt-4o" })

Tip

If you are running from a GitHub Codespace, the token is already configured for you.

Play

1. Create a GitHub personal access token. The token should not have any scopes or permissions.

2. Update the .env file with the token.

   .env

   GITHUB_TOKEN=...

To configure a specific model,

1. Open the GitHub Marketplace and find the model you want to use.

2. Copy the model name from the Javascript/Python samples

   const modelName = "Phi-3-mini-4k-instruct"

   to configure your script.

   script({
       model: "github:Phi-3-mini-4k-instruct",
   })

If you are already using GITHUB_TOKEN variable in your script and need a different one for GitHub Models, you can use the GITHUB_MODELS_TOKEN variable instead.

o1-preview and o1-mini models

Currently these models do not support streaming and system prompts. GenAIScript handles this internally.

script({
    model: "github:o1-mini",
})

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	gpt-4o
small	gpt-4o-mini
vision	gpt-4o
embeddings	text-embedding-3-small
reasoning	o1-preview
reasoning_small	o1-mini

Limitations

• Smaller context windows, and rate limiting
• listModels
• logprobs (and top logprobs) ignored
• Ignore prediction of output tokens
• topLogprobs

Azure OpenAI

The Azure OpenAI provider, azure uses the AZURE_OPENAI_... environment variables. You can use a managed identity (recommended) or an API key to authenticate with the Azure OpenAI service. You can also use a service principal as documented in automation.

script({ model: "azure:deployment-id" })

Tip

If you are a Visual Studio Subscriber, you can get free Azure credits to try the Azure OpenAI service.

Managed Identity (Entra ID)

1. Open your Azure OpenAI resource in the Azure Portal

2. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Cognitive Services OpenAI User/Contributor role. If you get a 401 error, click on Add, Add role assignment and add the Cognitive Services OpenAI User role to your user.

3. Navigate to Resource Management, then Keys and Endpoint.

4. Update the .env file with the endpoint.

   .env

   AZURE_OPENAI_API_ENDPOINT=https://....openai.azure.com

   Note

   Make sure to remove any AZURE_API_KEY, AZURE_OPENAI_API_KEY entries from .env file.

5. Navigate to deployments and make sure that you have your LLM deployed and copy the deployment-id, you will need it in the script.

6. Open a terminal and login with Azure CLI.

   az login

7. Update the model field in the script function to match the model deployment name in your Azure resource.

   script({
       model: "azure:deployment-id",
       ...
   })

Listing models

In order to allow GenAIScript to list deployments in your Azure OpenAI service, you need to provide the Subscription ID and you need to use Microsoft Entra!.

1. Open the Azure OpenAI resource in the Azure Portal, open the Overview tab and copy the Subscription ID.

2. Update the .env file with the subscription id.

   .env

   AZURE_OPENAI_SUBSCRIPTION_ID="..."

3. Test your configuration by running

   npx genaiscript models azure

   Note

   This feature will probably not work with AZURE_OPENAI_API_KEY as the token does not have the proper scope to query the list of deployments.

Custom credentials

In some situations, the default credentials chain lookup may not work. In that case, you can specify an additional environment variable AZURE_OPENAI_API_CREDENTIALS with the type of credential that should be used.

.env

AZURE_OPENAI_API_CREDENTIALS=cli

The types are mapped directly to their @azure/identity credential types:

• cli - AzureCliCredential
• env - EnvironmentCredential
• powershell - AzurePowerShellCredential
• devcli - AzureDeveloperCliCredential
• managedidentity - ManagedIdentityCredential

Custom token scopes

The default token scope for Azure OpenAI access is https://cognitiveservices.azure.com/.default. You can override this value using the AZURE_OPENAI_TOKEN_SCOPES environment variable.

.env

AZURE_OPENAI_TOKEN_SCOPES=...

API Version

GenAIScript maintains a default API version to access Azure OpenAI. You can override this value using the AZURE_OPENAI_API_VERSION environment variable.

.env

AZURE_OPENAI_API_VERSION=2025-01-01-preview

You can also override the API version on a per-deployment basis by settings the AZURE_OPENAI_API_VERSION_<deployment-id> environment variable (where deployment-id is capitalized).

.env

AZURE_OPENAI_API_VERSION_GPT-4O=2025-01-01-preview

API Key

1. Open your Azure OpenAI resource and navigate to Resource Management, then Keys and Endpoint.

2. Update the .env file with the secret key (Key 1 or Key 2) and the endpoint.

   .env

   AZURE_OPENAI_API_KEY=...
   AZURE_OPENAI_API_ENDPOINT=https://....openai.azure.com

3. The rest of the steps are the same: Find the deployment name and use it in your script, model: "azure:deployment-id".

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier

Limitations

• listModels
• Ignore prediction of output tokens

Azure AI Serverless Deployments

You can deploy “serverless” models through Azure AI Foundry and pay as you go per token. You can browse the Azure AI model catalog and use the serverless API filter to see the available models.

There are two types of serverless deployments that require different configurations: OpenAI models and all other models. The OpenAI models, like gpt-4o, are deployed to .openai.azure.com endpoints, while the Azure AI models, like Meta-Llama-3.1-405B-Instruct are deployed to .models.ai.azure.com endpoints.

They are configured slightly differently.

Azure AI OpenAI Serverless

The azure_serverless provider supports OpenAI models deployed through the Azure AI Foundry serverless deployments. It supports both Entra ID and key-based authentication.

script({ model: "azure_serverless:deployment-id" })

Note

This kind of deployment is different from the Azure OpenAI deployments (azure provider).

Managed Identity (Entra ID)

1. Open https://ai.azure.com/ and open the Deployments page.

2. Deploy a base model from the catalog. You can use the Deployment Options -> Serverless API option to deploy a model as a serverless API.

3. Deploy an OpenAI base model. This will also create a new Azure OpenAI resource in your subscription (which may be invisible to you, more later).

4. Update the .env file with the deployment endpoint in the AZURE_SERVERLESS_OPENAI_API_ENDPOINT variable.

   .env

   AZURE_SERVERLESS_OPENAI_API_ENDPOINT=https://....openai.azure.com

5. Go back to the Overview tab in your Azure AI Foundry project and click on Open in Management center.

6. Click on the Azure OpenAI Service resource, then click on the Resource external link which will take you back to the (underlying) Azure OpenAI service in Azure Portal.

7. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Cognitive Services OpenAI User/Contributor role. If you get a 401 error, click on Add, Add role assignment and add the Cognitive Services OpenAI User role to your user.

At this point, you are ready to login with the Azure CLI and use the managed identity.

Note

The resources created by Azure AI Foundry are not visible by default in the Azure Portal. To make them visible, open All resources, click Manage view and select Show hidden types.

1. Install the Azure CLI.

2. Open a terminal and login

   az login

API Key

1. Open your Azure OpenAI resource and navigate to Resource Management, then Keys and Endpoint.

2. Update the .env file with the endpoint and the secret key (Key 1 or Key 2) and the endpoint.

   .env

   AZURE_SERVERLESS_OPENAI_API_ENDPOINT=https://....openai.azure.com
   AZURE_SERVERLESS_OPENAI_API_KEY=...

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	gpt-4o
small	gpt-4o-mini
vision	gpt-4o
vision_small	gpt-4o-mini
reasoning	o1
reasoning_small	o1-mini

Limitations

• listModels
• Ignore prediction of output tokens

Azure AI Models

The azure_serverless_models provider supports non-OpenAI models deployed through the Azure AI Foundary serverless deployments.

script({ model: "azure_serverless_models:deployment-id" })

Managed Identity (Entra ID)

1. Open your Azure AI Project resource in the Azure Portal

2. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Azure AI Developer role. If you get a 401 error, click on Add, Add role assignment and add the Azure AI Developer role to your user.

3. Configure the Endpoint Target URL as the AZURE_SERVERLESS_MODELS_API_ENDPOINT.

   .env

   AZURE_SERVERLESS_MODELS_API_ENDPOINT=https://...models.ai.azure.com

4. Navigate to deployments and make sure that you have your LLM deployed and copy the Deployment Info name, you will need it in the script.

5. Update the model field in the script function to match the model deployment name in your Azure resource.

   script({
       model: "azure_serverless:deployment-info-name",
       ...
   })

API Key

1. Open https://ai.azure.com/ and open the Deployments page.

2. Deploy a base model from the catalog. You can use the Deployment Options -> Serverless API option to deploy a model as a serverless API.

3. Configure the Endpoint Target URL as the AZURE_SERVERLESS_MODELS_API_ENDPOINT variable and the key in AZURE_SERVERLESS_MODELS_API_KEY in the .env file**.**

   .env

   AZURE_SERVERLESS_MODELS_API_ENDPOINT=https://...models.ai.azure.com
   AZURE_SERVERLESS_MODELS_API_KEY=...

4. Find the deployment name and use it in your script, model: "azure_serverless_models:deployment-id".

Support for multiple inference deployements

You can update the AZURE_SERVERLESS_MODELS_API_KEY with a list of deploymentid=key pairs to support multiple deployments (each deployment has a different key).

.env

AZURE_SERVERLESS_MODELS_API_KEY="
model1=key1
model2=key2
model3=key3
"

Limitations

• listModels
• Ignore prediction of output tokens

Google AI

The google provider allows you to use Google AI models. It gives you access

Note

GenAIScript uses the OpenAI compatibility layer of Google AI, so some limitations apply.

• seed is not supported and ignored.
• fallback tools are enabled using Google finishes the OpenAI compatibilty layer. (See forum).

1. Open Google AI Studio and create a new API key.

2. Update the .env file with the API key.

   .env

   GEMINI_API_KEY=...

3. Find the model identifier in the Gemini documentation and use it in your script or cli with the google provider.

   ...
   const model = genAI.getGenerativeModel({
     model: "gemini-1.5-pro-latest",
   });
   ...

   then use the model identifier in your script.

   script({ model: "google:gemini-1.5-pro-latest" })

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	gemini-1.5-flash-latest
small	gemini-1.5-flash-latest
vision	gemini-1.5-flash-latest
long	gemini-1.5-flash-latest
reasoning	gemini-2.0-flash-thinking-exp-1219
reasoning_small	gemini-2.0-flash-thinking-exp-1219
embeddings	text-embedding-004

Limitations

• Uses OpenAI compatibility layer
• listModels
• logprobs (and top logprobs) ignored
• Ignore prediction of output tokens
• Seed ignored
• Tools implemented as fallback tools automatically.
• topLogprobs

GitHub Copilot Chat Models

If you have access to GitHub Copilot Chat in Visual Studio Code, GenAIScript will be able to leverage those language models as well.

This mode is useful to run your scripts without having a separate LLM provider or local LLMs. However, those models are not available from the command line and have additional limitations and rate limiting defined by the GitHub Copilot platform.

There is no configuration needed as long as you have GitHub Copilot installed and configured in Visual Studio Code. You can force using this model by using github_copilot_chat:* as a model name.

Play

1. Install GitHub Copilot Chat (emphasis on Chat)

2. run your script

3. Confirm that you are allowing GenAIScript to use the GitHub Copilot Chat models.

4. select the best chat model that matches the one you have in your script

   

   (This step is skipped if you already have mappings in your settings)

The mapping of GenAIScript model names to Visual Studio Models is stored in the settings.

Anthropic

The anthropic provider access Anthropic models. Anthropic is an AI research company that offers powerful language models, including the Claude series.

script({ model: "anthropic:claude-2.1" })

To use Anthropic models with GenAIScript, follow these steps:

1. Sign up for an Anthropic account and obtain an API key from their console.

2. Add your Anthropic API key to the .env file:

   .env

   ANTHROPIC_API_KEY=sk-ant-api...

3. Find the model that best suits your needs by visiting the Anthropic model documentation.

4. Update your script to use the model you choose.

   script({
       ...
       model: "anthropic:claude-3-5-sonnet-20240620",
   })

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	claude-3-5-sonnet-latest
small	claude-3-5-haiku-latest
vision	claude-3-5-sonnet-latest

Limitations

• listModels
• logprobs (and top logprobs) ignored
• Ignore prediction of output tokens
• topLogprobs

Anthropic Bedrock

The anthropic_bedrock provider accesses Anthropic models on Amazon Bedrock. You can find the model names in the Anthropic model documentation.

GenAIScript assumes that you have configured AWS credentials in a way that the AWS Node SDK will recognise.

script({ model: "anthropic_bedrock:anthropic.claude-3-sonnet-20240229-v1:0" })

Hugging Face

The huggingface provider allows you to use Hugging Face Models using Text Generation Inference.

script({ model: "huggingface:microsoft/Phi-3-mini-4k-instruct" })

To use Hugging Face models with GenAIScript, follow these steps:

1. Sign up for a Hugging Face account and obtain an API key from their console. If you are creating a Fined Grained token, enable the Make calls to the serverless inference API option.

2. Add your Hugging Face API key to the .env file as HUGGINGFACE_API_KEY, HF_TOKEN or HUGGINGFACE_TOKEN variables.

   .env

   HUGGINGFACE_API_KEY=hf_...

3. Find the model that best suits your needs by visiting the HuggingFace models.

4. Update your script to use the model you choose.

   script({
       ...
       model: "huggingface:microsoft/Phi-3-mini-4k-instruct",
   })

Note

Some models may require a Pro account.

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	meta-llama/Llama-3.3-70B-Instruct
small	microsoft/phi-4
vision	meta-llama/Llama-3.2-11B-Vision-Instruct
embeddings	nomic-ai/nomic-embed-text-v1.5

Limitations

• Uses OpenAI compatibility layer
• listModels
• Ignore prediction of output tokens

Mistral AI

The mistral provider allows you to use Mistral AI Models using the Mistral API.

script({ model: "mistral:mistral-large-latest" })

1. Sign up for a Mistral AI account and obtain an API key from their console.

2. Add your Mistral AI API key to the .env file:

   .env

   MISTRAL_API_KEY=...

3. Update your script to use the model you choose.

   script({
       ...
       model: "mistral:mistral-large-latest",
   })

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	mistral-large-latest
small	mistral-small-latest
vision	pixtral-large-latest

Limitations

• Ignore prediction of output tokens

Alibaba Cloud

The alibaba provider access the Alibaba Cloud models.

script({
    model: "alibaba:qwen-max",
})

1. Sign up for a Alibaba Cloud account and obtain an API key from their console.

2. Add your Alibaba API key to the .env file:

   .env

   ALIBABA_API_KEY=sk_...

3. Find the model that best suits your needs by visiting the Alibaba models.

4. Update your script to use the model you choose.

   script({
       ...
       model: "alibaba:qwen-max",
   })

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	qwen-max
small	qwen-turbo
long	qwen-plus
embeddings	text-embedding-v3

Limitations

• Uses OpenAI compatibility layer
• listModels
• Ignore prediction of output tokens
• Tools implemented as fallback tools automatically.

Ollama

Ollama is a desktop application that lets you download and run models locally.

Running tools locally may require additional GPU resources depending on the model you are using.

Use the ollama provider to access Ollama models.

Note

GenAIScript is currently using the OpenAI API compatibility layer of Ollama.

1. Start the Ollama application or

   ollama serve

2. Update your script to use the ollama:phi3.5 model (or any other model or from Hugging Face).

   script({
       ...,
       model: "ollama:phi3.5",
   })

   GenAIScript will automatically pull the model, which may take some time depending on the model size. The model is cached locally by Ollama.

3. If Ollama runs on a server or a different computer or on a different port, you have to configure the OLLAMA_HOST environment variable to connect to a remote Ollama server.

   .env

   OLLAMA_HOST=https://<IP or domain>:<port>/ # server url
   OLLAMA_HOST=0.0.0.0:12345 # different port

You can specify the model size by adding the size to the model name, like ollama:llama3.2:3b.

script({
    ...,
    model: "ollama:llama3.2:3b",
})

Ollama with Hugging Face models

You can also use GGUF models from Hugging Face.

script({
    ...,
    model: "ollama:hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF",
})

Ollama with Docker

You can conviniately run Ollama in a Docker container.

• start the Ollama container

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

• stop and remove the Ollama containers

docker stop ollama && docker rm ollama

Tip

You can start the Ollama container in a GitHub Codespace or a GitHub Action!

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
embeddings	nomic-embed-text

Limitations

• Uses OpenAI compatibility layer
• logit_bias ignored
• Ignore prediction of output tokens

DeepSeek

deepseek is the DeepSeek chat model provider. It uses the DEEPSEEK_API_... environment variables.

1. Create a new secret key from the DeepSeek API Keys portal.

2. Update the .env file with the secret key.

   .env

   DEEPSEEK_API_KEY=sk_...

3. Set the model field in script to deepseek:deepseek:deepseek-chat which is currently the only supported model.

   script({
       model: "deepseek:deepseek-chat",
       ...
   })

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
large	deepseek-chat
small	deepseek-chat
vision	deepseek-chat

LM Studio

The lmstudio provider connects to the LMStudio headless server. and allows to run local LLMs.

1. Install LMStudio (v0.3.5+)

2. Open LMStudio

3. Open the Model Catalog, select your model and load it at least once so it is downloaded locally.

4. Open the settings (Gearwheel icon) and enable Enable Local LLM Service.

5. GenAIScript assumes the local server is at http://localhost:1234/v1 by default. Add a LMSTUDIO_API_BASE environment variable to change the server URL.

   .env

   LMSTUDIO_API_BASE=http://localhost:2345/v1

Find the model API identifier in the dialog of loaded models then use that identifier in your script:

script({
    model: "lmstudio:llama-3.2-1b-instruct",
})

• GenAIScript uses the LMStudio CLI to pull models on demand.
• Specifiying the quantization is currently not supported.

Aliases

The following model aliases are attempted by default in GenAIScript.

Alias	Model identifier
embeddings	text-embedding-nomic-embed-text-v1.5

Limitations

• Ignore prediction of output tokens

LM Studio and Hugging Face Models

Follow this guide to load Hugging Face models into LMStudio.

Jan

The jan provider connects to the Jan local server.

1. Jan

2. Open Jan and download the models you plan to use. You will find the model identifier in the model description page.

3. Click on the Local API Server icon (lower left), then Start Server.

   Keep the desktop application running!

To use Jan models, use the jan:modelid syntax. If you change the default server URL, you can set the JAN_API_BASE environment variable.

.env

JAN_API_BASE=http://localhost:1234/v1

Limitations

• Ignore prediction of output tokens
• top_p ignored

LocalAI

LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It uses free Open Source models and it runs on CPUs.

LocalAI acts as an OpenAI replacement, you can see the model name mapping used in the container, like gpt-4 is mapped to phi-2.

1. Install Docker. See the LocalAI documentation for more information.

2. Update the .env file and set the api type to localai.

   .env

   OPENAI_API_TYPE=localai

To start LocalAI in docker, run the following command:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-aio-cpu
docker start local-ai
docker stats
echo "LocalAI is running at http://127.0.0.1:8080"

Llamafile

https://llamafile.ai/ is a single file desktop application that allows you to run an LLM locally.

The provider is llamafile and the model name is ignored.

LLaMA.cpp

LLaMA.cpp also allow running models locally or interfacing with other LLM vendors.

1. Update the .env file with the local server information.

   .env

   OPENAI_API_BASE=http://localhost:...

OpenRouter

You can configure the OpenAI provider to use the OpenRouter service instead by setting the OPENAI_API_BASE to https://openrouter.ai/api/v1. You will also need an api key.

.env

OPENAI_API_BASE=https://openrouter.ai/api/v1
OPENAI_API_KEY=...

Then use the OpenRouter model name in your script:

script({ model: "openai:openai/gpt-4o-mini" })

By default, GenAIScript will set the site URL and name to GenAIScript but you can override these settings with your own values:

.env

OPENROUTER_SITE_URL=... # populates HTTP-Referer header
OPENROUTER_SITE_NAME=... # populate X-Title header

LiteLLM

The LiteLLM proxy gateway provides a OpenAI compatible API for running models locally. Configure the LITELLM_... keys to set the key and optionally the base url.

.env

LITELLM_API_KEY="..."
#LITELLM_API_BASE="..."

Hugging Face Transformer.js

This transformers provider runs models on device using Hugging Face Transformers.js.

The model syntax is transformers:<repo>:<dtype> where

• repo is the model repository on Hugging Face,
• dtype is the quantization type.

script({
    model: "transformers:onnx-community/Qwen2.5-Coder-0.5B-Instruct:q4",
})

The default transformers device is cpu, but you can changing it using HUGGINGFACE_TRANSFORMERS_DEVICE environment variable.

.env

HUGGINGFACE_TRANSFORMERS_DEVICE=gpu

Note

This provider is experimental and may not work with all models.

Limitations

• Ignore prediction of output tokens

Whisper ASR WebServices

This whisperasr provider allows to configure a transcription task to use the Whisper ASR WebService project.

const transcript = await transcribe("video.mp4", {
    model: "whisperasr:default",
})

This whisper service can run locally or in a docker container (see documentation).

CPU

docker run -d -p 9000:9000 -e ASR_MODEL=base -e ASR_ENGINE=openai_whisper onerahmet/openai-whisper-asr-webservice:latest

You can also override the transcription model alias to change the default model used by transcribe.

Model specific environment variables

You can provide different environment variables for each named model by using the PROVIDER_MODEL_API_... prefix or PROVIDER_API_... prefix. The model name is capitalized and all non-alphanumeric characters are converted to _.

This allows to have various sources of LLM computations for different models. For example, to enable the ollama:phi3 model running locally, while keeping the default openai model connection information.

.env

OLLAMA_PHI3_API_BASE=http://localhost:11434/v1

Running behind a proxy

You can set the HTTP_PROXY and/or HTTPS_PROXY environment variables to run GenAIScript behind a proxy.

.env

HTTP_PROXY=http://proxy.example.com:8080

Checking your configuration

You can check your configuration by running the genaiscript info env command. It will display the current configuration information parsed by GenAIScript.

genaiscript info env

Next steps

Write your first script.