Configuration

You will need to configure the LLM connection and authorizion secrets.

Tip

If you do not have access to an LLM, you can try GitHub Models, GitHub Copilot Models or use a local model for inferencing.

Model selection

The model used by the script is configured throught the model field in the script function. The model name is formatted as provider:model-name, where provider is the LLM provider and the model-name is provider specific.

script({
    model: "openai:gpt-4o",
})

Large and small models

You can also use the small and large aliases to use the default configured small and large models. Large models are typically in the OpenAI gpt-4 reasoning range and can be used for more complex tasks. Small models are in the OpenAI gpt-4o-mini range, and are useful for quick and simple tasks.

script({ model: "small" })

script({ model: "large" })

The model can also be overriden from the cli run command

genaiscript run ... --model largemodelid --small-model smallmodelid

or by adding the GENAISCRIPT_DEFAULT_MODEL and GENAISCRIPT_DEFAULT_SMALL_MODEL environment variables.

.env

GENAISCRIPT_DEFAULT_MODEL="azure_serverless:..."
GENAISCRIPT_DEFAULT_SMALL_MODEL="azure_serverless:..."

.env file

GenAIScript uses a .env file to store the secrets.

1. Create or update a .gitignore file in the root of your project and make it sure it includes .env. This ensures that you do not accidentally commit your secrets to your source control.

   .gitignore

   ...
   .env

2. Create a .env file in the root of your project.

   • .gitignore
   • .env

3. Update the .env file with the configuration information (see below).

Do Not Commit Secrets

The .env file should never be commited to your source control! If the .gitignore file is properly configured, the .env file will appear grayed out in Visual Studio Code.

OpenAI

This provider, openai, is the OpenAI chat model provider. It uses the OPENAI_API_... environment variables.

1. Create a new secret key from the OpenAI API Keys portal.

2. Update the .env file with the secret key.

   .env

   OPENAI_API_KEY=sk_...

3. Find the model you want to use from the OpenAI API Reference or the OpenAI Chat Playground.

   

4. Set the model field in script to the model you want to use.

   script({
       model: "openai:gpt-4o",
       ...
   })

Default Model Configuration

Use GENAISCRIPT_DEFAULT_MODEL and GENAISCRIPT_DEFAULT_SMALL_MODEL in your .env file to set the default model and small model.

GENAISCRIPT_DEFAULT_MODEL=openai:gpt-4o
GENAISCRIPT_DEFAULT_SMALL_MODEL=openai:gpt-4o-mini

GitHub Models

The GitHub Models provider, github, allows running models through the GitHub Marketplace. This provider is useful for prototyping and subject to rate limits depending on your subscription.

script({ model: "github:gpt-4" })

If you are running from a GitHub Codespace, the token is already configured for you.

1. Create a GitHub personal access token. The token should not have any scopes or permissions.

2. Update the .env file with the token.

   .env

   GITHUB_TOKEN=...

To configure a specific model,

1. Open the GitHub Marketplace and find the model you want to use.

2. Copy the model name from the Javascript/Python samples

   const modelName = "Phi-3-mini-4k-instruct"

   to configure your script.

   script({
       model: "github:Phi-3-mini-4k-instruct",
   })

If you are already using GITHUB_TOKEN variable in your script and need a different one for GitHub Models, you can use the GITHUB_MODELS_TOKEN variable instead.

o1-preview and o1-mini models

Currently these models do not support streaming and system prompts. GenAIScript handles this internally.

script({
    model: "github:o1-mini",
})

Azure OpenAI

The Azure OpenAI provider, azure uses the AZURE_OPENAI_... environment variables. You can use a managed identity (recommended) or a API key to authenticate with the Azure OpenAI service. You can also use a service principal as documented in automation.

script({ model: "azure:deployment-id" })

Tip

If your are a Visual Studio Subscriber, you can get free Azure credits to try the Azure OpenAI service.

Managed Identity (Entra ID)

1. Open your Azure OpenAI resource in the Azure Portal

2. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Cognitive Services OpenAI User/Contributor role. If you get a 401 error, click on Add, Add role assignment and add the Cognitive Services OpenAI User role to your user.

3. Navigate to Resource Management, then Keys and Endpoint.

4. Update the .env file with the endpoint.

   .env

   AZURE_OPENAI_API_ENDPOINT=https://....openai.azure.com

   Note

   Make sure to remove any AZURE_API_KEY, AZURE_OPENAI_API_KEY entries from .env file.

5. Navigate to deployments and make sure that you have your LLM deployed and copy the deployment-id, you will need it in the script.

6. Open a terminal and login with Azure CLI.

   az login

7. Update the model field in the script function to match the model deployment name in your Azure resource.

   script({
       model: "azure:deployment-id",
       ...
   })

Custom credentials

In some situation, the default credentials chain lookup may not work. In that case, you can specify an additional environment variable AZURE_SERVERLESS_OPENAI_API_CREDENTIALS with the type of credential that should be used.

.env

AZURE_SERVERLESS_OPENAI_API_CREDENTIALS=cli

The types are mapped directly to their @azure/identity credential types:

• cli - AzureCliCredential
• env - EnvironmentCredential
• powershell - AzurePowerShellCredential
• devcli - AzureDeveloperCliCredential
• managedidentity - ManagedIdentityCredential

API Key

1. Open your Azure OpenAI resource and navigate to Resource Management, then Keys and Endpoint.

2. Update the .env file with the secret key (Key 1 or Key 2) and the endpoint.

   .env

   AZURE_OPENAI_API_KEY=...
   AZURE_OPENAI_API_ENDPOINT=https://....openai.azure.com

3. The rest of the steps are the same: Find the deployment name and use it in your script, model: "azure:deployment-id".

Azure AI Serverless Deployments

You can deploy “serverless” models through Azure AI Studio and pay as you go per token. You can browse the Azure AI model catalog and use the serverless API filter to see the available models.

There two types of serverless deployments that require different configurations: OpenAI models and all other models. The OpenAI models, like gpt-4o, are deployed to .openai.azure.com endpoints, while the Azure AI models, like Meta-Llama-3.1-405B-Instruct are deployed to .models.ai.azure.com endpoints.

They are configured slightly differently.

Azure AI OpenAI

The azure_serverless provider supports OpenAI models deployed through the Azure AI Studio serverless deployments. It supports both Entra ID and key-based authentication.

script({ model: "azure_serverless:deployment-id" })

Managed Identity (Entra ID)

1. Open a terminal and login with Azure CLI.

   az login

2. Open https://ai.azure.com/ and open the Deployments page.

3. Deploy a base model from the catalog. You can use the Deployment Options -> Serverless API option to deploy a model as a serverless API.

4. Deploy an OpenAI base model. This will also create a new Azure OpenAI resource in your subscription.

5. Update the .env file with the deployment endpoint in the AZURE_SERVERLESS_OPENAI_API_ENDPOINT variable.

   .env

   AZURE_SERVERLESS_OPENAI_API_ENDPOINT=https://....openai.azure.com

6. Open your Azure OpenAI resource that was created by Azure AI in the Azure Portal

7. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Cognitive Services OpenAI User/Contributor role. If you get a 401 error, click on Add, Add role assignment and add the Cognitive Services OpenAI User role to your user.

API Key

1. Open your Azure OpenAI resource and navigate to Resource Management, then Keys and Endpoint.

2. Update the .env file with the endpoint and the secret key (Key 1 or Key 2) and the endpoint.

   .env

   AZURE_SERVERLESS_OPENAI_API_ENDPOINT=https://....openai.azure.com
   AZURE_SERVERLESS_OPENAI_API_KEY=...

Azure AI Models

The azure_serverless_models provider supports non-OpenAI models deployed through the Azure AI Studio serverless deployments.

script({ model: "azure_serverless_models:deployment-id" })

Managed Identity (Entra ID)

1. Open your Azure AI Project resource in the Azure Portal

2. Navigate to Access Control (IAM), then View My Access. Make sure your user or service principal has the Azure AI Developer role. If you get a 401 error, click on Add, Add role assignment and add the Azure AI Developer role to your user.

3. Configure the Endpoint Target URL as the AZURE_SERVERLESS_MODELS_API_ENDPOINT.

   .env

   AZURE_SERVERLESS_MODELS_API_ENDPOINT=https://...models.ai.azure.com

4. Navigate to deployments and make sure that you have your LLM deployed and copy the Deployment Info name, you will need it in the script.

5. Update the model field in the script function to match the model deployment name in your Azure resource.

   script({
       model: "azure_serverless:deployment-info-name",
       ...
   })

API Key

1. Open https://ai.azure.com/ and open the Deployments page.

2. Deploy a base model from the catalog. You can use the Deployment Options -> Serverless API option to deploy a model as a serverless API.

3. Configure the Endpoint Target URL as the AZURE_SERVERLESS_MODELS_API_ENDPOINT variable and the key in AZURE_SERVERLESS_MODELS_API_KEY in the .env file**.**

   .env

   AZURE_SERVERLESS_MODELS_API_ENDPOINT=https://...models.ai.azure.com
   AZURE_SERVERLESS_MODELS_API_KEY=...

4. Find the deployment name and use it in your script, model: "azure_serverless:deployment-id".

Support for multiple inference deployements

You can update the AZURE_SERVERLESS_MODELS_API_KEY with a list of deploymentid=key pairs to support multiple deployments (each deployment has a different key).

.env

AZURE_SERVERLESS_MODELS_API_KEY="
model1=key1
model2=key2
model3=key3
"

GitHub Copilot Chat Models

If you have access to GitHub Copilot Chat in Visual Studio Code, GenAIScript will be able to leverage those language models as well.

This mode is useful to run your scripts without having a separate LLM provider or local LLMs. However, those models are not available from the command line and have additional limitations and rate limiting defined by the GitHub Copilot platform.

There is no configuration needed as long as you have GitHub Copilot installed and configured in Visual Studio Code. You can force using this model by using client:* as a model name.

1. Install GitHub Copilot Chat (emphasis on Chat)

2. run your script

3. Confirm that you are allowing GenAIScript to use the GitHub Copilot Chat models.

4. select the best chat model that matches the one you have in your script

   

   (This step is skipped if you already have mappings in your settings)

The mapping of GenAIScript model names to Visual Studio Models is stored in the settings.

Anthropic

Anthropic is an AI research company that offers powerful language models, including the Claude series.

script({ model: "anthropic:claude-2.1" })

To use Anthropic models with GenAIScript, follow these steps:

1. Sign up for an Anthropic account and obtain an API key from their console.

2. Add your Anthropic API key to the .env file:

   .env

   ANTHROPIC_API_KEY=sk-ant-api...

3. Find the model that best suits your needs by visiting the Anthropic model documentation.

4. Update your script to use the model you choose.

   script({
       ...
       model: "anthropic:claude-3-5-sonnet-20240620",
   })

Hugging Face

The huggingface provider allows you to use Hugging Face Models using Text Generation Inference.

script({ model: "huggingface:microsoft/Phi-3-mini-4k-instruct" })

To use Hugging Face models with GenAIScript, follow these steps:

1. Sign up for a Hugging Face account and obtain an API key from their console. If you are creating a Fined Grained token, enable the Make calls to the serverless inference API option.

2. Add your Hugging Face API key to the .env file:

   .env

   HUGGINGFACE_API_KEY=hf_...

3. Find the model that best suits your needs by visiting the HuggingFace models.

4. Update your script to use the model you choose.

   script({
       ...
       model: "huggingface:microsoft/Phi-3-mini-4k-instruct",
   })

Note

Some models may require a Pro account.

LocalAI

LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It uses free Open Source models and it runs on CPUs.

LocalAI acts as an OpenAI replacement, you can see the model name mapping used in the container, like gpt-4 is mapped to phi-2.

1. Install Docker. See the LocalAI documentation for more information.

2. Update the .env file and set the api type to localai.

   .env

   OPENAI_API_TYPE=localai

Ollama

Ollama is a desktop application that let you download and run model locally.

Running tools locally may require additional GPU resources depending on the model you are using.

Use the ollama provider to access Ollama models.

Note

GenAIScript is currently using the OpenAI API compatibility layer of Ollama.

1. Start the Ollama application or

   ollama serve

2. Update your script to use the ollama:phi3.5 model (or any other model or from Hugging Face).

   script({
       ...,
       model: "ollama:phi3.5",
   })

   GenAIScript will automatically pull the model, which may take some time depending on the model size. The model is cached locally by Ollama.

3. If Ollama runs on a server or a different computer or on a different port, you have to configure the OLLAMA_HOST environment variable to connect to a remote Ollama server.

   .env

   OLLAMA_HOST=https://<IP or domain>:<port>/ # server url
   OLLAMA_HOST=0.0.0.0:12345 # different port

You can specify the model size by adding the size to the model name, like ollama:llama3.2:3b.

script({
    ...,
    model: "ollama:llama3.2:3b",
})

You can also use GGUF models from Hugging Face.

script({
    ...,
    model: "ollama:hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF",
})

Llamafile

https://llamafile.ai/ is a single file desktop application that allows you to run an LLM locally.

The provider is llamafile and the model name is ignored.

Jan, LMStudio, LLaMA.cpp

Jan, LMStudio, LLaMA.cpp also allow running models locally or interfacing with other LLM vendors.

1. Update the .env file with the local server information.

   .env

   OPENAI_API_BASE=http://localhost:...

OpenRouter

You can configure the OpenAI provider to use the OpenRouter service instead by setting the OPENAI_API_BASE to https://openrouter.ai/api/v1. You will also need an api key.

.env

OPENAI_API_BASE=https://openrouter.ai/api/v1
OPENAI_API_KEY=...

Then use the OpenRouter model name in your script:

script({ model: "openai:openai/gpt-4o-mini" })

By default, GenAIScript will set the site url and name to GenAIScript but you can override these settigns with your own values:

.env

OPENROUTER_SITE_URL=... # populates HTTP-Referer header
OPENROUTER_SITE_NAME=... # populate X-Title header

Model specific environment variables

You can provide different environment variables for each named model by using the PROVIDER_MODEL_API_... prefix or PROVIDER_API_... prefix. The model name is capitalized and all non-alphanumeric characters are converted to _.

This allows to have various sources of LLM computations for different models. For example, to enable the ollama:phi3 model running locally, while keeping the default openai model connection information.

.env

OLLAMA_PHI3_API_BASE=http://localhost:11434/v1

Checking your configuration

You can check your configuration by running the genaiscript info env command. It will display the current configuration information parsed by GenAIScript.

genaiscript info env

Next steps

Write your first script.