Skip to content
GenAIScript
Blog

Blog

Let there be videos!

Peli
Peli
GenAIScript developer

The latest release includes support for including videos and audio transcripts in your scripts.

const frames = await ffmpeg.extractFrames("demo.mp4", { transcription: true })
def("DEMO", frames)


$`Describe what happens in the <DEMO>.`

Say you want to analyze a video file. For most LLMs that support images, you would have to extract screenshots at particular timestamps then send them as a sequence of images. Choosing those timestamp could be a challenge since you will run out of context window. GenAIScript provides a helpers to solve this tedious tasks around video analysis using LLMs.

tools and agents

We also provides wrap the new functionalities in tools and agents so you can use them in your scripts.

For example, to include the frame extraction tool so that the LLM is able to call it, you can use the following snippet:

script({
    tools: "video_extract_frames",
})

Or just let the agent work on the video for you.

script({
    tools: "agent_video",
})

Playground, o1 and DeepSeek

Peli
Peli
GenAIScript developer

The newly 2025 release brings a number of new features and support for new models.

Playground

The Playground is a self-hosted web application that allows you to run GenAIScript scripts from a friendly user interface. It sits between the GenAIScript CLI and the GenAIScript Visual Studio Code integration.

A screenshot of the playground.

o1

GenAIScript supports the various flavors of the OpenAI o1 models (mini, preview, …). It also adds support for tools.

o1 is also available on GitHub Models!

script({ model: "github:o1" })
$`Prove that the sum of the angles of a triangle is 180 degrees.`

DeepSeek

GenAIScript supports DeepSeek V3 through their OpenAI API.

script({ model: "deepseek:deepseek-chat" })
$`Prove that the sum of the angles of a triangle is 180 degrees.`

Node.JS API

Peli
Peli
GenAIScript developer

A long standing feature request has been to run GenAIScript programmatically from other scripts. We are happy to announce that we have released a Node.JS API for GenAIScript. This API allows you to call GenAIScript from other TypeScript scripts (v1.83+).

Installation

You’ll want to add genaiscript as a (dev) dependency to your project.

Terminal window
npm i -D genaiscript

The run API

The run API is meant to mimic the behavior of the GenAIScript CLI. It takes the same arguments as the CLI and returns the same results. This allows you to call GenAIScript from other TypeScript scripts.

import { run } from "genaiscript/api"
const results = await run("summarize", ["myfile.txt"])

The result object contains the full list of messages, and additional parsed information like modified files, diagnostics and so forth.

Don’t mess with my process

On the caller side, the run implementation is a dependency free, side effect free function. It spawns a worker thread where GenAIScript does the work.

  • No global added
  • No package loaded
  • A few hundred b of memory used

Help us improve it!

Obviously this is a first draft and we could do a better job at providing callbacks for progress. Send us your feedback!

Hugging Face Transformers.js

Peli
Peli
GenAIScript developer

🤗

Hugging Face Transformers.js is a JavaScript library that provides a simple way to run LLMs in the browser or node.js (or Bun, Deno, …).

With the latest GenAIScript, you can use Text Generation Models directly in the script configuration using the transformers model provider.

script({
  model: "transformers:HuggingFaceTB/SmolLM2-1.7B-Instruct:q4f16"
})

GenAIScript will download and cache the model for you, and you can start using it right away fully locally.

There are plenty of models to choose from and you can also follow the Hugging Face documentation to fine tune your own.

Fallback Tools

Peli
Peli
GenAIScript developer

Tools is a powerful feature of LLM models that allows you to augment the LLM reasoning with external tools.

These days, many LLM models come with a built-in support for tools. However, some of them don’t… like OpenAI’s o1-preview and o1-mini.

Fallback tools

With GenAIScript 1.72.0, we introduce the concept of fallback tools. Basically, it consists of a system script that “teaches” the LLM model about available tools and how to call them.

$`## Tool support


You can call external tools to help generating the answer of the user questions.


- The list of tools is defined in TOOLS. Use the description to help you choose the best tools.
- Each tool has an id, description, and a JSON schema for the arguments.
...


\`\`\`tool_calls
<tool_id>: { <JSON_serialized_tool_call_arguments> }
<tool_id_2>: { <JSON_serialized_tool_call_arguments_2> }
...
\`\`\`

A tool example

Here is an example of a tool that generates a random number between 0 and 1.

defTool("random", "Generate a random number", {}, () => Math.random())
$`Generate a random number between 0 and 1.`
  • o1-mini trace (using GitHub Models)
prompting github:o1-mini (~490 tokens)
```tool_calls
random: {}
```


prompting github:o1-mini (~532 tokens)
Your random number between 0 and 1 is **0.7792901036554349**.
  • gemma2 model (using Ollama)
prompting ollama:gemma2 (~716 tokens)


```tool_calls
random: {}
```
prompting ollama:gemma2 (~758 tokens)


The random number is 0.9552638470626966.




Let me know if you'd like to generate another random number!

Activation

The fallback tool mode is automatically activated for known LLM models that don’t support tools natively. The list is not complete so open an issue if you stumble upon a model that should have fallback tools enabled.

It can be activated manually by setting the fallbackTools option to true in the script configuration.

script({
    fallbackTools: true,
})

or by setting the --fallback-tools flag in the CLI.

Terminal window
genaiscript run --fallback-tools ...