Images in Azure Blob Storage

It is possible to use the Azure Node.JS SDK to download images from Azure Blog Storage and use them in the prompt. The defImages function support the node.js [Buffer] type.

Configuration

Install the @azure/storage-blob and @azure/identity packages.

npm install -D @azure/storage-blob @azure/identity

Make sure to login with the Azure CLI and set the subscription.

az login

Reading blobs

Open a connection to the Azure Blob Storage and get a client to the container. We deconstruct the account and container from the env.vars object so that they can be set through the cli.

import { BlobServiceClient } from "@azure/storage-blob"
import { DefaultAzureCredential } from "@azure/identity"

const { account = "myblobs", container = "myimages" } = env.vars
const blobServiceClient = new BlobServiceClient(
    `https://${account}.blob.core.windows.net`,
    new DefaultAzureCredential()
)
const containerClient = blobServiceClient.getContainerClient(container)

If you do not have a specific blob in mind, you can iterate through the blobs, and download them into a buffer (buf).

import { buffer } from "node:stream/consumers"

for await (const blob of containerClient.listBlobsFlat()) {
    const blockBlobClient = containerClient.getBlockBlobClient(blob.name)
    const downloadBlockBlobResponse = await blockBlobClient.download(0)
    const body = await downloadBlockBlobResponse.readableStreamBody
    const image = await buffer(body)
    ...

Using images in the prompt

The image buffer can be passed in defImages to be used in the prompt.

    defImages(image, { detail: "low" })

However since images can be “heavy”, you will most likely have to use inline prompts to split into smaller queries. (Note the use of _.)

for await (const blob of containerClient.listBlobsFlat()) {
    ...
    const res = await runPrompt(_ => {
        _.defImages(image, { detail: "low" })
        _.$`Describe the image.`
    })
    // res contains the LLM response for the inner prompt
    ...

Summarizing results

To summarize all images, we store each image summary using the def function and add prompting to summarize the descriptions.

    ...
    def("IMAGES_SUMMARY", { filename: blob.name, content: res.text })
}
$`Summarize IMAGES_SUMMARY.`

Full source

import { BlobServiceClient } from "@azure/storage-blob"
import { DefaultAzureCredential } from "@azure/identity"
import { buffer } from "node:stream/consumers"

script({
    parameters: {
        account: {
            description: "Azure Storage Account Name",
            default: "genaiscript",
            type: "string",
        },
        container: {
            description: "Azure Storage Container Name",
            default: "images",
            type: "string",
        },
    },
})

const { account, container } = env.vars
const url = `https://${account}.blob.core.windows.net`
console.log(`analyzing images in ${account}/${container} at ${url}`)
const blobServiceClient = new BlobServiceClient(
    url,
    new DefaultAzureCredential()
)
const containerClient = blobServiceClient.getContainerClient(container)
for await (const blob of containerClient.listBlobsFlat()) {
    console.log(`blob: ` + blob.name)
    const blockBlobClient = containerClient.getBlockBlobClient(blob.name)
    const downloadBlockBlobResponse = await blockBlobClient.download(0)
    const body = await downloadBlockBlobResponse.readableStreamBody
    const image = await buffer(body)

    const res = await runPrompt(_ => {
        _.defImages(image, { detail: "low" })
        _.$`Describe the images.`
    })

    def("IMAGES_SUMMARY", { filename: blob.name, content: res.text })
}

$`Summarize IMAGES_SUMMARY.`