Annotations

Annotations are errors, warnings, or notes that can be added to the LLM output. They are extracted and integrated into VSCode or your CI environment.

$`Report issues with this code using annotations.`

Configuration

If you use annotation in your script text without specifying the system field, system.annotations will be added by default.

Utilizing the system.annotations system prompt enables the LLM to generate errors, warnings, and notes.

script({
    ...
    system: [..., "system.annotations"]
})

To get a pretty rendering in the Markdown preview, try the Markdown Preview for Github Alerts extension.

Line numbers

The system.annotations prompt automatically enables line number injection for all def sections. This enhances the precision of the LLM’s responses and reduces the likelihood of hallucinations.

GitHub Action Commands

By default, the annotations use the GitHub Action Commands syntax. This means that the annotations will automatically be extracted by GitHub if you run your script in a GitHub Action.

Github Pull Request Review Comments

Use the --pull-request-reviews (-prr) flag on the cli run to add annotations as review comments on a pull request.

npx --yes genaiscript run ... --pull-request-reviews

Visual Studio Code Programs

Annotations are converted into Visual Studio Diagnostics, which are presented to the user through the Problems panel. These diagnostics also manifest as squiggly lines in the editor.

Static Analysis Results Interchange Format (SARIF)

GenAIScript converts these annotations into SARIF files, which can be uploaded as security reports, akin to CodeQL reports.

The SARIF Viewer extension facilitates the visualization of these reports.

GitHub Action

name: "Upload SARIF"

# Run workflow each time code is pushed to your repository and on a schedule.
# The scheduled workflow runs every Thursday at 15:45 UTC.
on:
    push:
    schedule:
        - cron: "45 15 * * 4"
jobs:
    build:
        runs-on: ubuntu-latest
        permissions:
            # required for all workflows
            security-events: write
            # only required for workflows in private repositories
            actions: read
            contents: read
        steps:
            # This step checks out a copy of your repository.
            - name: Checkout repository
              uses: actions/checkout@v4
            # Run GenAIScript tools
            - name: Run GenAIScript
              run: npx --yes genaiscript ... -oa result.sarif
            # Upload the generated SARIF file to GitHub
            - name: Upload SARIF file
              if: success() || failure()
              uses: github/codeql-action/upload-sarif@v3
              with:
                  sarif_file: result.sarif

Limitations

• Access to security reports may vary based on your repository visibility and organizational rules. Refer to the GitHub Documentation for further assistance.
• Your organization may impose restrictions on the execution of GitHub Actions for Pull Requests. Consult the GitHub Documentation for additional guidance.

Filtering

You can use the defOutputProcessor function to filter the annotations.

defOutputProcessor((annotations) => {
    // only allow errors
    const errors = annotations.filter(({ level }) => level === "error")
    return { annotations: errors }
})