its-ai-01 Documentation

Overview

its-ai-01 is a GPU-accelerated AI inference server providing OpenAI-compatible LLM APIs, image generation, and a browser-based chat interface. It also runs as a DaVinci Resolve Studio remote render node.

All AI services are accessible over HTTPS at https://ai-01.its.ohio.edu. The server uses an NVIDIA A10 GPU (24 GB VRAM) for accelerated inference.

Getting Started

1. Create an Account

Visit https://ai-01.its.ohio.edu and click Sign Up to create an account. An administrator must then add you to the All-Models group before you can access any models.

2. Start Chatting

Once your account has model access, log in and select a model from the dropdown to start chatting. See the Models section below for descriptions of each model.

3. Get an API Key (for developers)

To use the AI models programmatically from your own code, you need an API key:

1. Log in to https://ai-01.its.ohio.edu
2. Click your profile icon (bottom left)
3. Go to Settings → Account
4. Under API Keys, click Create new secret key
5. Copy the key (starts with sk-) — you won't be able to see it again

Use this key as the api_key parameter in any OpenAI-compatible client library with https://ai-01.its.ohio.edu/v1 as the base URL.

Services

API Usage

The LLM API is OpenAI-compatible. Point any OpenAI client library at https://ai-01.its.ohio.edu as the base URL.

List Available Models

curl https://ai-01.its.ohio.edu/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

Chat Completions

curl https://ai-01.its.ohio.edu/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [
      {"role": "user", "content": "Explain quantum computing in simple terms."}
    ]
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="https://ai-01.its.ohio.edu/v1",
    api_key="YOUR_API_KEY"
)

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

JavaScript / TypeScript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://ai-01.its.ohio.edu/v1',
  apiKey: 'YOUR_API_KEY',
});

const response = await client.chat.completions.create({
  model: 'llama3.1:8b',
  messages: [{ role: 'user', content: 'Hello!' }],
});
console.log(response.choices[0].message.content);

Embeddings

curl https://ai-01.its.ohio.edu/v1/embeddings \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nomic-embed-text",
    "input": "The quick brown fox"
  }'

Vision (Image Analysis)

curl https://ai-01.its.ohio.edu/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llava:13b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]
    }]
  }'

Web Search

Models can search the web for current information using SearXNG, a self-hosted metasearch engine. In the chat UI, prefix your message with @web to trigger a web search:

@web What is the latest stable release of Ubuntu?

The search results are automatically retrieved and injected into the model's context (RAG), allowing it to answer with up-to-date information from the internet.

Available Models

LLM Models (ollama)

Models are loaded into GPU VRAM on demand. Only one large model can be active at a time.

Image Generation Models (ComfyUI)

MCP Servers (Model Context Protocol)

MCP allows LLMs to interact with external tools and services — file systems, databases, APIs, and more. Users can build and register their own MCP servers to extend model capabilities.

What is MCP?

The Model Context Protocol is an open standard that lets LLMs call external tools. For example, an MCP server could let a model query a database, read files, create GitHub issues, or interact with any API. Open WebUI acts as an MCP client and connects to your servers.

Registering an MCP Server

Option A: Streamable HTTP MCP Servers (direct connection)

If your MCP server uses Streamable HTTP transport (the recommended approach):

1. Go to Settings (your user settings, not Admin Panel)
2. Click Tools & Functions or External Tools
3. Click Add Server
4. Select type: MCP (Streamable HTTP)
5. Enter your server URL (e.g., http://your-server:8000)
6. Configure authentication if needed
7. Save

The tools provided by your MCP server will then be available to models in your conversations.

Option B: stdio MCP Servers (using MCPO proxy)

Many MCP servers use stdio transport (they run as a subprocess, not an HTTP server). These need MCPO (MCP-to-OpenAPI Proxy) to translate them into HTTP.

MCPO is installed on this server. To proxy a stdio MCP server:

# Example: proxy a filesystem MCP server
mcpo --port 8300 -- npx -y @modelcontextprotocol/server-filesystem /path/to/allowed/dir

# Example: proxy a GitHub MCP server
GITHUB_TOKEN=your_token mcpo --port 8301 -- npx -y @modelcontextprotocol/server-github

# Example: proxy any stdio MCP server
mcpo --port 8302 -- python my_mcp_server.py

Then register http://localhost:8300 (or the appropriate port) in Open WebUI as described in Option A.

Option C: MCPO via Docker

For persistent MCP proxies, run MCPO as a Docker container:

docker run -d --restart unless-stopped \
  -p 127.0.0.1:8300:8000 \
  --name my-mcp-proxy \
  ghcr.io/open-webui/mcpo:latest \
  -- npx -y @modelcontextprotocol/server-filesystem /data

Building Your Own MCP Server

MCP servers can be built in Python or TypeScript. A minimal Python example:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("My Tools")

@mcp.tool()
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"The weather in {city} is sunny and 72F."

if __name__ == "__main__":
    mcp.run(transport="streamable-http", host="0.0.0.0", port=8000)

Then register http://localhost:8000 in Open WebUI.

Pre-installed: VyOS-ECTSheet (Public)

A publicly accessible MCP server providing VyOS 1.4 router documentation and Ohio University ECT Lab Notebook cheat sheets. 1,500+ searchable content chunks covering VyOS configuration, networking commands, subnetting, and lab procedures. Data is refreshed daily at 3:00 AM.

Available tools:

External endpoint:

https://ai-01.its.ohio.edu/mcp/vyos-ectsheet/mcp
Authorization: Bearer YOUR_API_KEY

Pre-installed: NIST OSCAL Security Controls

An MCP server providing access to all NIST SP 800-53 Rev 5 security controls is pre-installed and available to all models. It includes 324 controls across 20 families, plus LOW, MODERATE, HIGH, and PRIVACY baselines.

Available tools:

Use the "(Web + OSCAL)" model variants in the chat UI for automatic access, or ask any model directly (e.g., "What are the FedRAMP Moderate access control requirements?").

External endpoint:

https://ai-01.its.ohio.edu/mcp/oscal/mcp
Authorization: Bearer YOUR_API_KEY

Hardware & Infrastructure

Architecture

Internet / LAN
      |
  [ Caddy :443 ]  <-- TLS termination, reverse proxy
      |
      +--> /v1/*                -->  [ Open WebUI :8080 ]   (Auth + API gateway)
      |                                  |
      |                                  +--> [ ollama :11434 ]  (LLM inference, GPU)
      |                                  +--> [ SearXNG :8888 ]  (Web search, @web)
      |
      +--> /comfyui/*           -->  [ ComfyUI :8188 ]     (Image gen, GPU)
      +--> /gpu/*               -->  [ GPU Monitor :8500 ]  (Real-time GPU dashboard)
      +--> /mcp/oscal/*         -->  [ OSCAL MCP :8400 ]    (NIST security controls)
      +--> /mcp/vyos-ectsheet/* -->  [ VyOS MCP :8401 ]     (VyOS docs + ECT sheets)
      +--> /docs/*              -->  [ Static HTML ]         (This documentation)
      +--> /health              -->  200 OK
      +--> /*                   -->  [ Open WebUI :8080 ]    (Chat UI + auth)

  [ DaVinci Resolve Render Node :15000 ]  <-- Headless GPU rendering (standalone, not proxied)

API requests to /v1/* are routed through Open WebUI, which handles per-user authentication via API keys before forwarding to ollama for inference. MCP and ComfyUI endpoints are protected by the same API key via forward auth. The GPU Monitor, docs, and health check are public.

Docker Containers

Systemd Services

System Plan

This server is being built in phases. Current status of each phase:

GPU Sharing Constraints

The NVIDIA A10 does not support MIG (Multi-Instance GPU) partitioning. All workloads share one 24 GB VRAM pool via time-slicing.

Recommendation: Run one heavy workload at a time. Lightweight combos (7B LLM + SD at batch=1) can coexist when Resolve isn't rendering.

Phase 6: DaVinci Resolve Project Server (Deferred)

Phase 7: DaVinci Resolve Render Node (Active)

Administration

Common Commands

# Check GPU status
nvidia-smi
nvtop
# Or visit https://ai-01.its.ohio.edu/gpu/

# View running containers
docker ps

# Restart a service
sudo docker restart ollama
sudo docker restart comfyui
sudo docker restart open-webui
sudo systemctl restart gpu-monitor
sudo systemctl restart oscal-mcp
sudo systemctl restart vyos-ectsheet-mcp
sudo systemctl restart resolve-render

# View container logs
sudo docker logs -f ollama
sudo docker logs -f comfyui

# View systemd service logs
sudo journalctl -u gpu-monitor -f
sudo journalctl -u oscal-mcp -f
sudo journalctl -u vyos-ectsheet-mcp -f
sudo journalctl -u resolve-render -f

# List installed LLM models
sudo docker exec ollama ollama list

# Pull a new LLM model
sudo docker exec ollama ollama pull <model-name>

# Remove an LLM model
sudo docker exec ollama ollama rm <model-name>

# Check Caddy status / TLS cert
sudo systemctl status caddy
sudo caddy validate --config /etc/caddy/Caddyfile

# Reload Caddy config after edits
sudo systemctl reload caddy

Key File Locations

Stopping AI Services for Resolve Rendering

Before running heavy DaVinci Resolve render jobs, free up GPU VRAM:

# Stop AI containers to free GPU
sudo docker stop ollama comfyui

# After rendering completes, restart them
sudo docker start ollama comfyui

# Restart the render node if needed
sudo systemctl restart resolve-render

User Management

Users self-register at https://ai-01.its.ohio.edu. After signup, an admin must grant model access:

1. Go to Admin Panel (top nav)
2. Click the Users tab to see all registered users
3. Add the user to the All-Models group

To grant model access to the group:

1. Go to Admin Panel → Settings → Models
2. Click each model
3. Under access control, add the All-Models group

Adding New Models

LLM models:

# Browse models at https://ollama.com/library
sudo docker exec ollama ollama pull <model>:<tag>

# Example: pull a coding model
sudo docker exec ollama ollama pull qwen2.5-coder:14b

Image generation models:

# Download .safetensors files into ComfyUI checkpoints directory
sudo docker exec -u root comfyui wget -O \
  /basedir/models/checkpoints/model_name.safetensors \
  "https://huggingface.co/org/repo/resolve/main/model.safetensors"

Full Server Setup / Rebuild

A setup script at /home/itsadmin/setup.sh can provision the entire server from a fresh Ubuntu 24.04 install. It handles NVIDIA drivers, Docker, Caddy, all containers, Python venvs, MCP servers, the GPU monitor, and systemd services.

# Run from a fresh Ubuntu 24.04 install (as itsadmin with sudo)
chmod +x ~/setup.sh && ~/setup.sh

Refresh VyOS MCP Data

The VyOS-ECTSheet MCP server data can be refreshed from upstream sources:

cd ~/mcp-servers/vyos-ectsheet && bash refresh.sh