llamacpp_embedding_nats/README.md

# llamacpp_embedding_1_nats

A Julia backend service that forwards messages from NATS to llama.cpp embedding API.

## Overview

This service acts as a bridge between a NATS message broker and a llama.cpp HTTP embedding endpoint. It receives messages via NATS queue groups and forwards them to the llama.cpp server for embedding processing.

## Architecture

```
┌─────────────┐     NATS      ┌──────────────────────┐     HTTP      ┌─────────────────┐
│   Requester │ ────────────> │  this service        │ ────────────> │  llama.cpp      │
│             │ <─────────── <│  (embedding)         │ <─────────── <│  /v1/embeddings │
│             │     Queue     │                      │               │                 │
│             │     Group     │                      │               │                 │
└─────────────┘     (free)    └──────────────────────┘               └─────────────────┘
```

## Features

- NATS queue group subscription for load balancing across multiple service instances
- HTTP connection pooling cleanup to prevent llama.cpp server timeouts
- Health check endpoint on port 3000
- Service registration with load balancer
- Message envelope packing/unpacking via `msghandler` package

## Prerequisites

- Julia 1.12+
- NATS server
- llama.cpp server running with embedding endpoint
- Docker (for containerized deployment)

## Configuration

Edit `appconfig.json` to configure the service:

```json
{
  "config_version": "0.1",
  "nats_server_info": {
      "url": "nats.yiem.cc"
  },
  "this_service_name": "embedding",
  "this_service_input_channel": {
      "nats": "sommpanion.backend.embedding_nats_1.v1.tonpc.inbox"
  },
  "externalService": {
    "fileserver": {
      "url": "https://fileserver.yiem.cc"
    },
    "servicesloadbalancer": {
      "url": "sommpanion.backend.servicesloadbalancer.v1.inbox"
    },
    "llamacpp_embedding": {
      "url": "http://192.168.88.50:9010/v1/embeddings"
    }
  }
}
```

## Local Development

```bash
julia --project -t 2 main.jl
```

The service exposes:
- Health check endpoints on port 3000: `/`, `/health`, `/ready`
- NATS subscriber on configured topic

## Docker Deployment

```bash
docker build -t llamacpp_embedding_1_nats .
docker run -d --name llamacpp_embedding_1_nats -p 3000:3000 llamacpp_embedding_1_nats
```

Or use docker-compose:

```bash
docker-compose up -d
```

## Usage Example

### Using NATS with msghandler

```julia
using JSON, NATS, msghandler

const config = JSON.parsefile("./appconfig.json")

# Prepare documents to embed
documents = [
    "System architecture specification for heart monitor logic.",
    "NATS broker cluster configuration options for low-latency.",
    "Julia signal processing worker initialization sequence."
]

documents_dict = Dict("documents" => documents)

# Pack message envelope
payloads = [("documents", documents_dict, "dictionary")]
message_envelope, forward_msg_envelope_json_str = msghandler.smartpack(
    config["this_service_input_channel"]["nats"], 
    payloads;   
    msg_purpose="embedding",  
    broker_url=config["nats_server_info"]["url"],
    fileserver_url=config["externalService"]["fileserver"]["url"])

# Send via NATS
connection = NATS.connect(config["nats_server_info"]["url"])
reply = NATS.request(connection, 
    config["this_service_input_channel"]["nats"], 
    forward_msg_envelope_json_str, 
    timeout=120)

# Unpack response
incoming_env_json_str = String(reply.payload)
incoming_env = msghandler.smartunpack(incoming_env_json_str)
embedding_response = incoming_env["payloads"][1][2]

# Process embeddings
for (i, item) in enumerate(embedding_response["data"])
    vector = item["embedding"]
    println("Document $i embedding vector size: ", length(vector))
end
```

### Using Direct HTTP (for testing)

```julia
using HTTP, JSON

url = "http://localhost:9010/v1/embeddings"
headers = [
    "Content-Type" => "application/json",
    "Authorization" => "Bearer no-key"
]

documents = [
    "System architecture specification for heart monitor logic.",
    "NATS broker cluster configuration options for low-latency."
]

body_data = Dict(
    "model" => "embeddinggemma-300M-BF16",
    "input" => documents
)

response = HTTP.post(url, headers, JSON.json(body_data))
parsed = JSON.parse(String(response.body))

for (i, item) in enumerate(parsed["data"])
    println("Document $i embedding size: ", length(item["embedding"]))
end
```

## Message Flow

1. **Registration**: Service registers with load balancer via NATS
2. **Subscription**: Subscribes to input topic with queue group "group1"
3. **Processing**:
   - Receives message via NATS
   - Unpacks message envelope
   - Forwards to llama.cpp HTTP endpoint
   - Cleans up HTTP connections
   - Repacks response and publishes to reply-to topic
   - Re-registers with load balancer

## Health Monitoring

```bash
curl http://localhost:3000/health
curl http://localhost:3000/ready
```

## Dependencies

- `NATS.jl` - NATS client
- `HTTP.jl` - HTTP client with connection pooling
- `JSON.jl` - JSON serialization
- `msghandler` - Message envelope handling
- `GeneralUtils` - Utility functions

## Scaling

Multiple instances can be deployed. NATS queue group "group1" ensures each message is delivered to only one instance.

## License

MIT