llamacpp_embedding_1_nats

A Julia backend service that forwards messages from NATS to llama.cpp embedding API.

Overview

This service acts as a bridge between a NATS message broker and a llama.cpp HTTP embedding endpoint. It receives messages via NATS queue groups and forwards them to the llama.cpp server for embedding processing.

Architecture

┌─────────────┐     NATS      ┌──────────────────────┐     HTTP      ┌─────────────────┐
│   Requester │ ────────────> │  this service        │ ────────────> │  llama.cpp      │
│             │ <─────────── <│  (embedding)         │ <─────────── <│  /v1/embeddings │
│             │     Queue     │                      │               │                 │
│             │     Group     │                      │               │                 │
└─────────────┘     (free)    └──────────────────────┘               └─────────────────┘

Features

• NATS queue group subscription for load balancing across multiple service instances
• HTTP connection pooling cleanup to prevent llama.cpp server timeouts
• Health check endpoint on port 3000
• Service registration with load balancer
• Message envelope packing/unpacking via msghandler package

Prerequisites

• Julia 1.12+
• NATS server
• llama.cpp server running with embedding endpoint
• Docker (for containerized deployment)

Configuration

Edit appconfig.json to configure the service:

{
  "config_version": "0.1",
  "nats_server_info": {
      "url": "nats.yiem.cc"
  },
  "this_service_name": "embedding",
  "this_service_input_channel": {
      "nats": "sommpanion.backend.embedding_nats_1.v1.tonpc.inbox"
  },
  "externalService": {
    "fileserver": {
      "url": "https://fileserver.yiem.cc"
    },
    "servicesloadbalancer": {
      "url": "sommpanion.backend.servicesloadbalancer.v1.inbox"
    },
    "llamacpp_embedding": {
      "url": "http://192.168.88.50:9010/v1/embeddings"
    }
  }
}

Local Development

julia --project -t 2 main.jl

The service exposes:

• Health check endpoints on port 3000: /, /health, /ready
• NATS subscriber on configured topic

Docker Deployment

docker build -t llamacpp_embedding_1_nats .
docker run -d --name llamacpp_embedding_1_nats -p 3000:3000 llamacpp_embedding_1_nats

Or use docker-compose:

docker-compose up -d

Usage Example

Using NATS with msghandler

using JSON, NATS, msghandler

const config = JSON.parsefile("./appconfig.json")

# Prepare documents to embed
documents = [
    "System architecture specification for heart monitor logic.",
    "NATS broker cluster configuration options for low-latency.",
    "Julia signal processing worker initialization sequence."
]

documents_dict = Dict("documents" => documents)

# Pack message envelope
payloads = [("documents", documents_dict, "dictionary")]
message_envelope, forward_msg_envelope_json_str = msghandler.smartpack(
    config["this_service_input_channel"]["nats"], 
    payloads;   
    msg_purpose="embedding",  
    broker_url=config["nats_server_info"]["url"],
    fileserver_url=config["externalService"]["fileserver"]["url"])

# Send via NATS
connection = NATS.connect(config["nats_server_info"]["url"])
reply = NATS.request(connection, 
    config["this_service_input_channel"]["nats"], 
    forward_msg_envelope_json_str, 
    timeout=120)

# Unpack response
incoming_env_json_str = String(reply.payload)
incoming_env = msghandler.smartunpack(incoming_env_json_str)
embedding_response = incoming_env["payloads"][1][2]

# Process embeddings
for (i, item) in enumerate(embedding_response["data"])
    vector = item["embedding"]
    println("Document $i embedding vector size: ", length(vector))
end

Using Direct HTTP (for testing)

using HTTP, JSON

url = "http://localhost:9010/v1/embeddings"
headers = [
    "Content-Type" => "application/json",
    "Authorization" => "Bearer no-key"
]

documents = [
    "System architecture specification for heart monitor logic.",
    "NATS broker cluster configuration options for low-latency."
]

body_data = Dict(
    "model" => "embeddinggemma-300M-BF16",
    "input" => documents
)

response = HTTP.post(url, headers, JSON.json(body_data))
parsed = JSON.parse(String(response.body))

for (i, item) in enumerate(parsed["data"])
    println("Document $i embedding size: ", length(item["embedding"]))
end

Message Flow

1. Registration: Service registers with load balancer via NATS
2. Subscription: Subscribes to input topic with queue group "group1"
3. Processing:
   • Receives message via NATS
   • Unpacks message envelope
   • Forwards to llama.cpp HTTP endpoint
   • Cleans up HTTP connections
   • Repacks response and publishes to reply-to topic
   • Re-registers with load balancer

Health Monitoring

curl http://localhost:3000/health
curl http://localhost:3000/ready

Dependencies

• NATS.jl - NATS client
• HTTP.jl - HTTP client with connection pooling
• JSON.jl - JSON serialization
• msghandler - Message envelope handling
• GeneralUtils - Utility functions

Scaling

Multiple instances can be deployed. NATS queue group "group1" ensures each message is delivered to only one instance.

License

MIT