1st commit

2026-09-22 05:53:59 +07:00
commit 92a23740c6
15 changed files with 2868 additions and 0 deletions
@@ -0,0 +1,194 @@
+# llamacpp_embedding_1_nats
+
+A Julia backend service that forwards messages from NATS to llama.cpp embedding API.
+
+## Overview
+
+This service acts as a bridge between a NATS message broker and a llama.cpp HTTP embedding endpoint. It receives messages via NATS queue groups and forwards them to the llama.cpp server for embedding processing.
+
+## Architecture
+
+```
+┌─────────────┐     NATS      ┌──────────────────────┐     HTTP      ┌─────────────────┐
+│   Requester │ ────────────> │  this service        │ ────────────> │  llama.cpp      │
+│             │ <─────────── <│  (embedding)         │ <─────────── <│  /v1/embeddings │
+│             │     Queue     │                      │               │                 │
+│             │     Group     │                      │               │                 │
+└─────────────┘     (free)    └──────────────────────┘               └─────────────────┘
+```
+
+## Features
+
+- NATS queue group subscription for load balancing across multiple service instances
+- HTTP connection pooling cleanup to prevent llama.cpp server timeouts
+- Health check endpoint on port 3000
+- Service registration with load balancer
+- Message envelope packing/unpacking via `msghandler` package
+
+## Prerequisites
+
+- Julia 1.12+
+- NATS server
+- llama.cpp server running with embedding endpoint
+- Docker (for containerized deployment)
+
+## Configuration
+
+Edit `appconfig.json` to configure the service:
+
+```json
+{
+  "config_version": "0.1",
+  "nats_server_info": {
+      "url": "nats.yiem.cc"
+  },
+  "this_service_name": "embedding",
+  "this_service_input_channel": {
+      "nats": "sommpanion.backend.embedding_nats_1.v1.tonpc.inbox"
+  },
+  "externalService": {
+    "fileserver": {
+      "url": "https://fileserver.yiem.cc"
+    },
+    "servicesloadbalancer": {
+      "url": "sommpanion.backend.servicesloadbalancer.v1.inbox"
+    },
+    "llamacpp_embedding": {
+      "url": "http://192.168.88.50:9010/v1/embeddings"
+    }
+  }
+}
+```
+
+## Local Development
+
+```bash
+julia --project -t 2 main.jl
+```
+
+The service exposes:
+- Health check endpoints on port 3000: `/`, `/health`, `/ready`
+- NATS subscriber on configured topic
+
+## Docker Deployment
+
+```bash
+docker build -t llamacpp_embedding_1_nats .
+docker run -d --name llamacpp_embedding_1_nats -p 3000:3000 llamacpp_embedding_1_nats
+```
+
+Or use docker-compose:
+
+```bash
+docker-compose up -d
+```
+
+## Usage Example
+
+### Using NATS with msghandler
+
+```julia
+using JSON, NATS, msghandler
+
+const config = JSON.parsefile("./appconfig.json")
+
+# Prepare documents to embed
+documents = [
+    "System architecture specification for heart monitor logic.",
+    "NATS broker cluster configuration options for low-latency.",
+    "Julia signal processing worker initialization sequence."
+]
+
+documents_dict = Dict("documents" => documents)
+
+# Pack message envelope
+payloads = [("documents", documents_dict, "dictionary")]
+message_envelope, forward_msg_envelope_json_str = msghandler.smartpack(
+    config["this_service_input_channel"]["nats"], 
+    payloads;   
+    msg_purpose="embedding",  
+    broker_url=config["nats_server_info"]["url"],
+    fileserver_url=config["externalService"]["fileserver"]["url"])
+
+# Send via NATS
+connection = NATS.connect(config["nats_server_info"]["url"])
+reply = NATS.request(connection, 
+    config["this_service_input_channel"]["nats"], 
+    forward_msg_envelope_json_str, 
+    timeout=120)
+
+# Unpack response
+incoming_env_json_str = String(reply.payload)
+incoming_env = msghandler.smartunpack(incoming_env_json_str)
+embedding_response = incoming_env["payloads"][1][2]
+
+# Process embeddings
+for (i, item) in enumerate(embedding_response["data"])
+    vector = item["embedding"]
+    println("Document $i embedding vector size: ", length(vector))
+end
+```
+
+### Using Direct HTTP (for testing)
+
+```julia
+using HTTP, JSON
+
+url = "http://localhost:9010/v1/embeddings"
+headers = [
+    "Content-Type" => "application/json",
+    "Authorization" => "Bearer no-key"
+]
+
+documents = [
+    "System architecture specification for heart monitor logic.",
+    "NATS broker cluster configuration options for low-latency."
+]
+
+body_data = Dict(
+    "model" => "embeddinggemma-300M-BF16",
+    "input" => documents
+)
+
+response = HTTP.post(url, headers, JSON.json(body_data))
+parsed = JSON.parse(String(response.body))
+
+for (i, item) in enumerate(parsed["data"])
+    println("Document $i embedding size: ", length(item["embedding"]))
+end
+```
+
+## Message Flow
+
+1. **Registration**: Service registers with load balancer via NATS
+2. **Subscription**: Subscribes to input topic with queue group "group1"
+3. **Processing**:
+   - Receives message via NATS
+   - Unpacks message envelope
+   - Forwards to llama.cpp HTTP endpoint
+   - Cleans up HTTP connections
+   - Repacks response and publishes to reply-to topic
+   - Re-registers with load balancer
+
+## Health Monitoring
+
+```bash
+curl http://localhost:3000/health
+curl http://localhost:3000/ready
+```
+
+## Dependencies
+
+- `NATS.jl` - NATS client
+- `HTTP.jl` - HTTP client with connection pooling
+- `JSON.jl` - JSON serialization
+- `msghandler` - Message envelope handling
+- `GeneralUtils` - Utility functions
+
+## Scaling
+
+Multiple instances can be deployed. NATS queue group "group1" ensures each message is delivered to only one instance.
+
+## License
+
+MIT