High-Performance Local AI Infrastructure

A private on-premise AI platform built on 2 dedicated servers, in partnership with IBM AI Factory (Tier 3 datacenter), for projects that demand stability, security, governance and scalability. It brings together, on the same GPU, LLM inference, embeddings, RAG/OCR, image/video/3D generation and 3D rendering — serving external applications only through an internal OpenAI-compatible API.

What the platform delivers

• High-performance LLM: a 256K-token context window, 512 concurrent sequences and up to ~1,300 inferences per second with queueing — ready for parallel agentic workloads.
• Co-resident embeddings on the same GPU, for semantic search and RAG.
• RAG and OCR (document indexing) on a dedicated node, with Qdrant and Redis.
• Image, video and 3D generation (ComfyUI) plus 3D rendering with Blender (Cycles + OptiX) on the GPU.
• OpenAI-compatible API: any existing SDK or client works without code changes.

Architecture

Two complementary nodes: a GPU node dedicated to heavy inference and a service node (gateway, RAG, OCR, vector search and TLS proxy). Nodes and consumers communicate over a mesh VPN and a reverse tunnel with a dedicated key; access is protected by Nginx + TLS and panel authentication.

Consumer applications run on external VPS instances and use only the internal API — no model runs outside the company's own infrastructure.

Data sovereignty and security

• Data never leaves the physical infrastructure: prompts, indexed documents (RAG), outputs and model weights stay in-house; nothing is sent to third-party providers or used to train third-party models.
• Residency and compliance: meets LGPD requirements and those of clients that forbid sending sensitive data to third-party clouds.
• Full control over retention and deletion: deleting data means deleting it locally, immediately and verifiably.
• No vendor lock-in: OpenAI-compatible API and a local model library — changing models does not break applications.
• Auditing: inference logs per service and an in-house metrics store, traceable inside the environment itself.

Cost: near-zero marginal cost per token

More than 1.4 billion tokens have already been processed — over 28,000 LLM requests and 58,000 embedding requests — with no per-token charge. The cost is electricity plus already-amortized hardware: marginal inference cost ≈ 0, with no throttling, no fair-use limits and no request caps. On comparable commercial APIs this volume would run into thousands of dollars — here, the extra cost is indistinguishable from the server's electricity bill.

Method: research, lab and documentation

Nothing here was improvised. The platform is the result of applied research and lab experimentation: comparing several models, testing different inference configurations and validating performance before moving anything to production.

This foundation combines more than 29 years of software development and architecture experience with the systematic reading of official technical documentation — OpenAI, Anthropic, Qwen and others — and of the configuration details and model cards published on Hugging Face. Every parameter (context, concurrency, VRAM partitioning, quantization, parsers) was chosen based on evidence, not assumption.

It is this combination — engineering experience plus research and lab discipline applied to AI — that makes it possible to get the best performance out of the GPU with stability and predictability.

Engineering and operations

• LLM tuned for production: speculative decoding (MTP), native tool-calling for code agents, VRAM partitioning (LLM + embeddings on the same GPU) and a configuration history kept as scripts, allowing changes without reimplementation.
• Mature operations: services under systemd with auto-restart, orchestration and health-check scripts, runbooks and a rollback procedure.
• In-house operations dashboard: status, historical metrics, start/stop control and a VRAM monitor.
• Multi-tenant on the GPU: a documented protocol for 3D rendering to share and release the GPU with the LLM — a single GPU serving inference, media generation and rendering at the same time.
• Proven stability: 120 days in production without incidents, with a redundant mirror.

In short: a private AI foundation that turns dedicated hardware into scalable, sovereign and cost-predictable inference capacity for products that require data governance.