Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Assessing the risks associated with cloud LLMs, including data retention, input-based training, and foreign jurisdictional issues.
- Examining the Ollama architecture, encompassing the model server, registry, and OpenAI-compatible API.
- Comparing Ollama with alternative solutions such as vLLM, llama.cpp, and Text Generation Inference.
- Reviewing licensing terms for models like Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux systems with support for both CUDA and ROCm.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2 instructions.
- Managing Docker deployments and configuring persistent volume mappings.
- Configuring multi-GPU setups and defining VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like 'ollama pull llama3'.
- Importing GGUF format models from HuggingFace and TheBloke.
- Evaluating the trade-offs of different quantization levels, such as Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding limits on concurrent model loading.
Custom Modelfiles
- Mastering Modelfile syntax, including the use of FROM, PARAMETER, SYSTEM, and TEMPLATE.
- Tuning generation parameters like temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce role-specific behaviors.
- Creating custom models and publishing them to a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output mode.
- Integrating local models with frameworks like LangChain, LlamaIndex, and bespoke applications.
- Implementing authentication and rate limiting through reverse proxy configurations.
Performance Optimization
- Optimizing context window sizing and managing KV caches effectively.
- Implementing batch inference techniques and handling parallel requests.
- Allocating CPU threads and maintaining NUMA awareness.
- Monitoring GPU utilization and addressing memory pressure issues.
Security and Compliance
- Establishing network isolation for model serving endpoints.
- Deploying input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and generated completions.
- Verifying model provenance and ensuring hash integrity.
Requirements
- Proficiency in intermediate-level Linux and container administration.
- A high-level understanding of machine learning principles and transformer architectures.
- Working knowledge of REST APIs and JSON data structures.
Target Audience
- AI engineers and developers looking to transition away from cloud-based LLM APIs.
- Organizations with strict data sensitivity requirements that preclude the use of cloud models.
- Government and defense teams necessitating air-gapped language model environments.
