PCIe 4.0 x8/x8 vs x16/x16 for Dual GPUs: Does Bandwidth Choke Local LLM Tensor Parallelism?
📌 The Executive Verdict: Does PCIe 4.0 x8/x8 Choke Dual-GPU Inference? In dual-GPU workstations running local LLMs under tensor parallelism (such as…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
discuss
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
security
showdev
testing
tutorial
typescript
webdev
Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4
This article gives a short background on Amazon SageMaker real-time endpoints, then measures Gemma 4 E2B's quantization-aware trained (QAT) checkpoi…
Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4
This article gives a short background on Amazon SageMaker real-time endpoints, then measures Gemma 4 E2B's quantization-aware trained (QAT) checkpoi…
Gemma 4 on a Tesla T4, Part 2: The Minimum GCE VM and a Script to Drive It
This article provides a step by step deployment guide for **Gemma 4 E2B * to a Tesla T4 hosted GPU enabled system. A suite of Python MCP tools is bu…
Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16
This article provides a step by step deployment guide for **Gemma 4 E2B * to a Tesla T4 hosted GPU enabled system. A suite of Python MCP tools is bu…
FastMCP Is Now MCPServer: Migrating a Python MCP Server to the MCP SDK 2.x
This article provides a step by step migration guide for a Python MCP server from the MCP Python SDK 1.x ( FastMCP ) to 2.x ( MCPServer ), followed…
KV Cache on 16 GB GPUs: Making Long Context Actually Fit
A model can advertise a 128K context window and still fail at 40K tokens on a 16 GB GPU. The architecture ceiling never promised that weights, KV ca…
From adapter to deployment: merging LoRA weights and serving with vLLM or a Space
A fine-tuned adapter sitting in a folder is not a product. FineTune Studio offers three inference paths — local, a vLLM server, or a Hugging Face Sp…
Deploying Inference Using NVIDIA Dynamo and vLLM
NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework for deploying large-scale generative AI and reasoning models acros…
vLLM v0.28.0: the breaking change small GPU users must read
vLLM v0.28.0 shipped on 26 August 2026 with 584 commits from 270 contributors (76 new) , and almost every headline in the release notes on GitHub is…
Serving Gemma4 with Rust on vLLM 🦀
This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4…
Installing Rust for vLLM on Graviton: a G5g walk-through 🦀
This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4…
Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU
A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstac…
Self-hosting a lite agent backend on one TPU: Gemma 4 E2B + vLLM on a v5e-1
Self-hosting a lite agent backend on one TPU chip A single Google Cloud TPU v5e chip — 16 GB of HBM, about $0.58/hour on spot — will serve google/ge…
Kimi K3 Open Weights Are Here: How to Self-Host the 2.8T-Parameter Model (Hardware, vLLM, and Data Sovereignty)
Kimi K3 Open Weights Are Here: How to Self-Host the 2.8T-Parameter Model On July 27, 2026, Moonshot AI released the open weights for Kimi K3 -- a 2.…
What 90% Line-Rate Utilization on a Single 100GbE Port Means: Analyzing Network Bottlenecks in Inference Storage
In LLM inference clusters, the core bottleneck for KV Cache storage acceleration often lies not in the storage medium itself, but in network bandwid…
Gemma 4 E2B on a Single TPU v6e Chip: A Serving Deep Dive
Measured 2026-07-21 on vllm/vllm-tpu:nightly (vLLM 0.23.1rc1.dev1076), a GCE flex-start ct6e-standard-1t (one TPU v6e chip, 32 GB HBM) in europe-wes…
Does a Second GPU Increase Ollama's Context Window? (Quadro P2000 + RTX 3090 Tested)
TL;DR Short version: no. I dropped a much older GPU ( Quadro P2000, 5GB, Pascal, 2016 ) next to an RTX 3090 (24GB, Ampere) on the same box, ran the…
Qwen3.6-35B NVFP4 runs on one H100 — A100 owners are out
NVIDIA published nvidia/Qwen3.6-35B-A3B-NVFP4 on May 28, 2026 — a post-training FP4-quantized variant of Alibaba's 35B MoE model that fits on a sing…
I built an open-source alternative to Microsoft's KAITO that works on ANY Kubernetes cluster
Six months ago, my team needed to deploy DeepSeek-R1 for internal use. We have a Kubernetes cluster — like everyone does in 2026 — so I started look…
Prefix caching at scale: when it saves you 80% of prefill cost, and the eviction policies that quietly turn it into 5%
Prefix caching at scale: when it saves you 80% of prefill cost, and the eviction policies that quietly turn it into 5% Your chatbot deploys 70B Llam…
KV cache quantization: what FP8/INT8 K and V actually buy you, and where they break
KV cache quantization: what FP8/INT8 K and V actually buy you, and where they break You just deployed a 70B Llama fine-tune on 8x H100s, and your se…
Ollama vs llama.cpp vs vLLM: Which Should You Use in 2026?
From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing. Three to…