Pure JAX on G5g: Serving Gemma 4 on Graviton and a T4G
This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g instance using pure JAX. The code is here: gith…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
The Matrix Wasn't A Battery Farm. It Was A GPU Cluster Made Of Human Brains.
Nvidia is worth more than most countries because we cannot figure out how to do cheap inference. There are eight billion units walking around that d…
Latest Trends in GPU Cloud Cost Reduction and Containerized Data Centers
📝 Originally published (in Japanese) at forge.workstyle.tech . Introduction In recent years, the demand for GPUs has surged with the widespread adop…
Deploying DeepSeek R1 Reasoning LLM Using SGLang
DeepSeek R1 is a first-generation reasoning model tuned for math, coding, and logical reasoning — reinforcement learning with a cold-start phase for…
The sm_120 shared-memory cliff: why FP8 KV cache crashes vLLM on workstation Blackwell
If you run vLLM with --kv-cache-dtype fp8 on a DeepSeek-family (MLA) model and your GPU is a GB10, an RTX PRO 6000, or any workstation or consumer B…
Stop Comparing GPU Clouds Only by $/hour
GPU Cloud Research · #002 — 11 min read The cheapest GPU instance is not always the cheapest way to finish a workload. GPU cloud comparison pages us…
My CUDA/GPU Journey: From "What Even Is a GPU?" to Actually Fascinated
A few months ago, if you'd asked me what a GPU actually does, I would've mumbled something about "graphics" and changed the subject. I'm an undergra…
I Got 28 TPS Out of Free Kaggle GPUs. Here's What It Took.
I want to be upfront about something: this whole project runs on free Kaggle T4 notebooks, an AWS EC2 t3.micro relay that costs almost nothing, and…
Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess
Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess OK, you're a senior SRE, you've been hearing incessantly about AI models, but a…
From API to GPU, Week 5: Tensors, the Data Structure Behind Every Model
Phase 2 of 8: Enough ML to understand inference. Week 5 of 32. Phase 1 was about running models. Phase 2 is about understanding what happens inside…
Mastering Low-Precision AI: FP8 and FP4 Support Across Frameworks in Mid-2026
In mid-2026, FP8 and FP4 have become essential tools for making large-scale AI training and inference more efficient . FP8 uses two main formats-E4M…
Rust SIMD Just Came to the GPU — and It Changes How We Think About Parallel Programming
For decades, GPU programming has meant one of two things: writing CUDA kernels in C++ or wrestling with OpenCL. Both require you to think in a funda…
I Reviewed 100 Reddit Threads About GPU Clouds. Price Was Only Part of the Story.
When engineers compare GPU clouds, the conversation usually starts with three familiar numbers: GPU model VRAM Hourly price Those numbers matter. Bu…
KV Cache Quantization: I Stretched Qwen 35B's Context 8 on 12GB VRAM
600 MiB of headroom My RTX 4070 was running Qwen 35B beautifully after the --cpu-moe trick from a previous run. The tokens/sec were where I wanted t…
local-llm: A Field Report on Running SOTA Models on Your Own Hardware
The most useful thing in jamesob/local-llm is not the GPU shopping list. It is the fifteen or so BIOS settings, kernel flags, and PCIe hacks that st…
GPUs keep falling off the PCIe bus, and standard node health does not notice
AWS engineers have published a field report from running Kubernetes at Amazon EKS scale with GPU workloads, and the recurring failure they call out…
The Same RTX 5090, but the GPU Sat Idle — a CPU-Bound Go Solver and the Case for L2 Cache
This is the second in a short series that benchmarks a single RTX 5090 by re-running published Go solvers — programs that don't just play Go but pro…
One RTX 5090 vs a 12-GPU Cluster — Benchmarking a Decade of GPUs on the Same Go Proof
You don't need to know anything about Go to read this. The game is just the fixed yardstick. The story is a hardware benchmark: the same program, th…
WebGPU Explained: The Browser’s New Graphics and Compute Engine
A practical introduction to WebGPU, WGSL, render pipelines, compute shaders, and the future of high-performance graphics on the web. Your browser ca…
GPUs for AI in 2026: NVIDIA, AMD, Intel Compared
The AI hardware landscape has shifted significantly in 2026, with NVIDIA, AMD, and Intel all competing for developers who need GPUs capable of runni…
Linux 7.2 Improves Multi-GPU Displays, M3 Support, Mesa Rusticl Defaults Arm Mali
Linux 7.2 Improves Multi-GPU Displays, M3 Support, Mesa Rusticl Defaults Arm Mali Today's Highlights This week's hardware and driver news highlights…
Does a Second GPU Increase Ollama's Context Window? (Quadro P2000 + RTX 3090 Tested)
TL;DR Short version: no. I dropped a much older GPU ( Quadro P2000, 5GB, Pascal, 2016 ) next to an RTX 3090 (24GB, Ampere) on the same box, ran the…
CPU vs GPU: Why Large Language Models Need GPUs — What Really Happens After You Press Enter?
The moment you press Enter, billions of mathematical operations begin. Let's follow that journey. Every day, millions of people ask ChatGPT, Gemini,…
DGX Spark hitting 83 C under sustained Ollama load — solved by clock-locking via nvidia-smi -lgc
DGX Spark hitting 83°C under sustained Ollama load — solved by clock-locking via nvidia-smi -lgc TL;DR: GB10 in the DGX Spark has no user-exposed po…
Resurrecting Kepler: Getting Modern LLMs Running on a GTX 770 (Kernel 7.x)
⚠️ Experimental hack : Use on non-critical systems. Ensure you have backups. This patches a proprietary binary at the instruction level — no warrant…
96% of cuBLAS, no `unsafe`: what cuTile Rust proves
GPU programming usually asks Rust developers to surrender the borrow checker at the launch boundary: references collapse into raw pointers, and alia…
Why stop gaming saved my tokens: Building my own local AI Lab
About a year ago, I turned my gaming PC into a local AI Lab. And yes, the most important word in that sentence is LOCAL . Let me tell you the story…
Blackwell MLPerf Dominance, Intel Nova Lake Compute Runtime, & Weston 16 Vulkan HDR
Blackwell MLPerf Dominance, Intel Nova Lake Compute Runtime, & Weston 16 Vulkan HDR Today's Highlights NVIDIA's Blackwell architecture showcased…
Why You Need to Become a Neuro-Punk Right Now
A short essay on why the developer community should invest as much effort as possible into LLMs that are free from corporations and states. ML resea…
nvidia-smi Reports 97% Utilization While the GPU Sits Idle
TL;DR A GPU shows 97% utilization in nvidia-smi , but training throughput is a fraction of what benchmarks promise. The GPU is not computing; it is…