Popular
Trace a Token Through TP + CP + SP + EP + DP + PP in MoE Inference
Let’s trace one prompt token through prefill of a Mixture-of-Experts (MoE) model on 32 GPUs with six parallelisms: data (DP), context (CP), sequence (SP), tensor...
5.5x Faster HuggingFace Tokenizers for a Single Sequence
I made HuggingFace tokenizers encode a single long input on all cores. The Llama 3 tokenizer now encodes 4MB of text 4.9x faster, in 163ms...
Accelerate LLM Inference with Speculative Decoding
Many inference speedup techniques mirror the classic systems regime—such as caching, paging, tiling, pipelining, and speculative execution (e.g. branch prediction and cache prefetch). Speculative decoding,...
Parallelizing Multi-Head Attention
In the multi-head attention mechanism, why after reshaping the projection matrices for Q/K/V from 3 dimensions to 4, we need to transpose the tokens dimension...
Enterprise Sales
How to do product-led growth and hands-on outbound sales at the same time?
More Career Advice
Make sure to check out the previous post: Advice I wish I got at the start of my career.
Interviewing Adrien Treuille, Founder CEO of Streamlit
Streamlit, about to raise its Series-C, was acquired by Snowflake for $800M in March 2022. In this conversation with Adrien, we chatted about OSS metrics,...
Accounting Advice for Founders
Notes derived from a guest lecture by Danny Wallace, Partner at PwC’s Silicon Valley practice. For informational purposes only. Errors and omissions are my own....
Intellectual Property and Entrepreneurship
Notes on Intellectual Property (IP) law for founders and busy professionals. Not legal advice. For informational purposes only. Laws can change, so this article may...
Software Engineering Levels and Promotion
This post explains the expectation of each engineering level in the most concise and company-agnostic way and reveals the steps towards promotion.
What to Talk about in Effective 1-on-1s
Unlike in school when we get grades on every assignment and in every course, we get less frequent feedback in professional life, usually once or...
Advice I wish I got at the start of my career
When I was a kid playing chess with my dad, he sometimes would offer me hints on some good moves. I would never make those...
New Posts
Notes: The Lean Startup
Careful planning and execution work for general management but not for startups. Perfect execution is futile if you end up building something nobody wants (waste). The real progress for startups...
Notes: Venture Deals
Before Fundraise: Allow minimum three to six months to raise money. Have a clean cut from last job to avoid IP disputes. Prepare data site (Certificate of Incorporation, Bylaws, board...
Scaling Istio
In a large, busy cluster, how do you scale Istio to address Istio-proxy Container being OOM-Killed and Istiod crashes if too many connected istio-proxies?
Work Around Max Count of Security Group Rules on EKS
AWS EKS on VPC networks need AWS Security Group Rules (SG) to receipt ingress traffic. But what if you reach the max rules count in your SG?
Layer-4 Load Balancer & Zero-downtime Autoscaling and Upgrade
Your Kubernetes cluster probably has a shared ingress for north-south traffic, coming from a cloud load balancer and lands on your favorite proxies like Envoy, or Istio gateways, or Nginx....
Kubernetes Networking From the First Principles
We go from containers and network namespace to Pod-to-Pod, Pod-to-Service, and external-client-to-Service networking.
Never miss a story from me, subscribe to my newsletter