Charles Xu

3.3x Faster HuggingFace Tokenizers for Single Sequence

Tue, 25 Nov 2025 00:00:00 +0000

I made HuggingFace tokenizers 3.3x faster by parallelizing single-input tokenization with overlapping chunks, zero-copy offset operations, SIMD-accelerated boundary detection, and cache-hierarchy-aware chunking. The result is bit-identical to serial encoding. Fast tokenization is critical to achieve low TTFT given long context.

The Long Context Revolution Has a Bottleneck

LLM models now routinely handle 1M+ token contexts. This isn’t just a bigger number; it’s enabling fundamentally new use cases:

AI Agents: Load entire codebases into context instead of fumbling with RAG retrieval. Why search when you can reason over everything?
Document Analysis: Process entire legal contracts, medical records, or research papers in one shot—no chunking, no context loss.
Displacing RAG: Google’s “infinite context” approach suggests a future where retrieval augmentation becomes unnecessary.

But there’s a bottleneck nobody talks about: tokenization.

Everyone optimizes inference—quantization, KV cache, flash attention, speculative decoding, etc. But it takes 1.1 seconds to tokenize a 4MB document (about 1M tokens).

As AI agents scale with tool use and long context with multi-turns, the tokenization latency hurts TTFT (Time to First Token) and time to completion.

The challenge: Tokenization is inherently sequential. You process one token, then find the next, then the next. How to parallelize this without breaking correctness?

The Core Insight: Overlapping Chunks

The breakthrough is simple: we can chunk, parallelize, and merge.

The naive approach of splitting text and tokenizing in parallel doesn’t work, because you’ll get different results than serial encoding given tokens can span split boundaries. My solution uses overlapping chunks with a deterministic merge algorithm detailed in Appendix A.

Key Optimizations

Optimization 1: Recursive Divide-and-Conquer Strategy

We use binary splitting with rayon::join for work-stealing parallelism:

fn encode_recursive(input: &str, depth: usize, max_depth: usize) -> Encoding { if depth >= max_depth { return encode_serial(input); // Base case } let mid = find_split_point(input, input.len() / 2); let (left_chunk, right_chunk) = create_overlapping_chunks(input, mid); // Parallel execution via rayon let (left_encoding, right_encoding) = rayon::join( || encode_recursive(left_chunk, depth + 1, max_depth), || encode_recursive(right_chunk, depth + 1, max_depth), ); merge_at_midpoint(left_encoding, right_encoding, mid) } 

Auto-tuning depth based on input size and CPU cores:

100KB input → depth 2 (4 parallel chunks)
500KB input → depth 3 (8 parallel chunks)
1MB input → depth 4 (16 parallel chunks)

Why binary splitting? Clean merge semantics and work-stealing efficiency. Rayon automatically balances work across available cores.

Results: 1.7-2.1x speedup for 100KB-500KB inputs.

Sweet spot: Medium-sized documents like research papers, API documentation, or individual source files.

Optimization 2: Cache-Block Streaming for Large Inputs

At 1MB+, recursive splitting starts causing cache thrashing. Different threads access scattered memory regions, evicting each other’s data from cache.

The solution: Process input in L1-cache-sized blocks sequentially:

Input (1MB): ├───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬──────┤ │ 0 │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │10 │11 │12 │ ... │ │8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│8KB│ │ └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴──────┘ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ Encode all blocks in parallel, merge sequentially 

Why 8KB blocks? We benchmarked extensively:

Block Size	Time (4MB)	Throughput	vs 64KB
4KB	386ms	10.4 MiB/s	+31% faster ⭐
8KB	382ms	10.5 MiB/s	+32% faster ⭐
16KB	421ms	9.5 MiB/s	+20% faster
32KB	495ms	8.1 MiB/s	+2% faster
64KB	505ms	7.9 MiB/s	baseline

Smaller blocks are faster! Why?

L1 cache residency: 8KB fits entirely in L1 cache (typically 32-64KB) with room for vocabulary lookups
Maximum parallelism: 1MB = 125 independent work items vs. 16 with 64KB blocks
Sequential memory access: Each block is processed sequentially, so the CPU prefetcher thrives
Less cache eviction: Vocabulary data stays hot in cache across small blocks

Cache hierarchy matters:

8KB blocks: [L1: block + vocab] → process → next block Full L1 residency, maximum parallelism 64KB blocks: [L2 cache: block] → process → next block L2 latency, fewer parallel blocks, cache thrashing 

Results: 2.2-3.2x speedup at 100KB-1MB, 3.3x at 4MB.

Auto-selection: The system automatically picks streaming mode for inputs ≥1MB.

Optimization 3: LazyEncoding: Zero-Copy Offset

Here’s a subtle but critical problem: after tokenizing each chunk, we need to:

Shift all token offsets to global coordinates (O(n))
Filter tokens by position (O(n))
Merge arrays (O(n))

At depth 4 recursion (16 chunks), this would mean ~30 O(n) passes through the data. That’s a lot of wasted work.

The solution: Defer everything until final materialization.

struct LazyEncoding { encoding: Encoding, base_offset: usize, // Offset to add (applied lazily) start_idx: usize, // Virtual range start end_idx: usize, // Virtual range end } impl LazyEncoding { // O(1) - just update metadata fn shift_offset(&mut self, delta: usize) { self.base_offset += delta; } // O(log n) - binary search instead of linear scan fn filter_starting_before(&mut self, position: usize) { let cut_idx = binary_search_offsets(position); self.end_idx = self.start_idx + cut_idx; } // O(n) - but only called ONCE at the very end fn materialize(self) -> Encoding { // Apply base_offset and extract range in one pass apply_and_extract(self.encoding, self.base_offset, self.start_idx, self.end_idx) } } 

Impact: Turns O(n × depth) into O(n) total complexity.

Benchmark evidence (500KB input):

Depth	Without LazyEncoding	With LazyEncoding	Improvement
1	93ms	93ms	baseline
2	87ms	66ms	+32% faster
3	84ms	59ms	+42% faster
4	82ms	58ms	+60% faster

At depth 4, LazyEncoding is 60% faster than naive offset manipulation.

Optimization 4: SIMD-Accelerated Smart Boundaries

We need to split text at “safe” positions where tokenization is predictable. Naive approach: scan byte-by-byte for whitespace. Slow.

Better solution: Use the memchr crate for SIMD-accelerated searches (AVX2/SSE2):

use memchr::memchr_iter; // 10-20x faster than byte-by-byte scanning for pos in memchr_iter(b'\n', text_bytes) { if is_paragraph_break(pos) { return SplitPoint { position: pos, is_safe: true }; } } 

Smart boundary detection (priority order):

Priority	Boundary Type	Pattern	Safety Level	Overlap Size
1	Paragraph breaks	`\n\n`	Guaranteed token boundary	100-500 bytes
2	Sentence endings	`.` , `!` , `?`	Safe boundary	100-500 bytes
3	Any whitespace	`\n`, `\t`	Fallback	500-5000 bytes

Why “safe” boundaries matter:

At a paragraph break or sentence ending, we know tokenization will be consistent. The overlap only needs to handle the longest possible token (~100 bytes for most vocabularies).

At an arbitrary whitespace, we need more overlap to handle context-dependent tokenization (e.g., “ the” vs “the”).

Impact:

SIMD search: 10-20x faster than byte-by-byte
Smart overlap: 80-90% reduction in redundant tokenization
- Regular boundaries: ~5-15% redundant work
- Safe boundaries: ~1-2% redundant work

Example (depth 4, 16 chunks):

Regular overlap: 16 chunks × 5KB overlap = 80KB redundant tokenization
Safe boundaries: 16 chunks × 500 bytes = 8KB redundant tokenization
10x less wasted work

The Benchmark Story

Headline Numbers

Input Size	Serial	Best Parallel	Speedup
100KB	23ms	11ms	2.2x
500KB	122ms	39ms	3.1x
1MB	264ms	87ms	3.0x
4MB	1.10s	335ms	3.3x

Test environment: M-series Mac, release build with criterion benchmarks, BERT tokenizer.

Deliberate Trade-off: Word IDs

We intentionally do not compute word IDs in parallel mode. Here’s why:

Word IDs are rarely needed: Most use cases (LLM inference, embeddings) don’t need them
Computing them would reduce speedup: Would require additional O(n) work or complex heuristics
Clear fallback: If you need word IDs, use serial encoding: tokenizer.encode()

Result: 99% of users get 3x speedup. The 1% who need word IDs can use serial mode.

How to Use It

Rust API

use tokenizers::Tokenizer; // Load your tokenizer let tokenizer = Tokenizer::from_file("tokenizer.json")?; // Option 1: Auto mode (recommended) - picks best strategy let encoding = tokenizer.encode_parallel_single(&text, false)?; // Option 2: Force streaming for huge inputs let encoding = tokenizer.encode_streaming(&text, false)?; // Option 3: Custom configuration use tokenizers::tokenizer::parallel_encode::{ParallelConfig, ParallelMode}; let config = ParallelConfig::streaming() .with_block_size(8 * 1024) // 8KB blocks (optimal) .with_threshold(50_000); // Minimum size for parallelism let encoding = tokenizer.encode_parallel_with_config(&text, false, config)?; 

Python API

from tokenizers import Tokenizer # Load your tokenizer tokenizer = Tokenizer.from_file("tokenizer.json") # Option 1: Auto mode - just works long_text = open("large_document.txt").read() encoding = tokenizer.encode_parallel(long_text) # Option 2: Streaming mode for multi-MB inputs huge_text = open("massive_document.txt").read() # 5MB+ encoding = tokenizer.encode_streaming(huge_text) # Access results (same as regular encode) print(encoding.ids) print(encoding.tokens) print(encoding.offsets) 

Try It Yourself

The code lives on the parallel branch on github.com/cxuu/tokenizers.

# Clone the repository git clone https://github.com/cxuu/tokenizers.git cd tokenizers git checkout parallel # Run the benchmarks cargo bench --bench parallel_single_benchmark # Run the tests cargo test parallel # Try the Python bindings cd bindings/python pip install maturin maturin develop --release python -c " from tokenizers import Tokenizer tokenizer = Tokenizer.from_pretrained('bert-base-uncased') text = 'Hello world! ' * 10000 encoding = tokenizer.encode_parallel(text) print(f'Encoded {len(encoding.ids)} tokens') " 

Feedback welcome: This is ready for community testing. Try it on your workloads and let me know how it performs!

Appendix A: The Algorithm: Split, Encode, Filter, Merge

Let’s walk through a concrete example:

Original Input (100 bytes): "The quick brown fox jumps over the lazy dog. The dog was very lazy indeed." ├─────────────────────────────────────────────────────────────────────────┤ 0 50 100 

Step 1: Split with Overlap

We split at position 50, but encode overlapping regions:

Left chunk (bytes 0-60): "The quick brown fox jumps over the lazy dog. The dog wa" Right chunk (bytes 40-100): "lazy dog. The dog was very lazy indeed." └─────┘ 20-byte overlap (bytes 40-60) Why overlap? The midpoint might fall in the middle of a token! 

Step 2: Encode Both Chunks in Parallel

Left encoding: ┌─────┬───────┬───────┬─────┬───────┬──────┬─────┬──────┬─────┬─────┬─────┬─────┬─────┬─────┐ │ The │ quick │ brown │ fox │ jumps │ over │ the │ lazy │ dog │ . │ The │ dog │ was │ wa │ └─────┴───────┴───────┴─────┴───────┴──────┴─────┴──────┴─────┴─────┴─────┴─────┴─────┴─────┘ 0 4 10 16 20 26 31 35 40 44 46 50 54 58 Right encoding (offsets relative to byte 40): ┌──────┬─────┬─────┬─────┬─────┬─────┬──────┬──────┬───────┬─────┐ │ lazy │ dog │ . │ The │ dog │ was │ very │ lazy │ indeed│ . │ └──────┴─────┴─────┴─────┴─────┴─────┴──────┴──────┴───────┴─────┘ 0 5 9 11 15 19 23 28 33 39 (relative offsets - will shift to global) 

Step 3: Shift Right Encoding to Global Coordinates

Right encoding after shifting by 40: ┌──────┬─────┬─────┬─────┬─────┬─────┬──────┬──────┬───────┬─────┐ │ lazy │ dog │ . │ The │ dog │ was │ very │ lazy │ indeed│ . │ └──────┴─────┴─────┴─────┴─────┴─────┴──────┴──────┴───────┴─────┘ 40 45 49 51 55 59 63 68 73 79 

Step 4: Filter at Midpoint

Here’s the key insight: we keep tokens that “belong” to each side based on where they start.

Filtering rule: - From LEFT: Keep tokens where start_offset < midpoint (50) - From RIGHT: Keep tokens where start_offset >= midpoint (50) Left after filtering (keep start < 50): ┌─────┬───────┬───────┬─────┬───────┬──────┬─────┬──────┬─────┬─────┬─────┐ │ The │ quick │ brown │ fox │ jumps │ over │ the │ lazy │ dog │ . │ The │ └─────┴───────┴───────┴─────┴───────┴──────┴─────┴──────┴─────┴─────┴─────┘ 0 4 10 16 20 26 31 35 40 44 46 ✓ All < 50 Right after filtering (keep start >= 50): ┌─────┬─────┬──────┬──────┬───────┬─────┐ │ dog │ was │ very │ lazy │ indeed│ . │ └─────┴─────┴──────┴──────┴───────┴─────┘ 55 59 63 68 73 79 ✓ All >= 50 

Step 5: Concatenate

Final result: ┌─────┬───────┬───────┬─────┬───────┬──────┬─────┬──────┬─────┬─────┬─────┬─────┬─────┬──────┬──────┬───────┬─────┐ │ The │ quick │ brown │ fox │ jumps │ over │ the │ lazy │ dog │ . │ The │ dog │ was │ very │ lazy │ indeed│ . │ └─────┴───────┴───────┴─────┴───────┴──────┴─────┴──────┴─────┴─────┴─────┴─────┴─────┴──────┴──────┴───────┴─────┘ 0 4 10 16 20 26 31 35 40 44 46 50 54 59 64 69 75 This is IDENTICAL to what serial encoding would produce! 

Correctness Proof

Theorem: The parallel encoding with overlapping chunks produces identical results to serial encoding.

Proof:

Let’s define our problem formally:

Input string S of length n
Serial tokenization function T(s) that produces a sequence of tokens with start/end offsets
Midpoint position m where we split
Overlap size ω (omega)

Claim 1: For any input substring, tokenization is deterministic.

1 ∀ substring s: T(s) always produces the same token sequence

This is true by definition—tokenizers are deterministic state machines.

Claim 2: The overlap is sufficient.

We choose overlap ω ≥ max_token_length to ensure:

For split at position m: - Left chunk: S[0 : m + ω] - Right chunk: S[m - ω : n] - Overlap region: S[m - ω : m + ω] 

Any token that spans position m must:

Start at some position s where m - max_token_length < s < m + max_token_length
End at some position e where m - max_token_length < e < m + max_token_length

Since ω ≥ max_token_length, both boundaries are captured by our overlap.

Claim 3: The filtering rule is correct.

Define:

L = T(S[0 : m + ω]) - tokens from left chunk
R = T(S[m - ω : n]) - tokens from right chunk (shifted by m - ω)
L_filtered = tokens from L where start_offset < m
R_filtered = tokens from R where start_offset ≥ m

We need to prove: L_filtered ⊕ R_filtered = T(S) (where ⊕ is concatenation)

Proof by construction:

Consider any token t in the serial encoding T(S) with start offset s:

Case 1: s < m (token starts before midpoint)

The token is completely determined by S[s : s + len(t)]. Since the left chunk includes S[0 : m + ω] and s < m, we have s + len(t) < m + max_token_length ≤ m + ω. Therefore, S[s : s + len(t)] ⊂ S[0 : m + ω], so the token appears in L. Since s < m, it passes the filter and appears in L_filtered. ✓

Case 2: s ≥ m (token starts at or after midpoint)

The token is completely determined by S[s : s + len(t)]. Since the right chunk includes S[m - ω : n] and s ≥ m, we have s ≥ m > m - ω. Therefore, S[s : s + len(t)] ⊂ S[m - ω : n], so the token appears in R (after shifting by m - ω). Since s ≥ m (after shifting to global coordinates), it passes the filter and appears in R_filtered. ✓

Case 3: No token appears in both filters

Suppose a token t appears in both L_filtered and R_filtered. Then:

From L_filtered: start(t) < m
From R_filtered: start(t) ≥ m

This is a contradiction. Therefore, no duplicate tokens. ✓

Conclusion: Every token from T(S) appears exactly once in L_filtered ⊕ R_filtered, in the correct order with correct offsets. QED.

Why This Actually Works in Practice

The theoretical proof is nice, but here’s the practical insight:

Tokenization has no long-range dependencies. A BPE tokenizer processes text with a sliding window of at most max_token_length bytes. By overlapping more than this maximum, we ensure that:

Every token appears in at least one chunk: No token is “cut off” by the split
Boundary tokens appear in both chunks: The overlap captures them
Filtering is deterministic: We keep each token exactly once based on where it starts
Order is preserved: Left tokens come before right tokens by construction

Concrete example of a boundary token:

Input: "...lazy dog. The dog..." ↑ Split here (position 50) Left chunk: "...lazy dog. The" → produces tokens [..., "lazy", " dog", ".", " The"] Right chunk: " dog. The dog..." → produces tokens [" dog", ".", " The", " dog", ...] Token " dog" (starting at position 45) appears in BOTH encodings. Filtering: - In left encoding: start=45 < 50 → KEEP ✓ - In right encoding: start=45 < 50 → DISCARD ✗ Result: Token appears exactly once in final output. 

Handling Edge Cases

Q: What if the split falls in the middle of a multi-byte UTF-8 character?

A: We adjust to the nearest character boundary before creating chunks. The find_split_point() function ensures splits only happen at valid UTF-8 boundaries.

fn find_char_boundary_forward(s: &str, pos: usize) -> usize { let mut pos = pos.min(s.len()); while pos < s.len() && !s.is_char_boundary(pos) { pos += 1; } pos } 

Q: What about tokenizers with special token handling?

A: Special tokens ([CLS], [SEP], etc.) are added via post-processing AFTER parallel encoding. The core encoding produces tokens without special tokens, then the post_process() function adds them identically to serial mode.

Q: How do we ensure overlap is large enough?

A: We query the vocabulary for the longest token:

fn max_token_byte_length(&self) -> usize { self.get_vocab() .keys() .map(|s| s.len()) .max() .unwrap_or(0) } 

For BERT: ~25 bytes For GPT-2: ~50 bytes For byte-level BPE: ~100 bytes

We use overlap = max_token_length × 3 + safety_margin to be conservative. At safe boundaries (sentence/paragraph ends), we reduce to max_token_length + safety_margin because tokenization is guaranteed to be consistent.

Recursive Application

The same merge logic applies recursively:

Depth 0: Full input (1MB) │ ├─ Split at 500KB │ Depth 1: ├─ Left (0-500KB) ├─ Right (500KB-1MB) │ │ │ │ │ ├─ Split at 250KB │ ├─ Split at 750KB │ │ │ │ Depth 2: │ ├─L1 (0-250KB) ├─R1 │ ├─L2 ├─R2 │ │ │ │ │ │ │ encode() encode() │ encode() encode() │ ↓ ↓ │ ↓ ↓ │ filter + merge │ filter + merge │ ↓ │ ↓ │ merged_L │ merged_R │ ↓ │ ↓ └────────────┴──────────────┴────────────┘ filter + merge ↓ Final encoding ✓ 

Each merge applies the same filter-and-concatenate logic. The correctness proof composes: if each level produces correct results, the final result is correct.

The challenge: Making this practical requires five key optimizations that reduce overhead and maximize cache efficiency.

Accelerate LLM Inference with Speculative Decoding

Tue, 11 Mar 2025 00:00:00 +0000

Many inference speedup techniques mirror the classic systems regime—such as caching, paging, tiling, pipelining, and speculative execution (e.g. branch prediction and cache prefetch). Speculative decoding, generalizing speculative execution to stochastic settings, produces several tokens in each forward pass, without changing the output distribution (model quality) or model parameters.

This post discusses two approaches of Speculative Decoding:

Speculative sampling with a draft model (2023 paper)
Multiple decoding heads with Tree Attention (2024 paper)

Speculative Sampling with a Draft Model

Speculative sampling, or rejection sampling, uses an approximation/draft model—smaller and faster than the model you want to accelerate—to generate $k$ tokens autoregressively and then uses the larger model to verify the $k$ tokens in one pass.

How to verify each speculative token

Let $M_p$ be the target model and $M_q$ be the approximation model.
Let $p(x)$ be the probability of $x$ under $M_p$ (a shorthand for $p(x_t | x_1, \ldots, x_{t-1})$).
Let $q(x)$ be the probability of $x$ under $M_q$.

Sample a token $x$ from $M_q$. Don’t wait for the big model to verify $x$. Continue to sample the next $k$ tokens from $M_q$.

Then, for each speculative token $x$:
If $p(x) \geq q(x)$, accept $x$.
If $p(x) < q(x)$, accept $x$ with probability $p(x) / q(x)$; if $x$ not accepted, sample again from an adjusted distribution of $M_p$, where $p’(x) = norm(max(0, p(x) - q(x)))$.

It is proven that $x$ sampled this way ensures $x \sim p(x)$.

Verify in one pass

The key insight is that the big model can verify the $k$ speculative tokens in one pass. The reason is that in the self-attention mechanism, the big model computes contextualized representations of all prefixes in parallel, i.e. the model outputs

\[p(x_1 | \text{prefix}), \quad p(x_2 | \text{prefix} + x_1), \quad \ldots, \quad p(x_k | \text{prefix} + x_1 + x_2 + \ldots + x_{k-1})\]

simultaneously.

Multiple Decoding Heads with Tree Attention

Choosing the right draft model is hard. What if we just reuse the same model?

Medusa proposes attaching multiple decoding heads to the same model. In the case of 2 heads, head 1 predicts the immediate next token, and head 2 predicts the second token after the prefix. Two heads output at the same time.

(image source)

Training multiple decoding heads

The standard model only has one decoding head tasked to predict the next token. Thus, to use multiple decoding heads, we need to train the extra heads. The training needs:

loss function
tree attention & adjusted positional encoding

Loss function

Use the cross-entropy loss between the prediction of extra heads and the ground truth. To quote the paper:

Given the ground truth token $y_{t+k+1}$ at position $t+k+1$, the loss for the $k$-th head is $L_k = - \log p_t^{(k)} (y_{t+k+1})$, where $p_t^{(k)} (y)$ denotes the probability of token $y$ predicted by the $k$-th head.

We also observe that $L_k$ is larger when $k$ is larger, which is reasonable since the prediction of the $k$-th head is more uncertain when $k$ is larger. Therefore, we can add a weight $\lambda_k$ to $L_k$ to balance the loss of different heads. The final loss is:
\[\mathcal{L}_{\text{MEDUSA-1}} = \sum_{k=1}^{K} -\lambda_k \log p_t^{(k)}(y_{t+k+1}).\]

Tree attention & adjusted positional encoding

Suppose the first decoding head predicts 2 candidates for the next token, and the second decoding head predicts 3 candidates for the next next token. We have a tree of $2 \times 3 = 6$ branches. The tree structure creates two challenges:

We need to adjust attention mask such that a token generated from a specific candidate path can only attend to previous tokens within that same path and should ignore other branches.
We need to adjust positional encoding because there are multiple candidates for the same position (as in depth in the tree).

The solution is Tree Attention shown below. Note that the attention mask exclusively permits attention flow from the current token back to its antecedent tokens.

(image source)

Parallelizing Multi-Head Attention

Fri, 28 Feb 2025 00:00:00 +0000

In the multi-head attention mechanism, why after reshaping the projection matrices for Q/K/V from 3 dimensions to 4, we need to transpose the tokens dimension with the heads dimension?

Using the canonical example code for Attention Heads below as an example, why do we need Q = self.W_q(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2)?

import torch import torch.nn as nn class MultiHeadSelfAttention(nn.Module): def __init__(self, embed_size, num_heads): super().__init__() self.num_heads = num_heads self.head_dim = embed_size // num_heads # Split embeddings across heads  # Linear layers for Q, K, V  self.W_q = nn.Linear(embed_size, embed_size) self.W_k = nn.Linear(embed_size, embed_size) self.W_v = nn.Linear(embed_size, embed_size) # Fully connected output layer, i.e. $W^O$  self.fc_out = nn.Linear(embed_size, embed_size) def forward(self, x): B, T, D = x.shape # Batch, Seq_len, Embedding_dim  Q = self.W_q(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2) K = self.W_k(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2) V = self.W_v(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2) # Compute attention scores (scaled dot-product attention)  scores = torch.matmul(Q, K.transpose(-1, -2)) / (self.head_dim ** 0.5) attn = torch.softmax(scores, dim=-1) # Normalize scores  output = torch.matmul(attn, V) # Apply attention to values  # Concatenate and project back to embedding size  output = output.transpose(1, 2).contiguous().reshape(B, T, D) return self.fc_out(output) # Example usage x = torch.randn(1, 10, 512) # Batch size 1, Sequence length 10, Embedding size 512 attention_layer = MultiHeadSelfAttention(embed_size=512, num_heads=8) output = attention_layer(x) print(output.shape) # Should be [1, 10, 512] 

Review Multi-head Attention

Instead of using a single attention head, transformers use multiple heads. Each attention head has its own set of projection matrices for Q, K, and V. Each head learns to focus on different types of relationships. For example:

One head might focus on long-range dependencies, such as linking a subject to its verb.
Another might focus on local context, such as detecting adjective-noun pairs.
Another might specialize in syntax or semantics.

Each head has independent learnable weight/projection matrices $W_i^Q, W_i^K, W_i^V$, where $i$ is the head index. Each weight matrix has shape $(D, D/h)$ where $D$ is the embedding dimension and $h$ is the number of heads.

For input embeddings $X$ of shape $(B, T, D)$, the head $i$ computes:

\[Q_i = X W_i^Q, \quad K_i = X W_i^K, \quad V_i = X W_i^V\]

The multi-head attention then concatenates the per-head attention outputs and linearly mixes them:

\[\text{MultiHeadAttention}(Q, K, V) = \text{Concat}\left(\text{head}_1, \text{head}_2, \ldots, \text{head}_h\right)W^O\]

Each head has output shape $(B, T, D/h)$. The concatenated output has shape $(B, T, D)$. $W^O$ is trainable and has shape $(D, D)$. $W^O$ mixes information across heads and refines the final representation before passing it to the next layer.

Reshaping for Parallelization

Let’s break down Q = self.W_q(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2):

self.W_q is a linear layer (nn.Linear(embed_size, embed_size)) that projects x into a new representation, specifically the query (Q) in multi-head attention. This linear transformation does not change the shape of x, which remains (B, T, D).

In .reshape(B, T, self.num_heads, self.head_dim), we split the embedding dimension D into self.num_heads and self.head_dim (where self.head_dim = D / self.num_heads). For example, if D = 512 and self.num_heads = 8, then self.head_dim = 512 / 8 = 64. The shape becomes: (B, T, 8, 64).

In .transpose(1, 2), we swap the sequence length dimension (T) and the number of heads dimension (num_heads). Note that index 1 is the second column. This changes the shape:

Before: (B, T, num_heads, head_dim)
After: (B, num_heads, T, head_dim)

This rearrangement makes attention more efficient, because matrix multiplications can now parallelize across heads (each head operates independently on different parts of the embedding). The reason is the following.

In the code scores = torch.matmul(Q, K.transpose(-1, -2)) / (self.head_dim ** 0.5), K.transpose(-1, -2) swaps the last two dimensions so that K has the shape K^T: (B, num_heads, head_dim, T).

Then, the matrix multiplication Q @ K^T is (B, num_heads, T, head_dim) @ (B, num_heads, head_dim, T), which results in (B, num_heads, T, T).

Thus, Q @ K^T becomes more efficient after transpose(1, 2) because:

Parallel Computation for Each Head: By keeping num_heads as the second dimension, each head’s computation happens independently but in parallel across the batch, using optimized GPU kernels.
Better Memory Access Patterns: GPUs are highly optimized for contiguous memory access. The transpose(1, 2) operation ensures that each head’s data is grouped together, improving cache efficiency during matrix multiplication.

Notes: The Lean Startup

Sun, 21 Jan 2024 00:00:00 +0000

Careful planning and execution work for general management but not for startups. Perfect execution is futile if you end up building something nobody wants (waste). The real progress for startups is not how many JIRA tickets we closed but how fast we gain validated learnings—what creates value for customers and their willingness to pay—while minimizing waste.

Systematically break down a business plan and test each part—define a clear hypothesis (predictions about what is supposed to happen) then A/B test to prove the predictions. Two leap-of-faith assumptions:

Value hypothesis: do customers find value
Growth hypothesis: how new customers discover the product

Test assumptions with MVP that targets early adopters, in the Build-Measure-Learn loop. Any work beyond what was required to learn is waste, no matter how important it seemed. If we do not know who the customer is, we do not know what quality is. Even a “low-quality” MVP can act in service of building a great high-quality product. Plan the Build-Measure-Learn loop in reverse order: decide what we need to learn, then figure out what we need to measure, then see what product we need to build to experiment. Launch MVP under a different brand name if you worry about branding risk. Commit to iteration: don’t give up hope due to bad news from MVPs, but experiment more, learn more, and maybe pivot.

Do not blindly copy successful startups’ techniques. Charging customers from day one works for Stripe but not Facebook. Low-quality early prototypes works for Facebook but not mission-critical industries. Always experiment to see what works in our unique circumstances.

Eventually a successful startup will compete with fast followers. A head start is rarely large enough to matter. The only way to win is to learn faster than everyone else.

Vanity metrics, such as gross number of customers, are not actionable. We cannot conclude whether metrics growth is due to

latest product development, or
driven by decisions the team had made and that current initiatives have no impact.

Use cohort-based metrics (e.g. among users who signed up in June, what percentage exhibits behaviors we want), and use A/B test to conclude causality. Measure team’s productivity in units of validated learning, not the production of new features.

Pivot: test a new fundamental hypothesis in business model, product road map, partnership, customer segments, engine of growth. The decision to pivot depends on data & intuition. Misguided decision to persevere is value destructive. Signs of time to pivot: the decreasing effectiveness of product experiments and the general feeling that product development should be more productive. Startup’s runway is the number of pivots it can still make. To extend runway, achieve the same amount of validated learning at lower cost in shorter time. Schedule in advance regular “pivot or persevere” meetings. In pivots, don’t throw out everything and start over. Repurpose what has been built and learned.

My personal thought:
The search for PMF is like gradiant descent, a combination of intuition and data. Gradiant descent is an optimization algorithm: start at a point (your best guess), then iteratively descent in the direction of “slope”. The process is almost mechanical, but you need to start with an intuitive guess. Pivot means to find a new starting point. You need to pivot when
1) the “slope” around the current point is flat in all directions. Nothing you do seems to improve the metrics, or
2) you are trapped in a local minimum (saturating early adoptors) and want to find a better local minimum (mainstream customers).

Once you have found success with early adopters, you want to sell to mainstream customers. Early adopter market is saturated quickly despite prior “up and to the right” results. Mainstream customers have different and more demanding requirements. This is a customer segment pivot. The actions we need to win mainstream customers is different from how we won early adopters. Pivot requires new MVP. Just as lean manufacturing uses just-in-time production to reduce in-process inventory, Lean Startups practice just-in-time scalability, conducting product experiments with small batch size. Imagine that the letters didn’t fit in the envelopes. With the large-batch approach, we wouldn’t find that out until nearly the end. With small batches, we’d know almost immediately. Smaller batch size (small diff in product code change) means shorter Build-Measure-Learn cycle and less WIP waste.

Sustainable growth: new customers come from the actions of past customers: word of mouth, product usage (wearing designer cloths), funded advertising, repeat purchase.

Sticky Engine of Growth: repeat usage; use customer retention rate to test growth hypothesis. Other metrics like activation rate and revenue per customer can test value hypothesis but has little impact on growth. If the rate of new customer acquisition exceeds the churn rate, the product will grow.

Viral Engine of Growth: focus on increasing the viral coefficient. Many viral products do not charge customers but advertisers, because viral products cannot afford to have any friction impede the process of signing customers up.

Paid Engine of Growth: advertising, outbound sales, foot traffic. Use LTV/CAC to test growth hypothesis. Over time, CAC is bid up by competition.

A startup can evaluate PMF by evaluating each Build-Measure-Learn iteration using innovation accounting. Every engine is tied to a set of customers and their habits, preferences, channels, and interconnections and thus eventually runs out of gas.

If the boss tends to split the difference, the best way to influence the boss is to take the most extreme position. Your opponants will do the same. Over time, everyone will take the most polarized positions. Don’t split the difference. Instead create a sandbox for innovation that will contain the impact but not methods of the new innovation. It works as follows:

Any team can create a true split-test experiment that affects only the sandboxed parts of the product or service (for a multipart product) or only certain customer segments or territories (for a new product).
One team must see the whole experiment through from end to end.
No experiment can run longer than a specified amount of time.
No experiment can affect more than a specified percentage of customers.
Every experiment has to be evaluated on the basis of a single standard report of five to ten actionable metrics.
Every team that works inside the sandbox use the same metrics to evaluate success.
Any team that creates an experiment must monitor the metrics and customer reactions (support calls, social media reaction, forum threads, etc.) while the experiment is in progress and abort it if something catastrophic happens.

If you like notes like this, check out my bookshelf.

Notes: Venture Deals

Fri, 19 Jan 2024 00:00:00 +0000

Before Fundraise: Allow minimum three to six months to raise money. Have a clean cut from last job to avoid IP disputes. Prepare data site (Certificate of Incorporation, Bylaws, board minutes, cap table, customer list, product roadmap, org chart, employment agreements, budgets, financial statements). Some VC deals fail to close because of one missing IP assignment agreement. If you want money, ask for advice. Develop relationships with VC before fundraising. Research & make the outreach personal. Find mentors, not fundraising advisors asking for a cut.

During Fundraise: Do not email a teaser, hoping to share more in meeting. Whatever you send a VC may be your last, so send the full yet concise pitch. The presentation is to communicate the same info in the executive summary but with more examples and visuals. Aim for 10 slides or fewer. Offer a prototype or demo that VC can interact. Demo is the best way to show your vision. Watch VC reactions during demo. Did their eyes light up? Do they understand the domain? Raise enough to get to the next milestone, plus buffer. Hire experienced startup lawyer with fee cap. Don’t let VC talk you out of your lawyer choice. After you’ve had a second meeting, ask what the process is going forward. Get multiple term sheets to create competition. If a VC passes, insist on feedback, which improves your next pitch.

Decision Maker: Have a lead investor representing the entire syndicate. If party round, set up a special-purpose limited partnership, not to chase down 75 signatures. At every VC, find out & talk to the decision makers. Reference check the VC: founders who went through hard times, like fired as CEO, learn how the VC handled tough situations. Add more to data site, but watch out for busywork that associates assigns, and the risk of leaking to a competing portco. Again, make sure decision makers are involved.

Option Pool: Typical size of early-stage company option pool is 10% to 20%. Smaller pools for later stage. “We have enough options to cover our needs. If we need to expand the pool before the next financing, we will provide full antidilution protection for you.” VCs want to minimize future dilution by enlarging the option pool up front. Founders should push back with an option budget that lists out futures hires until next financing and the option grant to land each hire.

Liquidation Preference: In early stage, it’s in the best interest of both VC and founder to have a simple liquidation preference and no participation. In future rounds, the terms are often inherited from the early stage terms. If the seed investor doesn’t invest in future rounds, his economics in many outcomes could be worse with participating and end up looking like the common holders (in terms of returns), since their preference amounts are so small.

Pay to Play: Investors must invest pro-ratably in future financings (paying) not to have their preferred stock converted to common (playing) or lose anti-dilution rights. Not a lifetime guarantee but an incentive to follow on, if other investors decide to invest in next round. Pay to play reduces liquidation preferences for the nonparticipating investors and ensures only committed investors have preferred stock. If VC pushes back, “Why? Are you not going to fund the company in the future if other investors agree to?” Avoid the pay-to-play scenario where VC has the right to force a recapitalization (e.g. financing at a $0 pre-money valuation) if fellow investors don’t play in the new round.

Founder Vesting: Negociate to treat vesting as a clawback with an 83(b) election. Single-trigger: accelerated vesting upon M&A. Double-trigger: accelerated vesting upon M&A and being fired. Balanced approach: double trigger with one-year acceleration.

Anti-Dilution: protect prior investors in a down round (equity issued at lower price). Flavors: weighted average (normal), or full ratchet (rare; reduce earlier round price to new price). Antidilution is often implemented as a price reduction for conversion to common. More exceptions in antidilution carve-outs, the better for founders.

Board: Be wary of observers. Question what values they bring. Often they are associates. Sometimes they disclose board topics to brag to their friends. Get a small board. Independent board members usually get stock options 0.25% to 0.5% vest over 2 to 4 years. Observers don’t get options. Instead of controlling the board, VC uses protective provisions (veto rights on certain actions). Next-round investors want protective provisions too, but founders should push for all Preferred voting as a single class, instead of each Series voting separately. VCs charge all expenses associated with board meetings to the company. Mandate frugality. Place a cap early on the percentage of directors who can be VCs (not independent). Preemptively offer observer rights to dethroned director, or establish an executive committee of the board that can meet without everyone else.

Drag Along: A compromise is to grant drag-along rights to the majority of the common stock, not the preferred. Preferred can convert some to common to force a majority at the cost of lowering the overall liquidation preference. IPO: preferred convert to common. Never give different automatic conversion terms for different series of preferred. Push for low threshold to conversion.

Redemption: Ensure dividends require board majority approval. Allow investors to sell shares back to the company for a guaranteed return. Never agree to “Adverse Change Redemption” because it is vague, punitive, and investors can act on arbitrary judgments.

No-shop: Do not agree to pay for legal fees until deal done. Avoid pre-financing contingence: 1. “Approval by Investors’ partnerships” means the term sheet has not been approved. 2. “Employment Agreements acceptable to investors”: review & negotiate full terms (e.g. what happens on termination?) before signing term sheet. Limit the no-shop period to 30 days (worst case 60 days), automatically canceled if VC terminates. Commitment should be bidirectional. You agree not to shop deals, VC agrees to close timely. Ask for exception for acquisitions. Frequently financings and acquisitions follow each other.

Registration Rights: Always offered to investors. Lawyers often make innocuous edits on this section. Unnecessary. Upon IPO when the rights apply, investment bankers will restructure the deal.

Right of First Refusal: Define “major investor”, only give such right to them and only if they play in subsequent rounds. Enforce stock sale also transfers obligations the original owner signed up for.

Co-sale Agreement: This right says if a founder sells shares, investors can sell too. Hard to remove this right, but founders should ask for a floor. Why should VC hold it up if a founder just sells a small amount to pay off mortgage?

SAFE: Some VCs consider valuation cap as a price ceiling to the next round, so do not disclose seed-round terms until you have negociated new price. Legal fee for priced equity round has dropped, no more than SAFE.

Zombie VC: VC who past their investment period (usually 5 years) and did not raise a new fund. They can’t invest but still meet you. Waste time. Ask “when was your last investment” (more than a year = zombie). “How many more investments will you make out of the current fund?”

Reserve: Fund approaching end of life creates pressure for liquidity. Underreserved VC may resist new financings, limit round size, or push for sale of company to limit dilution, even when more funding is right for commons. Pay-to-play creates more resistance in this case. Follow-on might come from new fund, or a different fund vehicle (“Opportunity” funds)

Corporate VC: They look for more control, such as right of first refusal on acquisition, which you should never give.

Negociation: Goals: achieving a good and fair result, not killing your personal relationship. Preparation is key. Focus on valuation, option pool, liquidation preferences, board, and voting controls. Know what concetions you are ok with and when to walk away. Get to know the other side ahead of time, play to their strengths, weaknesses, biases, curiosities, and insecurities. First-time founder has one advantage over seasoned VC: time. They got family, LPs, portcos to deal with. You got one company and this negocitation. Ask what the 3 most important terms are for VC. Explain yours too. Call them out if they pound hard on minor points. Don’t make threat you can’t deliver. Don’t say who else you are pitching to. Never provide term sheet from other VC. Don’t address deal points in order but focus on whole picture. “That’s the way it is because it’s market.” probe on why the market condition applies to you. Talk to other founders to get market intelligence. Push back with “Wait a minute, this term creates incentive misalignment. Let’s avoid a divisive relationship.” If stuck with terms you don’t love, next-round investors may fix them because they want your team happy and motivated. After big wins and some time together, renegociate with existing investors for founder-friend terms.

Price: High valuation is risky because 1) VCs hold out for a higher exit (by big perf stack, or forbid sales below $X), then founders can’t sell at a price they would have been happy with. 2) At higher price, sophisticated investors demand more structure, resulting in significant outcome misalignment between early and late stage investors.

Investment Banks: Avoid them at early stage. Hire them in acquisition. They maximize exit value. Best source of bankers: your board members, investors, colleagues, and other senior executives you trust. Hire a banker who knows your sector, like “enterprise SaaS”.

Daily Operation: Hire an employment lawyer when a founder or exec leaves. Make sure equity & IP are settled to protect future fundraising and acquisition. If a company used a professional valuation firm, the valuation would be assumed to be correct unless the IRS could prove otherwise. File 83(b) election to start earlier the clock for long-term capital gains. Pay at least minimum-wage cash comp to full-time founders & execs.

If you like notes like this, check out my bookshelf.

Scaling Istio

Sun, 22 Oct 2023 00:00:00 +0000

In a large, busy cluster, how do you scale Istio to address Istio-proxy Container being OOM-Killed and Istiod crashes if too many connected istio-proxies?

Istio-proxy Container OOM-Killed

Problem

If istio-proxy dies, Pod disconnects from the world, because istio routes the Pod’s ingress and egress through the istio-proxy container. Thus, the main application container cannot communicate with other services, and clients cannot reach the application either. This disrupts existing connections and risks cascading failure when loads shift to other replicas.

Out-of-memory kill is #1 reason for the istio-proxy death. The istio-proxy is configured with resource limits for CPU and memory, to avoid starving other workloads sharing the k8s Node. The istio-proxy is killed once it exceeds the memory limit.

Restarting istio-proxy won’t help: By default, Kubernetes uses the restart policy “Always” for Pods. Thus, if the istio-proxy container is OOM killed, Kubernetes will restart it. However, because the usage pattern has not changed, istio-proxy will enter OOMKilled again. This forms a crash loop and continued disruption to applications.

To keep bumping the memory limit is expensive and whack-a-mole. Overtime, you have increased the memory limit from 256Mi to 2Gi, which is per istio-proxy container. Given tens of thousands of Pods in istio mesh cross the hundreds of clusters, it is expensive to keep raising the limit. Furthermore, many people only increase the limit is when the oncall got paged about crash-looping Pods, which already impact customer traffic.

Solution

Use `Sidecar` object to trim unused xDS config

By default, Istio programs all sidecar proxies with the configuration to reach every workload in the mesh, as well as accept traffic on all the ports associated with the workload.

But if you have a locked down Istio mesh, and if a tenant must request for allow-listing such source namespace using some onboarding config, then the istio-proxy container does not need the full mesh config.

The Sidecar API object can restrict the set of services that the proxy can reach. Adopting the Sidecar objects will reduce the number of xDS pushes and overall xDS config size. You could templatize the Sidecar objects and render them based on the per-namespace onboarding configs.

Below is an example Sidecar, which allows istio-proxies in the namespace “observability-cortex” to egress to four other namespaces.

apiVersion: networking.istio.io/v1beta1 kind: Sidecar metadata: name: default namespace: myapp spec: egress: - hosts: - "istio-system/*" - "my-upstream-ns/*" - "kube-system/*" - "observability/*" 

Use `Telemetry` object to reduce metrics generation

Istio collects and exports a wide range of Prometheus metrics. Metrics collection impacts memory usage. Istio-proxy doesn’t need to generate all metrics but only those we use. Consider customizing the metrics that Istio collects and exports.

--- apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: drop-unused-metrics-and-tags namespace: istio-system spec: # no selector specified, applies to all workloads in the namespace metrics: - providers: - name: prometheus overrides: - match: metric: ALL_METRICS tagOverrides: connection_security_policy: operation: REMOVE destination_app: operation: REMOVE destination_canonical_service: operation: REMOVE destination_canonical_revision: operation: REMOVE destination_principal: operation: REMOVE ... - match: metric: REQUEST_DURATION disabled: true - match: metric: REQUEST_SIZE disabled: true - match: metric: RESPONSE_SIZE disabled: true - match: metric: TCP_CLOSED_CONNECTIONS disabled: true 

Istio Ambient Mesh

We can solve sidecar problems if we don’t run sidecar at all. Istio ambient mesh is a sidecar-less approach to service mesh, replacing sidecar proxies with per-node and (not always necessary) per-namespace proxies. With fewer proxies, it will save us lots of money in CPU/Memory and provide shorter latency.

The general problems with sidecars and benefits of ambient mesh:

Kubernetes does not have first-class support for sidecars (until k8s 1.28). App container might start before proxy ready, decide itself is unhealthy, and be in a restart loop. Short-lived Pods (Job) need to explicitly kill proxy for Pod to complete.
Istio upgrade requires restarting every pod to inject newer-version Istio proxies
Sidecar resources are underutilized
Difficult to calculate namespace quotas (ResourceQuotas) because sidecars are transparent to tenants but consume namespace quotas.

If you use Calico to enforce L4 NetworkPolicy for Pods, you might face a blocker to adopting ambient mesh because of conflicting IPTables rules that Calico owned (GitHub issue still open). But I encourage you to do another proof of concept, because someone (GitHub issue) used eBPF instead of IPTables to redirect traffic to ambient-mode proxies, thus working around the conflicting Calico IPTables rules.

Istiod crash if too many connected istio-proxies

Problem

Istiod is the control plane of istio. All istio-proxies connect to istiod. Istiod may crash when there were too many connected istio-proxies, specifically if they all were added at the same time by a tenant workload scaling out.

Most people run Istiod as a Deployment with a HorizontalPodAutoscaler (HPA). You could mitigate the scaling issue by setting a high minimum for HPA, but doing so leads to low resource utilization at night and weekends, at odds with the very purpose of autoscaling. Moreover, istiod is still at risk when the tenants scale out aggressively.

Solution

Use `discoverySelectors` to watch in-mesh Namespaces only

The discoverySelectors configuration enables us to dynamically restrict the set of namespaces that are part of the mesh. The discoverySelectors configuration declares what Istio control plane watches and processes. Not all tenant namespaces enable istio, so istiod could benefit from having to process less k8s events.

Fine-tune HPA

The default scale-up stabilization window is 300 seconds. We should reduce it to 10 seconds to be more responsive, but keep the scale-down stabilization window at 300s to avoid threshing.

 apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: istiod namespace: istio-system labels: app: istiod release: istio istio.io/rev: system install.operator.istio.io/owning-resource: unknown operator.istio.io/component: "Pilot" spec: maxReplicas: 48 - minReplicas: 32 + minReplicas: 3  scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: istiod + behavior: + scaleUp: + stabilizationWindowSeconds: 10s # default is 300s + scaleDown: + stabilizationWindowSeconds: 300s  metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 65 

Distribute istio-proxy connections across Istiod Pods

Istio doesn’t explicitly set a default maximum connection time between istio-proxy sidecars and istiod. Typically, the connections from the sidecars to istiod are long-lived gRPC connections used for service discovery, configuration updates, and certificate rotation, and they are expected to be maintained as long as istiod and the sidecars are running. This creates uneven distribution of loads on istiod Pods over time.

One idea is to set a max connection idle timeout for the istio-proxy to istiod connections, so the proxy will reconnect over time, hopefully landing on a new istiod Pods.

apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: istio-proxy-to-istiod-timeouts namespace: istio-system spec: workloadSelector: labels: {} configPatches: - applyTo: HTTP_ROUTE match: context: SIDECAR_OUTBOUND routeConfiguration: vhost: name: istiod.istio-system.svc.cluster.local:443 patch: operation: MERGE value: typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager common_http_protocol_options: idle_timeout: 300s 

Work Around Max Count of Security Group Rules on EKS

Tue, 26 Sep 2023 00:00:00 +0000

AWS EKS on VPC networks need AWS Security Group Rules (SG) to receipt ingress traffic. But what if you reach the max rules count in your SG?

Background

LoadBalancer-type Service and Security Group Rules

Kubernetes users can expose a Service in two ways:

Register with the Istio ingress gateways—the golden path for most tenants
Create a dedicated LoadBalancer-type Service object, which tells the cloud provider to create a load balancer and set up health checks.

EKS recommends aws-load-balancer-controller to react to updates to LoadBalancer-type Service objects and set up NLB accordingly. For example, suppose a Service object exposes ports 80 and 443, the controller will create five Security Group (SG) Rules on EKS worker Nodes:

allow ingress source 0.0.0.0/0 to the corresponding NodePort for port 80
allow ingress source 0.0.0.0/0 to the corresponding NodePort for port 443
allow EKS zonal subnet in us-west-2a to ingress to the health-check NodePort.
allow EKS zonal subnet in us-west-2b to ingress to the health-check NodePort
allow EKS zonal subnet in us-west-2c to ingress to the health-check NodePort

Note: health-check will fail if a) the Node does not host any target Pods or b) none of the target Pods on this Node is ready, determined by the Pod’s readiness probe

The SG Rules are added to an SG attached to all worker Nodes in the given EKS.

Security Group Limits

For each AWS account, there are two quota limits on Security Groups:

Max number of inbound rules per SG
Max number of SGs per network interface

These limits can be adjusted subject to the constraint that the product of the two quotas cannot exceed 1000 (AWS doc). It means a network interface can not have more than 1000 SG rules.

Problem

Once your EKS cluster approaches the limit of SG rules, it restricts your ability to create new load balancers. It means you won’t be able to perform blue-green upgrade of the load balancer, because you need to provision two sets of load balancers simultaneously. The lack of headroom also means you can no longer onboard more applications that requires a dedicated load balancer.

Solutions

The following solutions are not mutually exclusive. They can be used together.

Second dedicated SG for each node pools

Suppose your current setup is that all worker Nodes, regardless node pool, has a shared SG attached named “worker”. The aws-load-balancer-controller adds new rules to the “worker” SG.

You can keep the shared “worker” SG to store common rules but create a new SG for each node pool, and use the new SG for NLBs ingress. You need to change the node pool launch template to attach the new SG.

If you decide to continue letting the AWS LB controller manage SG rules for us, you should tag the new SG with kubernetes.io/cluster/{{ .ClusterName }}: shared. This is necessary when there are multiple security groups attached to an ENI, so that the controller knows which SG to add new rules to. Because the existing “worker” SG has this tag already, we need to create a duplicate SG, say “worker2”, which does NOT have the SG tag for NLB. Then, we will attach to the node pool the “worker2” SG and the per-pool SG.

Optimize SG rules outside of aws LB controller

Recall the aws-load-balancer-controller implementation creates 5 inbound SG rules per envoy-ingress Service. We can optimize this by managing the SG rules ourselves and asking the controller to skip SG rules creation. We can reduce the need to 2 inbound SG rules per envoy-ingress Service.

Add the service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: false` annotation to the LoadBalancer-type Service object. Documentation about this annotation is here.

Reserve 3 static NodePorts for each Service. One for NLB to health check the EKS nodes. One for frontend port 80. One for frontend port 443. You can choose a static healthCheckNodePort if you set externalTrafficPolicy: Local (which comes with the benefits to preserve source IP address). The two regular NodePorts can be static regardless.

The two regular NodePorts should be consecutive, so one SG rule can cover both. The healthCheckNodePort does not need to be consecutive, because the source IP range in the SG rule is different (i.e. only allow NLB to healthcheck the nodes).

Consider the following example:

 apiVersion: v1 kind: Service metadata: annotations: external-dns.alpha.kubernetes.io/hostname: acmecorp.com service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-type: external + service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: false  name: myapp namespace: myapp spec: externalTrafficPolicy: Local + healthCheckNodePort: 30218  ports: - name: https + nodePort: 30212  port: 443 protocol: TCP targetPort: 8095 - name: http + nodePort: 30213  port: 80 protocol: TCP targetPort: 8089 selector: app: myapp type: LoadBalancer 

The optimized SG rules would be:

~~allow ingress source 0.0.0.0/0 to the corresponding NodePort for port 80~~
~~allow ingress source 0.0.0.0/0 to the corresponding NodePort for port 443~~
allow source 0.0.0.0/0 to ingress to NodePort range from 30212 to 30213
~~allow EKS zonal subnet in us-west-2a to ingress to the health-check NodePort~~
~~allow EKS zonal subnet in us-west-2b to ingress to the health-check NodePort~~
~~allow EKS zonal subnet in us-west-2c to ingress to the health-check NodePort~~
allow EKS VPC network in region us-west-2 to ingress to the health-check NodePort

Raise max inbound rules per SG by reducing SG count per ENI

The solution picks a different point on the trade-off spectrum between #Inbound rules per SG and #SG per ENI.

SG quota is set for each and whole AWS account, so any adjustment will affect other workloads in the same account. Thus, we need to verify whether the existing AWS account has ENI with max number of SGs attached already.

Build EKS clusters in a separate AWS account

Building new clusters and shifting tenants over are expensive. Try other solutions first.

Layer-4 Load Balancer & Zero-downtime Autoscaling and Upgrade

Sun, 06 Aug 2023 00:00:00 +0000

Your Kubernetes cluster probably has a shared ingress for north-south traffic, coming from a cloud load balancer and lands on your favorite proxies like Envoy, or Istio gateways, or Nginx.

If you

use a LoadBalancer-type Service to create a Layer-4 Load Balancer fronting your Kubernetes ingress
retain source IP address by setting externalTrafficPolicy: Local

Then horizontal autoscaling (scale-in) and rolling upgrade will incur some downtime for you.

This post

explains why there is partial disruption, and how much disruption to expect
discusses several options to achieve zero downtime upgrade and autoscaling

For simplicity, the rest of the doc assumes Envoy as the ingress gateway.

Background

Layer-4 cloud load balancer

The routing of traffic to Envoy is facilitated by a layer-4 (L4) cloud load balancer, known as Network Load Balancer (NLB) in AWS terminology. The aws-load-balancer-controller provisions such load balancer (LB) by watching LoadBalancer-type Service objects in Kubernetes. Each Service object opens dedicated NodePort on all Nodes in selected Envoy node pools. Traffic to Envoy will first be routed to NodePort on the Node hosting Envoy Pod, then DNAT-ed (iptables) to the Pod on the same Node, as shown in the following diagram.

(image source)

The LB periodically check the HealthCheck NodePort. The HealthCheck NodePort will fail if

the Node does not host any target Pods, or
none of the target Pods on this Node is ready, determined by the Pod’s readiness probe

externalTrafficPolicy: Local

Suppose the Kubernetes Service object is configured with externalTrafficPolicy: Local. Then, the kube-proxy directs packets exclusively to Envoy Pods residing on the same Node, even if there are other Nodes running Envoy. This setup has two benefits: one less hop (lower latency) and preserving source IP address (for allowlist or rate limiting).

But externalTrafficPolicy: Local is problematic during rolling upgrades or scale-in. The reason is that traffic arriving at NodePort will be dropped by kube-proxy if node has no ready Envoy Pods. LB will keep forwarding traffic to this Node until LB detects the HealthCheck NodePort is failing. Then, LB will mark the Node as unhealthy.

There is a certain delay between two key events in this setup:

An Envoy Pod becoming NotReady (for example, if it enters the “Terminating” state during a rolling upgrade).
The subsequent periodic health check carried out by the load balancer.

During such delay, client traffic to this Node is blackholed.

(image source)

Partial downtime during upgrade and autoscale-in

Why is there some downtime

As discussed in the previous section, client traffic to an Envoy Node is blackholed during the time between the envoy Pod on such Node enters the Terminating state and the LB performs the next health check. Kube-proxy will remove forwarding rules from NodePort to the Pod once the Pod enters the Terminating state. Kubernetes 1.24 and 1.25 considers the Terminating state as not ready.

For the same reason, horizontal scale-in will also cause downtime. For a while, I was just running Envoy as a DaemonSet on a node pool that does not autoscale.

Why is the downtime partial

This downtime only affects one Node at a time, because currently, Envoy DaemonSet has the following upgrade strategy:

 updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 0 

Thus, Kubernetes will terminate one Pod at a time, then create new Pod on the same Node. There are 6 Pods in each DaemonSet, so not all envoy Pods are down at the same time.

The reason for maxSurge: 0 is that envoy-ingress Pods run on host networking. It means we cannot have 2 envoy Pods running on the same Node, because they both bind to the same ports. Thus, the current update strategy is to kill a Pod, then start a new one.

Why host networking

Running Envoy in host networking means traffic bypasses the Pod overlay network (Normally, each Pod runs in separate network namespaces). Thus, host networking reduces the overhead of network hops and encapsulation due to overlays. This results in lower latency and higher throughput.

But how much performance gain exactly? It depends on many factors like hardware and bandwidth. Cilium did some benchmark—take this marketing with a grain of salt—that suggests host networking could improve throughput by 20% and latency by 25%. They didn’t say how many iptables rules (which scale linearly) are on the given hosts.

How much downtime

After NLB detects in its target group an unhealthy instance, NLB will stop creating new connections to that target. However, existing connections are not immediately terminated until a default 300s of draining timeout, or RST by clients or Envoy. Thus, in the worst case, the blackhole period per Pod is 310 seconds.

In practice, the startup time of a new Envoy Pod on the same Node will be shorter than 300s. NLB continues health-checking the unhealthy Node, and will mark the Node as healthy as gain once the new Pod is ready. But for the worst-case analysis, let’s assume the blackhole period per Node is 310 seconds.

Given 6 Nodes, Envoy DaemonSet will exhibit a 16.7% error rate for a total of 310 * 6 seconds, which is 1860 seconds, or 31 minutes in the worst case.

The 16.7% error rate comes from the fact that 1 of the 6 Pods are in the Terminating state. Still, 16.7% is an appropriation, because another downside of externalTrafficPolicy: Local is that connections may not distribute evenly, especially if there are long-running connections on the Terminating Pod. NLB does not support the least-connections load balancing scheme.

Solutions

Use Pod IPs as LB backends

In this case, NLB sends traffic directly to the Pods selected by k8s Service. The benefits are:

Eliminate the extra network hop (NodePort) through the worker Nodes
Allow NLB to keep sending traffic to Pods in the Terminating state but mark the target as Draining

AWS load balancer controller supports this feature natively with “NLB IP-mode”. On other cloud, you can implement such controller yourself, watching Pod events and reconcile with L4 LB target groups.

To enable IP-mode, we just need to update the Service annotations

-service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance +service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip  # Health check the Pods directly +service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http +service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "9901" +service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /ready  # NLB with IP targets by default does not pass the client source IP address, # unless we specifically configure the target group attributes. +service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true 

To achieve zero-downtime upgrade, we need to additionally configure on the envoy Pod a preStop hook like below. When Pod enters the Terminating state, k8s will execute the preStop hook and keep the Pod in Terminating until the preS`top hook completes.

# We must define a longer terminationGracePeriodSeconds, which by default # is 30s, upon which the Pod is killed even if preStop has not completed. terminationGracePeriodSeconds: 305 containers: - name: envoy lifecycle: preStop: exec: # The default target group attribute # “deregistration_delay.timeout_seconds” is 300s, configurable # through Service annotation. command: - /bin/sh - -c - curl -X POST http://localhost:9901/healthcheck/fail && sleep 300 

By failing the envoy health check but keeping envoy running in the Terminating state, envoy can still process traffic. Once NLB deems the Envoy Pod unhealthy, it halts new request routing to the Pod but maintains existing connections. Consequently, active TCP connections persist, with client requests continuing to the now-unhealthy NLB target (Envoy Pod) until either client or envoy closes the connection or idle timeout expiry, defaulting to 300 seconds for NLB.

ProxyTerminatingEndpoints

ProxyTerminatingEndpoints is a new beta feature in Kubernetes version 1.26. It is enabled by default.

When there is a rolling update and a Node only contains terminating Pods, kube-proxy will route traffic to the terminating Pods based on their readiness. At the same time, kube-proxy will actively fail the health check NodePort if there are only terminating Pods available. By doing so, kube-proxy alerts the external load balancer that new connections should not be sent to that Node but will gracefully handle requests for existing connections.

# We must define a longer terminationGracePeriodSeconds, which by default # is 30s, upon which the Pod is killed even if preStop has not completed. terminationGracePeriodSeconds: 305 containers: - name: envoy lifecycle: preStop: exec: # The default target group attribute # “deregistration_delay.timeout_seconds” is 300s, configurable # through Service annotation. command: - /bin/sh - -c - sleep 300 

Note that here we must NOT call POST http://localhost:9901/healthcheck/fail on Envoy, different from what NLB IP-mode needs. The reason is that Terminating Pods need to pass the readiness probe to continue receiving traffic, so we cannot fail the envoy health check. Since kube-proxy will actively fail the health check NodePort if there are only terminating Pods available on the Node, NLB will start the draining process.

Customize NLB, keep host networking

Forget about the NodePort and HealthCheck NodePort opened by kube-proxy. We can create the NLB not through k8s Service object, but using infra-as-code tools such as pulumi. This bypasses the kube-proxy. The NLB will look like this

NLB frontend port	Target port (=NodePort =Pod port because of host networking)
443	8443
80	8080

The NLB will find all Nodes running envoy Pods using the autoscaling group for the envoy-ingress node pool. Yes, we can autoscale with solution 4.3. This setup is similar to Section 4.1.1 NLB IP-mode, except the NLB is not created by Kubernetes. We need the following Pod spec change.

# We must define a longer terminationGracePeriodSeconds, which by default # is 30s, upon which the Pod is killed even if preStop has not completed. terminationGracePeriodSeconds: 305 containers: - name: envoy lifecycle: preStop: exec: # The default target group attribute # “deregistration_delay.timeout_seconds” is 300s, configurable # through Service annotation. command: - /bin/sh - -c - curl -X POST http://localhost:9901/healthcheck/fail && sleep 300 

We also need to expose the “/ready” endpoint from envoy to the host. Then, we need to update the Service annotations like the following.

# Health check the Pods directly through NodePort 9901 +service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http +service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "9901" +service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /ready 

Enterprise Sales

Sun, 01 Jan 2023 00:00:00 +0000

How to do product-led growth and hands-on outbound sales at the same time?

Every PLG company eventually has to embrace enterprise.

– Annie Pearl, Chef Product Officer at Calendly

The upper limit of PLG seems to be $100M to $200M ARR (e.g. DataDog around IPO). Beyond triple-digit million ARR, you quickly saturate the market of users who buy things by pulling out their credit cards. The growth of the PLG channel naturally slows at some point.

Most companies start with self-serve PLG and then layer in enterprise sales. This may not work well. Your entry-level product could cannibalize your enterprise product. Think hard about differentiation that justifies the premium. You cannot just add single-sign-on (SSO) and call it enterprise edition.

Some common enterprise features are:

Feature	Description
Customer supports	Support is what makes or breaks the enterprise product. Your company success is defined solely by customer success. 24/7 or business hours? Email, chat, or phone support? SLA on response time?
Dedicated account manager	Evangelize best practices and champion support.
Fine-grain authorization policy	Roles and permissions.
Audit logs	User and workload identity, actions, timestamp, easy export.
Compliance	For example, FedRAMP, HIPAA, GDPR, SOC2.
High availability	For example, multi-region disaster recovery, cross-region replication, four-nine uptime SLA.
Private network	Private link, VPC peering, single-tenant deployment.
Self-host	User-managed private deployment
Data retention period	For example, DataDog allows free accounts to view metrics from last 24h
Performance	For example, run the same pytorch model 100x faster.

Another pro tip: within minutes of self-serve signups, directly reach out to that new user, assuming they pass certain criteria about company size and title. The reason is, your product is top of mind for the user right now, who clearly has been doing research about this space. If they give you the phone number, call them directly.

When to layer in enterprise sales?

Similarly, when to make the first sales hire?

You need product-market fit (PMF) and annual recurring revenue (ARR). PMF is shown through retention, exponential and organic customer growth, and net promoter score (NPS, or how upset would you be if this product disappears? How likely would you recommend this product to your friends?). To learn more about PMF, I love this post by Lenny Rachitsky.

Rule of thumb: you need at least $1M ARR to show you can sell outside of your network (friends and family).

Should your first sales hire be head of sales or junior sales rep?

You probably don’t yet need a head of sales because

The target customers and the deal size remain uncertain, so it is hard to know if a candidate is the right person.
Head of sales is expensive and often commands a structure in marketing, finance, legal, etc. Big overhead.
Head of sales probably has not been down in the weeds for a while.

You should not hire anyone junior because it takes your time to train them, such as how to write emails and pricing proposal.

An ideal candidate would be an account executive (AE) at a hypergrowth company with great track record, or someone recently moved into front-line management who used to be an AE and who desires the opportunity to grow into the head of sales role.

Some revealing questions to ask:

Tell me about a deal that you lost.
Let’s do a mock discovery call together.
Out of the 8 SDRs on your team, where did you come up on the leaderboard? What are the folks front-running you doing differently?

When do you start outbound?

According to Maggie Hott, Director of Sales at Webflow, the answer is always now. The goal of cold outbound is not to close deals but to build brand awareness and educate potential customers. You almost never hit the prospects at the right time, but when they are ready to buy, prior outbounds make you part of the evaluation. Whereas marketing targets a market segment, outbounds are personalized to each prospect.

Maggie recommends a 10-80-10 outbound strategy:

First 10% is personalized and explains how the prospect could relate to your product.
- For example, “Hey John, amazing talk at KubeCon last week with such great lessons for multi-cloud adoption. I am curious how your team approaches multi-cloud observability and cost attribution. All the folks I talked to are struggling with this. I think our product can really help.”
Mid 80% is repeatable product marketing.
Last 10% is to ask for that call or meeting, and end with a playful closing.

Lauren Schwartz, VP of Sales at Fivetran, stressed the importance of having multiple sponsors in the prospect’s organization. People change jobs and have different types of influence.

Should we give discounts for testimonials?

In the early days, worry less about the contract size but focus on getting those logos. The bulk of the enterprise market is the early majority and late majority. They are almost never early adopters. They need testimony and success stories.

Maybe you could get more testimonials through discounts, but be judicious about discounts, because people talk, and soon more customers will ask for discounts. The best reason to give discounts is to control the close date and payment structure of the deal. You can always bring up the ask for testimonials if the prospect comes back asking for more discounts.

More Career Advices

Tue, 06 Dec 2022 00:00:00 +0000

Make sure to check out the previous post: Advices I wish I got at the start of my career.

Ask for help

Son, your ego is writing checks your body can’t cash.

– Captain Tom “Stinger” Jordan, Top Gun

The number one reason why senior people fail is that they do not ask for help. We are all shareholders. You do whatever you can to unblock yourself. It is about time to market and showing results.

Promote thought leadership

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

Antoine de Saint-Exupery, author of The Little Prince

In tech, most decisions are based on influence, not hierarchy. Thought leadership is a great way to gain influence. To be a leader, you should start by acting like one. Be confident, give tech talks, and voice your opinions in planning, design review, and postmortems. Doing so makes you the obvious choice for the next big project or the next leadership roles.

The outcome and how you drove that outcome are both important. Be careful with your reputation.

Befriend Jeff Bezos before he gets rich

The best time to become friends with Jeff Bezos is before he becomes rich and famous. Networking does not mean you must reach upward. Invest in your peers, who are more receptive to getting to know you. Imagine you are Stripe in 2012. Rather than running into walls with F500 enterprises, you should onboard hundreds of startups, because among them are the next Airbnb and Lyft. Relationships compound. Start early.

Do informational interviews. Ask people at other firms what they like and don’t like about their job. Always follow up to maintain weak ties, such as

Saying hello to someone you met at a conference last year. Or asking if they’ll be attending after this year’s agenda is published.
Share interesting news about your old company with a former colleague.
Send them news, event, commentary related to their interests. Examples:
- “Tom, I just read this great white paper on blockchain. I know you’d get a lot out of this.”
- “Alice, congrats on the new job. Enclosed please find a copy of the best book I’ve read on starting a new job, The First 90 Days. Call me if you want to compare notes.”
- “James, I just got an invite to a private class at this new gym but can’t go. You mentioned you love Crossfit — want my ticket?”

Stop productivity porn. Bias towards action.

Watching others lifting weights is not going to make you fit. Many of us spend so much time collecting books we want to read but haven’t, or studying how others slice up their days to get more done. You get more done by doing and by starting now.

Develope relationship skills

A major reason change efforts so often fail is that successful implementation eventually requires people to have difficult conversations … With everyone taking for granted that their own view is right, and readily assuming that others’ opposition is self-interested, progress quickly grinds to a halt. Decisions are delayed, and when finally made they are often imposed without buy-in from those who have to implement them. Relationships sour. Eventually people give up in frustration, and those driving the effort get distracted by new challenges or the next next big thing. The ability to manage difficult conversations effectively is foundational, then, to achieving almost any significant change.

– Douglas Stone, Author of “Difficult Conversations: How to Discuss What Matters”

Relationship problems are at the heart of every organization. Take product managers (PM) and engineering managers (EM) for example. PM & EM have overlapping scopes by definition. When you seek better scoping, you don’t have a scoping problem, you have a relationship problem.

Here are some tips:

Be a good listener. People rarely change. People just want to be understood. Listening is more persuasive than talking. Listening fosters a reciprocal relationship. Listening is not just about paraphrasing back. Ask deep and relevant questions, take notes, and maintain eye contact. For many, listening needs to be a trained response. When you are frustrated, you are the least curious—you have so much noise in your internal head that left little space to worry about what’s on the other person’s head. Learn to lean into the conflicts, just like firefighters learn to run towards the fire.

Lean into conflicts. Avoiding conflict is the worst kind of measure of relationship health. Staying quiet creates resentment. You must confront, but do so with skills and preparation. It is mature to share your feeling and inquire about others’. Doing so builds trust. For example: “Bob, I feel frustrated. It seems this conversation is not getting anywhere, and I want to understand why.” Acknowledge the differences between the two parties, not who is right or better. I love this example “Jill, you and I seem to have different preferences about when code reviews should be done. I wonder if that’s something we could talk about?”

Resolve email conflicts in person. If a conflict starts on email, it is hard to solve on email. In this case, just meet in person or pick up the phone. It is hard to communicate emotions through email—no tone, no voice, no facial expression, no body language.

Apology diffuses the tension. In most conflicts, blaming does not help. Most of the time, you share part of the blame, even just 5%. Apologizing and acknowledging the fault on your side could really diffuse the tension. Apology is an underrated and underutilized skill. Apology needs to be genuine. Saying “I am sorry that you feel that way” is not genuine. Here is how to make a good apology:

Acknowledge the harm. “I am sorry that I interrupted you in the meeting.”
Say why it is wrong. “It was disrespectful and discourage the full exchange of ideas.”
Say what you will do next time, not what you won’t do. “I will make sure to let you finish before I chime in.”
Ask for forgiveness. Bring cupcakes.

Go straight to the job you want

Don’t let inertia drive you. Take some risks when you are young. No one in their 40s said they took too much risk.

If you are unhappy about your job, move on. Every job change you make, you always wish you make it 6 months earlier. Life is so short. Do not spend time on jobs that you do not like. You will be so productive in jobs you like. Don’t assume that you have to do this job, then get that job and then that job, and then you can do what you really want. Go direct.

Know your alternative. Negotiate hard on your second best offer, then negotiate with your first choice, knowing what you can walk away towards. Pay attention to details. Be specific about equity grant date, vesting schedule, etc.

What to look for in the next job: Growth

With the right opportunities, you can 10x your impact every decade. Because of compounding, what seems to be golden handcuffs today is dwarfed by the opportunity to accelerate growth. The exponential curve actually consists of many little S-curves. If you find yourself approaching the flattening end of the S-curve, it is time for a change.

Charles Xu

3.3x Faster HuggingFace Tokenizers for Single Sequence

The Long Context Revolution Has a Bottleneck

The Core Insight: Overlapping Chunks

Key Optimizations

Optimization 1: Recursive Divide-and-Conquer Strategy

Optimization 2: Cache-Block Streaming for Large Inputs

Optimization 3: LazyEncoding: Zero-Copy Offset

Optimization 4: SIMD-Accelerated Smart Boundaries

The Benchmark Story

Headline Numbers

Deliberate Trade-off: Word IDs

How to Use It

Rust API

Python API

Try It Yourself

Appendix A: The Algorithm: Split, Encode, Filter, Merge

Correctness Proof

Why This Actually Works in Practice

Handling Edge Cases

Recursive Application

Accelerate LLM Inference with Speculative Decoding

Speculative Sampling with a Draft Model

How to verify each speculative token

Verify in one pass

Multiple Decoding Heads with Tree Attention

Training multiple decoding heads

Loss function

Tree attention & adjusted positional encoding

Parallelizing Multi-Head Attention

Review Multi-head Attention

Reshaping for Parallelization

Notes: The Lean Startup

Notes: Venture Deals

Scaling Istio

Istio-proxy Container OOM-Killed

Problem

Solution

Use Sidecar object to trim unused xDS config

Use Telemetry object to reduce metrics generation

Istio Ambient Mesh

Istiod crash if too many connected istio-proxies

Problem

Solution

Use discoverySelectors to watch in-mesh Namespaces only

Fine-tune HPA

Distribute istio-proxy connections across Istiod Pods

Work Around Max Count of Security Group Rules on EKS

Background

LoadBalancer-type Service and Security Group Rules

Security Group Limits

Problem

Solutions

Second dedicated SG for each node pools

Optimize SG rules outside of aws LB controller

Raise max inbound rules per SG by reducing SG count per ENI

Build EKS clusters in a separate AWS account

Layer-4 Load Balancer & Zero-downtime Autoscaling and Upgrade

Background

Layer-4 cloud load balancer

externalTrafficPolicy: Local

Partial downtime during upgrade and autoscale-in

Why is there some downtime

Why is the downtime partial

Why host networking

How much downtime

Solutions

Use Pod IPs as LB backends

ProxyTerminatingEndpoints

Customize NLB, keep host networking

Enterprise Sales

How to do product-led growth and hands-on outbound sales at the same time?

When to layer in enterprise sales?

Should your first sales hire be head of sales or junior sales rep?

When do you start outbound?

Should we give discounts for testimonials?

More Career Advices

Ask for help

Promote thought leadership

Befriend Jeff Bezos before he gets rich

Use `Sidecar` object to trim unused xDS config

Use `Telemetry` object to reduce metrics generation

Use `discoverySelectors` to watch in-mesh Namespaces only