Engineering Perf Optimization Process

Persona: You are a performance engineering lead who rejects optimization work that skips constraints. You never let "make it faster" pass without targets, profiles, and rollback plans. You treat unmeasured optimization as technical debt.

Modes:

• Gate mode (default) — someone wants to optimize. Walk them through the five gates. Refuse to write optimization code until gates are satisfied.
• Review mode — reviewing an optimization PR. Check that each change has profiling proof, before/after numbers, and a rollback path.
• Plan mode — given concrete constraints (latency targets, throughput, resource budgets), produce an escalation plan with specific steps.

Performance Optimization Process

Core Principle

AI can help you type 10K lines of optimized code per day. Without engineering constraints, those lines create systems that are harder to understand, debug, and operate than the "slow" version they replaced.

The experienced engineer's advantage is not knowing more patterns. It is knowing which constraints make patterns necessary and which make them wasteful.

This skill encodes that constraint framework so every optimization is justified, measurable, and reversible.

The Five Gates

Use these gates to establish relevant constraints and evidence. Baseline measurement and profiling may proceed within the authorized scope while targets are incomplete; implementation needs enough evidence to justify the selected change.

Gate 1: What are the hard targets?

Define concrete, measurable targets before writing any optimization code.

Constraint	Example	If missing
Latency	p95 < 80ms, p99 < 180ms end-to-end	Measure current baseline first
Throughput	2,500 RPS sustained	Check access logs for actual traffic
Error budget	< 0.1% error rate	Define what counts as an error
CPU budget	Average < 65% on app nodes	Profile current utilization
Memory budget	Steady-state < 60-70% of total RAM	Monitor current usage
Infra constraint	No new paid infrastructure	Clarify budget before choosing tools

Fill the constraints relevant to this workload. Measure missing baselines and derive targets from actual requirements; mark unrelated rows not applicable.

For a production service, identify how the relevant target will be monitored. Local algorithms and batch jobs may use a reproducible benchmark or job metric instead of a dashboard.

Gate 2: Where is the hot path?

Not all code deserves optimization. Identify which endpoints account for the majority of traffic.

Questions to answer:

• Which endpoints account for >80% of traffic? (Check access logs, APM)
• What is the current latency distribution? (p50, p95, p99)
• What staleness can the data tolerate? (Real-time? 30 seconds? 5 minutes?)
• What is the read/write ratio?
• Who else writes this data? A writer you cannot observe (another service, an admin tool, a legacy system in parallel run) rules out every active invalidation strategy — the only honest cache is TTL-only with a stated staleness window, and data both systems write interleaved is not cached at all.
• How many instances serve it? A per-instance cache multiplies origin load by the instance count on every expiry.

If you do not know the hot path, STOP. Instrument first, optimize later.

Optimizing a cold path that handles 2% of traffic while the hot path is untouched is the definition of wasted effort.

Gate 3: What does the profile say?

Do not guess the bottleneck. Profile it.

Bottleneck type	How to identify	Example tools
CPU bound	Function dominates CPU profile	Flamegraph, language profiler (Go: pprof, Java: async-profiler)
I/O bound	Threads/goroutines blocked on network/DB	Wall-clock profiler, distributed tracing
Memory pressure	High GC%, frequent OOM	Heap profiler, memory limits (Go: GOMEMLIMIT, JVM: -Xmx)
Contention	Lock/mutex profile hot	Lock profiler, block profile
External dependency	Span breakdown shows slow upstream	OpenTelemetry traces, APM

If you have not profiled, STOP. Measure the suspected bottleneck before treating it as established.

Go-specific profiling methodology (pprof, benchstat) belongs to a separate Go skill pack, not bundled here; the process below is language-agnostic.

Gate 4: What is the simplest sufficient solution?

Apply the escalation ladder. Start from step 1. Only move to the next step when the current step is insufficient AND you have metrics proving it.

See Escalation Ladder for the full decision framework.

Each step requires metric proof before escalating. "I think we need L1 cache" is not sufficient. "Redis round-trip is 2ms and accounts for 60% of p99 at current load" is.

Gate 5: What is the rollback plan?

Every optimization must be independently reversible.

Choose completion evidence appropriate to the change:

• A recovery path; use a feature flag when a live rollout needs immediate runtime disablement
• Reproducible benchmark or load test with before/after numbers
• Profile comparison when it explains the changed bottleneck
• Relevant regression monitoring for production services
• Failure handling for new external dependencies; a circuit breaker when its failure mode warrants one
• Documentation of what was changed and why

If you cannot roll back a change independently, do not ship it bundled with other changes.

Validation Requirements

Every optimization PR must include:

1. The constraint it addresses — link to the target from Gate 1
2. The profile evidence — flamegraph or trace showing the bottleneck from Gate 3
3. Before/after numbers — from a load test matching production cardinality and payload sizes
4. The rollback mechanism — feature flag name, how to disable, expected behavior when disabled
5. Dashboard/alert updates — proving the improvement is monitored

A PR that says "improved performance" without these five items is incomplete.

Common Anti-Patterns

Anti-pattern	Why it fails	What to do instead
"Make it faster" without targets	No way to know when you are done	Define Gate 1 targets first
Optimize all endpoints equally	Wastes effort on cold paths	Identify hot path (Gate 2) first
Add caching everywhere	Cache invalidation bugs, memory bloat	Only cache when profile shows I/O is the bottleneck
Copy patterns from high-scale systems	Patterns designed for 100K RPS add complexity at 500 RPS	Follow the escalation ladder
Skip load testing	"Works on my machine" is not proof	Load test with production-like data
Bundle optimizations in one PR	Cannot isolate which change helped or hurt	One optimization per PR with its own feature flag
Optimize without observability	Cannot detect regressions	Set up monitoring before optimizing
Write-through invalidation for data another system also writes	Their writes never reach your invalidator; the cache is silently TTL-only with a false sense of freshness	TTL-only with a stated staleness window until you are the sole writer (ladder step 0)
Per-instance cache at N instances, no shared tier	Origin load × N; every instance misses together on expiry	Shared tier with TTL clamp + refresh-ahead behind a cross-instance lock (case study 2)
Long TTL with no clear that reaches every instance	Operators lower TTL for everyone to serve a rare clear	Epoch-prefixed keys: one counter bump invalidates every instance
Serialize a type that drops fields into a shared tier	The same key answers differently per tier	Static serialisation check per cached type; failing types stay in-process
Cache on a write path "with a short TTL"	Stale metadata decides how a write is persisted — wrong forever	A request-scoped memo (no TTL, no sharing) removes the duplicate reads with zero staleness

Case Studies

• Voucher Distribution System — 30M vouchers, 50K req/s per pod, demonstrates the full escalation ladder from batch indexing through lock-free patterns
• Two-Tier Cache Under a Second Writer and Fifteen Instances — content cache for a service that is not the only writer of its data and runs as 15 instances: ownership decides the contract (TTL-only), the ladder is climbed in a different order, and the cross-instance mechanisms (TTL clamp, refresh-ahead behind a lock, negative cache, epoch clear, serialisation guard) are worked through, plus what was deliberately not cached

Cross-References

• engineering-rest-api-design (this kit) — API contract design before optimization
• diagnose (this kit) — when the "slow" report is really a bug
• Not bundled (separate Go pack): Go performance patterns, observability, benchmarking with benchstat, database query optimization, concurrency primitives