PCIe/CXL Deep Dive · All levels
Coherency and Memory Expansion: Tricky Q&A
Senior interview and review questions for Coherency and Memory Expansion.
Section Q&A bank
Use these drills after completing all topics in Coherency and Memory Expansion. Answer with workload context, mechanism proof, artifact, owner, and release decision.
What symptom distinguishes a coherency bug from a simple bandwidth limit?
diagram
[INT][PCIE-CXL][COHERENCY-MEMORY-EXPANSION]
Q: What symptom distinguishes a coherency bug from a simple bandwidth limit?
A:
Coherency bugs produce correctness failures or stale reads under contention, often rare and thread-count dependent. Bandwidth limits show smooth throughput ceilings without data corruption. Prove with litmus tests and ownership traces.
FOLLOW-UP TRAP: Explaining rare wrong answers as timing noise without coherency litmus tests.Why do HDM windows require careful alignment with OS NUMA policies?
diagram
[INT][PCIE-CXL][COHERENCY-MEMORY-EXPANSION]
Q: Why do HDM windows require careful alignment with OS NUMA policies?
A:
OS page allocation and migration assume node distance and capacity metadata. Misaligned HDM regions cause remote access penalties or stranded capacity when the OS cannot map pages efficiently.
FOLLOW-UP TRAP: Exposing memory capacity without publishing accurate proximity and interleave attributes.How do ownership upgrades differ from simple reads in CXL.cache?
diagram
[INT][PCIE-CXL][COHERENCY-MEMORY-EXPANSION]
Q: How do ownership upgrades differ from simple reads in CXL.cache?
A:
Upgrades request write permission and may invalidate sharers, adding snoop latency and retry paths. Treating them like reads underestimates contention cost and can deadlock if evictions are mis-ordered.
FOLLOW-UP TRAP: Benchmarking read-only traffic and expecting write-heavy performance.What RAS policies matter most for fabric-attached memory?
diagram
[INT][PCIE-CXL][COHERENCY-MEMORY-EXPANSION]
Q: What RAS policies matter most for fabric-attached memory?
A:
Poison propagation, page offline, and selective retire must work across host and device without breaking interleave groups. Partial failure handling should preserve system stability while reclaiming affected capacity.
FOLLOW-UP TRAP: Ignoring device-side error reporting when only host DIMM RAS is configured.Q&A drill guide
diagram
WORKLOAD -> PCIe/CXL SYMPTOM -> TIMING/QUEUE METRIC -> ROOT CAUSE -> FIX -> REGRESSIONSketch while answering
diagram
COHERENCY + HDM
CPU caches <-> CXL.cache <-> device memory (CXL.mem/HDM)Key takeaways
Always tie controller and PHY counter shifts to application latency and throughput outcomes.
Lock firmware timing profile, thermal condition, and DIMM state before comparing PCIe/CXL captures.
Common pitfalls
Chasing peak bandwidth while ignoring p99 latency and fairness tails.
Changing timing guardbands without separating SI noise from scheduling issues.
Declaring closure without reliability gates, fault injection, and regression replay.