Fabric-Attached Memory System Design: Inputs and Outputs

Inputs and outputs contract

Inputs and Outputs for Fabric-Attached Memory System Design focuses on Effective mem bandwidth, tail latency across NUMA nodes, and RAS event rate. The purpose is to turn memory observations into mechanism-backed actions with explicit owners and release-safe validation.

Use this contract for architecture, controller firmware, PHY, and validation handoffs. Missing inputs create expensive late-stage rework and inconclusive debug loops.

INPUTS
  - workload distribution and QoS target
  - firmware revision, controller policy profile, timing registers
  - data-rate / voltage / temperature operating state
  - training snapshot and reliability policy status

OUTPUTS
  - bottleneck classification with command-level evidence
  - owner-signed mitigation proposal
  - before/after trend for latency, bandwidth, and reliability
  - regression matrix with rollback triggers

Ownership split

OWNERSHIP LAYERS - Fabric-Attached Memory System Design

layer              owner
-----------------  ----------------
protocol/RTL       platform architect
PHY/SI             PHY + SI/PI owner
firmware/OS        FW + driver owner
validation         compliance + post-silicon

PCIe/CXL deep dive

Memory expansion and coherency require HDM windows, ownership discipline, and NUMA-aware software policies.

Concept diagram

COHERENCY + HDM

CPU caches <-> CXL.cache <-> device memory (CXL.mem/HDM)

Metric graph

EXPANSION BOTTLENECK SHARE

remote latency      ██████
ownership retry     ████
interleave skew     ███

Reports and artifacts

• HDM decode table

• ownership transition trace

• NUMA distance profile

• RAS region policy

Mini case study

Fabric-attached memory increased capacity but p99 regressed until page placement respected NUMA distance.

Debug branches

• Map HDM windows and interleave groups

• Run ownership litmus under contention

• Correlate RAS events with region offline policy

Senior review question

Ask: which latency, bandwidth, and reliability evidence proves this PCIe/CXL topic is closed under real traffic?

Key takeaways

• Always tie controller and PHY counter shifts to application latency and throughput outcomes.

• Lock firmware timing profile, thermal condition, and DIMM state before comparing PCIe/CXL captures.

Common pitfalls

• Chasing peak bandwidth while ignoring p99 latency and fairness tails.

• Changing timing guardbands without separating SI noise from scheduling issues.

• Declaring closure without reliability gates, fault injection, and regression replay.

Handoff explanation

Inputs extend beyond timing registers. PCIe/CXL analysis inputs include traffic distribution, address map, queue policy, training state, SI/PI condition, thermal state, and firmware version.

Outputs must be action-ready: Effective mem bandwidth, tail latency across NUMA nodes, and RAS event rate, artifact packet (NUMA distance table, bandwidth/latency profile, and RAS policy doc), bottleneck class, owner, expected gain, and rollback scope. "Bandwidth improved" without this packet is not signoff-ready.

The safest handoff is a before/after evidence set: environment tags, traces, hypothesis, chosen fix, rejected alternatives, and regression criteria.