Difficulty: Hard · Puzzle 5 of 6 · Topic: Real Interview Rounds: End-to-End Debug Walkthrough
Interviewer: 'The burst test hangs whenever max_outstanding is 8; at 4 or less it passes. The DUT is idle and healthy.' Evidence at the hang (from a debug print): req_mbx.num()=4 rsp_mbx.num()=0 monitor thread: blocked in req_mbx.put at line 88 scoreboard thread: blocked in rsp_mbx.get at line 131 Walk through the diagnosis and fix.
mailbox #(req_t) req_mbx = new(4); // bounded, depth 4
mailbox #(rsp_t) rsp_mbx = new(4);
// monitor: one thread samples both channels every cycle
task run_phase(uvm_phase phase);
forever begin
@(posedge vif.clk);
if (vif.req_valid && vif.req_ready) req_mbx.put(sample_req()); // line 88
if (vif.rsp_valid && vif.rsp_ready) rsp_mbx.put(sample_rsp());
end
endtask
// scoreboard
task run_phase(uvm_phase phase);
rsp_t p; req_t r;
forever begin
rsp_mbx.get(p); // line 131: waits for a response first
req_mbx.get(r);
compare(r, p);
end
endtaskDraw who is waiting for whom: monitor, scoreboard, and the mailbox occupancy.
1) Symptom: hang only when more than 4 requests are outstanding; DUT healthy and idle.
2) Hypotheses: (a) circular wait through the bounded mailboxes; (b) a lost response (would show rsp pending in the DUT, not a full request mailbox); (c) ordinary scoreboard starvation (fails the 'monitor blocked' evidence).
3) Confirm (a) from the evidence: req_mbx is full (4 of 4) so the monitor blocks in put; the scoreboard waits for a response first (rsp_mbx empty); the monitor is the only producer of responses and it is blocked on the request put. Each waits for the other: monitor waits for the scoreboard to free a request slot, the scoreboard waits for the monitor to deliver a response (verified in simulation: nothing is ever checked).
4) Root cause: a bounded buffer smaller than the outstanding window, combined with a consumer that waits on the other channel first and a single monitor thread. This is a circular wait through TLM/mailbox backpressure, not a lock.
5) Fix: never let a monitor block on downstream storage: use unbounded uvm_tlm_analysis_fifo (or queues) per channel and write() from the monitor, give each channel its own consumer or an id-keyed matcher instead of lockstep gets, and size any bounded buffer at least to the maximum outstanding count. Unblocking only the mailbox is not enough: while the single monitor thread is blocked it also misses handshakes on the other channel, so sample each channel in its own thread. Add occupancy counters/assertions (max occupancy < depth) to catch near-misses.Answer: Root cause: a single monitor thread blocks on a full bounded request mailbox while the scoreboard waits for a response the monitor can no longer deliver: a circular wait through backpressure. Fix: monitors must never block (analysis fifos or queues), consume channels independently or keyed by id, and size any bounded buffer to at least the outstanding window.
Checked by running the code in Verilator 5; the printed output supports the answer above. Plain-SV bounded-mailbox model: a single monitor thread deadlocks against the scoreboard once the window exceeds the depth, per-channel threads do not.
It passes at 4 outstanding because depth 4 hides it, so the failure looks like a rare 'load' problem. Dependency cycles through bounded buffers show up only when the window exceeds the depth; look at occupancy, not just thread states.
How would you catch this class of bug before it hangs a regression? Model answer: Track per-buffer high-water marks and fail the test (or at least warn) when any bounded buffer reaches its depth, and run a stress configuration that sets the outstanding window to its maximum; a watchdog that dumps who is blocked on which buffer after N idle cycles turns the hang into a one-line diagnosis.
It passes at four outstanding and fails above that, so I look at buffer occupancy, not just thread states. The root cause is a single monitor thread blocking on a full bounded request mailbox while the scoreboard waits for a response the monitor can no longer deliver, which is a circular wait through backpressure. I confirm it by checking mailbox depth against the outstanding window. The fix is that monitors never block, using analysis fifos or queues, consuming channels independently or keyed by id, and sizing any bounded buffer to at least the outstanding window.