Please enable JavaScript.
Coggle requires JavaScript to display documents.
NVMe TCP request double free - Coggle Diagram
NVMe TCP request double free
how to trigger
blktests nvme/012
two timeouts
NVMe timeout handling
controller is LIVE
queue_work on err_work
teardown
holding teardown lock
controller isn't LIVE
abort this request directly
with holding teardown lock
but the controller may not be disabled when aborting the request
timeout vs. normal completion is triggered
no race between normal completion and direct abort from timeout handler
blk-mq related parts
remote/async complete
timeout
hold one request ref before calling driver's timeout()
release the refcount after the timeout() is done
free the request if the refcount drops to zero
refcount underflow is triggered there
rq->ref
refcount_set(1)
blk_mq_rq_ctx_init
blk_rq_init
refcount_dec_and_test
blk_mq_free_request
flush_end_io
blk_mq_check_expired
refcount_inc_not_zero
blk_mq_check_expired
flush timeout vs. normal completion
observation
two timeout triggered
first timeout is flush request(not always)
second timeout is write request
refcount underflow is triggered on write request
request->state
INFLIGHT observed in nvme timeout handler
IDLE oberved when refcount underflow is triggered
controller state
RESETTING when two timeouts are fired
enter CONNECTING when refcount underflow is triggered, teardown is done
misc
the issue can't be triggered if rq->ref is read in timeout handler
conclusion
problem 1
blk_mq_tagset_wait_completed_request() hang forever
flush_rq's state isn't updated to IDLE
problem 2
blk_mq_request_completed() should cover IDLE
.complete() can update request's state
problem 3
nvme-tcp's timeout handler
time out request vs. normal completion
NVMe timeout should have been run after normal completion is stopped
direct completion is added for avoiding to block setup controller sequenence