Bug or expected behaviour? The value of getMouseDownY() changes if your component moves.
getOffsetFromDragStart() to figure out how far mouse has moved in drag operation and then update internal state based on that. If I get near the edge of the viewport it auto scrolls so user can keep dragging. Now mouse down position is incorrect relative to where user clicked in my component and user can no longer drag because mouse down position is moving as fast as component scrolls.