Viewport: Enable scrollOnDrag while dragging

I am enabling the scrollOnDrag() of a viewport, while I’m in drag mode. I do this because I only want scrolling to be enabled, when I do a vertical scroll. So I calculate the angle of the drag in my mouse listener and when its vertically enough I enable scrollOnDrag of the viewport.

The problem is this doesn’t work, because the viewport creates a new DragToScrollListener when I enable scrollOnDrag(). When this happens the DragToScrollListener resets the “numTouches”. So when I keep dragging the listener thinks no touch is active and no scrolling happens.

What is the best way to fix this?