Its a combination of all of the above. Different MIDI interfaces/driver combinations will have different latency. Differing OS’es have different latency issues, too. This is compounded by the fact that MIDI events are processed by JUCE in the audio block transaction, which you’ve discovered with your findings.
Lower your buffer sizes as minimal as possible, and if you really have to have tightest-possible MIDI, include a calibration process that calculates the MIDI latency of the current user and compensates accordingly.
(I once saw an engineer at Yamaha butcher a MIDI cable, extract the I/O lines, and wire them up to an extra audio input on their interface … so that the MIDI ‘noise’ could be recorded as an audio track, and they could visually see the latency between the MIDI events and the audio that was subsequently generated. This made me realize that MIDI latency is something we will, forever, have to contend with in some way …)