Oversampling Upsampling latency?

I am trying to implement a True Peak Limiter using the upsampling method from the juce Oversampling class. I cant figure out how to get the latency of just the upsampler. I thought the latency should be half of the getLatencySamples() (if you were to calculate the delay on the base sampleRate). if i upsample my signal by a factor of 4 and combine 4 upsampled samples into one sample, the latency is off.
Am i missing something here?