Well it is easier to use time if you can and certainly from a UI point of view. For example, if your are creating a delay class you would want to specify the delay time in ms something like delay.setDelayTimeMs(1000). However, a delay time of 1000ms is meaningless without a sample rate. This equates to a delay of 44100 samples at a sample rate of 44100 and a delay of 96000 samples at a sample rate of 96000.
You cant assume a specific sample rate as this will change depending on the users hardware. It is probably most straightforward if you specify the sample rate when you change the time something like delay.setDelayTime(getSampleRate(), 1000). This saves you from forgetting to update the sample rate when it changes (usually in a prepareToPlay method).
I think the most straightforward way would be to append what unit you are setting to the method name eg. setDelayTimeMs(…, …) or setDelaySamples(…). This means you shouldn’t call the wrong method by accident.
By the way I am assuming you are dealing with some audio processing here (please correct me if I am wrong). If you can give a more detailed description of your application we could probably tell you more.