I sometimes use the outline for a simple mouse over effect over labels. Applying your workaround would mean that the text would physically jump around as the user mouses over the label.
Well I don’t want to break the current behavior.
It would mean that all text will grow up after the change.
Allowing to change those value is ihmo the safest move.
I don’t find disturbing that there is a small border by default… just annoying that you can’t remove it.