browseForFileToSave & ! warnAboutOverwrite not working?

When using the native FileChooser (tested on el capitan), browseForFileToSave (false) will still warn about overwriting the file.
Is that expected?

I think that might be something that the OS does…