I have the top corners that are not rounded (because there is like a light grey background at the top) when opening a native fileChooser in a plugin on osx :
FileChooser fc ("Choose a file...",
File::getSpecialLocation (File::userDesktopDirectory),
"*", true);
if (fc.browseForFileToOpen())
{
...
}
The native file chooser is designed to be attached on the main window, thats why it has rectangle borders on the top. But its a very bad idea to use in a plugin, because it has modal behavior (the message thread gets wrapped). Some hosts may crash because of this behavior. You can choose the file-choose from juce, and use it an non modal window.