You don't need to worry about the START_JUCE_APPLICATION macro, that's just what gets everything else up and running for you.
The only reason any of your GUI gets shown at all is because your app's initialise(const String& cmdline) function sets such things in motion (e.g. creating a window, etc..). Since this function is given the parameters the app was launched with, you just need to examine these before you create any GUI, and decide what you might want to do instead.
How you do this is up to you and your requirements, of course. Here's a super dumb example...
void initialise (const String& cmdLineParams) override
{
if (cmdline.containsWholeWord("nogui"))
{
// create and configure some kind of 'headless' execution object...
}
else
{
// create the main window as usual
mainWindow = new MainWindow();
}
}
Obviously the above is way too simplistic, and only to be considered an illustration - your parameter parsing would need to be a lot more sophisticated than that! Also, how you implement what happens instead of creating the UI is down to your requirements, like everything else.
Hope this helps!
EDIT: I am of course assuming that you already have a GUI application at this point. If not, then this probably doesn't help much!