How to set up an audio app on Linux without desktop?

For the graphics part, it seems what you want is direct access to the framebuffer in Linux.

I’m not familiar with that, but that question has been asked on the forum before. See if the answers in the topics below are somehow helpful for your case:

BTW this is by no mean a complete list, you could find more searching with the search feature of the forum, or with Google