as said here and there, I don’t have a solid computer science or software engineering background. Instead I’m an electrical engineer student, doing scientific real time DSP university projects that really benefit from JUCE. We now got an exiting application up and running and see a lot more potential in starting to build our own modules for better code reusability. There are quite a lot of ideas what could be contained in these modules, but before getting started on implementing features over features, I’d really like to gather some information on how the professionals out there ensure high code quality.
Unit tests and CI are terms I came across and google gave me some hints to basically understand the aims of these approaches. However wisely choosing one approach, one special framework, etc. for a project that should last for maybe years usually needs some experience I don’t have. And I think I just scratched the surface in understanding why this is done but have no experience on what working with those setups in a daily productive environment looks like.
At the moment it will be just me working on the codebase, but others might join over the time and have developed code in the past. We use a private GitLab Server, hosted by the university, which displays me the possibility of trying out a new CI build/test solution called “AutoDevOps” since month. I don’t know if this is one of the possible tools to use, especially if it’s marked as beta?
What I’d expect would be a system behaving like that:
When implementing a new class, I would first implement a unit test (based on the JUCE UnitTest class?) that does a bunch of tests to see if the class behaves as expected. Then I would go on and implement the actual features.
When I’m done, I’d push a commit containing these changes
Now there is some kind of server (hosted where, running which OS?) that gets a notification if I pushed a commit. It will then automatically run all unit tests (using the UnitTestRunner?) and give me a notification through whichever communication channel if any test failed
Are my expectations right until here?
Now if there are right, what do I need to build such a system? Do I need to set up an independent physical build server with every possible OS the code should run on (–>would that mean I have to set up a Mac Mini somewhere running the Mac OS tests ) or is this usually done with virtual machines running a specific version of the target OS? What about mobile target platforms like iOS? How does interaction between those server(s) and the GitLab server work (assumed that it’s not part of the GitLab server through this autoDevOps feature mentioned above)?
A lot of questions. I’d love to hear how the professionals out there handle those tasks! Thank you in advance!
I hope what I wrote was not too complicated - should I clarify better what I’m asking for?
To make it short: I’m still strongly interested in some input on how the “pros” out here do unit tests for their own in-house JUCE-module codebase (I think there might be some out here that chose JUCE modules as the way to manage their own extensions to the JUCE codebase), what tools and workflows are used and so on, to derive a setup that might be suiting my plans. Maybe even someone from Roli might want to give some input on how they do it with the original JUCE codebase?
In the JUCE team we now use Jenkins on a several dedicated build machines (1x MacMini for macOS/iOS, 1x Windows machine for Windows, 1x Linux machine for Linux/Android).
For a little over a year though, we just had a single mac mini with two VMs (Windows/Linux) running on it. That was terribly slow though - also if you go with this solution be sure to get plenty of RAM. Jenkins has an option where it can simply poll a git repo for changes every minute or so (easier) or you can use a git hook to trigger jenkins to build.
This really depends on the class you are writing. I think test driven development lends itself very good when you are implementing something to a spec. For example, the OSC module, MIDI MPE and many parts of the DSP module were written this way.
Other classes, like the plug-in wrappers, don’t lend themselves very good to JUCE unit tests as the point of these classes (and likely failure methods) is in the interaction between the plug-in and the host - which is hard (impossible?) to simulate in a JUCE unit test. You really want to test this code with external testing tools like auval or steinberg’s plug-in validator which can load and unload your .dll etc.
Also, any code producing audio can be difficult to test as the exact result often depends on optimisation flags, which FFT library you use and so on (that’s the nature of floating point values). At ROLI, we have a few ground truth audio files. We can compare the output of some ROLI software with the ground truth audio files and if they are “similar enough” it passes the test where the definition of “similar enough” depends on your application.
Thank you Fabian, this really helped me a lot! I‘ll definitely take a look at Jenkins.
I assume that Jenkins will also build a simple application that runs the tests using the UnitTestRunner class for everything a unit test was written for and then executes it? Maybe this will get clear as soon as I take a deeper look at Jenkins, but how are the test results reported to Jenkins and the user, is there some standard interface to report the JUCE unit test results to the system?
Another point: Is a virtual machine running MacOS an option as well? I thought I read somewhere that Apple allows running MacOS in virtual machines for development purposes, but I‘m not sure
Yes exactly. In JUCE we have the UnitTestRunner app (in JUCE/extras/UnitTestRunner) which gets compiled&run every time we push.
Jenkins offers many plug-ins and there is probably a nice way to do this, but Jenkins works by executing shell (Linux/macOS) or batch (windows) scripts which have to succeed for the build to succeed. Our shell scripts simply fail if the UnitTestRunner fails.
You are legally only allowed to run a macOS virtual machine on macOS hardware - so using a “hackint0sh” (that’s the term you want to google for ) on a VM is illegal.
Ok, I din‘t know that this permission is Apple hardware bound, but I also don’t plan to do anything illegal. I always thought of a non-Apple personal computer illegally running MacOS when talking about a hackintosh, not a dedicated hardware server running a MacOS vm just for building software, but yeah, I understand that it’s nearly the same from a legal point of view.
You don’t have to use JUCE for tetsing JUCE modules (I don’t). If you have gitlab, then git lab is your git repository, so you should be able to trigger anything from there. You should be able to instruct gitlab to call something on your build farm (whether it’s Jenkins on the one you mentioned) to trigger the build.
I know that there are alternative unit testing frameworks out there, however I thought using the tools that already come with juce would reduce the number of additional dependencies. Which one do you use and why?
Well, Fabians post was more like „the build server polls the repo for changes on a regular interval“ if I got everything right? In this setup it shouldn’t matter if my git server is based on gitlab, github or whichever git server out there.
However if especially gitlab has some built-in tooling that actively invokes the build and test after a commit, this would be a maybe even better setup. Part of my initial question was indeed if this gitlab dev-ops thing it always shows me on top of every repo might be about exactly such a feature. Any gitlab experts round here that have set up a gitlab-triggered build system?
Haha, thank you. Maybe this is a matter of the “language barrier” but as a non-native english speaker I wouldn’t have considered “triggers” to be the correct search term. I actually performed some google searching on CI but actually drowned in a lot of results. This thread really gave me a good insight, some keywords and some clarification and I think I’m now ready to figure out my way through all this on my own. I really like this community
If your code is hosted on GitHub, you can also use AppVeyor (Windows/Linux) and Travis CI (Linux/macOS). They both have pretty good documentation (for instance, https://docs.travis-ci.com/user/for-beginners is where you would start to learn about Travis CI) and they are used by a lot of projects, so it’s easy to find some help.
I’m using both AppVeyor and Travis CI to build and test https://github.com/McMartin/FRUT, an alternative to Reprojucer for building JUCE projects using CMake.
) or is this usually done with virtual machines running a specific version of the target OS? What about mobile target platforms like iOS? How does interaction between those server(s) and the GitLab server work (assumed that it’s not part of the GitLab server through this autoDevOps feature mentioned above)?
- should I clarify better what I’m asking for?
) on a VM is illegal.