Automated testing with REAPER on MacOS

Useful Tools and Components
1

I have lately run into a few regression issues while developing my plugins, so I threw together an automated test, which uses REAPER on MacOS, to do a kind of ‘smoke test’ of the results of a plugin rendering session.

The idea is to create a REAPER project which is fully set up to test plugin features, and which is configured to render the project and compare against a ‘reference’ rendering of the project. This is so that a ‘gold reference’ rendering can be made, and then subsequent tests run and compared against that gold reference.

This script has the command-line “–create-gold-reference”, which creates the reference against which subsequent execution of the script (ommitting the argument) will be compared. If there are differences, the test is considered a ‘fail’, but if the new test rendering matches the prior ‘gold reference’ rendering, it is considered a pass.

So, I have this in my plugins development repo under the “tests/” subdirectory, which for example looks like this:

.
├── 00_plugin_all_formats_test.sh
├── Makefile
├── Reaper
│   ├── 00_plugin_all_formats.RPP
│   ├── Backups
│   └── Media
│       ├── testaudiofile1.wav
│       ├── testaudiofile2.wav
│       └── testaudiofile3.wav
├── Reference
│   ├── 00_plugin_all_formats_reaper_reference.wav
│   └── 00_plugin_all_formats_reaper_results.txt
└── Results

Here you can see that I’ve created a REAPER project named “00_plugin_all_formats.RPP” - this project is set up to test the plugin features, and in this case includes both VST3 and AU versions, on different tracks, albeit with the content inserted sequentially, such that when rendered, both versions of the plugins are executed with the same media files.

Here, the ./00_plugin_all_formats_test.sh file contents:

#!/bin/bash

# Script: 00_plugin_all_formats_reaper_test.sh
# Purpose: Automate plugin testing in REAPER on macOS by rendering a project and comparing output with a reference file.
# Specification: for the plugin under test, AU and VST3 versions of the plugin are set up on separate tracks
# Requirements: REAPER installed, project file with saved render settings, reference audio file, sox for comparison.
# Features:
#   - Normal mode: Render project, compare with reference, log results.
#   - --create-gold-reference: Render project, move output to reference path, skip comparison.

# Configuration variables
REAPER_PATH="/Applications/REAPER.app/Contents/MacOS/REAPER"  					# Path to REAPER executable
PROJECT_PATH="Reaper/00_plugin_all_formats.rpp"        							# Path to REAPER project file
OUTPUT_PATH="Results/00_plugin_all_formats_reaper.wav"  						# Path where REAPER will save rendered output
REFERENCE_PATH="Reference/00_plugin_all_formats_reaper_reference.wav"   		# Path to reference audio file
LOG_FILE="Reference/00_plugin_all_formats_reaper_results.txt"           		# Log file for REAPER output
TIMEOUT=300                                                 					# Maximum time (seconds) to wait for render
SOX_DIFF_OUTPUT="Results/00_plugin_all_formats_reaper_test_diff_output.wav"     # Output file for audio difference
SOX_ERROR_LOG="Results/sox_error_log.txt"                                      	# Log file for sox errors
TEST_HISTORY_LOG="Results/test_history.log"                                    	# Log file for test results

# Check for --create-gold-reference argument
CREATE_GOLD_REFERENCE=0
if [[ "$1" == "--create-gold-reference" ]]; then
    CREATE_GOLD_REFERENCE=1
fi

# Step 1: Validate environment
echo "Validating environment..."

# Check if REAPER executable exists
if [[ ! -x "$REAPER_PATH" ]]; then
    echo "Error: REAPER executable not found at $REAPER_PATH"
    exit 1
fi

# Check if project file exists
if [[ ! -f "$PROJECT_PATH" ]]; then
    echo "Error: Project file not found at $PROJECT_PATH"
    exit 1
fi

# If not creating gold reference, check if reference file exists
if [[ $CREATE_GOLD_REFERENCE -eq 0 && ! -f "$REFERENCE_PATH" ]]; then
    echo "Error: Reference file not found at $REFERENCE_PATH"
    exit 1
fi

# If creating gold reference, check if reference file already exists (prevent overwrite)
if [[ $CREATE_GOLD_REFERENCE -eq 1 && -f "$REFERENCE_PATH" ]]; then
    echo "Error: Reference file $REFERENCE_PATH already exists. Remove or rename it first."
    exit 1
fi

# Check if sox is installed (only needed for comparison)
if [[ $CREATE_GOLD_REFERENCE -eq 0 ]]; then
    if ! command -v sox &> /dev/null; then
        echo "Error: sox not installed. Install it using 'brew install sox'."
        exit 1
    fi

    # Check if soxi is installed (for debugging file info)
    if ! command -v soxi &> /dev/null; then
        echo "Error: soxi not installed. Install it using 'brew install sox'."
        exit 1
    fi
fi

# Step 2: Ensure output directory exists
# For gold reference, create Reference/ directory; otherwise, create Results/
if [[ $CREATE_GOLD_REFERENCE -eq 1 ]]; then
    OUTPUT_DIR=$(dirname "$REFERENCE_PATH")
else
    OUTPUT_DIR=$(dirname "$OUTPUT_PATH")
fi
if [[ ! -d "$OUTPUT_DIR" ]]; then
    echo "Creating output directory: $OUTPUT_DIR"
    mkdir -p "$OUTPUT_DIR"
fi

# Always create Results/ directory since REAPER renders to OUTPUT_PATH
RESULTS_DIR=$(dirname "$OUTPUT_PATH")
if [[ ! -d "$RESULTS_DIR" ]]; then
    echo "Creating results directory: $RESULTS_DIR"
    mkdir -p "$RESULTS_DIR"
fi

# Step 3: Run REAPER to render the project
if [[ $CREATE_GOLD_REFERENCE -eq 1 ]]; then
    echo "Rendering project to create gold reference: $OUTPUT_PATH"
else
    echo "Rendering project: $PROJECT_PATH"
fi
"$REAPER_PATH" -renderproject "$PROJECT_PATH" > "$LOG_FILE" 2>&1 &

# Get REAPER process ID
REAPER_PID=$!

# Wait for rendering to complete or timeout
echo "Waiting for render to complete (timeout: ${TIMEOUT}s)..."
wait_timeout=$TIMEOUT
while [[ $wait_timeout -gt 0 ]]; do
    if ! ps -p $REAPER_PID > /dev/null; then
        echo "Rendering completed."
        break
    fi
    sleep 1
    ((wait_timeout--))
done

# Check if REAPER is still running (timeout reached)
if ps -p $REAPER_PID > /dev/null; then
    echo "Error: Rendering timed out after ${TIMEOUT}s. Killing REAPER process."
    kill -9 $REAPER_PID
    exit 1
fi

# Step 4: Verify rendered output exists
if [[ ! -f "$OUTPUT_PATH" ]]; then
    echo "Error: Rendered output not found at $OUTPUT_PATH. Check $LOG_FILE for details."
    exit 1
fi

# If creating gold reference, move the rendered file to REFERENCE_PATH
if [[ $CREATE_GOLD_REFERENCE -eq 1 ]]; then
    echo "Moving rendered output to gold reference: $REFERENCE_PATH"
    if ! mv "$OUTPUT_PATH" "$REFERENCE_PATH"; then
        echo "Error: Failed to move $OUTPUT_PATH to $REFERENCE_PATH."
        exit 1
    fi
    echo "Successfully created gold reference file: $REFERENCE_PATH"
    echo "$(date): Created gold reference file ($REFERENCE_PATH)" >> "$TEST_HISTORY_LOG"
    exit 0
fi

# Step 5: Compare rendered output with reference
echo "Comparing rendered output with reference..."

# Debug: Print file info for reference and output files
echo "Reference file info:"
soxi "$REFERENCE_PATH"
echo "Rendered output file info:"
soxi "$OUTPUT_PATH"

# Step 5a: Mix the files with inverted phase to create difference file
if ! sox -V2 -m "$REFERENCE_PATH" -v -1 "$OUTPUT_PATH" "$SOX_DIFF_OUTPUT" 2> "$SOX_ERROR_LOG"; then
    echo "Error: sox failed to compute difference. Details:"
    cat "$SOX_ERROR_LOG"
    echo "Check input files, formats, or sox installation."
    exit 1
fi

# Verify difference file was created
if [[ ! -f "$SOX_DIFF_OUTPUT" ]]; then
    echo "Error: Difference file $SOX_DIFF_OUTPUT was not created."
    exit 1
fi

# Step 5b: Analyze difference file with sox stat
sox "$SOX_DIFF_OUTPUT" -n stat 2> "$SOX_ERROR_LOG"

# Extract RMS amplitude from stat output
RMS=$(grep "RMS     amplitude" "$SOX_ERROR_LOG" | awk '{print $3}')
if [[ -z "$RMS" ]]; then
    echo "Error: Failed to extract RMS amplitude from stat output."
    cat "$SOX_ERROR_LOG"
    exit 1
fi

# Debug: Print difference file size
DIFF_SIZE=$(stat -f %z "$SOX_DIFF_OUTPUT" 2>/dev/null)
if [[ $? -ne 0 || -z "$DIFF_SIZE" ]]; then
    echo "Error: Failed to get size of difference file $SOX_DIFF_OUTPUT."
    exit 1
fi
echo "Difference file size: $DIFF_SIZE bytes"

# Check if difference file is silent (RMS amplitude < 0.0001)
if (( $(echo "$RMS < 0.0001" | bc -l) )); then
    echo "Success: Rendered output matches reference (RMS: $RMS, size: $DIFF_SIZE bytes)."
    echo "$(date): Success (RMS: $RMS, size: $DIFF_SIZE bytes)" >> "$TEST_HISTORY_LOG"
else
    echo "Warning: Rendered output differs from reference (RMS: $RMS, size: $DIFF_SIZE bytes). Check $SOX_DIFF_OUTPUT for differences."
    echo "$(date): Warning (RMS: $RMS, size: $DIFF_SIZE bytes)" >> "$TEST_HISTORY_LOG"
    exit 1
fi

# Step 6: Clean up
echo "Cleaning up..."
rm -f "$SOX_DIFF_OUTPUT" "$SOX_ERROR_LOG"

echo "Test completed successfully."
exit 0

And lastly, a Makefile to simplify testing initiation:

all:	testplugin

gold:
	rm -rf Reference/00_plugin_all_formats_reaper_reference.wav
	./00_plugin_all_formats_test.sh --create-gold-reference

testplugin:
	./00_plugin_all_formats_test.sh 

reset:	clean	gold	testplugin

clean:
	rm -rf Results/*

With this setup (assuming a properly configured REAPER project), it is possible to do this:

$ cd tests; make reset

.. which will render the ‘gold reference’ .wav file, which should of course be manually verified and checked - and then, for all subsequent testing:

$ make test

.. will validate the results, going forward, of any changes to the plugin as the project evolves.

Here’s an example test session, which first ‘resets’ the project with a new gold reference, and a subsequent test run which validates the plugin:


➜  tests git:(main) ✗ make reset
rm -rf Results/*
rm -rf Reference/00_plugin_all_formats_reaper_reference.wav
./00_plugin_all_formats_test.sh --create-gold-reference
Validating environment...
Rendering project to create gold reference: Results/00_plugin_all_formats_reaper.wav
Waiting for render to complete (timeout: 300s)...
Rendering completed.
Moving rendered output to gold reference: Reference/00_plugin_all_formats_reaper_reference.wav
Successfully created gold reference file: Reference/00_plugin_all_formats_reaper_reference.wav
./00_plugin_all_formats_test.sh
Validating environment...
Rendering project: Reaper/00_plugin_all_formats.rpp
Waiting for render to complete (timeout: 300s)...
Rendering completed.
Comparing rendered output with reference...
Reference file info:

Input File     : 'Reference/00_plugin_all_formats_reaper_reference.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:19.00 = 837900 samples = 1425 CDDA sectors
File Size      : 5.03M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

Rendered output file info:

Input File     : 'Results/00_plugin_all_formats_reaper.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:19.00 = 837900 samples = 1425 CDDA sectors
File Size      : 5.03M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

Difference file size: 5027480 bytes
Success: Rendered output matches reference (RMS: 0.000000, size: 5027480 bytes).
Cleaning up...
Test completed successfully.

➜  tests git:(main) ✗ make testplugin
./00_plugin_all_formats_test.sh
Validating environment...
Rendering project: Reaper/00_plugin_all_formats.rpp
Waiting for render to complete (timeout: 300s)...
Rendering completed.
Comparing rendered output with reference...
Reference file info:

Input File     : 'Reference/00_plugin_all_formats_reaper_reference.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:19.00 = 837900 samples = 1425 CDDA sectors
File Size      : 5.03M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

Rendered output file info:

Input File     : 'Results/00_plugin_all_formats_reaper.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:19.00 = 837900 samples = 1425 CDDA sectors
File Size      : 5.03M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

Difference file size: 5027480 bytes
Success: Rendered output matches reference (RMS: 0.000000, size: 5027480 bytes).
Cleaning up...
Test completed successfully.
➜  tests git:(main) ✗ tree
├── 00_plugin_all_formats_test.sh
├── Makefile
├── Reaper
│   ├── 00_plugin_all_formats.RPP
│   ├── Backups
│   └── Media
│       ├── testaudiofile1.wav
│       ├── testaudiofile2.wav
│       └── testaudiofile3.wav
├── Reference
│   ├── 00_plugin_all_formats_reaper_reference.wav
│   └── 00_plugin_all_formats_reaper_results.txt
└── Results
 ├── 00_polardesigner_all_formats_reaper.wav
    └── test_history.log

I thought this might be useful for others, so I’m sharing it - and I would love to hear some feedback on the comparison/diff result testing techniques I’m using, which rely on sox and some offline analysis of the test .wav file against the gold reference .wav file.

Thoughts? I’ll probably flesh this out a bit for Logic, Cubase and ProTools testing going forward, so I’d love to get some feedback from anyone else who might be interested in this situation ..