Run Python scripts from within JUCE app

General JUCE discussion
1

I am starting out with extending JUCE to run Python scripts.

This post follows on from http://www.juce.com/forum/topic/juce-python

I'm looking at JuceDemo/Source/Demos/JavaScript.cpp

I'm planning on creating a Python.cpp does exactly the same thing for Python

Someone else seems to have done the same thing for Lua.

This makes me wonder whether there may be some sensible JUCE restructuring: we could have PythonEngine, LuaEngine and JavascriptEngine all inherit from ScriptEngine

ScriptEngine could have a

virtual void ScriptDidCallback(const var::NativeFunctionArgs& args);

and virtual StartRuntime, ExecuteScript, ExecuteCommand methods.

This would be enough for complete interoperability I think. If the script can call a function in the JUCE wrapper, it can also pass data out through the parameter-list. And vice versa. So we would have bidirectional signalling and data transfer.

Maybe StartRuntime could have a 'bool createOnNewThread' which could allow multiple environments to coexist.

Jules, what do you think of this? If I write a solid implementation, might you consider merging it?

π

EDIT: oh I've just noticed a problem -- the Python C-sourcecode probably couldn't be shipped with JUCE under the same license. So it would have to exist as an extra module I guess. But nevertheless it may make sense to create a generic ScriptEngine object...

1 Like
2

Well, good luck with that! I don't think a common base class would make much sense though - javascript is a special case because the var class already exists as a representation of a javascript object, and the engine relies on that. To do this for python, I guess you'd also need to build an equivalent for python, or provide a different interface to the engine that avoids the issue.

3

This is a much harder problem than I first imagined.

For a start the CPython tarball is ~20Meg of code, 2 for the core and 15 for the libraries.

Also there is involved build process where it figures out your OS and assembles different headers depending.

Somewhere in this source tree there may be code that takes care of the core mechanics. For example, getting a random number will involve a platform specific API-call, but iterating over a list should be platform agnostic. So somewhere we might expect some header that provides an interface for all of these platform specific operations, and it may be possible to bind them to JUCE functions.

Also if a lot of work is required, this work may need to be repeated for subsequent versions of Python.

Another option is the standard approach given in the official documentation, i.e. you link your project against libpython.a, #include "Python.h" then you can use Py_Initialize() and friends to crank up a Python runtime, load .py files into it, execute functions from them, etc. However this would require supplying a different libpython.a for each target platform, and that would be some work on iOS for example, I think a few people have succeeded in compiling it for iOS.  But also this approach is a bit messy because it requires linking a per-platform static library. Even if IntroJucer can do this, it breaks pattern with the rest of the framework.

Besides, it is overkill. The original task was not to get a complete and ultraefficient Python implementation, but to allow Python scripting.

Maybe somebody has written a more lightweight Python C implementation.

Turns out there are a couple of interest: https://wiki.python.org/moin/PythonImplementations

I'm currently looking through these to see whether it will be possible to knock one into shape.

π

4

Finally I have a promising candidate: MicroPython (https://github.com/micropython/micropython)

Someone on the Python-dev IRC channel pointed me towards it.

I emailed the author and got the following reply:

Hi,

Thanks for your interest in Micro Python.

Indeed Micro Python was written from scratch with the intention of
keeping it compact and self contained.  While it's main aim is to run
on a microcontroller, it does also run on linux/posix, windows and
mac.  It would be relatively straight forward to embed it into another
application (such as JUCE).  The 2 main things that need attention
are:

1. memory management.  It comes with a custom garbage collector that
does some tricky things.  This might or might not work within your
app.  You can easily replace the memory manager with your own, but it
needs to be a garbage collected manager.

2. exceptions.  It uses setjmp/longjmp for this, which may or may not
wreak havoc with C++'s exceptions.  You could get around this by not
using exceptions in your Python scripts.

Cheers,
Damien.

I am at a very early-stage of figuring out how all of this will have to fit together.

Could someone help me bring it into focus by filling me in with the JUCE perspective on memory management and exceptions?

This is exciting I think. MicroPython is MIT license. This is doable!

π

5

I've run into a serious problem with MicroPython.  It requires a complicated build script for each target platform that requires building, then digging through the intermediate generated files for data which then gets added into some source file, then everything gets built again.  etc.  There is complicated interplay between the preprocessor, the build-scripts and the compiler.

So I'm going back to looking at linking libpython.a for each target platform.

It should be straightforward to get hold of this library for the three desktop platforms (windows, OS X, Linux).

However, iOS and Android may be more tricky.

Kivy is a multiplatform Python engine, so they have figured out how to build Python for iOS/Android.

kived on IRC freenode #kivy-dev has just shown me how to get it for Android. I will list the steps here before I forget.

Get https://github.com/kivy/python-for-android

Now create the following environment variables (applying common sense):

export ANDROIDSDK="bla/Android SDK"
export ANDROIDNDK="ble/Android_ndk"
export ANDROIDNDKVER=r9d <-- check the RELEASE.TXT in the ndk folder
export ANDROIDAPI=19 <-- in bla/Android SDK, look in /platforms/ (*)

(*)or run /tools/android (to get the SDK manager)

Of course this requires first installing android SDK and NDK.  This above step is explained (though not so clearly) in the README.rst file

Now:

cd python-for-android-master/

./distribute.sh -m 'kivy'

and after several minutes of scrolling text you should get:

foo/python-for-android-master/dist/default/python-install/lib/python2.7/config/libpython2.7.a

Yay!

I don't know about Python 3, and I haven't yet investigated iOS

6

Kivy's iOS Python build seems to be buggy -- I've reported it here: https://github.com/kivy/kivy-ios/issues/102

It may be due to a new version of XCode coming out last week.

But assuming that gets sorted out I should be able to get an iOS Python library.

So I can get the Python library for all five platforms.

I'm going to start getting it working on OS X, and then add the others. 

I've managed to find and rename my python lib to /path/to/myProject/Builds/OSX/libpython3_3.dylib -- is this the "right place" to put it?

I am just a little confused after reading Jules' comment on this thread: http://www.juce.com/forum/topic/introjucer-xcodes-link-binary-libraries -- it seems to be saying that I should use the .a.  But wouldn't I want the .dylib (or for Windows it would be .dll)?  Because I don't want to bloat the executable.  I want my project to ship with the executable and libpython3_3.dylib.

From the same post, I can see that in the IntroJucer's "external libraries to link" field I should just put "python3_3".  But if I now launch Xcode, go into the project settings, build phases tab, it appears that this library is not linked.  I can only see the 13 Standard iOS libraries that were linked before. Does this mean it failed?

π

7

I've been looking at using boost-python, I would like to share results.

Firstly, why not just use the standard C-API for bridging between C and Python https://docs.python.org/3.4/extending/index.html ?

Mainly because it is a bit clunky, there is going to be some amount of boilerplate and repetition. How to transfer umpteen datatypes like lists and dictionaries between C and Python?

Also there are multithreading issues. What if several C++ threads independently call different functions in the Python runtime? How about if one C++ thread locks the runtime, calls a Py function, but that function then calls back out to C++? In that case the runtime surrenders the lock, so the original C++ thread maybe thinking it is still locking the runtime but it isn't.

I'm not sure to tell the truth. And I think actually it may be the best way to go; something simple and platform independent.

Boost is a huge C++ library, and boost-python digs its tendrils deep into Boost.  "According to http://www.pdimov.com/tmp/report-d5ca5be/module-levels.html Boost.Python is at module level 11, roughly requiring half of Boost" (thanks jhunold on irc.freenode.net #boost)

This means there is no chance of using Boost's BCP tool to just extract the necessary files, Boost is >300Megs and it is going to pull out a sizeable chunk. So the only practical solution will be to use boost-python library.

On OS X I have homebrew, and it is as simple as 'brew install boost-python' then sudo find / -iname "libboost_python*". Note the '*-mt' mean multithreaded.

But realise that to make this multiplatform, we would need to build boost-python on all five platforms just as we have to build python on all five platforms. And it looks like a rocky ride for mobile platforms.  (Android: http://www.codexperiments.com/android/2011/05/tips-tricks-building-boost-with-ndk-r5/).

So that may be a showstopper right there. At least building Python on all five platforms is already being done by the Kivy team. But for boost-python no such luck.

Anyway, trying it out on OS X, using an empty Xcode command line project, I straightaway run into a bug: http://boost.2283326.n4.nabble.com/boost-python-embedding-error-runtime-Mac-OS-X-1-38-td2702075.html

And there I am currently stuck. I don't want to have to dig into the source code for Boost.Python. But that may be the only way through.

What other options?

I have put some links up at http://mathpad.wikidot.com/embedpython

ffpython ( http://www.codeproject.com/Tips/587652/Cplusplus-Easier-to-Embed-Pytho ) looks interesting, although it doesn't seem to be actively maintained and it doesn't support Python 3.x

A couple of articles that look worth reading:

http://realmike.org/blog/2012/07/08/embedding-python-tutorial-part-1/

http://codextechnicanum.blogspot.co.uk/2013/12/embedding-python-in-c-converting-c.html

At the moment I am more interested in non-boost alternatives, mainly because it props the door open for a multiplatform solution, and also because it wouldn't run the risk of hitting bugs deep in boost. Also, boost.python provides way more than is needed for a simple bridge.

π

8

Thanks for posting your findings - did you make any more progress on this?