It seems OpenGL use may greatly help, there was a similar question (but not about svg) here:
But maybe just caching your svg rendered bitmap and reusing the cache might suffice, don’t know yet how it works there but I could have a look if you’re really stuck.