A modern code editor for JUCE?

#1

Hey all -

I have written a prototype of a modernized code editor component for JUCE, and I’d like to ask for your opinions, or just gauge the level of interest.

To start with, here is a link to the code:

Example project here: https://github.com/jzrake/MclModules

I did this because I wanted a custom editor for needs of my own project. But I’m also kind of a “Sublime-o-phile” - its performance and editing features are just awesome and I was curious whether an editor built with JUCE could compete in terms of speed. Glyph rendering in Sublime is (I believe) done with a mix of Skia and some custom code. I should say that I’m on OSX, with the CoreGraphics backend, and I’m using JUCE to do the glyph placement, not the drawTextLayout low level graphics method.

I did the rendering using a data structure that can quickly build a GlyphArrangement from the visible subset of the document. There’s no viewport, I just scroll the text directly with a transform.

The underlying data structure will allow for multi-caret editing, syntax highlighting, code folding etc. Multi-caret is 90% there (it needs an algorithm to shift the downstream selections). I have not done anything toward syntax highlighting, but I know more-or-less how it’ll work (should be compatible with CodeTokeniser). Word wrap and justification options are not yet implemented.

So anyway — I think I succeeded in making it fast. It’s also kind of elegant, to the extent that I shamefully lifted a couple of design elements from Sublime. Such things could of course be made more generic if folks are interested in adopting this into JUCE.

I did this over the course of 4 days (in addition to my day job) as proof of principle. So it’s not up to scale by any means. But, I’d really love to see a snappy, fully featured code-or-text editor evolving out there in the JUCE wilderness. Does anyone think this could be a step in that direction? Or, is everyone perfectly happy with the TextEditor and CodeEditorComponent options?

Best! -Zrake

#2

WOW! This is very cool and definitely something JUCE needs!

#3

Screenshot?

#4

Well, it’s hard to convey performance with an image. But here you go…

MclTextEditor-6-23-18
MclTextEditor-6-23-18.png1772x1506 307 KB

#5

Wow, this is really good.

I’ve compiled it on macOS and Windows, but with VS2017, the mcl::TextLayout::getSelections method fails to compile because of the lambda usage in the ternary operator. I’ve quickly added a std:function object to get it to compile, but then the selection doesn’t work anymore.

But really great stuff and a more than capable proof of concept app. I think these things should be next in order to get a useable replacement for the CodeEditorComponent:

  • code highlighting (definitely try to use the existing CodeTokeniser classes)
  • colour scheme (you’re using hardcoded colours all over the place, I’d suggest you add ColourIds like the rest of the JUCE components).
  • word wrap
  • extendability. One of the things I don’t like about the CodeEditorComponent is that you’re quickly heads deep in its internals to hack around because you want to change stuff or add behavior.
  • make sure you stay compatible to the JUCE editor. Ideally it would be a drop in replacement that uses the same constructors / basic methods.

I’ll definitely keep an eye on your progress here. I am currently using the CodeEditorComponent for my app and I always found its haptics to be a little clunky.

#6

I’m also watching this with keen interest. It would be great to have more options in the code editing department.

#7

Thanks for the feedback!

Minor updates:

  • Mulit-caret editing is functional
  • VS2017 build issue raised by chrisboy2000 should be fixed
  • Scheme is derived from Component::findColour rather than hard-coded

The next priority should probably be word-wrap ;-/

With regard to syntax highlighting, it’s a priority to support CodeTokeniser. But another priority is to support general grammars, instead of writing a new parser for every language, or inheriting CppCodeTokeniser. Ideally the editor would maintain its own syntax tree. I’m not sure whether those goals are compatible or if the latter is too ambitious. Opinions?

With regard to extensibility, AceEditor and Sublime offer good models. I plan on parameterizing behaviors (such as where to put the caret after a newline, how to comment a sequence of lines, navigation rules, etc) around a bunch of lambda’s in an EditingMode struct. If scripting is desired, those lambdas could wrap calls to a JavascriptEngine or what-have-you.

#8

you might check out https://github.com/textmate/textmate

#9

make sure it supports multi-language though (e.g. display Chinese characters among English)

#10

I’d also say emojis for marketability purposes, but JUCE’s font rendering doesn’t work with emojis :frowning:

#11

I’m not sure whether those goals are compatible or if the latter is too ambitious. Opinions?

I’d say don’t try to reinvent the wheel. There are a lot of code tokeniser classes available (XML, Lua, CPP, Javascript) and writing another one for a unsupported language is not that much harder than with whatever concept you come up with. The only reason not to support the JUCE code tokeniser is if you decide to use a readymade library for syntax highlighting, eg like this:

http://colorer.sourceforge.net/

(Disclaimer: I haven’t checked it out thoroughly, it was the first thing that Google came back with)

BTW, how would you approach adding Syntax highlighting to your editor? I’ve read the source code and the TextLayout class just holds a StringArray with the lines that are converted to GlyphArrangements and for Syntax highlighting you have to separate the tokens in some way (AFAIK a GlyphArrangement can only be rendered monochromatically).

But the idea of making it expandable using lambdas is awesome. Making it scriptable using the Javascript engine could also be useful, but I think for starters, let people expand it with C++ :slight_smile:

#12

Yup, sounds about right. I don’t see any promising options for actively maintained coloring libraries. Of course for full generality there is Antlr… and associated literature - that requires addition of dynamic libs, so not good for JUCE itself but maybe OK for outside projects. I’ll stick with the JUCE options and for now.

You’re right about the data structure - GlyphArrangement is a single font and color. Once I have a data structure for maintaining the tokens, I’ll modify getGlyphsForRow to return an array of arrangements, and associated colors (styles?). The rendering will need to loop over the arrangements and set the color.

#13

If this could be integrated with a NeoVim backend, or made flexible to support different editor backends, that would be a big win. See e.g. https://github.com/sassanh/qnvim

#14

Nice, looking forward to the Syntax additions. Go Regex! :slight_smile:

I’ve noticed a few quirks:

  • gutter component width is not resized when zooming in out using pinch.
  • when zooming, the top x/y position is not retained, which makes it weird because you loose track where you are.
  • I would prefer a tiny margin between the gutter and the text, but that should not be too problematic…
#15

Wow, thanks for following along. You’re gonna see all the mindless bugs I introduce if you keep that up. I’m aware of the glitches you mentioned and I’ll fix em.

BTW – there is an obstacle to re-using the CodeTokeniser methods: they depend on CodeDocument::Iterator which in turn requires a CodeDocument. I am not sure how to deal with that yet.

#16

Maybe use a CodeDocument as base data holder instead of a StringArray in TextLayout? Not sure if this messes up the multi caret data processing though. It would definitely improve the replacability (I am using a subclassed CodeDocument for my scripting stuff so the transition would be 100x easier if your editor accepts a CodeDocument as data object).

The most hacky solution would be to create one CodeDocument per line from the StringArray member, but I hardly think that’s performant or nice code.

#17

I’ve taken another look at the CodeTokeniser code and it appears that it’s just a wrapper class and the core functions in CPlusPlusCodeTokeniserFunctions.h are templated so you can pass in any Iterator class you want.

Furthermore there’s a generic StringIterator that can be used instead of a CodeDocument::Iterator.

If I understood this problem right, you just need to

  1. Write a wrapper class like the existing CodeTokenisers but with a suitable Iterator for your TextLayout class (I think the StringIterator should be fine, but maybe there’s something I’ve missed).
  2. Use these classes for syntax highlighting. The token type is returned as int, so you definitely need to store it somewhere in the form of
struct Token
{
    String content;
    int tokenType;
    Point<int> position; // or whatever
};

However this definitely means breaking up the GlyphArrangement for one line into multiple ones per token, and this is where I stop understanding what you did :slight_smile:

EDIT: I’ve poked around with the stuff and I got basic syntax highlighting to work in your editor by simply replacing the TextEditor::paint() method with this content:

    auto rows = layout.findRowsIntersecting(g.getClipBounds()
                                .toFloat()
                                .transformedBy (transform.inverted()));
    
    for(const auto& r: rows)
    {
        auto line = layout.getLine(r.rowNumber);
        
        // Everybody knows the 9000 characters per line limit :)
        auto bounds = layout.getBoundsOnRow(r.rowNumber, {0, 9000}).transformedBy(transform);
        
        AttributedString s;
        
        float originalHeight = layout.getFont().getHeight();
        auto font = layout.getFont().withHeight(originalHeight * transform.getScaleFactor());
        
        // Just create this to get the colour scheme...
        CPlusPlusCodeTokeniser tokeniser;
        auto colourScheme = tokeniser.getDefaultColourScheme();

        CppTokeniserFunctions::StringIterator si(line);
        auto previous = si.t;
        
        while(!si.isEOF())
        {
            auto tokenType = CppTokeniserFunctions::readNextToken(si);
            auto colour = colourScheme.types[tokenType].colour;
            auto token = String(previous, si.t);
            
            previous = si.t;
            s.append(token, font, colour);
        }
        
        s.draw(g, bounds);
    }

Now obviously this is just a proof of concept, you definitely want to cache the AttributedStrings and not run this in every paint callback (but even here the performance is not 100% terrible).

#18

Yup, that works! It misses multi-line comments of course, and it’s a little slower since it has to compute tokens on every paint. But not as bad as I might have thought.

In the approach I was aiming for (which basically works but I’m not sure will survive), the text layout is queried for the onscreen glyphs of a particular style; you call getGlyphsIntersecting once for each of your styles. You run the parser to generate an array of “style zones”, which are just Selection's (which now contain a style flag). The parser uses a TextLayout::Iterator, which I added.

To be worth the time, this needs to be a substantive improvement to the existing options and not just a sideways step. I would like for the document to maintain a proper syntax tree that can aid in things like auto-indent and enable extension behavior, but which could also be retro-fitted to the existing parsers.

I’ll clean up the code so that the existing features are well represented, and then I’ll step back and see what seems possible.

#19

The profiler said that 86% of the time in the paint routine is spent in AttributedStrings::draw() so even if generating the tokens in the paint callback is ugly, you will not see a huge performance increase by caching them.

If you generate one glyph arrangement per token style and then draw these, things might get faster, but I am not sure if the AttributedString class does this under the hood too.

Not sure if a syntax tree makes sense, this would require a lot more intelligence from the editor and definitely needs to understand each language you want to use it with. But if this is necessary for code folding, then it might be worth it.

#20

For those interested in text manipulation