CodeEditor and Tokenizer - user types?


The CodeEditor looks nice, and I’m keen to use it. Looking at the code though, I don’t see how/where it would detect defined types and be able to syntax color them?

I suppose an ideal is to parse the code (or includes, in C++ - although I’m thinking of lua) and pick out defined classes and types, maybe even instances.

I’d probably be fine with just picking out the types and global variables I’ve pre-defined - I guess that’s just subclassing a tokenizer? Is that setup for inheritance? So I can do a

BruceScriptTokenizer : public LuaTokenizer



I’ve not addressed that yet, but it’s on the to-do list!


i support several coding syntaxes in my new editor (soon to release now) and all-in-all i found the new code editor to be terrific. one problem i had with the tokenizer routines is that the iterator didnt let me see what the previous char was sometime i need more context to colorize the current token than what the iterator provides because its looks forward only so i had to write hack code to do this.
for example to colorize Lisp you might only want to color a special form symbol if it is directly adjacent to an open parens but you cant ask the interator what the last char was. im not sure why readNextToken couldnt just take a Position and then that method could see the context (backwards and forwards) if it needed to. positions were a godsend, i do lots of parsing backwards over expressions and this isnt even possible with iterators.
i also wanted to have more than == and != comparisons for positions (ie stop if pos1 is > pos2) and a few other things


I suspect I’ll probably have to tweak the way the tokeniser works, but haven’t space in my brain to think about it just yet. When the jucer’s component editor is working, I’ll move on to its code editor and start figuring this stuff out.