Syntax Highlighting Is a Waste of an Information Channel (2020)

I've used the Kate editor for years, it has a short list of strings that are auto-hilited... and I use frequently. If only I could edit and to that list ... wherever it's located!

If only there were a way I could highlite -one- string, and then use a single key to move from that instance to the next!

All that said, I'm one who appreciates information density! How about coloring branching code paths/call stacks?

My keyboard has a concept of "layers," which allows each key to map differently depending on the layer. I've seen this used to make a numpad or to have a QWERTY and DVORAK layer. What if highlighting was the same? Instead of competing for priority over the color channels, developers could explicitly swap layers?

I agree with the broader point of the article that color is underused, but the state of the art has moved way past what the author’s tools are currently configured to provide.

I used to reply that the color pens made it easier to keep context such as what teacher said was important, what I found difficult, when in the note I had an "aha!" moment, side comment from me, Q&A asked by student during lecture, or how certain things written down now is related to the point made earlier/later in the lecture/notes.

Text (note) is the content but our (at least mine) attention are not really made for plain text. There's so much more you can play with visual information.

To solve it we need to be able to describe the structured content of a document without rendering it, and that means we need an embedding language for code documents.

I hope this doesn't sound overly technical: I'm just borrowing ideas from web browsers. I think of my project as being the creation of a DOM for code documents. The DOM serves a similar function. A semantic HTML documents has meaning independent of its rendered presentation and so it can be rendered many ways.

CSTML is my novel embedding language for code. You could think of it like a safe way to hold or serialize an arbitrary parse tree. Like HTML a CSTML document has "inner text" which this case is the source text if the program the parser saw. E.g. a tiny document might be `<Boolean> 'true' </>`. The parser injects node tags into the source text, creating what is essentially the perfect data stream to feed a syntax highlighter. To do the highlighting you print the string content if the document and use the control tags to decide on color. This is actually already how we syntax highlight the output from our own CLI as it happens. We use our streaming parser technology to parse our log output into a CSTML tag stream (in real time) and then we just swap out open and close node tags for ANSI escape codes, print the strings, and send that stream to stdout.

Here's a more complicated document generated from a real parse: https://gist.github.com/conartist6/412920886d52cb3f4fdcb90e3...

Before that, I completed arguably the 2nd largest syntax highlighting: ISC Bind9 (most versions)

https://github.com/egberts/vim-syntax-nftables

https://github.com/egberts/vim-syntax-bind-named

My secret weapon was using a smaller highlighted syntax to project even faster completion of these larger syntax tree: EBNF

http://github.com/egberts/vim-syntax-ebnf

The real magic trick is that I used S-expression to pull up all the first-encounter/deeply-nested keywords touched to its Vim syntax 'nextgroup=', and region block-offs.

Basically said, I complied complex EBNF into Vimscript zeal and need for pure-deterministic LL(1) syntax tree. (Vim regex is weird, must order by largest static pattern first to most wildly wildcard pattern lastly within single regex string).

Rainbow nest braces, command/statement/keyword/unit/integer coloring.

For my next trick, I need to determine which route to go next (maybe HN can help me here).

- JetBrain's properitary LSP

- VSCode textmate LSP?

- treesitter

- or something more LSP mainstream, if any.

Kinda disappointed that there is no holy grail for both syntax-highlighting and autocompletion.

Was looking forward to adding hint-hover as well.

Also I think it's a bit off the mark to think of it as being a wasted information channel. Redundancy is a feature of human languages, because our languages are not optimizing solely for density. A bit of redundancy helps our brains pattern match more consistently, almost like a form of forward error correction. Syntax highlighting is like that, at least for me, where it makes a big difference in seeing the structure at a glance, and more overly complex coloring rules thwart that for me. Like I don't want to be trying to match up rainbow shades of parens.

i feel like HN might be the only place i can explain the "AI syntax highlighting" angle and have somebody get it. the Codemaps UX isnt exactly tuned for the exact form factor of syntax highlighting, but the general idea of "hey you can selectively highlight code based on what you're currently trying to do, and btw also reorganize your filesystem accordingly" is kinda cool and would love ideas on how to best express it.

[1]: https://github.com/willcrichton/flowistry

Give me proper indentation and the languages use of parenthesis, semicolons, etc and I'm good and I can find everything I'm looking for

Color could tell you whether your

  ,',',var

In run of the mill blub code not doing any metaprogramming, indentation is more than visible enough for indicating levels of nesting.

Years ago I used to work on C++ code in a commercial editor called Source Insight [1]. At its default settings, it would do things like:

1. Show function and class names in HUGE fonts in declarations and definitions, so you always knew what was a declaration as opposed to a use

2. Show nested parentheses with the outermost parens being biggest and getting smaller as you got further in

3. Show comments in a proportional sans-serif font instead of a monospaced one so that you could tell where the comments were even if you have color blindness

Those features, along with having a C++ parser and code relationship visualizer much faster than the Visual Studio of the day without having to parse ahead of time (a la ctags), made Source Insight a near standard in my company. I still miss it on occasion.

[1] https://www.sourceinsight.com/feature-details/

They are in my JetBrains IDE, IntelliJ. Just use proper tools instead of toy text editors? It's free and Open Source even.

(Some are difficult to implement with current systems, but there are ways to design systems that they do work (possibly with a IDE rather than just a ordinary editor).)

That way the colour can be defined at write time, languages don't need to implement them, they can be like whitespace and you can use color for whatever you want. Colour would be ignored by both compile and runtime of course... unless it wasn't

For example, in a loop body, putting the cursor on a "continue" statement will highlight all other loop control statements along with the "for" statement the pointed at "continue" is associated with. This helps massively.

Shame the author missed this.

To some extent syntax highlighting distinguishes bad syntax from good; it can be configured to flag deviations from token-level or even grammar-level syntax. For instance, not closing a string literal, or bungling a numeric token with nondigits.

In all the syntax that is good, it mainly just assigns colors to lexical categories, which have a role within syntax.

It does that while retaining some channel capacity for other functions,like matches for regex searches still being highlighted, and visual selections of text being highlighted. (The channel is not exclusively occupied by syntax coloring to the point that no other information can squeeze in there concurrently.)

https://snapwiki.miraheze.org/wiki/Snap!

>Zebra Coloring: This feature, introduced in BYOB 3.0, is representative of the careful attention to the user interface in BYOB/Snap!. When same-color blocks are nested, it's hard to see the borders between them. Zebra coloring assigns two colors to each palette category, the normal color and a lighter color. When same-color blocks are nested, the outermost one has the normal color, and an inner block takes the opposite color from the one just outside it. The text inside light color blocks is black, instead of white as usual.

Snap Manual, Page 11:

https://snap.berkeley.edu/snap/help/SnapManual.pdf#page=11

>The round block rounds 35.3905… to 35, and the + block adds 100 to that. (By the way, the round block is in the Operators palette, just like +, but in this script it’s a lighter color with black lettering because Snap!alternates light and dark versions of the palette colors when a block is nested inside another block from the same palette:

>This aid to readability is called zebra coloring.)

Snap Manual, Page 150:

https://snap.berkeley.edu/snap/help/SnapManual.pdf#page=150

>Non-goal: Emulate the terse APL syntax. It’s too bad, in a way; as noted above, the terseness of expressing a computation affects APL programmers’ sense of what’s difficult and what isn’t. But you can’t say “terse” and “block language” in the same sentence. Our whole raison d’être is to make it possible to build a program without having to memorize the syntax or the names of functions, and to allow those names to be long enough to be self-documenting. And APL’s syntax has its own issues, of which the biggest is that it’s hard to use functions with more than two inputs; because most mathematical dyadic functions use infix notation (the function symbol between the two inputs), the notion of “left argument” and “right argument” is universal in APL documentation. The thing people most complain about, that there is no operator precedence (like the multiplication-before-addition rule in normal arithmetic notation), really doesn’t turn out to be a problem. Function grouping is strictly right to left, so 2×3+4 means two times seven, not six plus four. That takes some getting used to, but it really doesn’t take long if you immerse yourself in APL. The reason is that there are too many infix operators for people to memorize a precedence table. But in any case, block notation eliminates the problem, especially with Snap!’s zebra coloring. You can see and control the grouping by which block is inside which other block’s input slot. Another problem with APL’s syntax is that it bends over backward not to have reserved words, as opposed to Fortran, its main competition back then. So the dyadic ○ “circular functions” function uses the left argument to select a trig function. 1○x is sin(x), 2○x is cos(x), and so on. ‾1○x is arcsin(x). What’s 0○x? Glad you asked; it’s √1 − W!.

I didn't actually find the uncoloured circle to be that much harder to spot.

Before looking at the concrete examples, I thought:

> Understanding the syntax similarly isn't hard scanning over the code. But if the words were coloured in a way that didn't correspond to syntax, I expect I would find it distracting, in the same way as those experiments with colour words written in a different colour from what the word indicates. Trying to make use of that second information channel isn't necessarily a good idea.

> I like the aesthetic of syntax-coloured code. But also, it's reassuring. I'm not using it to help understand the syntax, but to confirm that there's software in place that understands the syntax the same way I do.

After: the rainbow parentheses honestly don't help that much, not for a language like LISP anyway. The context highlighting seems to work much better. But then, as I go on through the examples... these are all really doing the same kind of thing that syntax highlighting does! They're about the structure of the code. And I'd have to shift my expectation, but once aligned, it's again something I'd perceive the same way — as something to confirm my understanding rather than eliciting that understanding.

Perhaps being able to switch between colouring modes would change that, but I don't know how quickly I could get used to that technique. And then, the kind of code where most of these things would help is smelly anyway. I already try to avoid this kind of nesting. Maybe import and argument highlighting (which could be used at the same time, along with highlighting for class attributes and closures in Python?), though...

It also highlights shaded variables (in Go), private/exported functions, class members, package names, etc.

This kind of highlighting as a secondary information channel for compiler feedback is great. Color, weight, italics, underlines - all help increase information density when reading code.

One thing I am still asking for: I want to be able to clearly and obviously distinguish mutable state from immutable state. IntelliJ can't do it yet.

1. Performance 2. Developer ergonomics 3. Syntax highlighting 4. Semantic highlighting 5. LSP

that's a lot.

The only language I can think of is Assembly :-)

Neovim can use treesitter for it.

- Making variables, functions,... grey if they're not used - Temporarily coloring variables + their usages when the cursor is on one - Showing wiggled red underlines for syntax errors

Especially the unused code feature is wonderful.

Also most modern IDE’s already contextually highlight usages of what you’ve selected too

The colour channel is being well and truly used to close to it's maximum.

¹https://www.cs.cmu.edu/~emc/15817-s11/lect-2011-03-23.pdf

Here is another use case i found helpful for html: i use a different, dimmer color for the closing html tags.

I now only use my own tree-sitter syntax and my own neovim highlight colors, i literally stripped all the defaults.

I started writting code with literally white text on a black background, and i only added new highlight groups when my brain really needed it. This way every color becomes deliberate and i gradually add in just what i need