Kefir: Solo-Developed Full C17/c23 Compiler with Extensive Validation

(I thought that the announcement has completely faded, so haven't even checked the replies).

I'll immediately reveal some issues with the project. On the compilation speed, it is unfortunately atrocious. There are multiple reasons for that:

1. Initially the compiler was much less ambitious, and was basically generating stack-based threaded code, so everything was much simpler. I have managed to make it more reasonable in terms of code generation (now it has real code generator, optimization pipeline), but there is still huge legacy in the code base. There is a whole layer of stack-based IR which is translated from the AST, and then transformed into equivalent SSA-based IR. Removing that means rewriting the whole translator part, for which I am not ready.

2. You've outlined some appealing points (standard compliance, debug info, optimization passes), but again -- this came at the expense of being over-engineered and bloated inside. Whenever, I have to implement some new feature, I hedge and over-abstract to keep it manageable/avoid hitting unanticipated problems in the future. This has served quite well in terms of development velocity and extending the scope (many parts have not seen complete refactoring since the initial implementation in 2020/2021, I just build up), but efficiency of the compiler itself suffered.

3. I have not particularly prioritized this issue. Basically, I start optimizing the compiler itself only when something gets too unreasonable (either, in terms of run time, or memory). There are all kinds of inefficiencies, O(n^2) algorithms and such simply because I knew that I would be able to swap that part out should that be necessary, but never actually did. I think the compiler efficiency has been the most de-prioritized concern for me.

Basically, if one is concerned with compilation speed, it is literally better to pick gcc, not even talking about something like tcc. Kefir is abysmal in that respect. I terms, of code base size, it is 144k (sans tests, 260k in total) which is again not exactly small. It's manageable for me, but not hacker-friendly.

With respect to marketing, I am kind of torn. I cannot work on this thing full time, unless somebody is ready to provide sufficient full-time funding for myself and also other expenses (machines for tests, etc). Otherwise, I'll just increase the workload on myself and reduce the amount of time I can spend actually working on the code, so it'll probably be net loss for the project. Either way, for now I treat it closer to an art project than a production tool.

As for compiled code performance, I have addressed it here https://lobste.rs/s/fxyvwf -- it's better than, say, tcc, but nowhere near well-established compilers. I think this is reasonable to expect, and the exact ways to improve that a bit are also clear to me, it's only question of development effort

P.S. JKU is a great school, although by the time I enrolled there the project has already been on the verge of bootstrapping itself.

EDIT: formatting

I have addressed compiler inefficiency in the sibling comment. This is indeed a problem. Empty arrays of such size should be compile-able (there is sparse representation for arrays). However, I would say that this use case is not particularly practical, at least in none of the projects from my test suite this has been an issue.

I am going to hijack this comment for a little rant about libc and independent compilers. Existing libc implementations have varying levels of hostility towards non-gcc/clang compilers. Glibc is probably the worst offender here, musl is the most compliant (but even there are some assumptions).

I am in no way accusing any of libc developers, they got much higher priority things to do than supporting obscure compilers, and this is my problem as a compiler developer, but nevertheless. For instance, glibc may simply override __attribute__ keyword with an empty macro, or omit packed attributes for non GNU C compilers (breaking epoll ABI along the way). Of course, strictly standard-compliant compiler may not have attributes at all, but in my opinion #warning/#error directive would have been more appropriate than silently producing ABI breakage.

Although, I have not engaged with glibc developers on these topics, and mostly documented encountered challenges as patches in my own test suite.

(I'll address the parallel build part in addition to what has been said, but please respond to my email address/sourcehut mailing list for any further communication on these issues).

I should have communicated this more clearly in README. Kefir requires relatively recent GNU Make versions (see dist/Dockerfile for full and complete set of dependencies), as it basically uses make for all project lifecycle tasks. I remember there were some parallel build issues with older GNU Make versions. This could be an issue.

(using throwaway account, as I don't have any access to my main one from this machine)

For out-of-tree build, use KEFIR_BIN_DIR make parameter. I use it quite extensively in https://git.sr.ht/~jprotopopov/kefir/tree/master/item/script... so this should work.

DESTDIR is supported as a parameter of make install: https://git.sr.ht/~jprotopopov/kefir/tree/master/item/instal...

As for the parallel build issues, contact me over the email please

Yes, you can use it as a library. I do not guarantee stable API, however

Upon researching that, I got some questions.

Why does kefir use the shell to invoke itself?

Why does it look for its library under /usr/local/bin/../lib/tls/x86_64/x86_64/libkefir.so ? Shouldn't it know where its supposed to be from the build configuration? Is it intended, that it access a file after trying to unlink it?

    unlink("/tmp/kefir-154946-hu9HGw")      = -1 EISDIR (Is a directory)
    rmdir("/tmp/kefir-154946-hu9HGw")       = 0
    access("/tmp/kefir-154946-hu9HGw/object-file.0.ivTxnw", F_OK) = -1 ENOENT (No such file or directory)

Can you provide me with more details of your system configuration? Dockerfile, ideally. Please send me an email.