Product Launch
anonymous
2 points
43 comments
Postedabout 1 month agoActive20 days ago
Freetype Port
twitter.commvplibrarydevelopers
Key Features
freetype portingfontconfig portingharfbuzz portinggraphite porting
Tech Stack
Key Features
Tech Stack
In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
But great work on this nonetheless.
> What is the actual catch with Fil-C?
Two things:
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
That's exactly my point.
sudo compiled with Fil-C: uses pam compiled with Fil-C, and all of pam's dependencies are compiled with Fil-C, so the whole thing is memory safe.
sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe. pam is quite big and pulls in other unsafe dependencies
Well it is, it's just that it doesn't magically make PAM safe either.
We're not disagreeing about anything technical, I just think it's slightly unfair to say Rust isn't as safe as Fil-C based on that. It is as safe; it just can't automatically make all C code safer like Fil-C can (and CHERI, etc.).
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not people actually create valuable libraries is an open question. But the energy is definitely there. Its just not necessarily pointed in the direction of economically useful, long lived software.
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." This is a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
The tech landscape was completely different back then. Linus himself stated that had BSD been free at the time, he wouldn't have started working on a free MINIX clone. He worked to solve a concrete problem (the lack of a free UNIX-like kernel), he didn't partially rewrite something that exists and works well just to "create his own ecosystem."
Its weird talking to people in this thread. I get the impression lots of folks would really like it if the actual C code making up linux stayed in place forever. Like it would be a good thing if humans exploring the cosmos in the year 3025 were still logging in using pam, written in C. And not just C, but the current lines of code we have right now. Why? Because! Because C is the best at things. Things like dynamic linking! Because don't change what works. Because if you rewrite something you might get bugs. Because C is fast and efficient. Oh, never mind fast and efficient - lets compile it with Fil-C even though it makes it really slow, just so long as we don't have to change the code!
I think gnu/linux could be way better than it is now, in lots of ways. I get the impression that we - as a species - have barely scratched the surface of the design space for operating systems. The UNIX model is a really early design attempt. Is it the best design that will ever exist? Of course not. Do you know how the unix socket API came about? It was an intern / grad student at berkley who happened to be at the right place and right time. They invented an API and its basically never been changed since then. Because its the right API? No. Just because they were there and it was too hard to change it later.
I want proper capabilities. I want a microkernel. I want faster syscalls. I want storage that's relational, not just file based. I want atomic disk operations. I want programs which can be migrated between computers while they're running. I want supervisor trees. I want data that can be moved or shared between computers. I want my programming languages which can talk to each other using something richer than the C ABI. I want debugging tools that work with lots of languages. I want to be able to download and run programs from the internet safely, without risking all my data. This is all technically possible today.
But the idea of just making a bug for bug compatible, drop in replacement of a single tool like sudo is met with so much hostility and distrust from the community. Sometimes I think I live in a different world than most people. I think things can be better. I want things to be better. But we can't get there if we can't accept any change.
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
In general dynamic linking support is something that rust, go and C# could definitely improve upon. As I understand it, Rust does support dynamic linking for pure rust code. It just doesn't have a stable ABI yet. So dynamic linking is only supported within the same version of the rust compiler.
Taking a look at pam, it looks like there's 2 parts:
- The configuration parser & dynamic loader, which would be trivial to rewrite in rust or C# or something.
- The actual dynamically loaded modules, and the ABI bridge to call into them.
I think the obvious way to port pam to rust or C# would be to rewrite the core. Then just statically link in all the standard, built-in pam modules (like pam_unix). And - at least for now - handle "external" modules in exactly the same "yolo C" dyld way they are handled today. Yes, that means a dozen lines of boilerplate unsafe rust code across the C ABI bridge. It wouldn't be quite as safe as Fil-C or C# JITting for everything. But its within my personal safety paranoia tolerance. It would certainly be much safer than pam is today, given currently every single line is in yolo mode. It would also mean those modules wouldn't need to all be rewritten in rust all at once, or at all.
If you want a fully safe replacement for the pam bridge maybe wasm+wasi could work here? I'm not sure. I certainly hear you that this is a problem in rust, go and c#, and I'm not entirely sure what a good solution would look like. Well, other than rust stabilizing its ABI. At the pace rust moves at, that could be any decade now.
> I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
Forget every program for a moment. C#'s JIT would certainly be fast enough for pam. And most linux programs don't make such heavy use of dynamic linking to work. So they could be AOT compiled.
On the topic of performance, I'm not sure I want to pay the fil-C performance overhead for all 500+ processes on my system either. I assume the performance ceiling for fil-c is going to be somewhere around the performance of other well optimized GC runtimes. I don't know of any GC language that does better than about a 2x memory usage overhead.
Call me crazy, but for general computing, I like the speed of native code.
Not quite. To safely run a Rust program against a Rust dynamic library, you had to have compiled and linked that program against exactly that version of the Rust library. This is necessary for fundamental Rust type system reasons. It’s not simply a matter of stabilizing ABI. It’s not like updating the library without relinking would work if you used exactly the same rustc version - it would still be unsafe.
So no, Rust doesn’t support dynamic linking in any meaningful way.
> And most linux programs don't make such heavy use of dynamic linking to work.
This is not true at all.
Linux relies on dynamic linking extensively in two ways:
- A dynamic library may be updated independently of its dependents. You can’t do this in Rust, see above.
- Lots of Linux programs rely on the dynamically loaded module architecture like what pam does. All of the scripting language runtimes do this for example.
Rust has full blown safe dynamic linking - so long as you use rust, and the the same compiler version, for everything. What it doesn't have is runtime libdl support for that. Which would be a problem for the current pam architecture.
I'm not particularly convinced you couldn't just re-architect pam to use static linking though. Or a scripting language.
This is UB in rust as far as I understand:
1) you compile bar using version 1.5 of libfoo.so at link/compile time
2) you update libfoo.so to version 1.6. It has a lot of changes, but is still source compatible and some kind of reasonable precautions were taken, like that struct layouts are the same
3) you run bar with the new libfoo.so version 1.6
This is UB even if you use the same compiler.
If this is UB then that’s not “full blown safe dynamic linking”.
Note that Fil-C does safely support this. This is safe in Fil-C even if you don’t take appropriate precautions (like if you change function signatures, change struct layouts, etc - it’ll still be safe and have well defined outcomes in Fil-C).
Native AOT, and the oldie NGEN only do dynamic linking.
https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
.NET Native can do both,
Applications are statically compiled, libraries are compiled into WinRT components, COM's evolution, introduced in Windows 8.
https://learn.microsoft.com/en-us/windows/uwp/dotnet-native/...
Just watched the 2024 talk by the way, quite interesting.
The only three Linux kernel powered systems that have actually achieved The Year of Linux Desktop success among non technical users.
Maybe. There's rust version of libc and SSL already. On the other hand fil-c won't help with dependencies written in C++, Fortran, and lots of others.
The "less safe" gets very context dependent.
IIRC Fil-C is already documented to work with C++. It's not immediately obvious to me that there are any fundamental reasons Fil-C wouldn't work for Fortran and "lots of others" as well; a fairly large chunk of its work is done via a pass on LLVM IR [0] so anything that compiles to LLVM could reap Fil-C's benefits with (probably?) relatively minor alterations to frontends.
It sounds like Fil-C combines the ergonomics of C with the performance of Python.
It might have a beautiful niche for safely sandboxing legacy code. But I don’t see any compelling reason to use it for new code. Modern GC languages are much more ergonomic than C and better optimised. C#, Java, JavaScript and Go are all easier to write, they have better tooling and they will probably all perform as well or better than Fil-C in its current state.
That is a matter of opinion.
Personally when I write C, I miss memory safety. And thread safety. I miss non-nullable types. I miss having a good way to return results. And generics. And tuples. And real enums (sum types). I miss having a sane build system. I miss having IDEs that just work with my projects. I miss having a well designed standard library. I miss being able to hit build and have it more or less work on every OS. I miss having a good package manager. And a sane macro language. I wish I didn't need to write header files. And I wish my header files didn't need to be parsed MxN times. I miss fixed integer types just being built into the language - like u8/u16/u32/... And real strings. And on and on.
C compilers are fast and mature, so that's a plus. And C binaries are tiny. I mean, C has aged surprisingly well given the language is 53 years old. But its a really old language. And it shows. I would never use it for a new project, because it seems worse in every way compared to languages like Zig, Odin and Rust.
> and the performance seems to be more like Java or C# than Python.
Alright, I was exaggerating. But I still don't want my linux workstation to feel like a big java program.
> The point is that you don't have to rewrite anything, so even the ergonomics of C don't matter.
As I said in another comment, I really agree with this. Fil-C makes a lot of sense to me if the goal is to simply take existing legacy software and recompile it in a more memory safe way. But I don't think that's how the developers see it. I think they want fil-C so rust developers shut up, and they can go back to writing C code forever and ever and ever.
I thought, "for sure this is a use-after-free" ... but it happened to be safe because some function performed an undocumented incref, and the return value of that function was kept alive.
So this is definitely a high priority thing to port!
To my understanding, there's no way to do this: Fil-C is basically a managed runtime for native code, so the FFI implications are similar to those for Go or any other intrusive managed runtime. Which is to say that Fil-C could offer an FFI, but without losing the blanket exit-instead-of-memory-unsafety guarantees it aims to offer.
Not affiliated with Hacker News or Y Combinator. We simply enrich the public API with analytics.