Pytorch Monarch

Apparently PyTorch oxidation has started.

> Monarch is split into a Python-based frontend, and a backend implemented in Rust.

Other than that, looks like a quite interesting project.

I assume this is similar to Ray?

  class Example(Actor):
     @endpoint
     def say_hello(self, txt):
         return f"hello {txt}"

  procs = this_host().spawn_procs({"gpus": 8})
  actors = procs.spawn("actors", Example)
  hello_future = actors.say_hello.call("world")
  hello_future.get()

  @ray.remote(num_gpus=1)
  class Example:
      def say_hello(self, txt):
          return f"hello {txt}"

  actors = [Example.remote() for _ in range(8)]
  hello_object_refs = [a.say_hello.remote("world") for a in actors]
  ray.get(hello_object_refs)

Cool! Essentially Fortran coarrays from 2008.

I made my own single controller PyTorch extension [1], though mines doesn't yet support cross node communication. I found it interesting to compare how Monarch makes things performant. I believe Monarch also uses cloudpickle for code to be shared among all nodes, which is probably the only way to performantly have various nodes execute work as that ends up being a one time setup cost. I found the fanning out of sending messages from the single controller to be really interesting, so the controller is unlikely to be the bottleneck besides any synchronous operations.

As far as things that might be a performance loss here, one thing I'm wondering is if custom kernels are supported. I'm also wondering how much granularity of control there is with communication between different actors calling a function. Overall, I really like this project and hope to see it used over multi-controller setups.

[1] https://github.com/alyxya/mycelya-torch

This seems strictly less powerful than Jax, which comes with a powerful compiler that optimises how cross-node communication is conducted.

FB should create a pytorch foundation and set it free before they fuck it up.

Interesting - this seems to target a different layer than services like Tinker (https://thinkingmachines.ai/blog/announcing-tinker/). Monarch provides the infrastructure primitives while Tinker is a managed finetuning service. Could someone build something like Tinker on top of Monarch?

"Our Rust-based backend facilitates our performance, scale, and robustness — we amply use Rust’s fearless concurrency in Monarch’s implementation"

Found a few typo's. The em dash makes me suspect an LLM was involved in proofreading

this could become a major thing in coarray world, but the issues start already:

> ...Note that this does not support tensor engine, which is tied to CUDA and RDMA (via ibverbs).

I.e. yet another CUDA married approach: the issue is not ibverbs but the code shows they use GPUDirect RDMA, going from there this can only get worse - more CUDA dependencies. There would have been OpenUCX.

> This lets us avoid single-host bottlenecks, effectively using the whole mesh as a distributed cluster for message forwarding. (Cite scalability numbers here.)

In case someone that can fix this is reading here

Beowulf then.

It is a nice project. I have questions.

- Is this similar to openMPI?

- How is a mesh established? Do they need to be on the same host?

> Monarch lets you program distributed systems the way you’d program a single machine, hiding the complexity of distributed computing:

There are some infamous tech based on the "hiding" paradigm. PHP comes to mind. By hiding how the http request/response cycle actually works it fostered a generation of web developers who didn't know what a session cookie was, resulting in login systems that leaked like a sieve. Distributed computing is complicated. There are many parameters you need to tweak and many design decisions you need to take to make distributed model training run smoothly. I think explicit and transparent architectures are way better. Distributed model training shouldn't "feel" like running on a single device because it isn't.