PyTorch: A Reference Language

Posted by matt_d 12 hours ago

Counter74Comment9OpenOriginal

Comments

Comment by ainch 1 hour ago

I think this idea is exactly right. Many ML papers that include pseudocode basically just write out a minimal PyTorch training loop - down to the loss.backward(). It has, along with Python, become something of a linga franca for research in the field.

Comment by ansk 10 hours ago

This kind of sounds like just giving up on making the compiler better.

Comment by ezyang 2 hours ago

One of the reasons why I have been thinking and writing this topic is because there is indeed a malaise among the team, which is along the lines of "AI agents are blowing up coding everywhere, what place does a compiler have in this brave new world?" If compilers truly are pointless, then I would like to update on this and save people time and grief!

But I also don't think compilers are pointless. In this post, which looks at the maximalist AI situation, the compiler resurrects itself as the verifier: the two types of systems do very similar things! You can also defend compilers on more banal grounds: not everyone has a ton of kernel engineers to hand optimize your kernels; token efficient ways of getting good baselines is useful; there's more to ML than transformer models and compiler leverage is really useful there. There's also a historical point which is that torch.compile... by all objective metrics, has really been quite successful!

I do agree it's hard to tell the future. These days, I ask myself, "Do I think this is likely to happen in six months." I don't think compilers die in six months. Ask me again in six months :)

Comment by ansk 10 minutes ago

I can see how LLMs may make compiler development seem futile when it comes to optimizing the local graph structure (ie kernel-sized chunks of the computation).

But optimizing the global structure -- things like defining the boundaries between kernels, the lifetimes of allocations (or when/how to reuse buffers), and scheduling/parallelizing operations -- seems inextricably tied to the type of traditional program analysis used by compilers.

Even tiling/layout decisions, which you mentioned can be optimized by hand or by an LLM in the context of a single kernel, must be optimized in the broader context of the computation to minimize copies/relayouts between kernels.

I think part of the problem is that massive models make the compiler problem very uninteresting, since global program optimization reduces to local kernel-level optimization.

Comment by pjmlp 2 hours ago

The folks submitting papers to SIGPLAN and PLDI seem to be the opinion use of AI technology in compilers will only increase merging both approaches, of natural language as frontend, and backends.

I think many that don't follow compilers haven't yet grasp how much there is already there, from machine learning based optimisation passes, JIT compiler heuristics, and so on, which never have been deterministic to start with.

Comment by antonvs 8 hours ago

The point is there are limits on how much better the compilers can get, without modifying the language to be closer to the target environments. And there are multiple target environments, so a one-size-fits-all solution is not really possible. You have to compromise somewhere - you can’t have a single high level language that automatically produces maximally efficient code for every target, because the targets are not general-purpose computing environments, they have significant constraints.

If what you’re looking for is a way to avoid having to get into the operational details of the targets, and just provide a high level description that somehow gets figured out, then an LLM-based solution is almost certainly going to be your best bet.

Comment by ofou 2 hours ago

this is just https://github.com/tinygrad/tinygrad

Comment by insumanth 4 hours ago

Anyone not doing distributed training across thousands of GPU, pytorch is the best option. Others have an infra team to work on it.

Comment by madhu_ghalame 10 hours ago

The idea makes sense because researchers already think in PyTorch. The tooling should adapt to them, not the other way around.

Comment by dmzxnico 5 hours ago

[flagged]