Companies like Polars are showing how with Rust, Python developers now have a better, smoother path towards building high-performance libraries.
If you’re interested in learning more about Rust, data and Python, apply to our 🦀🐍📊 event in San Francisco on February 9. We’re lucky to be joined by BCV portfolio company Polars founder Ritchie Vink, Cube founder Artyom Keydunov, Dagster’s head of data Pedram Navid and Lance founder Chang She.
Rust is replacing C as the “backend” workhorse language for high performance Python packages. Why?
Let’s start with the motivating problem: Python is easy to write, but slow to run. It’s so slow that you can’t write high-performance libraries in pure Python, and in particular you can’t write data processing libraries that way. And yet Python is the dominant language for ML and data engineering. So, if you want to write a library for data engineers, ML engineers, etc., you’re going to find yourself in a situation where:
Historically, this meant that would-be library writers had to do one of two things:
“Well,” the C-philes may be asking, “what’s so bad about that?” Maybe most library authors can get the job done by outsourcing the number crunching to numpy or scipy. And in the handful of cases where it’s really necessary, surely they can learn a little C. It builds character.
In practice, it just doesn’t work that well. Being able to outsource things to numpy, scipy and the rest is nice when it works, but having to vectorize every function and not being able to write for loops is a pain. Wondering whether something will be GIL-blocked is a pain. And on and on. Not everything you want to do fits neatly into existing libraries.
OK, so what about Option #2: Why not write the library from scratch in C, and throw on some Python bindings? The problem is that if you’re coming from a Python background, programming in C is going to seem very low-level; it takes some work to learn the language. Null pointer dereferences, buffer overflows, memory leaks… these are just a few of the many fun Ways to Shoot Yourself in the Foot with C, all of which are foreign to native Python programmers.
If only there were a better way. If only there were a language that’s as fast and memory-efficient as C, but which didn’t require manual memory management or garbage collection. And if only that language had great Python tooling and a thriving community of existing developers. If only.
Rust is fast and memory-efficient. It makes parallel and concurrent programming easier. It has great tooling and a friendly compiler. It has a large, happy community of developers. Rust will let you run faster, jump higher and make more friends at school.
And, most importantly, it’s easier for Python developers to learn Rust than it is for them to learn C.
The “first level” experience is better, because it’s easier for newcomers to write “safe” code in Rust. The learning curve is smoother, and lets you pick up more advanced language features over time.
As a result, over the last couple of years, we’ve seen several high-performance libraries with Python frontends choose Rust for their backends. For example:
1) Polars is a fast, highly parallel, memory efficient library for working with DataFrames. Polars author Ritchie Vink considered several different languages for Polars, and ultimately chose Rust. Here’s how Ritchie sees the advantages.
“I think data-engineering/science will remain dominated by a high-level language that connects low-level compiled binaries. Multi-threading, performance in that host language doesn’t matter, as the work will be passed down to the tool they dispatch to.
So we are down to either C , C++, Rust, Zig or Fortran.
Zig is very young, but it might be well set up to become the new C.
Rust, IMO is well set up to become the new C++.
Of these languages, Rust has the best tooling. (Maybe Zig will get there.) …
The borrow checker guarantees safe memory use and safe concurrency. This together with its great tooling (crates.io, pyo3), makes it the best language to build low-level tooling in.
Because it is safe by default, new language learners can also easily start building tools and learn incrementally. I see this happening today, with a lot of Python users writing Rust, getting the job done and being happy with the speedup they gain.
In short, I think Rust has the best correctness guarantees and is a modern systems language.”
2) Lance is a high-performance, low-cost vector database. The founders of Lance, Chang She and Lei Xu, originally wrote the codebase in C++ and decided to switch to Rust later, even though the team had plenty of C development experience. Here how Chang explained it to me.
“For us, the decision to switch to Rust from C++ was primarily the fact that we could be so much more productive without losing performance, and not having to deal with CMake. We learned Rust from scratch essentially and while we were doing that, Lei and I re-wrote Lance in roughly three weeks into Rust (replacing about four months of C++ code). And moreover, each release we had way more confidence releasing new features in Rust without the fear that it was going to segfault every other command.”
Rust isn’t just for data processing — it’s a useful backend for many other Python packages with high-performance requirements. For example:
3) Ruff is an extremely fast Python linter, written in Rust. Here’s Ruff creator and Astral founder Charlie Marsh on the reasons for Rust’s success within the Python ecosystem.
“As a newcomer to systems programming, Rust raised the ceiling for the kind of software I could build. As a language, it lets you (or: forces you to) care about the details that matter when writing performant code: where and when you allocate, how you layout your data in memory, and more. Higher-level languages protect you from these details, and for good reasons; but for certain categories of software, they’re core to what you’re building. Rust gives you the ability to operate at those lower levels without sacrificing abstraction.
Ultimately, though, I believe that much of Rust’s success comes from its tooling (Cargo), and the accessibility of the language and ecosystem itself. I’ve spent most of my career writing Python, TypeScript, and Java, and Ruff was my first attempt to build something in Rust, from scratch. Rust is not an easy language to learn, but in my experience, the difficulty comes from learning new concepts and new modes of thinking, rather than struggling with build issues or opaque errors. For me, Rust made systems programming approachable, which makes me very bullish on its future as a complement to Python and other higher-level languages.”
4) Pydantic is a developer-friendly validation library for Python. The Pydantic team wrote their V2 in Rust, and saw a 20x performance improvement, even for simple models. There are other benefits to Rust besides performance. Pydantic founder Samuel Colvin called some out some of them for me.
“Another point about Rust is that it’s not just faster, it can be significantly easier to write and maintain rock solid code in Rust. In particular, Rust forces you to catch and process every possible error, while Python’s (and TypeScript’s) type system ignores errors, so I can call ‘foobar()’ and I have no idea what exceptions it might raise in what scenarios. I basically have to use trial and error to find out what might fail.”
See this article by [Python Software Foundation’s] Seth Larson, announcing that Python has been added to the US government’s list of memory safe languages. In its response to the US government, the PSF specifically called out Rust and PyO3 and made a point about encouraging more packages to adopt Rust to harden them against memory errors.”
We think we’ll continue to see more libraries with Python frontends for ergonomics and Rust backends for performance. The upshot is that Python developers now have a better, smoother path towards building high-performance libraries.
Are you writing high-performance Python libraries with Rust? Are you or a loved one the victim of a segfault? If so, reach out to Slater at sstich@baincapital.com. And we’d love to see you at our upcoming San Francisco event on Feb 9!
We are delighted to have been part of Rubrik’s journey from data protection startup to data security corporation.
Kamal Shah and Vibhav Sreekanti are serial entrepreneurs whose Prophet AI platform uses automation to cut mean time to response to cybersecurity threats by 10x.
The Cyber Leaders are innovative CISOs and other top cybersecurity leaders, comprising a community that will benefit each member and the Bain Capital network.
