Show HN: Typol – Static typing layer for Polars

Posted by mrrpdt 2 days ago

Hello! Wanted to share Typol, a thin static typing layer around Polars that lets you enforce columnar schemas. We've been hesitant in the past to go with dataframes for processing reporting data, especially with Pandas, due to the long-term maintainability burden of tooling not understanding the data we're processing, or the library itself. Polars is well typed and encourages constructing shapes up rather than modifying in-place, so adding schema typing to it seemed like a natural extension. If Polars DataFrames are dicts, then Typol's are TypedDicts.

With Typol, it's easy to define your schemas, which should feel familiar if you're moving from dataclass-style code or from Polars' own schemas, and then build well-typed Polars expressions on these that enforce: (1) valid columns are referenced, (2) column values are used in a valid way for their type, and (3) expressions generate target valid columns in resulting schemas with the correct type.

  class Account(tp.Shape):
      name = tp.dimension(str)
      website = tp.dimension(str)
      uid = tp.dimension(int)

  # Works, with the type: Expr[Account, Account, str]
  email_address = accounts.s.name.str.to_lowercase() + "@" + accounts.s.website

  # Caught statically:
  # Unsupported `+` operation: `BoundDimension[Account, int]` + `Literal["@"]`
  email_address = accounts.s.uid + "@" + accounts.s.website

These types are checked statically using ty, which supports spelling the intersection types needed to infer join results, with a little dynamic enforcement filling in where static analysis can't reach. This allows you to make use of tooling both to check and guide your code (dot completion coming in handy). Existing tools, like Pandera, do provide dynamic verification of dataframe shapes. Whilst this can be good, it bites you at runtime which is well after a problem should be caught, and doesn't provide any tooling benefit.

Typol is great for production data processing pipelines, where narrowing your data to well-defined schemas at each processing stage can be appropriate and powerful. It's not well suited to a lot of data science, where columns generally get added and dropped quite freely. It covers most core Polars expression operations (laziness, arithmetic, strings, datetimes, lists, filtering, joins, aggregations), but we'd love to extend it further, and we'd love for you to try it out!

Comments

Comment by diziet_sma 1 day ago

Very cool! I would imagine this helps LLMs catch errors while refactoring code.

How hard is it to migrate existing pandas/Polars code to Typol?

Most importantly, how did you come up with the name?

Comment by mrrpdt 1 day ago

Definitely one of the advantages of improved tooling visibility, tooling's importance here is obstensibly one of the reasons for the recent acquisition of Astral.

Migrating Pandas to canonical Polars can require some rethinking, since Polars couldn't make a cleaner API model without making things different. From Polars to Typol can really depend: if you have relatively fixed `pl.Schema`s which you join, filter, aggregate, transform between etc., then it should be pretty trivial; the interface is specifically designed to deviate from Polars only where necessary or there is particularly strong case ergonomically. If you really need to add and drop columns all the time, then it might require some more effort. Worst case, you can always have the intermediates in some of your functions still be in Polars, but expose the right shapes with Typol. It's trivial to switch back and forward by doing `typol_df.dataframe` and `tp.DataFrame(MyShape, polars_df)`. This way you're enforcing shape types between sections of your code, and can push that typing inside your functions later.

Naming can end up as a bit of bikeshedding, but if you're interested, right now it needs Ty (until other checkers support intersections), and it's based on Polars, so Ty+Pol seemed the most obvious to users and Googleable.