Pandas Should Go Extinct

I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.

Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.

Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.

Don't skip your profiling

https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/

The i5-1135G7 has only 4 cores. Hyperthreading is enabled, so it reports 8 "virtual processors".

Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).

If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.

> TODO: rewrite this entire service

remains.

If I still need to do db-like things in python I think duckdb is better.

I think we can be certain that reduced cash use didn't cause the pandemic, and a common cause for both is also hard to imagine. That only leaves the pandemic causing a reduction in cash use. Am I missing something?

The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"

Something going wonky on their blog, where two posts got their links swapped.

In many cases I’ve found directly using python primitives to be less confusing than pandas.

Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.

I do not understand the aggressiveness towards previous evolutionary steps. Who knows, may they come up with newer methods. I still use it. I also use DuckDB and Polars. DuckDB is more useful to me these days from either. Still Pandas has a place in my arsenal. Mostly because I know it better and I like it.

There is also libraries for doing type checking.

Hoping OP can fix this on their end so the url has the expected content. Whoops!

DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.

I'm shocked! Shocked, I say!

Sometimes you just want something that works with all the things. That's pandas. But, like the bamboo eaters, it wont be long...

For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.

But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.

And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api

Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries