Neki – Sharded Postgres
Posted by simon_weber 1 day ago
Comments
Comment by gk1 1 day ago
- Intro: Explains problem but not what is Neki or what it's for.
- Why Neki: Explains alternatives and why they suck but not what is Neki or what it's for.
- How does Neki work: Describes the technical components but not what is Neki or what it's for.
- What you get beyond sharding: Explains how to run/deploy but not what is Neki or what it's for.
- What is a platform preview + Try Neki today: 140 words on the definition of "preview" and links to get started... but not what is Neki or what it's for.
Edit: The landing page has 100x more useful information up front -> https://neki.dev/
Edit #2: They've since added a "What is Neki" section. Good reaction time.
Comment by DrammBA 1 day ago
Edit: your marketing site actually has excellent marketing information.
Edit #2: They've since replaced the blog post with a link to the marketing page. Good reaction time.
Comment by ahoka 1 day ago
Comment by dwb 1 day ago
Comment by fastball 1 day ago
- short paragraph saying "we launched neki"
- short paragraph saying why neki was needed
- paragraphs about what neki is (with a header)
That seems like a very reasonable structure for a blog post like this one. Not being able to get to the third paragraph of a blog post seems like a "you" problem, not a blog problem.
Comment by crooked-v 1 day ago
Comment by simonw 1 day ago
Comment by entrope 1 day ago
Comment by simonw 1 day ago
Why do people post screenshots of an article on social media instead of linking to it?
Sometimes ignorance, but often because the web is such a hostile experience these days - paywalls and subscribe banners and cookie warnings and slow pages.
That's sad.
Comment by owebmaster 1 day ago
If you know that to be true and still add your links everywhere ("helping"), isn't this a spammer's confession?
Comment by simonw 1 day ago
Comment by Atreiden 1 day ago
Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.
Comment by mcpherrinm 1 day ago
The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be.
I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)
Comment by ronfriedhaber 1 day ago
Comment by mcpherrinm 1 day ago
Comment by osalberger 21 hours ago
If you have serializable on each shard and you chop transactions so you have no cross shard transactions, the application is technically still serializable, it just behaves like two independent deployments of the application for different shards
Comment by arrowleaf 1 day ago
Comment by harshitgangal 1 day ago
Comment by fsuts 1 day ago
And now planetscale have made a Vitess like equivalent for Postgres which they have made proprietary
Comment by hiyer 1 day ago
Google is big enough to survive open-sourcing Vitess, k8s, angular, etc. Planetscale isn't.
Comment by samlambert 1 day ago
Comment by Onavo 21 hours ago
Comment by figmert 1 day ago
Comment by ksec 23 hours ago
That Youtube built. I know at the time Youtube was already part of Google but I don't think of Vitess as a Google Stack.
Comment by xyzzy_plugh 1 day ago
Sam, it feels like you're just playing games. It's hard to take anything you say or do seriously. Are you going to OSS this thing? Or have you decided to backpedal?
Congrats on the checks notes closed launch, I guess?
Comment by karlmush 1 day ago
Just launch the product and let it speak for itself. The constant need to take shots at competitors comes across as pretty childish.
Comment by vincnetas 1 day ago
Comment by tpetry 1 day ago
Comment by pm90 1 day ago
Comment by samlambert 1 day ago
Comment by margalabargala 1 day ago
Open source being by far the more important of the two, since it means the user is more insulated by terrible corporate decisions thus making it more suitable for long term planning.
Comment by samlambert 1 day ago
Comment by tvink 1 day ago
Comment by therein 1 day ago
Their CEO always immediately comes in and makes a fool out of himself. He is what you get when you mix narcissism and being a midwit.
Comment by margalabargala 1 day ago
Unlike you, probably, if we're slinging unfounded accusations.
Comment by xyzzy_plugh 1 day ago
Commerical open source is fine. It's nobody else's fault if you raised too much to make it work sustainably.
Comment by samlambert 1 day ago
Comment by cschmatzler 1 day ago
> When we are ready we will release Neki as an open source project suitable for running the most demanding Postgres workloads.
Has this changed?
Comment by 190n 19 hours ago
https://web.archive.org/web/20260516145550/https://planetsca...
https://web.archive.org/web/20260910204345/https://planetsca...
Comment by dkgs 1 day ago
Sad to see this coming from a company that was itself built on the foundation of an open-source project like Vitess.
Comment by samlambert 1 day ago
We maintain Vitess using our company funds. Why are you going after a small company for defending itself against companies like Amazon but not criticizing the reason we have to do this.
I would love to open source Neki and I have not said we won't.
Comment by xyzzy_plugh 1 day ago
Comment by xyzzy_plugh 1 day ago
Comment by fwip 1 day ago
Comment by Twirrim 1 day ago
A remarkable own goal given that if they'd literally held their tongue, I'd be at worst indifferent.
Comment by samlambert 1 day ago
I want to let you know that I have customers that pay PlanetScale millions of dollars and have told me they came to the product because of my Twitter account. So you will never make me feel bad about this.
Comment by fwip 21 hours ago
Comment by kolanos 1 day ago
Comment by Yannik_Sc 1 day ago
They claim to provide unlimited IOPS (I/O operations Per Second).
I would like to get some of these drives myself. Sounds like a technical miracle.
On the other hand claiming such miracles makes me wonder which other aspects in this are are actually a working miracle or rather a mythos/false marketing claim.
Comment by hanspagel 1 day ago
Comment by ehfeng 1 day ago
Comment by erulabs 1 day ago
Selfish doubts aside, congrats to Planetscale on the launch!
Comment by imjosh-dev 1 day ago
Comment by vt_throwaway 1 day ago
Comment by khy 1 day ago
Comment by imjosh-dev 1 day ago
Comment by 1121redblackgo 1 day ago
Comment by ahknight 1 day ago
Comment by dvtkrlbs 1 day ago
Comment by kaycebasques 1 day ago
Comment by kelp 1 day ago
Comment by devy 1 day ago
Comment by samlambert 1 day ago
Comment by levkk 1 day ago
- PgDog is open source
- It runs anywhere, not just your cloud
- It's built for Postgres, not a MySQL pivot
Comment by samlambert 1 day ago
Comment by ripe 1 day ago
Found this from neki.dev.
Comment by fastball 1 day ago
Comment by Tepix 1 day ago
Apparently not.
Comment by kelp 23 hours ago
However we have a whole team working to improve Postgres and we always prefer to contribute those efforts to the Postgres community.
Comment by haul_up 13 hours ago
Comment by wewewedxfgdf 1 day ago
Comment by gk1 1 day ago
Comment by eile23 1 day ago
Comment by imjosh-dev 20 hours ago
Comment by aamederen 1 day ago
Comment by imjosh-dev 1 day ago
Comment by ahachete 22 hours ago
> This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck.
Apart from being able to use any node as a coordinator (and you can load balance them to avoid having "multiple connection strings), there's a new pattern which effectively allows you to have as many coordinators as you want. They are effectively "data-less" nodes. We have devised and implemented this pattern in StackGres [1].
> Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus.
Adding nodes (infrastructure) is what operators solve. In StackGres, adding new nodes means editing one/two characters from your YAML file: the integer number that represents the number of workers that you have.
Shard rebalancing is fully built-into Citus as a UDF, which you can call (manually or in an automated manner) over Postgres protocol.
> Managing backups is also external to Citus, so operators still need to build the proper infrastructure
Agreed, but it's also solved (see distributed backups in StackGres [2]).
> PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially.
Unsubstantiated why. I assume it's because of the assumption that a proxy model is superior than Citus. To which I have to say that Citus model is also a proxy model, where the proxy just happens to be Postgres, which unsurprisingly, speaks Postgres protocol. Sure, there are nuances that we could debate in this area and we can say that Citus is not a "pure proxy", but that doesn't lead to concluding that a proxy model is better --it's arguably not.
[1]: https://stackgres.io/doc/latest/administration/sharded-clust...
[2]: https://stackgres.io/doc/latest/administration/sharded-clust...
Comment by tjohnell 1 day ago
Comment by arrowleaf 1 day ago
Comment by klaussilveira 1 day ago
Comment by imjosh-dev 20 hours ago
Comment by SkiFire13 19 hours ago
Comment by ragall 1 day ago
Comment by aabhay 1 day ago
Comment by spongeboi 1 day ago
Comment by levkk 1 day ago
Comment by ragall 1 day ago
For example: point 08 says "Assign different tables or workloads to different shard groups" and point 02 says "Split hot shards as workloads grow". How do those interact ? Can a single table be split across multiple shards ? If so don't you need 2pc to enforce primary key constraints ?
The intro pages just read as AI slop.
Comment by kamikazechaser 1 day ago
Comment by vira28 1 day ago
Comment by samlambert 1 day ago
Comment by OrangeDelonge 1 day ago
Comment by theanirudh 1 day ago
Comment by tedd4u 1 day ago
Comment by samlambert 1 day ago
Comment by phoenix24 1 day ago
Comment by Hasz 1 day ago
Comment by karuppiah7890 1 day ago
Comment by xnx 1 day ago
Comment by jbonatakis 1 day ago
Comment by kelp 1 day ago
Comment by jbonatakis 1 day ago
Comment by kelp 1 day ago
Vitess today is not the Vitess of 2021. About 70% of the current codebase was written in the last five years, based on lessons we learned running it at scale.
Neki is built for Postgres from the ground up, by the engineers who did that work.
On the Multigres comparison, I'll admit we get a bit salty, mostly because "Vitess for Postgres" invites the comparison. But to date there's no public evidence of a working sharding implementation in Multigres. Their sharding design doc landed this week, and it lists cross-shard query planning and resharding execution as explicitly out of scope.
Neki does sharding today, including online resharding that switches traffic without a maintenance window. It isn't open source, but you can spin up a cluster and try it yourself.
Comment by uberman123 1 day ago
Comment by karlmush 1 day ago
Comment by samlambert 1 day ago
Comment by samlambert 1 day ago
Comment by rs_rs_rs_rs_rs 1 day ago
Comment by samlambert 1 day ago
Comment by ezekg 1 day ago
Comment by _1tan 1 day ago
Comment by tedd4u 1 day ago
Comment by IceWreck 1 day ago
Comment by kelp 1 day ago
Comment by vt_throwaway 1 day ago
Comment by dveeden2 1 day ago
Comment by MrDrMcCoy 1 day ago
Comment by manlymuppet 1 day ago
Comment by behole 1 day ago
Comment by samlambert 1 day ago
Comment by xyst 1 day ago
I am looking for interdimensional and interuniversal scale. Which of you trust fund babies has a startup which is working on this problem?
Comment by samlambert 1 day ago
Comment by _kidlike 1 day ago
Comment by parthdesai 1 day ago
Comment by tonyhb 1 day ago
They build good stuff and care.
Comment by mbesto 1 day ago
Comment by buremba 1 day ago
Comment by browningstreet 1 day ago
Comment by samlambert 1 day ago
Comment by cwisecarver 1 day ago
Comment by JeremyNT 1 day ago
Based on what I've read so far, with his shit-talking the competition, he seems like... a bit of an asshole.
Comment by overfeed 1 day ago
Never heard of the guy before today, but his dickish, edge-lord aura (on X and in this thread) makes me not want to meet him. If I had to choose between buying from him, or some other random dude, I'll hear out the random dude first.
Comment by tgma 1 day ago
If you have such luxuries, I bet you don't have enough scale to be really profitable for him.
Comment by overfeed 1 day ago
I can think of at least 4 other ways of scaling horizontally without switching DB engines. I'd have to be mightily constrained to have this particular product as the only option.
Comment by tgma 1 day ago
Comment by overfeed 18 hours ago
Comment by __s 1 day ago
Comment by samlambert 1 day ago
Comment by parthdesai 1 day ago
(I'm a paying PS customer)
Comment by tombert 1 day ago
Comment by parthdesai 1 day ago
Comment by samlambert 1 day ago
Comment by mbesto 1 day ago
PS - Just some advice that I learned in sales 101 early in my career...focusing on shit talking the competition generally isn't a strategy that works well.
Comment by omnicognate 1 day ago
"You're not wrong Walter, you're just an asshole." - The Dude
Comment by prisonguard 1 day ago
Comment by ragall 1 day ago
Only petty people can hold that view.
Comment by margalabargala 1 day ago
No one accused you of dishonesty.
Comment by nateberkopec 1 day ago
Comment by halilBB 1 day ago
Comment by arpinum 1 day ago
Comment by fatata123 1 day ago
Comment by nicechianti 1 day ago
Comment by vaibhav2614 1 day ago
The pitch is compelling. I wonder how many teams will be able to operate sharded database setups in production as a result of this.