118M Queries per Second on Neki

Posted by joshmgross 1 day ago

Counter97Comment49OpenOriginal

Comments

Comment by farazbabar 23 hours ago

In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.

Comment by tomnipotent 22 hours ago

> all the signals from the offline files in mere seconds

That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.

Comment by farazbabar 19 hours ago

Lots of EBS volumes mounted via 25GBPs or higher network, positioned carefully onto a single rack where possible and carefully tuned network/ip/os for both clients and servers. It is not feasible in most deployments as this would not have scaled to real production loads (RAID configurations requiring redundancy alone would slow you down, not to mention costs of using that many EBS volumes on extra high network/IO/provisioned IOPS nodes). This was only meant to prove what was possible in AWS at the time.

Comment by tomnipotent 18 hours ago

Not in 2015. 25GBps didn't even arrive until 2016, and dedicated EBS bandwidth didn't arrive until 2017.

Comment by jeffbee 19 hours ago

Your statement about SSD availability and performance does not fit with my personal recollections of history. Smoking fast SSDs were completely vanilla items you could order from anyone in 2015. For example, the Intel DC P35/6/700 series. Each of these was as good as 25 HDDs for linear reads and more like 5000 HDDs for random reads. The U.2 specification for putting SSDs in boxes was formalized in 2011. Outside of the mass market, cloud builders had private SSD designs long before 2015.

Comment by tomnipotent 17 hours ago

Less than 5% of commercial storage capacity was SSD of any flavor in 2015 (NVMe probably less than 1%), and the price-per-GB was anywhere from 5-15x. NVMe hardware RAID didn't happen until 2017, and AWS SSD via EBS was capped at 320 MB/s (~500 MB/s with striped EBS volumes). Before 2015 it was capped at 160 MB/s.

In practice even NVMe SSDs at the time were maybe 10x faster for sequential R/W and maybe only 200-500x for random I/O. Most databases in 2015 were limited by serial dependency chains like walking a B-tree to figure out what nodes to read next and couldn't max out what an NVMe was capable of. Not to mention we were stuck with libaio in 2015 that made O_DIRECT ops synchronous.

So the OPs claim of reading multiple terabytes of data in seconds on AWS in 2015 literally wasn't technically possible.

SSDs simply were not as ubiquitous in 2015 as they are today, and I didn't know anyone using them outside of mission critical systems were the cost made sense.

Comment by kinduff 20 hours ago

What happened next?

Comment by farazbabar 19 hours ago

That's a good question, I was very high up in the leadership, couple levels below the CIOs and the biggest surprise for the team was that I could write even a single line of code. In large organizations, changing direction is not always easy so I could not get them off of using a data platform, which to be honest was acceptable because we could not ask the team to write such custom code for each job going forward, I did manage to shift them towards flink which I think was better suited to our actual real-time needs.

Comment by stephenlf 22 hours ago

I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.

It’s cool to see the flip side of that argument. Planet scale is answering the question, “what does it look like when you DO fan out your workload to >100 nodes?”

There’s a place for both technologies. Very cool stuff.

[1] https://youtu.be/ONxwjqFjP3A?is=awlEJwGLxQmRE25i

Comment by malisper 18 hours ago

> I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.

Do you have the timestamp where they are talking about this? The claim doesn't pass the smell test for me. If you're talking about latency, then perhaps. On throughput, I don't understand how a single-node system could deliver higher throughput than a three-node system

Comment by titanomachy 21 hours ago

IME people usually go multi-node for availability and durability reasons, not throughput.

Comment by jamesblonde 20 hours ago

In 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [ref]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB has now become RonDB, but is still based on a non-blocking 2-phase commit protocol and is GPL-v2.

RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.

[ref] https://www.slideshare.net/frazerClement/200-million-qps-on-...

Comment by cbg0 22 hours ago

I did some quick ChatGPT math for the same performance/storage as the benchmark:

Neki 1 primary + 2 replicas: ~$5.0M/month (just the AWS bill)

Google Spanner w/ 3 replicas built in: ~$3.85M/month

Comment by weekendcode 18 hours ago

Its great to see scale and progress, but being closed source is HUGE DEALBREAKER.

Clickhouse is also on the right track of building some amazing opensource integrations with postgres, they have superior*[1] managed postgres looks like from their recent blog. I hope they do some OSS sharded postgres solution.

[1] - https://clickhouse.com/blog/benchmarking-nvme-managed-postgr...

Comment by saisrirampur 17 hours ago

Sai from ClickHouse here, I lead the Postgres efforts at ClickHouse. Expect news from us on this soon! Many of us here are ex-Citus and have done this for Postgres before.

Comment by ahachete 17 hours ago

If you want an OSS sharded Postgres, just use Citus. Feature-full, proven, mature, boring.

Comment by weekendcode 15 hours ago

its not feature-full. like schema changes locking, coordinator node..

Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.

Comment by ahachete 6 hours ago

> schema changes locking

Schema changes need locking... everywhere. Citus is no different. And they need proper design everywhere. If you mean that it requires distributed transactions, well, yes, again: expected and solved. Not even all sharding solutions support this.

> coordinator node

Not sure what the problem is here. If what you mean is that a single coordinator, even with an HA replica, can saturate, that's true, but you can add multiple "query routers" (that's our name in StackGres, see [1]).

> Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.

For example, we have a customer that ingests some 30TB/day, and it's ramping up towards 200TB/day of ingestion. On 24 worker nodes.

[1]: https://stackgres.io/doc/latest/administration/sharded-clust...

Comment by saisrirampur 14 hours ago

Most Citus workloads were 10s of TB with largest at around a few PB or so. Heap was a couple PB, back then, if I remember correctly. It is a brilliant piece of technology that supported mission critical workloads across mid/late stage startups to huge enterprises. The planner/executor are very advanced supporting a multitude of features and decade of intricate effort.

The biggest problem of Citus was migration effort, transition from single node to multi-node was not trivial. Here I’m not talking about single table use-cases, more classic relational, multi-tenant apps with 100s to 1000s of tables. This is partly expected with most sharding technologies, though.

Sharing some insights based on my multiple years of experience working with Citus!

Here are few customer use-cases I could found:

https://docs.citusdata.com/en/v10.0/get_started/what_is_citu...

https://info.citusdata.com/rs/235-CNE-301/images/Citus_Data_...?

https://www.youtube.com/watch?v=F6df3HV6kP0

Comment by danbruc 23 hours ago

87.3 % served from cache. Does that mean it returned a result existing in the cache because the very same query was executed before? Probably still a relevant result, if you have to process millions of queries every second, it seems not unlikely that you will see a lot of repeated queries. But at that point you are measuring cache performance more than query performance. But unless you run some standardized query benchmark, a single queries per second number is not that informative anyway because query complexity and therefore execution time can span many others of magnitude. Looking up a name by ID and aggregating across a billion rows from seventeen tables joined together are both a single query.

Comment by samlambert 1 day ago

It cost $250,000 to do this run but it feels worth it.

Comment by jeffbee 1 day ago

That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).

Comment by dataviz1000 23 hours ago

  > 512 shards, each with one Postgres primary each on an r8g.16xlarge
  > 480 Neki routers, each on its own 8xlarge instance
  > We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267.

  Component                      Detail                           Monthly  Hourly  16-min burst
  ---------------------------------------------------------------------------------------------
  Shard compute                  512x r8g.16xlarge                 $1.41M  $1,930          $515
  Router compute                 480x r8g.8xlarge*                  $661K    $905          $241
  Storage (gp3 floor)            1.22 PiB @ $0.08/GB-mo             $102K    $140           $37
  Storage (io2 floor)            1.22 PiB @ $0.125/GB-mo            $160K    $219           $58
  IOPS (io2, light)              5K IOPS/shard, tiered rate         $166K    $228           $61
  IOPS (io2, medium)             20K IOPS/shard, tiered rate        $666K    $912          $243
  IOPS (io2, worst-case)         231,517 IOPS/shard (0% cache)     $4.56M  $6,251        $1,667
  ---------------------------------------------------------------------------------------------
  Total (gp3 floor)                                                $2.17M  $2,975          $793
  Total (io2 floor)                                                $2.23M  $3,054          $814
  Total (io2 + light IOPS)                                         $2.40M  $3,282          $875
  Total (io2 + medium IOPS)                                        $2.90M  $3,966        $1,058
  Total (io2 + worst-case IOPS)                                    $6.79M  $9,305        $2,481

Comment by svuiv 22 hours ago

your calculation does not account for the load generation and cross-az network cost

Comment by secondcoming 21 hours ago

I see the machine sizes some people are allowed to use and I cry. Here I am being asked to downsize our VMs to only use 32 GB RAM.

Comment by jeffbee 22 hours ago

Sounds more right. Where did $250k come from?

Comment by fizx 22 hours ago

I think Sam is an exec at the company.

Presumably it took more than one attempt, or there's some humanpower in the budget.

Comment by 22 hours ago

Comment by handfuloflight 1 day ago

I did not know men could build such things.

Comment by whalesalad 1 day ago

I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.

Comment by svuiv 1 day ago

We used r8g.16xlarge instances with EBS disks, no nvmes

Comment by whalesalad 23 hours ago

would love to hear more about the ebs volumes, iops/size/raid configuration

Comment by svuiv 23 hours ago

each shard had a 4TiB volume with 65k IOPS and 1,500 mbps of throughput

Comment by rcrowley 23 hours ago

It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).

Comment by 22 hours ago

Comment by AdamProut 1 day ago

I'm curious why the test needed so many router hosts:

512 shards, each with one Postgres primary each on an r8g.16xlarge

480 Neki routers, each on its own 8xlarge instance

That's ~250K queries/sec per router which seems lowish for this type of workload? The routers won't be doing very much (parse query, route it to proper shard?).

Comment by svuiv 23 hours ago

that's over 13k queries/sec per router core, about 50% of it is spent doing syscalls, the other 50%: parsing, doing grpc, tls, go gc, resolving the shards, waiting for the responses

neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming

Comment by _zoltan_ 23 hours ago

> The benchmark was very simple. A single-shard point select, one row fetched per-query by primary key. No writes, joins, or cross-shard queries. The workload that each shard receives is isolated, in that there are no single queries that span multiple shards.

I mean... What's the point of this "benchmark"?

Comment by znpy 1 day ago

If this is closed source then i have zero interest in it.

Comment by jjice 23 hours ago

I believe multigress is the similarly aged open equivalent from Supabase. Haven't used it myself and don't know what the differences are in usability, but I'm a bit more interested in that since it's open.

Comment by samlambert 23 hours ago

Neki right now:

Multiple live shards: yes

Query routing across shards: yes

Online shard splitting: yes

Zero-downtime resharding: yes

Multiple independent shard groups: yes

Data topology management: yes

HA / automated failover: yes

Multi-AZ: yes

Connection pooling: yes

Online schema changes: yes

Workflow-driven migrations/cutovers: yes

Zero-downtime imports: yes

CDC / logical replication: yes

Online Postgres version upgrade workflows: yes

Cross-shard transactions: coming

Multigres today:

Multiple live shards: no

Query routing across shards: no

Online shard splitting: no

Resharding: no

Multiple shard groups: no

HA / failover: yes

Multi-AZ: yes

Connection pooling: yes

Logical replication/import work: in progress

Distributed migration/resharding workflows: no

How it is an it's an alternative? Do you just say things without validating?

Comment by znpy 4 hours ago

Hey, just to clarify: i’m not the person you’re responding to (my comment is gp comment) and i’m not criticising your product and your work. It truly is a great achievement.

However, in general i would not trust a product that is not open source (foss).

I do like what grafana has been doing for example, with licensing options at the two extremes: gpl/lgpl v3 on one side and proprietary hosted saas version on the other.

We run our own grafana/lgtm stack at work, we’re outgrowing it and we’re evaluating the saas offerings.

But then again, not criticising your work and product, you know better than me about it and about your business, just writing down my proverbial two cents.

Comment by jjice 23 hours ago

Comes off a bit rude, especially from the CEO.

I get that PlanetScale has the currently superior product, and I listed the closest thing to it. You're right it's not ready, but in the case that the parent had mentioned of closed-source making it a deal breaker, it's the best alternative.

Neki looks like a good product, and I trust that something coming from the folks behind Vitess is going to be a damn fine offering, but some of us don't love tying our workloads to fully proprietary products. I wouldn't even want to host something like this myself, but knowing there's a path out of the proprietary hosting (which we've seen time and time again is something we should plan for) is critical for a lot of us when choosing solutions as important as our data layer.

If I needed a truly sharded PG at the moment, Neki is probably the best option I could find.

As for the "alternative" argument specifically, it's not on par at the moment, you're right, but that's their goal and they're attempting to fill the same void. In the case of someone who has a requirement their solution be open source, then Neki is flat out of the running too.

I'm rooting for a good Postgres sharding solution to come out. Your team appears to be doing some good work. I would also like to see where both Neki and Multigres are in five years.

Comment by samlambert 22 hours ago

Not being rude, I just do not like the work of my company being written off and under appreciated with comments like yours. Saying it's an open source alternative when it is clearly not is incredibly disingenuous.

Anyone can start a project and say it's an alternative until it achieves the same results is not an alternative. They have raised 10x more venture funding than my company, and we have out executed them with hard work. I will absolutely defend that work.

Supabase hire companies to write fake AI slop to slander competitors. https://dev.to/sivarampg/how-a-neutral-supabase-vs-convex-co... They also claim ownership over projects they dont contribute to https://x.com/jrdntgn/status/2022003873171620160 https://news.ycombinator.com/item?id=45321794 (they have no vitess maintainers on staff now and they did not then either)

This is who you are carrying water for.

Comment by jjice 22 hours ago

> I just do not like the work of my company being written off and under appreciated with comments like yours

Not sure how I wrote your product off. I spoke about how it wasn't open, which it isn't.

> Anyone can start a project and say it's an alternative until it achieves the same results is not an alternative. They have raised 10x more venture funding than my company, and we have out executed them with hard work. I will absolutely defend that work.

Good on your team. I believe the animosity here is mostly in our differing opinions of the word "alternative".

> This is who you are carrying water for.

I am not "carrying water". I listed another company. I've never used a Supabase product.

All this because I mentioned something that was open and in the same lane as your new product.

How's this: I won't refer to it as an alternative because it isn't up to at feature parity yet. That's fair. I will however say that you've taken multiple angles unrelated to the original core of my first reply's parent that was that Neki is not open source. That is true and many of us will see it as a downside when chosing a product. That isn't bad. That's just a thing that is.

Comment by 20 hours ago

Comment by spongeboi 23 hours ago

unfortunately they haven't been able to move forward w/ the project, it can't even shard yet

Comment by jjice 23 hours ago

It appears to be under active development, but you're right that sharding doesn't appear implemented. I'm excited to see what they can build out over the next few years.

Comment by throw10920 12 hours ago

What company do you work for that makes its tech decisions based purely on ideology without regard to engineering merits?

Comment by noir_lord 23 hours ago

Someone posted a twitch conversation yesterday about this, I poked around on the page realised there was no open source version and noped out immediately.

I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.

Comment by jeffbee 1 day ago

The fact that you can just pay to scale out point reads is not news to anyone.

Comment by AdamProut 1 day ago

yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).

Comment by 1 day ago