GitHub's new Git storage architecture

Posted by DAlperin 6 hours ago

Counter64Comment9OpenOriginal

Comments

Comment by yakkomajuri 1 hour ago

If I'm reading this right, this is (really solid) future planning but won't help with the current outages?

I'd have thought so from the headline + initial content but then they say that the horizontal scale techniques are basically carrying over from the existing architecture (where it seems the problem is?) and this is mostly to help with what they call "vertical scale" (intra-repo scalability). They also admit right after that the most active repos today will receive short bursts of at most dozens of writes.

So this seems like really useful planning for a near future that likely helps prevent more issues but isn't really meant to help current issues. Am I reading this right?

Of course improvements like this are always useful either way. Not dunking here at all.

Comment by rockwotj 27 minutes ago

personally I see more issues with actions over pushing

Comment by pksunkara 42 minutes ago

Since they wouldn't need to actually have the inactive repositories on disk anymore, I reckon it should solve the horizontal scale issue too by reducing the compute/disk capacity that needs to be maintained.

Disclaimer: I am building a code forge myself at https://juju.bi, so I am trying to be objective.

Comment by pksunkara 50 minutes ago

Most of this is similar to what https://enroute.sh already did scaling Git. https://enroute.sh/blog/everyone-is-doing-git-on-s3-wrong

Comment by cedel2k1 15 minutes ago

At this point GitHub should likely just try snatching the Enroute guys :)

Comment by rochacon 2 hours ago

So that is why I got a bunch of 500's during push this week :)

Hope it helps improve reliability, it was getting hard to defend GitHub with all the recent outages.

Comment by mococa 1 hour ago

So we can have one more 9?

Comment by cr125rider 37 minutes ago

It will be nice having maybe a few at the beginning of the number!

Comment by chjail-11 2 hours ago

As if more developers means better code. Wasn't it Bonhoeffer who said stupidity is when smart people don't question authority?