| ▲ | simonw 8 hours ago | |
Scaling ChatGPT and scaling GitHub are very different problems. For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard. GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors. | ||
| ▲ | bhouston 8 hours ago | parent [-] | |
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors. Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state. And they have been working on this semi-shared state design for over 10 years now. | ||