Schema-Driven Development

ziyadmir 2 hours ago | parent | next [-]

I can't help thinking the author would actually enjoy a fully protobuffed environment :)

This reminds me of the data-driven programming and state modeling they taught us at Waterloo: https://news.ycombinator.com/item?id=35758587

A few thoughts:

1) Sorry in advance; I spent too long at Google and am tainted by that. Maybe the proto/thrift/avro style of typed schemas + generated client libraries is overkill, but it made some of this automatic. You probably don’t need the binary wire format everywhere, but schema annotations (e.g., field-level PII tags, etc.) and a centralized automatic registry for these objects did help.

2) I’m mostly curious why I don’t feel this pain heavily anymore. Maybe I’ve aligned on (or am just OK with) the other side — of engineers paying the tax and dealing with it? I still code often, but heavily optimize for speed of delivery which seems to be one metric that helps career wise. Not sure yet how to square those thoughts yet.

I think this is refreshing and I like it when people push for cleanup and improvements to how we develop. I'll suggest engineers at work try and see what the feedback is.

▲

dneighman 2 hours ago | parent [-]

The schema-driven development isn't really about a particular format tbh. It's a set of principles that can be applied to any development process. Decide on the shape, then derive the rest. Protobufs fit into this category.

I really loved protobuf in the past. I've used them at Square and implemented gRPC at Opendoor. Protobufs have been really helpful with the runtime compiled codegen. They are heavy though, and there are some issues.

Getting a data schema to go all the way from the backend to the front end is challenging with Protos. You can use proto over json but the clients on the fe are not well looked after. You have to fight the tooling (or add copious amounts of it) to have the same contracts flow all the way through your code.

In my experience, protos work well inside a company for backend, but they're not really made for public use (agent to agent) or runtime use.

Even with the strong contracts in protobufs, which I've personally seen improve teams using them! There's still the questions... but, what does X or Y mean!?

One of the issues with writing meaning into your proto files in terms of field or message options is that only the authors can annotate it for the authors purposes. What we need is a way for authors to write it an annotate it, but _also_ for humans and agent consumers to add their own meaning to the schemas, fields and uses. Use-cases, gotachas, relationships, service annotations.

We've been re-building slightly different versions of the same things in each company / application, and every time we do, the reason for a field, the meaning is usually buried in email or slack threads, remembered only by "those who came before". In the age of agents where they need context, that type of tribal knowledge is holding us back.

▲

ziyadmir an hour ago | parent [-]

Interesting — is the annotation graph the actual product, and why key it on objects rather than workflows? In any case, I agree that this graph has network effects and compounds.

Maybe what we want is for agents to both consume annotations and write them back into an open annotation registry that overlays a schema registry?

	▲	dneighman an hour ago \| parent [-]
		This is what I've been working on for a little while and what helped me to condense my thoughts on it to this point. I've tried to separate out the _principles_ vs the product though which is why I haven't spoken about a product.

▲

abewheeler 3 hours ago | parent | prev | next [-]

This is neat! There's definitely a missing formal, human layer sitting above code. Two thoughts come to mind:

1. For applications with a database, the database schema for core models strikes me as the canonical definition, except it's missing descriptions. 2. It feels like this is only a partial solution. Why not go all the way? It seems to me that more aspects of code needs this level of governance than just data models.

	▲	dneighman 3 hours ago \| parent [-]
		Absolutely agree that many things need governance, human and agent. It seems to me that mostly, the things downstream of the data structures: storage, input, transport etc are derived from the data or can be applied as a layer on top. I'm not trying to boil the entire ocean for this but my thesis is that if you can agree on the shape of the data and what that means, you've got a great foundation for all work that follows.

▲

dneighman 4 hours ago | parent | prev [-]

[dead]