I should have been clear. Yes it is avoidable but only by adding more checking machinery. The bare language doesnt help.

In Go etc. If a struct doesn't a field foo then there will not be a field foo at runtime. In JS there might be. Unless you bring in libraries to help prevent it.

You are relying on someone remembering to use zod on every fetch

It's a convenience and a labor saver, so of course it's fundamentally at odds with security. It's all trade-offs.

No, once-valid input can be rejected after a period of depreciation.

What actually makes sense is versioning your interfaces (and actually anything you serialize at all), with the version designator being easily accessible without parsing the entire message. (An easy way to have that is to version the endpoint URLs: /api/v1, /api/v2, etc).

For some time, you support two (or more) versions. Eventually you drop the old version if it's problematic. You never have to guess, and can always reject unknown fields.

ORMs love to put people and projects in the situation where it's much easier for data types in your program to match exactly with the tables in your database, but reality often requires them to be decoupled instead. That "create user" request contains "isAdmin" smells like someone mapped that exact request to a table, and just because the table has "isAdmin", the data type now also has it.

Except in the case of encoding/xml garbage, that's really a limitation of the Go parser. It's a very non-strict parser and doesn't actually validate that the input conforms to XML. Think of it more as a query language for a stream of XML fragments.

What? It's about parsers in Go's standard library.

1. As shown in the article, exposing the internal model to APIs directly is a security footgun. I've seen sites get hacked because the developer serialized internal objects without any oversight just like in the article and accidentally exposed secrets.

2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI).

In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems.

All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs.

Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs.

Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end.

Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult.

There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.

> also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway.

Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea.

Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.

I don't know about Go, but in Java and .NET world this is trivially solvable with libraries like MapStruct. If you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC):

  record Profile(int id, String login, boolean isAdmin) {}

  interface UserMapper {
    // arrays are just one example, plain models and plenty
    // of other data structures are supported
    Profile[] usersToProfiles(User[] user);

    // other mappers...
  }

  class UserController {
    //
    @GET("/profiles")
    Profile[] getUserProfiles() {
      var users = userRepo.getUsers();
      return userMapper.usersToProfiles(users);
    }
  }

It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read.

https://mapstruct.org

Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.

I imagine that one of the points of a solid protocol buffers library would be to align the types even across programming languages. E.g. explicitly force a 64-bit integer rather than "int" relying on the platform. And to have some custom "string" type which is always UTF-8 encoded in memory rather than depending on the platform-specific encoding.

(I have no idea if that is the case with protobuf, I don't have enough experience with it.)

> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )

I think calling it a convention is misleading.

In Python, you can access an _field just by writing obj._field. It's not enforced, only a note to the user that they shouldn't do that.

But in Go, obj.field is a compiler error. Fields that start with a lowercase letter really are private, and this is enforced.

So I think it's better to think of it as true private fields, just with a... unique syntax.

> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )

Go actually ties visibility to casing, instead of using separate annotations. And it will not serialise private fields, only public.

Python has no concept of visibility at all, conventionally you should not access attributes prefixed with `_` but it won't stop you.

Not just weird, it’s dangerous to do this - to easy to miss validation as fields are added over time. Better to explicitly validate all fields IMO.

> The reason its like that is that Go philosophically is very much against the idea of annotations and macros, and very strongly about the idea of a clear upfront control flow

Annotations have no control flow, they just attach metadata to items. The difference with struct tags being that that metadata is structured.

I find the most annoying thing to be that there is no standard for separate values or field in the tag string. Sometimes you use a comma (json) sometimes a semicolon (Gorm IIRC) etc. This has caused me several headaches.

I understand your feeling. There are many magical frameworks like e.g. Spring that do these things and it's super hard to figure out what's going on.

The solution is usually to have an even better language. One, where the typesystem is so powerful, that such hacks are not necessary. Unfortunately, that also means you have to learn that typesystem to be productive in language, and you have to learn it more or less upfront - which is not something that Google wanted for golang due to the turnover.

Go's simplifications often introduce complexities elsewhere, however, as this article demonstrates with the complexities of correctness of a stringly-typed DSL.

There's no free lunch here, and the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.

Starting to get feeling that 80/20 design is not good thing. Many things seems to be driven by worse is better but, looking at things like Climate Change... was it worth it?

> struct tags greatly reduce the boilerplate code required to map fields to fields.

Are you somehow under the impression that Go is unique in having a terse way to map fields to fields?

> It’s really quite novel once you understand it.

It's the opposite of novel, putting ad-hoc annotations in unstructured contexts is what people used to do before java 5.

It's not very novel. There's far better ways of solving this than allowing a random string to be embedded as aux information to a struct field. Examples: F# type providers, or OCamls PPX system for extending the language in a well defined way. Macro rewriting systems also allow for better safety in this area.

This allows you to derive a safe parser from the structural data, and you can make said parser be really strict. See e.g., Wuffs or Langsec for examples of approaches here.

If you use the same struct in both an HTTP API and an ORM, you're Doing It Wrong in my opinion. These should be completely separated. Exactly to prevent accidental leaking or injection of data.

Covered in the article along with a potential snafu - adding anything else (`-,omitempty`) turns it into a key "-", not the "ignore this field" indicator.

You can just make an IsAdmin() public function to expose it.

The correct conclusion is: https://news.ycombinator.com/item?id=44337330

The problem of trying to ensure that each parser behaves the same for all input is twofold: - JSON and XML specifications are complex, lots of quirks. So not feasible. - Does not solve the fundamental issue of the processing layer not using the same data that is verified in the verification layer.

Note: the processing layer parses the original input bytes, while the verification layer verifies a struct that is parsed using another parser.

Processed: Proc(input) Verified: VerifyingParser(input)

> The fact that it allows case-insensitive key matching is just insane.

It's probably a side effect of what is IMO another bad design of that language: letter casing determining field visibility, instead of using a keyword or a sigil. If your field has to be named "User" to be public, and the corresponding entry in the JSON has all-lowercase "user" as the key (probably because the JSON was defined first, and most languages have "field names start with lowercase" as part of their naming conventions), you have to either ignore case when matching, or manually map every field. They probably wanted to be "intuitive" and not require manual mapping.