- I don't find skills, I create them

- Keep them organised in software repos that you install with symlinks for all coding harnesses that you have. Progressive disclosure based on the frontmatter does the rest.

- I make sure they work with AI evals. Think of them like integration tests to prove behaviour. They're useful to optimize your flows. I try to make my skills be mostly a translation between natural language and good small fast tools that they call.

- I change them as a new problem arises. Not just because.

Skills can't be eaten by model capabilities if skills represent a workflow that is custom to my team or my person.

I wrote about a good mental model in the past:

https://alexhans.github.io/posts/series/evals/building-agent...

▲

stingraycharles 2 hours ago | parent [-]

People always say this about the evals, but I find it hard to have a practical implementation of such a thing where you won’t end up spending 100x the amount of time on the evals than building the skill itself.

Like, ok, I have a debugging skill, now how do I make evals except for the most trivial things?

▲

RicDan 12 minutes ago | parent | next [-]

You don't. If you're using skills to force the AI to fullfill some must criterias, it's not going to work. Must criterias need deterministic checks -> be it hooks or what not.

This is also my biggest gripe with AI. I.e. for specifications, no matter what hype machine I tried, it never fulfilled my criterias, which are: easily verifiable, concise, small specs. Hence I built https://github.com/RicardoMonteiroSimoes/Yamlet initially for claude code, but then decided to use extend it for pi.dev. I now have a dedicated docker image for pi.dev, that only contains Yamlet plugin, and whenever I work on spec I spin it up.

The end result is a .yaml file that easily works in git + git diff, so that I can then proceed with the technical specs-

▲

TobTobXX 2 hours ago | parent | prev [-]

A skill should only document behaviour the LLM didn't/couldn't exhibit on its own.

So you take your failed case (eg. working with gdb or whatever), write a skill and then test for that failed case.

	▲	hakunin 2 hours ago \| parent [-]
		There are also skills that help LLM do the thing it can do without the skill, but faster (by cutting out unnecessary discovery). I guess for such skills the fail case is "being slow"? I imagine many fail cases can burn a lot of tokens/usage/time because failing LLMs can be very persistent. Maybe some upper bound (turn count, timeout) would help too.