One of my tasks is to take various AI models the data science team has produced/fine-tuned and make them runnable with optimizations on GPUs (TensorRT, vLLM, Triton Inference Server, etc.), involving conversion, deploy, smoke-test, packaging, documentation, all within a uniform framework. The documentation includes model-provenance, simple deployment instructions (usually a Docker run), simple deployment test. The DevOps team takes this and tailors it to Kubernetes / helm-charts or whatever the target environment needs.

I had worked on some earlier deployment environments for a few models where we focused on one version of Triton and one runtime technology. Even upgrading to a newer Triton version was brittle, involving a lot of command-line changes at various phases. This was written mostly pre-Claude-getting-real-good. I decided that probably Claude had matured and was way better at understanding the particulars of AI-model-GPU-deployment-and-technologies. I worked with Claude to make the framework a much more lightweight wrapper, ignorant for the most part of a lot of the deployment internals.

After this refactoring and doing the first couple of models, it's quite amazing at how well Claude can figure things out. For any new model we now set up its "specialization" directory and its documentation subdirectory, point Claude at the proper AI-model files, point Claude to the sample non-optimized inference code and test data, and point Claude to a similar conversion we've done in the past. There a multi-layer class hierarchy dealing with various tiers. I ask Claude to explore the existing conversion/packaging, the model, any documentation that comes with it (a lot of times there are unexpected twists), the sample code, and the desired multiple use cases the model is meant to address, and the test data. Claude has been trained, I'm sure, on a lot of AI model conversion, so it's able to synthesize the full multi-stage conversion/deployment pipeline, come up with appropriate test cases for all the use cases involved. There are usually between 3 and 10 refinements after initial synthesis, fixing outright errors, refinemnts that the data-science team requests after playing with test deployments, etc. The options and pitfalls are vast, and without Claude each preparation likely would 10x or more longer. I just put most details in Claude's hands, and make sure the general framework is good enough to provide external uniformity. When all is working, it takes a couple of hours to make sure the documentation is good.

All this to say that at least for this domain, Claude / AI has been a game-changer and has sped up the process amazingly.