Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts):

  git clone --recursive https://github.com/SharpAI/SwiftLM.git
  cd SwiftLM

  swift build -c release

  # Trick to copy in that missing mlx.metallib file
  uv run --with mlx-metal python -c "
  import importlib.metadata, pathlib, shutil
  d = importlib.metadata.distribution('mlx-metal')
  metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib'
  shutil.copy(metallib, '.build/release/')
  print(f'Copied {metallib} -> .build/release/mlx.metallib')

  # Now start the server (downloads 69GB Qwen model)
  .build/release/SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

  freed pointer was not the last allocation

the Python mlx-metal trick is actually what's crashing it. The mlx.metallib from pip is a different version of MLX than what your Swift binary was built against. It gets past the startup error but then corrupts the GPU memory allocator at inference time → freed pointer was not the last allocation.

Use the version-matched metallib that's already in the repo:

cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/ .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 This is the exact metallib that was compiled alongside the Swift code — no version mismatch. Future pre-built releases will bundle it automatically.