Server Inference

• GitHub - sgl-project/sglang: SGLang is yet another fast serving framework for large language models and vision language models.

  • GitHub - sgl-project/sgl-learning-materials: Materials for learning SGLang

• GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs

• GitHub - flashinfer-ai/flashinfer: FlashInfer: Kernel Library for LLM Serving

Triton-inference-server

• Serving a Torch-TensorRT model with Triton — Torch-TensorRT v1.4.0.dev0+d0af394 documentation

ONNX Runtime

https://github.com/microsoft/onnxruntime-inference-examples

https://github.com/microsoft/DeepSpeed-MII

https://github.com/microsoft/onnx-script

GitHub - open-mmlab/mmdeploy: OpenMMLab Model Deployment Framework