Hanjun Cho

oss · 2026-05-11

[feat] Add Qwen3 MoE support to TensorRT backend

#TensorRT-LLM#Qwen3

Implements TensorRT-LLM support for Qwen3 MoE models with architectural differences from Qwen2 MoE:

Implement

  • Add 'Qwen3MoeForCausalLM' to MODEL_MAP for automatic detection
  • Add 'qwen3_moe' to valid model types in QWenConfig
  • Use standard MOE implementation for Qwen3 MoE without shared experts
  • Update weight conversion logic to handle Qwen3 MoE's expert structure
  • Refactor trtllm_modules_to_hf_modules mapping to distinguish between Qwen2 MoE (with shared experts) and Qwen3 MoE (without shared experts)

Testing

  • Successfully creates Qwen3-30B-A3B MoE Engine (FP16, FP8)

ref) vllm vs tensorrt-llm latency (Qwen3-30B-A3B MoE, H100 PCIe GPU, 2048 input_length, 3072 max_length, batch 16)

Precision Framework TTFT (ms) TPS (tokens/s)
FP16 vLLM 152.750 727.76
TRT (torch backend) 127.477 785.82
TRT (trt backend) 73.729 764.59
FP8 vLLM 175.951 993.81
TRT (torch backend) 128.124 1164.08
TRT (trt backend) 84.660 1206.60

Dear Maintainers,

I would like to kindly submit a pull request that adds support for building the Qwen3-MoE model as a TensorRT engine. I would be truly grateful if you could take the time to review it at your convenience.

Thank you very much for your consideration.

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features
    • Added support for the "qwen3_moe" model variant, including activation functions, model mappings, and weight loading.
    • Introduced new configuration options for layer-specific MLP control and decoder sparsity steps.
  • Refactor
    • Enhanced conditional logic for handling Mixture of Experts (MoE) models, improving clarity and extensibility.

Link ↗