Hanjun Cho

oss · 2026-05-11

Fix score layer quantization for sequence classification models - Qwen3 (VL) Reranker

#vllm#Quantization#Reranker

Purpose

Fix FP8/NVFP4 quantization bug for sequence classification models (e.g., Qwen3 Reranker).

The score layer created by as_seq_cls_model() is a dynamic classification head (output_dim=1) with no checkpoint weights. Passing quant_config to this layer causes:

  • FP8: all scores return 0.0 (Marlin tile alignment violation — output_dim=1 not divisible by tile_size=64)
  • NVFP4: model load crash (only weight_packed registered, .weight access raises AttributeError)

Fix by passing quant_config=None to the score layer (so LinearBase uses UnquantizedLinearMethod), and removing quant_config from temporary ParallelLMHead instances that are created only to extract token embeddings and immediately deleted.

Test Plan

  1. Existing BF16 test:
python -m pytest tests/models/language/pooling_mteb_test/test_qwen3_reranker.py -x -v
  1. FP8 online quantization smoke test:
from vllm import LLM
hf_overrides = {
    "architectures": ["Qwen3ForSequenceClassification"],
    "classifier_from_token": ["no", "yes"],
    "is_original_qwen3_reranker": True,
}
llm = LLM(model="Qwen/Qwen3-Reranker-0.6B",
          hf_overrides=hf_overrides, runner="pooling",
          quantization="fp8")
outputs = llm.score("What is the capital of France?",
                     ["Paris is the capital of France.",
                      "Berlin is the capital of Germany."])
for o in outputs:
    print(o.outputs.score)  # should be non-zero

Test Result

Before (vLLM 0.16.0, FP8):

doc[0] score = 0.000000
doc[1] score = 0.000000
doc[2] score = 0.000000

After (patched, FP8):

doc[0] score = 0.972774   # Paris — capital of France (highest)
doc[1] score = 0.896133   # Eiffel Tower — related
doc[2] score = 0.705444   # Berlin — unrelated (lowest)

Accuracy (BF16 as ground truth, 100 query-doc pairs)

Model FP8 Spearman FP8 Top-10 NVFP4 Spearman NVFP4 Top-10
Qwen3-VL-Reranker-2B 0.994 100% 0.950 90%
Qwen3-VL-Reranker-8B 0.993 90% 0.962 90%
Qwen3-Reranker-0.6B 0.986 90% 0.839 80%
Qwen3-Reranker-4B 0.993 90% 0.925 70%
Qwen3-Reranker-8B 0.995 100% 0.966 80%

Latency (P50 ms, batch=16, seq_len=2048)

Model BF16 FP8 NVFP4
Qwen3-VL-Reranker-2B 226.2 168.3 146.7
Qwen3-VL-Reranker-8B 768.6 496.1 376.8
Qwen3-Reranker-0.6B 131.6 112.3 103.9
Qwen3-Reranker-4B 470.0 323.8 269.8
Qwen3-Reranker-8B 757.1 485.1 367.5

GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM 0.16.0, CUDA 13.0.

Related Issue: #33970

Link ↗