Skip to content
ModelsThatRun

RT-DETRv4

The fourth RT-DETR: an HGNetv2 detector distilled from a DINOv3 teacher during training, four sizes on COCO's 80 classes, Apache-2.0.

Size

Commercial use OK for size s: code Apache-2.0, weights Apache-2.0. Measured on our RTX 4090: 24.5 ms median, 639 MiB VRAM peak. We scored it 49.6 box AP on COCO val2017. PyTorch weights, 161 MB.

License

Commercial use OK

Code
Apache-2.0checked 26 Oct 2026
Weights
Apache-2.0all sizes, checked 26 Oct 2026

The README links the weights on Google Drive and gives them no separate license, so the repository's Apache-2.0 applies. Training distills from DINOv3 ViT-B; whether DINOv3's terms reach the weights isn't settled.

On our RTX 4090, measured

24.5ms median

p90 25.8 ms, fp32, batch 1, 640x640

VRAM peak
639 MiB
Status
Runs with fixes (fixes/README.md)
Environment, 27 Oct 2026

NVIDIA GeForce RTX 4090, driver 580.173.02, CUDA 13.0, torch 2.14.1+cu130, Python 3.12.9, RT-DETRs/RT-DETRv4@df4412261c6162f62aa4fb84e90b2df3c1b5f223. Baseline 848 MiB. 10 warm-up + 100 timed. Scope: predict(): pre-processing, forward pass and post-processing; input file decoded beforehand.

Accuracy

49.6box APmeasured

COCO val2017, box AP, 5,000 images, fp32, pycocotools==2.0.11 · 26 Oct 2026

Paper 49.8 COCO APREADME, model zoo table

Weights

PyTorch

Size
161 MB
sha256
238a3f6537bf…b5a3ee
Download s weights

Same photos and one score threshold for every model of this task. Failures are marked.

Sizes

SizeLatencyVRAMAccuracyWeightsRunsLicense
s24.5 ms640x640639 MiB49.6 box AP, measured49.8 COCO AP paper161 MBWith fixesCommercial OK
m27.4 ms640x640691 MiB53.4 box AP, measured53.7 COCO AP paper303 MBWith fixesCommercial OK
l31.3 ms640x640768 MiB55.2 box AP, measured55.4 COCO AP paper482 MBWith fixesCommercial OK
x36.9 ms640x640901 MiB56.8 box AP, measured57 COCO AP paper964 MBWith fixesCommercial OK

From the authors

In their words, with their numbers. Ours are in the cards at the top of this page.

Paper

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Zijun Liao, Yian Zhao, Xin Shan, Yu Yan, Chang Liu, Lei Lu and 2 others

arXiv 2510.25257, 29 Oct 2025

Abstract

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a \textbf{Deep Semantic Injector (DSI)} module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a \textbf{Gradient-guided Adaptive Modulation (GAM)} strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of $49.8/53.7/55.4/57.0$ at corresponding speeds of $273/169/124/78$ FPS. Code is publicly available at https://github.com/RT-DETRs/RT-DETRv4.

README

Copied from the authors' repository at commit df44122 on 3 Nov 2026. Numbers and license claims in it are theirs, not measured or checked by us. We removed badges, replaced images with links and adjusted headings.

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models


This is the official implementation of the paper:

🚀 Overview

RT-DETRv4 is the new version of the state-of-the-art real-time object detector family, RT-DETR. It introduces a cost-effective and adaptable distillation framework that leverages the powerful representations of Vision Foundation Models (VFMs) to enhance lightweight detectors.

Figure rtv4 ap latency

🎥 Demo: Furious 7

We challenge RT-DETRv4 with the high-speed scene from Furious 7. Even under extreme motion blur, rapid camera pans, and chaotic lighting, the model maintains consistent tracking and high recall, proving its reliability in dynamic environments.

Video

⚡ Performance

RT-DETRv4 achieves new state-of-the-art results on the COCO dataset, outperforming previous real-time detectors.

ModelAPAP50AP75Latency (T4)FPS (T4)ConfigLogCheckpoint
RT-DETRv4-S49.867.154.03.66 ms273ymllogckpt
RT-DETRv4-M53.771.058.45.91 ms169ymllogckpt
RT-DETRv4-L55.473.060.38.07 ms124ymllogckpt
RT-DETRv4-X57.074.662.112.90 ms78ymllogckpt

📣 News

  • [2025.11.17] Code, configs and checkpoints fully released! Thanks for your attention, and feel free to ask any questions!
  • [2025.10.30] Repo created, and code will be open-sourced very soon!

Content


This repository also supports the reproduction of DEIM, D-FINE, and RT-DETRv2. Simply run the corresponding configuration files.

1. Getting Started

Setup
conda create -n rtv4 python=3.11.9
conda activate rtv4
pip install -r requirements.txt
Data Preparation
COCO2017 Dataset
  1. Download COCO2017 from OpenDataLab or COCO.
  2. Modify paths in coco_detection.yml
train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json
Custom Dataset

To train on your custom dataset, you need to organize it in the COCO format. Follow the steps below to prepare your dataset:

  1. Set remap_mscoco_category to False:

    This prevents the automatic remapping of category IDs to match the MSCOCO categories.

    remap_mscoco_category: False
    
  2. Organize Images:

    Structure your dataset directories as follows:

    dataset/
    ├── images/
    │   ├── train/
    │   │   ├── image1.jpg
    │   │   ├── image2.jpg
    │   │   └── ...
    │   ├── val/
    │   │   ├── image1.jpg
    │   │   ├── image2.jpg
    │   │   └── ...
    └── annotations/
        ├── instances_train.json
        ├── instances_val.json
        └── ...
    
    • images/train/: Contains all training images.
    • images/val/: Contains all validation images.
    • annotations/: Contains COCO-formatted annotation files.
  3. Convert Annotations to COCO Format:

    If your annotations are not already in COCO format, you'll need to convert them. You can use the following Python script as a reference or utilize existing tools:

    import json
    
    def convert_to_coco(input_annotations, output_annotations):
        # Implement conversion logic here
        pass
    
    if __name__ == "__main__":
        convert_to_coco('path/to/your_annotations.json', 'dataset/annotations/instances_train.json')
    
  4. Update Configuration Files:

    Modify your custom_detection.yml.

    task: detection
    
    evaluator:
      type: CocoEvaluator
      iou_types: ['bbox', ]
    
    num_classes: 777 # your dataset classes
    remap_mscoco_category: False
    
    train_dataloader:
      type: DataLoader
      dataset:
        type: CocoDetection
        img_folder: /data/yourdataset/train
        ann_file: /data/yourdataset/train/train.json
        return_masks: False
        transforms:
          type: Compose
          ops: ~
      shuffle: True
      num_workers: 4
      drop_last: True
      collate_fn:
        type: BatchImageCollateFunction
    
    val_dataloader:
      type: DataLoader
      dataset:
        type: CocoDetection
        img_folder: /data/yourdataset/val
        ann_file: /data/yourdataset/val/ann.json
        return_masks: False
        transforms:
          type: Compose
          ops: ~
      shuffle: False
      num_workers: 4
      drop_last: False
      collate_fn:
        type: BatchImageCollateFunction
    
Teacher Model Preparation

Our framework uses a pre-trained Vision Foundation Model (VFM) as the teacher. We use the ViT-B/16-LVD-1689M model from DINOv3.

Configuring DINOv3 Teacher

Specify the paths to your local DINOv3 repository and the downloaded checkpoint in the model's configuration file ./configs/rtv4/rtv4_hgnetv2_${model}_coco.yml and find the teacher_model section:

teacher_model:
  type: "DINOv3TeacherModel"
  dinov3_repo_path: dinov3/
  dinov3_weights_path: pretrain/dinov3_vitb16_pretrain_lvd1689m.pth

Update the dinov3_repo_path and dinov3_weights_path to match your local setup.

2. Usage

COCO2017
  1. Training

    CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --use-amp --seed=0
    
  2. Testing

    CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --test-only -r model.pth
    
  3. Tuning

    CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --use-amp --seed=0 -t model.pth
    
Customizing Batch Size

For example, if you want to double the total batch size when training RT-DETRv4-L on COCO2017, here are the steps you should follow:

  1. Modify your dataloader.yml to increase the total_batch_size:

    train_dataloader:
        total_batch_size: 64  # Previously it was 32, now doubled
    
  2. Modify your rtv4_hgnetv2_l_coco.yml. Here’s how the key parameters should be adjusted:

    optimizer:
      type: AdamW
      params:
        -
          params: '^(?=.*backbone)(?!.*norm|bn).*$'
          lr: 0.000025  # doubled, linear scaling law
        -
          params: '^(?=.*(?:encoder|decoder))(?=.*(?:norm|bn)).*$'
          weight_decay: 0.
    
    lr: 0.0005  # doubled, linear scaling law
    betas: [0.9, 0.999]
    weight_decay: 0.0001  # need a grid search
    
    ema:  # added EMA settings
        decay: 0.9998  # adjusted by 1 - (1 - decay) * 2
        warmups: 500  # halved
    
    lr_warmup_scheduler:
        warmup_duration: 250  # halved
    
Customizing Input Size

If you'd like to train RT-DETRv4 on COCO2017 with an input size of 320x320, follow these steps:

  1. Modify your dataloader.yml:

    train_dataloader:
      dataset:
          transforms:
              ops:
                  - {type: Resize, size: [320, 320], }
      collate_fn:
          base_size: 320
    
    val_dataloader:
      dataset:
          transforms:
              ops:
                  - {type: Resize, size: [320, 320], }
    
  2. Modify your rtv4_base.yml (or the relevant base config file):

    eval_spatial_size: [320, 320]
    

3. Tools

Deployment
  1. Setup

    pip install onnx onnxsim
    
  2. Export onnx

    python tools/deployment/export_onnx.py --check -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
    
  3. Export tensorrt

    trtexec --onnx="model.onnx" --saveEngine="model.engine" --fp16
    
Inference (Visualization)
  1. Setup

    pip install -r tools/inference/requirements.txt
    
  2. Inference (onnxruntime / tensorrt / torch)

    Inference on images and videos is now supported.

    python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg  # or video.mp4
    python tools/inference/trt_inf.py --trt model.engine --input image.jpg
    python tools/inference/torch_inf.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0
    
Benchmark
  1. Setup

    pip install -r tools/benchmark/requirements.txt
    
  2. Model FLOPs, MACs, and Params

    python tools/benchmark/get_info.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml
    
  3. TensorRT Latency

    python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine
    
Fiftyone Visualization
  1. Setup

    pip install fiftyone
    
  2. Voxel51 Fiftyone Visualization (fiftyone)

    python tools/visualization/fiftyone_vis.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
    
Others
  1. Auto Resume Training

    bash tools/reference/safe_training.sh
    
  2. Converting Model Weights

    python tools/reference/convert_weight.py model.pth
    

4. Citation

If you find this work helpful, please consider citing:

@article{liao2025rtdetrv4,
  title={RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models},
  author={Zijun Liao and Yian Zhao and Xin Shan and Yu Yan and Chang Liu and Lei Lu and Xiangyang Ji and Jie Chen},
  journal={arXiv preprint arXiv:2510.25257},
  year={2025}
}

5. Acknowledgement

Our work is built upon RT-DETR, D-FINE, DEIM and Teacher Model DINOv3. Thanks to these remarkable works!

Alternatives

Model output
Input test photo
⇆
InputOutput