Multi-GPU Validation fails

[abhiagwl4262]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

NCCL timeout in multi-GPU training

I trained for 2 epochs on single-GPU machine and then wanted to run the training on multi-GPU machine for faster training but I am seeing NCCL timeout after being stuck for a long time at the end of eval

Epoch 2: 100%|███████████████████████████████████████████████████████| 2289/2289 [19:11<00:00,  1.99it/s, train/lr=0.0001, train/lr_min=4.04e-6, train/lr_max=0.0001^
[  idation DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 303/303 [02:01<00:00,  2.49it/s]
                                                                                                                                                                     

^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[  [rank1]:[E407 10:02:26.551756518 ProcessGroupNCCL.cpp:683] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=492
09, OpType=ALLGATHER, NumelIn=43, NumelOut=172, Timeout(ms)=1800000) ran for 1800056 milliseconds before timing out.                                                 
[rank1]:[E407 10:02:26.551997332 ProcessGroupNCCL.cpp:2241] [PG ID 0 PG GUID 0(default_pg) Rank 1]  failure detected by watchdog at work sequence id: 49209 PG status
: last enqueued work: 49209, last completed work: 49208                                                                                                              
[rank1]:[E407 10:02:26.552039573 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can en
able it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.                                                                                                 
[rank1]:[E407 10:02:26.552111664 ProcessGroupNCCL.cpp:2573] [PG ID 0 PG GUID 0(default_pg) Rank 1] First PG on this rank to signal dumping.                          
[rank3]:[E407 10:02:26.564157461 ProcessGroupNCCL.cpp:683] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=49210, OpType=ALLREDUCE, NumelIn=1,
 NumelOut=1, Timeout(ms)=1800000) ran for 1800069 milliseconds before timing out.                                                                                    
[rank3]:[E407 10:02:26.564359555 ProcessGroupNCCL.cpp:2241] [PG ID 0 PG GUID 0(default_pg) Rank 3]  failure detected by watchdog at work sequence id: 49210 PG status
: last enqueued work: 49210, last completed work: 49209                                                                                                              
[rank3]:[E407 10:02:26.564374605 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can en
able it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value

Environment

• rfdetr=latest
• num-gpus = 4
• Driver Version: 580.95.05
• CUDA Version: 13.0
• 2.9.1+cu128

Minimal Reproducible Example

from rfdetr import RFDETRBase

model = RFDETRBase()

model.train(
    dataset_dir="IE_Z22_043/images",
    epochs=200,
    batch_size=8,
    grad_accum_steps=4,
    task="segment",
    resolution=624,
    output_dir="rfdetr_ie_z22_043_finetuning",
    progress_bar=True,
    # fp16_eval=True,
    early_stopping=True,
    devices="auto",  # required — see note below
    strategy='ddp_find_unused_parameters_true',
    persistent_workers=True,
    device='cuda',
    checkpoint_interval=1,
    num_workers=4,
    warmup_epochs=5,
    pin_memory = True,
    resume="rfdetr_ie_z22_043_finetuning/checkpoint_1.ckpt"
)

I ran with torchrun --nproc_per_node=4 train_rfdetr.py

Additional

No response

Are you willing to submit a PR?

• Yes, I'd like to help by submitting a PR!

[Borda]

Hi @abhiagwl4262 - the NCCL timeout is almost certainly caused by running torchrun --nproc_per_node=4 and passing strategy='ddp_find_unused_parameters_true' + devices="auto" to .train(). That's a double DDP setup -- torchrun initializes the process group, then PTL tries to manage it again, and the ranks desync during the post-validation all_gather calls.

Here's what to change:

1. Drop torchrun -- just run python train_rfdetr.py. PTL handles multi-GPU automatically when you pass devices=4 (or "auto").
2. Remove strategy='ddp_find_unused_parameters_true' -- the default strategy works. Only set this if you hit a specific "parameter not used in backward" error.
3. Check your dataset for images with empty annotations (no bounding boxes or masks in the JSON). That was the root cause in The multi-GPU training program gets stuck #908 -- empty-annotation images cause uneven batch distribution across ranks, which also triggers this exact NCCL hang.

Your updated script would look like:

from rfdetr import RFDETRBase

model = RFDETRBase()

model.train(
    dataset_dir="IE_Z22_043/images",
    epochs=200,
    batch_size=8,
    grad_accum_steps=4,
    task="segment",
    resolution=624,
    output_dir="rfdetr_ie_z22_043_finetuning",
    progress_bar=True,
    early_stopping=True,
    devices=4,
    persistent_workers=True,
    device="cuda",
    checkpoint_interval=1,
    num_workers=4,
    warmup_epochs=5,
    pin_memory=True,
    resume="rfdetr_ie_z22_043_finetuning/checkpoint_1.ckpt",
)

Then launch with: python train_rfdetr.py (no torchrun).

Let me know if that resolves it.

[abhiagwl4262]

@Borda I used without torchrun as well. it still fails. I added ddp_find_unused_parameters_true later to handle the "parameter not used in backward" error. Empty Annotations is something I am yet to try but I kept them deliberately to add negative samples so ideally I want to have empty annotations.

[Borda]

I used without torchrun as well. it still fails. I added ddp_find_unused_parameters_true later to handle the "parameter not used in backward" error. Empty Annotations is something I am yet to try but I kept them deliberately to add negative samples so ideally I want to have empty annotations.

Thank you for confirming. Could you pls update the example with a standard dataset so I can try to reproduce? I also will need to figure out where to run this multi-GPU setup...

[jiesonshan]

It should be related to the version of the Python package. But there will be problems with multi-GPU validation and it needs to be modified

[abhiagwl4262]

I used without torchrun as well. it still fails. I added ddp_find_unused_parameters_true later to handle the "parameter not used in backward" error. Empty Annotations is something I am yet to try but I kept them deliberately to add negative samples so ideally I want to have empty annotations.

Thank you for confirming. Could you pls update the example with a standard dataset so I can try to reproduce? I also will need to figure out where to run this multi-GPU setup...

When I got rid of images with no annotations then I didn't get the NCCL timeout issue but this is a hack not the solution.

I can't share my custom data. Try at your end with any data. Just add some images with no labels.

[Borda]

I can't share my custom data. Try at your end with any data. Just add some images with no labels.

We have a function to generate synthetic dataset, could you please try reproduce it with that one?

[abhiagwl4262]

@Borda I am facing an issue of memory increase as well. So yesterday my training crashed after 10 epochs saying OOM. Is there a memory leak or expected increase in memory during training ?

[Borda]

Borda I am facing an issue of memory increase as well. So yesterday my training crashed after 10 epochs saying OOM. Is there a memory leak or expected increase in memory during training ?

@abhiagwl4262, could you please try reproducing it with that one?