class

DETRForObjectDetection

extendsObjectDetectionModel

DETRForObjectDetection(config: DETRConfig)

source edit

DETR end-to-end object detector (Carion et al., ECCV 2020).

The first detection model to formulate object detection as a direct set prediction problem, eliminating anchors and NMS. A ResNet backbone produces a stride-32 feature map $(B, C, H', W')$ that is projected to $d_\mathrm{model}$ channels, flattened to a sequence, and processed by a stack of transformer encoder layers. A learned set of $N$ object queries (default 100) is then decoded against this memory by the transformer decoder, with each query producing one class logit vector (K+1, including "no-object") and one box prediction $(c_x, c_y, w, h)$ normalised to $[0, 1]$ .

During training, predictions and ground-truth boxes are matched by the Hungarian algorithm under a cost combining class probability and box L1 + GIoU, and the same combination defines the per-pair loss.

Parameters

configDETRConfig

Frozen architecture spec. Use detr_resnet50 / detr_resnet101 for the paper-cited variants.

Attributes

configDETRConfig

Stored copy of the config that built this model.

backbone_ResNetC5

ResNet trunk through stage 5 (frozen BN), producing a stride-32 feature map.

input_projnn.Conv2d

1x1 conv projecting backbone output to d_model.

query_embednn.Embedding

Learned object queries of shape (num_queries, d_model).

transformer_DETRTransformer

Reference DETR transformer (encoder.layers.{N} with no final encoder norm; decoder.layers.{N} + a final decoder.norm), with per-layer positional re-injection.

class_embednn.Linear

Per-query class head with output dim num_classes + 1 (the extra "no-object" slot is essential for matching).

bbox_embed_MLP

Three-layer MLP producing per-query 4-D box predictions; the final activation is a sigmoid so boxes lie in

[0, 1]

Notes

See Carion et al., "End-to-End Object Detection with Transformers", ECCV 2020 (arXiv:2005.12872). The bipartite Hungarian assignment finds the permutation $\hat{\sigma} \in \mathfrak{S}_N$ that minimises

\hat{\sigma} = \arg\min_{\sigma \in \mathfrak{S}_N} \sum_{i=1}^{N} \mathcal{L}_\mathrm{match}\bigl(y_i, \hat{y}_{\sigma(i)}\bigr),

with the pair cost

\mathcal{L}_\mathrm{match}(y, \hat{y}) = -\mathbb{1}_{c \ne \varnothing}\,\hat{p}(c) + \mathbb{1}_{c \ne \varnothing}\, \bigl(\lambda_{\mathrm{L_1}}\|b - \hat{b}\|_1 + \lambda_\mathrm{GIoU}\, (1 - \mathrm{GIoU}(b, \hat{b}))\bigr).

The final loss reuses the same per-pair terms, with "no-object" matches penalised at reduced weight. Removing NMS / anchors makes DETR conceptually simple but training-data hungry — the paper reports needing 500 epochs to fully converge on COCO.

Examples

Inference returns class logits and normalised cxcywh boxes for every
query:
>>> import lucid
>>> from lucid.models.vision.detr import detr_resnet50
>>> model = detr_resnet50()
>>> x = lucid.randn(1, 3, 800, 800)
>>> out = model(x)
>>> out.logits.shape   # (B, num_queries, K + 1)
(1, 100, 92)
>>> out.pred_boxes.shape
(1, 100, 4)
Training pass with normalised xyxy ground-truth boxes (Hungarian
matching computes the set loss):
>>> targets = [{
...     "boxes":  lucid.tensor([[0.1, 0.1, 0.5, 0.5]]),
...     "labels": lucid.tensor([3], dtype=lucid.int64),
... }]
>>> out = model(x, targets=targets)
>>> out.loss.shape
()

Used by 2

Constructors

dunder

init

→None

__init__(config: DETRConfig)

source edit

Instance methods

forward

→ObjectDetectionOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, pixel_mask: Tensor | None = None)

source edit

Run DETR.

Parameters

xTensor

(B, C, H, W) image batch.

pixel_maskTensor | None= None

Optional (B, H, W) bool/float mask, True on real pixels and False on padding. Supply it whenever images were letter-boxed onto a common canvas, so queries do not attend to the padded region.

targetslist[dict[str, Tensor]] | None= None

Optional training targets (list of dicts with "boxes" and "labels"). Divergence, deliberate: boxes are xyxy normalised to [0, 1], while the reference criterion takes cxcywh. Lucid's other detection families all take xyxy targets, so DETR matches its siblings rather than its paper's code and converts internally for the L1 term — feeding cxcywh here trains against the wrong corners silently. Ported reference training loops must convert their targets first.

Returns

ObjectDetectionOutput

ObjectDetectionOutput: logits : (B, N, K+1) raw class logits. pred_boxes: (B, N, 4) cxcywh normalised boxes. loss : set-prediction loss when targets provided.

postprocess

→list[dict[str, Tensor]]

postprocess(output: ObjectDetectionOutput, image_sizes: list[tuple[int, int]])

source edit

Filter queries by score threshold and convert to pixel coordinates.

Parameters

outputObjectDetectionOutput

Forward pass output.

image_sizeslist[tuple[int, int]]

List of (H, W) per image.

Returns

list[dict[str, Tensor]]

List of per-image result dicts with "boxes" (xyxy pixels),

Inference returns class logits and normalised cxcywh boxes for every query: >>> import lucid >>> from lucid.models.vision.detr import detr_resnet50 >>> model = detr_resnet50() >>> x = lucid.randn(1, 3, 800, 800) >>> out = model(x) >>> out.logits.shape # (B, num_queries, K + 1) (1, 100, 92) >>> out.pred_boxes.shape (1, 100, 4) Training pass with normalised xyxy ground-truth boxes (Hungarian matching computes the set loss): >>> targets = [{ ... "boxes": lucid.tensor([[0.1, 0.1, 0.5, 0.5]]), ... "labels": lucid.tensor([3], dtype=lucid.int64), ... }] >>> out = model(x, targets=targets) >>> out.loss.shape ()

forward

→ObjectDetectionOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, pixel_mask: Tensor | None = None)

source edit

Run DETR.

Parameters

xTensor

(B, C, H, W) image batch.

pixel_maskTensor | None= None

Optional (B, H, W) bool/float mask, True on real pixels and False on padding. Supply it whenever images were letter-boxed onto a common canvas, so queries do not attend to the padded region.

targetslist[dict[str, Tensor]] | None= None

Returns

ObjectDetectionOutput

ObjectDetectionOutput: logits : (B, N, K+1) raw class logits. pred_boxes: (B, N, 4) cxcywh normalised boxes. loss : set-prediction loss when targets provided.