ViT-Base/16 backbone (Dosovitskiy et al., 2020).
Builds the canonical ViT-Base configuration with patch size 16:
dim=768, depth=12, num_heads=12, mlp_ratio=4.0. With
the default 224x224 input this yields patch
tokens plus one CLS token, for an input sequence length of 197 to
the transformer encoder. Approximately 86M parameters.
Model Size
Parameters
pretrainedbool= FalseNo checkpoint is published for
vit_base_16, so True raises
NotImplementedError rather than return randomly initialised weights.
vit_base_16_cls is the checkpointed variant.**overridesobject= {}Keyword overrides applied on top of the canonical ViT-Base/16
config — e.g.
image_size=384, in_channels=1, or
dropout=0.1. Each override must match a field of
ViTConfig.Returns
ViTA ViT backbone instance returning a flat (B, 768) CLS
feature.
Notes
ViT-Base/16 was pretrained on JFT-300M / ImageNet-21k and reported 84.15% top-1 / 97.20% top-5 on ImageNet-1k after fine-tuning at 384x384 in Dosovitskiy et al. (2020, Table 5). See arXiv:2010.11929.
Examples
>>> import lucid
>>> from lucid.models.vision.vit import vit_base_16
>>> model = vit_base_16()
>>> x = lucid.randn(1, 3, 224, 224)
>>> feat = model.forward_features(x)
>>> feat.shape
(1, 768)