Compact Transformers

Preprint Link: Escaping the Big Data Paradigm with Compact Transformers

By Ali Hassani^[1]*, Steven Walton^[1]*, Nikhil Shah^[1], Abulikemu Abuduweili^[1], Jiachen Li^[1,2], and Humphrey Shi^[1,2,3]

*Ali Hassani and Steven Walton contributed equal work

In association with SHI Lab @ University of Oregon^[1] and UIUC^[2], and Picsart AI Research (PAIR)^[3]

Other implementations & resources

[PyTorch blog]: check out our official blog post with PyTorch to learn more about our work and vision transformers in general.

[Keras]: check out Compact Convolutional Transformers on keras.io by Sayak Paul.

[vit-pytorch]: CCT is also available through Phil Wang's vit-pytorch, simply use pip install vit-pytorch

Abstract

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, along with their unprecedented size and amounts of training data, many have come to believe that they are not suitable for small sets of data. This trend leads to great concerns, including but not limited to: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we dispel the myth that transformers are “data hungry” and therefore can only be applied to large sets of data. We show for the first time that with the right size and tokenization, transformers can perform head-to-head with state-of-the-art CNNs on small datasets, often with bet-ter accuracy and fewer parameters. Our model eliminates the requirement for class token and positional embeddings through a novel sequence pooling strategy and the use of convolution/s. It is flexible in terms of model size, and can have as little as 0.28M parameters while achieving good results. Our model can reach 98.00% accuracy when training from scratch on CIFAR-10, which is a significant improvement over previous Transformer based models. It also outperforms many modern CNN based approaches, such as ResNet, and even some recent NAS-based approaches,such as Proxyless-NAS. Our simple and compact design democratizes transformers by making them accessible to those with limited computing resources and/or dealing with small datasets. Our method also works on larger datasets, such as ImageNet (82.71% accuracy with 29% parameters of ViT),and NLP tasks as well.

ViT-Lite: Lightweight ViT

Different from ViT we show that an image is not always worth 16x16 words and the image patch size matters. Transformers are not in fact ''data-hungry,'' as the authors proposed, and smaller patching can be used to train efficiently on smaller datasets.

CVT: Compact Vision Transformers

Compact Vision Transformers better utilize information with Sequence Pooling post encoder, eliminating the need for the class token while achieving better accuracy.

CCT: Compact Convolutional Transformers

Compact Convolutional Transformers not only use the sequence pooling but also replace the patch embedding with a convolutional embedding, allowing for better inductive bias and making positional embeddings optional. CCT achieves better accuracy than ViT-Lite and CVT and increases the flexibility of the input parameters.

How to run

Install locally

Our base model is in pure PyTorch and Torchvision. No extra packages are required. Please refer to PyTorch's Getting Started page for detailed instructions.

Here are some of the models that can be imported from src (full list available in Variants.md):

Model	Resolution	PE	Name	Pretrained Weights	Config
CCT-7/3x1	32x32	Learnable	cct_7_3x1_32	CIFAR-10/300 Epochs	pretrained/cct_7-3x1_cifar10_300epochs.yml
		Sinusoidal	cct_7_3x1_32_sine	CIFAR-10/5000 Epochs	pretrained/cct_7-3x1_cifar10_5000epochs.yml
		Learnable	cct_7_3x1_32_c100	CIFAR-100/300 Epochs	pretrained/cct_7-3x1_cifar100_300epochs.yml
		Sinusoidal	cct_7_3x1_32_sine_c100	CIFAR-100/5000 Epochs	pretrained/cct_7-3x1_cifar100_5000epochs.yml
CCT-7/7x2	224x224	Sinusoidal	cct_7_7x2_224_sine	Flowers-102/300 Epochs	pretrained/cct_7-7x2_flowers102.yml
CCT-14/7x2	224x224	Learnable	cct_14_7x2_224	ImageNet-1k/300 Epochs	pretrained/cct_14-7x2_imagenet.yml
	384x384		cct_14_7x2_384	ImageNet-1k/Finetuned/30 Epochs	finetuned/cct_14-7x2_imagenet384.yml
	384x384		cct_14_7x2_384_fl	Flowers102/Finetuned/300 Epochs	finetuned/cct_14-7x2_flowers102.yml

You can simply import the names provided in the Name column:

from src import cct_14_7x2_384
model = cct_14_7x2_384(pretrained=True, progress=True)

The config files are provided both to specify the training settings and hyperparameters, and allow easier reproduction.

Please note that the models missing pretrained weights will be updated soon. They were previously trained using our old training script, and we're working on training them again with the new script for consistency.

You could even create your own models with different image resolutions, positional embeddings, and number of classes:

from src import cct_14_7x2_384, cct_7_7x2_224_sine
model = cct_14_7x2_384(img_size=256)
model = cct_7_7x2_224_sine(img_size=256, positional_embedding='sine')

Changing resolution and setting pretrained=True will interpolate the PE vector to support the new size, just like ViT.

These models are also based on experiments in the paper. You can create your own versions:

from src import cct_14
model = cct_14(arch='custom', pretrained=False, progress=False, kernel_size=5, n_conv_layers=3)

You can even go further and create your own custom variant by importing the class CCT.

All of these apply to CVT and ViT as well.

Training

timm is recommended for image classification training and required for the training script provided in this repository:

Distributed training

./dist_classification.sh $NUM_GPUS -c $CONFIG_FILE /path/to/dataset

You can use our training configurations provided in configs/:

./dist_classification.sh 8 -c configs/imagenet.yml --model cct_14_7x2_224 /path/to/ImageNet

Non-distributed training

python train.py -c configs/datasets/cifar10.yml --model cct_7_3x1_32 /path/to/cifar10

Models and config files

We've updated this repository and moved the previous training script and the checkpoints associated with it to examples/. The new training script here is just the timm training script. We've provided the checkpoints associated with it in the next section, and the hyperparameters are all provided in configs/pretrained for models trained from scratch, and configs/finetuned for fine-tuned models.

Results

Type can be read in the format L/PxC where L is the number of transformer layers, P is the patch/convolution size, and C (CCT only) is the number of convolutional layers.

CIFAR-10 and CIFAR-100

Model	Pretraining	Epochs	PE	CIFAR-10	CIFAR-100
CCT-7/3x1	None	300	Learnable	96.53%	80.92%
		1500	Sinusoidal	97.48%	82.72%
		5000	Sinusoidal	98.00%	82.87%

Flowers-102

Model	Pre-training	PE	Image Size	Accuracy
CCT-7/7x2	None	Sinusoidal	224x224	97.19%
CCT-14/7x2	ImageNet-1k	Learnable	384x384	99.76%

ImageNet

Model	Type	Resolution	Epochs	Top-1 Accuracy	# Params	MACs
ViT	12/16	384	300	77.91%	86.8M	17.6G
CCT	14/7x2	224	310	80.67%	22.36M	5.11G
	14/7x2	384	310 + 30	82.71%	22.51M	15.02G

NLP

NLP results and instructions have been moved to nlp/.

Citation

@article{hassani2021escaping,
	title        = {Escaping the Big Data Paradigm with Compact Transformers},
	author       = {Ali Hassani and Steven Walton and Nikhil Shah and Abulikemu Abuduweili and Jiachen Li and Humphrey Shi},
	year         = 2021,
	url          = {https://arxiv.org/abs/2104.05704},
	eprint       = {2104.05704},
	archiveprefix = {arXiv},
	primaryclass = {cs.CV}
}