Can't reproduce benchmark results for classification task on eurosat

[rbavery]

I'm trying to load a model checkpoint and use it for inference via a ClassificationTask and and Trainer like so. The model appears untrained though even though I am loading weights that appear to have benchmarks performing above 90% total accuracy associated with them: https://torchgeo.readthedocs.io/en/stable/api/models.html#sentinel-2 I assume I'm doing something incorrect but I can't find info on how to use torchgeo to run just the inference pipeline with the correct transforms.

My torchgeo version is v0.5.2

[{'test_loss': 2.3217544555664062,
  'test_AverageAccuracy': 0.09963898360729218,
  'test_F1Score': 0.10222221910953522,
  'test_JaccardIndex': 0.010842664167284966,
  'test_OverallAccuracy': 0.10222221910953522}]

import os
import tempfile

import torch
from lightning.pytorch import Trainer
from lightning.pytorch.callbacks import EarlyStopping, ModelCheckpoint
from lightning.pytorch.loggers import TensorBoardLogger

from torchgeo.datamodules import EuroSATDataModule
from torchgeo.models import ResNet50_Weights
from torchgeo.trainers import ClassificationTask

batch_size = 10
num_workers = 8
max_epochs = 3
fast_dev_run = False

root = os.path.join(tempfile.gettempdir(), 'eurosat')
datamodule = EuroSATDataModule(
    root=root, batch_size=batch_size, num_workers=num_workers, download=True
)

weights = ResNet50_Weights.SENTINEL2_ALL_MOCO
task = ClassificationTask(
    model='resnet50',
    loss='ce',
    weights=weights,
    in_channels=13,
    num_classes=10,
    lr=0.001,
    patience=5,
)

accelerator = 'gpu' if torch.cuda.is_available() else 'cpu'
default_root_dir = os.path.join(tempfile.gettempdir(), 'experiments')
checkpoint_callback = ModelCheckpoint(
    monitor='val_loss', dirpath=default_root_dir, save_top_k=1, save_last=True
)
early_stopping_callback = EarlyStopping(monitor='val_loss', min_delta=0.00, patience=10)
logger = TensorBoardLogger(save_dir=default_root_dir, name='tutorial_logs')
trainer = Trainer(
    accelerator=accelerator,
    callbacks=[checkpoint_callback, early_stopping_callback],
    fast_dev_run=fast_dev_run,
    log_every_n_steps=1,
    logger=logger,
    min_epochs=1,
    max_epochs=max_epochs,
)

trainer.test(model=task, datamodule=datamodule)

[isaaccorley]

In your example code you are just loading the pretrained encoder weights with a random init head. The numbers you're referencing are achieved by fine-tuning on the dataset. I don't believe we save the weights with the head anywhere.

[rbavery]

Ok got it, thanks. I think that would be worth stating above these tables: https://torchgeo.readthedocs.io/en/stable/api/models.html#sentinel-2

the tables are formatted such that it looks like the links to the weights are right next to the accuracy benchmarks on specific datasets. I assumed that the scores were produced from the linked weights.

[isaaccorley]

Agreed. In the future we plan to actually have model weights for each dataset from our benchmarks but as of now basically all of the weights are pretrained from various SSL methods to be used for fine-tuning/linear probing.