might be a waste of resources

[SDaoer]

        while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
            # prepare model inputs
            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)

            # forward pass to get next token
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )

            if synced_gpus and this_peer_finished:
                continue  # don't waste resources running the code we don't need

            ...

Why is this condition checked after the outputs is generated? Can this be considered a form of resource wastage? Could this part be moved to the beginning of the while loop?

if synced_gpus and this_peer_finished:
    continue  # don't waste resources running the code we don't need

The code comes from transformers/generation/utils.py: GenerationMixin._sample

[amyeroberts]

cc @gante @zucchini-nlp

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[gante]

Hi @SDaoer 👋 Thank you for opening this issue 🤗

TL;DR without it the code will hang in specific settings.

The answer is documented in the code:

transformers/src/transformers/generation/utils.py

Line 2198 in 083e13b

# Under synced_gpus the `forward` call must continue until all gpus complete their sequence.

You can complement this comment with the meaning of synced_gpus:

synced_gpus (`bool`, *optional*):
                Whether to continue running the while loop until max_length. Unless overridden this flag will be set to
                `True` under DeepSpeed ZeRO Stage 3 multiple GPUs environment to avoid hanging if one GPU finished
                generating before other GPUs. Otherwise it'll be set to `False`.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.