Reward assignment during recording #518

ChorntonYoel · 2024-11-22T17:14:28Z

What this does

Examples:

Title	Label
Item1 of Reward Classifier in issue #504	(Feature)

Those PR is meant to add the possibility for the teleoperator to add rewards when performing the tasks.
The current approach simply consists in having frames labeled 0 until the experimenter presses the space bar. Then the reward for each frame becomes 1. The experimenter can press the bar again (eg. in case of subsequent failure) and the frame labeling will return to 0. Each time the space bar is pressed, the labeling switches

This is one way to go about it, but lmk if you have a better idea!

How it was tested

I made datasets with Moss with and without rewards. Checking that I got the expected behavior.

How to checkout & try? (for the reviewer)

python lerobot/scripts/control_robot.py record \
    --robot-path lerobot/configs/robot/moss.yaml \
    --fps 30 \
    --root data \
    --repo-id ${HF_USER}/moss_test \
    --tags moss tutorial \
    --warmup-time-s 5 \
    --episode-time-s 40 \
    --reset-time-s 10 \
    --num-episodes 2 \
    --push-to-hub 1 \
    --assign_rewards

Cadene

Really cool! I like this design ;)

We are in the process of switching to dataset v2. We removed populate_dataset.py and updated scripts/control_robot.py.
By any chance could you rebase on top of user/aliberts/2024_09_25_reshape_dataset? Sorry for that!!!

You might need to solve some conflicts. Happy to jump on a call if needed or message me on Discord ;)

git fetch origin user/aliberts/2024_09_25_reshape_dataset
git rebase origin/user/aliberts/2024_09_25_reshape_dataset

lerobot/scripts/control_robot.py

lerobot/common/datasets/push_dataset_to_hub/aloha_hdf5_format.py

Cadene · 2024-11-22T18:27:41Z

Also style check failing was failing. Could you please run our pre-commit? :D

pre-commit install
pre-commit run --all-files

https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr

…uggingface#450)

…ggingface#489)

Co-authored-by: Remi <[email protected]>

lerobot/scripts/control_robot.py

ChorntonYoel · 2024-11-23T13:04:33Z

lerobot/scripts/control_robot.py

-        nargs="*",
-        help="Add tags to your dataset on the hub.",
-    )
+    # parser_record.add_argument(


I commented those because they were raising "unexpected keyword argument" errors. Probably shouldn't be done in this PR. Lmk if you want me to bring them back

We will bring them back after merging Simon's PR ;) thanks for flagging ; same for force-override
cc @aliberts

lerobot/scripts/control_robot.py

Cadene

Thanks ;) Let's wait for Simon's to merge the PR, then we can merge

Cadene · 2024-11-25T19:54:28Z

lerobot/scripts/control_robot.py

-        nargs="*",
-        help="Add tags to your dataset on the hub.",
-    )
+    # parser_record.add_argument(


We will bring them back after merging Simon's PR ;) thanks for flagging ; same for force-override
cc @aliberts

lerobot/scripts/control_robot.py

lerobot/common/robot_devices/control_utils.py

lerobot/scripts/control_robot.py

Cadene · 2024-11-25T21:21:31Z

lerobot/scripts/control_robot.py

-        dataset.push_to_hub(private=True)
+        dataset.push_to_hub()


cc @aliberts

Co-authored-by: Remi <[email protected]>

michel-aractingi

Nice work! Left few comments but in general it's ready to merge.

michel-aractingi · 2024-12-03T16:15:01Z

lerobot/common/datasets/lerobot_dataset.py

+            features = {} if features is None else features
+            features.update(get_features_from_robot(robot, use_videos))


What do you think of writing it like this?

Suggested change

features = {} if features is None else features

features.update(get_features_from_robot(robot, use_videos))

features = {**(features or {}), **get_features_from_robot(robot)}

michel-aractingi · 2024-12-03T16:56:23Z

lerobot/common/robot_devices/control_utils.py

    # Allow to exit early while recording an episode or resetting the environment,
    # by tapping the right arrow key '->'. This might require a sudo permission
    # to allow your terminal to monitor keyboard events.


Should we add some comments here to explain assign_rewards?

Suggested change

# Allow to exit early while recording an episode or resetting the environment,

# by tapping the right arrow key '->'. This might require a sudo permission

# to allow your terminal to monitor keyboard events.

"""

Initializes a keyboard listener to enable early termination of an episode

or environment reset by pressing the right arrow key ('->'). This may require

sudo permissions to allow the terminal to monitor keyboard events.

Args:

assign_rewards (bool): If True, allows annotating the collected trajectory

with a binary reward at the end of the episode to indicate success.

"""

Also the same in the comments header of lerobot/scripts/control_robot.py

Cadene requested review from michel-aractingi and Cadene November 22, 2024 17:22

Cadene reviewed Nov 22, 2024

View reviewed changes

lerobot/scripts/control_robot.py Outdated Show resolved Hide resolved

lerobot/common/datasets/push_dataset_to_hub/aloha_hdf5_format.py Outdated Show resolved Hide resolved

lerobot/common/datasets/push_dataset_to_hub/aloha_hdf5_format.py Outdated Show resolved Hide resolved

ChorntonYoel and others added 6 commits November 22, 2024 23:49

add reward assignment during teleoperation

2e15499

nit

0c3faff

pre commit

e7805ed

Add support for Windows (huggingface#494)

0151ec5

bug causes error uploading to huggingface, unicode issue on windows. (h…

b8bf366

…uggingface#450)

Add distinction between two unallowed cases in name check "eval_" (hu…

2001f16

…ggingface#489)

ChorntonYoel force-pushed the add_reward_assignment branch from 766ed5a to 2001f16 Compare November 22, 2024 22:51

ChorntonYoel changed the base branch from main to user/aliberts/2024_09_25_reshape_dataset November 22, 2024 22:51

ChorntonYoel and others added 7 commits November 22, 2024 23:53

remove populate dataset

5a60728

take off useless code

abf5798

fix find motor port

6ee99dd

Update lerobot/scripts/control_robot.py

81a926f

Co-authored-by: Remi <[email protected]>

adapt to v2

c7eeff4

nit

a4f7db9

nit

e123a1f

ChorntonYoel commented Nov 23, 2024

View reviewed changes

lerobot/scripts/control_robot.py Show resolved Hide resolved

ChorntonYoel commented Nov 23, 2024

View reviewed changes

ChorntonYoel added 3 commits November 23, 2024 14:18

fix

56447f9

cleanup rebase

62d3116

nit from rebase

cdc723e

ChorntonYoel marked this pull request as ready for review November 23, 2024 15:33

ChorntonYoel requested a review from Cadene November 23, 2024 16:32

ChorntonYoel commented Nov 23, 2024

View reviewed changes

lerobot/scripts/control_robot.py Outdated Show resolved Hide resolved

ChorntonYoel added 3 commits November 24, 2024 00:02

fix reward leak between episodes

57f58d8

next.reward

515487f

nit

63b23f6

Cadene reviewed Nov 25, 2024

View reviewed changes

lerobot/scripts/control_robot.py Outdated

dataset.push_to_hub(private=True)

dataset.push_to_hub()

Copy link

Collaborator

Cadene Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @aliberts

ChorntonYoel and others added 3 commits November 25, 2024 22:58

Update lerobot/common/robot_devices/control_utils.py

1eb1f3b

Co-authored-by: Remi <[email protected]>

Update lerobot/common/robot_devices/control_utils.py

4b78469

Co-authored-by: Remi <[email protected]>

int in arg parser

62db861

ChorntonYoel mentioned this pull request Nov 26, 2024

Reward classifier and training #528

Open

ChorntonYoel changed the base branch from user/aliberts/2024_09_25_reshape_dataset to user/michel-aractingi/2024-11-27-port-hil-serl November 27, 2024 17:34

michel-aractingi reviewed Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reward assignment during recording #518

Reward assignment during recording #518

ChorntonYoel commented Nov 22, 2024 •

edited

Loading

Cadene left a comment •

edited

Loading

Cadene commented Nov 22, 2024

ChorntonYoel Nov 23, 2024

Cadene Nov 25, 2024

Cadene left a comment

Cadene Nov 25, 2024

Cadene Nov 25, 2024

michel-aractingi left a comment

michel-aractingi Dec 3, 2024

michel-aractingi Dec 3, 2024

michel-aractingi Dec 3, 2024

		features = {} if features is None else features
		features.update(get_features_from_robot(robot, use_videos))

	features = {} if features is None else features
	features.update(get_features_from_robot(robot, use_videos))
	features = {(features or {}), get_features_from_robot(robot)}

-    # Allow to exit early while recording an episode or resetting the environment,
-    # by tapping the right arrow key '->'. This might require a sudo permission
-    # to allow your terminal to monitor keyboard events.
+    """
+    Initializes a keyboard listener to enable early termination of an episode
+    or environment reset by pressing the right arrow key ('->'). This may require
+    sudo permissions to allow the terminal to monitor keyboard events.
+    Args:
+        assign_rewards (bool): If True, allows annotating the collected trajectory
+        with a binary reward at the end of the episode to indicate success.
+    """

Reward assignment during recording #518

Are you sure you want to change the base?

Reward assignment during recording #518

Conversation

ChorntonYoel commented Nov 22, 2024 • edited Loading

What this does

How it was tested

How to checkout & try? (for the reviewer)

Cadene left a comment • edited Loading

Choose a reason for hiding this comment

Cadene commented Nov 22, 2024

ChorntonYoel Nov 23, 2024

Choose a reason for hiding this comment

Cadene Nov 25, 2024

Choose a reason for hiding this comment

Cadene left a comment

Choose a reason for hiding this comment

Cadene Nov 25, 2024

Choose a reason for hiding this comment

Cadene Nov 25, 2024

Choose a reason for hiding this comment

michel-aractingi left a comment

Choose a reason for hiding this comment

michel-aractingi Dec 3, 2024

Choose a reason for hiding this comment

michel-aractingi Dec 3, 2024

Choose a reason for hiding this comment

michel-aractingi Dec 3, 2024

Choose a reason for hiding this comment

ChorntonYoel commented Nov 22, 2024 •

edited

Loading

Cadene left a comment •

edited

Loading