RoboGen: Automated Robotic Skill Learning at Scale via Generative Simulation

Abstract

We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates corresponding simulation environments by populating pertinent objects and assets with proper spatial configurations. Afterwards, the agent decomposes the proposed high-level task into sub-tasks, selects the optimal learning approach (reinforcement learning, motion planning, or trajectory optimization), generates required training supervision, and then learns policies to acquire the proposed skill. Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics. Our fully generative pipeline can be queried repeatedly, producing an endless stream of skill demonstrations associated with diverse tasks and environments.

RoboGen Pipeline


RoboGen is a fully automated pipeline for endless and diverse skill acquisition. RoboGen pipeline consists of 4 stages: A) task proposal, B) scene generation, C) training supervision generation, and D) skill learning with generated information.

Long-horizon task generation and learning


RoboGen is able to propose high-level tasks, generate corresponding environments, decompose the high-level goal into low-level subtasks, and then learn the sub-skills sequentially.

RoboGen generated tasks, scenes, training supervisions, and learned skills

Please select an image below to view the results.


<b>store an item into the storagefurniture</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/store_an_item_into_the_storagefurniture.txt <b>Heat up a bowl of soup in microwave</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Heat_up_a_bowl_of_soup_in_microwave.txt <b>Slide Window Halfway</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Slide_Window_Halfway.txt <b>Gallop</b>, RoboGen Results: [sep] assets/robogen_results/locomotion_gallop.txt <b>Rotate in place</b>, RoboGen Results: [sep] assets/robogen_results/locomotion_rotate_in_place.txt <b>retrieve an item from safe</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/retrieve_an_item_from_safe.txt <b>Unload Objects from Cart</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Unload_Objects_from_Cart.txt <b>Change Lamp Direction</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Change_Lamp_Direction.txt <b>Retrieve item from box</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Retrieve_item_from_box.txt <b>store food in refrigerator</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/store_food_in_refrigerator.txt <b>jump up</b>, RoboGen Results: [sep] assets/robogen_results/locomotion_jump.txt <b>Extend Suitcase Handle</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Extend_Suitcase_Handle.txt <b>Pull Lever to Start Coffee Brewing</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Pull_Lever_to_Start_Coffee_Brewing.txt <b>Flush the Toilet</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Flush_the_Toilet.txt <b>Close laptop lid</b>, RoboGen Results (scene configuration, task decomposition and supervisions): [sep] assets/robogen_results/Close_Laptop_Lid.txt <b>Flip leftwards</b>, RoboGen Results: [sep] assets/robogen_results/locomotion_flip_leftwards.txt
Select an image above to show the RoboGen results:
RoboGen response shown within code block.

Gallery



Change lamp light direction

Pull lever to start coffee brewing

Store the toy into the storage

Stand upright and walk forward using only hind legs

Bend noodle into a U-shape

Open stapler lid

Jump up

Retrieve the gold bar from safe

Flatten the rice ball

Do a headstand

Uncover trashcan lid

Close the drawer of the table

Heat up a bowl of soup in microwave

Kick the soccer ball to the left

Fold the chair

Spin clockwise without right hind foot touching the ground

Walk forward

Retrieve the toy car from box

Jump backward

Flip leftward

Slide in table drawer

Do a headstand

Store the apple in refrigerator

Lift up toilet lid

Set the clock back by 5 minutes

Flip backward

Take jumps forward

Throw trash away

Walk backward

Lie face down and cawl forward

Do a headstead

Slide window halfway

Load dish into dishwasher

Stay balanced and move forward

Climp up the stairs

Run forward fast

Open microwave door

Arrange three different cans in a row

Walk backward

Unload the milk from cart

Open the door of trash can

Pull drawer out

Take out a bottle of water from fridge

Tilt display screen

Flip rightward

Flip rightward

Press and rotate dispenser lid

Put a book in the drawer

Cut the dough in half

Rotate globe horizontally

Collapse chair

Turn on faucet

Store a toy inside the box

Close dispenser lid

Flip backward

Turn off faucet

Jump as high as possible

Put an toy into the storage

Direct and press dispenser

Flip forward

Do a backflip

Lift up kettle lid

Kick the soccer ball to the left

Crawl forward

Flush the toilet

Open washing machine door

Raise laptop screen

Activate water faucet

Slide both windows

Rotate fan rotor

Open fridges freezer door

Twist kettle handle

Open oven door

Jump higher than 5 meters

Spin left without using right hind leg

Flip forward

Run backward fast using only front and left hind legs

Take a big jump backward

Pull out the top drawer of the cabinet

Press the button to turn on the printer

Close trashcan lid

Run forward

Shut toilet lid

Push the ball forward

Open box lid halfway

Push the ball forward

Wrap the dumpling wrapper

Lift box lid

Spin leftward

Crawl backward at 1m/s

Start dishwasher by pressing the start button

Opening both refrigerator doors

Shape the dough into a baguette

Jump over the hurdle

Close window

Slide down display screen

Put a fruit into a fruit bowl

Jump and kick the basketball

Turning on coffee machine using the knob

Turn rightward continuously

Flip forward

Open door

Kick the soccer ball to the right

Extend suitcase handle

Spin clockwise

Lift up the dumpling

Stand upright on two hind legs

Spin clockwise

Rotate bucket handle

Draw out drawer

Do a backflip and kick the soccer ball to the right

Remove pot lid

Close laptop screen

Roll out the doll

Spin counter-clockwise

Close door

Put filling onto the wrapper

Turn on lamp by pressing the toggle button

Move clock ahead for daylight saving

Shut the oven door