Competing in RoboCup@Dwelling and successful the 2026 greatest paper award: an interview with Raphael Memmesheimer

Crew NimbRo’s robotic in motion in the course of the RoboCup@Dwelling competitors at RoboCup 2026. Raphael leads the staff.

Raphael Memmesheimer has been efficiently competing within the RoboCup@Dwelling league successful the RoboCup@Dwelling competitors 5 occasions. He and his colleagues gained the very best paper award at RoboCup 2026 for his or her work “Environment friendly Picture Annotation by way of Semi-Supervised Object Segmentation with Label Propagation”. We caught up with Raphael to seek out out extra about this analysis, his historical past with RoboCup, and the challenges of the @Dwelling competitors.

May you begin by telling us how you bought concerned with RoboCup@Dwelling?

I’ve really been concerned within the @Dwelling league since I used to be a scholar, and my first time going to a World Cup for @Dwelling was in 2013. That competitors motivated me to do a PhD and contaminated me in a approach that meant I mainly aligned my educational profession to it. I all the time return to robotic competitions as a result of this is likely one of the uncommon locations the place you possibly can benchmark robots in an actual atmosphere and have an actual competitors. Generally in analysis you tune your approaches on knowledge units and present that they carry out in a one-shot video. In robotic competitions, you mainly need to roll out your robotic, it enters the world, after which it operates autonomously by itself. It’s all the time fascinating to see how new developments make the robots carry out higher.

You and your co-authors gained the very best paper award on the RoboCup symposium this 12 months. Congratulations! May you give us a fast introduction to the paper?

The motivation for this work is that we solely have two preparation days on the RoboCup competitions. So we arrive there, we’ve got to assemble our robots, and the robots must get used to the brand new atmosphere [a model apartment] that we haven’t seen earlier than. Additionally, the robots don’t have a clue about something on this residence. And inside the two preparation days, the robots must be taught the objects they’re interacting with. For instance, they need to recognise what a spoon is, or a particular drink.

Our paper mainly helps us in very effectively producing this knowledge such that the robots can extra rapidly be taught fashions for the objects which might be concerned on this competitors. And the fascinating factor is it’s not just some objects that we are able to distinguish between; there may be as much as 50 completely different objects that we haven’t seen earlier than. Our method mainly optimizes the method in order that we are able to rapidly get a very good imaginative and prescient mannequin for the robotic. We’re conscious of open-vocabulary approaches however discovered them to have points distinguishing between objects of a class like drinks.

Is it doable to do some pre-training to anticipate objects that is likely to be used within the competitors?

Partially that’s doable, however most objects can’t actually be anticipated. We all know that there are drinks, there are snacks, that there are cutlery gadgets within the residence. However what can’t be anticipated is the small print as a result of objects come from completely different areas on the planet. The organisers will often go to a shopping center and simply get native groceries there. These objects are then distributed within the residence. A part of the thing set comes from the publicly out there YCB Dataset, for these we are able to collect knowledge beforehand, and naturally pre-training with giant basic object units.

May you discuss your methodology?

We use a multi-stage method. Firstly, we section, such that we be taught what an object really is on a segmentation degree. We take a look at all of the objects that robots noticed prior to now editions of those competitions, and we practice one mannequin that simply says, “oh, there’s something which is likely to be an object”. After which we get reference photographs of the particular objects and we semi-supervised or semi-automatically label these objects. The concept is that we mix figuring out one thing as an object with a reference of what may really be there. That permits us to simplify the labeling method. Round 60% of the information is mechanically labeled primarily based on our method. That will take plenty of hours by human annotation.

Labeled instance photographs from completely different competitors venues. Determine from “Environment friendly Picture Annotation by way of Semi-Supervised Object Segmentation with Label Propagation”.

Have you ever been in a position to make use of the strategy already in RoboCup competitions?

Sure, we used it in Brazil for RoboCup 2025 and at RoboCup GermanOpen for the native RoboCup competitions. The strategy is constantly evolving. Now that we’ve got this superb imaginative and prescient mannequin, it implies that we save plenty of time in the course of the two preparation days and we are able to focus extra on testing and attaining the precise duties as an alternative of simply specializing in the imaginative and prescient component. One facet word: these days there are promptable object detection or segmentation strategies, however we discovered that they don’t scale but to the variety of lessons which might be concerned on this competitors. They work fantastic with 5 to 10 completely different lessons, however we had points distinguishing all 50 lessons simply by prompting them.

On this 12 months’s World Cup (in Korea) we used an advanced method the place we used movies to generate much more knowledge.

Crew NimbRo’s robotic selecting laundry from the washer.

How did your staff get on within the competitors this time?

It went effectively. We had points, as all the time throughout competitions, however we ended up in third place. There was a brand new “Doing Laundry” job launched this 12 months. The robotic needed to carry a laundry basket from a washer to a folding desk and open the washer. It needed to detect the washer door, then discover the particular level to understand the deal with to open the machine door. Then it needed to pull out t-shirts and fold them.

Our robotic carried out the very best on this job. It was capable of open the washer and begin folding the laundry. And this was doable as a result of we used these semi-automatic labeling approaches.

What have been the opposite duties that groups wanted to hold out?

The primary massive hurdle on the competitors is to make the robotic cross the preliminary inspection. Robots should fulfill sure security and interactive capabilities and in the event that they fail they don’t seem to be allowed to participate within the competitors.

Within the competitors itself there are 5 important duties.

One is human-robot interplay. The robots need to open the door for celebration friends and greet them. Right here, the robotic has to go to the door, discover the deal with, align the end-effector to the deal with, open the door, let the human enter, communicate with the visitor whereas holding eye-contact, information them to the lounge, discover them a free seat, and introduce this visitor to some other friends. This course of continues on repeat so long as there are new friends arriving.

Crew NimbRo’s robotic opening the door.

One other job is a decide and place job, and this time we needed to clear a desk after breakfast. So the robotic needed to carry the cutlery to the dishwasher, put a dishwasher pill into the dishwasher, shut the dishwasher, after which put together the desk prepared for the subsequent meal. That’s a really manipulation intensive job.

There may be additionally a general-purpose service robotic job. This job is undefined as a result of the robotic has to cope with a random pure language command that may incorporate all parts of the residence. For instance it is likely to be requested to go and fetch a selected drink or snack and hand it to somebody. Up to now there have been individuals from the viewers referred to as into the world and requested to offer a command. It’s fascinating to see how good these robots are at interacting with a brand new individual and never somebody they’re used to receiving instructions from.

One other fascinating job is the restaurant job. Right here, the robots should not within the identified atmosphere that we skilled them in for 2 days, as an alternative they’re taken to an unknown restaurant location. The robotic is positioned there and must serve naturally. Solely the menu is thought to the groups prematurely. The robots must scan the restaurant to autonomously function and discover waving prospects. The order is then given by voice, and often these restaurant environments are very noisy. This can be a very fascinating job for the viewers as a result of it’s within the wild and other people collect round it to look at it.

Then, after all, there’s the brand new laundry job that I described above. So as to add to what I mentioned earlier than, this can be a very exhausting job as a result of the robotic can’t see into the washer. So it has to really feel if it has a T-shirt in its hand and take the t-shirt out of the machine. Then the robotic has to go to a desk and begin folding different t-shirts.

Crew NimbRo

Attention-grabbing. I used to be at RoboCup in Eindhoven and I discovered the @Dwelling competitors fascinating.

We really gained the competitors that 12 months!

The league was extra divided then, with three completely different leagues: two commonplace platforms and an open platform league. It was the open platform league that we gained.

Now, all of these leagues have been merged. This 12 months there have been 24 groups from all around the world collaborating in only one league, which is superb. It additionally drives competitors quite a bit.

How a lot of an enchancment have you ever seen over time that you simply’ve been doing RoboCup@Dwelling?

There are all the time phases of enchancment, as I’d describe them. I began pre-deep studying, and there have been no imaginative and prescient sturdy fashions for seeing objects in unknown environments. What was working effectively was mapping and navigation. Once I began within the league, the robots have been already speaking, however typically, as an alternative of speaking to a human like they have been alleged to, they could discuss to a potted plant or a coat hanger as a result of that they had a form much like a human. That is not the case – these days we understand people fairly effectively. We will additionally acknowledge speech fairly effectively. Process planning continues to be a problem, nevertheless it has improved. Then there are issues that seem and push the boundaries, akin to giant language fashions, open vocabulary approaches and, most just lately, agentic methods. However now and again there’s stagnation as a result of groups are simply overwhelmed by the extensive quantity of various challenges that this competitors throws at you. Principally your robotic has to do all the pieces. It must navigate in an residence, it wants to have the ability to work together with people, it has to govern gadgets, and it has to do job planning. And all of these are very intensive analysis areas which might be difficult in themselves. As well as the implementations have to be very sturdy, as there are only some trials.

How are you and the staff planning to enhance the robotic for the subsequent competitors?

We’re engaged on enhancing manipulation capabilities in order that the robotic can extra rapidly decide varied objects. We’re additionally constantly testing our robots’ efficiency to make our robots extra human-like, and utilizing two arms for manipulation is one thing we need to enhance.

About Raphael

Raphael Memmesheimer’s long-term analysis purpose is to develop autonomous service robots that help individuals in on a regular basis environments: robots that understand, perceive, and be taught from human habits utilizing multi-modal sensory data and that show themselves outdoors the lab. He’s at the moment a postdoctoral researcher with the Autonomous Clever Programs (AIS) group on the College of Bonn, the place he re-established and leads the home service robotic staff NimbRo@Dwelling. He’s additionally a founding member of the Heart for Robotics on the College of Bonn. Raphael Memmesheimer has intensive expertise in benchmark robotics competitions, having led groups at two universities to a number of worldwide titles. Notably, he has gained the RoboCup@Dwelling World Championship 5 occasions (together with three consecutive titles), together with a number of victories on the RoboCup German Open and high placements within the European Robotics League and the World Robotic Summit.

tags: family