Making a mobile cobot plug and play using real-time pose tracking | Cobots manual - part 6

This article was created by ACRO (KU Leuven) within the framework of the COOCK+ ROBUST project, in collaboration with Sirris.

Smarter automation for high-mix, low-volume production

In today’s manufacturing environments, especially in sheet metal and discrete industries, production demands are changing rapidly. Orders shift constantly, batch sizes are shrinking, and high product variability is becoming the norm. Traditional automation solutions are often too rigid to keep up especially for small and medium-sized enterprises (SMEs). The result? Costly manual labour, underused engineering talent, and limited scalability.

To bridge this gap, the COOCK+ ROBUST project, led by Sirris and the ACRO research unit at KU Leuven, is developing a reconfigurable, mobile cobot workstation. A key enabling technology in this initiative is 6D object pose tracking: the real-time detection of an object’s exact position and orientation in space. This capability is essential for mobile cobots to reliably interact with parts, tools, and machines no matter where they are deployed.

What is 6D pose tracking and why does it matter?

6D pose tracking refers to the continuous estimation of an object’s position (x, y, z) and orientation (roll, pitch, yaw) in a 3D space. Unlike single-frame pose estimation, which treats every camera image independently, tracking considers the temporal sequence of frames. This approach uses the previously known pose to reduce the search space for the current frame, making the system more efficient and robust.

Why does this matter for cobots? Because in a mobile setup, even the slightest deviation from a known location can introduce errors. If the cobot cannot detect where a part or tool is relative to itself, tasks like grasping, placing, or machine tending become unreliable. 6D pose tracking ensures that the robot remains spatially aware even if it’s moved or the lighting changes.
 

Enabling intuitive and adaptive interaction

6D pose tracking is a foundational building block for several advanced cobot features, including:

• Precise grasping and placement: especially useful for irregular or texture-less objects
• Task status recognition: the robot can infer human intent by tracking how tools or workpieces move
• Programming by demonstration: human operators can demonstrate a task, which the cobot then replicates autonomously

This technology plays a vital role in enabling more natural and flexible collaboration between humans and robots, particularly in SMEs where production setups evolve rapidly.

Tracking methods and modeling approaches

There are three main types of 6D pose tracking based on prior object knowledge:

• Instance-level tracking: works with 3D CAD models of specific objects
• Category-level tracking: uses general knowledge of object types (e.g. all power drills)
• Novel object tracking: requires no prior data, very flexible but also more error-prone

In the ROBUST project, we focus on instance-level tracking of rigid objects, using known 3D models for optimal accuracy. These are the most practical for industry today.

Technically, the project explores three modeling approaches:
 

Use case: tracking a handheld power drill

To validate the tracking method, a test case was created involving a handheld power drill, a common tool in manual assembly tasks. By training the pose tracker on data from CAD models and randomly generated backgrounds, the cobot learns to reliably track the drill’s 6D pose in real time.

The ability to track tools with high precision without the need for physical markers or extensive setup opens the door to mobile cobots that are truly plug-and-play across different stations.
 

Next steps: integration and demonstration

6D pose tracking is now being integrated into the mobile cobot demonstrator, forming part of a larger system that includes:

• Flexible mounting on a cobot cart
• Automated positioning via floor-based docking or visual localisation
• Communication with MES/ERP systems through fieldbus or MQTT
• Real-time safety zone configuration via location-based coding
   

Power drill tracking result using a deep-learning-based 6D pose tracking method