Attack of the Franka

• Computer Vision
• Motion
• Conclusions
• Future Work
• Team Organization

First, the ROS 2 wrapper for Intel RealSense devices is used to connect to the RealSense and publish color image data and depth data that is aligned with the color image camera reference frame.

Next, the April Robotics AprilTag library is used along with Christian Rauch’s ROS 2 AprilTag node and messages packages to detect AprilTags within the camera’s feed. These fiducial markers of known size allow for determination of the 6-DoF pose of the object to which they are attached.

This project employs three tags - one to determine the location of the robot’s table and two to determine the bounds of the work area. In order to improve accuracy and consistency of the detected tags, we created a calibration process. Each tag’s transformation is sampled 100 times and the average taken and broadcast by the camera_processor node. If any tag is obscured from the camera - for example by a falling enemy - this average transformation is still retained and the system operation is not affected.

Especially important was the robot table tag, the large tag shown in the left of the above feed. The size of this tag was increased to improve transformation accuracy. This tag was placed in a fixed location relative to the Panda’s base reference frame. Therefore, knowing the pose of this tag in the camera’s reference frame allowed us to know the position of the robot relative to the camera.

The camera_processor node subscribes to the color and aligned depth image data and processes it using OpenCV. The color data is converted into the HSV color space, which helps reduce problems in color detection caused by lighting changes. A certain range of HSV values representing red colors is defined as “enemy” and likewise another blue range defines “allies.”

Pixels that fall into these ranges are grouped by contouring, and each contour is considered a detected object. Opening and closing morphological transformations are applied on the contours to improve their quality. Then, the centroid of each contour is calculated, which is considered the detected object’s position in the camera feed. This pixel position can be combined with the depth data for that pixel and deprojected into 3D real world coordinates of the object in the camera’s reference frame. Knowing the position of the object relative to the camera reference frame means that, since we already know the position of the robot relative to the camera, we can now determine the position of the detected objects relative to the robot.

Finally, a few filters are applied to ensure we only consider detected objects we care about. The position of each object is compared with the work area bounds (as defined by the AprilTags), and any objects outside of the work bounds are filtered out. Also, each object’s depth from the camera is compared to a range from the location of the work table, and objects outside of that range are filtered out. This allows us to ignore enemies that have been knocked over, which we consider “vanquished.”

The 3D real world coordinates of the detected objects are broadcast as transformations, which gives the motion subsystem the information needed to plan out the Panda’s attacks.

Our API handled:

• Reading and publishing to the planning scene to ensure the robot and the lightsaber it wielded did not collide with objects in the environment
• Interfacing with the compute_ik service to determine joint positions that would move the end-effector to a desired pose
• Interfacing with the move_action action to plan a collision-free trajectory to the desired joint positions
• Interfacing with the execute_trajectory action to execute previously planned trajectories

The lightsaber is added to the planning scene as a special attached collision object - meaning it moves with the robot and is allowed to collide with certain robot links (like the end-effector in which it is grasped). When grasping the lightsaber, it starts as a collision object so the arm will avoid it. Once grasped, the lightsaber changes to an attached collision object so collision-free trajectory plans account for it.

The process for selecting an attack style has two checks. First, there is a simple geometric check - using the position of the objects from CV - to determine if a selected attack style will cause an enemy to fall into an ally. Secondly, the node attempts to plan motion to attack in that selected attack style. For this check, the allies are added as collision objects in the planning scene. If a collision were to occur during the motion, this plan fails.

If either of these checks fail for a particular attack style, the same checks are attempted on the next attack style. If no attack style would successfully knock down an enemy without affecting an ally, the robot simply does not attack that enemy.

These attacks use several joint state waypoints, adjusted for the position of the enemy, in order to ensure the motion planning reliably completes.

The most impactful future improvement to this project would be to increase the robustness of the motion planning algorithms. With arbitrary enemy and ally positions, there are many scenarios and edge cases, not all of which the team was able to test. Planning for more of these edge cases and better defining specific motions to handle them would improve the performance of the system. Adding even more possible attack styles could also reduce the number of enemies that are unreachable due to configuration of allies in the work area.

This project was based on color detection as a central method of detecting and differentiating object types. It would be an exciting challenge to expand this object detection capability. For example, one could use YOLO or other neural networks to analyze the image and detect multiple different types of objects that represent allies and enemies based on a training dataset.