Perception

Lupin's perception stack reads the greenhouse two ways: it locates AprilTag climate stations from the Orbbec RGB stream, and it classifies tulips and pest anomalies from the gripper camera. An aggregator then fuses both into one map-localised, debounced registry that the mission orchestrator and the digital twin consume, and it places every detected bloom into the real planter rectangle (from the known greenhouse layout) instead of stacking dots on the marker.

This page is the reference for that fusion: the two detectors, the sim-versus-hardware swap point, the known-layout box geometry that registers the greenhouse's real planter rectangles onto the live map and turns a 4 cm marker into a bed of located flowers, and the temporal-co-location rule that stands in for a shared 3-D frame Lupin does not have.

Four console scripts ship in lupin_perception (setup.py), not three:

• tag_annotator, OpenCV-ArUco 36h11 detection, live-CameraInfo solvePnP pose, one TF per tag, overlay JSON for the HMI. Light enough for the Orange Pi.
• yolo_detector, Ultralytics YOLO on the gripper cam, four classes (tulip_red / tulip_white / tulip_pink + bug). Heavy (torch), runs laptop/Jetson-side.
• sim_flower_detector, an HSV colour-segmentation stand-in that emits the identical /yolo/detections JSON so the rest of the stack never knows it is in Gazebo. This is the only node that differs between sim and hardware.
• perception_aggregator, TF-projects tags into map, debounces the discovery registry, temporally co-locates the flower class with the tag being scanned, places blooms inside a tag-anchored box, and serves /perception/confirm_tag. Torch-free, runs on the robot.

AprilTag detection, no apriltag_ros

tag_annotator.py uses OpenCV's native ArUco detector against the AprilTag 36h11 dictionary (cv2.aruco.DICT_APRILTAG_36h11), there is no apriltag_ros dependency in the pipeline. That keeps the build small and means the same node runs in Gazebo and on the real robot.

Runtime API selection (OpenCV 4.6 vs 4.7 and up)

OpenCV 4.7 reworked the ArUco API, and that split burned the team twice in one day on the sim host. The node picks the right API at runtime so the identical file runs on the dev laptop (>= 4.7) and the OpenCV-4.6 sim host:

if hasattr(cv2.aruco, 'ArucoDetector'):
    self.aruco_params = cv2.aruco.DetectorParameters()
    _detector = cv2.aruco.ArucoDetector(self.aruco_dict, self.aruco_params)
    self._detect_markers = _detector.detectMarkers
else:
    self.aruco_params = cv2.aruco.DetectorParameters_create()
    self._detect_markers = lambda img: cv2.aruco.detectMarkers(...)

The whole hot path then calls one self._detect_markers(img) callable, so the branch is paid once at construction and never appears in the per-frame loop. Test any perception change on both a 4.6 sim host and a 4.7+ laptop before calling it done.

Live intrinsics + solvePnP pose

Intrinsics are never hard-coded. The first valid sensor_msgs/CameraInfo message latches K and the (variable-length) distortion D; subsequent messages are ignored. Each detected tag's 3-D pose is solved with cv2.solvePnP from its four corners and the configured tag_size_m, then broadcast as a TF transform:

• parent frame, Image.header.frame_id verbatim (the driver owns the optical-frame name, e.g. camera_color_optical_frame).
• child frame, f"{tf_frame_prefix}{id}", i.e. tag_<id>.
• stamped with Image.header.stamp.

The overlay JSON published on /camera/tag_detections_json is an array of {id, corners, dist}, where dist is the camera-frame z of the tag. An empty array is published for every frame with no tags, which clears the HMI overlay.

Graceful corners-only degrade

solvePnP needs a non-degenerate K. The uncalibrated gripper cam ships with K all zeros, so the node checks fx > 0 and fy > 0 before solving. If intrinsics are missing or zero, it skips pose and TF, still emits corners + id for the HMI box overlay (dist is 0.0), and logs a 30-second-throttled warning. The moment a real calibration lands, pose computation resumes automatically.

fallback_intrinsics, the sim-only PnP rescue

In Gazebo the camera plugin sometimes publishes CameraInfo late, or a stripped sensor omits it, which would leave the detector corners-only and starve the aggregator of the tag_<id> TF it needs to localise anything. The fallback_intrinsics parameter is an opt-in [fx, fy, cx, cy] pinhole K, default all-zeros so it is disabled by default. When armed and no real CameraInfo has latched, the first image frame installs the fallback K so PnP and the tag-to-map TF still happen, and the node switches to live CameraInfo the instant it arrives.

Configuration

All overridable via launch or --ros-args. Defaults from tag_annotator.py:

YOLO flower detection

yolo_detector_node.py subscribes to the gripper camera's compressed stream /gripper_camera/image_raw/compressed, decodes each frame with cv2.imdecode, and runs Ultralytics YOLO under a threading.Lock. It ships a trained model, best.pt, installed to share/lupin_perception/models/best.pt, with four classes: tulip_red, tulip_white, tulip_pink, and a bug anomaly class.

It publishes:

• yolo/image_detections (sensor_msgs/Image), the annotated frame (results[0].plot()), published every frame, useful in rqt_image_view.
• yolo/detections (std_msgs/String), a JSON array of {class, confidence, bbox_xyxy}, published only when there is at least one detection.

The threading.Lock around model.predict makes inference single-flight: a frame that arrives mid-inference waits, and on a slow box that is effective frame-dropping rather than a backlog, which is the behaviour you want when the camera outruns the GPU. Parameters: model_path (resolved via get_package_share_directory), conf (default 0.25), imgsz (default 640).

best.pt provenance, and the only-white-tulips myth

best.pt is trained on photographs of physical dahlias, so it fires on nothing in Gazebo, which is exactly why the HSV stand-in below exists. On hardware the recurring "it only detects white tulips" symptom is not a classifier bug, it is camera aim: the wrist cam framing the white tag backing plate or grey walls, where the largest blob is white. The trained classes and the HSV bands are fine; reframe the shot before suspecting the model.

The HSV sim stand-in, one detector, one swap

sim_flower_detector_node.py is the load-bearing reason the sim-to-hardware gap is small. Because best.pt sees nothing in Gazebo, in sim Lupin segments the bright emissive blossom colours the greenhouse world paints, then publishes the exact same /yolo/detections JSON contract the trained YOLO emits on hardware. Everything downstream, the aggregator, the twin, the HMI, is byte-identical across the two targets, and the only thing that changes is which detector node is running.

It segments four classes in HSV (cv2.cvtColor(..., COLOR_BGR2HSV)), separating them by saturation, not just hue:

For each class it takes the largest blob over min_area_px, scales confidence with blob area, and emits one {class, confidence, bbox_xyxy} entry, the same shape yolo_detector produces. It also publishes an annotated debug image on /sim_flower/overlay (green boxes on the segmented blossoms) so you can re-tune the bands against the Gazebo render, which shifts them from their initial estimate.

The aggregator, fusion, discovery, confirmation

perception_aggregator.py is where two un-correlated 2-D detectors become the located, typed flowers the rest of the stack needs. It subscribes five inputs: tag-detection JSON, YOLO JSON, /mission/state, and /joint_states (for the arm pan angle that drives bloom placement), and it broadcasts no TF of its own.

TF-project tags into map, debounced

For every tag in the detection JSON, the aggregator looks up tag_<id> into the map frame and maintains a de-duplicated registry. A tag becomes discovered only after min_sightings (default 3) successful map-localised sightings, which debounces a single noisy frame. It keeps the pose from the closest view (best_dist), and detections beyond max_tag_distance_m (default 2.5 m) are counted in-frame but not pose-committed, because solvePnP error grows with range for a 4 cm tag. Those three knobs together, keep-closest-view, the distance gate, and the sightings debounce, are how a trustworthy map pose is wrung out of a small, noisy marker.

The TF lookup is zero-timeout by design:

tf = self._tf_buffer.lookup_transform(
    self._map_frame, child, Time(), timeout=Duration(seconds=0.0),
)

A blocking timeout would deadlock the single-threaded executor that also fills the TF buffer: the lookup would wait for a transform that only the same, now-blocked, executor can deliver. The detector re-broadcasts each tag TF every frame, so asking for Time() (latest available) is current to within one frame anyway, and the zero-timeout lookup either succeeds immediately or returns None for the silent-guard path below.

Known-layout bloom placement (Stream B)

This is the cleverest idea in the stack, and it used to be invisible. The greenhouse layout is known: the course tag_locations.json (the same file that seeds the sim world and the mission's approach poses) gives every planter bench as an exact rectangle, the tables dict, in metres, the real benches are 1.10 x 0.25 m. So a bloom's map position is fully determined without depth, register that known layout onto the live map, then place the bloom inside the real rectangle.

The geometry is split producer / consumer so the aggregator stays a pure fuser. A small box_layout_publisher node is the producer; the aggregator consumes its boxes off /perception/box_geometry_json (a MapBox per tag: centre, outward-normal yaw, width, depth) and only falls back to a tag-anchored box when none arrive. Both sides live in box_geometry.py, a pure, ROS-free, unit-tested module (it mirrors how lupin_mission/approach.py isolates the approach-pose geometry); an OpenCV/occupancy node could publish the same contract.

The producer turns the tag detections into map-frame boxes in two steps:

• Register the layout. The detected tags form a constellation whose map positions should match their tag_locations.json coordinates up to a rigid motion. solve_rigid_2d fits that motion, a closed-form 2-D Kabsch (rotation + translation, no scale) over every discovered tag's (json_xy, map_xy) pair. Orientation comes from the whole constellation of tag positions, far more stable than any single tag's quaternion. Where the map is already aligned to the layout (the sim case) the fit is the identity; on hardware the boxes snap into the registered orientation once a second tag is seen.
• Emit the rectangle. For each tag, nearest_table_rect picks its bench, the four corners are transformed into the map frame, and map_box_from_rect packs them into a MapBox: long edge → width (lateral l), short edge → depth (normal n), yaw signed toward the tag so the box front faces the aisle. The aggregator's box_from_map_box reconstructs the frame, and a bloom at lateral fraction f in [-1, 1], depth d in [0, 1] maps to origin + l*(f*width/2) - n*(d*depth). Drawing the full rectangle is also what "fills in" the cells the lidar can't see behind the near face.

When no box arrives for a tag (producer down, package missing, or a tag with no table) the aggregator falls back to the legacy box: a fixed StandardBox (fallback width=0.80, depth=0.40) hung off the single tag pose via box_from_tag, with the in-plane normal n from the tag quaternion, nx = 2(qx*qz + qw*qy), ny = 2(qy*qz - qw*qx). That legacy path is the old, wrong-sized overlay (a 0.80 x 0.40 box per tag, so ~4 overlapping rects per bench); the producer is what makes the real rectangle the normal case.

Each YOLO detection's lateral fraction comes from fusing two cues: the arm pan angle (coarse, and the primary cue once the Stream-C sweep lands) and the detection's image-x (fine). box_geometry.lateral_fraction computes bearing = (pan - pan_center) + bbox_cx_norm * camera_half_fov, then scales it by pan_half_span + camera_half_fov and clamps to [-1, 1]. Pre-Stream-C, with a static pan, image-x alone drives placement, and detection_image_width=640 is only a spread-tuning constant used to normalise bbox centre-x to [-1, 1], the real camera may differ, but getting it wrong shifts spread magnitude, not correctness.

bin_detections then groups the per-scan detections into lateral_columns (default 7) across the box, picks the dominant non-anomaly species per occupied column, raises anomaly if the bug class lands anywhere in that column, and scatters depth with a deterministic golden-ratio jitter (RNG-free, so the same scan always renders the same bed) before calling geom.place(f, d). The output is a FlowerPoint[] of map positions plus a 4-corner box_footprint polygon.

Why association is temporal, not geometric

The two detectors live on different, differently-calibrated cameras, the Orbbec body cam reads the tags, the uncalibrated gripper cam reads the flowers, so there is no shared 3-D frame to overlap their boxes in. Association is therefore temporal: whatever the gripper cam classifies while the robot is parked SCANNING tag X is attributed to tag X, and the mission's SCANNING gate is the substitute for geometric overlap.

_focus_tag() reads /mission/state: it attributes only when lifecycle_state is MONITORING or INSPECTING and mission_phase contains SCANNING, targeting current_target. Run standalone with no mission, it falls back to the nearest detected tag in the latest frame. While the mission is running but not scanning, e.g. EXPLORING or driving between beds, it refuses attribution, the explicit guard against drive-by misassociation.

The per-scan accumulator is the other half of this. The dominant-species summary is derived from _scan_dets, a tag-scoped accumulator that is cleared the instant the focus tag changes, so a bug or a species seen at the previous pot cannot bleed into this pot's summary.

One registry, three outputs

The latched discovery feed, the KIND_FLOWER observations, and the confirm_tag service all read the same registry state.

/perception/discovered_tags (lupin_msgs/DiscoveredTags) is published at publish_rate_hz (default 2.0) with RELIABLE + TRANSIENT_LOCAL (depth 1), so a late-subscribing orchestrator instantly sees the whole set. Only entries with sightings >= min_sightings and a non-null pose are included. Each DiscoveredTag carries tag_id, pose_in_map, species, species_confidence, anomaly, and sightings.

/floranova/observations carries KIND_FLOWER Observations (the twin's intake, RELIABLE + TRANSIENT_LOCAL depth 50) whenever the fused state changes. The re-emit trigger fires on a change in species, anomaly, a confidence shift past a > 0.1 deadband, or a change in flower_count (the number of occupied lateral columns). Position-only shifts at the same count are deliberately not a trigger, the twin is latest-wins, so the next summary or count update carries the new positions anyway. The embedded FlowerObservation carries the tag's map pose as pose (the tag anchor, not a bloom), plus species, confidence, anomaly, the per-bloom flowers array, and the box_footprint polygon; obs.tag_pose_in_map is set too.

/perception/confirm_tag (lupin_msgs/srv/ConfirmTag) is the hardware-only visual-confirmation gate the orchestrator calls before a bridge scan. Given expected_tag_id, it returns detected = true only when the registry has that tag, it is pose-localised, has >= min_sightings, and was last seen within freshness_s (default 5.0 s). Otherwise detected = false with an error_message that prefixes the tag id: tag <id> not discovered, tag <id> below sighting threshold, or tag <id> stale. detection_confidence is min(1.0, sightings / min_sightings).

Message and service contracts

FlowerObservation.pose is the tag anchor, while each FlowerPoint.position is a per-bloom map xy (z=0 in v1). That distinction is the whole v2 redesign: the HMI stops stacking every flower on the marker and renders them scattered inside box_footprint.

Topic and service contract

Running it

ros2 launch lupin_perception perception.launch.py \
  image_topic:=/camera/image_raw \
  image_qos:=reliable \
  use_sim_time:=true

ros2 launch lupin_perception perception_stack.launch.py

The gripper cam can also feed the tag detector for in-hand tag tracking:

ros2 launch lupin_perception perception.launch.py \
  image_topic:=/gripper_camera/image_raw \
  camera_info_topic:=/gripper_camera/camera_info

Inspecting the pipeline

# Camera stack alive?
ros2 topic hz /camera/color/image_raw
ros2 topic echo --once /camera/color/camera_info

# Live tag overlay JSON
ros2 topic echo /camera/tag_detections_json --no-arr

# The discovery registry as the robot explores
ros2 topic echo /perception/discovered_tags

# A tag TF (transient, only while the tag is in view)
ros2 run tf2_ros tf2_echo camera_color_optical_frame tag_1

If the HMI shows the feed but never any boxes, check the JSON rate with ros2 topic hz /camera/tag_detections_json: zero means the node isn't receiving either the image or CameraInfo. The aggregator's fusion, the debounce, the per-scan species summary, the bloom placement (flowers land inside the box, never on the tag, and bug raises anomaly on a located flower), and confirm_tag, is covered by desktop-runnable unit tests that monkeypatch the TF lookup, so box_geometry.py and the placement logic can be validated with no robot and no Gazebo.

See also