Deploy & networking

Lupin runs on two machines. The robot keeps the controller-facing nodes alive on systemd the moment it boots, so manual drive, arm, and gripper work with no laptop attached. The laptop brings up the whole autonomy and HMI stack, HMI, SLAM, Nav2, twin, mission, perception, RViz, with a single ros2 launch. The two halves find each other over WiFi through the vendor MIRTE image's FastDDS discovery server, and that link is where most demo-day pain lives.

This split is deliberate: the robot only does what must live close to the hardware; everything CPU-heavy (JSON encoding for the HMI, Nav2, SLAM, the torch-based YOLO detector) runs on the laptop where there is headroom. This page is the runbook for both halves, the systemd units on the robot, the one-command launch on the laptop, the DDS link that joins them, and the six distinct ways that link breaks.

The robot / laptop split

Robot-side services

All three Lupin units are PartOf=mirte-ros.service, so when the vendor service stops or restarts (which respawns its own cameras and controllers) the Lupin units stop and restart with it. Each is User=mirte, sources /opt/ros/humble, mirte_ws, and ros2_ws, and runs Restart=on-failure with a short RestartSec.

lupin-onboard.service

Runs onboard.launch.py, which starts the controller-facing nodes (all use_sim_time=false):

• twist_mux, arbitrates the three command-velocity inputs and remaps cmd_vel_out to /mirte_base_controller/cmd_vel (the stamped sink the real Mirte's mecanum controller listens on). The arbitration priorities are Xbox dead-man 100, web HMI joystick 50, Nav2 smoother 10, the operator override always wins. See /docs/teleop.
• arm_preset_server, serves /lupin/arm/preset (named poses: home/tuck/pick/place). lupin-auto-home calls it at boot.
• arm_calibrate_server, serves /lupin/arm/calibrate (Hiwonder zero-offset flow, hardware-only; the underlying bus services exist only on the real Mirte).
• gripper_action_bridge, translates /lupin/gripper/set_angle_with_speed into a goal on mirte_master_gripper_controller/gripper_cmd.
• light_strip_bridge, owns /lupin/leds/{set,auto} with the per-robot color_order: BRG fix for Mirte-247264's wired LED strip.

lupin-cameras.service

An ExecStartPre runs scripts/kill-vendor-cameras.sh to release the /dev/video* and USB device handles, then cameras.launch.py relaunches the cameras from config/cameras.yaml:

• The Orbbec astra via the vendor astra.launch.py with FPS overrides (default color 30 fps at 640x480; depth and point cloud off by default, they are the single biggest producer-side CPU drain on the A55 Pi). Point cloud requires depth, so the launch couples them.
• One usb_cam_node_exe per auto-detected gripper-camera v4l device (default 20 fps), using the same v4l name filter as the vendor's gripper_camera.launch.py.

Relaunched topic names are identical to the vendor's (/camera/color/image_raw, /gripper_camera/image_raw) so downstream consumers see no interface change.

A robot-side web_video_server also starts here, bound to 0.0.0.0:8091 with default_stream_type: ros_compressed. Running it on the robot keeps the camera-subscriber loop on the same host as the publisher, so only JPEG-encoded MJPEG crosses WiFi, not raw RGB. The Vite /_video same-origin proxy in the HMI forwards browser requests to the robot's :8091.

The unit is sized MemoryMax=384M for color@30 (plus depth + point cloud when toggled on).

lupin-auto-home.service

A Type=oneshot that runs after lupin-onboard is up. It heals controllers stuck by the vendor first-boot race and then drives the arm to its low-load home pose via /lupin/arm/preset. It is Restart=on-failure with RestartSec=15 and StartLimitBurst=4, so if the discovery server or controller_manager is slow to settle on a cold boot, a later attempt picks up the heal.

Installing the robot-side services

The units are not in the stock MIRTE image. Install once per robot image:

sudo apt install ros-humble-twist-mux v4l-utils

cd ~/ros2_ws && colcon build --packages-up-to lupin_bringup --symlink-install
source ~/ros2_ws/install/setup.bash

sudo bash $(ros2 pkg prefix lupin_bringup)/share/lupin_bringup/scripts/install-onboard-systemd.sh
sudo bash $(ros2 pkg prefix lupin_bringup)/share/lupin_bringup/scripts/install-cameras-systemd.sh

Both install scripts are idempotent and accept --uninstall. The onboard installer also installs lupin-auto-home alongside (enabled, but not restarted, it only fires on the next boot), adds mirte to the input group for the Xbox controller, and installs the Xbox-rebind udev rule.

Laptop-side bring-up

One command brings up everything on top of the robot's onboard services:

ros2 launch lupin_bringup hardware.launch.py

Each subsystem is behind a boolean flag, so you can opt in or out without editing files:

use_sim_time is forced false throughout. Common invocations:

# Default operator mode, HMI + SLAM + Nav2 + twin + RViz.
ros2 launch lupin_bringup hardware.launch.py

# Headless smoke test, no HMI, no RViz.
ros2 launch lupin_bringup hardware.launch.py web:=false rviz:=false

# Full mission run.
ros2 launch lupin_bringup hardware.launch.py mission:=true

# Just the HMI on top of a parked robot.
ros2 launch lupin_bringup hardware.launch.py slam:=false nav2:=false

The launch prints a clickable HMI URL banner with the auto-detected LAN IP. Open https://<laptop-ip>:8090 once Vite logs ready in NNN ms; accept the self-signed cert once per device.

Event-driven sentinel cascade

Instead of fixed sleeps, the launch chains tiny ros2 topic echo --once sentinels through OnProcessExit handlers. Each stage only fires when its subsystem flag is enabled:

wait_for_scan (/scan) ──▶ slam_toolbox
                              │
                              ▼
wait_for_map (/map) ──▶ wait_for_tf (odom→base_link) ──▶ Nav2 lifecycle

The slam_toolbox node is spawned directly (not via online_async_launch.py) with respawn=True, so the HMI's /lupin/nav/clear_map service can SIGTERM it and let it respawn with an empty pose graph. The wait_for_tf stage matters on hardware: a cold DDS-over-WiFi /tf subscription takes a few seconds to warm up, and Nav2's costmap activation does a single canTransform() call that otherwise fails and wedges the lifecycle. The full cascade and its self-heal logic live in /docs/operations.

Self-sourced DDS env

Before anything else, hardware.launch.py calls _apply_dds_env_from_ros_env_sh(), which reads ~/.config/lupin/ros-env.sh (if present) and applies its export FOO=bar lines into its own process environment. This makes the launch self-configuring: whichever terminal you run it from, every child, rosbridge, Nav2, RViz, inherits a consistent DDS-over-WiFi env, even from a terminal opened before the DDS setup script ran. When it applies anything, it prints a banner naming the variables it set.

The same function feeds _default_robot_ip(), which derives the HMI camera-proxy target from ROS_DISCOVERY_SERVER (the <ip>:11811 endpoint it just applied). So whichever IP you pointed DDS at, robot AP 192.168.42.1 or wired 10.42.0.1, the video tab follows automatically, one IP drives the whole laptop side.

Laptop ↔ robot DDS over WiFi

ROS 2's default FastDDS uses UDP multicast for participant discovery, which most WiFi access points (the robot's own AP, university captive portals, the iPhone hotspot) silently drop. The symptom: ssh and ping to the robot work, but ros2 topic list from the laptop returns only /parameter_events + /rosout, and the HMI loads but every panel is blank.

The vendor MIRTE image ships a FastDDS discovery server for exactly this case, and understanding how it routes traffic explains every failure mode below. A discovery server splits the DDS graph into two channels: metatraffic (who exists, what topics they offer, what QoS) is mediated by the server on the wildcard 11811 port, while user data (the actual samples) flows peer-to-peer over the unicast locators each participant announces during discovery. The server is a phone book, not a relay.

Both halves point their metatraffic at the same server and receive the full graph back (that is what SUPER_CLIENT means, a plain CLIENT would only learn the participants it explicitly asked about). Then they talk data directly. This is why the server fixes the multicast drop, and it is also the seed of the multi-homing trap: discovery and data ride different channels, so one can work while the other silently dies.

ros2 launch lupin_bringup sim_full.launch.py

echo 'export MIRTE_FASTDDS=true' >> /home/mirte/.mirte_settings.sh
sudo systemctl restart mirte-ros.service lupin-onboard.service lupin-cameras.service
sudo ss -lnup | grep 11811        # expect a listener on 0.0.0.0:11811 (UDP)

./lupin_bringup/scripts/setup-laptop-dds-env.sh                  # uses 192.168.42.1
./lupin_bringup/scripts/setup-laptop-dds-env.sh 10.42.0.1        # wired robot IP
./lupin_bringup/scripts/setup-laptop-dds-env.sh --uninstall      # back out

ros2 daemon start && sleep 5
ros2 topic list | wc -l                          # expect 50+, not 2
ros2 topic echo /io/power/power_watcher --once    # should print BatteryState

Three env vars, three jobs

setup-laptop-dds-env.sh writes ~/.config/lupin/ros-env.sh with exactly three exports and nothing else (it is pure, with no side effects, because every new terminal sources it):

export FASTRTPS_DEFAULT_PROFILES_FILE="$HOME/.config/lupin/fastdds_super_client.xml"
export ROS_DISCOVERY_SERVER="<robot-ip>:11811"
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp

Each var does a separate job, and both of the first two are load-bearing:

• FASTRTPS_DEFAULT_PROFILES_FILE, points FastDDS at the generated XML, which applies the SUPER_CLIENT discovery protocol and the remote-server locator. The profile carries the fixed GUID prefix 44.53.00.5f.45.50.52.4f.53.49.4d.41 (the bytes spell DS_PROSIMA), which must match the server's identity, the robot's participants use the same prefix, so everyone agrees on which server is server 0.
• ROS_DISCOVERY_SERVER, the FastDDS 2.6 auto-config shortcut that flips the participant into discovery-server mode and supplies the same <ip>:11811 endpoint. On Humble this env var alone does not apply SUPER_CLIENT semantics, and the XML alone does not reliably trigger the connection. You need both.
• RMW_IMPLEMENTATION, forces rmw_fastrtps_cpp so the XML and the env var are actually read, a Cyclone-DDS default would ignore both.

The robot side never changes, its participants always point at 127.0.0.1:11811. The default laptop-side robot IP is 192.168.42.1 (the robot's own AP); pass a different IP when you swap networks.

Source order matters

Two source lines bring up every laptop terminal, and the order is not free:

source ~/ros2_ws/install/setup.bash          # FIRST, lupin_* packages + RViz meshes
source ~/.config/lupin/ros-env.sh            # LAST, DDS env

Source them backwards and ROS_DISCOVERY_SERVER ends up empty. The reason is a vendor colcon hook: mirte_ws's mirte_fastdds_discovery_setup blanks ROS_DISCOVERY_SERVER when setup.bash runs (it expects to own DDS config itself). So ros-env.sh must be sourced after setup.bash, or the workspace overlay quietly wipes the var you just set, the terminal falls back to multicast, and you get the classic LIVE-but-dead HMI with no obvious cause.

The per-process LIVE-but-dead trap

A node's DDS mode is fixed at launch from its environment, not from whatever the current shell holds. This is the single most common demo-day confusion: the HMI shows LIVE, but no battery, no telemetry, no drive, while a ros2 topic list in a different terminal shows the robot's topics just fine.

Why only MJPEG crosses WiFi

Camera frames are the heaviest payload, so web_video_server runs on the robot. With default_stream_type: ros_compressed it subscribes to the <topic>/compressed JPEG companion topics that the cameras already publish, re-emits MJPEG, and the laptop's Vite /_video proxy fetches that. Raw RGB never leaves the Pi. An earlier layout that ran web_video_server on the laptop pushed every raw frame across the 2.4 GHz AP and produced lag, frame pile-ups, and black screens.

When the link breaks: a triage tree

Six distinct failure modes wear the same disguise, a blank HMI or a ros2 topic list that returns 2. The key is that each one breaks a different part of the discovery-server topology, so the discriminating questions are narrow. Walk the tree from the top; each leaf names the fix.

WiFi roam masquerades as dead controllers

When everything dies at once, suspect the laptop's network before the robot. This one sent a whole debug session down a non-existent rabbit hole more than once; the full story is in /docs/war-stories.

Robot multi-homing: discovery survives, data dies

This is the subtlest failure on the stack, and it is a hard blocker, not a caveat. When the robot is dual-homed, eth0 on the cable plus wlan0 still running its AP, its participants announce both locators, 10.42.0.1 and 192.168.42.1. Discovery rides the server, so it works. Robot-to-laptop best-effort streams (scan, telemetry) reach the laptop, so the HMI looks LIVE. But the laptop-to-robot data channel tries to reach the robot's announced locator, and if it picks the unroutable AP address, every command silently lands nowhere.

A multi-homed robot poisons DDS by advertising an unroutable locator; one-way comms that work for discovery but not data are the signature, and the full root-cause walk is in /docs/war-stories.

Cold-boot discovery-server race

The vendor's mirte_ros.sh forks fast-discovery-server exactly once, with no retry and no postcondition check, and at cold boot that fork races two things that may not be ready: wlan0 coming up, and a fully-populated PATH. If it loses, the journal logs Discovery Server wasn't able to allocate the specified listening port or fast-discovery-server tool not found, the script silently continues, and systemctl is-active mirte-ros reports active while nothing is bound to 11811. The service lied.

The recovery exploits the fact that the robot's nodes are clients retrying 127.0.0.1:11811 forever, so you do not need to restart the stack, just bind the server they are waiting for:

ssh lupin 'sudo ss -lnup | grep 11811' || echo NOTHING-BOUND
# nothing bound -> launch the server standalone; live nodes attach within seconds:
ssh lupin 'env -u FASTRTPS_DEFAULT_PROFILES_FILE setsid bash -c \
  "exec fast-discovery-server -i 0 -l 0.0.0.0 -p 11811" >/tmp/lupin-disc.log 2>&1 </dev/null &'
ssh lupin 'sudo ss -lnup | grep 11811'   # expect UNCONN 0.0.0.0:11811 users:(("fast-discovery-",…))

SHM late-joiner: names but no samples

On the discovery-server stack, a participant that joins after the stack is up, a late ros2 topic echo, an rclpy subscriber, a ros2 bag record, discovers every topic name through metatraffic fine, but the shared-memory user-data path to a late joiner never delivers samples. It looks like a dead stack, but the live nodes are working.

Zombie hijack and stale SHM

A Ctrl-C'd launch leaves zombie ROS processes still binding DDS ports 7410-7447; the discovery server hands topic info to the zombie and your fresh participant gets nothing. Separately, a dirty kill (kill -9, OOM, a Ctrl-C during a hang) leaves /dev/shm/fastrtps_* segments that the next participant fails to lock, so it comes up alive but never completes discovery. Sweep both before relaunching:

pkill -9 -f 'ros2|rviz2|rosbridge|slam_toolbox|twin_node|tag_annotator|web_video_server' 2>/dev/null
ros2 daemon stop; pkill -9 -f ros2cli
rm -f /dev/shm/fastrtps_* /dev/shm/sem.fastrtps_*
ls /dev/shm | grep -i fast || echo "laptop SHM clean"

Clock sync on a robot with no RTC

The Orange Pi 3B has no hardware RTC, and in AP mode it has no NTP source, so it boots with whatever clock it had at shutdown (caught 4h34m behind once). A wrong clock floods slam_toolbox with Message Filter dropping message and stops /map, but the dangerous part is the cure: running date -s on a live ros2_control stack steps system_clock forward thousands of seconds in one tick, controller_manager sees everything expire at once, the activation spawner dies with exit code -11, controller_state goes silent, and the wheels go dead. Worse, the clock step leaves stale FastDDS SHM that a plain systemctl restart cannot clear, so the wedge survives a restart.

The chrony fix (now live)

The proper fix, deployed and verified on Mirte-247264 on 2026-06-08, stops stepping the clock mid-session at all. install-clock-sync.sh puts a lupin-clock-sync oneshot on the robot, ordered Before=mirte-ros.service, that disciplines the robot to the laptop's chrony with a single chronyc makestep before ros2_control ever starts, so the one big correction lands on a clock nothing is watching yet, and for the rest of the session chrony only slews. It also comments out Ubuntu's stock makestep 1 3, which would otherwise let chrony step a live stack the first time it reaches a far-off source (the war story behind this).

The catch is that the robot syncs from the laptop, so the laptop has to actually serve NTP, and chrony being installed is not the same as serving. On the wired LAN that is a one-line drop-in on the laptop:

# laptop: serve UTC to the robot LAN (chrony installed != serving)
printf 'allow 10.42.0.0/24\nlocal stratum 10\n' | sudo tee /etc/chrony/conf.d/lupin-server.conf
sudo systemctl restart chrony && ss -lnup | grep ':123'   # expect chronyd on 0.0.0.0:123

allow admits the robot; local stratum 10 makes the laptop serve its own clock even with no upstream, so AP mode (no internet) still works. After a reboot the robot's chronyc tracking shows Reference ID … (10.42.0.2) at stratum 4 and drift collapses to microseconds. The robot-side install and the operator runbook are in DEMO_DAY_WIRED.md (§A.4 robot, §B7 laptop).

When the laptop NTP isn't reachable at boot (laptop joined late, or the server drop-in was never added), the robot still comes up skewed, and post-boot-sync.sh is the fallback that corrects it. It encodes the safe procedure: it only steps the clock when the drift exceeds 3 s, and only with the stack stopped, then wipes SHM, then restarts. Re-running it mid-session as a health check is a no-op on the clock.

Install the chrony oneshot on the robot once (it enables itself and activates on the next reboot; it also disables the stock makestep):

ssh lupin-wired 'sudo bash -s' < lupin_bringup/scripts/install-clock-sync.sh
ssh lupin-wired sudo reboot

The post-boot-sync.sh fallback is set up once per laptop as a symlink (so a future colcon build of lupin_bringup updates it in place):

mkdir -p ~/.config/lupin
ln -s "$(ros2 pkg prefix lupin_bringup)/share/lupin_bringup/scripts/post-boot-sync.sh" \
      ~/.config/lupin/post-boot-sync.sh

~/.config/lupin/post-boot-sync.sh                       # mirte@192.168.42.1 (default)
ROBOT=mirte@10.42.0.1 ~/.config/lupin/post-boot-sync.sh # wired

Demo transport choices

Three substrates can carry the laptop ↔ robot DDS link, and they are not equal. The recurring failure mode at a venue is networking jitter and discovery flakiness, so the ranking is deliberate.

The router and the cable both rely on the laptop dual-homing: eduroam WiFi for internet (Gemini Live voice is a demo wow-factor and needs it), a separate interface for the DDS LAN. Linux routes per-destination, the robot subnet over ethernet, everything else over WiFi, as long as the DDS interface carries ipv4.never-default yes so the cable never becomes the default route.

The wired demo kit

The wired path is one static-IP scheme, a private /24 that avoids the AP's 192.168.42.0/24 and typical eduroam ranges:

The robot-side delta is small: give eth0 a static IP (nmcli ... ipv4.addresses 10.42.0.1/24 ipv4.never-default yes), then single-home by disabling the AP (the multi-homing fix above). The laptop-side delta is even smaller: setup-laptop-dds-env.sh 10.42.0.1. That one IP drives everything, because hardware.launch.py auto-derives the camera-proxy target from ROS_DISCOVERY_SERVER, the video tab follows with no second place to edit. The robot's discovery server binds 0.0.0.0:11811 (all interfaces), so it answers on the cable with no robot-side reconfiguration; if ss shows it pinned to a specific AP address instead, the wired link will not work and the durable fix is to make the server listen on the wildcard. The full scan-able runbook is DEMO_DAY_WIRED.md, and the AP twin is DEMO_DAY.md.

See also