Operations & troubleshooting

This is the runbook for operating Lupin on the real MIRTE Master V2. It says what the bring-up code actually does, where the hardware lies to you about its own health, and how to walk a wedged stack back to life without guessing. Read this page as the runbook, every failure mode below is written as Symptom → Cause → Fix, and almost every one has cost the team a real hour at the robot.

The runtime is split across two machines: the robot runs three Lupin systemd units on top of the vendor mirte-ros.service, and the laptop brings up the HMI, SLAM, Nav2, the digital twin, and perception in a single ros2 launch lupin_bringup hardware.launch.py. See Deploy & networking for the install steps and the laptop ↔ robot DDS setup.

The event-driven bring-up cascade

hardware.launch.py does not sleep between stages. It chains tiny "wait for a topic" sentinel processes, each one is a real one-shot ros2 topic echo --once (or a TF lookup) that exits the moment its target appears, and RegisterEventHandler(OnProcessExit) fires the next stage on that exit. SLAM, Nav2, and the TF gate run as a dependency chain; the web HMI, the twin, perception, YOLO, RViz, and the mission orchestrator are fire-and-forget at t=0 because they tolerate their inputs arriving late.

Why event-driven and not delay-driven: a fixed sleep either wastes time on a fast boot or fires too early on a slow one, and when it fires too early the downstream stage fails silently under a flood of retry warnings. With sentinels a stalled stage is immediately visible, the launch is simply parked on wait_for_scan or wait_for_map, which names the exact dependency that never came up.

Each sentinel is gated by its own subsystem flag, so disabling one stage does not block the rest:

• wait_for_scan, echoes one sensor_msgs/msg/LaserScan on /scan, then starts slam_toolbox. On hardware the robot's RPLidar is already publishing before the laptop launches, so this usually clears instantly.
• wait_for_map, echoes /map (nav_msgs/msg/OccupancyGrid) with RELIABLE + TRANSIENT_LOCAL QoS. The echo must match slam_toolbox's QoS or the subscription never connects and the cascade parks here forever. This is the textbook QoS-negotiation footgun: a --qos-reliability reliable --qos-durability transient_local echo against a best-effort default would never see a sample even though /map is publishing.
• wait_for_tf, runs ros2 run lupin_bringup wait_for_tf odom base_link 30.0, blocking until the odom → base_link transform resolves in the laptop's tf2 buffer (timeout 30 s, exits 0 on success, 1 on timeout).

The launch self-sources ~/.config/lupin/ros-env.sh into its own process before spawning anything, so every child (rosbridge, Nav2, RViz) inherits a consistent discovery-server env regardless of which terminal you launched from. It prints a clickable HMI URL banner with the auto-detected LAN IP, and if the env was applied it prints [lupin_bringup] applied DDS env from ….

One command, or eight terminals

The single launch is the one-command path and the right default. But DEMO_DAY.md deliberately recommends an 8-terminal layout for on-stage work, one subsystem per terminal, so any single piece (SLAM, Nav2, rosbridge, the joystick) is killable and restartable without taking the rest of the stack down with it. Pick the unified launch for a clean bring-up, pick the terminal-per-subsystem layout when you expect to debug live in front of an audience. Running both at once double-launches everything, so do one or the other, never both.

systemd units and lifecycle coupling

On the robot, three Lupin units hang off the vendor mirte-ros.service, which is the anchor that owns ros2_control, the discovery server bind on port 11811, and the vendor camera nodes. The wiring is After + Wants for ordering, and crucially PartOf=mirte-ros.service on all three Lupin units, so a vendor mirte-ros restart re-fires the entire Lupin stack, including the auto_home controller heal and the camera kill-and-relaunch.

The ordering is not arbitrary. lupin-onboard is After=mirte-ros because its twist_mux, arm_preset_server, and estop_bridge connect HMI commands to controllers that have to exist first (the nodes tolerate late controllers, the action clients retry). lupin-auto-home is After=lupin-onboard because the heal calls /lupin/arm/preset, which arm_preset_server inside lupin-onboard serves. lupin-cameras runs in parallel, and is PartOf precisely so that when mirte-ros restarts and respawns its vendor cameras, our kill-and-relaunch step runs again to take the devices back.

Two unit-file asymmetries are load-bearing and were both paid for in a boot journal:

The units also carry MemoryMax caps, lupin-onboard at 256M, lupin-cameras at 384M, the latter sized for color at 5 fps plus optional depth and point-cloud on the A55-class Pi. These are real limits, not advisory: crank the camera frame rates past the budget and the OOM killer starts reaping the camera pipeline, which presents as cameras randomly die with nothing obviously wrong in the launch.

Boot-time controller self-heal (auto_home)

lupin-auto-home.service runs the auto_home one-shot after lupin-onboard. Its first job is to self-heal the ros2_control stack, it closes the vendor first-boot spawner race before parking the arm. The vendor mirte-ros stack races ros2_control_node's YAML auto-load against three controller-spawner launches, and when that race fires the five target controllers end up stranded in one of three bad states.

heal_controllers is a true three-state repair machine, not a blind reload, and the missing-entirely case is the dangerous one. The old heal handled only unconfigured and inactive; a controller absent from list_controllers reads as None, None not in ('unconfigured', 'inactive') is true, and the empty needs-activate set tripped a "nothing to do" early exit, a false positive that silenced exactly the wedge the heal was meant to catch. The fix uses a positive done-check (every target must explicitly read active) and loads anything missing.

The five targets driven to active are joint_state_broadcaster, pid_wheels_controller, mirte_master_arm_controller, mirte_master_gripper_controller, and mirte_base_controller. The three-valued return is what prevents a restart storm:

• True, every target ended active. The node continues to home the arm and exits 0.
• False, list_controllers worked at least once and the heal ran, but not all targets reached active. main() exits 1, and systemd's Restart=on-failure gives a later attempt a shot, by which point mirte-ros has usually settled.
• None, list_controllers never returned valid data within the poll budget, so the state is genuinely unverifiable and blindly LoadController-ing each target is what once caused an "already loaded" retry storm. main() converts None into exit 0 so systemd does not spin, and the operator re-triggers via post-boot-sync.sh.

After the heal the node waits up to 30 s for /joint_states (all four arm joints) and up to 30 s for /lupin/arm/preset, settles 2 s, and calls the preset service with name='home' so the arm parks in its measured low-gravity rest pose, which dodges the Hiwonder shoulder_lift thermal trip. Opt out with LUPIN_AUTO_HOME=false in ~/.mirte_settings.sh. The unit is Type=oneshot with RemainAfterExit=yes, Restart=on-failure, RestartSec=15, capped at StartLimitBurst=4 over a 240 s window.

SLAM map reset via process respawn

The HMI "Erase map" button calls /lupin/nav/clear_map (std_srvs/srv/Trigger), served by slam_reset_node. There is no clean way to wipe the map in slam_toolbox 2.6, /slam_toolbox/clear_changes only drops the localization edit buffer, not the scan-built occupancy grid. So the pragmatic v1 SIGTERMs the slam_toolbox process and lets respawn bring it back empty.

This only works because hardware.launch.py spawns slam_toolbox directly (not via online_async_launch.py) specifically so it can pin respawn=True with respawn_delay=1.0. The reset:

1. Best-effort clears the Nav2 costmaps (/local_costmap/clear_entirely_local_costmap, /global_costmap/clear_entirely_global_costmap) so stale lethal cells do not stay inflated under the robot while SLAM is down.
2. pgrep/SIGTERMs every process matching async_slam_toolbox_node.
3. The launch's respawn=True brings slam_toolbox back with an empty pose graph, and /map repopulates after roughly 5-10 s.

If no matching process is found, the service returns success=false with a message saying slam_toolbox may not be running, a useful signal that you reset the wrong thing or SLAM was never up. Do not switch the launch back to online_async_launch.py, that would leave the SIGTERM dead with no respawn and the erase button broken.

The deepest gotcha: "active" is a lie about the motors

Before any of the runbook entries, internalize the one distinction that explains the worst Lupin failure. The telemetrix board's MCU runs on logic/USB power, separate from the motor H-bridge that runs on battery power. So the board can keep reading wheel encoders and streaming /joint_states at 10 Hz, and every controller can report active, while the H-bridge output is dead and un-energized. Every software signal looks healthy. Only a physical test reveals the truth.

This is why the triage tree below leads with feel the wheels by hand, and why systemctl is-active and ros2 control list_controllers are not allowed to be the final word on whether the robot can drive.

Triage: HMI is LIVE but nothing flows

Four different failures look almost identical from the operator's chair, the robot will not drive or telemetry is blank, yet the HMI pill reads LIVE. They route apart on cheap, specific tests.

Robot will not drive, but every controller reads "active"

Symptom. The robot will not move, not from the HMI, not from a raw cmd_vel, not even from the priority-100 /cmd_vel_joy. Yet ros2 control list_controllers shows every controller active, /joint_states streams at 10 Hz, the battery is good, and the e-stop button is out.

Cause. The telemetrix motor-driver board is wedged. Its MCU runs on logic/USB power so it keeps reading encoders, which is why /joint_states stay healthy and every controller reports active, but the motor H-bridge output has gone dead. systemctl is-active reports lifecycle state, not whether the real-time loop is pushing current to the motors.

Fix. Feel the wheels by hand first. If they free-spin the motors are un-energized, because a PID-outputs-zero fault would still hold and energize the wheel. Do a genuine hard power-cycle, main switch off about 15 s, then on. A soft reboot or systemctl restart mirte-ros keeps the USB-powered board alive and will not reset it.

Restarting mirte-ros itself drops the drive controller

Symptom. You systemctl restart mirte-ros to fix something else, and now the chassis silently ignores all cmd_vel, and HMI services and discovery-server-registered Lupin nodes time out until you also restart lupin-onboard.

Cause. A mirte-ros restart drops mirte_base_controller (the MecanumDriveController) on respawn, so the pid_wheels_controller reference interfaces go unclaimed and nothing converts cmd_vel into wheel commands. The same restart bounces the FastDDS discovery server (the vendor owns the 11811 bind), which de-registers the Lupin onboard services until lupin-onboard restarts too. And because every Lupin unit is PartOf=mirte-ros, the restart also re-fires the whole stack, the auto_home heal and the camera kill-and-relaunch, so the bounce is heavier than it looks.

Fix. Avoid restarting mirte-ros as a casual fix. After a vendor apt upgrade specifically, never restart it, a cold power-cycle is the only clean reload. When you do restart it deliberately, expect auto_home to need a re-trigger via post-boot-sync.sh to confirm 5 controllers active.

rosbridge says LIVE but accepts subscriptions and delivers nothing

Symptom. The HMI loads, the rosbridge pill reads LIVE, the journal even logs Subscribed to /xxx, but no callback ever fires, the battery pill is empty, the joystick does not drive, and an /rosapi/topics RPC also stalls. Robot-side ros2 topic echo shows the data publishing fine.

Cause. After uptime, mirte-ros's rosbridge_websocket executor stalls, the worker threads park and never deliver. This is a third distinct failure from the two blank-HMI DDS-env modes, and it sends people down the wrong path because the connection itself is healthy.

Fix. Prove it with a raw Python WebSocket subscribe plus an /rosapi/topics RPC, if both stall it is the executor, not the browser, the proxy, or localStorage. One systemctl restart mirte-ros clears it. Restarts are budgeted, a power-cycle is needed after three or more.

HMI shows LIVE but every panel is blank

Symptom. The HMI loads, the pill is green, but battery, IMU, lidar, and telemetry are all empty and you cannot drive. ssh and ping to the robot work fine.

Cause. The laptop's rosbridge cannot subscribe over DDS. Most commonly ROS 2's default FastDDS uses UDP multicast for discovery, which the WiFi AP silently drops, so ros2 topic list from the laptop returns only /parameter_events and /rosout.

Fix. Run the laptop DDS setup once and relaunch from a fresh shell. Full procedure in Deploy & networking. Quick confirmation:

ros2 daemon start && sleep 5
ros2 topic list | wc -l                          # expect 50+, not 2

HMI blank, but a fresh terminal does see robot topics

Symptom. The HMI is blank, yet ros2 topic list in a brand-new terminal shows all the robot topics.

Cause. The DDS env is per-process. The HMI's rosbridge was launched from a terminal that never sourced ros-env.sh, so only it is on multicast while your new shell is on the discovery server.

Fix. Confirm by reading the running node's environment, not a fresh shell, then relaunch the HMI from a sourced shell:

tr '\0' '\n' </proc/$(pgrep -f rosbridge_websocket)/environ | grep ROS_DISCOVERY_SERVER
# empty output confirms the rosbridge is on multicast

The lupin_web launch prints a rosbridge DDS mode: banner to catch this early.

ros2 topic list returns 2 even with the right DDS env

Symptom. Setup ran, you opened a fresh shell, and the topic count is still 2 (or it was 50+ and dropped to 2 after switching WiFi).

Cause. Either the daemon raced discovery, the network changed under a stale daemon config, stale shared-memory segments are wedging new participants, or the robot's discovery server is not listening.

Fixes, in order.

1. Daemon raced discovery. ros2 topic list auto-starts the daemon but does not wait for discovery. Run ros2 daemon start && sleep 5 first, then list.
2. Network changed. Re-run setup-laptop-dds-env.sh <new-robot-ip> and open a fresh shell. The robot side never changes, its participants always point at 127.0.0.1:11811.
3. Stale SHM after a hard kill. A kill -9 or Ctrl-C wedge leaves stale /dev/shm/fastrtps_* segments that silently break fresh participants. Wipe them before relaunching: rm -f /dev/shm/fastrtps_*.
4. Server not listening. SSH in and check the port, if empty MIRTE_FASTDDS was not picked up: sudo ss -lnup | grep 11811.

Clock skew wedges the controllers (and the SHM-wipe cure)

The single most causally tangled failure on the robot is clock skew, because the symptom appears after you try to fix it. The Orange Pi 3B has no hardware RTC, and in AP mode there is no NTP source, so it boots with whatever clock it had at shutdown. Run a naive date -s on a live stack and the recovery becomes the new outage.

post-boot-sync.sh encodes the entire dance, and it is more careful than a one-line date -s. It measures drift first and only steps the clock when drift exceeds 3 s, so re-running it mid-session as a health check does not gratuitously restart a healthy stack. When a real step is needed it stops lupin-onboard, lupin-cameras, and mirte-ros, steps the clock, removes the /dev/shm/fastrtps_* and sem.fastrtps_* segments, restarts mirte-ros, waits 25 s for controllers to re-activate, then restarts lupin-onboard and lupin-cameras.

Symptom. SLAM and RViz spam Message Filter dropping message with timestamps hundreds of seconds in the past, /map never publishes, and after a date -s "fix" ros2 control list_controllers times out and /scan stops.

Cause. A live clock step jumps system_clock forward in one tick, controller_manager's timers see everything expired at once and wedge, and the step leaves stale FastDDS SHM that jams the new controller_manager so a plain restart cannot unwedge it.

Fix. Sync the clock as the first SSH command on every cold boot, before ROS does real work, via post-boot-sync.sh. If it already wedged, fix the clock then do the reboot-free SHM wipe, stop services, rm /dev/shm/fastrtps_*, start with the clock already stable. A power-cycle is worse, the dead RTC just re-creates the wrong-clock-then-step sequence.

~/.config/lupin/post-boot-sync.sh

Cameras will not come up / device stays busy

Symptom. lupin-cameras.service restarts in a loop or its launch errors out opening /dev/video*.

Cause. lupin-cameras kills the vendor camera nodes via ExecStartPre (kill-vendor-cameras.sh) so the V4L/USB handles are free, then relaunches at config-driven low FPS on the same vendor topic names. The kill is a four-phase device-handle handoff, not a blind signal, and the wait-for-device-free part is load-bearing.

Phase 1 waits up to 30 s because mirte-ros spawns the vendor cameras several seconds after its own start, so racing it would land the SIGTERM before they exist. Phase 3 drains while the orbbec container takes 1-2 s to actually release its USB claim, which is the real reason for the wait, cameras.launch.py fails to open a busy V4L device. Phase 4 only SIGKILLs a process stuck in D-state.

Fix. The kill script already SIGTERMs, drains, then SIGKILLs stragglers, and Restart=on-failure re-runs ExecStartPre on the next attempt. If it stays stuck, restart the unit so cleanup re-runs: sudo systemctl restart lupin-cameras.

A related quirk: the gripper-cam usb_cam lazy-publishes and destroys its publisher when nothing is subscribed, after long stops, restarting lupin-cameras is the reliable way to wake it.

Controllers wedged after a "reboot" that was really a resume

Symptom. You power the robot back on and the controllers are wedged, stuck unconfigured, the wheel and gripper controllers never loaded, even though mirte-ros came up clean. The auto_home heal never re-ran.

Cause. The Orange Pi resumed from suspend, which feels like a reboot and resets /proc/uptime, but systemd's boot ID did not change. A failed Type=oneshot unit (lupin-auto-home) stays failed across a resume and systemd never re-attempts it.

Fix. last -x reboot reveals the truth, a still running line that is hours or days old means it was not a fresh boot even when uptime looks small. Re-trigger via post-boot-sync.sh, which does the idempotent reset-failed + start. Any unit that must fire per operator session, not per boot, needs an operator-driven re-trigger, not just WantedBy=multi-user.target.

Mission orchestrator goes FAULT on a cold start

Symptom. With mission:=true, the orchestrator transitions to FAULT shortly after launch.

Cause. Nav2 lifecycle activation in SLAM mode takes longer than the orchestrator's default dependency wait on a cold start.

Fix. The bring-up already raises dependency_timeout_s to 120.0 (up from the 30 s default). If your start is unusually slow, bump it further:

ros2 launch lupin_bringup hardware.launch.py mission:=true dependency_timeout_s:=180.0

A collapsed index of failure modes

Routine boot sequence

ros2 launch lupin_bringup hardware.launch.py

See also