Skip to content

Fault Decision Matrix

SymptomRead-Only Inspection & DiagnosisControlled Recovery ProcedureStrictly Prohibited Actions (Stop Conditions)
Connection FailureCheck LAN, local subnet, ping, REST 9091, SSH 22Fix dev PC NIC configuration or use a known healthy connection pathDo NOT reset router, change robot static IP, or restore factory settings
Service List active but Node Info shows no nodeSource both ROS 2 environments; run ros2 daemon stop and re-check; match exact process namesRebuild ROS 2 daemon; restart recorder ONLY when no process is running, no active recording exists, and authorization is givenDo NOT proceed if same-named processes exist, recording is active, or software version is unknown
Topic Data MissingRun topic type/info -v/echo --once; verify publisher, QoS profile, and Domain IDFix message package dependencies, QoS settings, or subscription configurationsDo NOT switch operating modes, reboot system, or stop unknown services due to single Topic timeout
Control Command Ineffective (No Motion)Check control authority, mode, E-stop, enable state, error bits, joint limits, units, and readbacksExecute verified stop command; capture baseline state before recoveryDo NOT continuously resend commands, force-enable, clear error bits, or alter joint limits upon initial success response
Recording Failure / TimeoutVerify 6 required fields and integer operate parameter; check Service, exact process, recording/completed directoriesOn "Unknown Outcome", execute stop command first and query unique run IDDo NOT blindly retry start commands, spawn secondary instances, or clear root directories
Slow System Service Restart / Startup Error LogsCheck systemctl show/status, MainPID, Result, and journal logs for exact time window; re-verify version, Topic/Service, error bits, zero velocity, cameras, and recording servicesAllow a SINGLE authorized start recovery attempt; record exit timeouts and non-TTY commands for unit service rectificationDo NOT repeatedly restart services, force-terminate processes using fuzzy matching, or declare full stack failure/success based on single error logs
Camera Frame Rate FluctuationMeasure one channel at a time; check camera FPS status array and MCAP frame count/duration; compare camera_info vs. image topicsEvaluate reducing stream count/resolution or staggering processing workloadsDo NOT use multi-channel parallel topic hz commands for root cause assignment or apply global tuning blindly
Camera FPS Array Reads 0, but H.264 Streams ActiveAccess target .../h264 stream using BEST_EFFORT QoS to read headers; check encoding log frame counts, queue depth, frame drops, and latencyConfirm monitoring array mappings first; validate based on continuous decoding results of target streamDo NOT reboot cameras due to zero-value FPS array elements or declare the stream as working properly based on isolated frame timestamps
Aggregated Health Score 100, but Log Reports Component DisconnectionCross-verify current hardware variant, target endpoints, /diagnostics topic, joint error bits, and log entries within exact time windowDocument configuration if optional component is not applicable; escalate to support if required component is unreachableDo NOT rely on aggregated health scores as sole judgment criteria, clear errors blindly, or continuously ping unknown devices
Data Tracing Heartbeat Frequency Misaligned with DefinitionInspect TraceHeartbeat interface, node startup logs, current configuration, and observe over a window exceeding 35 secondsInterface comments and configurations are subject to unified corrections in later releases. Do not use this document until those updates have been applied.Do NOT treat /trace/heartbeat/status as an availability criterion or invoke write-configuration interfaces prior to fix
Recording Directory Exists but NVMe Not MountedCross-verify using findmnt -T /home/realman/ssd, df -h, block device origins, and directory disk usage levelsStop new recording sessions immediately; request administrator to restore correct mount pointDo NOT mistake empty mount point directories on root partition for SSD storage or allow data logging to fall back to system disk
Incomplete TF TreeCheck /tf, /tf_static, frame connectivity, and TRANSIENT_LOCAL durability policyReconstruct TF using URDF + joint angles for arms/head/waist, and extrinsic calibration parameters for camerasDo NOT forge static transforms to mask broken transform trees
HTTP Bridge 401 / 502 / Timeout401: Inspect token injection without printing plain text; 502/Timeout: Verify ROS 2 state and actual recording statusRestore credentials, SSH tunnels, or bridge services while maintaining loopback bindingDo NOT rebind service to 0.0.0.0 to bypass faults or blindly retry start commands
Insufficient Disk SpaceExecute df, run du on exact directories, and verify run ID, fetch, parsing, and checksum resultsPurge EXACT confirmed and authorized subdirectories ONLYStrictly PROHIBIT wildcards, prefix matching, root directory operations, or "delete-upon-fetch" execution