v0.1.0 · ROS 2 Humble

Natural language
for any robot.

ros2_lingua is a ROS 2 library that translates natural language instructions into validated, dependency-aware execution plans — dispatched over your existing ROS 2 actions and services.

ROS 2 Humble Python 3.10+ Apache 2.0 41 tests passing 16 integration tests
terminal
# Send a natural language instruction to your robot
ros2 run ros2_lingua_mock cli "go to the table and pick up the bottle"

# The library figures out the rest:
✅  Plan generated — 3 step(s):

   1.  stabilize_robot        ← auto-chained prerequisite
   2.  navigate_to_location   params: {location_name: "table"}
   3.  pick_up_object         params: {object_name: "bottle"}

🚀  Dispatching to robot...

Why ros2_lingua?

verified_user

Hallucination Protection

Every LLM-suggested action is validated against registered capabilities before dispatch. Nothing runs on your robot that wasn't explicitly declared.

account_tree

Auto Prerequisite Chaining

A backward-chaining planner resolves dependencies automatically. Say "navigate" — the system inserts "stabilize" first if needed.

device_hub

Robot-Agnostic

Works on any ROS 2 robot — wheeled AMRs, robotic arms, humanoids, drones, AUVs. Your nodes describe what they can do. The library handles the rest.

science

ROS-Free Core

The schema, registry, and grounding engine have zero ROS 2 dependencies. 41 unit tests run in 0.2s — no nodes, no DDS, no networking required.

Quick Start

Get ros2_lingua running in under 10 minutes.

1

Clone and build

mkdir -p ~/ros2_lingua_ws/src && cd ~/ros2_lingua_ws/src
git clone https://github.com/purahan/ros2_lingua.git
cp -r ros2_lingua/ros2_lingua_core .
cp -r ros2_lingua/ros2_lingua .
cp -r ros2_lingua/ros2_lingua_mock .

cd ~/ros2_lingua_ws
source /opt/ros/humble/setup.bash
colcon build && source install/setup.bash
2

Install rosbridge + pull a model

sudo apt install ros-humble-rosbridge-suite
ollama pull llama3.1    # or llama3.2 for a smaller download
3

Launch the demo system

# Terminal 1 — rosbridge
ros2 launch rosbridge_server rosbridge_websocket_launch.xml

# Terminal 2 — full demo
ros2 launch ros2_lingua_mock demo.launch.py
4

Send an instruction

ros2 run ros2_lingua_mock cli "go to the table and pick up the bottle"
tips_and_updates

Open http://localhost:8080 to see the mission control dashboard with live capability visualization and plan animation.

Installation

Requirements and LLM backend setup.

Requirements

Dependency Version Notes
ROS 2HumbleUbuntu 22.04 recommended
Python3.10+Included with Humble
LLM backendanyOllama (local), OpenAI, or Anthropic
rosbridge_suiteanyOnly needed for web dashboard

LLM Backend Setup

bash
# Option A — Ollama (local, no API key, recommended)
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1

# Option B — OpenAI
pip install openai --break-system-packages

# Option C — Anthropic Claude
pip install anthropic --break-system-packages

Capability Schema

A Capability is the fundamental unit of ros2_lingua — the structured description of one thing a node can do.

python
from ros2_lingua_core import Capability, CapabilityParameter, Tags

navigate = Capability(
    # Identity — used in plans and LLM prompts
    name="navigate_to_location",
    description="Moves the robot to a named location. Known locations: table, door, dock.",

    # ROS 2 interface — exactly one of these
    ros_action="robot/navigate_to_pose",  # or ros_service=

    # Input parameters
    parameters=[
        CapabilityParameter(
            name="location_name",
            type="string",
            description="Where to go, e.g. 'table', 'door', 'charging_dock'",
            required=True,
        ),
    ],

    # Preconditions: must be true before this can run
    preconditions=["robot_is_balanced"],

    # Postconditions: become true after this runs
    postconditions=["robot_at_location"],

    # Tags: for filtering and categorization
    tags=[Tags.LOCOMOTION, Tags.NAVIGATION],
)
tips_and_updates

WRITE FOR THE LLM, NOT FOR HUMANS

The description and parameter description fields are injected directly into the LLM prompt. Be specific. Mention example values. Mention constraints. The more precise the description, the better the grounding quality.

Standard Tags

Constant Value Use for
Tags.LOCOMOTION"locomotion"Navigation, driving, walking
Tags.MANIPULATION"manipulation"Arms, grippers, pick/place
Tags.BALANCE"balance"Stabilization, posture
Tags.SPEECH"speech"TTS, STT, voice I/O
Tags.PERCEPTION"perception"Cameras, lidar, detection
Tags.SAFETY"safety"E-stop, collision avoidance
Tags.INSPECTION"inspection"ROVs, drones, industrial

Registry & State

The CapabilityRegistry is the single source of truth — it stores all registered capabilities and the robot's current symbolic state.

python
from ros2_lingua_core import CapabilityRegistry

registry = CapabilityRegistry()

# Registration
registry.register(navigate)       # raises if already registered
registry.update(navigate)         # register or overwrite
registry.unregister("navigate_to_location")

# Querying
registry.get("navigate_to_location")
registry.get_all()
registry.get_by_tag("locomotion")
registry.get_by_tags(["locomotion", "manipulation"], match="any")
registry.get_by_tags(["manipulation", "social"], match="all")
registry.get_all_tags()           # sorted unique tag list
registry.get_untagged()           # caps with no tags

# State management
registry.set_state("robot_is_balanced")
registry.clear_state("robot_is_balanced")
registry.get_state()              # current Set[str]
registry.is_satisfied(["robot_is_balanced"])

Grounding Engine

Translates natural language into ActionPlan objects. Validates all LLM output against registered capabilities.

python
from ros2_lingua_core import GroundingEngine, OllamaBackend

engine = GroundingEngine(
    registry=registry,
    backend=OllamaBackend(model="llama3.1"),
    auto_chain=True,   # insert prerequisites automatically
    max_retries=3,      # retry on timeout/rate limit
    retry_delay=1.0,    # initial wait in seconds
    retry_backoff=2.0,  # multiply wait on each retry
)

# Ground an instruction
plan = engine.ground("go to the table and pick up the bottle")

# Ground with tag filter — only consider locomotion caps
plan = engine.ground("move to position A", tag_filter=["locomotion"])

# Check the result
if plan.feasible:
    for step in plan.steps:
        print(step.capability_name, step.parameters)
else:
    print("Not feasible:", plan.reason)

ActionPlan fields

Field Type Description
stepsList[ActionStep]Ordered steps to execute
feasibleboolFalse if instruction cannot be executed
reasonstrWhy it's infeasible (if applicable)
original_instructionstrThe original input string

Backward Chaining

The planner automatically resolves capability prerequisites without you hardcoding execution sequences. It works backward from the goal through the dependency graph.

python
# Given:
# stabilize_robot  → produces: [robot_is_balanced]
# navigate         → requires: [robot_is_balanced]  produces: [robot_at_location]
# pick_up_object   → requires: [robot_at_location, arm_is_free]

# Current state: {arm_is_free}  ← robot_is_balanced NOT yet true

chain = registry.resolve_chain(
    "pick_up_object",
    current_state={"arm_is_free"}
)

# Result: [stabilize_robot, navigate, pick_up_object]
# stabilize was auto-inserted because navigate needed robot_is_balanced
warning

DESIGN TOKEN NAMES CAREFULLY

Use specific, unambiguous state tokens. Prefer "robot_is_balanced" over "ready". Each token should have exactly one capability that produces it to avoid ambiguous dependency resolution.

Integrating With Your Robot

Adding ros2_lingua to an existing robot takes four steps. Each is independent — add them incrementally.

1

Declare capabilities

my_robot/capabilities.py
from ros2_lingua_core import Capability, CapabilityParameter, Tags

NAVIGATE = Capability(
    name="navigate_to_location",
    description="Drives the robot to a named location. Known: kitchen, office, dock.",
    ros_action="my_robot/navigate",
    parameters=[CapabilityParameter("location_name", "string", "Destination name")],
    preconditions=["robot_is_ready"],
    postconditions=["robot_at_location"],
    tags=[Tags.LOCOMOTION],
)
2

Add LinguaMixin to your node

from rclpy.node import Node
from ros2_lingua import LinguaMixin
from my_robot.capabilities import NAVIGATE

class MyNavigationNode(LinguaMixin, Node):
    def __init__(self):
        Node.__init__(self, "my_navigation_node")
        LinguaMixin.__init__(self)
        # ... your existing __init__ code ...
        self.register_lingua_capability(NAVIGATE)  # ← this is all it takes
3

Subclass the dispatcher

from ros2_lingua.dispatcher_node import DispatcherNode
from my_robot_interfaces.action import NavigateTo

class MyRobotDispatcher(DispatcherNode):
    def _call_action(self, action_name, cap_name, params):
        if cap_name == "navigate_to_location":
            client = ActionClient(self, NavigateTo, action_name)
            goal = NavigateTo.Goal()
            goal.location_name = params.get("location_name", "")
            # ... send goal, wait for result ...
            return True
        return super()._call_action(action_name, cap_name, params)
4

Report state changes

async def _execute_navigate(self, goal_handle):
    result = await self._do_navigation(goal_handle.request)
    if result.success:
        self.update_lingua_state(set_tokens=["robot_at_location"])
        goal_handle.succeed()
    else:
        self.update_lingua_state(clear_tokens=["robot_at_location"])
        goal_handle.abort()

LLM Backends

ros2_lingua is LLM-agnostic. Swap backends without changing any other code.

python
from ros2_lingua_core import OllamaBackend, OpenAIBackend, AnthropicBackend, RetryConfig

# Ollama — local, no API key
backend = OllamaBackend(model="llama3.1")

# OpenAI
backend = OpenAIBackend(api_key="sk-...", model="gpt-4o")

# Anthropic
backend = AnthropicBackend(api_key="sk-ant-...", model="claude-sonnet-4-20250514")

# Custom retry config
retry = RetryConfig(max_retries=5, base_delay_sec=2.0, backoff_factor=2.0)
backend = OllamaBackend(model="llama3.1", retry=retry)

# Custom backend — implement one method
class MyBackend:
    def complete(self, messages: list[dict]) -> str:
        return my_llm_call(messages)  # return response string
Backend API Key Best for
OllamaBackendNone requiredOffline robots, privacy-sensitive
OpenAIBackendOpenAI keyBest grounding quality
AnthropicBackendAnthropic keyComplex multi-step instructions
MockBackendNone requiredTesting, CI/CD

Error Handling

11 specific exception types, all inheriting from LinguaError. Catch exactly what you care about.

python
from ros2_lingua_core import (
    LinguaError,            # catch all ros2_lingua errors
    LLMBackendError,        # any LLM connectivity problem
    LLMTimeoutError,        # LLM took too long
    LLMRateLimitError,      # API rate limit hit
    LLMModelNotFoundError,  # model doesn't exist
    HallucinationError,     # LLM referenced unknown capability
    UnsatisfiablePreconditionError,
    StepTimeoutError,
    StepFailedError,
)

try:
    plan = engine.ground("pick up the bottle")
except LLMModelNotFoundError as e:
    print(f"Model not found: {e}. Try: ollama pull llama3.1")
except LLMBackendError as e:
    print(f"LLM connectivity issue: {e}")
info

GROUNDING NEVER RAISES BY DEFAULT

GroundingEngine.ground() catches most errors internally and returns an infeasible ActionPlan with a descriptive reason string rather than raising. This keeps the grounding node stable under all conditions.

API Reference

Complete reference for the ros2_lingua public API, ROS 2 services, and topics.

Python API

Name Type Package Description
CapabilitydataclasscoreDescribes one robot capability
CapabilityParameterdataclasscoreOne input parameter for a capability
TagsclasscoreStandard tag string constants
CapabilityRegistryclasscoreStores capabilities and robot state
GroundingEngineclasscoreTranslates instructions to plans
ActionPlandataclasscoreOrdered execution plan
RetryConfigdataclasscoreControls LLM retry behaviour
LinguaMixinmixinros2_linguaSelf-registration for ROS 2 nodes
DispatcherNodeNoderos2_linguaExecutes plans — subclass for your robot

ROS 2 Services

Service Type Description
/lingua/register_capabilityRegisterCapabilityRegister a capability from any node
/lingua/groundGroundInstructionGround a natural language instruction
/lingua/update_stateUpdateStateSet or clear state tokens

ROS 2 Topics

Topic Msg Type Description
/lingua/current_planstd_msgs/StringLatest ActionPlan as JSON
/lingua/capabilitiesstd_msgs/StringAll registered capabilities as JSON (5Hz)
/lingua/execution_statusExecutionStatusStep-by-step execution progress

Examples

Common usage patterns and complete working examples.

Run demo without a robot
cd examples/humanoid_demo
PYTHONPATH=../../ros2_lingua_core python3 humanoid_demo.py
Run all unit tests (no ROS required)
cd ros2_lingua_core
python3 -m pytest tests/test_core.py -v   # 41 tests, 0.2s
Run integration tests (Ollama must be running)
cd ~/ros2_lingua_ws
colcon test --packages-select ros2_lingua
colcon test-result --verbose
Launch with OpenAI
ros2 launch ros2_lingua_mock demo.launch.py \
    llm_backend:=openai \
    llm_model:=gpt-4o \
    llm_api_key:=sk-...
Use grounding engine standalone (no ROS)
import sys; sys.path.insert(0, "path/to/ros2_lingua_core")
from ros2_lingua_core import Capability, CapabilityRegistry, GroundingEngine, OllamaBackend

registry = CapabilityRegistry()
registry.register(Capability(
    name="navigate", description="Moves robot to named location",
    ros_action="robot/navigate",
    parameters=[CapabilityParameter("location", "string", "where")],
))
plan = GroundingEngine(registry, OllamaBackend("llama3.1")).ground("go to the kitchen")
print(plan.to_json())

Architecture

The two-layer architecture keeps intelligence separate from ROS 2 plumbing — enabling independent testing and clean separation of concerns.

package structure
ros2_lingua_core          ← pure Python, zero ROS dependencies
├── schema.py             Capability, CapabilityParameter, Tags
├── registry.py           CapabilityRegistry + backward chain planner
├── grounding.py          GroundingEngine, ActionPlan, ActionStep
├── backends.py           OpenAI, Anthropic, Ollama, Mock, RetryConfig
└── errors.py             LinguaError hierarchy (11 specific types)

ros2_lingua               ← ROS 2 interface layer
├── grounding_node.py     Service server wrapping GroundingEngine
├── dispatcher_node.py    Executes ActionPlans — subclass for your robot
└── capability_mixin.py   LinguaMixin for self-registering nodes

ros2_lingua_interfaces    ← custom service/message definitions
├── srv/RegisterCapability.srv
├── srv/GroundInstruction.srv
└── msg/ExecutionStatus.msg

ros2_lingua_mock          ← simulation for testing and demos
├── balance_node.py       Simulates balance controller
├── navigation_node.py    Simulates locomotion
├── manipulation_node.py  Simulates arm/gripper
├── speech_node.py        Simulates TTS
├── cli.py                Command line instruction tool
├── dashboard_server.py   HTTP server for web dashboard (port 8080)
└── launch/demo.launch.py  Starts everything with one command

Data Flow

User instruction
      │
      ▼
GroundingNode  (/lingua/ground service)
      │  calls registry.to_llm_context(tags=...)
      │  calls LLM backend with capability list + instruction
      │  parses + validates LLM JSON response
      │  runs backward chain planner (auto_chain=True)
      │
      ▼
ActionPlan  (validated, ordered steps)
      │
      ├──▶  service response (plan_json)
      └──▶  /lingua/current_plan topic
                │
                ▼
          DispatcherNode
                │  for each step:
                │  look up capability in cache
                │  call ros_action or ros_service
                │  wait for result
                │  update state via /lingua/update_state
                │
                ▼
          Your robot nodes