Natural language
for any robot.
ros2_lingua is a ROS 2 library that translates natural language instructions into validated, dependency-aware execution plans — dispatched over your existing ROS 2 actions and services.
# Send a natural language instruction to your robot
ros2 run ros2_lingua_mock cli "go to the table and pick up the bottle"
# The library figures out the rest:
✅ Plan generated — 3 step(s):
1. stabilize_robot ← auto-chained prerequisite
2. navigate_to_location params: {location_name: "table"}
3. pick_up_object params: {object_name: "bottle"}
🚀 Dispatching to robot...
Why ros2_lingua?
Hallucination Protection
Every LLM-suggested action is validated against registered capabilities before dispatch. Nothing runs on your robot that wasn't explicitly declared.
Auto Prerequisite Chaining
A backward-chaining planner resolves dependencies automatically. Say "navigate" — the system inserts "stabilize" first if needed.
Robot-Agnostic
Works on any ROS 2 robot — wheeled AMRs, robotic arms, humanoids, drones, AUVs. Your nodes describe what they can do. The library handles the rest.
ROS-Free Core
The schema, registry, and grounding engine have zero ROS 2 dependencies. 41 unit tests run in 0.2s — no nodes, no DDS, no networking required.
Quick Start
Get ros2_lingua running in under 10 minutes.
Clone and build
mkdir -p ~/ros2_lingua_ws/src && cd ~/ros2_lingua_ws/src
git clone https://github.com/purahan/ros2_lingua.git
cp -r ros2_lingua/ros2_lingua_core .
cp -r ros2_lingua/ros2_lingua .
cp -r ros2_lingua/ros2_lingua_mock .
cd ~/ros2_lingua_ws
source /opt/ros/humble/setup.bash
colcon build && source install/setup.bash
Install rosbridge + pull a model
sudo apt install ros-humble-rosbridge-suite
ollama pull llama3.1 # or llama3.2 for a smaller download
Launch the demo system
# Terminal 1 — rosbridge
ros2 launch rosbridge_server rosbridge_websocket_launch.xml
# Terminal 2 — full demo
ros2 launch ros2_lingua_mock demo.launch.py
Send an instruction
ros2 run ros2_lingua_mock cli "go to the table and pick up the bottle"
Open http://localhost:8080 to see the mission control dashboard with live capability visualization and plan animation.
Installation
Requirements and LLM backend setup.
Requirements
| Dependency | Version | Notes |
|---|---|---|
| ROS 2 | Humble | Ubuntu 22.04 recommended |
| Python | 3.10+ | Included with Humble |
| LLM backend | any | Ollama (local), OpenAI, or Anthropic |
| rosbridge_suite | any | Only needed for web dashboard |
LLM Backend Setup
# Option A — Ollama (local, no API key, recommended)
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1
# Option B — OpenAI
pip install openai --break-system-packages
# Option C — Anthropic Claude
pip install anthropic --break-system-packages
Capability Schema
A Capability is the fundamental unit of ros2_lingua — the structured description of one thing a node can do.
from ros2_lingua_core import Capability, CapabilityParameter, Tags
navigate = Capability(
# Identity — used in plans and LLM prompts
name="navigate_to_location",
description="Moves the robot to a named location. Known locations: table, door, dock.",
# ROS 2 interface — exactly one of these
ros_action="robot/navigate_to_pose", # or ros_service=
# Input parameters
parameters=[
CapabilityParameter(
name="location_name",
type="string",
description="Where to go, e.g. 'table', 'door', 'charging_dock'",
required=True,
),
],
# Preconditions: must be true before this can run
preconditions=["robot_is_balanced"],
# Postconditions: become true after this runs
postconditions=["robot_at_location"],
# Tags: for filtering and categorization
tags=[Tags.LOCOMOTION, Tags.NAVIGATION],
)
WRITE FOR THE LLM, NOT FOR HUMANS
The description and parameter description fields are injected directly into the LLM prompt. Be specific. Mention example values. Mention constraints. The more precise the description, the better the grounding quality.
Standard Tags
| Constant | Value | Use for |
|---|---|---|
| Tags.LOCOMOTION | "locomotion" | Navigation, driving, walking |
| Tags.MANIPULATION | "manipulation" | Arms, grippers, pick/place |
| Tags.BALANCE | "balance" | Stabilization, posture |
| Tags.SPEECH | "speech" | TTS, STT, voice I/O |
| Tags.PERCEPTION | "perception" | Cameras, lidar, detection |
| Tags.SAFETY | "safety" | E-stop, collision avoidance |
| Tags.INSPECTION | "inspection" | ROVs, drones, industrial |
Registry & State
The CapabilityRegistry is the single source of truth — it stores all registered capabilities and the robot's current symbolic state.
from ros2_lingua_core import CapabilityRegistry
registry = CapabilityRegistry()
# Registration
registry.register(navigate) # raises if already registered
registry.update(navigate) # register or overwrite
registry.unregister("navigate_to_location")
# Querying
registry.get("navigate_to_location")
registry.get_all()
registry.get_by_tag("locomotion")
registry.get_by_tags(["locomotion", "manipulation"], match="any")
registry.get_by_tags(["manipulation", "social"], match="all")
registry.get_all_tags() # sorted unique tag list
registry.get_untagged() # caps with no tags
# State management
registry.set_state("robot_is_balanced")
registry.clear_state("robot_is_balanced")
registry.get_state() # current Set[str]
registry.is_satisfied(["robot_is_balanced"])
Grounding Engine
Translates natural language into ActionPlan objects. Validates all LLM output against registered capabilities.
from ros2_lingua_core import GroundingEngine, OllamaBackend
engine = GroundingEngine(
registry=registry,
backend=OllamaBackend(model="llama3.1"),
auto_chain=True, # insert prerequisites automatically
max_retries=3, # retry on timeout/rate limit
retry_delay=1.0, # initial wait in seconds
retry_backoff=2.0, # multiply wait on each retry
)
# Ground an instruction
plan = engine.ground("go to the table and pick up the bottle")
# Ground with tag filter — only consider locomotion caps
plan = engine.ground("move to position A", tag_filter=["locomotion"])
# Check the result
if plan.feasible:
for step in plan.steps:
print(step.capability_name, step.parameters)
else:
print("Not feasible:", plan.reason)
ActionPlan fields
| Field | Type | Description |
|---|---|---|
| steps | List[ActionStep] | Ordered steps to execute |
| feasible | bool | False if instruction cannot be executed |
| reason | str | Why it's infeasible (if applicable) |
| original_instruction | str | The original input string |
Backward Chaining
The planner automatically resolves capability prerequisites without you hardcoding execution sequences. It works backward from the goal through the dependency graph.
# Given:
# stabilize_robot → produces: [robot_is_balanced]
# navigate → requires: [robot_is_balanced] produces: [robot_at_location]
# pick_up_object → requires: [robot_at_location, arm_is_free]
# Current state: {arm_is_free} ← robot_is_balanced NOT yet true
chain = registry.resolve_chain(
"pick_up_object",
current_state={"arm_is_free"}
)
# Result: [stabilize_robot, navigate, pick_up_object]
# stabilize was auto-inserted because navigate needed robot_is_balanced
DESIGN TOKEN NAMES CAREFULLY
Use specific, unambiguous state tokens. Prefer "robot_is_balanced" over "ready". Each token should have exactly one capability that produces it to avoid ambiguous dependency resolution.
Integrating With Your Robot
Adding ros2_lingua to an existing robot takes four steps. Each is independent — add them incrementally.
Declare capabilities
my_robot/capabilities.pyfrom ros2_lingua_core import Capability, CapabilityParameter, Tags
NAVIGATE = Capability(
name="navigate_to_location",
description="Drives the robot to a named location. Known: kitchen, office, dock.",
ros_action="my_robot/navigate",
parameters=[CapabilityParameter("location_name", "string", "Destination name")],
preconditions=["robot_is_ready"],
postconditions=["robot_at_location"],
tags=[Tags.LOCOMOTION],
)
Add LinguaMixin to your node
from rclpy.node import Node
from ros2_lingua import LinguaMixin
from my_robot.capabilities import NAVIGATE
class MyNavigationNode(LinguaMixin, Node):
def __init__(self):
Node.__init__(self, "my_navigation_node")
LinguaMixin.__init__(self)
# ... your existing __init__ code ...
self.register_lingua_capability(NAVIGATE) # ← this is all it takes
Subclass the dispatcher
from ros2_lingua.dispatcher_node import DispatcherNode
from my_robot_interfaces.action import NavigateTo
class MyRobotDispatcher(DispatcherNode):
def _call_action(self, action_name, cap_name, params):
if cap_name == "navigate_to_location":
client = ActionClient(self, NavigateTo, action_name)
goal = NavigateTo.Goal()
goal.location_name = params.get("location_name", "")
# ... send goal, wait for result ...
return True
return super()._call_action(action_name, cap_name, params)
Report state changes
async def _execute_navigate(self, goal_handle):
result = await self._do_navigation(goal_handle.request)
if result.success:
self.update_lingua_state(set_tokens=["robot_at_location"])
goal_handle.succeed()
else:
self.update_lingua_state(clear_tokens=["robot_at_location"])
goal_handle.abort()
LLM Backends
ros2_lingua is LLM-agnostic. Swap backends without changing any other code.
from ros2_lingua_core import OllamaBackend, OpenAIBackend, AnthropicBackend, RetryConfig
# Ollama — local, no API key
backend = OllamaBackend(model="llama3.1")
# OpenAI
backend = OpenAIBackend(api_key="sk-...", model="gpt-4o")
# Anthropic
backend = AnthropicBackend(api_key="sk-ant-...", model="claude-sonnet-4-20250514")
# Custom retry config
retry = RetryConfig(max_retries=5, base_delay_sec=2.0, backoff_factor=2.0)
backend = OllamaBackend(model="llama3.1", retry=retry)
# Custom backend — implement one method
class MyBackend:
def complete(self, messages: list[dict]) -> str:
return my_llm_call(messages) # return response string
| Backend | API Key | Best for |
|---|---|---|
| OllamaBackend | None required | Offline robots, privacy-sensitive |
| OpenAIBackend | OpenAI key | Best grounding quality |
| AnthropicBackend | Anthropic key | Complex multi-step instructions |
| MockBackend | None required | Testing, CI/CD |
Error Handling
11 specific exception types, all inheriting from LinguaError. Catch exactly what you care about.
from ros2_lingua_core import (
LinguaError, # catch all ros2_lingua errors
LLMBackendError, # any LLM connectivity problem
LLMTimeoutError, # LLM took too long
LLMRateLimitError, # API rate limit hit
LLMModelNotFoundError, # model doesn't exist
HallucinationError, # LLM referenced unknown capability
UnsatisfiablePreconditionError,
StepTimeoutError,
StepFailedError,
)
try:
plan = engine.ground("pick up the bottle")
except LLMModelNotFoundError as e:
print(f"Model not found: {e}. Try: ollama pull llama3.1")
except LLMBackendError as e:
print(f"LLM connectivity issue: {e}")
GROUNDING NEVER RAISES BY DEFAULT
GroundingEngine.ground() catches most errors internally and returns an infeasible ActionPlan with a descriptive reason string rather than raising. This keeps the grounding node stable under all conditions.
API Reference
Complete reference for the ros2_lingua public API, ROS 2 services, and topics.
Python API
| Name | Type | Package | Description |
|---|---|---|---|
| Capability | dataclass | core | Describes one robot capability |
| CapabilityParameter | dataclass | core | One input parameter for a capability |
| Tags | class | core | Standard tag string constants |
| CapabilityRegistry | class | core | Stores capabilities and robot state |
| GroundingEngine | class | core | Translates instructions to plans |
| ActionPlan | dataclass | core | Ordered execution plan |
| RetryConfig | dataclass | core | Controls LLM retry behaviour |
| LinguaMixin | mixin | ros2_lingua | Self-registration for ROS 2 nodes |
| DispatcherNode | Node | ros2_lingua | Executes plans — subclass for your robot |
ROS 2 Services
| Service | Type | Description |
|---|---|---|
| /lingua/register_capability | RegisterCapability | Register a capability from any node |
| /lingua/ground | GroundInstruction | Ground a natural language instruction |
| /lingua/update_state | UpdateState | Set or clear state tokens |
ROS 2 Topics
| Topic | Msg Type | Description |
|---|---|---|
| /lingua/current_plan | std_msgs/String | Latest ActionPlan as JSON |
| /lingua/capabilities | std_msgs/String | All registered capabilities as JSON (5Hz) |
| /lingua/execution_status | ExecutionStatus | Step-by-step execution progress |
Examples
Common usage patterns and complete working examples.
cd examples/humanoid_demo
PYTHONPATH=../../ros2_lingua_core python3 humanoid_demo.py
cd ros2_lingua_core
python3 -m pytest tests/test_core.py -v # 41 tests, 0.2s
cd ~/ros2_lingua_ws
colcon test --packages-select ros2_lingua
colcon test-result --verbose
ros2 launch ros2_lingua_mock demo.launch.py \
llm_backend:=openai \
llm_model:=gpt-4o \
llm_api_key:=sk-...
import sys; sys.path.insert(0, "path/to/ros2_lingua_core")
from ros2_lingua_core import Capability, CapabilityRegistry, GroundingEngine, OllamaBackend
registry = CapabilityRegistry()
registry.register(Capability(
name="navigate", description="Moves robot to named location",
ros_action="robot/navigate",
parameters=[CapabilityParameter("location", "string", "where")],
))
plan = GroundingEngine(registry, OllamaBackend("llama3.1")).ground("go to the kitchen")
print(plan.to_json())
Architecture
The two-layer architecture keeps intelligence separate from ROS 2 plumbing — enabling independent testing and clean separation of concerns.
ros2_lingua_core ← pure Python, zero ROS dependencies
├── schema.py Capability, CapabilityParameter, Tags
├── registry.py CapabilityRegistry + backward chain planner
├── grounding.py GroundingEngine, ActionPlan, ActionStep
├── backends.py OpenAI, Anthropic, Ollama, Mock, RetryConfig
└── errors.py LinguaError hierarchy (11 specific types)
ros2_lingua ← ROS 2 interface layer
├── grounding_node.py Service server wrapping GroundingEngine
├── dispatcher_node.py Executes ActionPlans — subclass for your robot
└── capability_mixin.py LinguaMixin for self-registering nodes
ros2_lingua_interfaces ← custom service/message definitions
├── srv/RegisterCapability.srv
├── srv/GroundInstruction.srv
└── msg/ExecutionStatus.msg
ros2_lingua_mock ← simulation for testing and demos
├── balance_node.py Simulates balance controller
├── navigation_node.py Simulates locomotion
├── manipulation_node.py Simulates arm/gripper
├── speech_node.py Simulates TTS
├── cli.py Command line instruction tool
├── dashboard_server.py HTTP server for web dashboard (port 8080)
└── launch/demo.launch.py Starts everything with one command
Data Flow
User instruction
│
▼
GroundingNode (/lingua/ground service)
│ calls registry.to_llm_context(tags=...)
│ calls LLM backend with capability list + instruction
│ parses + validates LLM JSON response
│ runs backward chain planner (auto_chain=True)
│
▼
ActionPlan (validated, ordered steps)
│
├──▶ service response (plan_json)
└──▶ /lingua/current_plan topic
│
▼
DispatcherNode
│ for each step:
│ look up capability in cache
│ call ros_action or ros_service
│ wait for result
│ update state via /lingua/update_state
│
▼
Your robot nodes