Learning

Ockam Routing

Nazmul Idris

Published 2023-07-31

Learning

Ockam is a suite of programming libraries, command line tools, and managed cloud services to orchestrate end-to-end encryption, mutual authentication, key management, credential management, and authorization policy enforcement — all at massive scale. Ockam's end-to-end secure channels guarantee authenticity, integrity, and confidentiality of all data-in-motion at the application layer.

One of the key features that makes this possible is Ockam Routing. Routing allows us to create secure channels over multi-hop, multi-protocol routes which can span various network topologies (servers behind NAT firewalls with no external ports open, etc) and transport protocols (TCP, UDP, WebSockets, BLE, etc).

In this blog post we will explore the Ockam Rust Library and see how routing works in Ockam. We will work with Rust code and look at some code examples that demonstrate the simple case, and more advanced use cases.

Mitigating risk

Before we get started, let's quickly discuss the pitfalls of using existing approaches to securing communications within applications. Security is not something that most of us think about when we are building systems and are focused on getting things working and shipping.

Traditional secure communication implementations are typically tightly coupled with transport protocols in a way that all their security is limited to the length and duration of one underlying transport connection.

For example, most TLS implementations are tightly coupled with the underlying TCP connection. If your application's data and requests travel over two TCP connection hops (TCP → TCP), then all TLS guarantees break at the bridge between the two networks. This bridge, gateway, or load balancer then becomes a point of weakness for application data.
Traditional secure communication protocols are also unable to protect your application's data if it travels over multiple different transport protocols. They can't guarantee data authenticity or data integrity if your application's communication path is UDP → TCP or BLE → TCP.

In other words using traditional secure communication implementations you may be opening the doors to losing trust in the data that your apps are working on. Here are some aspects of your apps that may be at risk:

Lack of trust in the data your app receives.
- Who sent it to my app?
- Is it actually the data they sent my app?
- Missing authentication, data integrity.
Lack of trust in the data your app sends.
- Who am I sending the data to?
- Would someone else, other than them, be able to see it?

Our journey

In this blog post we will create two examples of Ockam nodes communicating with each other using Ockam Routing and Ockam Transports. We will use the Rust library to create these Ockam nodes and setup routing. Ockam Routing and transports enable other Ockam protocols to provide end-to-end guarantees like trust, security, privacy, reliable delivery, and ordering at the application layer.

Ockam Routing: is a simple and lightweight message-based protocol that makes it possible to bidirectionally exchange messages over a large variety of communication topologies: TCP -> TCP or TCP -> TCP -> TCP or BLE -> UDP -> TCP or BLE -> TCP -> TCP or TCP -> Kafka -> TCP or any other topology you can imagine.
Ockam Transports: adapt Ockam Routing to various transport protocols.

An Ockam node is any running application that can communicate with other applications using various Ockam protocols like Routing, Relays, and Portals, Secure Channels, etc.

An Ockam node can be defined as any independent process which provides an API supporting the Ockam Routing protocol. We can create Ockam nodes using the Ockam command line interface (CLI) (ockam command) or using various Ockam programming libraries like our Rust and Elixir libraries. We will be using the Rust library in this blog post.

Let's dive in

To get started please follow this guide to get the Rust toolchain setup on your machine along with an empty project.

The empty project is named hello_ockam.
This will be the starting point for all our examples in this blog post.

Simple example

For our first example, we will create a simple Ockam node that will send a message over some hops (in the same node) to a worker (in the same node) that just echoes the message back. There are no TCP transports involved and all the messages are being passed back and forth inside the same node. This will give us a feel for building workers and routing at a basic level.

When a worker is started on a node, it is given one or more addresses. The node maintains a mailbox for each address and whenever a message arrives for a specific address it delivers that message to the corresponding registered worker.

For more information on creating nodes and workers using the Rust library, please refer to this guide.

We will need to create a Rust source file with a main() program, and two other Rust source files with two workers: Hopper and Echoer. We can then send a string message and see if we can get it echoed back.

Before we begin let's consider routing. When we send a message inside of a node it carries with it 2 metadata fields, onward_route and return_route, where a route is simply a list of addresses. Each worker gets an address in a node.

So, if we wanted to send a message from the app address to the echoer address, with 3 hops in the middle, we can build a route like the following.

┌───────────────────────┐
│  Node 1               │
├───────────────────────┤
│  ┌────────────────┐   │
│  │ Address:       │   │
│  │ 'app'          │   │
│  └─┬────────────▲─┘   │
│  ┌─▼────────────┴─┐   │
│  │ Address:       │   │
│  │ 'hopper1..3'   │x3 │
│  └─┬────────────▲─┘   │
│  ┌─▼────────────┴─┐   │
│  │ Address:       │   │
│  │ 'echoer'       │   │
│  └────────────────┘   │
└───────────────────────┘

Here's the Rust code to build this route.

/// Send a message to the echoer worker via the "hopper1", "hopper2", and "hopper3" workers.
let route = route!["hopper1", "hopper2", "hopper3", "echoer"];

Let's add some source code to make this happen next. The first thing we will do is add one more dependency to this empty hello_ockam project. The colored crate will give us colorized console output which will make the output from our examples so much easier to read and understand.

cargo add colored

Then we add the echoer worker (in our hello_ockam project) by creating a new /src/echoer.rs file and copy / pasting the following code in it.

use colored::Colorize;
use ockam::{Context, Result, Routed, Worker};

pub struct Echoer;

/// When a worker is started on a node, it is given one or more addresses. The node
/// maintains a mailbox for each address and whenever a message arrives for a specific
/// address it delivers that message to the corresponding registered worker.
///
/// Workers can handle messages from other workers running on the same or a different
/// node. In response to a message, an worker can: make local decisions, change its
/// internal state, create more workers, or send more messages to other workers running on
/// the same or a different node.
#[ockam::worker]
impl Worker for Echoer {
    type Context = Context;
    type Message = String;

    async fn handle_message(&mut self, ctx: &mut Context, msg: Routed<String>) -> Result<()> {
        // Echo the message body back on its return_route.
        let addr_str = ctx.address().to_string();
        let msg_str = msg.as_body().to_string();
        let new_msg_str = format!("👈 echo back: {}", msg);

        // Formatting stdout output.
        let lines = [
            format!("📣 'echoer' worker → Address: {}", addr_str.bright_yellow()),
            format!("    Received: '{}'", msg_str.green()),
            format!("    Sent: '{}'", new_msg_str.cyan()),
        ];
        lines
            .iter()
            .for_each(|line| println!("{}", line.white().on_black()));

        ctx.send(msg.return_route(), new_msg_str).await
    }
}

Next we add the hopper worker (in our hello_ockam project) by creating a new /src/hopper.rs file and copy / pasting the following code in it.

Note how this worker manipulates the onward_route & return_route fields of the message to send it to the next hop. We will actually see this in the console output when we run this code soon.

use colored::Colorize;
use ockam::{Any, Context, Result, Routed, Worker};

pub struct Hopper;

#[ockam::worker]
impl Worker for Hopper {
    type Context = Context;
    type Message = Any;

    /// This handle function takes any incoming message and forwards. it to the next hop
    /// in it's onward route.
    async fn handle_message(&mut self, ctx: &mut Context, msg: Routed<Any>) -> Result<()> {
        // Cast the msg to a Routed<String>
        let msg: Routed<String> = msg.cast()?;

        let msg_str = msg.to_string().white().on_bright_black();
        let addr_str = ctx.address().to_string().white().on_bright_black();

        // Some type conversion.
        let mut message = msg.into_local_message();
        let transport_message = message.transport_mut();

        // Remove my address from the onward_route.
        let removed_address = transport_message.onward_route.step()?;
        let removed_addr_str = removed_address
            .to_string()
            .white()
            .on_bright_black()
            .strikethrough();

        // Formatting stdout output.
        let lines = [
            format!("🐇 'hopper' worker → Addr: '{}'", addr_str),
            format!("    Received: '{}'", msg_str),
            format!("    onward_route -> remove: '{}'", removed_addr_str),
            format!("    return_route -> prepend: '{}'", addr_str),
        ];
        lines
            .iter()
            .for_each(|line| println!("{}", line.black().on_yellow()));

        // Insert my address at the beginning return_route.
        transport_message
            .return_route
            .modify()
            .prepend(ctx.address());

        // Send the message on its onward_route.
        ctx.forward(message).await
    }
}

And finally let's add a main() to our hello_ockam project. This will be the entry point for our example.

When a new node starts and calls an async main function, it turns that function into a worker with an address of app. This makes it easy to send and receive messages from the main function (i.e the app worker).

Create an empty file /examples/03-routing-many.hops.rs (note this is in the examples/ folder and not src/ folder like the workers above).

use colored::Colorize;
use hello_ockam::{Echoer, Hopper};
use ockam::{node, route, Context, Result};

#[rustfmt::skip]
const HELP_TEXT: &str =r#"
┌───────────────────────┐
│  Node 1               │
├───────────────────────┤
│  ┌────────────────┐   │
│  │ Address:       │   │
│  │ 'app'          │   │
│  └─┬────────────▲─┘   │
│  ┌─▼────────────┴─┐   │
│  │ Address:       │   │
│  │ 'hopper1..3'   │x3 │
│  └─┬────────────▲─┘   │
│  ┌─▼────────────┴─┐   │
│  │ Address:       │   │
│  │ 'echoer'       │   │
│  └────────────────┘   │
└───────────────────────┘
"#;

/// This node routes a message through many hops.
#[ockam::node]
async fn main(ctx: Context) -> Result<()> {
    println!("{}", HELP_TEXT.green());

    print_title(vec![
        "Run a node w/ 'app', 'echoer' and 'hopper1', 'hopper2', 'hopper3' workers",
        "then send a message over 3 hops",
        "finally stop the node",
    ]);

    // Create a node with default implementations.
    let mut node = node(ctx);

    // Start an Echoer worker at address "echoer".
    node.start_worker("echoer", Echoer).await?;

    // Start 3 hop workers at addresses "hopper1", "hopper2" and "hopper3".
    node.start_worker("hopper1", Hopper).await?;
    node.start_worker("hopper2", Hopper).await?;
    node.start_worker("hopper3", Hopper).await?;

    // Send a message to the echoer worker via the "hopper1", "hopper2", and "hopper3" workers.
    let route = route!["hopper1", "hopper2", "hopper3", "echoer"];
    let route_msg = format!("{:?}", route);
    let msg = "Hello Ockam!";
    node.send(route, msg.to_string()).await?;

    // Wait to receive a reply and print it.
    let reply = node.receive::<String>().await?;

    // Formatting stdout output.
    let lines = [
        "🏃 Node 1 →".to_string(),
        format!("    sending: {}", msg.green()),
        format!("    over route: {}", route_msg.blue()),
        format!("    and receiving: '{}'", reply.purple()), // Should print "👈 echo back:  Hello Ockam!"
        format!("    then {}", "stopping".bold().red()),
    ];
    lines
        .iter()
        .for_each(|line| println!("{}", line.black().on_white()));

    // Stop all workers, stop the node, cleanup and return.
    node.stop().await
}

fn print_title(title: Vec<&str>) {
    let line = format!("🚀 {}", title.join("\n  → ").white());
    println!("{}", line.black().on_bright_black())
}

Now it is time to run our program to see what it does! 🎉

In your terminal app, run the following command. Note that OCKAM_LOG=none is used to disable logging output from the Ockam library. This is done to make the output of the example easier to read.

OCKAM_LOG=none cargo run --example 03-routing-many-hops

And you should see something like the following. Our example program creates multiple hop workers (three hopper workers) between the app and the echoer and route our message through them 🚀.

Complex example

This example continues from the simple example above, we are going to reuse all the dependencies and workers in this example so please make sure to complete the simple example before working on this one.

In this example, we will introduce TCP transports in between the hops. Instead of passing messages around between workers in the same node, we will spawn multiple nodes. Then we will have a few TCP transports (TCP socket client and listener combos) that will connect the nodes.

An Ockam transport is a plugin for Ockam Routing. It moves Ockam Routing messages using a specific transport protocol like TCP, UDP, WebSockets, Bluetooth, etc.

We will have three nodes:

node_initiator: The first node initiates sending the message over TCP to the middle node (port 3000).
node_middle: Then middle node simply forwards this message on to the last node over TCP again (port 4000 this time).
node_responder: And finally the responder node receives the message and sends a reply back to the initiator node.

The following diagram depicts what we will build next. In this example all these nodes are on the same machine, but they can easy just be nodes on different machines.

┌──────────────────────┐
│node_initiator        │
├──────────────────────┤
│ ┌──────────────────┐ │
│ │Address:          │ │     ┌───────────────────────────┐
│ │'app'             │ │     │node_middle                │
│ └──┬────────────▲──┘ │     ├───────────────────────────┤
│ ┌──▼────────────┴──┐ │     │ ┌──────────────────┐      │
│ │TCP transport     └─┼─────┼─►TCP transport     │      │
│ │connect to 3000   ◄─┼─────┼─┐listening on 3000 │      │
│ └──────────────────┘ │     │ └──┬────────────▲──┘      │
└──────────────────────┘     │ ┌──▼────────────┴───────┐ │
                             │ │Address:               │ │   ┌──────────────────────┐
                             │ │'forward_to_responder' │ │   │node_responder        │
                             │ └──┬────────────▲───────┘ │   ├──────────────────────┤
                             │ ┌──▼────────────┴──┐      │   │ ┌──────────────────┐ │
                             │ │TCP transport     └──────┼───┼─►TCP transport     │ │
                             │ │connect to 4000   ◄──────┼───┼─┐listening on 4000 │ │
                             │ └──────────────────┘      │   │ └──┬────────────▲──┘ │
                             └───────────────────────────┘   │ ┌──▼────────────┴──┐ │
                                                             │ │Address:          │ │
                                                             │ │'echoer'          │ │
                                                             │ └──────────────────┘ │
                                                             └──────────────────────┘

Let's start by creating a new file /examples/04-routing-over-two-transport-hops.rs (in the /examples/ folder and not /src/ folder). Then copy / paste the following code in that file.

use colored::Colorize;
use hello_ockam::{Echoer, Forwarder};
use ockam::{
    node, route, AsyncTryClone, Context, Result, TcpConnectionOptions, TcpListenerOptions,
    TcpTransportExtension,
};

#[rustfmt::skip]
const HELP_TEXT: &str =r#"
┌──────────────────────┐
│node_initiator        │
├──────────────────────┤
│ ┌──────────────────┐ │
│ │Address:          │ │     ┌───────────────────────────┐
│ │'app'             │ │     │node_middle                │
│ └──┬────────────▲──┘ │     ├───────────────────────────┤
│ ┌──▼────────────┴──┐ │     │ ┌──────────────────┐      │
│ │TCP transport     └─┼─────┼─►TCP transport     │      │
│ │connect to 3000   ◄─┼─────┼─┐listening on 3000 │      │
│ └──────────────────┘ │     │ └──┬────────────▲──┘      │
└──────────────────────┘     │ ┌──▼────────────┴───────┐ │
                             │ │Address:               │ │   ┌──────────────────────┐
                             │ │'forward_to_responder' │ │   │node_responder        │
                             │ └──┬────────────▲───────┘ │   ├──────────────────────┤
                             │ ┌──▼────────────┴──┐      │   │ ┌──────────────────┐ │
                             │ │TCP transport     └──────┼───┼─►TCP transport     │ │
                             │ │connect to 4000   ◄──────┼───┼─┐listening on 4000 │ │
                             │ └──────────────────┘      │   │ └──┬────────────▲──┘ │
                             └───────────────────────────┘   │ ┌──▼────────────┴──┐ │
                                                             │ │Address:          │ │
                                                             │ │'echoer'          │ │
                                                             │ └──────────────────┘ │
                                                             └──────────────────────┘
"#;

#[ockam::node]
async fn main(ctx: Context) -> Result<()> {
    println!("{}", HELP_TEXT.green());

    let ctx_clone = ctx.async_try_clone().await?;
    let ctx_clone_2 = ctx.async_try_clone().await?;

    let mut node_responder = create_responder_node(ctx).await.unwrap();

    let mut node_middle = create_middle_node(ctx_clone).await.unwrap();

    create_initiator_node(ctx_clone_2).await.unwrap();

    node_responder.stop().await.ok();
    node_middle.stop().await.ok();

    println!(
        "{}",
        "App finished, stopping node_responder & node_middle".red()
    );

    Ok(())
}

fn print_title(title: Vec<&str>) {
    let line = format!("🚀 {}", title.join("\n  → ").white());
    println!("{}", line.black().on_bright_black())
}

This code won't actually compile, since there are 3 functions missing from this source file. We are just adding this file first in order to stage the rest of the code we will write next.

This main() function creates the three nodes like we see in the diagram above, and it also stops them after the example is done running.

Initiator node

So let's write the function that creates the initiator node first. Copy the following into the source file we created earlier (/examples/04-routing-over-two-transport-hops.rs), and paste it below the existing code there:

/// This node routes a message, to a worker on a different node, over two TCP transport
/// hops.
async fn create_initiator_node(ctx: Context) -> Result<()> {
    print_title(vec![
        "Create node_initiator that routes a message, over 2 TCP transport hops, to 'echoer' worker on node_responder",
        "stop",
    ]);

    // Create a node with default implementations.
    let mut node = node(ctx);

    // Initialize the TCP transport.
    let tcp_transport = node.create_tcp_transport().await?;

    // Create a TCP connection to the middle node.
    let connection_to_middle_node = tcp_transport
        .connect("localhost:3000", TcpConnectionOptions::new())
        .await?;

    // Send a message to the "echoer" worker, on a different node, over two TCP hops. Wait
    // to receive a reply and print it.
    let route = route![connection_to_middle_node, "forward_to_responder", "echoer"];
    let route_str = format!("{:?}", route);
    let msg = "Hello Ockam!";
    let reply = node
        .send_and_receive::<String>(route, msg.to_string())
        .await?;

    // Formatting stdout output.
    let lines = [
        "🏃 node_initiator →".to_string(),
        format!("    sending: {}", msg.green()),
        format!("    over route: '{}'", route_str.blue()),
        format!("    and received: '{}'", reply.purple()), // Should print "👈 echo back:  Hello Ockam!"
        format!("    then {}", "stopping".bold().red()),
    ];
    lines
        .iter()
        .for_each(|line| println!("{}", line.black().on_white()));

    // Stop all workers, stop the node, cleanup and return.
    node.stop().await
}

This (initiator) node will send a message to the responder using the following route.

let route = route![connection_to_middle_node, "forward_to_responder", "echoer"];

Note the use of a mix of TCP transport routes as well as addresses for other workers. Also note that this node does not have to be aware of the full topology of the network of nodes. It just knows that it has to jump over the TCP transport connection_to_middle_node and then have its message routed to forward_to_responder address followed by echoer address.

Middle node

Let's create the middle node next, which will run the worker Forwarder on this address: forward_to_responder.

Copy and paste the following into the source file we created above (/examples/04-routing-over-two-transport-hops.rs).

This middle node simply forwards whatever comes into its TCP listener (on 3000) to port 4000.
This node has a Forwarder worker on address forward_to_responder, so that's how the initiator can reach this address specified in its route at the start of this example.

/// - Starts a TCP listener at 127.0.0.1:3000.
/// - This node creates a TCP connection to a node at 127.0.0.1:4000.
/// - Starts a forwarder worker to forward messages to 127.0.0.1:4000.
/// - Then runs forever waiting to route messages.
async fn create_middle_node(ctx: Context) -> Result<ockam::Node> {
    print_title(vec![
        "Create node_middle that listens on 3000 and forwards to 4000",
        "wait for messages until stopped",
    ]);

    // Create a node with default implementations.
    let node = node(ctx);

    // Initialize the TCP transport.
    let tcp_transport = node.create_tcp_transport().await?;

    // Create a TCP connection to the responder node.
    let connection_to_responder = tcp_transport
        .connect("127.0.0.1:4000", TcpConnectionOptions::new())
        .await?;

    // Create a Forwarder worker.
    node.start_worker(
        "forward_to_responder",
        Forwarder {
            address: connection_to_responder.into(),
        },
    )
    .await?;

    // Create a TCP listener and wait for incoming connections.
    let listener = tcp_transport
        .listen("127.0.0.1:3000", TcpListenerOptions::new())
        .await?;

    // Allow access to the Forwarder via TCP connections from the TCP listener.
    node.flow_controls()
        .add_consumer("forward_to_responder", listener.flow_control_id());

    // Don't call node.stop() here so this node runs forever.
    Ok(node)
}

We will also need the Forwarder worker that we use in the middle node. Copy and paste the following into a new source file /src/forwarder.rs. This file goes in the src folder and not the examples folder.

use colored::Colorize;
use ockam::{Address, Any, Context, LocalMessage, Result, Routed, Worker};

pub struct Forwarder {
    pub address: Address,
}

#[ockam::worker]
impl Worker for Forwarder {
    type Context = Context;
    type Message = Any;

    /// This handle function takes any incoming message and forwards it to the next hop in
    /// it's onward route.
    async fn handle_message(&mut self, ctx: &mut Context, msg: Routed<Any>) -> Result<()> {
        let address_str = ctx.address().to_string();
        let msg_str = msg.to_string();

        // Formatting stdout output.
        let lines = [
            format!("👉 'forwarder' worker → Address: {}", address_str.blue()),
            format!("    message: {}", msg_str.purple()),
        ];
        lines
            .iter()
            .for_each(|line| println!("{}", line.black().on_cyan()));

        // Do some type conversion.
        let mut transport_message = msg.into_local_message().into_transport_message();

        transport_message
            .onward_route
            .modify()
            .pop_front() // Remove my address from the onward_route.
            .prepend(self.address.clone()); // Prepend predefined address to the onward_route.

        let prev_hop = transport_message.return_route.next()?.clone();

        // Wipe all local info (e.g. transport types).
        let message = LocalMessage::new(transport_message, vec![]);

        if let Some(info) = ctx
            .flow_controls()
            .find_flow_control_with_producer_address(&self.address)
        {
            ctx.flow_controls()
                .add_consumer(prev_hop.clone(), info.flow_control_id());
        }

        if let Some(info) = ctx
            .flow_controls()
            .find_flow_control_with_producer_address(&prev_hop)
        {
            ctx.flow_controls()
                .add_consumer(self.address.clone(), info.flow_control_id());
        }

        // Send the message on its onward_route.
        ctx.forward(message).await
    }
}

Responder node

Finally, we will create the responder node. This node will run the worker echoer which actually echoes the message back to the initiator. Copy and paste the following into the source file above (/examples/04-routing-over-two-transport-hops.rs).

This node has an Echoer worker on address echoer, so that's how the initiator can reach this address specified in its route at the start of this example.

/// This node starts a TCP listener and an echoer worker. It then runs forever waiting for
/// messages.
async fn create_responder_node(ctx: Context) -> Result<ockam::Node> {
    print_title(vec![
        "Create node_responder that runs tcp listener on 4000 and 'echoer' worker",
        "wait for messages until stopped",
    ]);

    // Create a node with default implementations.
    let node = node(ctx);

    // Initialize the TCP transport.
    let tcp_transport = node.create_tcp_transport().await?;

    // Create an echoer worker.
    node.start_worker("echoer", Echoer).await?;

    // Create a TCP listener and wait for incoming connections.
    let listener = tcp_transport
        .listen("127.0.0.1:4000", TcpListenerOptions::new())
        .await?;

    // Allow access to the Echoer via TCP connections from the TCP listener.
    node.flow_controls()
        .add_consumer("echoer", listener.flow_control_id());

    Ok(node)
}

Let's run this example to see what it does 🎉.

In your terminal app, run the following command. Note that OCKAM_LOG=none is used to disable logging output from the Ockam library. This is done to make the output of the example easier to read.

cargo run --example 04-routing-over-two-transport-hops

This should produce output similar to the following. Our example program creates a route that traverses multiple nodes and TCP transports from the app to the echoer and routes our message through them 🚀.

Next steps

Ockam Routing and transports are extremely powerful and flexible. They are one of the key features that enables Ockam Secure Channels to be implemented. By layering Ockam Secure Channels and other protocols over Ockam Routing, we can provide end-to-end guarantees over arbitrary transport topologies that span many networks and clouds.

In a future blog post we will be covering Ockam Secure Channels and how they can be used to provide end-to-end guarantees over arbitrary transport topologies. So stay tuned!

In the meantime here are some good jumping off points to learn more about Ockam:

Deep dive into Ockam Routing.
Install Ockam command line interface (CLI) (ockam command) on your computer and try to create end-to-end encrypted communication between two apps. That will give you a taste of the experience of using Ockam on the command line in addition to our Rust library.

Authenticate & authorize every access decision

Rewriting Ockam in Rust

Edit on Github

Table of content