Skip to main content

TCP/IP Offload

TCP/IP offload APIs to seamlessly integrate and leverage TCP Offload Engine (TOE) with the existing network infrastructure. In general, we recommend using LD_PRELOAD which hooks standard POSIX socket APIs to support non-modified applications (e.g., iPerf, Nginx). However, if you want to develop more optimizd TCP applications, these API will be the best option.

There are three types of handles for TCP APIs:

  • IPC IPC server to orchestrate requests and responses.
  • TCP Per-device TCP connection management.
  • Socket Per-session socket buffer management.

Example​

Here is an example of using TCP offload APIs.

example_toe_client.cc
#include <libmango.h>

int main() {
  // open an IPC server
  mango_ipc_h ipc;
  mango_ipc_open ("localhost:32847", false, &ipc);

  // create TCP connected with IPC server
  mango_tcp_h tcp;
  mango_tcp_create (ipc, &tcp);
  mango_tcp_set_ipc (tcp, ipc);

  // create TCP socket;
  mango_socket_h sock;
  mango_socket_create (tcp, &sock);
  mango_socket_open (sock, NULL);

  // connect to the remote server
  struct sockaddr_in addr = { ... };
  mango_socket_connect (sock, (struct sockaddr *) &addr, sizeof (addr));

  // send data to the remote server
  char buf[128];
  while (1) {
    mango_socket_send (sock, buf, 128, NULL);
    ...
  }

  return 0;
}

Datatypes​

mango_tcp_h​

typedef void * mango_tcp_h;

A handle of mango tcp.

mango_socket_h​

typedef void * mango_socket_h;

A handle of mango socket.

mango_ipc_h​

typedef void * mango_ipc_h;

A handle of mango ipc.

mango_ipc_prot_e​

typedef enum _mango_ipc_prot mango_ipc_prot_e;

Define available IPC protocol type.

Values

MANGO_IPC_PROT_GRPCgRPC IPC protocol
MANGO_IPC_PROT_ICEORYXIceoryx IPC protocol

mango_ipc_msg_e​

typedef enum _mango_ipc_msg mango_ipc_msg_e;

Define available IPC message type.

Values

MANGO_IPC_MSG_HEALTH_CHECKHealth check message
MANGO_IPC_MSG_SET_CONFIGSet configuration message
MANGO_IPC_MSG_GET_CONFIGGet configuration message
MANGO_IPC_MSG_TCP_LISTENTCP listen message
MANGO_IPC_MSG_TCP_ACCEPTTCP accept message
MANGO_IPC_MSG_TCP_CONNECTTCP connect message
MANGO_IPC_MSG_TCP_CLOSETCP close message
MANGO_IPC_MSG_TCP_LISTEN_CLOSETCP listen close message
MANGO_IPC_MSG_TCP_ABORTTCP abort message

Functions​

mango_tcp_create​

mango_status_e mango_tcp_create(mango_tcp_h *tcp)

Create an tcp handle.

Parameters

  • out tcp The output tcp handle.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_destroy​

mango_status_e mango_tcp_destroy(mango_tcp_h tcp)

Destroy the tcp handle.

Parameters

  • in tcp The tcp handle to be destroyed.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_reset​

mango_status_e mango_tcp_reset(mango_tcp_h tcp, bool preserved)

Reset the tcp handle.

Parameters

  • in tcp The tcp handle to be reset.
  • in preserved Preserve internal states (e.g., IP/MAC address)

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_set_address​

mango_status_e mango_tcp_set_address(mango_tcp_h tcp,
                                     const char *ipaddr,
                                     const char *macaddr)

Set ethernet addresses to the tcp.

Parameters

  • in tcp the tcp handle.
  • in ipaddr the IP address.
  • in macaddr the MAC address.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_get_ip_address​

mango_status_e mango_tcp_get_ip_address(mango_tcp_h tcp, const char **ipaddr)

Get IP addresses from the tcp.

Parameters

  • in tcp the tcp handle.
  • out ipaddr the IP address.

Returns 0 on success, Otherwise, a negative error value.

note

this API uses inet_ntoa() internally to be rewritten.

mango_tcp_get_address​

mango_status_e mango_tcp_get_address(mango_tcp_h tcp,
                                     struct in_addr *ipaddr,
                                     struct in_addr *subnet,
                                     struct ether_addr *macaddr)

Get addresses from the tcp.

Parameters

  • in tcp the tcp handle.
  • out ipaddr (Optional) the IP address.
  • out subnet (Optional) the subnet mask.
  • out macaddr (Optional) the MAC address.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_set_context​

mango_status_e mango_tcp_set_context(mango_tcp_h tcp, mango_context_h context)

Set the DPU context to the tcp.

Parameters

  • in tcp the tcp handle.
  • in context the context handle.

Returns 0 on success, Otherwise, a negative error value.

note

Context and IPC should be set exclusively.

mango_tcp_set_ipc​

mango_status_e mango_tcp_set_ipc(mango_tcp_h tcp, mango_ipc_h ipc)

Set the IPC channel to the tcp.

Parameters

  • in tcp the tcp handle.
  • in ipc the ipc handle.

Returns 0 on success, Otherwise, a negative error value.

note

In TCP, context and ipc should be set exclusively.

mango_tcp_connect​

mango_status_e mango_tcp_connect(mango_tcp_h tcp,
                                 const struct sockaddr_in *saddr,
                                 struct sockaddr_in *caddr,
                                 int *session)

Initiate a TCP connection.

Parameters

  • in tcp The tcp handle.
  • in saddr The server sockaddr structure pointer.
  • out caddr The client sockaddr structure pointer.
  • out session The output session id.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_listen​

mango_status_e mango_tcp_listen(mango_tcp_h tcp, struct sockaddr_in *addr)

Listen for incoming connections.

Parameters

  • in tcp The tcp handle.
  • in addr The sockaddr structure pointer.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_listen_close​

mango_status_e mango_tcp_listen_close(mango_tcp_h tcp, struct sockaddr_in *addr)

Stop listening to incoming connections.

Parameters

  • in tcp The tcp handle.
  • in addr The sockaddr structure pointer.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_accept​

mango_status_e mango_tcp_accept(mango_tcp_h tcp,
                                struct sockaddr_in *saddr,
                                struct sockaddr_in *caddr,
                                bool non_blocking,
                                int *session)

Accept a TCP connection.

Parameters

  • in tcp The tcp handle.
  • inout saddr The server sockaddr structure pointer.
  • out caddr (Optional) The client sockaddr structure pointer.
  • in non_blocking Indicates non blocking mode.
  • out session The output session id.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_wait​

mango_status_e mango_tcp_wait(mango_tcp_h tcp, int session)

Wait completions of previous send requests.

Parameters

  • in tcp The tcp handle.
  • in session The session id.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_close​

mango_status_e mango_tcp_close(mango_tcp_h tcp, int session)

Close the TCP connection.

Parameters

  • in tcp The tcp handle.
  • in session The session id.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_abort​

mango_status_e mango_tcp_abort(mango_tcp_h tcp, int session)

Abort the TCP connection.

Parameters

  • in tcp The tcp handle.
  • in session The session id.

Returns 0 on success, Otherwise, a negative error value.

note

It does not close the socket.

mango_tcp_session_reset​

mango_status_e mango_tcp_session_reset(mango_tcp_h tcp,
                                       int session,
                                       suseconds_t timeout)

Reset the TCP connection.

Parameters

  • in tcp The tcp handle.
  • in session The session id.
  • in timeout The reset timeout in usec. (-1: block indefinitely, 0: return immediately)

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_epoll_create​

mango_status_e mango_tcp_epoll_create(mango_tcp_h tcp, int epfd)

Create a new epoll instance.

Parameters

  • in tcp The tcp handle.
  • in epfd The file descriptor referring to the epoll instance.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_epoll_ctl​

mango_status_e mango_tcp_epoll_ctl(mango_tcp_h tcp,
                                   int epfd,
                                   int op,
                                   int fd,
                                   struct epoll_event *event)

Register epoll operations.

Parameters

  • in tcp The tcp handle.
  • in epfd The epoll file descriptor.
  • in op The epoll opcode.
  • in fd The file descriptor
  • in event The epoll event.

Returns 0 on success, Otherwise, a negative error value.

note

A negative session id indicates the TCP socket itself.

mango_tcp_epoll_wait​

mango_status_e mango_tcp_epoll_wait(mango_tcp_h tcp,
                                    int epfd,
                                    struct epoll_event *event,
                                    int max_events,
                                    int timeout,
                                    int *num_events)

Wait epoll events.

Parameters

  • in tcp The tcp handle.
  • in epfd The epoll file descriptor.
  • in event The epoll event
  • in max_events The number of maximum events
  • in timeout The epoll timeout
  • out num_events The number of detected events

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_epoll_check​

mango_status_e mango_tcp_epoll_check(mango_tcp_h tcp, int fd)

Check whether a fd is an epoll instance of a tcp instance.

Parameters

  • in tcp The tcp handle.
  • in fd The file descriptor.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_epoll_close​

mango_status_e mango_tcp_epoll_close(mango_tcp_h tcp, int epfd)

Close an epoll instance.

Parameters

  • in tcp The tcp handle.
  • in epfd The epoll file descriptor.

Returns 0 on success, Otherwise, a negative error value.

mango_tcp_poll​

mango_status_e mango_tcp_poll(mango_tcp_h tcp,
                              struct pollfd *fds,
                              nfds_t nfds,
                              int timeout,
                              int *num_events)

wait for events on TCP sockets.

Parameters

  • in tcp The tcp handle.
  • in fds The array of poll file descriptors.
  • in nfds The number of poll file descriptors.
  • in timeout The poll timeout
  • out num_events The number of detected events

Returns 0 on success, Otherwise, a negative error value.

mango_socket_create​

mango_status_e mango_socket_create(mango_tcp_h tcp, mango_socket_h *socket)

Create a TCP socket.

Parameters

  • in tcp The tcp handle.
  • out socket The output socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_destroy​

mango_status_e mango_socket_destroy(mango_socket_h socket)

Destroy the TCP socket.

Parameters

  • in socket The created socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_open​

mango_status_e mango_socket_open(mango_socket_h socket, int *fd)

Open the TCP socket.

Parameters

  • in socket The socket handle.
  • out fd The file descriptor referring to the socket.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_close​

mango_status_e mango_socket_close(mango_socket_h socket)

Close the TCP socket.

Parameters

  • in socket The socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_detach​

mango_status_e mango_socket_detach(mango_socket_h socket)

Detach the TCP socket and keep its file desriptor.

Parameters

  • in socket The socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_fd​

mango_status_e mango_socket_get_fd(mango_socket_h socket, int *fd)

Get the file descriptor of the TCP socket.

Parameters

  • in socket The socket handle.
  • out fd The file descriptor referring to the socket.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_session​

mango_status_e mango_socket_get_session(mango_socket_h socket, int *session)

Get the session ID of the TCP socket.

Parameters

  • in socket The socket handle.
  • out session The session ID referring to the socket.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_sequence​

mango_status_e mango_socket_get_sequence(mango_socket_h socket,
                                         uint32_t *tx,
                                         uint32_t *rx)

Get the sequence numbers of the TCP socket.

Parameters

  • in socket The socket handle.
  • out tx The TX sequence number of the socket.
  • out rx The RX sequence number of the socket.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_name​

mango_status_e mango_socket_get_name(mango_socket_h socket,
                                     struct sockaddr *addr,
                                     socklen_t *addrlen)

Get the address name of the TCP socket.

Parameters

  • in socket The socket handle.
  • out addr The sockaddr structure pointer.
  • out addrlen The sockaddr structure size.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_peername​

mango_status_e mango_socket_get_peername(mango_socket_h socket,
                                         struct sockaddr *addr,
                                         socklen_t *addrlen)

Get the peer address name of the TCP socket.

Parameters

  • in socket The socket handle.
  • out addr The sockaddr structure pointer.
  • out addrlen The sockaddr structure size.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_find​

mango_status_e mango_socket_find(int fd, mango_socket_h *socket)

Find the socket handle using a file descriptor.

Parameters

  • in fd The socket file descriptor.
  • out socket (Optional) The output socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_set_ssl​

mango_status_e mango_socket_set_ssl(mango_socket_h socket, void *ssl)

Set the SSL data to the TCP socket.

Parameters

  • in socket The socket handle.
  • in ssl The SSL data.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_set_option​

mango_status_e mango_socket_set_option(mango_socket_h socket,
                                       int level,
                                       int optname,
                                       const void *optval,
                                       socklen_t optlen)

Set TCP socket option.

Parameters

  • in socket The socket handle.
  • in level The protocol level.
  • in optname The option name.
  • in optval The option value.
  • in optlen The option length.

Returns 0 on success, Otehrwise, a negative error value.

mango_socket_get_option​

mango_status_e mango_socket_get_option(mango_socket_h socket,
                                       int level,
                                       int optname,
                                       void *optval,
                                       socklen_t *optlen)

Get TCP socket option.

Parameters

  • in socket The socket handle.
  • in level The protocol level.
  • in optname The option name.
  • out optval The option value.
  • out optlen The option length.

Returns 0 on success, Otehrwise, a negative error value.

mango_socket_set_nonblocking​

mango_status_e mango_socket_set_nonblocking(mango_socket_h socket, bool enable)

Set non-blocking socket.

Parameters

  • in socket The socket handle.
  • in enable Flag to enable a non-blocking mode.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_bind​

mango_status_e mango_socket_bind(mango_socket_h socket,
                                 const struct sockaddr *addr,
                                 socklen_t addrlen)

Bind address info. to the TCP socket.

Parameters

  • in socket The tcp socket handle.
  • in addr The sockaddr structure pointer.
  • in addrlen The sockaddr structure size.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_listen​

mango_status_e mango_socket_listen(mango_socket_h socket, int backlog)

Mark the socket as a passive one to be used to accept incoming connections.

Parameters

  • in socket The tcp socket handle.
  • in backlog The maximum number of pending connections.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_connect​

mango_status_e mango_socket_connect(mango_socket_h socket,
                                    const struct sockaddr *addr,
                                    socklen_t addrlen)

Initiate a TCP socket connection.

Parameters

  • in socket The tcp socket handle.
  • in addr The sockaddr structure pointer.
  • in addrlen The sockaddr structure size.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_accept​

mango_status_e mango_socket_accept(mango_socket_h socket,
                                   struct sockaddr *addr,
                                   socklen_t *addrlen,
                                   mango_socket_h *client)

Accept a TCP socket connection.

Parameters

  • in socket The tcp socket handle.
  • out addr (Optional) The sockaddr structure pointer.
  • inout addrlen (Optional) The sockaddr structure size.
  • out client The client socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_send​

mango_status_e mango_socket_send(mango_socket_h socket,
                                 const void *buf,
                                 size_t len,
                                 size_t *sent)

Send a TCP packet on the socket.

Parameters

  • in socket The socket handle
  • in buf The buffer to be sent
  • in len The buffer size
  • out sent The number of bytes sent

Returns 0 on success, Otherwise, a negative error value.

mango_socket_send_iov​

mango_status_e mango_socket_send_iov(mango_socket_h socket,
                                     const struct iovec *iov,
                                     int iovcnt,
                                     size_t *sent)

Send a TCP packet using I/O vector.

Parameters

  • in socket The socket handle
  • in iov The pointer of I/O vector
  • in iovcnt The number of I/O vector entries
  • out sent The number of bytes sent

Returns 0 on success, Otherwise, a negative error value.

mango_socket_send_tls​

mango_status_e mango_socket_send_tls(mango_socket_h socket,
                                     const void *buf,
                                     size_t len,
                                     size_t *sent)

Send a TLS packet on the socket.

Parameters

  • in socket The socket handle
  • in buf The buffer to be sent
  • in len The buffer size
  • out sent The number of bytes sent

Returns 0 on success, Otherwise, a negative error value.

note

the socket should have its SSL data already assigned.

mango_socket_recv​

mango_status_e mango_socket_recv(mango_socket_h socket,
                                 void *buf,
                                 size_t len,
                                 size_t *received)

Receive the TCP packet on the socket.

Parameters

  • in socket The socket handle
  • in buf The buffer to receive
  • in len The buffer size
  • out received The number of bytes received

Returns 0 on success, Otherwise, a negative error value.

mango_socket_recv_iov​

mango_status_e mango_socket_recv_iov(mango_socket_h socket,
                                     const struct iovec *iov,
                                     int iovcnt,
                                     size_t *received)

Receive the TCP packet using I/O vector.

Parameters

  • in socket The socket handle
  • in iov The pointer of I/O vector
  • in iovcnt The number of I/O vector entries
  • out received The number of bytes received

Returns 0 on success, Otherwise, a negative error value.

mango_socket_recv_tls​

mango_status_e mango_socket_recv_tls(mango_socket_h socket,
                                     void *buf,
                                     size_t len,
                                     size_t *received)

Receive a TLS packet on the socket.

Parameters

  • in socket The socket handle
  • in buf The buffer to receive
  • in len The buffer size
  • out received The number of bytes received

Returns 0 on success, Otherwise, a negative error value.

note

the socket should have its SSL data already assigned.

mango_socket_wait​

mango_status_e mango_socket_wait(mango_socket_h socket)

Wait completions of previous send requests.

Parameters

  • in socket The socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_get_num_avail​

mango_status_e mango_socket_get_num_avail(mango_socket_h socket, int *nbytes)

Get a number of recv data bytes immediately availble.

Parameters

  • in socket The socket handle.
  • out nbytes The number of recv bytes.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_shutdown​

mango_status_e mango_socket_shutdown(mango_socket_h socket, int how)

Shut down part of a full-duplex connection.

Parameters

  • in socket The socket handle.
  • in how The shutdown direction (in sys/socket.h).

Returns 0 on success, Otherwise, a negative error value.

mango_socket_abort​

mango_status_e mango_socket_abort(mango_socket_h socket)

Abort the TCP socket.

Parameters

  • in socket The socket handle.

Returns 0 on success, Otherwise, a negative error value.

note

It does not close the socket.

mango_socket_reset​

mango_status_e mango_socket_reset(mango_socket_h socket)

Reset the TCP socket.

Parameters

  • in socket The socket handle.

Returns 0 on success, Otherwise, a negative error value.

mango_socket_reset_blocking​

mango_status_e mango_socket_reset_blocking(mango_socket_h socket,
                                           useconds_t usec)

Reset the TCP socket, and wait for finishing it.

Parameters

  • in socket The socket handle.
  • in usec The microseconds to wait.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_open​

mango_status_e mango_ipc_open(const char *path, bool owner, mango_ipc_h *ipc)

Open a IPC channel.

Parameters

  • in path The socket path.
  • in owner The socket owner.
  • out ipc The IPC handle.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_set_index​

mango_status_e mango_ipc_set_index(mango_ipc_h ipc, int idx)

Set DPU index to the IPC.

Parameters

  • in ipc The ipc handle.
  • in idx The DPU index.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_get_config​

mango_status_e mango_ipc_get_config(mango_ipc_h ipc,
                                    const char *field,
                                    char **value)

Get a config value from the ipc server.

Parameters

  • in ipc The context handle.
  • in field The config field.
  • out value The config value newly allocated.

Returns 0 on success, Otherwise, a negative error value.

note

The caller should free the option value using free().

mango_ipc_set_config​

mango_status_e mango_ipc_set_config(mango_ipc_h ipc,
                                    const char *field,
                                    const char *value)

Set a config value to the ipc server.

Parameters

  • in ipc The context handle.
  • in field The config field.
  • in value The config value.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_get_server_path​

mango_status_e mango_ipc_get_server_path(mango_ipc_h ipc, const char **path)

Get the IPC server path.

Parameters

  • in ipc The ipc handle.
  • out path The ipc server path.

Returns 0 on success, Otherwise, a negative error value.

note

The path is statically allocated. No need to free.

mango_ipc_set_context​

mango_status_e mango_ipc_set_context(mango_ipc_h ipc, mango_context_h ctx)

Set a context handle to enable context message handling.

Parameters

  • in ipc The ipc handle.
  • in ctx The context handle.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_set_tcp​

mango_status_e mango_ipc_set_tcp(mango_ipc_h ipc, mango_tcp_h tcp)

Set a TCP handle to enable TCP message handling.

Parameters

  • in ipc The ipc handle.
  • in tcp The tcp handle.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_wait​

mango_status_e mango_ipc_wait(mango_ipc_h ipc)

Wait until the IPC server is shutdown.

Parameters

  • in ipc The created IPC handle.

Returns 0 on success, Otherwise, a negative error value.

note

It's not supported for non-server IPC handles.

mango_ipc_shutdown​

mango_status_e mango_ipc_shutdown(mango_ipc_h ipc)

Shutdown the IPC server.

Parameters

  • in ipc The IPC created handle.

Returns 0 on success, Otherwise, a negative error value.

mango_ipc_close​

mango_status_e mango_ipc_close(mango_ipc_h ipc)

Close The IPC connection.

Parameters

  • in ipc The IPC created handle.

Returns 0 on success, Otherwise, a negative error value.