Protocol Documentation

Table of Contents

• ocellus_module_service.proto

  • BindRequest
  • DownloadRefRequest
  • DownloadRefResponse
  • IOData
  • ModuleData
  • OutputData
  • OutputData.Image
  • PointCloud

  • StateEvent

  • OcellusModuleService

• ocellus_types.proto

  • Camera
  • Color
  • Contour
  • Contours
  • Data
  • Data.CamerasEntry
  • Data.ContoursEntry
  • Data.DecimalsEntry
  • Data.IntegersEntry
  • Data.ItemsEntry
  • Data.PointCloudsEntry
  • Data.PositionsEntry
  • Data.QuaternionsEntry
  • Data.RgbaImagesEntry
  • Data.StringsEntry
  • Dimension
  • Error
  • EulerAngles
  • ExtremePoint
  • Extrinsics
  • FileData
  • GlobalConfig
  • Histogram
  • Intrinsics
  • Item
  • Item.ExtremePointsEntry
  • Items
  • PageanationData
  • PathPoint
  • Point
  • Points
  • Position
  • Quaternion
  • Resolution

  • Vector3Dim

  • CameraType

  • DataType
  • Intrinsics.Distortion

• Scalar Value Types

Top

ocellus_module_service.proto

License

Copyright 2021 Byte Motion AB

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Summary

The Ocellus system allows for remote modules to be integrated into it's core module system.

This protobuf describes an API for remote modules to bind themselves to Ocellus and therein recieve streaming events and provide streaming responses.

The API communcates over gRPC with protocol buffers which are described in detail here.

The afformentioned link will also provide instructions for building clients for most languages. More details on bidirectional streaming can be found here.

Click here to download the modules service protobuf file

Click here to download the types protobuf file

See a python example here

See a javascript example here

BindRequest

Bind payload of the initial request registering the remote module with Ocellus

Field	Type	Label	Description
name	string		The human readible name of the module. Must be unique
inputs	IOData	repeated	Deprecated. The input data types the module requires. May have several of the same input data types mapped with different names.
outputs	IOData	repeated	Deprecated. Data the module will output, made available to other modules.

DownloadRefRequest

Download reference payload

Field	Type	Label	Description
urn	string

DownloadRefResponse

Field	Type	Label	Description
data	bytes
point_cloud	PointCloud

IOData

Module input/output data specification

Field	Type	Label	Description
name	string		Unique name of the input/output data
data_type	DataType		Data type of the input/output data
key	string		The full key of the input/output data used when wiring module inputs and outputs

ModuleData

Bind payload

Field	Type	Label	Description
bind_request	BindRequest
data	OutputData
timestamp	int64		This should be set to the unix time in milliseconds which data was aquired or the start of the binding process. For example, the time at which the shutter opened when aquiring a mat from a camera. Or a critical time in the event generation process

OutputData

Field	Type	Label	Description
intrinsics	Intrinsics		intrinsics data
contours	Contour	repeated	contours data
camera	Camera		cameras data
image	OutputData.Image		images data
point_cloud	PointCloud		point clouds data
items	Item	repeated	items data
decimal	double		generic decimal data
integer	int32		generic integer data
string	string		generic string data
errors	Error	repeated	errors, will cause ocellus to go to an error state
position	Position		position data
rotation	Quaternion		rotation data

OutputData.Image

Field	Type	Label	Description
data	bytes		Currently, the following file formats are supported:

• Windows bitmaps - .bmp, .dib (always supported)

• JPEG files - .jpeg, .jpg, *.jpe (see the Note section)

• JPEG 2000 files - *.jp2 (see the Note section)

• Portable Network Graphics - *.png (see the Note section)

• WebP - *.webp (see the Note section)

• Portable image format - .pbm, .pgm, .ppm .pxm, *.pnm (always supported)

• Sun rasters - .sr, .ras (always supported)

• TIFF files - .tiff, .tif (see the Note section)

• OpenEXR Image files - *.exr (see the Note section)

• Radiance HDR - .hdr, .pic (always supported)

• Raster and Vector geospatial data supported by GDAL (see the Note section) |

PointCloud

Field	Type	Label	Description
width	int32
height	int32
vertices	Vector3Dim	repeated	Not included in response if "format=depth" included in URN
mat32_fc1_data	bytes		Only included in response if "format=depth" included in URN 32-bit floats encoded as byte array, every pixel is a value in meters

StateEvent

Ocellus works by passing a state through several reducers internaly. The remote module is treated no different except it passes only the data specified in the remote modules inputs, in the UI for example.

Field	Type	Label	Description
data	Data		Available input names (keys in each map) are found in the OcellusModuleInfoService#GetModules rpc endpoints GetModulesResponse.outputParameters.key fields.
frame_id	int64
timestamp	int64		unix time in milliseconds which this event was created in ocellus

OcellusModuleService

Service for remote modules

Method Name	Request Type	Response Type	Description
Bind	ModuleData stream	StateEvent stream	Used to bind a remote module to the Ocellus. Initial bind response will always receive an empty StateEvent containing only the initial frame id. This can be used in two ways:

1. If the remote module is generating data for the rest of the pipeline it may begin. For example, it may begin sending images and pointclouds.

2. If the remote module is processing data only, the data will begin to arrive in the next message. | | DownloadRef | DownloadRefRequest | DownloadRefResponse | Used to download reference URNs provided by StateEvents. The data will only be available from after an event has been received and a response is sent. |

Top

ocellus_types.proto

License

Copyright 2021 Byte Motion AB

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Camera

Field	Type	Label	Description
name	string		name of camera
position	Position		Position relative scene center point
quaternion	Quaternion		Quaternion relative scene center point
euler_angles	EulerAngles		EulerAngles relative scene center point
intrinsics	Intrinsics
camera_type	CameraType

Color

The three-channel RGB color model supplemented with a fourth alpha channel.

Field	Type	Label	Description
r	float		red channel intensity, a number between 0 and 255
g	float		green channel intensity, a number between 0 and 255
b	float		blue channel intensity, a number between 0 and 255
a	float		a number between 0.0 (fully transparent) and 1.0 (fully opaque).

Contour

Contours can be explained simply as a curve joining all the continuous points (along the boundary).

Field	Type	Label	Description
outer_contour	Points
hole_contours	Points	repeated
area	double

Contours

Field	Type	Label	Description
contours	Contour	repeated

Data

A collection of maps, each of which's key must be of the format "<module-id>/<output-name>" Example: MyBlobModule/RGBA_IMAGE

Field	Type	Label	Description
cameras	Data.CamerasEntry	repeated	cameras data
contours	Data.ContoursEntry	repeated	contours data
items	Data.ItemsEntry	repeated	items data
rgba_images	Data.RgbaImagesEntry	repeated	rgbaImages data, name to urn. Use the DownloadRef rpc to fetch
point_clouds	Data.PointCloudsEntry	repeated	point clouds data, name to urn. Use the DownloadRef rpc to fetch
decimals	Data.DecimalsEntry	repeated	generic double data
integers	Data.IntegersEntry	repeated	generic int32 data
strings	Data.StringsEntry	repeated	generic string data
positions	Data.PositionsEntry	repeated	generic positional data
quaternions	Data.QuaternionsEntry	repeated	generic quaternion data

Data.CamerasEntry

Field	Type	Label	Description
key	string
value	Camera

Data.ContoursEntry

Field	Type	Label	Description
key	string
value	Contours

Data.DecimalsEntry

Field	Type	Label	Description
key	string
value	double

Data.IntegersEntry

Field	Type	Label	Description
key	string
value	int32

Data.ItemsEntry

Field	Type	Label	Description
key	string
value	Items

Data.PointCloudsEntry

Field	Type	Label	Description
key	string
value	string

Data.PositionsEntry

Field	Type	Label	Description
key	string
value	Position

Data.QuaternionsEntry

Field	Type	Label	Description
key	string
value	Quaternion

Data.RgbaImagesEntry

Field	Type	Label	Description
key	string
value	string

Data.StringsEntry

Field	Type	Label	Description
key	string
value	string

Dimension

Field	Type	Label	Description
width	float		width in meters
height	float		height in meters
depth	float		depth in meters

Error

Field	Type	Label	Description
code	int32		a well defined error code
message	string		short description of error
description	string		long description of error (e.g. includes stack trace)

EulerAngles

Field	Type	Label	Description
yaw	float		z-axis in degrees (-180 to +180)
pitch	float		y-axis in degrees (-180 to +180)
roll	float		x-axis in degrees (-180 to +180)
local	EulerAngles		The rotation as Euler angles in degrees relative to the parent transform's rotation.

ExtremePoint

Field	Type	Label	Description
position	Position
pixel	Point		pixel data relative to the 2D canvas calculated with respect to intrinsics
accuracy	float		value between 0 and 1 how confident the position is

Extrinsics

Rotation and translation vectors which translates a coordinates of a 3D point to a coordinate system

Field	Type	Label	Description
rotation	float	repeated	The rvec (rotation vector) represents the rotation of the 3D object (including, but not limited to another camera) with respect to the camera in 3D space. The vector’s direction represents the axis of rotation, and its magnitude represents the angle of rotation.
translation	float	repeated	The tvec (translation vector) represents the translation of the 3D object (including, but not limited to another camera) with respect to the camera in 3D space. It defines how much and in which direction the object has moved from the origin (camera position).

FileData

Field	Type	Label	Description
name	string
date_time	string
mime_type	string
size	int64
is_directory	bool
path	string

GlobalConfig

Field	Type	Label	Description
position	Vector3Dim
orientation	Vector3Dim

Histogram

Field	Type	Label	Description
red	int32	repeated
green	int32	repeated
blue	int32	repeated

Intrinsics

Field	Type	Label	Description
width	int32		number of columns in pixels
height	int32		number of rows pixels
ppx	float		describe the x pixel coordinates of the principal point (center of projection)
ppy	float		describe the y pixel coordinates of the principal point (center of projection)
fx	float		describe the focal length of the image, as a multiple of pixel width and height
fy	float		describe the focal length of the image, as a multiple of pixel width and height
model	Intrinsics.Distortion		describes which of several supported distortion models was used to calibrate the image
coeffs	float	repeated	provides an array of up to five coefficients describing the distortion model
fov	float	repeated	array of two elements in degrees defining field of view. first element is horizontal, second is vertical

Item

Field	Type	Label	Description
id	int64		required can be used to identify the item
name	string		required human readible item name
parent_camera	Camera		required camera item position is relative to
weight	float		estimated weight in kilograms
confidence	float		value between 0 and 1 how confident the object detection is
color	Color		rendition color of the item in the Ocellus viewer
delta_timestamp	int64		unix style timestamp in ms
dimension	Dimension
position	Position		required Position relative to the camera that produces this item
quaternion	Quaternion		required unless eulerAngles is set Quaternion relative to the camera that produces this item
euler_angles	EulerAngles		required unless quaternion is set Eular rotation relative to the camera that produces this item
extreme_points	Item.ExtremePointsEntry	repeated	most extreme points of the item's contour
children	Item	repeated	nested items
path_plan	PathPoint	repeated
contour	Contour
pixel	Point		pixel data relative to the 2D canvas calculated with respect to intrinsics
histogram	Histogram
accuracy	float		value between 0 and 1 how confident the position is

Item.ExtremePointsEntry

Field	Type	Label	Description
key	string
value	ExtremePoint

Items

Field	Type	Label	Description
data	Item	repeated

PageanationData

Field	Type	Label	Description
from	int32
to	int32
max	int32
limit	int32
path	string

PathPoint

Represents a single point in a path plan.

Field	Type	Label	Description
delta_time	int64		time in ms since previous planned path point (e.g. previos PathPoint object in current array)
position	Position		Position relative to the item that is owner of the PathPlan
euler_angles	EulerAngles		EulerAngles relative to the item that is owner of the PathPlan
quaternion	Quaternion		Quaternion relative to the item that is owner of the PathPlan
pixel	Point		pixel data relative to the 2D canvas calculated with respect to intrinsics

Point

Describes a single pixel position.

Field	Type	Label	Description
x	double		Increases from left to right
y	double		Increases from top to bottom

Points

Field	Type	Label	Description
points	Point	repeated

Position

Value relative to the camera source's position in meters

Field	Type	Label	Description
x	float
y	float
z	float
local	Position		Position of the transform relative to the parent transform.

Quaternion

Set of numbers that comprise a four-dimensional vector space

Field	Type	Label	Description
x	float
y	float
z	float
w	float
local	Quaternion		The rotation of the transform relative to the transform rotation of the parent.

Resolution

Field	Type	Label	Description
width	int32
height	int32

Vector3Dim

Field	Type	Label	Description
x	float
y	float
z	float

CameraType

Name	Number	Description
CAMERA_UNKNOWN	0
CAMERA_2D	1	2D camera, produces rgba image output.
CAMERA_3D	2	3D camera, The module's pointcloud output will be derrived from the camera instead of producing a flat pointcloud. If running in buddy mode with a 2D camera, the cameras need to be re-calibrated with the normal calibration procudure. Once this is done, 2D to 3D and 3D to 2D point conversions will be accurate between the 2D camera's rgba image outputs and the 3D camera's pointcloud outputs.
CAMERA_STEREO	3	Same as 2D, except requires a buddy camera to be configured. UNIMPLEMENTED as of writing, just a placeholder for now.

DataType

Name	Number	Description
DATA_TYPE_UNKNOWN	0
DATA_TYPE_CAMERA	1
DATA_TYPE_CONTOURS	2
DATA_TYPE_ITEMS	3
DATA_TYPE_RGBA_IMAGE	4
DATA_TYPE_POINTCLOUD	5
DATA_TYPE_DECIMAL	6
DATA_TYPE_INTEGER	7
DATA_TYPE_STRING	8
DATA_TYPE_ERROR	9
DATA_TYPE_POSITION	10
DATA_TYPE_ROTATION	11

Intrinsics.Distortion

Name	Number	Description
distortion_none	0
distortion_modified_brown_conrady	1
distortion_inverse_brown_conrady	2
distortion_ftheta	3
distortion_brown_conrady	4

Scalar Value Types

.proto Type	Notes	C++	Java	Python	Go	C#	PHP	Ruby
double		double	double	float	float64	double	float	Float
float		float	float	float	float32	float	float	Float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
int64	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.	int64	long	int/long	int64	long	integer/string	Bignum
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	uint	integer	Bignum or Fixnum (as required)
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	ulong	integer/string	Bignum or Fixnum (as required)
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long	int64	long	integer/string	Bignum
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int	uint32	uint	integer	Bignum or Fixnum (as required)
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long	uint64	ulong	integer/string	Bignum
sfixed32	Always four bytes.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sfixed64	Always eight bytes.	int64	long	int/long	int64	long	integer/string	Bignum
bool		bool	boolean	boolean	bool	bool	boolean	TrueClass/FalseClass
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode	string	string	string	String (UTF-8)
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str	[]byte	ByteString	string	String (ASCII-8BIT)