Interact

Starting off...

Before we dive in, it is important to note that the pipelines in the Core Engine are created, executed and evaluated within the scope of a workspace. Thus, the following interactions such as listing, pulling or pushing are only possible in the context of a workspace

List your pipelines

Python SDK
CLI
CE Dashboard
Python SDK

In order to get a list of registered pipelines through the SDK, execute the following steps:

import cengine
# Create a client
client = cengine.Client(username='my_username', password='my_password')
# Obtain the work
pipelines = client.get_pipelines(workspace_id='my_workspace_id')

Once executed, get_pipelines will return a list of cengine.Pipeline objects, which reside within the workspace with the corresponding workspace_id.

CLI

In order to list the registered pipelines within the context of a workspace through the CLI, use the following command:

cengine pipeline list

Example for the expected output:

You have the following workspace as the active workspace:
- Workspace Name: Demo Workspace
- Workspace ID: e51k399a
Fetching pipeline(s). This might take a few seconds...
PIPELINE NAME: DemoPipeline PIPELINE ID: 3da711ec
-------------------------------------------------
RUN ID TYPE CPUs PER WORKER WORKERS DATASOURCE AUTHOR CREATED AT
3jf73y85 training 4 4 Demo Datasource foo@maiot.io 2020-07-27 13:43:18
PIPELINE NAME: DemoPipeline PIPELINE ID: ec646376
-------------------------------------------------
RUN ID TYPE CPUs PER WORKER WORKERS DATASOURCE AUTHOR CREATED AT
61h314f8 training 4 4 Demo Datasource bar@maiot.io 2020-07-28 16:02:48
CE Dashboard

Currently, this feature is unavailable.

We are working hard to create a dashboard for the Core Engine. Please see our roadmap for an indication on when this feature will be released.

Pull a pipeline

Python SDK
CLI
CE Dashboard
Python SDK

In order to pull the configuration of selected pipeline, you can use the pull_pipeline method of your client.

import cengine
# Create a client
client = cengine.Client(username='my_username', password='my_password')
# Obtain the work
config = client.pull_pipeline(pipeline_id='my_pipeline_id')

This method will return an instance cengine.PipelineConfig which represents the pipeline with the corresponding pipeline_id.

CLI
cengine pipeline pull PIPELINE_ID [--output_path=OUTPUT_PATH]

where:

  • PIPELINE_ID is a string value which identifies the selected pipeline. As a convenience function, you can also set this i.e to 2a3 if you have only one pipeline whose id starts 2a3.

  • output_path is an optional value and it is a path which is utilized to download the config file. If not provided, it will download the configuration to the working directory and name it ce_config.yaml.

CE Dashboard

Currently, this feature is unavailable.

We are working hard to create working hard to create a dashboard for the Core Engine. Please see our roadmap for an indication on when this feature will be released.

Push a pipeline

Python SDK
CLI
CE Dashboard
Python SDK

In order to register a new pipeline in your workspace, you can use the method push_pipeline:

import cengine
# Create a client
client = cengine.Client(username='my_username', password='my_password')
# Generate a configuration
config = cengine.PipelineConfig()
...
...
# Obtain the work
pipelines = client.push_pipeline(name='New Pipeline',
config=config,
workspace_id='my_workspace_id')

Once executed, this method will return the instance of cengine.Pipeline which represents the newly registered pipeline.

CLI
cengine pipeline push CONFIG_PATH PIPELINE_NAME

ARGUMENT

TYPE

DESCRIPTION

CONFIG_PATH

path

Path to the config file to submit

PIPELINE_NAME

string

Name of the new pipeline

CE Dashboard

Currently, this feature is unavailable.

We are working hard to create a dashboard for the Core Engine. Please see our roadmap for an indication on when this feature will be released.