chariots.config

The config module allows you to define all of your Chariot’s app configuration in one place. Once this is done, the ChariotsConfiguration will allow you to

• create all of the necessary servers, workers and have them all work together

• get the clients you want to interact with.

The easiest way to create your config is to use a yaml file with all of the “static” configuration that don’t need to be created programmatically. here is an example:

op_store:
  op_store_db_url: 'sqlite:///memory:'
  saver_kwargs:
    root_path: /tmp/op_store_test
  saver_type: file-saver
  server_host: localhost
  server_port: 5000
pipelines:
  import_name: test_chariots_pipelines_server
  runner: sequential-runner
  server_host: localhost
  server_port: 80
pipelines_workers:
  use_for_all: true
  worker_pool_kwargs:
    redis_kwargs:
      host: localhost
      port: 8080
  worker_type: rq

Once this is done, you can load the configuration using the ChariotsConfig class:

>>> my_config = ChariotsConfiguration('config.yaml')  

Once you have loaded the configuration, you can always modify it programatically:

>>> my_config.pipelines_config.pipelines.append(some_pipeline)

You can than get the clients or servers you want to deploy/use:

>>> my_config.get_pipelines_server() 
>>> my_config.get_op_store_client() 

The ChariotsConfig is built around three main components:

• The pipelines config

• The op_store config

• The workers config

class chariots.config.ChariotsConfig(config_file=None, pipelines_config: Optional[chariots.config.PipelinesConfig] = None, pipelines_worker_config: Optional[chariots.config.WorkersConfig] = None, op_store_config: Optional[chariots.config.OpStoreConfig] = None)

Bases: object

full configuration for Chariots. This configuration encapsulates all the other configurations available in Chariots you can either instanciate your Chariots Config using a yaml file.:

>>> my_conf = ChariotsConfig('config.yaml')  

or you can instantite it in pure python:

..doctest:

>>> my_conf = ChariotsConfig(
...     pipelines_config=PipelinesConfig(**pipelines_kwargs),
...     pipelines_worker_config=WorkersConfig(**pipelines_workers_kwargs),
...     op_store_config=OpStoreConfig(**op_store_kwargs),
... )

__init__(config_file=None, pipelines_config: Optional[chariots.config.PipelinesConfig] = None, pipelines_worker_config: Optional[chariots.config.WorkersConfig] = None, op_store_config: Optional[chariots.config.OpStoreConfig] = None)

Parameters

• config_file – a file to load the configuration from. If any other arguments are set during the instantiation, the config in the file that portrays to this part of the configuration will be completely overridden (for instance if both config_file and pipelines_config are set, the configuration under pipelines of the config file will be ignored)

• pipelines_config – the pipelines specific configuration.

• pipelines_worker_config – the pipelines workers configuration

• op_store_config – the pipelines Op Store configuration

get_op_store_client() → chariots.op_store._op_store_client.OpStoreClient

gets the op store client as configured by this configuration

get_op_store_server() → chariots.op_store._op_store.OpStoreServer

gets the Op Store server as configured by this configuration

get_pipelines_client() → chariots.pipelines.pipelines_client.PipelinesClient

gets the pipelines client as configured by this configuration

get_pipelines_server() → chariots.pipelines.pipelines_server.PipelinesServer

gets the pipelines server as configured by this configuration

get_pipelines_worker_pool() → chariots.workers._base_worker_pool.BaseWorkerPool

gets the pipelines workers as configured by this configuration

class chariots.config.OpStoreConfig(server_host: Optional[str] = None, server_port: Union[str, int, None] = None, saver_type: Optional[str] = None, saver_kwargs: Optional[Dict[str, Any]] = None, op_store_db_url: Optional[str] = None)

Bases: object

configuration for the Op Store server and client

__init__(server_host: Optional[str] = None, server_port: Union[str, int, None] = None, saver_type: Optional[str] = None, saver_kwargs: Optional[Dict[str, Any]] = None, op_store_db_url: Optional[str] = None)

Parameters

• server_host – the host of the server (where the client should try to contact)

• server_port – the port the server should be run at

• saver_type – the type of saver to be used by the op store to save the serialized ops (this should be a string describing the saver type such as ‘file-saver’ or ‘google-storage-saver’

• saver_kwargs – additional keyword arguments to be used when instanciating the saver

• op_store_db_url – the url (sqlalchemy compatibale) to locate the Op Store database

get_client() → chariots.op_store._op_store_client.OpStoreClient

returns the op_store client as configured by this OpStoreConfig

get_saver() → chariots.op_store.savers._base_saver.BaseSaver

returns the op_store saver as configured by this OpStoreConfig

get_server() → chariots.op_store._op_store.OpStoreServer

returns the op_store server as configured by this OpStoreConfig

class chariots.config.PipelinesConfig(runner: Union[str, chariots.pipelines.runners._base_runner.BaseRunner, None] = None, server_host: Optional[str] = None, server_port: Union[str, int, None] = None, pipelines: Optional[List[chariots.pipelines._pipeline.Pipeline]] = None, pipeline_callbacks: Optional[List[chariots.pipelines.callbacks._pipeline_callback.PipelineCallback]] = None, import_name: Optional[str] = None)

Bases: object

Configuration of the pipelines. This mainly describes what and how your pipelines should be run and served

__init__(runner: Union[str, chariots.pipelines.runners._base_runner.BaseRunner, None] = None, server_host: Optional[str] = None, server_port: Union[str, int, None] = None, pipelines: Optional[List[chariots.pipelines._pipeline.Pipeline]] = None, pipeline_callbacks: Optional[List[chariots.pipelines.callbacks._pipeline_callback.PipelineCallback]] = None, import_name: Optional[str] = None)

Parameters

• runner – the runner to use to run the instance (both in the main server and in the workers. This should either be A BaseRunner instance or a string describing the type of runner to use ( ‘sequential-runner’ for instance

• server_host – the host of the server (mainly used to create the proper client)

• server_port – the port to run the server at

• pipelines – the pipelines to be present in the pipelines server (this cannot be filled using the config yaml file format and has to be added after loading the file (or at init)

• pipeline_callbacks – the callbacks to be used accross all the pipelines of the server. This parameter cannot be filled trough the config file and have to be filed programmatically.

• import_name – the import name of the resulting PipelinesServer

get_client() → chariots.pipelines.pipelines_client.PipelinesClient

creates a PipelinesClient configured according to this configuration

get_runner() → chariots.pipelines.runners._base_runner.BaseRunner

creates a runner according to the configuration

class chariots.config.WorkersConfig(use_for_all: bool = False, worker_type: Optional[str] = None, worker_pool_kwargs: Optional[Dict[str, Any]] = None)

Bases: object

the configuration of Chariots Workerpolls

__init__(use_for_all: bool = False, worker_type: Optional[str] = None, worker_pool_kwargs: Optional[Dict[str, Any]] = None)

Parameters

• use_for_all – whether or not all the pipelines should be executed asynchronously using workers

• worker_type – the type of worker pool to use. This should be a string such as ‘rq’ for instance

• worker_pool_kwargs – additional keyword arguments to be passed down to the init of the WorkerPool

get_worker_pool() → chariots.workers._base_worker_pool.BaseWorkerPool

get the WorkerPool corresponding to this configuration