ReasonChip is a distributed, pipeline-driven reasoning system designed to handle complex, asynchronous job execution across multiple components. A the heart of the system are three major roles: clients, brokers, and workers. This document outlines the responsibilities of each component, how they interact, and the overall flow of job execution within the infrastructure.

Purpose:
Clients are external systems, user interfaces, or automated scripts that initiate a job or pipeline. They act as the entry point for work into the ReasonChip system.

Responsibilities:

• Create and submit jobs to the system.
• Establish long-lived bidirectional connections to receive progress, logs, or output (e.g., via gRPC streaming).
• Authenticate and optionally specify job configuration or input data.

Purpose:
The broker serves as the central coordination point of the ReasonChip system. It accepts incoming client connections and routes work to available workers.

Responsibilities:

• Accept and manage incoming client sessions.
• Queue or throttle job submissions to control load.
• Dispatch jobs to available workers based on criteria like capacity, type, or affinity.
• Maintain job state and metadata.
• Optionally log or audit job lifecycle events.
• Communicate between client and worker for status updates or results.

Optional Capabilities:

• Load balancing across multiple workers.
• Secure communication enforcement (e.g., Unix sockets, SSL, gRPC TLS).
• Worker registration and health monitoring.

Purpose:
Workers are the executors of ReasonChip pipelines. Each worker processes tasks routed to it by a broker, potentially handling multiple tasks concurrently or sequentially depending on configuration.

Responsibilities:

• Register with a broker to accept work.
• Receive and execute jobs (often structured as YAML-defined ReasonChip pipelines).
• Isolate and manage job execution (e.g., fork, thread, process pool).
• Report progress, logs, and results back to the broker.
• Optionally support plugin-based task execution or custom pipeline extensions.

Execution Contexts:

• May run locally, on a container, or within sandboxed environments.
• Typically handle I/O, compute, or API-bound tasks as defined in the ReasonChip library.

Data flow

	Client -> Broker : Establish Connection
	Client -> Broker : Submit Job Request
	Broker -> Worker : Dispatch Job
	Worker -> Broker : Stream Logs/Updates
	Broker -> Client : Stream Logs/Status
	Worker --> Broker : Job Complete + Result
	Broker --> Client : Final Result
	

Transports

Client to Broker:

• Sockets (Unix, TCP) recommended
• gRPC
• HTTP/1 and HTTP/2 (slower because "connection"-per-request)

Worker to Broker:

• Sockets (Unix, TCP) recommended
• gRPC

Security

All transports provide for SSL/TLS encryption and authentication.

Secrets (e.g., API keys, tokens) should be stored in a secure vault or kept within isolated areas. The broker may be provided with secrets on startup and these will be combined to all requests received from the clients and sent to the workers. The workflows should not contain any secrets, and instead, the broker should be responsible for providing these to the workers.

Secrets are kept in memory during runtime. Depending on your log-levels, secrets MAY BE logged.

Logging

By default, all logging goes to syslog and different log-levels are provided.

Individual loggers (modules) within the system may be targetted for different log levels.