API Reference - Backend

The backend system provides a flexible storage and persistence layer with support for multiple storage backends.

Core Components

BackendConnectorBase

class BackendConnectorBase

Abstract base class for handling backend connections.

Attributes:

Parameters:

  • host – Backend server host

  • port – Backend server port

  • username – Authentication username

  • password – Authentication password

  • db – Database name/number

Methods:

connect()

Establish connection to the backend.

disconnect()

Close the backend connection.

is_connected() → bool

Check if currently connected to the backend.

Key-Value Store

KeyValueStoreBackendBase

class KeyValueStoreBackendBase

Abstract base class for key-value store implementations.

Attributes:

Parameters:

connector_klass – The connector class to use for this store

Methods:

exists(schema_name: str, record_key: str) → bool

Check if a record exists.

count(schema_name: str) → int

Get the number of records in a schema.

insert_record(schema_name: str, record_key: str, record: BackendIntegrationMixin)

Insert a new record.

update_record(schema_name: str, record_key: str, record: BackendIntegrationMixin)

Update an existing record.

delete_record(schema_name: str, record_key: str)

Delete a record.

get_record(schema_name: str, klass: Type[BackendIntegrationMixin], record_key: str | int)

Retrieve a specific record.

reload_record(schema_name: str, record: BackendIntegrationMixin)

Reload a record’s data from storage.

filter_record(schema_name: str, record_klass: Type[BackendIntegrationMixin], **filter_kwargs)

Filter records based on criteria.

Backend Implementations

InMemoryKeyValueStoreBackend

class InMemoryKeyValueStoreBackend

In-memory implementation of key-value store.

Uses a simple dictionary-based storage for testing and development.

RedisStoreBackend

class RedisStoreBackend

Redis-based implementation of key-value store.

Uses Redis hash sets for schema-based record storage with support for atomic operations.

HDFSStoreBackend

class HDFSStoreBackend

HDFS implementation of key-value store.

Stores records as JSON files in HDFS with schema-based directory organization.

Additional Configuration:

Parameters:

base_path – Base HDFS path for storage (default: “/volnux”)

Connectors

RedisConnector

class RedisConnector

Redis connection handler.

Parameters:

Parameters:

  • host – Redis server host

  • port – Redis server port

  • db – Redis database number

HDFSConnector

class HDFSConnector

HDFS connection handler.

Parameters:

Parameters:

  • host – HDFS namenode host

  • port – HDFS namenode port

  • username – HDFS username

Connection Management

ConnectionMode

class ConnectionMode

Enumeration of connection management modes.

Values:

  • SINGLE: Single shared connection

  • POOLED: Connection pooling

ConnectorManagerFactory

class ConnectorManagerFactory

Factory for creating appropriate connector managers.

Methods:

classmethod create_manager(connector_class, connector_config: dict, connection_mode: ConnectionMode = None, **kwargs)

Create a connector manager based on backend type and configuration.

SingleConnectorManager

class SingleConnectorManager

Manages a single shared connection to a backend store.

PooledConnectorManager

class PooledConnectorManager

Manages a pool of connections to a backend store.

Parameters:

Parameters:

  • max_connections – Maximum number of connections in pool

  • connection_timeout – Timeout for connection acquisition

  • idle_timeout – Maximum idle time before connection cleanup