Metastore
The Eleanor Framework metadata store (metastore), sometimes referred to at the relational database management system (RDBMS) is used to store persistent state in the EleanorAI Framework.
The primary objectives of the metastore configuration are to ensure data integrity, optimize performance, support long-running transactions, and provide a developer-friendly environment that abstracts the complexities of database interactions.
Technology Stack
The database system in the EleanorAI Framework is built upon the following components:
- Database Server: MariaDB with InnoDB storage engine.
- ORM (Object-Relational Mapping): SQLAlchemy ORM for object-relational mapping and database abstraction.
- Database Driver: PyMySQL as the MySQL database driver for Python.
- Session Management: SQLAlchemy’s sessionmaker and scoped_session for session creation and management.
- Connection Pooling: Connection pooling configurations to manage database connections efficiently.
Flush Behavior
Flushing synchronizes the session’s in-memory state with the database by issuing SQL statements for pending changes. Flushing does not commit the transaction; it only sends the changes to the database within the current transaction scope.
The EleanorAI Framework sets autoflush=True to automatically flush pending changes before query execution or when required by the session. Developers should rely on the autoflush behavior unless a specific need arises. Overusing manual flushes can lead to unnecessary performance overhead and bloated code. Scenarios where manual flushing is needed include:
- Accessing Generated Primary Keys or Defaults: When an immediate access to database-generated values (e.g., primary keys, default timestamps) is required before the next automatic flush.
- Dependent Operations Within a Transaction: When subsequent operations depend on the database state updated by earlier operations in the same transaction.
- Bulk Operations or Raw SQL Execution: When executing raw SQL queries or bulk operations that rely on the current session state being reflected in the database.
Thread Safety
SQLAlchemy sessions are NOT thread safe. Typically this is not an issue since eleanor.backend.api.dependencies.db_session_dependency injects a new session for every incoming connection coming from the FastAPI server. However, if there are additional thread pools managed my the service itself then a unique session must be created for each thread.
The following is general guidance to EleanorAI Framework developers when working with metastore sessions and multiple threads:
- Scoped Sessions: The use of scoped_session ties sessions to the current thread, ensuring that each thread works with its own session instance.
- Thread Safety: Developers must avoid sharing session instances across threads. When using thread pools or asynchronous execution, each thread or coroutine should acquire its own session via the session factory.
- Session Lifecycle Management: Sessions should be managed using context managers or explicit open and close operations to ensure proper resource cleanup.
Transaction Handling
In the EleanorAI Framework a new transaction begins automatically when a session is created and after each commit or rollback. Auto-commit is disabled to prevent automatic transaction commits and allows for precise error / exception handling. Developers should be cautious with long-running transactions due to potential issues with lock contention, increased risk of deadlocks, and data staleness.
Schema Versioning
The metastore uses Alembic to manage schema versioning. Whenever a change is made to the object relational mapping (ORM) under eleanor.backend.orm the following commands should be executed to generate an updated schema revision script:
To apply the schema revision script(s), execute the following make target:
Alternatively, alembic can generate an upgrade SQL script to STDOUT via the following command: