Skip to content

Packaging

Dependency Management

To help with reproducibility and distribution, dependencies are compiled via pip-compile. This will basically pin the dependencies that need to be pinned in the requirements/in/*.in files and dynamically compile them into the requirements/*.txt files.

Additionally, this allows for a dependency profile hierarchy to separate the various roles the EleanorAI Framework runs in.

graph TD;
    base.in --> backend.in;
    backend.in --> build.in;
    build.in --> development.in;
    base.in --> quantization.in;
    base.in --> ui.in;

Recompiling Dependencies

Any time a dependency is added, removed, or updated, the dependencies need to be re-compiled.

To re-compile the dependencies, run the following command:

make pkg-req-compile

Build Wheel

make clean pkg-build-whl

Troubleshooting

This section contains troubleshooting guides for common packaging issues.

Check for Conflicts

To output a list of dependency version incompatibilities, use the following command:

pip check

If a conflict is found you can manually downgrade the package:

pip install --upgrade 'numpy<2'

Check Package Compatibility

To manually scan the entire dependency tree for a package, use pipdeptree:

pipdeptree --reverse --packages numpy

Gives the following output:

numpy==2.1.1
├── accelerate==0.33.0 [requires: numpy>=1.17,<2.0.0]
│   ├── auto_gptq==0.7.1 [requires: accelerate>=0.26.0]
│   ├── autoawq==0.2.6+cu121 [requires: accelerate]
│   │   └── axolotl==0.4.1 [requires: autoawq>=0.2.5]
│   ├── axolotl==0.4.1 [requires: accelerate==0.33.0]
│   ├── peft==0.12.0 [requires: accelerate>=0.21.0]
│   │   ├── auto_gptq==0.7.1 [requires: peft>=0.5.0]
│   │   └── axolotl==0.4.1 [requires: peft==0.12.0]
│   └── trl==0.9.6 [requires: accelerate]
│       └── axolotl==0.4.1 [requires: trl==0.9.6]
├── auto_gptq==0.7.1 [requires: numpy]
├── bitsandbytes==0.43.3 [requires: numpy]
│   ├── axolotl==0.4.1 [requires: bitsandbytes==0.43.3]
│   └── galore-torch==1.0 [requires: bitsandbytes]
├── datasets==2.20.0 [requires: numpy>=1.17]
│   ├── auto_gptq==0.7.1 [requires: datasets]
│   ├── autoawq==0.2.6+cu121 [requires: datasets>=2.20]
│   │   └── axolotl==0.4.1 [requires: autoawq>=0.2.5]
│   ├── axolotl==0.4.1 [requires: datasets==2.20.0]
│   ├── evaluate==0.4.1 [requires: datasets>=2.0.0]
│   │   └── axolotl==0.4.1 [requires: evaluate==0.4.1]
│   ├── optimum==1.16.2 [requires: datasets]
│   │   └── axolotl==0.4.1 [requires: optimum==1.16.2]
│   └── trl==0.9.6 [requires: datasets]
│       └── axolotl==0.4.1 [requires: trl==0.9.6]
├...

Here, we can see that the entire tree of packages dependencies that require numpy, basically this can be used to see where conflicts are happening and a strategy can be developed to resolve.

Package Build Process

There are some important prerequisites to building the EleanorAI Framework Python package:

  • Ensure that src/ is in the PYTHONPATH
  • Python build dependencies: pip install requirements/build/build.txt
  • Required packages: GNU Make
  • Correct version information in eleanor/version.py, set via bin/bump_EleanorAI_version.py

Building the EleanorAI Framework Python package involves the following high-level steps:

  1. (Re)compile project dependencies via make pkg-req-compile
  2. Execute code quality checks via make pre_commit
  3. Build the EleanorAI Framework wheel
  4. (Optional) Commit version changes back to Git
  5. (Optional) Tag the release in Git, commit

Container Build Process

Since the EleanorAI Framework currently leverages a private Git repository, the SSH deploy key needs to be setup in the Git repository and available in the local build .ssh/ directory.

Some pre-work before building containers:

  • Refresh trust.pem with the environment-specific trust store
  • Re-build FastAPI client

Base Images

Both a GPU-enabled and CPU base images are available to the other images used by the framework. Broadly, these base images provide the following features:

  • Python
  • Git / Git LFS support
  • Updated CA certificates
  • Basic OS dependencies
  • Locale settings

Builder Image

The builder image is intended to get executed as a multistage build step for another container and been designed to be as lightweight as possible. Additionally, no dependencies from the source code itself are included in the container. This is important so that rebuilding the builder container are relatively infrequent which simplifies the build process over time.

The builder image provides the following features:

  • Docker-in-Docker support
  • Make
  • Builder user (non-root)
  • PyTorch
graph LR;
    IMAGE_CUDA["&lt;Image&gt;<br />**nvcr.io/nvidia/cuda:x.x.x**"]@{shape: rounded}
    IMAGE_BUILDER["&lt;Image&gt;<br />**eleanor-builder**"]@{shape: rounded}

    IMAGE_CUDA----IMAGE_BUILDER

Runtime Image

graph LR;
    IMAGE_BUILDER["&lt;Image&gt;<br />**eleanor-builder**"]@{shape: rounded}
    IMAGE_PYTHON["&lt;Image&gt;<br />**python:3.x**"]@{shape: rounded}
    STAGE_RUNTIME_BUILD["&lt;Stage&gt;<br />*Build Package*"]@{shape: rounded}
    STAGE_RUNTIME_FINAL["&lt;Stage&gt;<br />*Final*"]@{shape: rounded}
    IMAGE_RUNTIME["&lt;Image&gt;<br />**eleanor-runtime**"]@{shape: rounded}

    subgraph "Runtime Multistage"
        STAGE_RUNTIME_BUILD----STAGE_RUNTIME_FINAL
    end

    STAGE_RUNTIME_FINAL----IMAGE_RUNTIME
    IMAGE_BUILDER----STAGE_RUNTIME_BUILD
    IMAGE_PYTHON----STAGE_RUNTIME_FINAL

Runtime Image (GPU)

graph LR;
    IMAGE_BUILDER["&lt;Image&gt;<br />**eleanor-builder**"]@{shape: rounded}
    IMAGE_CUDA["&lt;Image&gt;<br />**nvcr.io/nvidia/cuda:x.x.x**"]@{shape: rounded}
    STAGE_RUNTIME_BUILD["&lt;Stage&gt;<br />*Build Package*"]@{shape: rounded}
    STAGE_RUNTIME_FINAL["&lt;Stage&gt;<br />*Final*"]@{shape: rounded}
    IMAGE_RUNTIME["&lt;Image&gt;<br />**eleanor-runtime-gpu**"]@{shape: rounded}

    subgraph "GPU Runtime Multistage"
        STAGE_RUNTIME_BUILD----STAGE_RUNTIME_FINAL
    end

    STAGE_RUNTIME_FINAL----IMAGE_RUNTIME
    IMAGE_BUILDER----STAGE_RUNTIME_BUILD
    IMAGE_CUDA----STAGE_RUNTIME_FINAL

Developer Image

graph LR;
    IMAGE_CUDA["&lt;Image&gt;<br />**nvcr.io/nvidia/cuda:x.x.x**"]@{shape: rounded}
    IMAGE_DEV["&lt;Image&gt;<br />**eleanor-dev**"]@{shape: rounded}

    IMAGE_CUDA----IMAGE_DEV