quantization_settings

Quantization settings module

Classes

GPTQSettings

Bases: BaseModel

GPTQ quantization settings

Attributes:

• hijack_fasterquant (bool) –

  Hijack FasterQuant

• model_kwargs (KwargsModel) –

  Model keyword arguments

• gptq_config_kwargs (KwargsModel) –

  GPTQ configuration

• save_pretrained_kwargs (KwargsModel) –

  Save pretrained model kwargs

Attributes

gptq_config_kwargs class-attribute instance-attribute

gptq_config_kwargs: KwargsModel = Field(
    default_factory=KwargsModel,
    title="GPTQ Config",
    description="GPTQ configuration",
)

hijack_fasterquant class-attribute instance-attribute

hijack_fasterquant: bool = Field(
    default=False,
    title="Hijack FasterQuant",
    description="Hijack FasterQuant",
)

model_config class-attribute instance-attribute

model_config = ConfigDict(protected_namespaces=())

model_kwargs class-attribute instance-attribute

model_kwargs: KwargsModel = Field(
    default_factory=KwargsModel,
    title="Model Kwargs",
    description="Model keyword arguments",
)

save_pretrained_kwargs class-attribute instance-attribute

save_pretrained_kwargs: KwargsModel = Field(
    default_factory=KwargsModel,
    title="Save Pretrained Kwargs",
    description="Save pretrained model kwargs",
)

ModelQuantizationSettings

Bases: BaseModel

Model quantization settings

Attributes:

• hijack_fasterquant (bool) –

  Hijack FasterQuant

• model_kwargs (KwargsModel) –

  Model keyword arguments

• quantization_config (IoCFactoryModel) –

  HuggingFace quantization configuration class

• save_pretrained_kwargs (KwargsModel) –

  Save pretrained model kwargs

• auto_class (str) –

  Pretrained model loader class name

Attributes

auto_class class-attribute instance-attribute

auto_class: str = Field(
    default="transformers.AutoModelForCausalLM",
    title="Auto Class",
    description="Pretrained model loader class name",
)

hijack_fasterquant class-attribute instance-attribute

hijack_fasterquant: bool = Field(
    default=False,
    title="Hijack FasterQuant",
    description="Hijack FasterQuant",
)

model_config class-attribute instance-attribute

model_config = ConfigDict(protected_namespaces=())

model_kwargs class-attribute instance-attribute

model_kwargs: KwargsModel = Field(
    default_factory=KwargsModel,
    title="Model Kwargs",
    description="Model load keyword arguments",
)

quantization_config class-attribute instance-attribute

quantization_config: IoCFactoryModel = Field(
    ...,
    title="Quantization Configuration",
    description="HuggingFace quantization configuration class",
)

save_pretrained_kwargs class-attribute instance-attribute

save_pretrained_kwargs: KwargsModel = Field(
    default_factory=KwargsModel,
    title="Save Pretrained Kwargs",
    description="Save pretrained model kwargs",
)

source class-attribute instance-attribute

source: str = Field(
    ..., title="Source", description="Model source path"
)

QuantizationSettings

Bases: BaseModel

Model quantization configuration settings

Attributes:

• models (Dict[str, ModelQuantizationSettings]) –

  Model quantization settings, indexed by path to quantized destination name

Attributes

models class-attribute instance-attribute

models: Dict[str, ModelQuantizationSettings] = Field(
    default_factory=dict,
    title="Model Settings",
    description="Model quantization settings, indexed by path to quantized destination name",
)

Functions