Skip to content

langfuse

Langfuse integration module

This module contains the Langfuse integration for the Eleanor Framework and includes callback that includes key generation metrics (logprobs and perplexity). Additionally, tis module includes utilities that weill emit the trace URL to the logs making it easier for developers to troubleshoot generation problems.

Warning

Due to the customizations implemented here it is not recommended to use tracing tool integration provided by the LiteLLM proxy.

Danger

As of langchain-openai==0.1.23, the LangChain OpenAI client does not support n generation callbacks properly and a monkeypatch is needed. See: https://github.com/langchain-ai/langchain/issues/19185

Attributes

Classes

EleanorLangfuseCallback

Bases: CallbackHandler

Custom LangFuse callback handler for Eleanor Framework.

Functions

on_llm_end 
on_llm_end(
    response: LLMResult,
    *,
    run_id: UUID,
    parent_run_id: Optional[UUID] = None,
    **kwargs: Any
) -> Any

Handle the ‘on_llm_end’ event.

Update the LangFuse trace with Eleanor Framework specific information.

Parameters:

  • response (LLMResult) –

    The response object containing the LLM result.

  • run_id (UUID) –

    The ID of the current run.

  • parent_run_id (Optional[UUID], default: None ) –

    The ID of the parent run. Defaults to None.

  • **kwargs (Any, default: {} ) –

    Additional keyword arguments.

Returns:

  • Any ( Any ) –

    The result of the event handling.

perplexity staticmethod 
perplexity(logprobs: List[float]) -> float

Calculates the perplexity given a list of log probabilities.

Parameters:

  • logprobs (List[float]) –

    A list of log probabilities.

Returns:

  • float ( float ) –

    The perplexity value.

GenerationMetrics

Bases: BaseModel

Attributes

mean_perplexity class-attribute instance-attribute 
mean_perplexity: float = Field(default=5.0)
mean_tokens class-attribute instance-attribute 
mean_tokens: float = Field(default=0.0)
metrics class-attribute instance-attribute 
metrics: List[NMetrics] = Field(default_factory=list)

NMetrics

Bases: BaseModel

Attributes

finish_reason class-attribute instance-attribute 
finish_reason: str = Field(default='')
logprobs class-attribute instance-attribute 
logprobs: List[Dict[str, float]] = Field(
    default_factory=list
)
perplexity class-attribute instance-attribute 
perplexity: float = Field(default=5.0)
tokens class-attribute instance-attribute 
tokens: int = Field(default=0)

ResponseMetrics

Bases: BaseModel

Attributes

generations class-attribute instance-attribute 
generations: List[GenerationMetrics] = Field(
    default_factory=list
)

Functions

append_trace_tag 

append_trace_tag(
    new_tag: str, tags: List[str] | None
) -> List[str]

Add a new trace tag to the list of tags, if it doesn’t already exist.

Note

This method copies the list of tags before appending the new tag to it and is non-destructive.

Parameters:

  • new_tag (str) –

    The new tag to add.

  • tags (List[str] | None) –

    The list of tags to append to.

Returns:

  • List[str] –

    List[str]: The updated list of tags.

build_langfuse_trace_url 

build_langfuse_trace_url(trace_id: str) -> str | None

Build the URL for retrieving a Langfuse trace based on the given trace ID.

Parameters:

  • trace_id (str) –

    The ID of the trace.

Returns:

  • str ( str | None ) –

    The complete URL for retrieving the Langfuse trace.

new_trace_id 

new_trace_id() -> str

Generates a new trace ID.

Returns:

  • str ( str ) –

    The generated trace ID.

new_trace_session_id 

new_trace_session_id(session_prefix: str) -> str

Generates a new trace session ID

Parameters:

  • session_prefix (str) –

    The prefix to use for the session ID.

Returns:

  • str ( str ) –

    The generated trace session ID.