Log Probability Visualizer
The Log Probability Visulizer App is used to help visualize the per-token log probabilities returned from a language model generation. The primary use case for this tool is to help prompt engineers understand the confidence level of a language model of a given generation.
Usage
When logging traces to LangFuse (internal link), the Eleanor Framework will automatically include token log probabilities in the trace metadata for supported inferencing endpoints.
First, identify the generation you would like to analyze in LangFuse. Next, navigate to the generation operation (step 1 in the diagram). Now, copy the generation metadata to the clipboard (step 2 in the diagram).
Finally, navigate to the Log Probability Visulizer App and past the generation metadata into the Input field.
The app will parse the generation metadata and color-code each token based on the confidence.
Demo
The generation json below is an example of the metadata that can be pasted into the Input field of the Log Probability Visulizer App.
[{"metrics":[{"tokens":1047,"logprobs":[{"**":0},{"Neutral":0},{" Observer":0},{"'s":0},{" Evaluation":0},{"**\n\n":0},{"As":-0.1269280463457108},{" the":0},{" Neutral":0},{" Observer":0},{",":0},{" I":0},{" will":0},{" provide":-0.0001233740040333942},{" an":-0.0003353986830916256},{" objective":0},{" analysis":0},{" of":0},{" each":0},{" response":0},{",":0},{" addressing":-0.0181499607861042},{" previous":0},{" points":0},{" raised":0},{" in":0},{" the":0},{" debate":0},{" and":0},{" evaluating":0},{" the":-8.344646857949556e-07},{" relevance":-0.002475651912391186},{",":0},{" accuracy":0},{",":0},{" completeness":0},{",":0},{" depth":0},{",":0},{" limitations":0},{",":0},{" tim":-0.002475651912391186},{"eliness":0},{",":0},{" and":0},{" missing":0},{" information":0},{"/context":0},{" for":0},{" each":0},{" response":0},{".\n\n":0},{"**":0},{"Response":0},{" ":0},{"1":0},{":":0},{" \"":0},{"George":0},{"\"":0},{"**\n\n":0},{"*":0},{" Re":0},{"levance":0},{":":0},{" I":0},{" agree":0},{" with":0},{" the":0},{" Crit":0},{"ic":0},{" that":0},{" the":0},{" response":0},{" is":-8.344646857949556e-07},{" partially":0},{" relevant":0},{",":0},{" as":-0.1269280463457108},{" it":0},{" mentions":0},{" a":0},{" name":0},{",":0},{" but":0},{" lacks":0},{" specificity":0},{".":0},{" However":0},{",":0},{" I":0},{" also":0},{" acknowledge":-0.0003353986830916256},{" the":0},{" Support":0},{"er":0},{"'s":0},{" point":0},{" that":0},{" it":0},{" shows":-4.541770613286644e-05},{" a":0},{" good":0},{" starting":0},{" point":0},{".\n":-8.344646857949556e-07},{"*":0},{" Accuracy":0},{":":0},{" The":0},{" response":0},{" is":0},{" incomplete":0},{",":0},{" as":0},{" there":0},{" are":0},{" multiple":0},{" individuals":0},{" with":0},{" the":0},{" name":0},{" George":0},{" who":0},{" could":0},{" be":0},{" referred":0},{" to":0},{".":0},{" While":-0.0181500781327486},{" the":-0.002475651912391186},{" Support":0},{"er":0},{" argues":-0.04858732968568802},{" that":0},{" it":-0.002475651912391186},{"'s":0},{" not":0},{" entirely":0},{" inaccurate":0},{",":0},{" I":0},{" believe":0},{" the":-0.1269280463457108},{" lack":0},{" of":0},{" specificity":0},{" makes":0},{" it":0},{" more":-0.009174452163279057},{" misleading":-0.3414012789726257},{" than":0},{" accurate":0},{".\n":0},{"*":0},{" Comple":0},{"teness":0},{":":0},{" The":-0.6931471824645996},{" response":0},{" is":0},{" lacking":0},{",":0},{" as":0},{" it":0},{" does":0},{" not":0},{" provide":0},{" enough":0},{" information":0},{" to":0},{" accurately":0},{" identify":0},{" the":0},{" first":0},{" U":0},{".S":0},{".":0},{" president":0},{".":0},{" The":-0.6931698322296143},{" Support":0},{"er":0},{"'s":0},{" argument":-6.198863957251888e-06},{" that":0},{" it":0},{"'s":-0.0003353986830916256},{" a":0},{" foundation":0},{" for":0},{" further":0},{" clarification":0},{" is":0},{" valid":0},{",":0},{" but":0},{" it":-6.198863957251888e-06},{"'s":0},{" essential":-0.002810221398249269},{" to":0},{" acknowledge":0},{" the":-4.541770613286644e-05},{" response":-2.264974000354414e-06},{"'s":0},{" limitations":0},{".\n":0},{"*":0},{" Depth":0},{":":0},{" The":0},{" response":0},{" is":0},{" shallow":0},{",":0},{" as":0},{" it":0},{" does":0},{" not":0},{" provide":0},{" any":0},{" additional":0},{" context":0},{" or":0},{" details":0},{".":-1.192092824453539e-07},{" However":-0.2395448386669159},{",":0},{" I":-0.0003808011533692479},{" agree":0},{" with":0},{" the":0},{" Support":0},{"er":0},{" that":0},{" it":0},{" demonstrates":0},{" a":0},{" basic":0},{" understanding":0},{" of":0},{" the":0},{" topic":0},{".\n":0},{"*":0},{" Limit":0},{"ations":0},{":":0},{" The":0},{" main":0},{" limitation":0},{" is":0},{" the":0},{" lack":0},{" of":0},{" specificity":0},{",":0},{" which":0},{" makes":0},{" the":-0.6931471824645996},{" response":0},{" open":0},{" to":0},{" interpretation":0},{".":-0.1269280463457108},{" The":-6.198863957251888e-06},{" Support":0},{"er":0},{"'s":0},{" suggestion":0},{" that":0},{" follow":-0.03597624972462654},{"-up":0},{" questions":0},{" or":0},{" clarification":0},{" could":0},{" address":0},{" this":0},{" limitation":0},{" is":0},{" a":-0.0181499607861042},{" good":-0.1269280463457108},{" point":0},{".\n":0},{"*":0},{" Tim":0},{"eliness":0},{":":0},{" Not":0},{" applicable":0},{",":0},{" as":0},{" the":0},{" question":0},{" is":0},{" historical":0},{" in":0},{" nature":0},{".\n":0},{"*":0},{" Missing":0},{" information":0},{"/context":0},{":":0},{" The":0},{" response":0},{" lacks":0},{" a":0},{" last":0},{" name":0},{",":0},{" title":0},{",":0},{" or":0},{" any":0},{" other":0},{" identifying":0},{" information":0},{",":-4.553691178443842e-05},{" which":-0.0003353986830916256},{" is":-0.0001233740040333942},{" essential":-2.126928091049194},{" for":-0.002475651912391186},{" accurately":0},{" identifying":-6.198863957251888e-06},{" the":0},{" first":0},{" U":0},{".S":0},{".":0},{" president":0},{".\n\n":0},{"**":0},{"Response":0},{" ":0},{"2":0},{":":0},{" \"":0},{"M":0},{"ickey":0},{" Mouse":0},{"\"":0},{"**\n\n":0},{"*":0},{" Re":0},{"levance":0},{":":0},{" I":0},{" agree":-8.344646857949556e-07},{" with":0},{" the":0},{" Crit":0},{"ic":0},{" that":0},{" the":0},{" response":0},{" is":0},{" not":0},{" relevant":0},{",":0},{" as":0},{" Mickey":0},{" Mouse":0},{" is":0},{" a":0},{" fictional":0},{" character":0},{" and":0},{" not":0},{" a":0},{" historical":0},{" figure":0},{".":0},{" The":-0.6931471824645996},{" Support":0},{"er":0},{"'s":0},{" argument":0},{" that":0},{" it":0},{" shows":0},{" creative":0},{" thinking":0},{" is":0},{" an":0},{" interesting":0},{" perspective":0},{",":0},{" but":0},{" it":0},{" does":-1.098612308502197},{" not":0},{" justify":-0.3132633864879608},{" the":0},{" response":-0.002475651912391186},{"'s":0},{" factual":0},{" in":0},{"accuracy":0},{".\n":0},{"*":0},{" Accuracy":0},{":":0},{" The":0},{" response":0},{" is":0},{" incorrect":-4.018149852752686},{",":0},{" as":0},{" Mickey":0},{" Mouse":0},{" was":0},{" not":0},{" the":0},{" first":0},{" U":0},{".S":0},{".":0},{" president":0},{".":0},{" While":-0.002475651912391186},{" the":0},{" Support":0},{"er":0},{" acknowledges":0},{" this":-0.002475651912391186},{",":0},{" I":0},{" believe":-8.344646857949556e-07},{" it":0},{"'s":0},{" essential":0},{" to":0},{" emphasize":0},{" the":0},{" importance":0},{" of":0},{" factual":0},{" accuracy":0},{" in":0},{" historical":0},{" questions":-6.198863957251888e-06},{".\n":0},{"*":0},{" Comple":0},{"teness":0},{":":0},{" The":0},{" response":0},{" is":0},{" complete":0},{" in":0},{" its":0},{" incorrect":0},{"ness":0},{",":0},{" which":0},{" demonstrates":0},{" a":0},{" level":0},{" of":0},{" confidence":0},{" in":0},{" the":0},{" respondent":0},{"'s":0},{" thought":0},{" process":0},{".":-0.6931471824645996},{" However":0},{",":0},{" this":-0.0003353986830916256},{" confidence":0},{" is":0},{" misplaced":0},{" due":-0.002520957263186574},{" to":0},{" the":0},{" factual":-0.0181499607861042},{" in":0},{"accuracy":0},{".\n":0},{"*":0},{" Depth":0},{":":0},{" The":0},{" response":0},{" is":0},{" shallow":0},{",":0},{" as":0},{" it":0},{" does":0},{" not":0},{" provide":0},{" any":0},{" additional":0},{" context":0},{" or":0},{" details":0},{".":0},{" The":0},{" Support":0},{"er":0},{"'s":0},{" argument":0},{" that":0},{" it":0},{" shows":0},{" a":0},{" willingness":0},{" to":0},{" think":0},{" creatively":0},{" is":0},{" valid":0},{",":0},{" but":0},{" it":0},{"'s":0},{" essential":-8.344646857949556e-07},{" to":0},{" balance":-6.198863957251888e-06},{" creativity":0},{" with":0},{" factual":0},{" accuracy":0},{".\n":0},{"*":0},{" Limit":0},{"ations":0},{":":0},{" The":0},{" main":0},{" limitation":0},{" is":0},{" the":0},{" factual":0},{" in":0},{"accuracy":0},{",":0},{" which":0},{" makes":0},{" the":0},{" response":0},{" uns":-0.0003808011533692479},{"uitable":0},{" as":0},{" a":-6.198863957251888e-06},{" correct":0},{" answer":0},{".":0},{" The":0},{" Support":0},{"er":0},{"'s":0},{" suggestion":0},{" that":0},{" education":0},{" and":0},{" correction":0},{" could":0},{" address":0},{" this":0},{" limitation":0},{" is":0},{" a":0},{" good":0},{" point":0},{".\n":0},{"*":0},{" Tim":0},{"eliness":0},{":":0},{" Not":0},{" applicable":0},{",":0},{" as":0},{" the":0},{" question":0},{" is":0},{" historical":0},{" in":0},{" nature":0},{".\n":0},{"*":0},{" Missing":0},{" information":0},{"/context":0},{":":0},{" The":0},{" response":0},{" lacks":0},{" historical":0},{" context":0},{" and":-4.541770613286644e-05},{" factual":0},{" accuracy":0},{",":0},{" which":-4.541770613286644e-05},{" are":-0.1269287765026093},{" essential":0},{" for":0},{" answering":-1.192092824453539e-07},{" the":0},{" question":0},{" correctly":0},{".\n\n":0},{"**":0},{"Response":0},{" ":0},{"3":0},{":":0},{" \"":0},{"George":0},{" Washington":0},{"\"":0},{"**\n\n":0},{"*":0},{" Re":0},{"levance":0},{":":0},{" I":0},{" agree":0},{" with":0},{" the":-8.344646857949556e-07},{" Crit":0},{"ic":0},{" and":-8.344646857949556e-07},{" Support":-8.344646857949556e-07},{"er":0},{" that":0},{" the":-8.344646857949556e-07},{" response":0},{" is":0},{" highly":0},{" relevant":0},{" and":0},{" accurate":0},{".":0},{" George":-4.541770613286644e-05},{" Washington":0},{" is":0},{" a":0},{" historical":0},{" figure":0},{" closely":0},{" associated":0},{" with":0},{" the":0},{" United":0},{" States":0},{",":-0.0003415954706724733},{" and":0},{" the":-6.198863957251888e-06},{" response":0},{" provides":0},{" a":0},{" clear":0},{" and":0},{" specific":0},{" answer":0},{".\n":0},{"*":0},{" Accuracy":0},{":":0},{" The":0},{" response":0},{" is":0},{" correct":0},{",":0},{" as":0},{" George":0},{" Washington":0},{" is":0},{" widely":0},{" recognized":0},{" as":0},{" the":0},{" first":0},{" U":0},{".S":0},{".":0},{" president":0},{".":-0.002475651912391186},{" The":-0.1269287765026093},{" Support":0},{"er":0},{"'s":0},{" argument":-0.0001710507058305666},{" that":0},{" the":0},{" respondent":0},{" demonstrates":0},{" a":0},{" clear":-4.541770613286644e-05},{" and":-0.002475651912391186},{" concise":-6.198863957251888e-06},{" answer":0},{" is":0},{" well":-4.541770613286644e-05},{"-t":0},{"aken":0},{".\n":0},{"*":0},{" Comple":0},{"teness":0},{":":0},{" The":0},{" response":0},{" is":0},{" complete":0},{",":0},{" as":0},{" it":0},{" provides":0},{" a":0},{" clear":0},{" and":0},{" specific":0},{" answer":0},{".":0},{" The":-1.192092824453539e-07},{" Support":0},{"er":0},{"'s":0},{" suggestion":0},{" that":0},{" the":0},{" respondent":-4.541770613286644e-05},{" could":0},{" explore":0},{" more":0},{" details":0},{" about":0},{" George":0},{" Washington":0},{"'s":0},{" presidency":0},{" is":0},{" a":0},{" good":0},{" point":0},{",":-4.541770613286644e-05},{" but":0},{" the":0},{" response":0},{" is":0},{" sufficient":0},{" as":0},{" a":-6.198863957251888e-06},{" correct":0},{" answer":0},{".\n":0},{"*":0},{" Depth":0},{":":0},{" The":0},{" response":0},{" demonstrates":0},{" a":0},{" moderate":0},{" level":0},{" of":0},{" depth":0},{",":0},{" as":0},{" it":0},{" provides":0},{" a":0},{" basic":0},{" but":0},{" accurate":0},{" answer":0},{".":0},{" The":0},{" Support":0},{"er":0},{"'s":0},{" encouragement":-4.541770613286644e-05},{" to":-0.6931471824645996},{" explore":0},{" more":0},{" details":0},{" is":0},{" a":0},{" good":0},{" suggestion":-6.198863957251888e-06},{" for":-0.1269680559635162},{" further":0},{" discussion":0},{".\n":0},{"*":0},{" Limit":0},{"ations":0},{":":0},{" The":0},{" main":0},{" limitation":0},{" is":0},{" the":0},{" lack":0},{" of":0},{" additional":0},{" context":0},{",":0},{" which":-0.002475651912391186},{" could":0},{" be":0},{" addressed":0},{" with":0},{" further":0},{" discussion":0},{".":0},{" However":-0.0003353986830916256},{",":0},{" the":0},{" response":0},{" is":0},{" sufficient":0},{" as":0},{" a":0},{" correct":0},{" answer":0},{".\n":-1.192092824453539e-07},{"*":0},{" Tim":0},{"eliness":0},{":":0},{" Not":0},{" applicable":0},{",":0},{" as":0},{" the":0},{" question":0},{" is":0},{" historical":0},{" in":0},{" nature":0},{".\n":0},{"*":0},{" Missing":0},{" information":0},{"/context":0},{":":0},{" The":0},{" response":0},{" could":0},{" benefit":0},{" from":0},{" additional":0},{" information":0},{" about":0},{" George":0},{" Washington":0},{"'s":0},{" role":0},{" in":0},{" American":0},{" history":0},{",":0},{" but":0},{" it":0},{"'s":0},{" a":0},{" solid":0},{" foundation":0},{".\n\n":-0.0003808011533692479},{"In":0},{" conclusion":0},{",":0},{" while":-0.1291089206933975},{" all":0},{" responses":0},{" have":0},{" some":0},{" merit":0},{",":0},{" Response":0},{" ":0},{"3":0},{":":0},{" \"":0},{"George":0},{" Washington":0},{"\"":0},{" is":0},{" the":0},{" most":0},{" accurate":0},{" and":0},{" complete":0},{" answer":0},{".":0},{" Response":0},{" ":0},{"1":0},{":":-0.002475651912391186},{" \"":0},{"George":0},{"\"":0},{" shows":0},{" a":0},{" good":0},{" starting":0},{" point":0},{",":-0.0003353986830916256},{" but":0},{" its":-1.192092824453539e-07},{" lack":0},{" of":0},{" specificity":0},{" makes":0},{" it":0},{" incomplete":-0.0181499607861042},{".":0},{" Response":0},{" ":0},{"2":0},{":":0},{" \"":0},{"M":0},{"ickey":0},{" Mouse":0},{"\"":0},{" demonstrates":-6.198863957251888e-06},{" creative":0},{" thinking":0},{",":0},{" but":0},{" its":0},{" factual":0},{" in":0},{"accuracy":0},{" makes":0},{" it":0},{" uns":0},{"uitable":0},{" as":0},{" a":0},{" correct":0},{" answer":0},{".":-4.541770613286644e-05},{" By":0},{" acknowledging":0},{" the":0},{" strengths":0},{" and":0},{" limitations":0},{" of":0},{" each":0},{" response":0},{",":0},{" we":0},{" can":0},{" encourage":0},{" further":0},{" discussion":0},{" and":0},{" exploration":0},{" of":0},{" the":0},{" topic":0},{".\n\n":0},{"Neutral":0},{" Observer":0},{"'s":0},{" score":0},{" summary":0},{":":0},{" Re":0},{"levance":0},{" Scores":0},{":":0},{" [":0},{"6":-0.002475651912391186},{",":0},{" ":0},{"2":0},{",":0},{" ":0},{"10":0},{"],":0},{" Accuracy":0},{" Scores":0},{":":0},{" [":0},{"5":0},{",":0},{" ":0},{"1":0},{",":0},{" ":0},{"10":0},{"],":0},{" Comple":0},{"teness":0},{" Scores":0},{":":0},{" [":0},{"5":0},{",":0},{" ":0},{"4":-4.541770613286644e-05},{",":0},{" ":0},{"9":0},{"],":0},{" Tim":0},{"eliness":0},{" Scores":0},{":":0},{" [":0},{"5":0},{",":0},{" ":0},{"5":0},{",":0},{" ":0},{"5":0},{"],":0},{" Overall":0},{" Scores":0},{":":0},{" [":0},{"6":-4.541770613286644e-05},{",":0},{" ":0},{"2":-0.3132617473602295},{",":0},{" ":0},{"9":0},{"],":0},{" Confidence":0},{" Scores":0},{":":0},{" [":0},{"8":0},{",":0},{" ":0},{"5":-1.192092213386786e-06},{",":0},{" ":0},{"9":0},{"]":0},{"":0}],"perplexity":1.01344354251844}],"mean_tokens":1047,"mean_perplexity":1.01344354251844}]