Package de.uni_trier.wi2.procake.utils.eval

Class RetrieverEvaluation<TCase extends DataObject>

java.lang.Object
de.uni_trier.wi2.procake.utils.eval.RetrieverEvaluation<TCase>
Direct Known Subclasses:
MACFACRetrieverEvaluation
public class RetrieverEvaluation<TCase extends DataObject> extends Object
Can be used to evaluate different retrievers or differently configured retrievers. Hence, some parameters can be specified in order to control the evaluation.
Author:
Maximilian Hoffmann

  • Field Details

    • logger

      protected final org.slf4j.Logger logger

    • retrievers

      protected final Map<String,Retriever<TCase extends DataObject,Query>> retrievers
      The list of retrievers to evaluate.

    • metrics

      protected final List<EvalMetric> metrics
      All metrics that are to computed during the evaluation.

    • trainingObjectPool

      protected TrainingObjectPool<TCase extends DataObject> trainingObjectPool
      The training object pool, that contains training case base, i.e., the case base to retrieve from, and testing case base, i.e., the case base to extract queries from.

    • groundTruthSimilarities

      protected List<SimpleSimilarityResult> groundTruthSimilarities
      The ground-truth similarities are stored as a list of simple similarity results that come from retrievals.

    • k

      protected Integer k
      Specifies the number of cases to inspect beginning with the most similar case. Metrics can use this constant for their computations.

    • metricResults

      protected Map<RetrieverMetricKeyPair,Double> metricResults
      Map containing the retrievers and the metrics as RetrieverMetricKeyPair and the corresponding metric values.

    • similarityResults

      protected Map<CasePair,Collection<RetrieverSimilarityPair>> similarityResults
      Map containing a case pair and a list of predictions from all retrievers for this pair. This data can be used to compare the predictions of all retrievers for each case pair.

    • retrievalTimeResultMap

      protected HashMap<String,List<Double>> retrievalTimeResultMap
      Map containing the names of the retrievers and the times required for each.

    • decimalFormat

      protected DecimalFormat decimalFormat
      Specifies the decimal format for the csv output.

    • trackSimilarityResults

      protected boolean trackSimilarityResults
      Tracks and prints the detailed pairwise similarities computed by every retriever. Enable this if you want to analyze this data later on. Be cautious since this can use a large amount of memory.

  • Constructor Details

  • Method Details

    • performEvaluation

      public Map<RetrieverMetricKeyPair,Double> performEvaluation() throws RetrieverEvaluationException
      Main method for performing the evaluation of the given retrievers according to the specified parameters. Assumes that ProCAKE was already started (CakeInstance.start()).
      Returns:
      the evaluation results as a map with a RetrieverMetricKeyPair as key and a double as value.
      Throws:
      RetrieverEvaluationException - if something goes wrong while evaluating

    • setGroundTruthSimilarities

      public void setGroundTruthSimilarities(List<SimpleSimilarityResult> gtSimilarities)
      Set the ground-truth similarities that are loaded outside of the evaluation.
      Parameters:
      gtSimilarities - list of simple similarity results where each entry represents a single retrieval

    • addRetrieverToEvaluate

      public void addRetrieverToEvaluate(String uniqueRetrieverName, Retriever<TCase,Query> retriever)
      Adds a retriever to evaluate.
      Parameters:
      uniqueRetrieverName - a unique retriever name to identify it in the results
      retriever - the pre-configured retriever

    • addMetricToEvaluate

      public void addMetricToEvaluate(EvalMetric metric)
      Adds a metric to compute during evaluation. Besides these metrics, the avg. retrieval time is always computed.
      Parameters:
      metric - the metric to compute during evaluation

    • setTrainTestCaseBase

      public void setTrainTestCaseBase(WriteableObjectPool<TCase> trainCaseBase, WriteableObjectPool<TCase> testCaseBase)
      Method used to store the training case base and the testing case base in the retriever evaluation, if this has not been done in the constructor. In the background, these are stored in a TrainingObjectPool.
      Parameters:
      trainCaseBase - The training case base, i.e., the case base to retrieve from.
      testCaseBase - The testing case base, i.e., the case base to extract queries from.

    • getK

      public Integer getK()

    • setK

      public void setK(Integer k)

    • setDecimalFormat

      public void setDecimalFormat(DecimalFormat decimalFormat)
      Change the DecimalFormat for a csv output.
      Parameters:
      decimalFormat - The desired decimal format.

    • addRetrieversToEvaluate

      public void addRetrieversToEvaluate(Map<String,Retriever<TCase,Query>> retrieverMap)
      Adds retrievers to evaluate.
      Parameters:
      retrieverMap - A map consisting of a unique retriever name to identify it in the results and the pre-configured retrievers.

    • addMetricsToEvaluate

      public void addMetricsToEvaluate(Collection<EvalMetric> metrics)
      Adds metrics to compute during evaluation. Besides these metrics, the avg. retrieval time is always computed.
      Parameters:
      metrics - Collection of the metrics to compute during evaluation

    • importGroundTruthSimilarities

      public void importGroundTruthSimilarities(String pathGroundTruthSimilarities)
      Loads the ground-truth similarities and tests them.
      Parameters:
      pathGroundTruthSimilarities - The path to load the ground-truth similarities from.

    • writeSimilarityResultsAsCSV

      public String writeSimilarityResultsAsCSV(String exportPathExportResults) throws IOException
      Writes similarity results as CSV file to file system.
      Parameters:
      exportPathExportResults - the path to write the results to
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException - if something goes wrong during export. Ignore if you do not want to export any values.

    • getSimilarityResultsAsCSVString

      public String getSimilarityResultsAsCSVString() throws IOException, RetrieverEvaluationException
      Writes similarity results as CSV to a string.
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException
      RetrieverEvaluationException

    • writeSimilarityResultsAsCSV

      public String writeSimilarityResultsAsCSV(OutputStream outputStream) throws IOException
      Writes similarity results as CSV file to an output stream. The calling method is responsible for closing the stream afterwards!
      Parameters:
      outputStream - the stream to write the results to
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException - if something goes wrong during export. Ignore if you do not want to export any values.

    • writeMetricResultsAsCSV

      public String writeMetricResultsAsCSV(String exportPathExportResults) throws IOException, RetrieverEvaluationException
      Writes metric results as CSV file to file system.
      Parameters:
      exportPathExportResults - the path to write the results to
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException - if something goes wrong during export. Ignore if you do not want to export any values.
      RetrieverEvaluationException - if something goes wrong while evaluating

    • getMetricResultsAsCSVString

      public String getMetricResultsAsCSVString() throws IOException, RetrieverEvaluationException
      Writes metric results as CSV to a string.
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException
      RetrieverEvaluationException

    • writeMetricResultsAsCSV

      public String writeMetricResultsAsCSV(OutputStream outputStream) throws IOException, RetrieverEvaluationException
      Writes metric results as CSV file to an output stream. The calling method is responsible for closing the stream afterwards!
      Parameters:
      outputStream - the stream to write the results to
      Returns:
      the evaluation results as a CSV string
      Throws:
      IOException - if something goes wrong during export. Ignore if you do not want to export any values.
      RetrieverEvaluationException - if something goes wrong while evaluating

    • printMetricResultsAsASCIITable

      public void printMetricResultsAsASCIITable()
      Writes metric results as ASCII table to standard output.

    • trackSimilarityResults

      public void trackSimilarityResults()
      Tracks and prints the detailed pairwise similarities computed by every retriever. Enable this if you want to analyze this data later on. Be cautious since this can use a large amount of memory.

    • getSimilarityResults

      public Map<CasePair,Collection<RetrieverSimilarityPair>> getSimilarityResults()