gov.sandia.cognition.learning.experiment

Class LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>

java.lang.Object

gov.sandia.cognition.util.AbstractCloneableSerializable

gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment<InputDataType,FoldDataType>

gov.sandia.cognition.learning.experiment.LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>

Type Parameters:

InputDataType - The type of the data to perform the experiment with. This will be passed to the fold creator to create a number of folds on which to validate the performance of the learning algorithms.

FoldDataType - The type of data created by the fold creator that will go into the learning algorithms. Typically, this is the same as the InputDataType, but it does not need to be. It just needs to match the output of the fold creator and the input of the learning algorithms.

LearnedType - The type of the output produced by the learning algorithms whose performance will be evaluated on each fold of data.

StatisticType - The type of the statistic generated by the performance evaluator on each learned objects for each fold. It is created by passing the learned object plus the test data for the fold into the performance evaluator.

SummaryType - The type produced by the summarizer at the end of the experiment from a collection of the given statistics (one for each fold). This represents the performance result for the comparison of the performance of the learning algorithms for the whole experiment.

All Implemented Interfaces:

LearningExperiment, CloneableSerializable, java.io.Serializable, java.lang.Cloneable

Direct Known Subclasses:

SupervisedLearnerComparisonExperiment

public class LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>
extends AbstractValidationFoldExperiment<InputDataType,FoldDataType>
implements java.io.Serializable

The LearnerComparisonExperiment compares the performance of two machine learning algorithms to determine (using a statistical test) if the two algorithms have significantly different performance.

Since:

2.0

Author:

Justin Basilico

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LearnerComparisonExperiment.Result<SummaryType> Encapsulates the results of the comparison experiment.

Field Summary

Fields

Modifier and Type	Field and Description
protected ConfidenceStatistic	confidence The confidence statistic generated from the underlying performance statistics.
protected Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>>	learners The learners that the experiment is being performed on.
protected PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType>	performanceEvaluator The evaluator to use to compute the performance of the learned object on each fold.
protected DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>>	statistics The performance evaluations made during the experiment.
protected DefaultPair<SummaryType,SummaryType>	summaries The summaries of performance.
protected Summarizer<? super StatisticType,? extends SummaryType>	summarizer The summarizer for summarizing the result of the performance evaluator from all the folds.

Fields inherited from class gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment

foldCreator, numTrials

Fields inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

listeners

Constructor Summary

Constructors

Constructor and Description
LearnerComparisonExperiment() Creates a new instance of LearnerComparisonExperiment.
LearnerComparisonExperiment(ValidationFoldCreator<InputDataType,FoldDataType> foldCreator, PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator, NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>> statisticalTest, Summarizer<? super StatisticType,? extends SummaryType> summarizer) Creates a new instance of LearnerComparisonExperiment.

Method Summary

Modifier and Type	Method and Description
LearnerComparisonExperiment.Result<SummaryType>	evaluate(BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType> learner1, BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType> learner2, java.util.Collection<? extends InputDataType> data) Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.
LearnerComparisonExperiment.Result<SummaryType>	evaluate(Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> learners, java.util.Collection<? extends InputDataType> data) Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.
ConfidenceStatistic	getConfidence() Gets the confidence statistic that the two learners are different.
Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>>	getLearners() Gets the learners the experiment is being run on.
PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType>	getPerformanceEvaluator() Gets the performance evaluator to apply to each fold.
NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>>	getStatisticalTest() Gets the statistical test to use to determine if the two learners are significantly different.
DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>>	getStatistics() Gets the performance evaluations for the trials of the experiment.
DefaultPair<SummaryType,SummaryType>	getSummaries() Gets the summaries of the experiment.
Summarizer<? super StatisticType,? extends SummaryType>	getSummarizer() Gets the summarizer of the performance evaluations.
protected void	runTrial(PartitionedDataset<FoldDataType> fold) Runs a single trial of the experiment on one fold of the data.
protected void	setConfidence(ConfidenceStatistic confidence) Sets the confidence statistic that the two learners are different.
protected void	setLearners(Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> learners) Sets the learners the experiment is being run on.
void	setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator) Sets the performance evaluator to apply to each fold.
void	setStatisticalTest(NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>> statisticalTest) Sets the statistical test to use to determine if the two learners are significantly different.
protected void	setStatistics(DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>> statistics) Sets the performance evaluations for the trials of the experiment.
protected void	setSummaries(DefaultPair<SummaryType,SummaryType> summaries) Sets the summaries of the experiment.
void	setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer) Sets the summarizer of the performance evaluations.

Methods inherited from class gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment

getFoldCreator, getNumTrials, runExperiment, setFoldCreator, setNumTrials

Methods inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

addListener, fireExperimentEnded, fireExperimentStarted, fireTrialEnded, fireTrialStarted, getListeners, removeListener, setListeners

Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable

clone

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

performanceEvaluator

protected PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator

The evaluator to use to compute the performance of the learned object on each fold.

summarizer

protected Summarizer<? super StatisticType,? extends SummaryType> summarizer

The summarizer for summarizing the result of the performance evaluator from all the folds.

learners

protected Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> learners

The learners that the experiment is being performed on.

statistics

protected DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>> statistics

The performance evaluations made during the experiment.

confidence

protected ConfidenceStatistic confidence

The confidence statistic generated from the underlying performance statistics.

summaries

protected DefaultPair<SummaryType,SummaryType> summaries

The summaries of performance.

Constructor Detail

LearnerComparisonExperiment

public LearnerComparisonExperiment()

Creates a new instance of LearnerComparisonExperiment.

LearnerComparisonExperiment

public LearnerComparisonExperiment(ValidationFoldCreator<InputDataType,FoldDataType> foldCreator,
                                   PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator,
                                   NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>> statisticalTest,
                                   Summarizer<? super StatisticType,? extends SummaryType> summarizer)

Creates a new instance of LearnerComparisonExperiment.

Parameters:

foldCreator - The object to use for creating the folds.

performanceEvaluator - The evaluator to use to compute the performance of the learned object on each fold.

statisticalTest - The statistical test to apply to the performance results of the two learners to determine if they are statistically different.

summarizer - The summarizer for summarizing the result of the performance evaluator from all the folds.

Method Detail

evaluate

public LearnerComparisonExperiment.Result<SummaryType> evaluate(Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> learners,
                                                                java.util.Collection<? extends InputDataType> data)

Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.

Parameters:

learners - The two learners.

data - The data to use.

Returns:

The experimental results.

runTrial

protected void runTrial(PartitionedDataset<FoldDataType> fold)

Runs a single trial of the experiment on one fold of the data.

Specified by:

runTrial in class AbstractValidationFoldExperiment<InputDataType,FoldDataType>

Parameters:

fold - The fold to run the trial of the experiment on.

evaluate

public LearnerComparisonExperiment.Result<SummaryType> evaluate(BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType> learner1,
                                                                BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType> learner2,
                                                                java.util.Collection<? extends InputDataType> data)

Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.

Parameters:

learner1 - The first learner.

learner2 - The second learner.

data - The data to use.

Returns:

The experimental results.

getPerformanceEvaluator

public PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> getPerformanceEvaluator()

Gets the performance evaluator to apply to each fold.

Returns:

The performance evaluator to apply to each fold.

setPerformanceEvaluator

public void setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator)

Sets the performance evaluator to apply to each fold.

Parameters:

performanceEvaluator - The performance evaluator to apply to each fold.

getSummarizer

public Summarizer<? super StatisticType,? extends SummaryType> getSummarizer()

Gets the summarizer of the performance evaluations.

Returns:

The summarizer of the performance evaluations.

setSummarizer

public void setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer)

Sets the summarizer of the performance evaluations.

Parameters:

summarizer - The summarizer of the performance evaluations.

getStatisticalTest

public NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>> getStatisticalTest()

Gets the statistical test to use to determine if the two learners are significantly different.

Returns:

The statistical test.

setStatisticalTest

public void setStatisticalTest(NullHypothesisEvaluator<java.util.Collection<? extends StatisticType>> statisticalTest)

Sets the statistical test to use to determine if the two learners are significantly different.

Parameters:

statisticalTest - The statistical test.

getLearners

public Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> getLearners()

Gets the learners the experiment is being run on.

Returns:

The learners.

setLearners

protected void setLearners(Pair<BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super java.util.Collection<? extends FoldDataType>,? extends LearnedType>> learners)

Sets the learners the experiment is being run on.

Parameters:

learners - The learners.

getStatistics

public DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>> getStatistics()

Gets the performance evaluations for the trials of the experiment.

Returns:

The performance evaluations for the trials of the experiment.

setStatistics

protected void setStatistics(DefaultPair<java.util.ArrayList<StatisticType>,java.util.ArrayList<StatisticType>> statistics)

Sets the performance evaluations for the trials of the experiment.

Parameters:

statistics - The performance evaluations for the trials of the experiment.

getConfidence

public ConfidenceStatistic getConfidence()

Gets the confidence statistic that the two learners are different.

Returns:

The confidence statistic of the experiment.

setConfidence

protected void setConfidence(ConfidenceStatistic confidence)

Sets the confidence statistic that the two learners are different.

Parameters:

confidence - The confidence statistic of the experiment.

getSummaries

public DefaultPair<SummaryType,SummaryType> getSummaries()

Gets the summaries of the experiment.

Returns:

The summaries of the experiment.

setSummaries

protected void setSummaries(DefaultPair<SummaryType,SummaryType> summaries)

Sets the summaries of the experiment.

Parameters:

summaries - The summaries of the experiment.