gov.sandia.cognition.learning.experiment

Class LearnerRepeatExperiment<InputDataType,LearnedType,StatisticType,SummaryType>

java.lang.Object

gov.sandia.cognition.util.AbstractCloneableSerializable

gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

gov.sandia.cognition.learning.experiment.LearnerRepeatExperiment<InputDataType,LearnedType,StatisticType,SummaryType>

Type Parameters:

InputDataType - The type of the data to perform the experiment with. It will be passed to the algorithm.

LearnedType - The type of the output produced by the learning algorithm whose performance will be evaluated on each fold of data.

StatisticType - The type of the statistic generated by the performance evaluator on the learned object for each fold. It is created by passing the learned object plus the test data for the fold into the performance evaluator.

SummaryType - The type produced by the summarizer at the end of the experiment from a collection of the given statistics (one for each fold). This represents the performance result for the learning algorithm for the whole experiment.

All Implemented Interfaces:

LearningExperiment, PerformanceEvaluator<BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType>,PartitionedDataset<? extends InputDataType>,SummaryType>, CloneableSerializable, java.io.Serializable, java.lang.Cloneable

public class LearnerRepeatExperiment<InputDataType,LearnedType,StatisticType,SummaryType>
extends AbstractLearningExperiment
implements PerformanceEvaluator<BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType>,PartitionedDataset<? extends InputDataType>,SummaryType>

Runs an experiment where the same learner is evaluated multiple times on the same data. Useful for cases where there is a randomized algorithm used in conjunction with a known training/test split. Should only be used with stochastic algorithms

Since:

3.1.1

Author:

Justin Basilico

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_NUM_TRIALS The default number of trials is 10.
protected int	numTrials The number of trials to repeat the learning.
protected PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType>	performanceEvaluator The evaluator to use to compute the performance of the learned object on each fold.
protected java.util.ArrayList<StatisticType>	statistics The performance evaluations made during the experiment.
protected Summarizer<? super StatisticType,? extends SummaryType>	summarizer The summarizer for summarizing the result of the performance evaluator from all the folds.
protected SummaryType	summary The summary of the performance evaluations made at the end of the experiment.

Fields inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

listeners

Constructor Summary

Constructors

Constructor and Description
LearnerRepeatExperiment() Creates a new instance of LearnerRepeatExperiment.
LearnerRepeatExperiment(int numTrials, PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> performanceEvaluator, Summarizer<? super StatisticType,? extends SummaryType> summarizer) Creates a new instance of LearnerRepeatExperiment.

Method Summary

Modifier and Type	Method and Description
SummaryType	evaluatePerformance(BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType> learner, PartitionedDataset<? extends InputDataType> data) Performs the experiment.
BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType>	getLearner() Gets the learner the experiment is being run on.
int	getNumTrials() Gets the number of trials in the experiment.
PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType>	getPerformanceEvaluator() Gets the performance evaluator to apply to each fold.
java.util.ArrayList<StatisticType>	getStatistics() Gets the performance evaluations for the trials of the experiment.
Summarizer<? super StatisticType,? extends SummaryType>	getSummarizer() Gets the summarizer of the performance evaluations.
SummaryType	getSummary() Gets the summary of the experiment.
protected void	runExperiment(PartitionedDataset<? extends InputDataType> data) Runs the experiment.
protected void	runTrial(PartitionedDataset<? extends InputDataType> data) Runs one trial in the experiment.
protected void	setLearner(BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType> learner) Sets the learner the experiment is being run on.
void	setNumTrials(int numTrials) Sets the number of trials for the experiment to repeatedly call the learning algorithm.
void	setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> performanceEvaluator) Sets the performance evaluator to apply to each fold.
protected void	setStatistics(java.util.ArrayList<StatisticType> statistics) Sets the performance evaluations for the trials of the experiment.
void	setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer) Sets the summarizer of the performance evaluations.
protected void	setSummary(SummaryType summary) Sets the summary of the experiment.

Methods inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment

addListener, fireExperimentEnded, fireExperimentStarted, fireTrialEnded, fireTrialStarted, getListeners, removeListener, setListeners

Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable

clone

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_NUM_TRIALS

public static final int DEFAULT_NUM_TRIALS

The default number of trials is 10.

See Also:

Constant Field Values

numTrials

protected int numTrials

The number of trials to repeat the learning.

performanceEvaluator

protected PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> performanceEvaluator

The evaluator to use to compute the performance of the learned object on each fold.

summarizer

protected Summarizer<? super StatisticType,? extends SummaryType> summarizer

The summarizer for summarizing the result of the performance evaluator from all the folds.

statistics

protected java.util.ArrayList<StatisticType> statistics

The performance evaluations made during the experiment.

summary

protected SummaryType summary

The summary of the performance evaluations made at the end of the experiment.

Constructor Detail

LearnerRepeatExperiment

public LearnerRepeatExperiment()

Creates a new instance of LearnerRepeatExperiment.

LearnerRepeatExperiment

public LearnerRepeatExperiment(int numTrials,
                               PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> performanceEvaluator,
                               Summarizer<? super StatisticType,? extends SummaryType> summarizer)

Creates a new instance of LearnerRepeatExperiment.

Parameters:

numTrials - The number of repeated trials to run.

performanceEvaluator - The evaluator to use to compute the performance of the learned object on each fold.

summarizer - The summarizer for summarizing the result of the performance evaluator from all the folds.

Method Detail

evaluatePerformance

public SummaryType evaluatePerformance(BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType> learner,
                                       PartitionedDataset<? extends InputDataType> data)

Performs the experiment.

Specified by:

evaluatePerformance in interface PerformanceEvaluator<BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType>,PartitionedDataset<? extends InputDataType>,SummaryType>

Parameters:

data - The data to use.

learner - The learner to perform the experiment on.

Returns:

The summary of the experiment.

runExperiment

protected void runExperiment(PartitionedDataset<? extends InputDataType> data)

Runs the experiment.

Parameters:

data - The data to use.

runTrial

protected void runTrial(PartitionedDataset<? extends InputDataType> data)

Runs one trial in the experiment.

Parameters:

data - The data to use.

getPerformanceEvaluator

public PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> getPerformanceEvaluator()

Gets the performance evaluator to apply to each fold.

Returns:

The performance evaluator to apply to each fold.

setPerformanceEvaluator

public void setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,? super java.util.Collection<? extends InputDataType>,? extends StatisticType> performanceEvaluator)

Sets the performance evaluator to apply to each fold.

Parameters:

performanceEvaluator - The performance evaluator to apply to each fold.

getSummarizer

public Summarizer<? super StatisticType,? extends SummaryType> getSummarizer()

Gets the summarizer of the performance evaluations.

Returns:

The summarizer of the performance evaluations.

setSummarizer

public void setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer)

Sets the summarizer of the performance evaluations.

Parameters:

summarizer - The summarizer of the performance evaluations.

getLearner

public BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType> getLearner()

Gets the learner the experiment is being run on.

Returns:

The learner.

setLearner

protected void setLearner(BatchLearner<? super java.util.Collection<? extends InputDataType>,? extends LearnedType> learner)

Sets the learner the experiment is being run on.

Parameters:

learner - The learner.

getStatistics

public java.util.ArrayList<StatisticType> getStatistics()

Gets the performance evaluations for the trials of the experiment.

Returns:

The performance evaluations for the trials of the experiment.

setStatistics

protected void setStatistics(java.util.ArrayList<StatisticType> statistics)

Sets the performance evaluations for the trials of the experiment.

Parameters:

statistics - The performance evaluations for the trials of the experiment.

getSummary

public SummaryType getSummary()

Gets the summary of the experiment.

Returns:

The summary of the experiment.

setSummary

protected void setSummary(SummaryType summary)

Sets the summary of the experiment.

Parameters:

summary - The summary of the experiment.

getNumTrials

public int getNumTrials()

Description copied from interface: LearningExperiment

Gets the number of trials in the experiment.

Specified by:

getNumTrials in interface LearningExperiment

Returns:

The number of trials in the experiment.

setNumTrials

public void setNumTrials(int numTrials)

Sets the number of trials for the experiment to repeatedly call the learning algorithm.

Parameters:

numTrials - The number of trials. Must be positive.