1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 7313487107622865920

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 0.496 ], [ 0.08 ] ],
    	[ [ -0.128 ], [ 0.7 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.496 ], [ 0.08 ] ],
    	[ [ -0.128 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.496 ] ],
    	[ [ -0.128 ], [ 0.08 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.08 ] ],
    	[ [ 0.7 ], [ 0.496 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.41 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4292753303081
Reset training subject: 4292805223287
Constructing line search parameters: GD
th(0)=0.12496960000000001;dx=-0.024993920000000003
New Minimum: 0.12496960000000001 > 0.07692240671112885
END: th(2.154434690031884)=0.07692240671112885; dx=-0.01960914317121183 evalInputDelta=0.048047193288871165
Fitness changed from 0.12496960000000001 to 0.07692240671112885
Iteration 1 complete. Error: 0.07692240671112885 Total: 0.1374; Orientation: 0.0043; Line Search: 0.0483
th(0)=0.07692240671112885;dx=-0.01538448134222577
New Minimum: 0.07692240671112885 > 0.02208640004804931
END: th(4.641588833612779)=0.02208640004804931; dx=-0.008243637661325841 evalInputDelta=0.05483600666307954
Fitness changed from 0.07692240671112885 to 0.02208640004804931
Iteration 2 complete. Error: 0.02208640004804931 Total: 0.0643; Orientation: 0.0013; Line Search: 0.0413
th(0)=0.02208640004804931;dx=-0.004417280009609862
New Minimum: 0.02208640004804931 > 1.0014835710813627E-33
WOLF (strong): th(10.000000000000002)=1.0014835710813627E-33; dx=8.251877414928983E-19 evalInputDelta=0.02208640004804931
END: th(5.000000000000001)=0.005521600012012325; dx=-0.0022086400048049308 evalInputDelta=0.016564800036036983
Fitness changed from 0.02208640004804931 to 1.0014835710813627E-33
Iteration 3 complete. Error: 1.0014835710813627E-33 Total: 0.0829; Orientation: 0.0013; Line Search: 0.0653
Zero gradient: 1.415262216750919E-17
th(0)=1.0014835710813627E-33;dx=-2.0029671421627258E-34
New Minimum: 1.0014835710813627E-33 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=1.0014835710813627E-33
Fitness changed from 1.0014835710813627E-33 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0557; Orientation: 0.0014; Line Search: 0.0362
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0555; Orientation: 0.0014; Line Search: 0.0378
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.397s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.43 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4293157048538
Reset training subject: 4293175706706
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.12496960000000001}, derivative=-0.024993920000000003}
New Minimum: 0.12496960000000001 > 0.1249695999975006
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.1249695999975006}, derivative=-0.024993919999750064}, evalInputDelta = -2.4994173397629993E-12
New Minimum: 0.1249695999975006 > 0.12496959998250423
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.12496959998250423}, derivative=-0.024993919998250427}, evalInputDelta = -1.7495782600462917E-11
New Minimum: 0.12496959998250423 > 0.1249695998775298
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.1249695998775298}, derivative=-0.02499391998775298}, evalInputDelta = -1.2247021452527207E-10
New Minimum: 0.1249695998775298 > 0.12496959914270853
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.12496959914270853}, derivative=-0.024993919914270857}, evalInputDelta = -8.572914877991167E-10
New Minimum: 0.12496959914270853 > 0.12496959399895988
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.12496959399895988}, derivative=-0.02499391939989598}, evalInputDelta = -6.001040137038061E-9
New Minimum: 0.12496959399895988 > 0.12496955799272218
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.12496955799272218}, derivative=-0.024993915799271866}, evalInputDelta = -4.200727783676417E-8
New Minimum: 0.12496955799272218 > 0.12496930594920355
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.12496930594920355}, derivative=-0.024993890594903062}, evalInputDelta = -2.9405079646216414E-7
New Minimum: 0.12496930594920355 > 0.12496754165168986
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.12496754165168986}, derivative=-0.024993714164321414}, evalInputDelta = -2.0583483101577205E-6
New Minimum: 0.12496754165168986 > 0.1249551919178096
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.1249551919178096}, derivative=-0.0249924791502499}, evalInputDelta = -1.4408082190414673E-5
New Minimum: 0.1249551919178096 > 0.12486876086771263
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.12486876086771263}, derivative=-0.024983834051749306}, evalInputDelta = -1.0083913228738084E-4
New Minimum: 0.12486876086771263 > 0.12426458078321163
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.12426458078321163}, derivative=-0.024923318362245143}, evalInputDelta = -7.050192167883801E-4
New Minimum: 0.12426458078321163 > 0.12007634623441103
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.12007634623441103}, derivative=-0.024499708535715977}, evalInputDelta = -0.004893253765588981
New Minimum: 0.12007634623441103 > 0.09276898048543189
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.09276898048543189}, derivative=-0.021534439750011822}, evalInputDelta = -0.032200619514568124
New Minimum: 0.09276898048543189 > 1.2094878119793442E-4
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=1.2094878119793442E-4}, derivative=-7.775582500827418E-4}, evalInputDelta = -0.12484865121880208
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=4.178257625243946}, derivative=0.1445206122494208}, evalInputDelta = 4.053288025243946
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.028588204469310353}, derivative=-0.0119543405961984}, evalInputDelta = -0.09638139553068967
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=0.8789045685928801}, derivative=0.06628313582661122}, evalInputDelta = 0.7539349685928801
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=0.06461864301419842}, derivative=-0.0179726080133376}, evalInputDelta = -0.06035095698580159
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=0.1167237420939314}, derivative=0.024155263906636807}, evalInputDelta = -0.008245857906068618
0.1167237420939314 <= 0.12496960000000001
New Minimum: 1.2094878119793442E-4 > 3.851859888774472E-35
F(10.0) = LineSearchPoint{point=PointSample{avg=3.851859888774472E-35}, derivative=1.154631945610163E-19}, evalInputDelta = -0.12496960000000001
Right bracket at 10.0
Converged to right
Fitness changed from 0.12496960000000001 to 3.851859888774472E-35
Iteration 1 complete. Error: 3.851859888774472E-35 Total: 0.3783; Orientation: 0.0012; Line Search: 0.3262
Zero gradient: 2.7755575615628915E-18
F(0.0) = LineSearchPoint{point=PointSample{avg=3.851859888774472E-35}, derivative=-7.703719777548945E-36}
New Minimum: 3.851859888774472E-35 > 0.0
F(10.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -3.851859888774472E-35
0.0 <= 3.851859888774472E-35
Converged to right
Fitness changed from 3.851859888774472E-35 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0340; Orientation: 0.0008; Line Search: 0.0226
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0163; Orientation: 0.0006; Line Search: 0.0081
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.428s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.29 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4293590178078
Reset training subject: 4293598667978
Adding measurement be479ab to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.12496960000000001 < 0.12496960000000001. Total: 1
th(0)=0.12496960000000001;dx=-0.024993920000000003
Adding measurement 7674de47 to history. Total: 1
New Minimum: 0.12496960000000001 > 0.07692240671112885
END: th(2.154434690031884)=0.07692240671112885; dx=-0.01960914317121183 evalInputDelta=0.048047193288871165
Fitness changed from 0.12496960000000001 to 0.07692240671112885
Iteration 1 complete. Error: 0.07692240671112885 Total: 0.0498; Orientation: 0.0035; Line Search: 0.0226
Non-optimal measurement 0.07692240671112885 < 0.07692240671112885. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.07692240671112885 < 0.07692240671112885. Total: 2
th(0)=0.07692240671112885;dx=-0.01538448134222577
Adding measurement 589ab7d3 to history. Total: 2
New Minimum: 0.07692240671112885 > 0.02208640004804931
END: th(4.641588833612779)=0.02208640004804931; dx=-0.008243637661325843 evalInputDelta=0.05483600666307954
Fitness changed from 0.07692240671112885 to 0.02208640004804931
Iteration 2 complete. Error: 0.02208640004804931 Total: 0.0345; Orientation: 0.0019; Line Search: 0.0245
Non-optimal measurement 0.02208640004804931 < 0.02208640004804931. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.02208640004804931 < 0.02208640004804931. Total: 3
th(0)=0.02208640004804931;dx=-0.004417280009609862
Adding measurement 52833d17 to history. Total: 3
New Minimum: 0.02208640004804931 > 1.0014835710813627E-33
WOLF (strong): th(10.000000000000002)=1.0014835710813627E-33; dx=8.251877414928983E-19 evalInputDelta=0.02208640004804931
Non-optimal measurement 0.005521600012012325 < 1.0014835710813627E-33. Total: 4
END: th(5.000000000000001)=0.005521600012012325; dx=-0.0022086400048049308 evalInputDelta=0.016564800036036983
Fitness changed from 0.02208640004804931 to 1.0014835710813627E-33
Iteration 3 complete. Error: 1.0014835710813627E-33 Total: 0.0437; Orientation: 0.0016; Line Search: 0.0293
Non-optimal measurement 1.0014835710813627E-33 < 1.0014835710813627E-33. Total: 4
Rejected: LBFGS Orientation magnitude: 1.415e-16, gradient 1.415e-17, dot -1.000; [24085b69-10b4-4ab6-9129-fbba99361cb5 = 1.000/1.000e+00, 62d35645-0373-4ed7-9d46-a62abc00026d = 1.000/1.000e+00, b030d946-7eb7-4252-8784-38d7141cd350 = 1.000/1.000e+00, 902999d0-b001-4038-aac8-90e47c76bcdb = 1.000/1.000e+00, 61e3b7e0-c3f1-4ee9-a903-877046ae3d88 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.0014835710813627E-33, 0.02208640004804931, 0.07692240671112885, 0.12496960000000001
LBFGS Accumulation History: 3 points
Removed measurement 52833d17 to history. Total: 3
Adding measurement 920ab82 to history. Total: 3
th(0)=1.0014835710813627E-33;dx=-2.0029671421627258E-34
Adding measurement 1ebedc30 to history. Total: 4
New Minimum: 1.0014835710813627E-33 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=1.0014835710813627E-33
Fitness changed from 1.0014835710813627E-33 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0674; Orientation: 0.0382; Line Search: 0.0206
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [24085b69-10b4-4ab6-9129-fbba99361cb5 = 0.000e+00, 61e3b7e0-c3f1-4ee9-a903-877046ae3d88 = 0.000e+00, 62d35645-0373-4ed7-9d46-a62abc00026d = 0.000e+00, b030d946-7eb7-4252-8784-38d7141cd350 = 0.000e+00, 902999d0-b001-4038-aac8-90e47c76bcdb = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.0014835710813627E-33, 0.02208640004804931, 0.07692240671112885, 0.12496960000000001
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [902999d0-b001-4038-aac8-90e47c76bcdb = 0.000e+00, 61e3b7e0-c3f1-4ee9-a903-877046ae3d88 = 0.000e+00, 24085b69-10b4-4ab6-9129-fbba99361cb5 = 0.000e+00, 62d35645-0373-4ed7-9d46-a62abc00026d = 0.000e+00, b030d946-7eb7-4252-8784-38d7141cd350 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.0014835710813627E-33, 0.02208640004804931, 0.07692240671112885
LBFGS Accumulation History: 3 points
Removed measurement 1ebedc30 to history. Total: 4
Removed measurement 920ab82 to history. Total: 3
Adding measurement 4c09ee2d to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0944; Orientation: 0.0622; Line Search: 0.0219
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.290s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.14 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -34.41432951870191], [4.0, -1.1139471362248807]; valueStats=DoubleSummaryStatistics{count=7, sum=0.198018, min=0.000000, average=0.028288, max=0.076922}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -34.41432951870191], [0.202, -1.1139471362248807]; valueStats=DoubleSummaryStatistics{count=7, sum=0.198018, min=0.000000, average=0.028288, max=0.076922}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.069",
      "gc_time": "0.272"
    },
    "created_on": 1586738924378,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Bottom",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayerTest.Bottom",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer/Bottom/trainingTest/202004134844",
    "id": "6caa9172-5f1c-4c5e-9ac9-5964450378e8",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPaddingLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer.java",
      "javaDoc": ""
    }
  }