1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 7331551951232087040

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 0.08 ], [ -0.128 ] ],
    	[ [ 0.496 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.496 ], [ -0.128 ] ],
    	[ [ 0.08 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.496 ] ],
    	[ [ -0.128 ], [ 0.08 ] ]
    ]
    [
    	[ [ 0.7 ], [ -0.128 ] ],
    	[ [ 0.08 ], [ 0.496 ] ]
    ]
    [
    	[ [ 0.7 ], [ -0.128 ] ],
    	[ [ 0.08 ], [ 0.496 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.42 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4326609687904
Reset training subject: 4326663411105
Constructing line search parameters: GD
th(0)=0.14177599999999999;dx=-0.028355200000000004
New Minimum: 0.14177599999999999 > 0.08726723246195077
END: th(2.154434690031884)=0.08726723246195077; dx=-0.02224625734772079 evalInputDelta=0.054508767538049216
Fitness changed from 0.14177599999999999 to 0.08726723246195077
Iteration 1 complete. Error: 0.08726723246195077 Total: 0.1573; Orientation: 0.0046; Line Search: 0.0622
th(0)=0.08726723246195077;dx=-0.017453446492390154
New Minimum: 0.08726723246195077 > 0.025056665406724826
END: th(4.641588833612779)=0.025056665406724826; dx=-0.009352274257676527 evalInputDelta=0.06221056705522594
Fitness changed from 0.08726723246195077 to 0.025056665406724826
Iteration 2 complete. Error: 0.025056665406724826 Total: 0.0660; Orientation: 0.0015; Line Search: 0.0498
th(0)=0.025056665406724826;dx=-0.005011333081344965
New Minimum: 0.025056665406724826 > 5.585196838722984E-34
WOLF (strong): th(10.000000000000002)=5.585196838722984E-34; dx=5.211097926339485E-19 evalInputDelta=0.025056665406724826
END: th(5.000000000000001)=0.006264166351681205; dx=-0.0025056665406724826 evalInputDelta=0.01879249905504362
Fitness changed from 0.025056665406724826 to 5.585196838722984E-34
Iteration 3 complete. Error: 5.585196838722984E-34 Total: 0.0849; Orientation: 0.0018; Line Search: 0.0658
Zero gradient: 1.056900831556394E-17
th(0)=5.585196838722984E-34;dx=-1.117039367744597E-34
New Minimum: 5.585196838722984E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=5.585196838722984E-34
Fitness changed from 5.585196838722984E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0515; Orientation: 0.0014; Line Search: 0.0366
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0495; Orientation: 0.0013; Line Search: 0.0325
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.410s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.48 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4327026645795
Reset training subject: 4327042254966
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.14177599999999999}, derivative=-0.028355200000000004}
New Minimum: 0.14177599999999999 > 0.14177599999716448
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.14177599999716448}, derivative=-0.02835519999971645}, evalInputDelta = -2.83550960489265E-12
New Minimum: 0.14177599999716448 > 0.14177599998015136
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.14177599998015136}, derivative=-0.028355199998015137}, evalInputDelta = -1.984862274539978E-11
New Minimum: 0.14177599998015136 > 0.14177599986105952
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.14177599986105952}, derivative=-0.02835519998610595}, evalInputDelta = -1.3894047024010092E-10
New Minimum: 0.14177599986105952 > 0.14177599902741664
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.14177599902741664}, derivative=-0.028355199902741664}, evalInputDelta = -9.725833471918577E-10
New Minimum: 0.14177599902741664 > 0.14177599319191656
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.14177599319191656}, derivative=-0.02835519931919165}, evalInputDelta = -6.808083430343004E-9
New Minimum: 0.14177599319191656 > 0.14177595234341936
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.14177595234341936}, derivative=-0.028355195234341535}, evalInputDelta = -4.76565806262208E-8
New Minimum: 0.14177595234341936 > 0.14177566640410375
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.14177566640410375}, derivative=-0.028355166640390755}, evalInputDelta = -3.3359589624026853E-7
New Minimum: 0.14177566640410375 > 0.14177366483696824
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.14177366483696824}, derivative=-0.028354966482735267}, evalInputDelta = -2.3351630317469674E-6
New Minimum: 0.14177366483696824 > 0.14175965426263165
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.14175965426263165}, derivative=-0.02835356537914685}, evalInputDelta = -1.6345737368334E-5
New Minimum: 0.14175965426263165 > 0.14166159962727598
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.14166159962727598}, derivative=-0.02834375765402794}, evalInputDelta = -1.1440037272400505E-4
New Minimum: 0.14166159962727598 > 0.14097616704479016
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.14097616704479016}, derivative=-0.028275103578195557}, evalInputDelta = -7.998329552098282E-4
New Minimum: 0.14097616704479016 > 0.13622468235258706
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.13622468235258706}, derivative=-0.027794525047368866}, evalInputDelta = -0.005551317647412929
New Minimum: 0.13622468235258706 > 0.1052449153818416
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.1052449153818416}, derivative=-0.024430475331582047}, evalInputDelta = -0.03653108461815839
New Minimum: 0.1052449153818416 > 1.3721444577816005E-4
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=1.3721444577816005E-4}, derivative=-8.821273210743315E-4}, evalInputDelta = -0.14163878555422182
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=4.74016603299191}, derivative=0.16395630875247968}, evalInputDelta = 4.59839003299191
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.03243285788576537}, derivative=-0.013562007019040024}, evalInputDelta = -0.10934314211423461
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=0.9971030884056933}, derivative=0.07519715086671984}, evalInputDelta = 0.8553270884056933
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=0.07330881055857581}, derivative=-0.020389634548713856}, evalInputDelta = -0.06846718944142417
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=0.13242120691039433}, derivative=0.027403758159002988}, evalInputDelta = -0.009354793089605656
0.13242120691039433 <= 0.14177599999999999
New Minimum: 1.3721444577816005E-4 > 9.629649721936179E-35
F(10.0) = LineSearchPoint{point=PointSample{avg=9.629649721936179E-35}, derivative=2.0039525594484078E-19}, evalInputDelta = -0.14177599999999999
Right bracket at 10.0
Converged to right
Fitness changed from 0.14177599999999999 to 9.629649721936179E-35
Iteration 1 complete. Error: 9.629649721936179E-35 Total: 0.4109; Orientation: 0.0010; Line Search: 0.3589
Zero gradient: 4.388541835720877E-18
F(0.0) = LineSearchPoint{point=PointSample{avg=9.629649721936179E-35}, derivative=-1.9259299443872365E-35}
New Minimum: 9.629649721936179E-35 > 0.0
F(10.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -9.629649721936179E-35
0.0 <= 9.629649721936179E-35
Converged to right
Fitness changed from 9.629649721936179E-35 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0444; Orientation: 0.0011; Line Search: 0.0307
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0262; Orientation: 0.0008; Line Search: 0.0133
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.482s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.29 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4327513274291
Reset training subject: 4327522517468
Adding measurement 36ee3336 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.14177599999999999 < 0.14177599999999999. Total: 1
th(0)=0.14177599999999999;dx=-0.028355200000000004
Adding measurement 122ee5b4 to history. Total: 1
New Minimum: 0.14177599999999999 > 0.08726723246195077
END: th(2.154434690031884)=0.08726723246195077; dx=-0.02224625734772079 evalInputDelta=0.054508767538049216
Fitness changed from 0.14177599999999999 to 0.08726723246195077
Iteration 1 complete. Error: 0.08726723246195077 Total: 0.0516; Orientation: 0.0035; Line Search: 0.0224
Non-optimal measurement 0.08726723246195077 < 0.08726723246195077. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.08726723246195077 < 0.08726723246195077. Total: 2
th(0)=0.08726723246195077;dx=-0.017453446492390154
Adding measurement 7a527b8 to history. Total: 2
New Minimum: 0.08726723246195077 > 0.025056665406724826
END: th(4.641588833612779)=0.025056665406724826; dx=-0.009352274257676527 evalInputDelta=0.06221056705522594
Fitness changed from 0.08726723246195077 to 0.025056665406724826
Iteration 2 complete. Error: 0.025056665406724826 Total: 0.0340; Orientation: 0.0015; Line Search: 0.0243
Non-optimal measurement 0.025056665406724826 < 0.025056665406724826. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.025056665406724826 < 0.025056665406724826. Total: 3
th(0)=0.025056665406724826;dx=-0.005011333081344965
Adding measurement 6ec24fe3 to history. Total: 3
New Minimum: 0.025056665406724826 > 5.585196838722984E-34
WOLF (strong): th(10.000000000000002)=5.585196838722984E-34; dx=5.211097926339485E-19 evalInputDelta=0.025056665406724826
Non-optimal measurement 0.006264166351681205 < 5.585196838722984E-34. Total: 4
END: th(5.000000000000001)=0.006264166351681205; dx=-0.0025056665406724826 evalInputDelta=0.01879249905504362
Fitness changed from 0.025056665406724826 to 5.585196838722984E-34
Iteration 3 complete. Error: 5.585196838722984E-34 Total: 0.0429; Orientation: 0.0020; Line Search: 0.0298
Non-optimal measurement 5.585196838722984E-34 < 5.585196838722984E-34. Total: 4
Rejected: LBFGS Orientation magnitude: 1.057e-16, gradient 1.057e-17, dot -1.000; [a478c63a-b7e8-4884-b416-19b5e536b622 = 0.000e+00, 29c6c741-832f-4677-b132-8cf65a0048d1 = 1.000/1.000e+00, 86b6d2a8-8d3c-4e62-8d55-1dc9f38e3f22 = 1.000/1.000e+00, cda7d862-e170-4892-9d08-ec2494c0246a = 0.000e+00, 3f37588f-dbf1-4280-a793-e6c4ede72f61 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 5.585196838722984E-34, 0.025056665406724826, 0.08726723246195077, 0.14177599999999999
LBFGS Accumulation History: 3 points
Removed measurement 6ec24fe3 to history. Total: 3
Adding measurement 5a1d3023 to history. Total: 3
th(0)=5.585196838722984E-34;dx=-1.117039367744597E-34
Adding measurement 75a735a1 to history. Total: 4
New Minimum: 5.585196838722984E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=5.585196838722984E-34
Fitness changed from 5.585196838722984E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0650; Orientation: 0.0323; Line Search: 0.0209
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [a478c63a-b7e8-4884-b416-19b5e536b622 = 0.000e+00, 86b6d2a8-8d3c-4e62-8d55-1dc9f38e3f22 = 0.000e+00, 29c6c741-832f-4677-b132-8cf65a0048d1 = 0.000e+00, 3f37588f-dbf1-4280-a793-e6c4ede72f61 = 0.000e+00, cda7d862-e170-4892-9d08-ec2494c0246a = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 5.585196838722984E-34, 0.025056665406724826, 0.08726723246195077, 0.14177599999999999
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [29c6c741-832f-4677-b132-8cf65a0048d1 = 0.000e+00, cda7d862-e170-4892-9d08-ec2494c0246a = 0.000e+00, a478c63a-b7e8-4884-b416-19b5e536b622 = 0.000e+00, 3f37588f-dbf1-4280-a793-e6c4ede72f61 = 0.000e+00, 86b6d2a8-8d3c-4e62-8d55-1dc9f38e3f22 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 5.585196838722984E-34, 0.025056665406724826, 0.08726723246195077
LBFGS Accumulation History: 3 points
Removed measurement 75a735a1 to history. Total: 4
Removed measurement 5a1d3023 to history. Total: 3
Adding measurement 1e34f143 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0984; Orientation: 0.0706; Line Search: 0.0193
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.292s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.13 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -34.016389510029875], [4.0, -1.0591487967565694]; valueStats=DoubleSummaryStatistics{count=7, sum=0.224648, min=0.000000, average=0.032093, max=0.087267}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -34.016389510029875], [0.202, -1.0591487967565694]; valueStats=DoubleSummaryStatistics{count=7, sum=0.224648, min=0.000000, average=0.032093, max=0.087267}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.143",
      "gc_time": "0.263"
    },
    "created_on": 1586738958230,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Center",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayerTest.Center",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer/Center/trainingTest/202004134918",
    "id": "004e35d8-aa16-408f-afa0-c52fef55465e",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPaddingLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer.java",
      "javaDoc": ""
    }
  }