1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 4180968091543351296

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 0.7 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ -0.128 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.496 ], [ 0.08 ] ],
    	[ [ -0.128 ], [ 0.7 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.496 ] ],
    	[ [ 0.08 ], [ -0.128 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.496 ] ],
    	[ [ 0.08 ], [ -0.128 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.41 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4392738521824
Reset training subject: 4392793116377
Constructing line search parameters: GD
th(0)=0.22772320000000001;dx=-0.045544640000000004
New Minimum: 0.22772320000000001 > 0.14017022226173195
END: th(2.154434690031884)=0.14017022226173195; dx=-0.035732344763898624 evalInputDelta=0.08755297773826806
Fitness changed from 0.22772320000000001 to 0.14017022226173195
Iteration 1 complete. Error: 0.14017022226173195 Total: 0.1511; Orientation: 0.0047; Line Search: 0.0564
th(0)=0.14017022226173195;dx=-0.02803404445234639
New Minimum: 0.14017022226173195 > 0.04024647350573214
END: th(4.641588833612779)=0.04024647350573214; dx=-0.015021793683244862 evalInputDelta=0.09992374875599981
Fitness changed from 0.14017022226173195 to 0.04024647350573214
Iteration 2 complete. Error: 0.04024647350573214 Total: 0.0610; Orientation: 0.0016; Line Search: 0.0418
th(0)=0.04024647350573214;dx=-0.008049294701146428
New Minimum: 0.04024647350573214 > 1.2518544638517033E-33
WOLF (strong): th(10.000000000000002)=1.2518544638517033E-33; dx=1.1988092273894284E-18 evalInputDelta=0.04024647350573214
END: th(5.000000000000001)=0.010061618376433032; dx=-0.004024647350573214 evalInputDelta=0.03018485512929911
Fitness changed from 0.04024647350573214 to 1.2518544638517033E-33
Iteration 3 complete. Error: 1.2518544638517033E-33 Total: 0.0858; Orientation: 0.0012; Line Search: 0.0644
Zero gradient: 1.5823112613210483E-17
th(0)=1.2518544638517033E-33;dx=-2.503708927703407E-34
New Minimum: 1.2518544638517033E-33 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=1.2518544638517033E-33
Fitness changed from 1.2518544638517033E-33 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0523; Orientation: 0.0015; Line Search: 0.0320
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0518; Orientation: 0.0010; Line Search: 0.0363
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.403s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.42 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4393148889021
Reset training subject: 4393168572841
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.22772320000000001}, derivative=-0.045544640000000004}
New Minimum: 0.22772320000000001 > 0.22772319999544552
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.22772319999544552}, derivative=-0.045544639999544556}, evalInputDelta = -4.554495669495395E-12
New Minimum: 0.22772319999544552 > 0.2277231999681187
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.2277231999681187}, derivative=-0.045544639996811874}, evalInputDelta = -3.1881303153014073E-11
New Minimum: 0.2277231999681187 > 0.22772319977683128
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.22772319977683128}, derivative=-0.04554463997768313}, evalInputDelta = -2.231687334930399E-10
New Minimum: 0.22772319977683128 > 0.22772319843781882
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.22772319843781882}, derivative=-0.04554463984378189}, evalInputDelta = -1.5621811899624305E-9
New Minimum: 0.22772319843781882 > 0.22772318906473205
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.22772318906473205}, derivative=-0.045544638906473194}, evalInputDelta = -1.093526796891453E-8
New Minimum: 0.22772318906473205 > 0.22772312345312998
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.22772312345312998}, derivative=-0.04554463234531236}, evalInputDelta = -7.654687003699756E-8
New Minimum: 0.22772312345312998 > 0.22772266417218004
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.22772266417218004}, derivative=-0.04554458641718649}, evalInputDelta = -5.358278199751876E-7
New Minimum: 0.22772266417218004 > 0.22771944921849877
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.22771944921849877}, derivative=-0.0455442649203054}, evalInputDelta = -3.7507815012494117E-6
New Minimum: 0.22771944921849877 > 0.22769694517816927
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.22769694517816927}, derivative=-0.04554201444213784}, evalInputDelta = -2.6254821830745678E-5
New Minimum: 0.22769694517816927 > 0.22753944803240386
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.22753944803240386}, derivative=-0.04552626109496483}, evalInputDelta = -1.8375196759615187E-4
New Minimum: 0.22753944803240386 > 0.22643849370256014
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.22643849370256014}, derivative=-0.04541598766475385}, evalInputDelta = -0.0012847062974398749
New Minimum: 0.22643849370256014 > 0.21880657222883032
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.21880657222883032}, derivative=-0.04464407365327693}, evalInputDelta = -0.008916627771169694
New Minimum: 0.21880657222883032 > 0.169046304836377
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.169046304836377}, derivative=-0.03924067557293848}, evalInputDelta = -0.05867689516362301
New Minimum: 0.169046304836377 > 2.2039634831585815E-4
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=2.2039634831585815E-4}, derivative=-0.0014168890105693082}, evalInputDelta = -0.22750280365168415
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=7.613741236628366}, derivative=0.2633496169260149}, evalInputDelta = 7.386018036628366
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.05209424855329341}, derivative=-0.021783543313383474}, evalInputDelta = -0.1756289514467066
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=1.6015651874903185}, derivative=0.12078303680631569}, evalInputDelta = 1.3738419874903185
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=0.11774995012267712}, derivative=-0.0327502033225911}, evalInputDelta = -0.10997324987732289
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=0.21269736052291727}, derivative=0.044016416741862296}, evalInputDelta = -0.015025839477082747
0.21269736052291727 <= 0.22772320000000001
New Minimum: 2.2039634831585815E-4 > 4.9688992565190686E-33
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=4.9688992565190686E-33}, derivative=-6.531997165382108E-18}, evalInputDelta = -0.22772320000000001
Left bracket at 9.999999999999998
Converged to left
Fitness changed from 0.22772320000000001 to 4.9688992565190686E-33
Iteration 1 complete. Error: 4.9688992565190686E-33 Total: 0.3616; Orientation: 0.0009; Line Search: 0.3112
Zero gradient: 3.1524274001217124E-17
F(0.0) = LineSearchPoint{point=PointSample{avg=4.9688992565190686E-33}, derivative=-9.937798513038139E-34}
New Minimum: 4.9688992565190686E-33 > 0.0
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -4.9688992565190686E-33
0.0 <= 4.9688992565190686E-33
Converged to right
Fitness changed from 4.9688992565190686E-33 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0324; Orientation: 0.0009; Line Search: 0.0213
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0208; Orientation: 0.0009; Line Search: 0.0107
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.415s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.32 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4393568974046
Reset training subject: 4393579441843
Adding measurement 473e2bd6 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.22772320000000001 < 0.22772320000000001. Total: 1
th(0)=0.22772320000000001;dx=-0.04554464
Adding measurement 780a3d2b to history. Total: 1
New Minimum: 0.22772320000000001 > 0.14017022226173195
END: th(2.154434690031884)=0.14017022226173195; dx=-0.035732344763898624 evalInputDelta=0.08755297773826806
Fitness changed from 0.22772320000000001 to 0.14017022226173195
Iteration 1 complete. Error: 0.14017022226173195 Total: 0.0685; Orientation: 0.0047; Line Search: 0.0281
Non-optimal measurement 0.14017022226173195 < 0.14017022226173195. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.14017022226173195 < 0.14017022226173195. Total: 2
th(0)=0.14017022226173195;dx=-0.02803404445234639
Adding measurement 5be7c42d to history. Total: 2
New Minimum: 0.14017022226173195 > 0.04024647350573214
END: th(4.641588833612779)=0.04024647350573214; dx=-0.015021793683244862 evalInputDelta=0.09992374875599981
Fitness changed from 0.14017022226173195 to 0.04024647350573214
Iteration 2 complete. Error: 0.04024647350573214 Total: 0.0337; Orientation: 0.0022; Line Search: 0.0219
Non-optimal measurement 0.04024647350573214 < 0.04024647350573214. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.04024647350573214 < 0.04024647350573214. Total: 3
th(0)=0.04024647350573214;dx=-0.008049294701146428
Adding measurement 8c620b2 to history. Total: 3
New Minimum: 0.04024647350573214 > 1.2518544638517033E-33
WOLF (strong): th(10.000000000000002)=1.2518544638517033E-33; dx=1.1988092273894284E-18 evalInputDelta=0.04024647350573214
Non-optimal measurement 0.010061618376433032 < 1.2518544638517033E-33. Total: 4
END: th(5.000000000000001)=0.010061618376433032; dx=-0.004024647350573214 evalInputDelta=0.03018485512929911
Fitness changed from 0.04024647350573214 to 1.2518544638517033E-33
Iteration 3 complete. Error: 1.2518544638517033E-33 Total: 0.0526; Orientation: 0.0019; Line Search: 0.0403
Non-optimal measurement 1.2518544638517033E-33 < 1.2518544638517033E-33. Total: 4
Rejected: LBFGS Orientation magnitude: 1.582e-16, gradient 1.582e-17, dot -1.000; [624f2151-f271-40ec-84cb-7ad928ccb6ca = 1.000/1.000e+00, 3a738ef5-b063-4047-a417-d27e979d27dc = 1.000/1.000e+00, 407d0db4-8584-4a0b-8827-7c42c7a23e8f = 1.000/1.000e+00, 3c6ab0cc-a5de-4005-89e9-afbeddf9f325 = 1.000/1.000e+00, 626cb621-cd8d-4696-b5f3-983bdf5ff2fb = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.2518544638517033E-33, 0.04024647350573214, 0.14017022226173195, 0.22772320000000001
LBFGS Accumulation History: 3 points
Removed measurement 8c620b2 to history. Total: 3
Adding measurement d10ff55 to history. Total: 3
th(0)=1.2518544638517033E-33;dx=-2.503708927703407E-34
Adding measurement 5f1ce78d to history. Total: 4
New Minimum: 1.2518544638517033E-33 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=1.2518544638517033E-33
Fitness changed from 1.2518544638517033E-33 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0683; Orientation: 0.0336; Line Search: 0.0255
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [624f2151-f271-40ec-84cb-7ad928ccb6ca = 0.000e+00, 407d0db4-8584-4a0b-8827-7c42c7a23e8f = 0.000e+00, 3a738ef5-b063-4047-a417-d27e979d27dc = 0.000e+00, 626cb621-cd8d-4696-b5f3-983bdf5ff2fb = 0.000e+00, 3c6ab0cc-a5de-4005-89e9-afbeddf9f325 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.2518544638517033E-33, 0.04024647350573214, 0.14017022226173195, 0.22772320000000001
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [3c6ab0cc-a5de-4005-89e9-afbeddf9f325 = 0.000e+00, 626cb621-cd8d-4696-b5f3-983bdf5ff2fb = 0.000e+00, 3a738ef5-b063-4047-a417-d27e979d27dc = 0.000e+00, 407d0db4-8584-4a0b-8827-7c42c7a23e8f = 0.000e+00, 624f2151-f271-40ec-84cb-7ad928ccb6ca = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.2518544638517033E-33, 0.04024647350573214, 0.14017022226173195
LBFGS Accumulation History: 3 points
Removed measurement 5f1ce78d to history. Total: 4
Removed measurement d10ff55 to history. Total: 3
Adding measurement c26e69d to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0988; Orientation: 0.0702; Line Search: 0.0198
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.323s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.14 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -32.90244615772304], [4.0, -0.8533442380168973]; valueStats=DoubleSummaryStatistics{count=7, sum=0.360833, min=0.000000, average=0.051548, max=0.140170}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -32.90244615772304], [0.198, -0.8533442380168973]; valueStats=DoubleSummaryStatistics{count=7, sum=0.360833, min=0.000000, average=0.051548, max=0.140170}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.033",
      "gc_time": "0.298"
    },
    "created_on": 1586739024360,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Right",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayerTest.Right",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer/Right/trainingTest/202004135024",
    "id": "8c6368ef-189e-4a94-a90e-162b12bb1b60",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPaddingLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer.java",
      "javaDoc": ""
    }
  }