1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 4457730160615260160

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 0.7 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ -0.128 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.08 ] ],
    	[ [ 0.7 ], [ 0.496 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.7 ] ],
    	[ [ 0.08 ], [ 0.496 ] ]
    ]
    [
    	[ [ -0.128 ], [ 0.496 ] ],
    	[ [ 0.7 ], [ 0.08 ] ]
    ]
    [
    	[ [ 0.08 ], [ 0.496 ] ],
    	[ [ -0.128 ], [ 0.7 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.39 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4358807639042
Reset training subject: 4358862597887
Constructing line search parameters: GD
th(0)=0.231304;dx=-0.046260800000000005
New Minimum: 0.231304 > 0.1423743083270727
END: th(2.154434690031884)=0.1423743083270727; dx=-0.0362942127691373 evalInputDelta=0.0889296916729273
Fitness changed from 0.231304 to 0.1423743083270727
Iteration 1 complete. Error: 0.1423743083270727 Total: 0.1415; Orientation: 0.0043; Line Search: 0.0486
th(0)=0.1423743083270727;dx=-0.028474861665414544
New Minimum: 0.1423743083270727 > 0.040879323265130066
END: th(4.641588833612779)=0.040879323265130066; dx=-0.01525800167092887 evalInputDelta=0.10149498506194264
Fitness changed from 0.1423743083270727 to 0.040879323265130066
Iteration 2 complete. Error: 0.040879323265130066 Total: 0.0555; Orientation: 0.0014; Line Search: 0.0383
th(0)=0.040879323265130066;dx=-0.008175864653026013
New Minimum: 0.040879323265130066 > 8.281498760865114E-34
WOLF (strong): th(10.000000000000002)=8.281498760865114E-34; dx=8.70461050839511E-19 evalInputDelta=0.040879323265130066
END: th(5.000000000000001)=0.010219830816282513; dx=-0.004087932326513006 evalInputDelta=0.030659492448847553
Fitness changed from 0.040879323265130066 to 8.281498760865114E-34
Iteration 3 complete. Error: 8.281498760865114E-34 Total: 0.0740; Orientation: 0.0012; Line Search: 0.0596
Zero gradient: 1.2869730969111294E-17
th(0)=8.281498760865114E-34;dx=-1.6562997521730232E-34
New Minimum: 8.281498760865114E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=8.281498760865114E-34
Fitness changed from 8.281498760865114E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0563; Orientation: 0.0010; Line Search: 0.0315
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0482; Orientation: 0.0015; Line Search: 0.0334
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.376s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.43 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4359190791763
Reset training subject: 4359206230802
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.231304}, derivative=-0.046260800000000005}
New Minimum: 0.231304 > 0.23130399999537393
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.23130399999537393}, derivative=-0.046260799999537396}, evalInputDelta = -4.626077299008102E-12
New Minimum: 0.23130399999537393 > 0.2313039999676174
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.2313039999676174}, derivative=-0.04626079999676175}, evalInputDelta = -3.238259660420795E-11
New Minimum: 0.2313039999676174 > 0.23130399977332208
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.23130399977332208}, derivative=-0.046260799977332207}, evalInputDelta = -2.266779264292751E-10
New Minimum: 0.23130399977332208 > 0.23130399841325455
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.23130399841325455}, derivative=-0.04626079984132546}, evalInputDelta = -1.58674545724935E-9
New Minimum: 0.23130399841325455 > 0.231303988892782
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.231303988892782}, derivative=-0.04626079888927819}, evalInputDelta = -1.110721800645642E-8
New Minimum: 0.231303988892782 > 0.23130392224947993
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.23130392224947993}, derivative=-0.04626079222494735}, evalInputDelta = -7.775052007774619E-8
New Minimum: 0.23130392224947993 > 0.2313034557466342
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.2313034557466342}, derivative=-0.04626074557463141}, evalInputDelta = -5.442533658195359E-7
New Minimum: 0.2313034557466342 > 0.23130019023988613
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.23130019023988613}, derivative=-0.046260419022419855}, evalInputDelta = -3.8097601138820103E-6
New Minimum: 0.23130019023988613 > 0.23127733233808087
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.23127733233808087}, derivative=-0.046258133156938996}, evalInputDelta = -2.6667661919138608E-5
New Minimum: 0.23127733233808087 > 0.23111735865158733
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.23111735865158733}, derivative=-0.04624213209857295}, evalInputDelta = -1.86641348412675E-4
New Minimum: 0.23111735865158733 > 0.2299990925271424
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.2299990925271424}, derivative=-0.04613012469001061}, evalInputDelta = -0.0013049074728576082
New Minimum: 0.2299990925271424 > 0.22224716402552472
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.22224716402552472}, derivative=-0.04534607283007426}, evalInputDelta = -0.009056835974475291
New Minimum: 0.22224716402552472 > 0.17170444861952294
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.17170444861952294}, derivative=-0.03985770981051979}, evalInputDelta = -0.05959955138047707
New Minimum: 0.17170444861952294 > 2.238619383130539E-4
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=2.238619383130539E-4}, derivative=-0.0014391686736385367}, evalInputDelta = -0.23108013806168695
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=7.7334623920491525}, derivative=0.2674906192845302}, evalInputDelta = 7.502158392049153
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.05291339691068357}, derivative=-0.02212607543965152}, evalInputDelta = -0.17839060308931642
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=1.6267487639698572}, derivative=0.12268227192243938}, evalInputDelta = 1.3954447639698573
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=0.11960149191288243}, derivative=-0.03326517908288928}, evalInputDelta = -0.11170250808711758
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=0.21604188891774245}, derivative=0.04470854641977505}, evalInputDelta = -0.015262111082257557
0.21604188891774245 <= 0.231304
New Minimum: 2.238619383130539E-4 > 4.198527278764174E-33
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=4.198527278764174E-33}, derivative=-5.976330541557218E-18}, evalInputDelta = -0.231304
Left bracket at 9.999999999999998
Converged to left
Fitness changed from 0.231304 to 4.198527278764174E-33
Iteration 1 complete. Error: 4.198527278764174E-33 Total: 0.3647; Orientation: 0.0014; Line Search: 0.3208
Zero gradient: 2.897767167584095E-17
F(0.0) = LineSearchPoint{point=PointSample{avg=4.198527278764174E-33}, derivative=-8.39705455752835E-34}
New Minimum: 4.198527278764174E-33 > 0.0
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -4.198527278764174E-33
0.0 <= 4.198527278764174E-33
Converged to right
Fitness changed from 4.198527278764174E-33 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0434; Orientation: 0.0011; Line Search: 0.0302
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0232; Orientation: 0.0007; Line Search: 0.0131
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.432s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.31 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 4359627783017
Reset training subject: 4359637363623
Adding measurement 7d8bbec7 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.231304 < 0.231304. Total: 1
th(0)=0.231304;dx=-0.046260800000000005
Adding measurement 1713c3fe to history. Total: 1
New Minimum: 0.231304 > 0.1423743083270727
END: th(2.154434690031884)=0.1423743083270727; dx=-0.0362942127691373 evalInputDelta=0.0889296916729273
Fitness changed from 0.231304 to 0.1423743083270727
Iteration 1 complete. Error: 0.1423743083270727 Total: 0.0573; Orientation: 0.0036; Line Search: 0.0238
Non-optimal measurement 0.1423743083270727 < 0.1423743083270727. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.1423743083270727 < 0.1423743083270727. Total: 2
th(0)=0.1423743083270727;dx=-0.028474861665414544
Adding measurement 2b3901e8 to history. Total: 2
New Minimum: 0.1423743083270727 > 0.040879323265130066
END: th(4.641588833612779)=0.040879323265130066; dx=-0.01525800167092887 evalInputDelta=0.10149498506194264
Fitness changed from 0.1423743083270727 to 0.040879323265130066
Iteration 2 complete. Error: 0.040879323265130066 Total: 0.0350; Orientation: 0.0022; Line Search: 0.0220
Non-optimal measurement 0.040879323265130066 < 0.040879323265130066. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.040879323265130066 < 0.040879323265130066. Total: 3
th(0)=0.040879323265130066;dx=-0.008175864653026013
Adding measurement 47d701ab to history. Total: 3
New Minimum: 0.040879323265130066 > 8.281498760865114E-34
WOLF (strong): th(10.000000000000002)=8.281498760865114E-34; dx=8.70461050839511E-19 evalInputDelta=0.040879323265130066
Non-optimal measurement 0.010219830816282513 < 8.281498760865114E-34. Total: 4
END: th(5.000000000000001)=0.010219830816282513; dx=-0.004087932326513006 evalInputDelta=0.030659492448847553
Fitness changed from 0.040879323265130066 to 8.281498760865114E-34
Iteration 3 complete. Error: 8.281498760865114E-34 Total: 0.0487; Orientation: 0.0020; Line Search: 0.0340
Non-optimal measurement 8.281498760865114E-34 < 8.281498760865114E-34. Total: 4
Rejected: LBFGS Orientation magnitude: 1.287e-16, gradient 1.287e-17, dot -1.000; [e5ae112c-ca5b-4526-8e3f-47fb98136642 = 1.000/1.000e+00, 548b16bb-18d1-4d0f-88a1-0e67a9ce1692 = 1.000/1.000e+00, d9152d01-5f02-4c4f-8db7-87c627e547ab = 1.000/1.000e+00, 97f46f25-ab1b-408e-bd33-86b16f94eec2 = 1.000/1.000e+00, d809fcd6-ba99-4afa-a8f5-9da270be46ec = 1.000/1.000e+00]
Orientation rejected. Popping history element from 8.281498760865114E-34, 0.040879323265130066, 0.1423743083270727, 0.231304
LBFGS Accumulation History: 3 points
Removed measurement 47d701ab to history. Total: 3
Adding measurement 6a9f43b7 to history. Total: 3
th(0)=8.281498760865114E-34;dx=-1.656299752173023E-34
Adding measurement 286f979a to history. Total: 4
New Minimum: 8.281498760865114E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=8.281498760865114E-34
Fitness changed from 8.281498760865114E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0704; Orientation: 0.0361; Line Search: 0.0250
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [e5ae112c-ca5b-4526-8e3f-47fb98136642 = 0.000e+00, 97f46f25-ab1b-408e-bd33-86b16f94eec2 = 0.000e+00, 548b16bb-18d1-4d0f-88a1-0e67a9ce1692 = 0.000e+00, d809fcd6-ba99-4afa-a8f5-9da270be46ec = 0.000e+00, d9152d01-5f02-4c4f-8db7-87c627e547ab = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 8.281498760865114E-34, 0.040879323265130066, 0.1423743083270727, 0.231304
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [e5ae112c-ca5b-4526-8e3f-47fb98136642 = 0.000e+00, 548b16bb-18d1-4d0f-88a1-0e67a9ce1692 = 0.000e+00, 97f46f25-ab1b-408e-bd33-86b16f94eec2 = 0.000e+00, d9152d01-5f02-4c4f-8db7-87c627e547ab = 0.000e+00, d809fcd6-ba99-4afa-a8f5-9da270be46ec = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 8.281498760865114E-34, 0.040879323265130066, 0.1423743083270727
LBFGS Accumulation History: 3 points
Removed measurement 286f979a to history. Total: 4
Removed measurement 6a9f43b7 to history. Total: 3
Adding measurement af55fc7 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0942; Orientation: 0.0619; Line Search: 0.0196
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.306s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.15 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, -33.081891058786304], [4.0, -0.8465683727668574]; valueStats=DoubleSummaryStatistics{count=7, sum=0.366507, min=0.000000, average=0.052358, max=0.142374}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, -33.081891058786304], [0.186, -0.8465683727668574]; valueStats=DoubleSummaryStatistics{count=7, sum=0.366507, min=0.000000, average=0.052358, max=0.142374}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.020",
      "gc_time": "0.280"
    },
    "created_on": 1586738990433,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Left",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayerTest.Left",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer/Left/trainingTest/202004134950",
    "id": "24c55487-e913-453b-a3ce-c4490bc156a3",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPaddingLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer.java",
      "javaDoc": ""
    }
  }