1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 2216536936730800128

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 0.08 ], [ 0.7 ] ],
    	[ [ 0.496 ], [ -0.128 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ -0.128 ] ]
    ]
    [
    	[ [ 0.7 ], [ 0.08 ] ],
    	[ [ 0.496 ], [ -0.128 ] ]
    ]
    [
    	[ [ 0.08 ], [ 0.7 ] ],
    	[ [ -0.128 ], [ 0.496 ] ]
    ]
    [
    	[ [ 0.7 ], [ -0.128 ] ],
    	[ [ 0.08 ], [ 0.496 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.41 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4425833023255
Reset training subject: 4425893092090
Constructing line search parameters: GD
th(0)=0.2444496;dx=-0.04888992
New Minimum: 0.2444496 > 0.1504658056965275
END: th(2.154434690031884)=0.1504658056965275; dx=-0.03835690603591164 evalInputDelta=0.09398379430347248
Fitness changed from 0.2444496 to 0.1504658056965275
Iteration 1 complete. Error: 0.1504658056965275 Total: 0.1434; Orientation: 0.0042; Line Search: 0.0459
th(0)=0.1504658056965275;dx=-0.030093161139305498
New Minimum: 0.1504658056965275 > 0.04320260012983666
END: th(4.641588833612779)=0.04320260012983666; dx=-0.016125153068074456 evalInputDelta=0.10726320556669085
Fitness changed from 0.1504658056965275 to 0.04320260012983666
Iteration 2 complete. Error: 0.04320260012983666 Total: 0.0670; Orientation: 0.0014; Line Search: 0.0475
th(0)=0.04320260012983666;dx=-0.008640520025967333
New Minimum: 0.04320260012983666 > 8.666684749742561E-34
WOLF (strong): th(10.000000000000002)=8.666684749742561E-34; dx=8.718612562832207E-19 evalInputDelta=0.04320260012983666
END: th(5.000000000000001)=0.01080065003245916; dx=-0.004320260012983666 evalInputDelta=0.0324019500973775
Fitness changed from 0.04320260012983666 to 8.666684749742561E-34
Iteration 3 complete. Error: 8.666684749742561E-34 Total: 0.0827; Orientation: 0.0014; Line Search: 0.0616
Zero gradient: 1.316562550716263E-17
th(0)=8.666684749742561E-34;dx=-1.7333369499485125E-34
New Minimum: 8.666684749742561E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=8.666684749742561E-34
Fitness changed from 8.666684749742561E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0561; Orientation: 0.0014; Line Search: 0.0317
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0477; Orientation: 0.0012; Line Search: 0.0312
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.397s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.39 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4426236613427
Reset training subject: 4426250462987
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.2444496}, derivative=-0.04888992}
New Minimum: 0.2444496 > 0.24444959999511098
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.24444959999511098}, derivative=-0.0488899199995111}, evalInputDelta = -4.889005866814955E-12
New Minimum: 0.24444959999511098 > 0.24444959996577703
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.24444959996577703}, derivative=-0.048889919996577706}, evalInputDelta = -3.422295780097784E-11
New Minimum: 0.24444959996577703 > 0.24444959976043937
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.24444959976043937}, derivative=-0.04888991997604394}, evalInputDelta = -2.39560621340118E-10
New Minimum: 0.24444959976043937 > 0.24444959832307575
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.24444959832307575}, derivative=-0.048889919832307574}, evalInputDelta = -1.6769242383585237E-9
New Minimum: 0.24444959832307575 > 0.24444958826153035
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.24444958826153035}, derivative=-0.04888991882615302}, evalInputDelta = -1.173846964075409E-8
New Minimum: 0.24444958826153035 > 0.24444951783071836
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.24444951783071836}, derivative=-0.048889911783071144}, evalInputDelta = -8.216928162885218E-8
New Minimum: 0.24444951783071836 > 0.24444902481531852
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.24444902481531852}, derivative=-0.04888986248149802}, evalInputDelta = -5.751846814672223E-7
New Minimum: 0.24444902481531852 > 0.24444557372144046
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.24444557372144046}, derivative=-0.048889517370486135}, evalInputDelta = -4.0262785595268635E-6
New Minimum: 0.24444557372144046 > 0.24442141674640708
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.24442141674640708}, derivative=-0.04888710159340294}, evalInputDelta = -2.8183253592911628E-5
New Minimum: 0.24442141674640708 > 0.2442523513447111
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.2442523513447111}, derivative=-0.04887019115382058}, evalInputDelta = -1.9724865528888968E-4
New Minimum: 0.2442523513447111 > 0.24307053128619893
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.24307053128619893}, derivative=-0.0487518180767441}, evalInputDelta = -0.0013790687138010571
New Minimum: 0.24307053128619893 > 0.2348780407912267
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.2348780407912267}, derivative=-0.0479232065372087}, evalInputDelta = -0.009571559208773284
New Minimum: 0.2348780407912267 > 0.18146285314245728
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.18146285314245728}, derivative=-0.04212292576046085}, evalInputDelta = -0.06298674685754271
New Minimum: 0.18146285314245728 > 2.3658458684610185E-4
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=2.3658458684610185E-4}, derivative=-0.0015209603232260187}, evalInputDelta = -0.24421301541315388
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=8.17297490900053}, derivative=0.28269279773741784}, evalInputDelta = 7.92852530900053
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.05592060106810879}, derivative=-0.023383557097121702}, evalInputDelta = -0.1885289989318912
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=1.7192010715462165}, derivative=0.1296546203201481}, evalInputDelta = 1.4747514715462164
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=0.12639875167531622}, derivative=-0.03515572459075784}, evalInputDelta = -0.11805084832468377
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=0.2283201039722036}, derivative=0.047249447864695135}, evalInputDelta = -0.016129496027796397
0.2283201039722036 <= 0.2444496
New Minimum: 2.3658458684610185E-4 > 5.4214927934500687E-33
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=5.4214927934500687E-33}, derivative=-7.045475314271244E-18}, evalInputDelta = -0.2444496
Left bracket at 9.999999999999998
Converged to left
Fitness changed from 0.2444496 to 5.4214927934500687E-33
Iteration 1 complete. Error: 5.4214927934500687E-33 Total: 0.3399; Orientation: 0.0010; Line Search: 0.2934
Zero gradient: 3.292868899136457E-17
F(0.0) = LineSearchPoint{point=PointSample{avg=5.4214927934500687E-33}, derivative=-1.084298558690014E-33}
New Minimum: 5.4214927934500687E-33 > 0.0
F(9.999999999999998) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -5.4214927934500687E-33
0.0 <= 5.4214927934500687E-33
Converged to right
Fitness changed from 5.4214927934500687E-33 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0306; Orientation: 0.0007; Line Search: 0.0188
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0177; Orientation: 0.0007; Line Search: 0.0095
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.389s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.29 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 4426630373666
Reset training subject: 4426639449028
Adding measurement 2857c136 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.2444496 < 0.2444496. Total: 1
th(0)=0.2444496;dx=-0.04888992
Adding measurement a1b8a5d to history. Total: 1
New Minimum: 0.2444496 > 0.1504658056965275
END: th(2.154434690031884)=0.1504658056965275; dx=-0.03835690603591164 evalInputDelta=0.09398379430347248
Fitness changed from 0.2444496 to 0.1504658056965275
Iteration 1 complete. Error: 0.1504658056965275 Total: 0.0527; Orientation: 0.0037; Line Search: 0.0228
Non-optimal measurement 0.1504658056965275 < 0.1504658056965275. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.1504658056965275 < 0.1504658056965275. Total: 2
th(0)=0.1504658056965275;dx=-0.030093161139305498
Adding measurement 557f6d5d to history. Total: 2
New Minimum: 0.1504658056965275 > 0.04320260012983666
END: th(4.641588833612779)=0.04320260012983666; dx=-0.016125153068074456 evalInputDelta=0.10726320556669085
Fitness changed from 0.1504658056965275 to 0.04320260012983666
Iteration 2 complete. Error: 0.04320260012983666 Total: 0.0391; Orientation: 0.0023; Line Search: 0.0284
Non-optimal measurement 0.04320260012983666 < 0.04320260012983666. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.04320260012983666 < 0.04320260012983666. Total: 3
th(0)=0.04320260012983666;dx=-0.008640520025967331
Adding measurement 6c06bf2d to history. Total: 3
New Minimum: 0.04320260012983666 > 8.666684749742561E-34
WOLF (strong): th(10.000000000000002)=8.666684749742561E-34; dx=8.718612562832207E-19 evalInputDelta=0.04320260012983666
Non-optimal measurement 0.01080065003245916 < 8.666684749742561E-34. Total: 4
END: th(5.000000000000001)=0.01080065003245916; dx=-0.004320260012983666 evalInputDelta=0.0324019500973775
Fitness changed from 0.04320260012983666 to 8.666684749742561E-34
Iteration 3 complete. Error: 8.666684749742561E-34 Total: 0.0424; Orientation: 0.0016; Line Search: 0.0305
Non-optimal measurement 8.666684749742561E-34 < 8.666684749742561E-34. Total: 4
Rejected: LBFGS Orientation magnitude: 1.317e-16, gradient 1.317e-17, dot -1.000; [ca179c90-b660-4bbc-a724-7b40a2805e48 = 1.000/1.000e+00, 5044c188-2a8f-43dc-9979-0be22623886f = 1.000/1.000e+00, e5b72ef0-8990-423d-9d16-3f387ad25810 = 1.000/1.000e+00, 13a91935-988c-4a67-a835-fa5db6e0567c = 1.000/1.000e+00, 45f0249d-9049-440a-a92a-0055ab110283 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 8.666684749742561E-34, 0.04320260012983666, 0.1504658056965275, 0.2444496
LBFGS Accumulation History: 3 points
Removed measurement 6c06bf2d to history. Total: 3
Adding measurement 757d23ad to history. Total: 3
th(0)=8.666684749742561E-34;dx=-1.7333369499485125E-34
Adding measurement 38830dab to history. Total: 4
New Minimum: 8.666684749742561E-34 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=8.666684749742561E-34
Fitness changed from 8.666684749742561E-34 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0730; Orientation: 0.0372; Line Search: 0.0195
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [ca179c90-b660-4bbc-a724-7b40a2805e48 = 0.000e+00, e5b72ef0-8990-423d-9d16-3f387ad25810 = 0.000e+00, 5044c188-2a8f-43dc-9979-0be22623886f = 0.000e+00, 45f0249d-9049-440a-a92a-0055ab110283 = 0.000e+00, 13a91935-988c-4a67-a835-fa5db6e0567c = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 8.666684749742561E-34, 0.04320260012983666, 0.1504658056965275, 0.2444496
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [5044c188-2a8f-43dc-9979-0be22623886f = 0.000e+00, 45f0249d-9049-440a-a92a-0055ab110283 = 0.000e+00, 13a91935-988c-4a67-a835-fa5db6e0567c = 0.000e+00, e5b72ef0-8990-423d-9d16-3f387ad25810 = 0.000e+00, ca179c90-b660-4bbc-a724-7b40a2805e48 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 8.666684749742561E-34, 0.04320260012983666, 0.1504658056965275
LBFGS Accumulation History: 3 points
Removed measurement 38830dab to history. Total: 4
Removed measurement 757d23ad to history. Total: 3
Adding measurement 1f885c46 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0830; Orientation: 0.0574; Line Search: 0.0181
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.290s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.14 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -33.06214700059055], [4.0, -0.8225621850176863]; valueStats=DoubleSummaryStatistics{count=7, sum=0.387337, min=0.000000, average=0.055334, max=0.150466}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -33.06214700059055], [0.205, -0.8225621850176863]; valueStats=DoubleSummaryStatistics{count=7, sum=0.387337, min=0.000000, average=0.055334, max=0.150466}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.053",
      "gc_time": "0.278"
    },
    "created_on": 1586739057450,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Top",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayerTest.Top",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer/Top/trainingTest/202004135057",
    "id": "6afd3df4-ba1b-4105-9d0c-17a7efa6bfa5",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPaddingLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgPaddingLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgPaddingLayer.java",
      "javaDoc": ""
    }
  }