1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 3774916106864890880

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.972, 1.356, -0.852, -1.856, -1.688, 1.556, -0.712, 1.704, ... ]
    [ 1.776 ]
    [ 0.7, 0.048, -1.616, -1.484, 1.356, -1.028, 0.496, -1.72, ... ]
    [ 1.776 ]
    [ 0.184, -0.032, 1.64, -1.476, -0.888, -1.16, 1.876, -0.768, ... ]
    [ 1.776 ]
    [ -0.032, 1.876, 1.764, 0.184, -0.892, 0.788, 0.972, -0.068, ... ]
    [ 1.776 ]
    [ -1.16, -1.34, 1.288, -1.832, 0.048, 1.156, -1.572, -1.552, ... ]
    [ 1.776 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.41 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1747215899687
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 1747253029810
Constructing line search parameters: GD
th(0)=225.59779679495037;dx=-1.1472741121011153E25
New Minimum: 225.59779679495037 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(1.077217345015942)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(0.3590724483386473)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(0.08976811208466183)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(0.017953622416932366)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(0.002992270402822061)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(4.2746720040315154E-4)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(5.343340005039394E-5)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(5.9370444500437714E-6)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(5.937044450043771E-7)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(5.397313136403428E-8)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(4.4977609470028565E-9)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(3.4598161130791205E-10)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(2.4712972236279432E-11)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Armijo: th(1.6475314824186289E-12)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Armijo: th(1.029707176511643E-13)=71.11150823820729; dx=-1.0325184005824358E24 evalInputDelta=154.4862885567431
Armijo: th(6.057101038303783E-15)=223.59470215363245; dx=-1.1472741120956867E25 evalInputDelta=2.003094641317915
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 225.59779679495037 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.3630; Orientation: 0.0047; Line Search: 0.3083
th(0)=0.0;dx=-6238.419052799998
Armijo: th(2.154434690031884E-15)=0.0; dx=-6238.419052799998 evalInputDelta=0.0
Armijo: th(1.077217345015942E-15)=0.0; dx=-6238.419052799998 evalInputDelta=0.0
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0342; Orientation: 0.0021; Line Search: 0.0266
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.398s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.75 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1747619851647
Reset training subject: 1747623439379
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=225.59779679495037}, derivative=-1.1472741121011153E25}
New Minimum: 225.59779679495037 > 0.0
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.1800296065442047E14}, evalInputDelta = -225.59779679495037
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.1800296065442047E14}, evalInputDelta = -225.59779679495037
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.1800296065442047E14}, evalInputDelta = -225.59779679495037
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.1800296065442047E14}, evalInputDelta = -225.59779679495037
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.1800296065442047E14}, evalInputDelta = -225.59779679495037
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
0.0 <= 225.59779679495037
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.180029606544205E14}, evalInputDelta = -225.59779679495037
Right bracket at 1.0E10
Converged to right
Fitness changed from 225.59779679495037 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.4358; Orientation: 0.0018; Line Search: 0.4241
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-6238.419052799998}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.3123; Orientation: 0.0012; Line Search: 0.3089
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.749s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.31 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1748372812197
Reset training subject: 1748375130876
Adding measurement 5effe14d to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 225.59779679495037 < 225.59779679495037. Total: 1
th(0)=225.59779679495037;dx=-1.1472741121011153E25
Adding measurement 7b2e9596 to history. Total: 1
New Minimum: 225.59779679495037 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.3590724483386473)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.08976811208466183)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.017953622416932366)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.002992270402822061)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(4.2746720040315154E-4)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.343340005039394E-5)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.9370444500437714E-6)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.937044450043771E-7)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(5.397313136403428E-8)=0.0; dx=-2.1800296065442047E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(4.4977609470028565E-9)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(3.4598161130791205E-10)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(2.4712972236279432E-11)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.6475314824186289E-12)=0.0; dx=-2.180029606544205E14 evalInputDelta=225.59779679495037
Non-optimal measurement 71.11150823820729 < 0.0. Total: 2
Armijo: th(1.029707176511643E-13)=71.11150823820729; dx=-1.0325184005824358E24 evalInputDelta=154.4862885567431
Non-optimal measurement 223.59470215363245 < 0.0. Total: 2
Armijo: th(6.057101038303783E-15)=223.59470215363245; dx=-1.1472741120956867E25 evalInputDelta=2.003094641317915
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 225.59779679495037 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.2921; Orientation: 0.0077; Line Search: 0.2775
Non-optimal measurement 0.0 < 0.0. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.0 < 0.0. Total: 2
th(0)=0.0;dx=-6238.419052799998
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(2.154434690031884E-15)=0.0; dx=-6238.419052799998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942E-15)=0.0; dx=-6238.419052799998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0215; Orientation: 0.0024; Line Search: 0.0166
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.314s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.10 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[0.0, 0.0], [2.0, 1.0]; valueStats=DoubleSummaryStatistics{count=0, sum=0.000000, min=Infinity, average=0.000000, max=-Infinity}
Only 0 points for GD
Only 0 points for CjGD
Only 0 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.00 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
No Data

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "2.029",
      "gc_time": "0.229"
    },
    "created_on": 1586736335018,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "N1Test",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.SumInputsLayerTest.N1Test",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/SumInputsLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/SumInputsLayer/N1Test/trainingTest/202004130535",
    "id": "e88f7ba4-0fca-4710-8b0e-525a41891634",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "SumInputsLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.SumInputsLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/SumInputsLayer.java",
      "javaDoc": ""
    }
  }