1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 9144630994940868608

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.7, -0.128, 0.08 ]
    [ 0.7, -0.128, 0.08 ]
    [ 0.7, -0.128, 0.08 ]
    [ 0.08, 0.7, -0.128 ]
    [ 0.08, 0.7, -0.128 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.33 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 3351486298880
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 3351513077598
Constructing line search parameters: GD
th(0)=9.733509875294144;dx=-3.9712E22
New Minimum: 9.733509875294144 > 0.0199838006405011
Armijo: th(2.154434690031884)=0.0199838006405011; dx=-3.971200000074223E10 evalInputDelta=9.713526074653643
Armijo: th(1.077217345015942)=0.02981580383737257; dx=-3.9712000000746826E10 evalInputDelta=9.703694071456772
Armijo: th(0.3590724483386473)=0.18600936308642044; dx=-3.9712000001059685E10 evalInputDelta=9.547500512207725
Armijo: th(0.08976811208466183)=0.6529971765171129; dx=-3.971200000313374E10 evalInputDelta=9.080512698777031
Armijo: th(0.017953622416932366)=0.9727767254728997; dx=-3.971200000663605E10 evalInputDelta=8.760733149821245
Armijo: th(0.002992270402822061)=1.0858797406445215; dx=-3.971200000866381E10 evalInputDelta=8.647630134649622
Armijo: th(4.2746720040315154E-4)=1.1087046443793687; dx=-3.971200000914338E10 evalInputDelta=8.624805230914776
Armijo: th(5.343340005039394E-5)=1.1121384614193723; dx=-3.9712000009217804E10 evalInputDelta=8.621371413874773
Armijo: th(5.9370444500437714E-6)=1.1125765003866877; dx=-3.971200000922734E10 evalInputDelta=8.620933374907457
Armijo: th(5.937044450043771E-7)=1.11262580813175; dx=-3.971200000922842E10 evalInputDelta=8.620884067162395
Armijo: th(5.397313136403428E-8)=1.1126307890312657; dx=-3.971200000922853E10 evalInputDelta=8.620879086262878
Armijo: th(4.4977609470028565E-9)=1.112631245616652; dx=-3.971200000922854E10 evalInputDelta=8.620878629677492
Armijo: th(3.4598161130791205E-10)=1.112631283931532; dx=-3.971200000922854E10 evalInputDelta=8.620878591362612
Armijo: th(2.4712972236279432E-11)=1.1548386903601189; dx=-4.111468038729657E10 evalInputDelta=8.578671184934025
Armijo: th(1.6475314824186289E-12)=2.634212378510317; dx=-5.120000003818638E20 evalInputDelta=7.099297496783827
Armijo: th(1.029707176511643E-13)=9.733509875293194; dx=-3.9712E22 evalInputDelta=9.50350909079134E-13
Armijo: th(6.057101038303783E-15)=9.73350987529409; dx=-3.9712E22 evalInputDelta=5.5067062021407764E-14
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0199838006405011
Fitness changed from 9.733509875294144 to 0.0199838006405011
Iteration 1 complete. Error: 0.0199838006405011 Total: 0.1278; Orientation: 0.0030; Line Search: 0.0895
th(0)=0.0199838006405011;dx=-0.1004048818499054
WOLFE (weak): th(2.154434690031884E-15)=0.0199838006405011; dx=-0.1004048818499054 evalInputDelta=0.0
New Minimum: 0.0199838006405011 > 0.019983800640501093
WOLFE (weak): th(4.308869380063768E-15)=0.019983800640501093; dx=-0.1004048818499054 evalInputDelta=6.938893903907228E-18
New Minimum: 0.019983800640501093 > 0.019983800640501075
WOLFE (weak): th(1.2926608140191303E-14)=0.019983800640501075; dx=-0.1004048818499054 evalInputDelta=2.42861286636753E-17
New Minimum: 0.019983800640501075 > 0.019983800640501
WOLFE (weak): th(5.1706432560765214E-14)=0.019983800640501; dx=-0.1004048818499054 evalInputDelta=1.0061396160665481E-16
New Minimum: 0.019983800640501 > 0.01998380064050061
WOLFE (weak): th(2.5853216280382605E-13)=0.01998380064050061; dx=-0.10040488184990537 evalInputDelta=4.891920202254596E-16
New Minimum: 0.01998380064050061 > 0.01998380064049816
WOLFE (weak): th(1.5511929768229563E-12)=0.01998380064049816; dx=-0.10040488184990526 evalInputDelta=2.9386215683047112E-15
New Minimum: 0.01998380064049816 > 0.019983800640480543
WOLFE (weak): th(1.0858350837760695E-11)=0.019983800640480543; dx=-0.10040488184990445 evalInputDelta=2.0556473190325164E-14
New Minimum: 0.019983800640480543 > 0.01998380064033667
WOLFE (weak): th(8.686680670208556E-11)=0.01998380064033667; dx=-0.10040488184989792 evalInputDelta=1.644309688408896E-13
New Minimum: 0.01998380064033667 > 0.01998380063902124
WOLFE (weak): th(7.8180126031877E-10)=0.01998380063902124; dx=-0.10040488184983806 evalInputDelta=1.4798579028862946E-12
New Minimum: 0.01998380063902124 > 0.019983800625702524
WOLFE (weak): th(7.818012603187701E-9)=0.019983800625702524; dx=-0.1004048818492321 evalInputDelta=1.4798575559415994E-11
New Minimum: 0.019983800625702524 > 0.01998380047771678
WOLFE (weak): th(8.599813863506471E-8)=0.01998380047771678; dx=-0.10040488184249913 evalInputDelta=1.6278432074523508E-10
New Minimum: 0.01998380047771678 > 0.01998379868708936
WOLFE (weak): th(1.0319776636207765E-6)=0.01998379868708936; dx=-0.10040488176103027 evalInputDelta=1.9534117379205185E-9
New Minimum: 0.01998379868708936 > 0.019983775246155588
WOLFE (weak): th(1.3415709627070094E-5)=0.019983775246155588; dx=-0.10040488069452953 evalInputDelta=2.539434551182551E-8
New Minimum: 0.019983775246155588 > 0.019983445121074517
WOLFE (weak): th(1.878199347789813E-4)=0.019983445121074517; dx=-0.10040486567478832 evalInputDelta=3.5551942658210156E-7
New Minimum: 0.019983445121074517 > 0.01997846816806357
WOLFE (weak): th(0.0028172990216847197)=0.01997846816806357; dx=-0.10040463925593687 evalInputDelta=5.332472437530322E-6
New Minimum: 0.01997846816806357 > 0.0198985629741787
WOLFE (weak): th(0.045076784346955515)=0.0198985629741787; dx=-0.10040100875646703 evalInputDelta=8.523766632239846E-5
New Minimum: 0.0198985629741787 > 0.01855791795365027
WOLFE (weak): th(0.7663053338982437)=0.01855791795365027; dx=-0.10034137979469585 evalInputDelta=0.0014258826868508295
New Minimum: 0.01855791795365027 > 9.411822435988591E-4
WOLFE (weak): th(13.793496010168386)=9.411822435988591E-4; dx=-0.09972680485350396 evalInputDelta=0.01904261839690224
New Minimum: 9.411822435988591E-4 > 0.0
WOLFE (weak): th(262.07642419319933)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(5241.528483863986)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(110072.09816114372)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(2421586.1595451618)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(1265829.1288531527)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(687950.6135071482)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(399011.355834146)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(254541.72699764484)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(182306.9125793943)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(218424.31978851958)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(200365.61618395694)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(191336.26438167563)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(195850.9402828163)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
WOLFE (weak): th(198108.27823338663)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
Armijo: th(199236.94720867177)=0.0; dx=-0.0996964446297697 evalInputDelta=0.0199838006405011
mu ~= nu (198108.27823338663): th(262.07642419319933)=0.0
Fitness changed from 0.0199838006405011 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.1172; Orientation: 0.0011; Line Search: 0.1134
th(0)=0.0;dx=-0.09927999999999998
Armijo: th(428027.16880545503)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(214013.58440272752)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(71337.86146757584)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(17834.46536689396)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(3566.893073378792)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(594.4821788964654)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(84.92602555663791)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(10.615753194579739)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(1.179528132731082)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(0.1179528132731082)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(0.010722983024828018)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(8.935819187356682E-4)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(6.873707067197447E-5)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(4.909790762283891E-6)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(3.273193841522594E-7)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(2.0457461509516212E-8)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(1.2033800887950713E-9)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(6.685444937750395E-11)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(3.518655230394945E-12)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(1.7593276151974725E-13)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Armijo: th(8.377750548559392E-15)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
MIN ALPHA (3.8080684311633604E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0738; Orientation: 0.0007; Line Search: 0.0714
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.319s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.13 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 3351812446680
Reset training subject: 3351814967098
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=9.733509875294144}, derivative=-3.9712E22}
New Minimum: 9.733509875294144 > 1.1126312862015841
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=1.1126312862015841}, derivative=-3.971200000922854E10}, evalInputDelta = -8.62087858909256
New Minimum: 1.1126312862015841 > 1.1126312806644574
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=1.1126312806644574}, derivative=-3.971200000922854E10}, evalInputDelta = -8.620878594629687
New Minimum: 1.1126312806644574 > 1.112631241904571
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=1.112631241904571}, derivative=-3.971200000922854E10}, evalInputDelta = -8.620878633389573
New Minimum: 1.112631241904571 > 1.112630970585467
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=1.112630970585467}, derivative=-3.971200000922853E10}, evalInputDelta = -8.620878904708677
New Minimum: 1.112630970585467 > 1.1126290713566047
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=1.1126290713566047}, derivative=-3.971200000922849E10}, evalInputDelta = -8.62088080393754
New Minimum: 1.1126290713566047 > 1.1126157769930176
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=1.1126157769930176}, derivative=-3.97120000092282E10}, evalInputDelta = -8.620894098301127
New Minimum: 1.1126157769930176 > 1.1125227281299772
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=1.1125227281299772}, derivative=-3.9712000009226166E10}, evalInputDelta = -8.620987147164167
New Minimum: 1.1125227281299772 > 1.1118719578258196
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=1.1118719578258196}, derivative=-3.9712000009212006E10}, evalInputDelta = -8.621637917468325
New Minimum: 1.1118719578258196 > 1.1073443483210548
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=1.1073443483210548}, derivative=-3.971200000911407E10}, evalInputDelta = -8.62616552697309
New Minimum: 1.1073443483210548 > 1.076937611177979
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=1.076937611177979}, derivative=-3.971200000848291E10}, evalInputDelta = -8.656572264116166
New Minimum: 1.076937611177979 > 0.9094320189797308
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.9094320189797308}, derivative=-3.9712000005717E10}, evalInputDelta = -8.824077856314414
New Minimum: 0.9094320189797308 > 0.40355341115735816
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.40355341115735816}, derivative=-3.971200000175302E10}, evalInputDelta = -9.329956464136787
New Minimum: 0.40355341115735816 > 0.0264871789555712
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0264871789555712}, derivative=-3.9712000000744965E10}, evalInputDelta = -9.707022696338573
New Minimum: 0.0264871789555712 > 0.002617448459314281
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.002617448459314281}, derivative=-3.971200000073898E10}, evalInputDelta = -9.73089242683483
New Minimum: 0.002617448459314281 > 0.0
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
0.0 <= 9.733509875294144
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-3.971200000073892E10}, evalInputDelta = -9.733509875294144
Right bracket at 1.0E10
Converged to right
Fitness changed from 9.733509875294144 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0717; Orientation: 0.0007; Line Search: 0.0655
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.09927999999999998}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0564; Orientation: 0.0006; Line Search: 0.0545
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.128s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.71 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 3351946139601
Reset training subject: 3351948366472
Adding measurement 644fb156 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 9.733509875294144 < 9.733509875294144. Total: 1
th(0)=9.733509875294144;dx=-3.9712E22
Adding measurement 6ccbcdf0 to history. Total: 1
New Minimum: 9.733509875294144 > 0.0199838006405011
Armijo: th(2.154434690031884)=0.0199838006405011; dx=-3.971200000074223E10 evalInputDelta=9.713526074653643
Non-optimal measurement 0.02981580383737257 < 0.0199838006405011. Total: 2
Armijo: th(1.077217345015942)=0.02981580383737257; dx=-3.971200000074683E10 evalInputDelta=9.703694071456772
Non-optimal measurement 0.18600936308642044 < 0.0199838006405011. Total: 2
Armijo: th(0.3590724483386473)=0.18600936308642044; dx=-3.9712000001059685E10 evalInputDelta=9.547500512207725
Non-optimal measurement 0.6529971765171129 < 0.0199838006405011. Total: 2
Armijo: th(0.08976811208466183)=0.6529971765171129; dx=-3.971200000313374E10 evalInputDelta=9.080512698777031
Non-optimal measurement 0.9727767254728997 < 0.0199838006405011. Total: 2
Armijo: th(0.017953622416932366)=0.9727767254728997; dx=-3.971200000663605E10 evalInputDelta=8.760733149821245
Non-optimal measurement 1.0858797406445215 < 0.0199838006405011. Total: 2
Armijo: th(0.002992270402822061)=1.0858797406445215; dx=-3.971200000866381E10 evalInputDelta=8.647630134649622
Non-optimal measurement 1.1087046443793687 < 0.0199838006405011. Total: 2
Armijo: th(4.2746720040315154E-4)=1.1087046443793687; dx=-3.971200000914339E10 evalInputDelta=8.624805230914776
Non-optimal measurement 1.1121384614193723 < 0.0199838006405011. Total: 2
Armijo: th(5.343340005039394E-5)=1.1121384614193723; dx=-3.9712000009217804E10 evalInputDelta=8.621371413874773
Non-optimal measurement 1.1125765003866877 < 0.0199838006405011. Total: 2
Armijo: th(5.9370444500437714E-6)=1.1125765003866877; dx=-3.971200000922735E10 evalInputDelta=8.620933374907457
Non-optimal measurement 1.11262580813175 < 0.0199838006405011. Total: 2
Armijo: th(5.937044450043771E-7)=1.11262580813175; dx=-3.9712000009228424E10 evalInputDelta=8.620884067162395
Non-optimal measurement 1.1126307890312657 < 0.0199838006405011. Total: 2
Armijo: th(5.397313136403428E-8)=1.1126307890312657; dx=-3.971200000922853E10 evalInputDelta=8.620879086262878
Non-optimal measurement 1.112631245616652 < 0.0199838006405011. Total: 2
Armijo: th(4.4977609470028565E-9)=1.112631245616652; dx=-3.971200000922854E10 evalInputDelta=8.620878629677492
Non-optimal measurement 1.112631283931532 < 0.0199838006405011. Total: 2
Armijo: th(3.4598161130791205E-10)=1.112631283931532; dx=-3.971200000922854E10 evalInputDelta=8.620878591362612
Non-optimal measurement 1.1548386903601189 < 0.0199838006405011. Total: 2
Armijo: th(2.4712972236279432E-11)=1.1548386903601189; dx=-4.111468038729657E10 evalInputDelta=8.578671184934025
Non-optimal measurement 2.634212378510317 < 0.0199838006405011. Total: 2
Armijo: th(1.6475314824186289E-12)=2.634212378510317; dx=-5.120000003818638E20 evalInputDelta=7.099297496783827
Non-optimal measurement 9.733509875293194 < 0.0199838006405011. Total: 2
Armijo: th(1.029707176511643E-13)=9.733509875293194; dx=-3.9712E22 evalInputDelta=9.50350909079134E-13
Non-optimal measurement 9.73350987529409 < 0.0199838006405011. Total: 2
Armijo: th(6.057101038303783E-15)=9.73350987529409; dx=-3.9712E22 evalInputDelta=5.5067062021407764E-14
Non-optimal measurement 0.0199838006405011 < 0.0199838006405011. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0199838006405011
Fitness changed from 9.733509875294144 to 0.0199838006405011
Iteration 1 complete. Error: 0.0199838006405011 Total: 0.0494; Orientation: 0.0037; Line Search: 0.0408
Non-optimal measurement 0.0199838006405011 < 0.0199838006405011. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.0199838006405011 < 0.0199838006405011. Total: 2
th(0)=0.0199838006405011;dx=-0.1004048818499054
Non-optimal measurement 0.0199838006405011 < 0.0199838006405011. Total: 2
WOLFE (weak): th(2.154434690031884E-15)=0.0199838006405011; dx=-0.1004048818499054 evalInputDelta=0.0
Adding measurement 2c4b324d to history. Total: 2
New Minimum: 0.0199838006405011 > 0.019983800640501093
WOLFE (weak): th(4.308869380063768E-15)=0.019983800640501093; dx=-0.1004048818499054 evalInputDelta=6.938893903907228E-18
Adding measurement 1566bf53 to history. Total: 3
New Minimum: 0.019983800640501093 > 0.019983800640501075
WOLFE (weak): th(1.2926608140191303E-14)=0.019983800640501075; dx=-0.1004048818499054 evalInputDelta=2.42861286636753E-17
Adding measurement 606dd40 to history. Total: 4
New Minimum: 0.019983800640501075 > 0.019983800640501
WOLFE (weak): th(5.1706432560765214E-14)=0.019983800640501; dx=-0.1004048818499054 evalInputDelta=1.0061396160665481E-16
Adding measurement fd6f9ba to history. Total: 5
New Minimum: 0.019983800640501 > 0.01998380064050061
WOLFE (weak): th(2.5853216280382605E-13)=0.01998380064050061; dx=-0.10040488184990537 evalInputDelta=4.891920202254596E-16
Adding measurement 10cb3e67 to history. Total: 6
New Minimum: 0.01998380064050061 > 0.01998380064049816
WOLFE (weak): th(1.5511929768229563E-12)=0.01998380064049816; dx=-0.10040488184990526 evalInputDelta=2.9386215683047112E-15
Adding measurement 5893094 to history. Total: 7
New Minimum: 0.01998380064049816 > 0.019983800640480543
WOLFE (weak): th(1.0858350837760695E-11)=0.019983800640480543; dx=-0.10040488184990445 evalInputDelta=2.0556473190325164E-14
Adding measurement 10a3dec2 to history. Total: 8
New Minimum: 0.019983800640480543 > 0.01998380064033667
WOLFE (weak): th(8.686680670208556E-11)=0.01998380064033667; dx=-0.10040488184989792 evalInputDelta=1.644309688408896E-13
Adding measurement 629e07ec to history. Total: 9
New Minimum: 0.01998380064033667 > 0.01998380063902124
WOLFE (weak): th(7.8180126031877E-10)=0.01998380063902124; dx=-0.10040488184983806 evalInputDelta=1.4798579028862946E-12
Adding measurement 2c3c37c6 to history. Total: 10
New Minimum: 0.01998380063902124 > 0.019983800625702524
WOLFE (weak): th(7.818012603187701E-9)=0.019983800625702524; dx=-0.1004048818492321 evalInputDelta=1.4798575559415994E-11
Adding measurement 7467e14b to history. Total: 11
New Minimum: 0.019983800625702524 > 0.01998380047771678
WOLFE (weak): th(8.599813863506471E-8)=0.01998380047771678; dx=-0.10040488184249913 evalInputDelta=1.6278432074523508E-10
Adding measurement 1587c0d9 to history. Total: 12
New Minimum: 0.01998380047771678 > 0.01998379868708936
WOLFE (weak): th(1.0319776636207765E-6)=0.01998379868708936; dx=-0.10040488176103027 evalInputDelta=1.9534117379205185E-9
Adding measurement 261335b5 to history. Total: 13
New Minimum: 0.01998379868708936 > 0.019983775246155588
WOLFE (weak): th(1.3415709627070094E-5)=0.019983775246155588; dx=-0.10040488069452953 evalInputDelta=2.539434551182551E-8
Adding measurement 42795595 to history. Total: 14
New Minimum: 0.019983775246155588 > 0.019983445121074517
WOLFE (weak): th(1.878199347789813E-4)=0.019983445121074517; dx=-0.10040486567478832 evalInputDelta=3.5551942658210156E-7
Adding measurement 50ce20bb to history. Total: 15
New Minimum: 0.019983445121074517 > 0.01997846816806357
WOLFE (weak): th(0.0028172990216847197)=0.01997846816806357; dx=-0.10040463925593687 evalInputDelta=5.332472437530322E-6
Adding measurement 692867a9 to history. Total: 16
New Minimum: 0.01997846816806357 > 0.0198985629741787
WOLFE (weak): th(0.045076784346955515)=0.0198985629741787; dx=-0.10040100875646703 evalInputDelta=8.523766632239846E-5
Adding measurement 3cef3516 to history. Total: 17
New Minimum: 0.0198985629741787 > 0.01855791795365027
WOLFE (weak): th(0.7663053338982437)=0.01855791795365027; dx=-0.10034137979469585 evalInputDelta=0.0014258826868508295
Adding measurement 5ad934a to history. Total: 18
New Minimum: 0.01855791795365027 > 9.411822435988591E-4
WOLFE (weak): th(13.793496010168386)=9.411822435988591E-4; dx=-0.09972680485350396 evalInputDelta=0.01904261839690224
Adding measurement 29c182b to history. Total: 19
New Minimum: 9.411822435988591E-4 > 0.0
WOLFE (weak): th(262.07642419319933)=0.0; dx=-0.0996964446297697 eval

...skipping 9089 bytes...

0e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517, 0.019983775246155588, 0.01998379868708936, 0.01998380047771678, 0.019983800625702524, 0.01998380063902124
Rejected: LBFGS Orientation magnitude: 5.569e+02, gradient 3.151e-01, dot -0.998; [5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00, 834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517, 0.019983775246155588, 0.01998379868708936, 0.01998380047771678, 0.019983800625702524
Rejected: LBFGS Orientation magnitude: 5.569e+02, gradient 3.151e-01, dot -0.998; [ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00, 834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517, 0.019983775246155588, 0.01998379868708936, 0.01998380047771678
Rejected: LBFGS Orientation magnitude: 5.569e+02, gradient 3.151e-01, dot -0.998; [834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517, 0.019983775246155588, 0.01998379868708936
Rejected: LBFGS Orientation magnitude: 5.569e+02, gradient 3.151e-01, dot -0.998; [afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, 834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517, 0.019983775246155588
Rejected: LBFGS Orientation magnitude: 7.305e+02, gradient 3.151e-01, dot -1.000; [ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, 834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357, 0.019983445121074517
Rejected: LBFGS Orientation magnitude: 7.336e+02, gradient 3.151e-01, dot -1.000; [834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787, 0.01997846816806357
Rejected: LBFGS Orientation magnitude: 7.621e+02, gradient 3.151e-01, dot -1.000; [834bc27b-89a3-4548-98f4-d5380716d8fd = 1.000/1.000e+00, afcecb11-f6e9-4aef-bc83-053addfac496 = 1.000/1.000e+00, 5179f9c5-862b-4804-8eb5-21191d31a98b = 1.000/1.000e+00, ed2b645b-25b7-424b-9bb4-210af33bb8b0 = 1.000/1.000e+00, 86628058-7f1b-4a23-ab1f-e36aa441953d = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 9.411822435988591E-4, 0.01855791795365027, 0.0198985629741787
LBFGS Accumulation History: 3 points
Removed measurement 29c182b to history. Total: 19
Removed measurement 5ad934a to history. Total: 18
Removed measurement 3cef3516 to history. Total: 17
Removed measurement 692867a9 to history. Total: 16
Removed measurement 50ce20bb to history. Total: 15
Removed measurement 42795595 to history. Total: 14
Removed measurement 261335b5 to history. Total: 13
Removed measurement 1587c0d9 to history. Total: 12
Removed measurement 7467e14b to history. Total: 11
Removed measurement 2c3c37c6 to history. Total: 10
Removed measurement 629e07ec to history. Total: 9
Removed measurement 10a3dec2 to history. Total: 8
Removed measurement 5893094 to history. Total: 7
Removed measurement 10cb3e67 to history. Total: 6
Removed measurement fd6f9ba to history. Total: 5
Removed measurement 606dd40 to history. Total: 4
Removed measurement 1566bf53 to history. Total: 3
Adding measurement 507fd9fc to history. Total: 3
th(0)=0.0;dx=-0.09927999999999998
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(428027.16880545503)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(214013.58440272752)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(71337.86146757584)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(17834.46536689396)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3566.893073378792)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(594.4821788964654)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(84.92602555663791)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(10.615753194579739)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.179528132731082)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.1179528132731082)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.010722983024828018)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.935819187356682E-4)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(6.873707067197447E-5)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.909790762283891E-6)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.273193841522594E-7)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2.0457461509516212E-8)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.2033800887950713E-9)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(6.685444937750395E-11)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.518655230394945E-12)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.7593276151974725E-13)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.377750548559392E-15)=0.0; dx=-0.09927999999999998 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
MIN ALPHA (3.8080684311633604E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.5966; Orientation: 0.5615; Line Search: 0.0339
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.709s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.10 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, -2.6993219114940668], [2.0, -0.6993219114940668]; valueStats=DoubleSummaryStatistics{count=2, sum=0.039968, min=0.019984, average=0.019984, max=0.019984}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, -2.6993219114940668], [0.117, -0.6993219114940668]; valueStats=DoubleSummaryStatistics{count=2, sum=0.039968, min=0.019984, average=0.019984, max=0.019984}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "1.938",
      "gc_time": "0.343"
    },
    "created_on": 1586737939294,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Basic",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.VariableLayerTest.Basic",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/VariableLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/VariableLayer/Basic/trainingTest/202004133219",
    "id": "314ec234-a3d6-4fdb-adb4-abaf28d3a4ba",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "VariableLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.VariableLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/VariableLayer.java",
      "javaDoc": ""
    }
  }