1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 1493664145123787776

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -1.408, 0.628, -0.988, 0.612, -0.38, -0.748, 1.036, 0.116, ... ] ]
    ]
    [
    	[ [ 0.508, -1.844, -0.648, -1.372, -0.432, -1.564, -1.652, 0.528, ... ] ]
    ]
    [
    	[ [ 1.708, 0.136, -0.38, -0.856, 1.556, -1.112, 1.588, -0.644, ... ] ]
    ]
    [
    	[ [ 0.2, 1.6, -0.256, -0.38, 1.88, 1.216, 1.736, -1.584, ... ] ]
    ]
    [
    	[ [ -1.644, 0.944, -0.448, 0.9, 0.016, 1.244, -1.128, 1.276, ... ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 4.10 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1452387635941
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 1452447903690
Constructing line search parameters: GD
th(0)=29449.11544690837;dx=-2.829203232E26
New Minimum: 29449.11544690837 > 4.551500382763309
Armijo: th(2.154434690031884)=4.551500382763309; dx=-2.829203232009954E14 evalInputDelta=29444.563946525606
Armijo: th(1.077217345015942)=18.86576220308195; dx=-2.829203232010005E14 evalInputDelta=29430.24968470529
Armijo: th(0.3590724483386473)=128.16524726047493; dx=-2.8292032320110025E14 evalInputDelta=29320.950199647894
Armijo: th(0.08976811208466183)=291.3251363701951; dx=-2.829203232015563E14 evalInputDelta=29157.790310538174
Armijo: th(0.017953622416932366)=402.9536024847596; dx=-2.829203232029708E14 evalInputDelta=29046.161844423612
Armijo: th(0.002992270402822061)=462.64660240782405; dx=-2.829203232071622E14 evalInputDelta=28986.468844500545
Armijo: th(4.2746720040315154E-4)=488.4999155587226; dx=-2.829203232179376E14 evalInputDelta=28960.61553134965
Armijo: th(5.343340005039394E-5)=497.7123328079371; dx=-2.829203232403026E14 evalInputDelta=28951.403114100434
Armijo: th(5.9370444500437714E-6)=499.96209942025496; dx=-2.829203232577198E14 evalInputDelta=28949.153347488114
Armijo: th(5.937044450043771E-7)=500.27919478989315; dx=-2.829203232613612E14 evalInputDelta=28948.836252118475
Armijo: th(5.397313136403428E-8)=500.3123448408686; dx=-2.829203232617664E14 evalInputDelta=28948.8031020675
Armijo: th(4.4977609470028565E-9)=500.315394578042; dx=-2.829203232618039E14 evalInputDelta=28948.800052330327
Armijo: th(3.4598161130791205E-10)=502.48262643971174; dx=-2.0800028292950868E20 evalInputDelta=28946.632820468658
Armijo: th(2.4712972236279432E-11)=884.8783045360626; dx=-5.231084802925781E23 evalInputDelta=28564.23714237231
Armijo: th(1.6475314824186289E-12)=23938.144314096913; dx=-2.1843052864079304E26 evalInputDelta=5510.971132811457
Armijo: th(1.029707176511643E-13)=29113.62765077263; dx=-2.784577324814516E26 evalInputDelta=335.4877961357415
Armijo: th(6.057101038303783E-15)=29421.406068657525; dx=-2.8263610240021833E26 evalInputDelta=27.709378250845475
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=4.551500382763309
Fitness changed from 29449.11544690837 to 4.551500382763309
Iteration 1 complete. Error: 4.551500382763309 Total: 0.9882; Orientation: 0.0100; Line Search: 0.8887
th(0)=4.551500382763309;dx=-564.7001699262673
New Minimum: 4.551500382763309 > 4.5515003827633045
WOLFE (weak): th(2.154434690031884E-15)=4.5515003827633045; dx=-564.7001699262673 evalInputDelta=4.440892098500626E-15
New Minimum: 4.5515003827633045 > 4.551500382763299
WOLFE (weak): th(4.308869380063768E-15)=4.551500382763299; dx=-564.7001699262673 evalInputDelta=9.769962616701378E-15
New Minimum: 4.551500382763299 > 4.55150038276328
WOLFE (weak): th(1.2926608140191303E-14)=4.55150038276328; dx=-564.7001699262673 evalInputDelta=2.930988785010413E-14
New Minimum: 4.55150038276328 > 4.55150038276319
WOLFE (weak): th(5.1706432560765214E-14)=4.55150038276319; dx=-564.7001699262673 evalInputDelta=1.1901590823981678E-13
New Minimum: 4.55150038276319 > 4.551500382762723
WOLFE (weak): th(2.5853216280382605E-13)=4.551500382762723; dx=-564.7001699262673 evalInputDelta=5.861977570020827E-13
New Minimum: 4.551500382762723 > 4.551500382759789
WOLFE (weak): th(1.5511929768229563E-12)=4.551500382759789; dx=-564.700169926267 evalInputDelta=3.5198510772715963E-12
New Minimum: 4.551500382759789 > 4.551500382738679
WOLFE (weak): th(1.0858350837760695E-11)=4.551500382738679; dx=-564.7001699262648 evalInputDelta=2.4630075756704173E-11
New Minimum: 4.551500382738679 > 4.551500382566277
WOLFE (weak): th(8.686680670208556E-11)=4.551500382566277; dx=-564.7001699262474 evalInputDelta=1.9703172426943638E-10
New Minimum: 4.551500382566277 > 4.551500380990031
WOLFE (weak): th(7.8180126031877E-10)=4.551500380990031; dx=-564.7001699260877 evalInputDelta=1.7732775248191501E-9
New Minimum: 4.551500380990031 > 4.551500365030537
WOLFE (weak): th(7.818012603187701E-9)=4.551500365030537; dx=-564.7001699244712 evalInputDelta=1.7732771695477823E-8
New Minimum: 4.551500365030537 > 4.5515001877028265
WOLFE (weak): th(8.599813863506471E-8)=4.5515001877028265; dx=-564.7001699065097 evalInputDelta=1.950604824330071E-7
New Minimum: 4.5515001877028265 > 4.5514980420376325
WOLFE (weak): th(1.0319776636207765E-6)=4.5514980420376325; dx=-564.7001696891758 evalInputDelta=2.340725676397426E-6
New Minimum: 4.5514980420376325 > 4.551469953348597
WOLFE (weak): th(1.3415709627070094E-5)=4.551469953348597; dx=-564.7001668440805 evalInputDelta=3.042941471154137E-5
New Minimum: 4.551469953348597 > 4.551074374720134
WOLFE (weak): th(1.878199347789813E-4)=4.551074374720134; dx=-564.7001267764381 evalInputDelta=4.2600804317505236E-4
New Minimum: 4.551074374720134 > 4.5451111129161585
WOLFE (weak): th(0.0028172990216847197)=4.5451111129161585; dx=-564.6995228565376 evalInputDelta=0.006389269847150381
New Minimum: 4.5451111129161585 > 4.451797582175036
WOLFE (weak): th(0.045076784346955515)=4.451797582175036; dx=-564.6901102835917 evalInputDelta=0.09970280058827274
New Minimum: 4.451797582175036 > 3.225261078628447
WOLFE (weak): th(0.7663053338982437)=3.225261078628447; dx=-564.5721582425631 evalInputDelta=1.326239304134862
New Minimum: 3.225261078628447 > 0.132813003271741
WOLFE (weak): th(13.793496010168386)=0.132813003271741; dx=-564.3464383882366 evalInputDelta=4.418687379491568
New Minimum: 0.132813003271741 > 4.3563740751283124E-5
WOLFE (weak): th(262.07642419319933)=4.3563740751283124E-5; dx=-564.3434967712031 evalInputDelta=4.551456819022557
New Minimum: 4.3563740751283124E-5 > 0.0
WOLFE (weak): th(5241.528483863986)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(110072.09816114372)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(57656.813322503855)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(31449.17090318392)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(18345.349693523953)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(11793.43908869397)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(8517.483786278977)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
WOLFE (weak): th(6879.506135071482)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
WOLFE (weak): th(7698.49496067523)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
Armijo: th(8107.989373477103)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
WOLFE (weak): th(7903.242167076167)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
WOLFE (weak): th(8005.615770276635)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
WOLFE (weak): th(8056.8025718768695)=0.0; dx=-564.3434967161069 evalInputDelta=4.551500382763309
mu ~= nu (8056.8025718768695): th(5241.528483863986)=0.0
Fitness changed from 4.551500382763309 to 0.0
Iteration 2 complete. Error: 0.0 Total: 1.7691; Orientation: 0.0036; Line Search: 1.7552
th(0)=0.0;dx=-564.0867232
Armijo: th(17412.99426210929)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(8706.497131054644)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(2902.165710351548)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(725.541427587887)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(145.1082855175774)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(24.184714252929567)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(3.454959178989938)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(0.43186989737374226)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(0.04798554415263803)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(0.004798554415263803)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(4.362322195694367E-4)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(3.6352684964119726E-5)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(2.7963603818553637E-6)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(1.9974002727538312E-7)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(1.3316001818358875E-8)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(8.322501136474297E-10)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(4.89558890380841E-11)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(2.7197716132268946E-12)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(1.4314587438036288E-13)=0.0; dx=-564.0867232 evalInputDelta=0.0
Armijo: th(7.157293719018144E-15)=0.0; dx=-564.0867232 evalInputDelta=0.0
MIN ALPHA (3.408235104294354E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 1.3261; Orientation: 0.0025; Line Search: 1.3190
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 4.084s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 2.61 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1456479276298
Reset training subject: 1456484282620
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=29449.11544690837}, derivative=-2.829203232E26}
New Minimum: 29449.11544690837 > 527.640418544999
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=527.640418544999}, derivative=-9.626880282949996E21}, evalInputDelta = -28921.475028363373
New Minimum: 527.640418544999 > 500.76240517461054
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=500.76240517461054}, derivative=-1.6000282920447767E19}, evalInputDelta = -28948.35304173376
New Minimum: 500.76240517461054 > 500.31536977592697
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=500.31536977592697}, derivative=-2.8292032326180356E14}, evalInputDelta = -28948.800077132444
New Minimum: 500.31536977592697 > 500.31355730032556
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=500.31355730032556}, derivative=-2.8292032326178125E14}, evalInputDelta = -28948.801889608047
New Minimum: 500.31355730032556 > 500.3008882856865
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=500.3008882856865}, derivative=-2.8292032326162575E14}, evalInputDelta = -28948.814558622686
New Minimum: 500.3008882856865 > 500.21308553319614
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=500.21308553319614}, derivative=-2.8292032326056756E14}, evalInputDelta = -28948.902361375174
New Minimum: 500.21308553319614 > 499.6365696983492
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=499.6365696983492}, derivative=-2.829203232543776E14}, evalInputDelta = -28949.478877210022
New Minimum: 499.6365696983492 > 496.63087793429185
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=496.63087793429185}, derivative=-2.829203232351239E14}, evalInputDelta = -28952.48456897408
New Minimum: 496.63087793429185 > 486.02993854161457
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=486.02993854161457}, derivative=-2.8292032321565906E14}, evalInputDelta = -28963.085508366756
New Minimum: 486.02993854161457 > 455.87445383821415
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=455.87445383821415}, derivative=-2.8292032320618125E14}, evalInputDelta = -28993.240993070158
New Minimum: 455.87445383821415 > 378.77493166672906
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=378.77493166672906}, derivative=-2.8292032320242956E14}, evalInputDelta = -29070.34051524164
New Minimum: 378.77493166672906 > 206.21460209281173
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=206.21460209281173}, derivative=-2.8292032320124325E14}, evalInputDelta = -29242.90084481556
New Minimum: 206.21460209281173 > 11.48506069331923
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=11.48506069331923}, derivative=-2.8292032320099744E14}, evalInputDelta = -29437.630386215053
New Minimum: 11.48506069331923 > 0.21016704679876988
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.21016704679876988}, derivative=-2.829203232009946E14}, evalInputDelta = -29448.90527986157
New Minimum: 0.21016704679876988 > 0.0039344376865718695
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0039344376865718695}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.111512470685
New Minimum: 0.0039344376865718695 > 0.0
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
0.0 <= 29449.11544690837
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.829203232009946E14}, evalInputDelta = -29449.11544690837
Right bracket at 1.0E10
Converged to right
Fitness changed from 29449.11544690837 to 0.0
Iteration 1 complete. Error: 0.0 Total: 1.4124; Orientation: 0.0026; Line Search: 1.3951
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-564.0867232}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 1.1944; Orientation: 0.1095; Line Search: 1.0812
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 2.608s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 44.52 seconds (0.080 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 1459091233375
Reset training subject: 1459248283222
Adding measurement 6426053f to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 29449.11544690837 < 29449.11544690837. Total: 1
th(0)=29449.11544690837;dx=-2.829203232E26
Adding measurement 6ae3844e to history. Total: 1
New Minimum: 29449.11544690837 > 4.551500382763309
Armijo: th(2.154434690031884)=4.551500382763309; dx=-2.829203232009954E14 evalInputDelta=29444.563946525606
Non-optimal measurement 18.86576220308195 < 4.551500382763309. Total: 2
Armijo: th(1.077217345015942)=18.86576220308195; dx=-2.829203232010005E14 evalInputDelta=29430.24968470529
Non-optimal measurement 128.16524726047493 < 4.551500382763309. Total: 2
Armijo: th(0.3590724483386473)=128.16524726047493; dx=-2.8292032320110025E14 evalInputDelta=29320.950199647894
Non-optimal measurement 291.3251363701951 < 4.551500382763309. Total: 2
Armijo: th(0.08976811208466183)=291.3251363701951; dx=-2.829203232015563E14 evalInputDelta=29157.790310538174
Non-optimal measurement 402.9536024847596 < 4.551500382763309. Total: 2
Armijo: th(0.017953622416932366)=402.9536024847596; dx=-2.829203232029708E14 evalInputDelta=29046.161844423612
Non-optimal measurement 462.64660240782405 < 4.551500382763309. Total: 2
Armijo: th(0.002992270402822061)=462.64660240782405; dx=-2.829203232071622E14 evalInputDelta=28986.468844500545
Non-optimal measurement 488.4999155587226 < 4.551500382763309. Total: 2
Armijo: th(4.2746720040315154E-4)=488.4999155587226; dx=-2.829203232179376E14 evalInputDelta=28960.61553134965
Non-optimal measurement 497.7123328079371 < 4.551500382763309. Total: 2
Armijo: th(5.343340005039394E-5)=497.7123328079371; dx=-2.829203232403026E14 evalInputDelta=28951.403114100434
Non-optimal measurement 499.96209942025496 < 4.551500382763309. Total: 2
Armijo: th(5.9370444500437714E-6)=499.96209942025496; dx=-2.829203232577198E14 evalInputDelta=28949.153347488114
Non-optimal measurement 500.27919478989315 < 4.551500382763309. Total: 2
Armijo: th(5.937044450043771E-7)=500.27919478989315; dx=-2.829203232613612E14 evalInputDelta=28948.836252118475
Non-optimal measurement 500.3123448408686 < 4.551500382763309. Total: 2
Armijo: th(5.397313136403428E-8)=500.3123448408686; dx=-2.829203232617664E14 evalInputDelta=28948.8031020675
Non-optimal measurement 500.315394578042 < 4.551500382763309. Total: 2
Armijo: th(4.4977609470028565E-9)=500.315394578042; dx=-2.829203232618039E14 evalInputDelta=28948.800052330327
Non-optimal measurement 502.48262643971174 < 4.551500382763309. Total: 2
Armijo: th(3.4598161130791205E-10)=502.48262643971174; dx=-2.0800028292950865E20 evalInputDelta=28946.632820468658
Non-optimal measurement 884.8783045360626 < 4.551500382763309. Total: 2
Armijo: th(2.4712972236279432E-11)=884.8783045360626; dx=-5.231084802925781E23 evalInputDelta=28564.23714237231
Non-optimal measurement 23938.144314096913 < 4.551500382763309. Total: 2
Armijo: th(1.6475314824186289E-12)=23938.144314096913; dx=-2.1843052864079304E26 evalInputDelta=5510.971132811457
Non-optimal measurement 29113.62765077263 < 4.551500382763309. Total: 2
Armijo: th(1.029707176511643E-13)=29113.62765077263; dx=-2.784577324814516E26 evalInputDelta=335.4877961357415
Non-optimal measurement 29421.406068657525 < 4.551500382763309. Total: 2
Armijo: th(6.057101038303783E-15)=29421.406068657525; dx=-2.8263610240021833E26 evalInputDelta=27.709378250845475
Non-optimal measurement 4.551500382763309 < 4.551500382763309. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=4.551500382763309
Fitness changed from 29449.11544690837 to 4.551500382763309
Iteration 1 complete. Error: 4.551500382763309 Total: 0.9835; Orientation: 0.0134; Line Search: 0.8046
Non-optimal measurement 4.551500382763309 < 4.551500382763309. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 4.551500382763309 < 4.551500382763309. Total: 2
th(0)=4.551500382763309;dx=-564.7001699262673
Adding measurement 6e548277 to history. Total: 2
New Minimum: 4.551500382763309 > 4.5515003827633045
WOLFE (weak): th(2.154434690031884E-15)=4.5515003827633045; dx=-564.7001699262673 evalInputDelta=4.440892098500626E-15
Adding measurement 2362772b to history. Total: 3
New Minimum: 4.5515003827633045 > 4.551500382763299
WOLFE (weak): th(4.308869380063768E-15)=4.551500382763299; dx=-564.7001699262673 evalInputDelta=9.769962616701378E-15
Adding measurement 164f5326 to history. Total: 4
New Minimum: 4.551500382763299 > 4.55150038276328
WOLFE (weak): th(1.2926608140191303E-14)=4.55150038276328; dx=-564.7001699262673 evalInputDelta=2.930988785010413E-14
Adding measurement 1212ca50 to history. Total: 5
New Minimum: 4.55150038276328 > 4.55150038276319
WOLFE (weak): th(5.1706432560765214E-14)=4.55150038276319; dx=-564.7001699262673 evalInputDelta=1.1901590823981678E-13
Adding measurement 1b637a41 to history. Total: 6
New Minimum: 4.55150038276319 > 4.551500382762723
WOLFE (weak): th(2.5853216280382605E-13)=4.551500382762723; dx=-564.7001699262673 evalInputDelta=5.861977570020827E-13
Adding measurement 6a5d010c to history. Total: 7
New Minimum: 4.551500382762723 > 4.551500382759789
WOLFE (weak): th(1.5511929768229563E-12)=4.551500382759789; dx=-564.700169926267 evalInputDelta=3.5198510772715963E-12
Adding measurement 466d2f31 to history. Total: 8
New Minimum: 4.551500382759789 > 4.551500382738679
WOLFE (weak): th(1.0858350837760695E-11)=4.551500382738679; dx=-564.7001699262648 evalInputDelta=2.4630075756704173E-11
Adding measurement 1b42ea2c to history. Total: 9
New Minimum: 4.551500382738679 > 4.551500382566277
WOLFE (weak): th(8.686680670208556E-11)=4.551500382566277; dx=-564.7001699262474 evalInputDelta=1.9703172426943638E-10
Adding measurement f456105 to history. Total: 10
New Minimum: 4.551500382566277 > 4.551500380990031
WOLFE (weak): th(7.8180126031877E-10)=4.551500380990031; dx=-564.7001699260877 evalInputDelta=1.7732775248191501E-9
Adding measurement 350d2c75 to history. Total: 11
New Minimum: 4.551500380990031 > 4.551500365030537
WOLFE (weak): th(7.818012603187701E-9)=4.551500365030537; dx=-564.7001699244712 evalInputDelta=1.7732771695477823E-8
Adding measurement 571de88f to history. Total: 12
New Minimum: 4.551500365030537 > 4.5515001877028265
WOLFE (weak): th(8.599813863506471E-8)=4.5515001877028265; dx=-564.7001699065097 evalInputDelta=1.950604824330071E-7
Adding measurement 6f525d1b to history. Total: 13
New Minimum: 4.5515001877028265 > 4.5514980420376325
WOLFE (weak): th(1.0319776636207765E-6)=4.5514980420376325; dx=-564.7001696891758 evalInputDelta=2.340725676397426E-6
Adding measurement 2dededfc to history. Total: 14
New Minimum: 4.5514980420376325 > 4.551469953348597
WOLFE (weak): th(1.3415709627070094E-5)=4.551469953348597; dx=-564.7001668440805 evalInputDelta=3.042941471154137E-5
Adding measurement 5086ebfc to history. Total: 15
New Minimum: 4.551469953348597 > 4.551074374720134
WOLFE (weak): th(1.878199347789813E-4)=4.551074374720134; dx=-564.7001267764381 evalInputDelta=4.2600804317505236E-4
Adding measurement 5540a487 to history. Total: 16
New Minimum: 4.551074374720134 > 4.5451111129161585
WOLFE (weak): th(0.0028172990216847197)=4.5451111129161585; dx=-564.6995228565376 evalInputDelta=0.006389269847150381
Adding measurement 33758a96 to history. Total: 17
New Minimum: 4.5451111129161585 > 4.451797582175036
WOLFE (weak): th(0.045076784346955515)=4.451797582175036; dx=-564.6901102835917 evalInputDelta=0.09970280058827274
Adding measurement 4a1ce07f to history. Total: 18
New Minimum: 4.451797582175036 > 3.225261078628447
WOLFE (weak): th(0.7663053338982437)=3.225261078628447; dx=-564.5721582425631 evalInputDelta=1.326239304134862
Adding measurement 5c0de568 to history. Total: 19
New Minimum: 3.225261078628447 > 0.132813003271741
WOLFE (weak): th(13.793496010168386)=0.132813003271741; dx=-564.3464383882366 evalInputDelta=4.418687379491568
Adding measurement 4a9ebb88 to history. Total: 20
New Minimum: 0.132813003271741 > 4.3563740751283124E-5
WOLFE (weak): th(262.07642419319933)=4.3563740751283124E-5; dx=-564.3434967712031 evalInputDelta=4.551456819022557
Adding measurement 13d38cfb to history. Total:

...skipping 10022 bytes...

376325, 4.5515001877028265, 4.551500365030537, 4.551500380990031
Rejected: LBFGS Orientation magnitude: 1.021e+05, gradient 2.375e+01, dot -0.956; [781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585, 4.551074374720134, 4.551469953348597, 4.5514980420376325, 4.5515001877028265, 4.551500365030537
Rejected: LBFGS Orientation magnitude: 1.021e+05, gradient 2.375e+01, dot -0.956; [781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585, 4.551074374720134, 4.551469953348597, 4.5514980420376325, 4.5515001877028265
Rejected: LBFGS Orientation magnitude: 1.021e+05, gradient 2.375e+01, dot -0.956; [67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585, 4.551074374720134, 4.551469953348597, 4.5514980420376325
Rejected: LBFGS Orientation magnitude: 1.426e+05, gradient 2.375e+01, dot -0.945; [fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585, 4.551074374720134, 4.551469953348597
Rejected: LBFGS Orientation magnitude: 1.158e+05, gradient 2.375e+01, dot -1.000; [67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585, 4.551074374720134
Rejected: LBFGS Orientation magnitude: 1.203e+05, gradient 2.375e+01, dot -1.000; [075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036, 4.5451111129161585
Rejected: LBFGS Orientation magnitude: 1.639e+05, gradient 2.375e+01, dot -1.000; [075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447, 4.451797582175036
Rejected: LBFGS Orientation magnitude: 1.547e+06, gradient 2.375e+01, dot -1.000; [781081b2-b1b0-4fa2-8ac8-51faa43edbfd = 1.000/1.000e+00, 075fc91b-725f-4895-ae9c-f1b5c683f387 = 1.000/1.000e+00, fc1d4eb7-032d-4ae0-b3af-2e14c99f6a67 = 1.000/1.000e+00, 1d1194b6-45ed-4524-92da-50aacc070349 = 1.000/1.000e+00, 67e34550-6b7d-40f7-b600-46f1aae5d378 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 4.3563740751283124E-5, 0.132813003271741, 3.225261078628447
LBFGS Accumulation History: 3 points
Removed measurement 13d38cfb to history. Total: 21
Removed measurement 4a9ebb88 to history. Total: 20
Removed measurement 5c0de568 to history. Total: 19
Removed measurement 4a1ce07f to history. Total: 18
Removed measurement 33758a96 to history. Total: 17
Removed measurement 5540a487 to history. Total: 16
Removed measurement 5086ebfc to history. Total: 15
Removed measurement 2dededfc to history. Total: 14
Removed measurement 6f525d1b to history. Total: 13
Removed measurement 571de88f to history. Total: 12
Removed measurement 350d2c75 to history. Total: 11
Removed measurement f456105 to history. Total: 10
Removed measurement 1b42ea2c to history. Total: 9
Removed measurement 466d2f31 to history. Total: 8
Removed measurement 6a5d010c to history. Total: 7
Removed measurement 1b637a41 to history. Total: 6
Removed measurement 1212ca50 to history. Total: 5
Removed measurement 164f5326 to history. Total: 4
Removed measurement 2362772b to history. Total: 3
Adding measurement 22f3f961 to history. Total: 3
th(0)=0.0;dx=-564.0867232
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(17412.99426210929)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8706.497131054644)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2902.165710351548)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(725.541427587887)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(145.1082855175774)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(24.184714252929567)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.454959178989938)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.43186989737374226)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.04798554415263803)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.004798554415263803)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.362322195694367E-4)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.6352684964119726E-5)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2.7963603818553637E-6)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.9974002727538312E-7)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.3316001818358875E-8)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.322501136474297E-10)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.89558890380841E-11)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2.7197716132268946E-12)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.4314587438036288E-13)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(7.157293719018144E-15)=0.0; dx=-564.0867232 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
MIN ALPHA (3.408235104294354E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 42.3170; Orientation: 41.0530; Line Search: 1.2599
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 44.525s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.09 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -0.3418454164103276], [2.0, 1.6581545835896723]; valueStats=DoubleSummaryStatistics{count=2, sum=9.103001, min=4.551500, average=4.551500, max=4.551500}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -0.3418454164103276], [1.768, 1.6581545835896723]; valueStats=DoubleSummaryStatistics{count=2, sum=9.103001, min=4.551500, average=4.551500, max=4.551500}
Plotting 2 points for GD
Only 1 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "51.869",
      "gc_time": "0.310"
    },
    "created_on": 1586736040170,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Big2",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.ReshapeLayerTest.Big2",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/ReshapeLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/ReshapeLayer/Big2/trainingTest/202004130040",
    "id": "78c498fc-8607-4a27-afd7-d821c336fdcf",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ReshapeLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.ReshapeLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/ReshapeLayer.java",
      "javaDoc": ""
    }
  }