1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 1879275274989395968

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 1.056, -0.736 ], [ 1.648, -0.7 ], [ -1.62, 1.316 ], [ -1.86, 0.24 ], [ 1.952, -1.156 ], [ 0.144, 1.728 ], [ -1.284, -1.684 ], [ -0.156, 0.596 ], ... ],
    	[ [ 0.652, -0.456 ], [ -0.008, -0.536 ], [ 1.876, 0.516 ], [ 0.08, 1.236 ], [ -1.936, 0.212 ], [ -1.188, -1.94 ], [ 0.088, -0.784 ], [ 0.212, -1.072 ], ... ],
    	[ [ -1.172, -0.36 ], [ -1.476, 0.48 ], [ -1.236, -1.132 ], [ 1.856, -0.512 ], [ 1.88, 0.936 ], [ -1.276, -0.808 ], [ 1.748, -0.576 ], [ -0.424, 1.376 ], ... ],
    	[ [ -1.876, -1.732 ], [ 0.212, -0.676 ], [ -1.592, -1.456 ], [ 0.856, 0.828 ], [ -1.656, -1.304 ], [ -0.536, 1.168 ], [ -1.196, -0.548 ], [ -1.368, -1.004 ], ... ],
    	[ [ -0.344, 1.636 ], [ 0.324, 0.06 ], [ -0.92, -0.716 ], [ -1.968, -0.9 ], [ 0.212, 0.604 ], [ 0.956, 1.508 ], [ 0.392, -1.356 ], [ 0.208, 0.776 ], ... ],
    	[ [ -1.872, -0.26 ], [ -0.812, -1.184 ], [ 0.336, 1.24 ], [ 0.476, 0.036 ], [ -0.952, 1.056 ], [ 0.384, -0.476 ], [ 0.04, -0.956 ], [ 0.996, -1.92 ], ... ],
    	[ [ 0.92, 0.34 ], [ -1.92, -0.76 ], [ -1.896, -1.988 ], [ -1.644, 0.884 ], [ 1.952, 1.048 ], [ -0.596, 0.796 ], [ -1.704, 1.98 ], [ 0.0, -1.968 ], ... ],
    	[ [ 0.36, -1.852 ], [ -1.432, -1.072 ], [ -0.26, 0.392 ], [ 1.084, -1.2 ], [ -1.248, 1.64 ], [ -0.216, 0.616 ], [ -0.784, 1.488 ], [ 1.672, -1.632 ], ... ],
    	...
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 20.14 seconds (0.468 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 11305259875240
Reset training subject: 11305381170037
Constructing line search parameters: GD
th(0)=2.6697513978380214;dx=-1.0679005588386296E-5
New Minimum: 2.6697513978380214 > 2.669728390667768
WOLFE (weak): th(2.154434690031884)=2.669728390667768; dx=-1.0678959576503472E-5 evalInputDelta=2.3007170253386278E-5
New Minimum: 2.669728390667768 > 2.6697053833272966
WOLFE (weak): th(4.308869380063768)=2.6697053833272966; dx=-1.0678913563070325E-5 evalInputDelta=4.601451072483442E-5
New Minimum: 2.6697053833272966 > 2.6696133562143696
WOLFE (weak): th(12.926608140191302)=2.6696133562143696; dx=-1.067872950259021E-5 evalInputDelta=1.3804162365183004E-4
New Minimum: 2.6696133562143696 > 2.669199252793554
WOLFE (weak): th(51.70643256076521)=2.669199252793554; dx=-1.0677901243055438E-5 evalInputDelta=5.521450444674336E-4
New Minimum: 2.669199252793554 > 2.666991245123198
WOLFE (weak): th(258.53216280382605)=2.666991245123198; dx=-1.0673483857551554E-5 evalInputDelta=0.0027601527148233274
New Minimum: 2.666991245123198 > 2.653211895051193
WOLFE (weak): th(1551.1929768229563)=2.653211895051193; dx=-1.0645875192129108E-5 evalInputDelta=0.016539502786828475
New Minimum: 2.653211895051193 > 2.555054103542744
WOLFE (weak): th(10858.350837760694)=2.555054103542744; dx=-1.0447092809267375E-5 evalInputDelta=0.11469729429527753
New Minimum: 2.555054103542744 > 1.8226823735040365
END: th(86866.80670208555)=1.8226823735040365; dx=-8.823703362362895E-6 evalInputDelta=0.8470690243339849
Fitness changed from 2.6697513978380214 to 1.8226823735040365
Iteration 1 complete. Error: 1.8226823735040365 Total: 2.3412; Orientation: 0.2742; Line Search: 1.8241
th(0)=1.8226823735040365;dx=-7.29072949560471E-6
New Minimum: 1.8226823735040365 > 0.7135862344799726
END: th(187148.86177126726)=0.7135862344799726; dx=-4.561826039784814E-6 evalInputDelta=1.109096139024064
Fitness changed from 1.8226823735040365 to 0.7135862344799726
Iteration 2 complete. Error: 0.7135862344799726 Total: 0.3435; Orientation: 0.0685; Line Search: 0.2156
th(0)=0.7135862344799726;dx=-2.8543449362378328E-6
New Minimum: 0.7135862344799726 > 0.026745897189992224
END: th(403200.00000000006)=0.026745897189992224; dx=-5.526011809312287E-7 evalInputDelta=0.6868403372899804
Fitness changed from 0.7135862344799726 to 0.026745897189992224
Iteration 3 complete. Error: 0.026745897189992224 Total: 0.3371; Orientation: 0.0644; Line Search: 0.2181
th(0)=0.026745897189992224;dx=-1.0698358887524603E-7
New Minimum: 0.026745897189992224 > 0.014540796830033113
WOLF (strong): th(868668.0670208557)=0.014540796830033113; dx=7.888286580176289E-8 evalInputDelta=0.012205100359959111
New Minimum: 0.014540796830033113 > 4.613152809402743E-4
END: th(434334.03351042786)=4.613152809402743E-4; dx=-1.4050361471193833E-8 evalInputDelta=0.02628458190905195
Fitness changed from 0.026745897189992224 to 4.613152809402743E-4
Iteration 4 complete. Error: 4.613152809402743E-4 Total: 0.5343; Orientation: 0.0730; Line Search: 0.3948
th(0)=4.613152809402743E-4;dx=-1.84526112413351E-9
New Minimum: 4.613152809402743E-4 > 3.5036545623860825E-4
WOLF (strong): th(935744.3088563365)=3.5036545623860825E-4; dx=1.6081240697838814E-9 evalInputDelta=1.1094982470166606E-4
New Minimum: 3.5036545623860825E-4 > 1.9046758454076025E-6
END: th(467872.15442816826)=1.9046758454076025E-6; dx=-1.1856853365070459E-10 evalInputDelta=4.5941060509486673E-4
Fitness changed from 4.613152809402743E-4 to 1.9046758454076025E-6
Iteration 5 complete. Error: 1.9046758454076025E-6 Total: 0.4413; Orientation: 0.0698; Line Search: 0.3194
Low gradient: 2.76019988227702E-6
th(0)=1.9046758454076025E-6;dx=-7.61870339012209E-12
Armijo: th(1008000.0000000003)=1.966113613010227E-6; dx=7.74060370723723E-12 evalInputDelta=-6.14377676026244E-8
New Minimum: 1.9046758454076025E-6 > 1.2190407070884681E-10
WOLF (strong): th(504000.0000000002)=1.2190407070884681E-10; dx=6.095048935449664E-14 evalInputDelta=1.9045539413368936E-6
END: th(168000.00000000006)=8.397641171699856E-7; dx=-5.0588195125177E-12 evalInputDelta=1.0649117282376168E-6
Fitness changed from 1.9046758454076025E-6 to 1.2190407070884681E-10
Iteration 6 complete. Error: 1.2190407070884681E-10 Total: 0.5116; Orientation: 0.0655; Line Search: 0.3936
Low gradient: 2.2082035298015877E-8
th(0)=1.2190407070884681E-10;dx=-4.876162829028221E-16
New Minimum: 1.2190407070884681E-10 > 9.292052359864694E-12
END: th(361945.0279253566)=9.292052359864694E-12; dx=-1.3461481813526794E-16 evalInputDelta=1.1261201834898212E-10
Fitness changed from 1.2190407070884681E-10 to 9.292052359864694E-12
Iteration 7 complete. Error: 9.292052359864694E-12 Total: 0.3422; Orientation: 0.0640; Line Search: 0.2285
Low gradient: 6.0965735915956966E-9
th(0)=9.292052359864694E-12;dx=-3.716820955774223E-17
New Minimum: 9.292052359864694E-12 > 2.9106852080821855E-12
WOLF (strong): th(779786.9240469472)=2.9106852080821855E-12; dx=2.0797603049097644E-17 evalInputDelta=6.381367151782509E-12
New Minimum: 2.9106852080821855E-12 > 4.520382673736819E-13
END: th(389893.4620234736)=4.520382673736819E-13; dx=-8.185516506346348E-18 evalInputDelta=8.840014092491013E-12
Fitness changed from 9.292052359864694E-12 to 4.520382673736819E-13
Iteration 8 complete. Error: 4.520382673736819E-13 Total: 0.4664; Orientation: 0.0910; Line Search: 0.3159
Low gradient: 1.3446758317563857E-9
th(0)=4.520382673736819E-13;dx=-1.808153092509737E-18
New Minimum: 4.520382673736819E-13 > 2.1134021185861815E-13
WOLF (strong): th(840000.0000000003)=2.1134021185861815E-13; dx=1.2324255505496151E-18 evalInputDelta=2.406980555150637E-13
New Minimum: 2.1134021185861815E-13 > 1.2725745514258993E-14
END: th(420000.0000000002)=1.2725745514258993E-14; dx=-2.8721023640070485E-19 evalInputDelta=4.3931252185942293E-13
Fitness changed from 4.520382673736819E-13 to 1.2725745514258993E-14
Iteration 9 complete. Error: 1.2725745514258993E-14 Total: 0.4334; Orientation: 0.0672; Line Search: 0.3148
Low gradient: 2.2561689251476262E-10
th(0)=1.2725745514258993E-14;dx=-5.0902982188017634E-20
New Minimum: 1.2725745514258993E-14 > 1.0766298873005281E-14
WOLF (strong): th(904862.5698133915)=1.0766298873005281E-14; dx=4.5043787804578004E-20 evalInputDelta=1.959446641253712E-15
New Minimum: 1.0766298873005281E-14 > 4.266844498550074E-16
END: th(452431.2849066958)=4.266844498550074E-16; dx=-2.4272799789823374E-21 evalInputDelta=1.2299061064403985E-14
Fitness changed from 1.2725745514258993E-14 to 4.266844498550074E-16
Iteration 10 complete. Error: 4.266844498550074E-16 Total: 0.4589; Orientation: 0.0880; Line Search: 0.3138
Zero gradient: 4.1312683173689E-11
th(0)=4.266844498550074E-16;dx=-1.7067377910096055E-21
Armijo: th(974733.6550586841)=4.37860200459659E-16; dx=1.722241513255413E-21 evalInputDelta=-1.1175750604651572E-17
New Minimum: 4.266844498550074E-16 > 6.460735985122294E-19
WOLF (strong): th(487366.82752934203)=6.460735985122294E-19; dx=4.711096609949932E-24 evalInputDelta=4.260383762564952E-16
END: th(162455.60917644735)=5.050164491746297E-18; dx=-2.2455312682503327E-23 evalInputDelta=4.216342853632611E-16
Fitness changed from 4.266844498550074E-16 to 6.460735985122294E-19
Iteration 11 complete. Error: 6.460735985122294E-19 Total: 0.5592; Orientation: 0.0658; Line Search: 0.4398
Zero gradient: 1.6075740670341783E-12
th(0)=6.460735985122294E-19;dx=-2.5842943810008095E-24
New Minimum: 6.460735985122294E-19 > 3.0948121399555842E-24
END: th(350000.00000000023)=3.0948121399555842E-24; dx=-4.126416229876024E-29 evalInputDelta=6.460705037000894E-19
Fitness changed from 6.460735985122294E-19 to 3.0948121399555842E-24
Iteration 12 complete. Error: 3.0948121399555842E-24 Total: 0.3244; Orientation: 0.0648; Line Search: 0.2118
Zero gradient: 3.518415639058617E-15
th(0)=3.0948121399555842E-24;dx=-1.2379248609172257E-29
New Minimum: 3.0948121399555842E-24 > 7.989875306574893E-25
WOLF (strong): th(754052.1415111598)=7.989875306574893E-25; dx=6.289949208372204E-30 evalInputDelta=2.295824609298095E-24
New Minimum: 7.989875306574893E-25 > 1.8720627035295685E-25
END: th(377026.0707555799)=1.8720627035295685E-25; dx=-3.0446497098451608E-30 evalInputDelta=2.9076058696026274E-24
Fitness changed from 3.094812139955

...skipping 9854 bytes...

206671676097894E-64
New Minimum: 1.380166799636106E-58 > 4.681270809849142E-59
WOLF (strong): th(791196.4699074087)=4.681270809849142E-59; dx=3.215197537860499E-64 evalInputDelta=9.120397186511919E-59
New Minimum: 4.681270809849142E-59 > 6.017377705242122E-60
END: th(395598.23495370435)=6.017377705242122E-60; dx=-1.1527348321943727E-64 evalInputDelta=1.3199930225836847E-58
Fitness changed from 1.380166799636106E-58 to 6.017377705242122E-60
Iteration 30 complete. Error: 6.017377705242122E-60 Total: 0.4081; Orientation: 0.0657; Line Search: 0.2945
Zero gradient: 4.9060687968434746E-33
th(0)=6.017377705242122E-60;dx=-2.406951103936118E-65
New Minimum: 6.017377705242122E-60 > 2.987234261110583E-60
WOLF (strong): th(852290.5606996444)=2.987234261110583E-60; dx=1.6958923081655236E-65 evalInputDelta=3.0301434441315393E-60
New Minimum: 2.987234261110583E-60 > 1.3128761559916963E-61
END: th(426145.2803498222)=1.3128761559916963E-61; dx=-3.5552939135624185E-66 evalInputDelta=5.886090089642952E-60
Fitness changed from 6.017377705242122E-60 to 1.3128761559916963E-61
Iteration 31 complete. Error: 1.3128761559916963E-61 Total: 0.4135; Orientation: 0.0646; Line Search: 0.3015
Zero gradient: 7.246726576378188E-34
th(0)=1.3128761559916963E-61;dx=-5.251504607278593E-67
New Minimum: 1.3128761559916963E-61 > 9.180125116855726E-62
WOLF (strong): th(918102.1749790194)=9.180125116855726E-62; dx=4.3913309839841285E-67 evalInputDelta=3.948636443061237E-62
New Minimum: 9.180125116855726E-62 > 8.805793812675172E-64
END: th(459051.0874895097)=8.805793812675172E-64; dx=-4.300868136657689E-68 evalInputDelta=1.304070362179021E-61
Fitness changed from 1.3128761559916963E-61 to 8.805793812675172E-64
Iteration 32 complete. Error: 8.805793812675172E-64 Total: 0.4419; Orientation: 0.0871; Line Search: 0.3054
Zero gradient: 5.934911565005712E-35
th(0)=8.805793812675172E-64;dx=-3.522317528443855E-69
New Minimum: 8.805793812675172E-64 > 8.422448853994972E-64
WOLF (strong): th(988995.5873842611)=8.422448853994972E-64; dx=3.4447954520594864E-69 evalInputDelta=3.833449586801997E-65
New Minimum: 8.422448853994972E-64 > 1.0663559395297872E-67
END: th(494497.79369213054)=1.0663559395297872E-67; dx=-3.876103312755283E-71 evalInputDelta=8.804727456735642E-64
Fitness changed from 8.805793812675172E-64 to 1.0663559395297872E-67
Iteration 33 complete. Error: 1.0663559395297872E-67 Total: 0.4135; Orientation: 0.0642; Line Search: 0.3021
Zero gradient: 6.53102117920447E-37
th(0)=1.0663559395297872E-67;dx=-4.2654237643217344E-73
Armijo: th(1065363.2008745556)=1.3633810687635258E-67; dx=4.8230272671758137E-73 evalInputDelta=-2.9702512923373863E-68
New Minimum: 1.0663559395297872E-67 > 4.5558434668590535E-70
WOLF (strong): th(532681.6004372778)=4.5558434668590535E-70; dx=2.7880173897598402E-74 evalInputDelta=1.0618000960629282E-67
END: th(177560.5334790926)=4.434642107659092E-68; dx=-2.7506819422103153E-73 evalInputDelta=6.22891728763878E-68
Fitness changed from 1.0663559395297872E-67 to 4.5558434668590535E-70
Iteration 34 complete. Error: 4.5558434668590535E-70 Total: 0.4826; Orientation: 0.0639; Line Search: 0.3677
Zero gradient: 4.268884318791947E-38
th(0)=4.5558434668590535E-70;dx=-1.8223373327227783E-75
New Minimum: 4.5558434668590535E-70 > 2.5141419851968827E-71
END: th(382542.5729079248)=2.5141419851968827E-71; dx=-4.280941620508584E-76 evalInputDelta=4.304429268339365E-70
Fitness changed from 4.5558434668590535E-70 to 2.5141419851968827E-71
Iteration 35 complete. Error: 2.5141419851968827E-71 Total: 0.3066; Orientation: 0.0647; Line Search: 0.1933
Zero gradient: 1.0028244620428543E-38
th(0)=2.5141419851968827E-71;dx=-1.0056569016715401E-76
New Minimum: 2.5141419851968827E-71 > 1.0567609603918396E-71
WOLF (strong): th(824162.9894868843)=1.0567609603918396E-71; dx=6.519934532222112E-77 evalInputDelta=1.4573810248050431E-71
New Minimum: 1.0567609603918396E-71 > 7.773385135464646E-73
END: th(412081.49474344216)=7.773385135464646E-73; dx=-1.7683171344587596E-77 evalInputDelta=2.4364081338422363E-71
Fitness changed from 2.5141419851968827E-71 to 7.773385135464646E-73
Iteration 36 complete. Error: 7.773385135464646E-73 Total: 0.4064; Orientation: 0.0645; Line Search: 0.2944
Zero gradient: 1.763336657099615E-39
th(0)=7.773385135464646E-73;dx=-3.109356166271245E-78
New Minimum: 7.773385135464646E-73 > 4.676193046497078E-73
WOLF (strong): th(887802.6673954632)=4.676193046497078E-73; dx=2.411635182805108E-78 evalInputDelta=3.097192088967568E-73
New Minimum: 4.676193046497078E-73 > 9.785272442027175E-75
END: th(443901.3336977316)=9.785272442027175E-75; dx=-3.488599678787957E-79 evalInputDelta=7.675532411044375E-73
Fitness changed from 7.773385135464646E-73 to 9.785272442027175E-75
Iteration 37 complete. Error: 9.785272442027175E-75 Total: 0.4527; Orientation: 0.0767; Line Search: 0.3173
Zero gradient: 1.9784081956570268E-40
th(0)=9.785272442027175E-75;dx=-3.914098988642893E-80
New Minimum: 9.785272442027175E-75 > 8.151526548511735E-75
WOLF (strong): th(956356.4322698122)=8.151526548511735E-75; dx=3.572464846445628E-80 evalInputDelta=1.6337458935154402E-75
New Minimum: 8.151526548511735E-75 > 1.8639689568473518E-77
END: th(478178.2161349061)=1.8639689568473518E-77; dx=-1.7079967641321121E-81 evalInputDelta=9.766632752458701E-75
Fitness changed from 9.785272442027175E-75 to 1.8639689568473518E-77
Iteration 38 complete. Error: 1.8639689568473518E-77 Total: 0.4809; Orientation: 0.0744; Line Search: 0.3502
Zero gradient: 8.633207985371635E-42
th(0)=1.8639689568473518E-77;dx=-7.453228011868456E-83
Armijo: th(1030203.7368586055)=2.0945279199095052E-77; dx=7.902764304127653E-83 evalInputDelta=-2.3055896306215345E-78
New Minimum: 1.8639689568473518E-77 > 1.6839360688330949E-80
WOLF (strong): th(515101.86842930276)=1.6839360688330949E-80; dx=2.2390964386438767E-84 evalInputDelta=1.8622850207785187E-77
END: th(171700.62280976758)=8.037485021068646E-78; dx=-4.8941515593355306E-83 evalInputDelta=1.0602204547404872E-77
Fitness changed from 1.8639689568473518E-77 to 1.6839360688330949E-80
Iteration 39 complete. Error: 1.6839360688330949E-80 Total: 0.4813; Orientation: 0.0646; Line Search: 0.3648
Zero gradient: 2.5997768910085167E-43
th(0)=1.6839360688330949E-80;dx=-6.75883988302191E-86
New Minimum: 1.6839360688330949E-80 > 1.162359336941063E-81
END: th(369917.778081443)=1.162359336941063E-81; dx=-1.7476372736459907E-86 evalInputDelta=1.5677001351389886E-80
Fitness changed from 1.6839360688330949E-80 to 1.162359336941063E-81
Iteration 40 complete. Error: 1.162359336941063E-81 Total: 0.3067; Orientation: 0.0645; Line Search: 0.1929
Zero gradient: 6.964326488408017E-44
th(0)=1.162359336941063E-81;dx=-4.850184343714154E-87
New Minimum: 1.162359336941063E-81 > 5.2017101970727394E-82
WOLF (strong): th(796963.6935581767)=5.2017101970727394E-82; dx=2.945456079178636E-87 evalInputDelta=6.42188317233789E-82
New Minimum: 5.2017101970727394E-82 > 7.11151391447811E-83
END: th(398481.8467790884)=7.11151391447811E-83; dx=-6.49963974805419E-88 evalInputDelta=1.091244197796282E-81
Fitness changed from 1.162359336941063E-81 to 7.11151391447811E-83
Iteration 41 complete. Error: 7.11151391447811E-83 Total: 0.4083; Orientation: 0.0647; Line Search: 0.2961
Zero gradient: 1.961817850054744E-44
th(0)=7.11151391447811E-83;dx=-3.8487292767934176E-88
Armijo: th(858503.1140488379)=1.3473820013638872E-82; dx=3.8487292767934176E-88 evalInputDelta=-6.3623060991607625E-83
New Minimum: 7.11151391447811E-83 > 3.2011178009361137E-83
END: th(429251.55702441896)=3.2011178009361137E-83; dx=0.0 evalInputDelta=3.9103961135419965E-83
Fitness changed from 7.11151391447811E-83 to 3.2011178009361137E-83
Iteration 42 complete. Error: 3.2011178009361137E-83 Total: 0.3940; Orientation: 0.0652; Line Search: 0.2817
Zero gradient: 0.0
th(0)=3.2011178009361137E-83;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 3.2011178009361137E-83 to 3.2011178009361137E-83
Static Iteration Total: 0.2864; Orientation: 0.0663; Line Search: 0.1715
Iteration 43 failed. Error: 3.2011178009361137E-83
Previous Error: 0.0 -> 3.2011178009361137E-83
Optimization terminated 43
Final threshold in iteration 43: 3.2011178009361137E-83 (> 0.0) after 20.127s (< 30.000s)

Returns

    3.2011178009361137E-83

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 5.65 seconds (0.070 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 11325411189859
Reset training subject: 11325458883497
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6697513978380214}, derivative=-1.0679005588386296E-5}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=2.6697513978380214}, derivative=-1.0679005588386296E-5}, evalInputDelta = 0.0
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=2.6697513978380214}, derivative=-1.0679005588386296E-5}, evalInputDelta = 0.0
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=2.6697513978380214}, derivative=-1.0679005588386296E-5}, evalInputDelta = 0.0
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=2.6697513978380214}, derivative=-1.0679005588386296E-5}, evalInputDelta = 0.0
New Minimum: 2.6697513978380214 > 2.6697513978380205
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=2.6697513978380205}, derivative=-1.0679005588386296E-5}, evalInputDelta = -8.881784197001252E-16
New Minimum: 2.6697513978380205 > 2.669751397838012
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=2.669751397838012}, derivative=-1.0679005588386296E-5}, evalInputDelta = -9.325873406851315E-15
New Minimum: 2.669751397838012 > 2.6697513978280267
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=2.6697513978280267}, derivative=-1.0679005588370875E-5}, evalInputDelta = -9.99467175688551E-12
New Minimum: 2.6697513978280267 > 2.669751397207893
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=2.669751397207893}, derivative=-1.067900558712351E-5}, evalInputDelta = -6.301283939080804E-10
New Minimum: 2.669751397207893 > 2.669751391620403
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=2.669751391620403}, derivative=-1.0679005575931813E-5}, evalInputDelta = -6.217618420123472E-9
New Minimum: 2.669751391620403 > 2.6697513544661295
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=2.6697513544661295}, derivative=-1.0679005501396321E-5}, evalInputDelta = -4.337189185577017E-8
New Minimum: 2.6697513544661295 > 2.6697510962880306
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=2.6697510962880306}, derivative=-1.0679004985927868E-5}, evalInputDelta = -3.015499907554897E-7
New Minimum: 2.6697510962880306 > 2.669749285979953
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=2.669749285979953}, derivative=-1.0679001366960797E-5}, evalInputDelta = -2.1118580684209576E-6
New Minimum: 2.669749285979953 > 2.6697366167271865
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=2.6697366167271865}, derivative=-1.0678976028460026E-5}, evalInputDelta = -1.4781110834860556E-5
New Minimum: 2.6697366167271865 > 2.6696479308185435
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=2.6696479308185435}, derivative=-1.0678798652945716E-5}, evalInputDelta = -1.0346701947794301E-4
New Minimum: 2.6696479308185435 > 2.6690271720608125
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=2.6690271720608125}, derivative=-1.0677557039752039E-5}, evalInputDelta = -7.242257772088756E-4
New Minimum: 2.6690271720608125 > 2.6646838812203684
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=2.6646838812203684}, derivative=-1.0668865742019203E-5}, evalInputDelta = -0.005067516617653034
New Minimum: 2.6646838812203684 > 2.6343798744695213
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=2.6343798744695213}, derivative=-1.0608026658770575E-5}, evalInputDelta = -0.035371523368500135
New Minimum: 2.6343798744695213 > 2.4271042947250363
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=2.4271042947250363}, derivative=-1.0182153078665868E-5}, evalInputDelta = -0.2426471031129851
New Minimum: 2.4271042947250363 > 1.2139460915202438
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=1.2139460915202438}, derivative=-7.201038006188067E-6}, evalInputDelta = -1.4558053063177776
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=4.372610632891192}, derivative=1.3666767485694708E-5}, evalInputDelta = 1.7028592350531704
F(87683.80911825495) = LineSearchPoint{point=PointSample{avg=1.8154805144687993}, derivative=-8.806253814172137E-6}, evalInputDelta = -0.8542708833692221
New Minimum: 1.2139460915202438 > 0.13826540873331047
F(613786.6638277847) = LineSearchPoint{point=PointSample{avg=0.13826540873331047}, derivative=2.430256835865448E-6}, evalInputDelta = -2.531485989104711
0.13826540873331047 <= 2.6697513978380214
New Minimum: 0.13826540873331047 > 2.5215324685304284E-15
F(500000.00007384265) = LineSearchPoint{point=PointSample{avg=2.5215324685304284E-15}, derivative=-3.7840657329198627E-16}, evalInputDelta = -2.6697513978380187
Left bracket at 500000.00007384265
Converged to left
Fitness changed from 2.6697513978380214 to 2.5215324685304284E-15
Iteration 1 complete. Error: 2.5215324685304284E-15 Total: 2.8926; Orientation: 0.0649; Line Search: 2.6817
Low gradient: 1.0042972598887299E-10
F(0.0) = LineSearchPoint{point=PointSample{avg=2.5215324685304284E-15}, derivative=-1.0086129862200127E-20}
New Minimum: 2.5215324685304284E-15 > 6.554756737386937E-19
F(500000.00007384265) = LineSearchPoint{point=PointSample{avg=6.554756737386937E-19}, derivative=5.442757310286009E-24}, evalInputDelta = -2.5208769928566896E-15
6.554756737386937E-19 <= 2.5215324685304284E-15
Converged to right
Fitness changed from 2.5215324685304284E-15 to 6.554756737386937E-19
Iteration 2 complete. Error: 6.554756737386937E-19 Total: 0.4743; Orientation: 0.1186; Line Search: 0.2805
Zero gradient: 1.6192290393009382E-12
F(0.0) = LineSearchPoint{point=PointSample{avg=6.554756737386937E-19}, derivative=-2.6219026817154392E-24}
New Minimum: 6.554756737386937E-19 > 1.1921712384417164E-48
F(500000.00007384265) = LineSearchPoint{point=PointSample{avg=1.1921712384417164E-48}, derivative=1.97897673514546E-48}, evalInputDelta = -6.554756737386937E-19
1.1921712384417164E-48 <= 6.554756737386937E-19
Converged to right
Fitness changed from 6.554756737386937E-19 to 1.1921712384417164E-48
Iteration 3 complete. Error: 1.1921712384417164E-48 Total: 0.3101; Orientation: 0.0654; Line Search: 0.1972
Zero gradient: 2.183731894957738E-27
F(0.0) = LineSearchPoint{point=PointSample{avg=1.1921712384417164E-48}, derivative=-4.7686849890557126E-54}
New Minimum: 1.1921712384417164E-48 > 1.1527776876567053E-63
F(500000.00007384265) = LineSearchPoint{point=PointSample{avg=1.1527776876567053E-63}, derivative=-1.6906674491921066E-62}, evalInputDelta = -1.1921712384417152E-48
F(3500000.0005168985) = LineSearchPoint{point=PointSample{avg=4.29181655163092E-47}, derivative=2.8612110350429523E-53}, evalInputDelta = 4.1725994277867485E-47
F(269230.76927053067) = LineSearchPoint{point=PointSample{avg=2.5395363769548763E-49}, derivative=-2.2009315366393326E-54}, evalInputDelta = -9.382176007462288E-49
F(1884615.3848937147) = LineSearchPoint{point=PointSample{avg=9.14233101731355E-48}, derivative=1.320558920272076E-53}, evalInputDelta = 7.950159778871834E-48
F(144970.41422259345) = LineSearchPoint{point=PointSample{avg=6.0107370248353845E-49}, derivative=-3.3860485425078484E-54}, evalInputDelta = -5.910975359581779E-49
F(1014792.8995581542) = LineSearchPoint{point=PointSample{avg=1.2637574430299417E-48}, derivative=4.9097702965501E-54}, evalInputDelta = 7.158620458822532E-50
F(78060.99227370416) = LineSearchPoint{point=PointSample{avg=8.489810337725056E-49}, derivative=-4.024188423147385E-54}, evalInputDelta = -3.4319020466921076E-49
F(546426.9459159292) = LineSearchPoint{point=PointSample{avg=1.0278715307348882E-50}, derivative=4.4279094118894775E-55}, evalInputDelta = -1.1818925231343676E-48
1.0278715307348882E-50 <= 1.1921712384417164E-48
New Minimum: 1.1527776876567053E-63 > 1.1377961531681311E-63
F(500000.00181954575) = LineSearchPoint{point=PointSample{avg=1.1377961531681311E-63}, derivative=-2.572583439499982E-64}, evalInputDelta = -1.1921712384417153E-48
Left bracket at 500000.00181954575
Converged to left
Fitness changed from 1.1921712384417164E-48 to 1.1377961531681311E-63
Iteration 4 complete. Error: 1.1377961531681311E-63 Total: 1.1413; Orientation: 0.0657; Line Search: 1.0277
Zero gradient: 6.746246808554963E-35
F(0.0) = LineSearchPoint{point=PointSample{avg=1.1377961531681311E-63}, derivative=-4.551184600193801E-69}
New Minimum: 1.1377961531681311E-63 > 2.0152957162862356E-78
F(500000.00181954575) = LineSearchPoint{point=PointSample{avg=2.0152957162862356E-78}, derivative=7.994487760437843E-77}, evalInputDelta = -1.1377961531681291E-63
2.0152957162862356E-78 <= 1.1377961531681311E-63
Converged to right
Fitness changed from 1.1377961531681311E-63 to 2.0152957162862356E-78
Iteration 5 complete. Error: 2.0152957162862356E-78 Total: 0.3178; Orientation: 0.0652; Line Search: 0.2055
Zero gradient: 2.839782766836325E-42
F(0.0) = LineSearchPoint{point=PointSample{avg=2.0152957162862356E-78}, derivative=-8.064366162820573E-84}
New Minimum: 2.0152957162862356E-78 > 4.0988407816183775E-83
F(500000.00181954575) = LineSearchPoint{point=PointSample{avg=4.0988407816183775E-83}, derivative=0.0}, evalInputDelta = -2.0152547278784193E-78
4.0988407816183775E-83 <= 2.0152957162862356E-78
Converged to right
Fitness changed from 2.0152957162862356E-78 to 4.0988407816183775E-83
Iteration 6 complete. Error: 4.0988407816183775E-83 Total: 0.3033; Orientation: 0.0654; Line Search: 0.1896
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=4.0988407816183775E-83}, derivative=0.0}
Fitness changed from 4.0988407816183775E-83 to 4.0988407816183775E-83
Static Iteration Total: 0.2058; Orientation: 0.0656; Line Search: 0.0934
Iteration 7 failed. Error: 4.0988407816183775E-83
Previous Error: 0.0 -> 4.0988407816183775E-83
Optimization terminated 7
Final threshold in iteration 7: 4.0988407816183775E-83 (> 0.0) after 5.645s (< 30.000s)

Returns

    4.0988407816183775E-83

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 30.71 seconds (0.682 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 11331076168366
Reset training subject: 11331132014007
Adding measurement 2a1c2d42 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 2.6697513978380214 < 2.6697513978380214. Total: 1
th(0)=2.6697513978380214;dx=-1.0679005588386296E-5
Adding measurement 5bb4cec to history. Total: 1
New Minimum: 2.6697513978380214 > 2.669728390667768
WOLFE (weak): th(2.154434690031884)=2.669728390667768; dx=-1.0678959576503472E-5 evalInputDelta=2.3007170253386278E-5
Adding measurement 52970584 to history. Total: 2
New Minimum: 2.669728390667768 > 2.6697053833272966
WOLFE (weak): th(4.308869380063768)=2.6697053833272966; dx=-1.0678913563070325E-5 evalInputDelta=4.601451072483442E-5
Adding measurement 5d1ce015 to history. Total: 3
New Minimum: 2.6697053833272966 > 2.6696133562143696
WOLFE (weak): th(12.926608140191302)=2.6696133562143696; dx=-1.067872950259021E-5 evalInputDelta=1.3804162365183004E-4
Adding measurement 1bec1514 to history. Total: 4
New Minimum: 2.6696133562143696 > 2.669199252793554
WOLFE (weak): th(51.70643256076521)=2.669199252793554; dx=-1.0677901243055438E-5 evalInputDelta=5.521450444674336E-4
Adding measurement 4d3eb006 to history. Total: 5
New Minimum: 2.669199252793554 > 2.666991245123198
WOLFE (weak): th(258.53216280382605)=2.666991245123198; dx=-1.0673483857551554E-5 evalInputDelta=0.0027601527148233274
Adding measurement 7556958c to history. Total: 6
New Minimum: 2.666991245123198 > 2.653211895051193
WOLFE (weak): th(1551.1929768229563)=2.653211895051193; dx=-1.0645875192129108E-5 evalInputDelta=0.016539502786828475
Adding measurement 6f7f1139 to history. Total: 7
New Minimum: 2.653211895051193 > 2.555054103542744
WOLFE (weak): th(10858.350837760694)=2.555054103542744; dx=-1.0447092809267375E-5 evalInputDelta=0.11469729429527753
Adding measurement 3b20ed80 to history. Total: 8
New Minimum: 2.555054103542744 > 1.8226823735040365
END: th(86866.80670208555)=1.8226823735040365; dx=-8.823703362362895E-6 evalInputDelta=0.8470690243339849
Fitness changed from 2.6697513978380214 to 1.8226823735040365
Iteration 1 complete. Error: 1.8226823735040365 Total: 1.9195; Orientation: 0.1192; Line Search: 1.6404
Non-optimal measurement 1.8226823735040365 < 1.8226823735040365. Total: 9
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198, 2.669199252793554, 2.6696133562143696, 2.6697053833272966, 2.669728390667768, 2.6697513978380214
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198, 2.669199252793554, 2.6696133562143696, 2.6697053833272966, 2.669728390667768
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198, 2.669199252793554, 2.6696133562143696, 2.6697053833272966
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198, 2.669199252793554, 2.6696133562143696
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198, 2.669199252793554
Rejected: LBFGS Orientation magnitude: 1.350e+03, gradient 2.700e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.8226823735040365, 2.555054103542744, 2.653211895051193, 2.666991245123198
LBFGS Accumulation History: 3 points
Removed measurement 3b20ed80 to history. Total: 8
Removed measurement 6f7f1139 to history. Total: 7
Removed measurement 7556958c to history. Total: 6
Removed measurement 4d3eb006 to history. Total: 5
Removed measurement 1bec1514 to history. Total: 4
Removed measurement 5d1ce015 to history. Total: 3
Adding measurement 36cc47d1 to history. Total: 3
th(0)=1.8226823735040365;dx=-7.29072949560471E-6
Adding measurement 249a4c8d to history. Total: 4
New Minimum: 1.8226823735040365 > 0.7135862344799726
END: th(187148.86177126726)=0.7135862344799726; dx=-4.561826039784814E-6 evalInputDelta=1.109096139024064
Fitness changed from 1.8226823735040365 to 0.7135862344799726
Iteration 2 complete. Error: 0.7135862344799726 Total: 19.3231; Orientation: 18.9183; Line Search: 0.3565
Non-optimal measurement 0.7135862344799726 < 0.7135862344799726. Total: 5
Rejected: LBFGS Orientation magnitude: 8.449e+02, gradient 1.689e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.7135862344799726, 1.8226823735040365, 2.6697053833272966, 2.669728390667768, 2.6697513978380214
Rejected: LBFGS Orientation magnitude: 8.449e+02, gradient 1.689e-03, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.7135862344799726, 1.8226823735040365, 2.6697053833272966, 2.669728390667768
LBFGS Accumulation History: 3 points
Removed measurement 249a4c8d to history. Total: 4
Removed measurement 36cc47d1 to history. Total: 3
Adding measurement 7654e6fc to history. Total: 3
th(0)=0.7135862344799726;dx=-2.8543449362378328E-6
Adding measurement 7e2d9138 to history. Total: 4
New Minimum: 0.7135862344799726 > 0.026745897189992224
END: th(403200.00000000006)=0.026745897189992224; dx=-5.526011809312287E-7 evalInputDelta=0.6868403372899804
Fitness changed from 0.7135862344799726 to 0.026745897189992224
Iteration 3 complete. Error: 0.026745897189992224 Total: 4.6815; Orientation: 4.3141; Line Search: 0.3186
Non-optimal measurement 0.026745897189992224 < 0.026745897189992224. Total: 5
Rejected: LBFGS Orientation magnitude: 1.636e+02, gradient 3.271e-04, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.026745897189992224, 0.7135862344799726, 2.6697053833272966, 2.669728390667768, 2.6697513978380214
Rejected: LBFGS Orientation magnitude: 1.636e+02, gradient 3.271e-04, dot -1.000; [e6ee4de6-eeac-4d31-867d-fb4fc944724c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.026745897189992224, 0.7135862344799726, 2.6697053833272966, 2.669728390667768
LBFGS Accumulation History: 3 points
Removed measurement 7e2d9138 to history. Total: 4
Removed measurement 7654e6fc to history. Total: 3
Adding measurement 1561caa to history. Total: 3
th(0)=0.026745897189992224;dx=-1.0698358887524603E-7
Adding measurement 60458bcc to history. Total: 4
New Minimum: 0.026745897189992224 > 0.014540796830033113
WOLF (strong): th(868668.0670208557)=0.014540796830033113; dx=7.888286580176289E-8 evalInputDelta=0.012205100359959111
Adding measurement 1582686f to history. Total: 5
New Minimum: 0.014540796830033113 > 4.613152809402743E-4
END: th(434334.03351042786)=4.613152809402743E-4; dx=-1.4050361471193833E-8 evalInputDelta=0.02628458190905195
Fitness changed from 0.026745897189992224 to 4.613152809402743E-4
Iteration 4 complete. Error: 4.613152809402743E-4 Total: 4.7888; Orientation: 4.2836; Line Search: 0.4581
Final threshold in iteration 4: 4.613152809402743E-4 (> 0.0) after 30.713s (< 30.000s)

Returns

    4.613152809402743E-4

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

And regressed input:

TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 1.0160389215696446, -1.7 ], [ 1.6395345349077723, 1.628 ], [ -1.590923084347253, -0.476 ], [ -1.8101012679501423, -1.348 ], [ 1.931914357570565, 0.04 ], [ 0.16813437360598255, 1.464 ], [ -1.2635462408421223, 1.808 ], [ -0.16041675895993715, 0.636 ], ... ],
    	[ [ 0.6191898796616264, 0.6 ], [ -0.020934762924807185, -1.832 ], [ 1.8483953866796063, -0.476 ], [ 0.055444966826467934, -0.3 ], [ -1.892936640971015, -1.588 ], [ -1.157818849946651, -0.28 ], [ 0.07874585764949749, -1.896 ], [ 0.19922297834515265, 1.704 ], ... ],
    	[ [ -1.1785726455649268, -1.064 ], [ -1.4439785453644205, 0.88 ], [ -1.204136220790901, 1.696 ], [ 1.8313398015619826, 0.064 ], [ 1.8370944480844544, 0.956 ], [ -1.2557039889374204, 1.964 ], [ 1.744319365066787, 0.088 ], [ -0.4427712032523162, 0.628 ], ... ],
    	[ [ -1.8762629303344658, 1.34 ], [ 0.20400778748152648, -1.412 ], [ -1.593577456941413, 1.152 ], [ 0.8416455858512898, -0.716 ], [ -1.6373865785907205, -1.404 ], [ -0.5183855989301455, 1.98 ], [ -1.1661343176731518, 1.156 ], [ -1.3422882442302781, -1.044 ], ... ],
    	[ [ -0.31434469231472956, 0.912 ], [ 0.3182687525252828, 1.864 ], [ -0.8913437049937237, 1.852 ], [ -1.9674742940925525, 1.652 ], [ 0.23481986224930138, 0.952 ], [ 0.9329697499476999, 1.74 ], [ 0.3811684452775488, -1.44 ], [ 0.22808568826796788, -0.436 ], ... ],
    	[ [ -1.8582239733635533, -1.176 ], [ -0.8111586862673265, -1.996 ], [ 0.3223291022475723, 1.6 ], [ 0.4925628102441324, -1.9 ], [ -0.9654605468561156, -1.608 ], [ 0.35797272465681107, 0.312 ], [ 0.025014601347874935, -1.124 ], [ 0.9852735907447512, 1.62 ], ... ],
    	[ [ 0.9311996437206822, 1.668 ], [ -1.9126387306308137, -0.448 ], [ -1.8791217569216605, -1.908 ], [ -1.605668977678079, 0.884 ], [ 1.9511061080158232, -0.984 ], [ -0.5682901775562356, 0.312 ], [ -1.6638286305431238, 0.984 ], [ 0.019296978378638863, 0.72 ], ... ],
    	[ [ 0.3327107997625883, -1.456 ], [ -1.38988313139722, -0.728 ], [ -0.26425901233917154, -1.284 ], [ 1.0573943517941415, -0.404 ], [ -1.2209211688849886, 0.376 ], [ -0.20280233112211873, -1.396 ], [ -0.7811080457117264, -1.664 ], [ 1.6711061518983206, -1.208 ], ... ],
    	...
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.02 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ 1.0160388946533203 ], [ 1.639534592628479 ], [ -1.5909230709075928 ], [ -1.8101012706756592 ], [ 1.9319143295288086 ], [ 0.16813437640666962 ], [ -1.2635462284088135 ], [ -0.16041675209999084 ], ... ],
    	[ [ 0.6191898584365845 ], [ -0.020934762433171272 ], [ 1.8483953475952148 ], [ 0.05544496700167656 ], [ -1.8929365873336792 ], [ -1.1578187942504883 ], [ 0.07874585688114166 ], [ 0.19922298192977905 ], ... ],
    	[ [ -1.178572654724121 ], [ -1.4439785480499268 ], [ -1.2041362524032593 ], [ 1.8313398361206055 ], [ 1.8370944261550903 ], [ -1.2557040452957153 ], [ 1.7443193197250366 ], [ -0.44277119636535645 ], ... ],
    	[ [ -1.876262903213501 ], [ 0.20400778949260712 ], [ -1.59357750415802 ], [ 0.8416455984115601 ], [ -1.6373865604400635 ], [ -0.5183855891227722 ], [ -1.1661343574523926 ], [ -1.3422882556915283 ], ... ],
    	[ [ -0.31434470415115356 ], [ 0.318268746137619 ], [ -0.8913437128067017 ], [ -1.967474341392517 ], [ 0.23481985926628113 ], [ 0.9329697489738464 ], [ 0.3811684548854828 ], [ 0.2280856817960739 ], ... ],
    	[ [ -1.8582239151000977 ], [ -0.8111586570739746 ], [ 0.3223291039466858 ], [ 0.49256280064582825 ], [ -0.9654605388641357 ], [ 0.3579727113246918 ], [ 0.02501460164785385 ], [ 0.9852735996246338 ], ... ],
    	[ [ 0.9311996698379517 ], [ -1.912638783454895 ], [ -1.8791217803955078 ], [ -1.6056690216064453 ], [ 1.951106071472168 ], [ -0.5682901740074158 ], [ -1.6638286113739014 ], [ 0.019296977669000626 ], ... ],
    	[ [ 0.33271080255508423 ], [ -1.3898831605911255 ], [ -0.26425901055336 ], [ 1.0573943853378296 ], [ -1.2209211587905884 ], [ -0.20280233025550842 ], [ -0.781108021736145 ], [ 1.6711061000823975 ], ... ],
    	...
    ]

TrainingTester.java:432 executed in 0.15 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -82.49469834355185], [42.0, 0.2607109936944505]; valueStats=DoubleSummaryStatistics{count=52, sum=5.126954, min=0.000000, average=0.098595, max=1.822682}
Plotting 42 points for GD
Plotting 6 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -82.49469834355185], [28.794, 0.2607109936944505]; valueStats=DoubleSummaryStatistics{count=52, sum=5.126954, min=0.000000, average=0.098595, max=1.822682}
Plotting 42 points for GD
Plotting 6 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "NonConverged", "value": 4.613152809402743E-4 }, "CjGD": { "type": "Converged", "value": 4.0988407816183775E-83 }, "GD": { "type": "Converged", "value": 3.2011178009361137E-83 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "NonConverged", "value": 4.613152809402743E-4 }, "CjGD": { "type": "Converged", "value": 4.0988407816183775E-83 }, "GD": { "type": "Converged", "value": 3.2011178009361137E-83 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "57.903",
      "gc_time": "1.497"
    },
    "created_on": 1586745936456,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Float",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgBandSelectLayerTest.Float",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 4.613152809402743E-4
        },
        "CjGD": {
          "type": "Converged",
          "value": 4.0988407816183775E-83
        },
        "GD": {
          "type": "Converged",
          "value": 3.2011178009361137E-83
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayer/Float/trainingTest/202004134536",
    "id": "fe7ccb04-b848-47c2-bfec-800bb8a18a73",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgBandSelectLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgBandSelectLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayer.java",
      "javaDoc": ""
    }
  }