1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 1408479387076581376

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -1.028, 1.764, -0.608 ], [ -0.128, 1.208, -1.72 ] ],
    	[ [ 0.7, 0.496, 1.524 ], [ 0.048, -0.384, 0.08 ] ]
    ]
    [
    	[ [ -0.384, -0.608, -0.128 ], [ 1.764, -1.028, -1.72 ] ],
    	[ [ 1.208, 1.524, 0.048 ], [ 0.08, 0.7, 0.496 ] ]
    ]
    [
    	[ [ 0.496, 0.7, -1.028 ], [ 0.048, -0.608, 1.208 ] ],
    	[ [ 1.764, 1.524, -0.128 ], [ -1.72, 0.08, -0.384 ] ]
    ]
    [
    	[ [ 0.08, 1.208, -1.028 ], [ 1.764, 0.496, -0.608 ] ],
    	[ [ 1.524, -0.128, -1.72 ], [ 0.7, -0.384, 0.048 ] ]
    ]
    [
    	[ [ 0.496, -1.72, -1.028 ], [ 1.764, 0.08, -0.608 ] ],
    	[ [ 0.7, 0.048, -0.384 ], [ 1.208, 1.524, -0.128 ] ]
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 1.39 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 551015479328
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 551049380409
Constructing line search parameters: GD
th(0)=6.348754967858289;dx=-0.35491685591903666
New Minimum: 6.348754967858289 > 5.625441380892864
END: th(2.154434690031884)=5.625441380892864; dx=-0.3152874878523754 evalInputDelta=0.7233135869654248
Fitness changed from 6.348754967858289 to 5.625441380892864
Iteration 1 complete. Error: 5.625441380892864 Total: 0.0792; Orientation: 0.0033; Line Search: 0.0269
th(0)=5.625441380892864;dx=-0.2809335642048333
New Minimum: 5.625441380892864 > 4.503504226472425
END: th(4.641588833612779)=4.503504226472425; dx=-0.19913884152845376 evalInputDelta=1.121937154420439
Fitness changed from 5.625441380892864 to 4.503504226472425
Iteration 2 complete. Error: 4.503504226472425 Total: 0.0261; Orientation: 0.0011; Line Search: 0.0184
th(0)=4.503504226472425;dx=-0.14367082183960722
New Minimum: 4.503504226472425 > 3.5458644308943468
END: th(10.000000000000002)=3.5458644308943468; dx=-0.04919854107464198 evalInputDelta=0.9576397955780784
Fitness changed from 4.503504226472425 to 3.5458644308943468
Iteration 3 complete. Error: 3.5458644308943468 Total: 0.0251; Orientation: 0.0010; Line Search: 0.0188
th(0)=3.5458644308943468;dx=-0.020786174232093904
New Minimum: 3.5458644308943468 > 3.357179346132849
WOLF (strong): th(21.544346900318843)=3.357179346132849; dx=0.0032114041979105857 evalInputDelta=0.1886850847614978
END: th(10.772173450159421)=3.385463036189157; dx=-0.008763586075308994 evalInputDelta=0.16040139470518966
Fitness changed from 3.5458644308943468 to 3.357179346132849
Iteration 4 complete. Error: 3.357179346132849 Total: 0.0309; Orientation: 0.0012; Line Search: 0.0245
th(0)=3.357179346132849;dx=-0.002662933092976458
New Minimum: 3.357179346132849 > 3.3417410728590546
WOLF (strong): th(23.2079441680639)=3.3417410728590546; dx=0.0014291079431895718 evalInputDelta=0.015438273273794323
New Minimum: 3.3417410728590546 > 3.337507554221825
END: th(11.60397208403195)=3.337507554221825; dx=-6.898703762979494E-4 evalInputDelta=0.019671791911024084
Fitness changed from 3.357179346132849 to 3.337507554221825
Iteration 5 complete. Error: 3.337507554221825 Total: 0.0326; Orientation: 0.0011; Line Search: 0.0241
th(0)=3.337507554221825;dx=-3.2339531895205815E-4
New Minimum: 3.337507554221825 > 3.3327700020265967
END: th(25.000000000000007)=3.3327700020265967; dx=-5.475549619588996E-5 evalInputDelta=0.004737552195228201
Fitness changed from 3.337507554221825 to 3.3327700020265967
Iteration 6 complete. Error: 3.3327700020265967 Total: 0.0235; Orientation: 0.0009; Line Search: 0.0176
th(0)=3.3327700020265967;dx=-1.0763615872089683E-4
Armijo: th(53.860867250797114)=3.333526998454155; dx=1.3206600961409315E-4 evalInputDelta=-7.569964275582208E-4
New Minimum: 3.3327700020265967 > 3.3315333763420427
WOLF (strong): th(26.930433625398557)=3.3315333763420427; dx=1.4976275950935624E-5 evalInputDelta=0.0012366256845539603
END: th(8.976811208466186)=3.3319900146317165; dx=-6.622551549319904E-5 evalInputDelta=7.799873948801661E-4
Fitness changed from 3.3327700020265967 to 3.3315333763420427
Iteration 7 complete. Error: 3.3315333763420427 Total: 0.0405; Orientation: 0.0008; Line Search: 0.0317
th(0)=3.3315333763420427;dx=-6.983020960653365E-5
New Minimum: 3.3315333763420427 > 3.3309535383446396
WOLF (strong): th(19.33995347338659)=3.3309535383446396; dx=1.0128846166892422E-5 evalInputDelta=5.79837997403132E-4
END: th(9.669976736693295)=3.331050120720545; dx=-3.0046726894923576E-5 evalInputDelta=4.832556214977579E-4
Fitness changed from 3.3315333763420427 to 3.3309535383446396
Iteration 8 complete. Error: 3.3309535383446396 Total: 0.0449; Orientation: 0.0016; Line Search: 0.0375
th(0)=3.3309535383446396;dx=-2.260189299861668E-5
New Minimum: 3.3309535383446396 > 3.330715764313993
END: th(20.833333333333346)=3.330715764313993; dx=-2.6970982640150527E-7 evalInputDelta=2.3777403064650926E-4
Fitness changed from 3.3309535383446396 to 3.330715764313993
Iteration 9 complete. Error: 3.330715764313993 Total: 0.0295; Orientation: 0.0009; Line Search: 0.0234
th(0)=3.330715764313993;dx=-1.0766896463390678E-5
New Minimum: 3.330715764313993 > 3.330687623193279
WOLF (strong): th(44.88405604233094)=3.330687623193279; dx=9.524612310590385E-6 evalInputDelta=2.814112071414243E-5
New Minimum: 3.330687623193279 > 3.3305878404786258
END: th(22.44202802116547)=3.3305878404786258; dx=-6.29891789842976E-7 evalInputDelta=1.2792383536730512E-4
Fitness changed from 3.330715764313993 to 3.3305878404786258
Iteration 10 complete. Error: 3.3305878404786258 Total: 0.0196; Orientation: 0.0007; Line Search: 0.0148
th(0)=3.3305878404786258;dx=-6.315247708954912E-6
Armijo: th(48.34988368346648)=3.3306041690093005; dx=6.958262173624438E-6 evalInputDelta=-1.6328530674680763E-5
New Minimum: 3.3305878404786258 > 3.3305157801280387
WOLF (strong): th(24.17494184173324)=3.3305157801280387; dx=3.4582105503653287E-7 evalInputDelta=7.20603505870443E-5
END: th(8.058313947244413)=3.3305459208510806; dx=-4.089675057515547E-6 evalInputDelta=4.191962754518386E-5
Fitness changed from 3.3305878404786258 to 3.3305157801280387
Iteration 11 complete. Error: 3.3305157801280387 Total: 0.0225; Orientation: 0.0005; Line Search: 0.0186
th(0)=3.3305157801280387;dx=-4.6021440992668666E-6
New Minimum: 3.3305157801280387 > 3.3304723985890106
END: th(17.36111111111112)=3.3304723985890106; dx=-3.9399633642799167E-7 evalInputDelta=4.338153902816444E-5
Fitness changed from 3.3305157801280387 to 3.3304723985890106
Iteration 12 complete. Error: 3.3304723985890106 Total: 0.0190; Orientation: 0.0006; Line Search: 0.0136
th(0)=3.3304723985890106;dx=-1.8060762141935708E-6
New Minimum: 3.3304723985890106 > 3.33044419120802
WOLF (strong): th(37.40338003527578)=3.33044419120802; dx=2.953632720999644E-7 evalInputDelta=2.8207380990519226E-5
END: th(18.70169001763789)=3.330448469778824; dx=-7.535316039171802E-7 evalInputDelta=2.3928810186646388E-5
Fitness changed from 3.3304723985890106 to 3.33044419120802
Iteration 13 complete. Error: 3.33044419120802 Total: 0.0199; Orientation: 0.0005; Line Search: 0.0162
th(0)=3.33044419120802;dx=-3.011198049170524E-6
Armijo: th(40.291569736222066)=3.3304991656973377; dx=5.748451975098455E-6 evalInputDelta=-5.4974489317682895E-5
New Minimum: 3.33044419120802 > 3.3304275600446545
WOLF (strong): th(20.145784868111033)=3.3304275600446545; dx=1.3623113998969145E-6 evalInputDelta=1.663116336558801E-5
END: th(6.715261622703678)=3.330428859326747; dx=-1.5548311921949493E-6 evalInputDelta=1.533188127300633E-5
Fitness changed from 3.33044419120802 to 3.3304275600446545
Iteration 14 complete. Error: 3.3304275600446545 Total: 0.0281; Orientation: 0.0008; Line Search: 0.0224
th(0)=3.3304275600446545;dx=-1.7172601083867402E-6
New Minimum: 3.3304275600446545 > 3.33041544931415
WOLF (strong): th(14.467592592592604)=3.33041544931415; dx=4.253149499197301E-8 evalInputDelta=1.2110730504666378E-5
END: th(7.233796296296302)=3.3304183221803996; dx=-8.369583847521737E-7 evalInputDelta=9.237864254885153E-6
Fitness changed from 3.3304275600446545 to 3.33041544931415
Iteration 15 complete. Error: 3.33041544931415 Total: 0.0228; Orientation: 0.0009; Line Search: 0.0180
th(0)=3.33041544931415;dx=-3.806423410953639E-7
New Minimum: 3.33041544931415 > 3.330410922207433
END: th(15.584741681364912)=3.330410922207433; dx=-2.0033777020681312E-7 evalInputDelta=4.527106716611939E-6
Fitness changed from 3.33041544931415 to 3.330410922207433
Iteration 16 complete. Error: 3.330410922207433 Total: 0.0130; Orientation: 0.0006; Line Search: 0.0092
th(0)=3.330410922207433;dx=-2.2419534225441603E-7
New Minimum: 3.330410922207433 > 3.3304052078995325
END: th(33.576308113518394)=3.3304052078995325; dx=-1.1626681170476468E-7 evalInputDelta=5.714307900639426E-6
Fitness changed from 3.330410922207433 to 3.3304052078995325
Iteration 17 complete. Error: 3.3304052078995325 Total: 0.0130; Orientation: 0.0007; Line Search: 0.0087
th(0)=3.3304052078995325;dx=-1.715630291122467E-7
Armijo: th(

...skipping 16420 bytes...

dx=-1.3723215422722298E-14 evalInputDelta=1.3296030942910875E-12
Fitness changed from 3.3303982701294332 to 3.3303982701279446
Iteration 50 complete. Error: 3.3303982701279446 Total: 0.0130; Orientation: 0.0004; Line Search: 0.0098
Low gradient: 3.7145126278406566E-7
th(0)=3.3303982701279446;dx=-1.3797604062387699E-13
Armijo: th(116.82984873397439)=3.3303982701959947; dx=1.3029159648235935E-12 evalInputDelta=-6.805001007137434E-11
Armijo: th(58.414924366987194)=3.3303982701409267; dx=5.8246916647342E-13 evalInputDelta=-1.298205987154688E-11
New Minimum: 3.3303982701279446 > 3.3303982701275956
WOLF (strong): th(19.471641455662397)=3.3303982701275956; dx=1.0217218492435847E-13 evalInputDelta=3.490541189421492E-13
New Minimum: 3.3303982701275956 > 3.330398270127419
END: th(4.867910363915599)=3.330398270127419; dx=-7.793900081090829E-14 evalInputDelta=5.258016244624741E-13
Fitness changed from 3.3303982701279446 to 3.330398270127419
Iteration 51 complete. Error: 3.330398270127419 Total: 0.0346; Orientation: 0.0004; Line Search: 0.0314
Low gradient: 2.1051271124333402E-7
th(0)=3.330398270127419;dx=-4.431560159501933E-14
New Minimum: 3.330398270127419 > 3.330398270127167
END: th(10.487594955985498)=3.330398270127167; dx=-3.7184601346819976E-15 evalInputDelta=2.517985819849855E-13
Fitness changed from 3.330398270127419 to 3.330398270127167
Iteration 52 complete. Error: 3.330398270127167 Total: 0.0104; Orientation: 0.0006; Line Search: 0.0071
Low gradient: 3.89297212570795E-8
th(0)=3.330398270127167;dx=-1.5155231971539078E-15
New Minimum: 3.330398270127167 > 3.330398270127138
END: th(22.594838388178566)=3.330398270127138; dx=-1.038055813484174E-15 evalInputDelta=2.886579864025407E-14
Fitness changed from 3.330398270127167 to 3.330398270127138
Iteration 53 complete. Error: 3.330398270127138 Total: 0.0106; Orientation: 0.0006; Line Search: 0.0073
Low gradient: 3.3309668431743716E-8
th(0)=3.330398270127138;dx=-1.109534011032704E-15
New Minimum: 3.330398270127138 > 3.330398270127106
END: th(48.679103639156)=3.330398270127106; dx=-2.0141635223415702E-16 evalInputDelta=3.197442310920451E-14
Fitness changed from 3.330398270127138 to 3.330398270127106
Iteration 54 complete. Error: 3.330398270127106 Total: 0.0198; Orientation: 0.0006; Line Search: 0.0164
Low gradient: 4.3490729826220976E-8
th(0)=3.330398270127106;dx=-1.8914435808173466E-15
Armijo: th(104.875949559855)=3.33039827012765; dx=1.2262508482765424E-14 evalInputDelta=-5.440092820663267E-13
Armijo: th(52.4379747799275)=3.3303982701271932; dx=5.185532272180291E-15 evalInputDelta=-8.704148513061227E-14
New Minimum: 3.330398270127106 > 3.3303982701270938
WOLF (strong): th(17.4793249266425)=3.3303982701270938; dx=4.675483309231152E-16 evalInputDelta=1.2434497875801753E-14
END: th(4.369831231660625)=3.3303982701270995; dx=-1.3016956067004388E-15 evalInputDelta=6.661338147750939E-15
Fitness changed from 3.330398270127106 to 3.3303982701270938
Iteration 55 complete. Error: 3.3303982701270938 Total: 0.0255; Orientation: 0.0005; Line Search: 0.0224
Low gradient: 2.8193509578400276E-8
th(0)=3.3303982701270938;dx=-7.948739823473481E-16
New Minimum: 3.3303982701270938 > 3.3303982701270884
END: th(9.414515995074405)=3.3303982701270884; dx=-3.7244309480496423E-16 evalInputDelta=5.329070518200751E-15
Fitness changed from 3.3303982701270938 to 3.3303982701270884
Iteration 56 complete. Error: 3.3303982701270884 Total: 0.0105; Orientation: 0.0005; Line Search: 0.0072
Low gradient: 1.7051577952713205E-8
th(0)=3.3303982701270884;dx=-2.9075631067745515E-16
New Minimum: 3.3303982701270884 > 3.3303982701270827
END: th(20.28295984964834)=3.3303982701270827; dx=-2.2976980827833806E-16 evalInputDelta=5.773159728050814E-15
Fitness changed from 3.3303982701270884 to 3.3303982701270827
Iteration 57 complete. Error: 3.3303982701270827 Total: 0.0103; Orientation: 0.0005; Line Search: 0.0068
Low gradient: 1.4564677494849596E-8
th(0)=3.3303982701270827;dx=-2.1212983052897829E-16
New Minimum: 3.3303982701270827 > 3.330398270127076
END: th(43.69831231660626)=3.330398270127076; dx=-1.1895108107153414E-16 evalInputDelta=6.661338147750939E-15
Fitness changed from 3.3303982701270827 to 3.330398270127076
Iteration 58 complete. Error: 3.330398270127076 Total: 0.0100; Orientation: 0.0005; Line Search: 0.0067
Low gradient: 1.2708297525282563E-8
th(0)=3.330398270127076;dx=-1.6150082599110294E-16
Armijo: th(94.14515995074406)=3.3303982701270898; dx=4.468268134146452E-16 evalInputDelta=-1.3766765505351941E-14
New Minimum: 3.330398270127076 > 3.3303982701270756
WOLF (strong): th(47.07257997537203)=3.3303982701270756; dx=1.4266300152594928E-16 evalInputDelta=4.440892098500626E-16
New Minimum: 3.3303982701270756 > 3.3303982701270742
END: th(15.690859991790676)=3.3303982701270742; dx=-6.011288165736689E-17 evalInputDelta=1.7763568394002505E-15
Fitness changed from 3.330398270127076 to 3.3303982701270742
Iteration 59 complete. Error: 3.3303982701270742 Total: 0.0165; Orientation: 0.0005; Line Search: 0.0134
Low gradient: 9.351772429675171E-9
th(0)=3.3303982701270742;dx=-8.745564757643264E-17
New Minimum: 3.3303982701270742 > 3.3303982701270725
END: th(33.80493308274723)=3.3303982701270725; dx=-3.6298083094598884E-17 evalInputDelta=1.7763568394002505E-15
Fitness changed from 3.3303982701270742 to 3.3303982701270725
Iteration 60 complete. Error: 3.3303982701270725 Total: 0.0102; Orientation: 0.0004; Line Search: 0.0071
Low gradient: 9.442635323658434E-9
th(0)=3.3303982701270725;dx=-8.916336185560202E-17
Armijo: th(72.83052052767711)=3.330398270127077; dx=2.2848467815728217E-16 evalInputDelta=-4.440892098500626E-15
New Minimum: 3.3303982701270725 > 3.330398270127072
WOLF (strong): th(36.415260263838555)=3.330398270127072; dx=6.966066050776752E-17 evalInputDelta=4.440892098500626E-16
New Minimum: 3.330398270127072 > 3.330398270127071
END: th(12.138420087946185)=3.330398270127071; dx=-3.622202053428895E-17 evalInputDelta=1.3322676295501878E-15
Fitness changed from 3.3303982701270725 to 3.330398270127071
Iteration 61 complete. Error: 3.330398270127071 Total: 0.0247; Orientation: 0.0005; Line Search: 0.0146
Low gradient: 6.112409192129648E-9
th(0)=3.330398270127071;dx=-3.736154613203101E-17
New Minimum: 3.330398270127071 > 3.3303982701270707
END: th(26.151433319651133)=3.3303982701270707; dx=-2.950345075164267E-17 evalInputDelta=4.440892098500626E-16
Fitness changed from 3.330398270127071 to 3.3303982701270707
Iteration 62 complete. Error: 3.3303982701270707 Total: 0.0115; Orientation: 0.0005; Line Search: 0.0075
Low gradient: 5.026566860590423E-9
th(0)=3.3303982701270707;dx=-2.526637440398586E-17
New Minimum: 3.3303982701270707 > 3.33039827012707
END: th(56.34155513791207)=3.33039827012707; dx=-1.1442698524866358E-17 evalInputDelta=8.881784197001252E-16
Fitness changed from 3.3303982701270707 to 3.33039827012707
Iteration 63 complete. Error: 3.33039827012707 Total: 0.0273; Orientation: 0.0005; Line Search: 0.0240
Low gradient: 4.69986286271162E-9
th(0)=3.33039827012707;dx=-2.208871092829587E-17
Armijo: th(121.38420087946187)=3.3303982701270756; dx=1.3206755697437827E-16 evalInputDelta=-5.773159728050814E-15
Armijo: th(60.692100439730936)=3.3303982701270707; dx=5.4989422224368006E-17 evalInputDelta=-8.881784197001252E-16
New Minimum: 3.33039827012707 > 3.330398270127069
WOLF (strong): th(20.23070014657698)=3.330398270127069; dx=3.603999979600333E-18 evalInputDelta=8.881784197001252E-16
END: th(5.057675036644245)=3.330398270127069; dx=-1.5665533170051036E-17 evalInputDelta=8.881784197001252E-16
Fitness changed from 3.33039827012707 to 3.330398270127069
Iteration 64 complete. Error: 3.330398270127069 Total: 0.0470; Orientation: 0.0005; Line Search: 0.0275
Low gradient: 3.971410086649308E-9
th(0)=3.330398270127069;dx=-1.577209807633986E-17
END: th(10.89643054985464)=3.330398270127069; dx=-6.038848349577467E-18 evalInputDelta=0.0
Fitness changed from 3.330398270127069 to 3.330398270127069
Static Iteration Total: 0.0105; Orientation: 0.0005; Line Search: 0.0071
Iteration 65 failed. Error: 3.330398270127069
Previous Error: 0.0 -> 3.330398270127069
Optimization terminated 65
Final threshold in iteration 65: 3.330398270127069 (> 0.0) after 1.381s (< 30.000s)

Returns

    3.330398270127069

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -2.114666666667231, 0.6773333333334438, -1.6946666666662125 ], [ 0.9226666666789263, 2.258666666676137, -0.6693333333550516 ] ],
    	[ [ 0.12533333333333554, -0.07866666666666454, 0.9493333333333333 ], [ 0.6586666666666812, 0.22666666666666907, 0.6906666666666557 ] ]
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ 0.053091023126679, 0.8661064807259734, 0.08080249614734755 ], [ 0.19970778712373427, 0.7596480498888759, 0.04064416298738974 ] ],
    	[ [ 0.24419652333032507, 0.1991330759506399, 0.5566704007190351 ], [ 0.372894050738625, 0.24208631424961954, 0.3850196350117555 ] ]
    ]

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 3.57 seconds (0.138 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 552415764584
Reset training subject: 552418557090
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=6.348754967858289}, derivative=-0.35491685591903666}
New Minimum: 6.348754967858289 > 6.3487549678227975
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=6.3487549678227975}, derivative=-0.35491685591737315}, evalInputDelta = -3.5491609651217004E-11
New Minimum: 6.3487549678227975 > 6.348754967609848
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=6.348754967609848}, derivative=-0.35491685590739197}, evalInputDelta = -2.4844126755851903E-10
New Minimum: 6.348754967609848 > 6.348754966119197
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=6.348754966119197}, derivative=-0.35491685583752386}, evalInputDelta = -1.739092425623312E-9
New Minimum: 6.348754966119197 > 6.34875495568464
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=6.34875495568464}, derivative=-0.3549168553484468}, evalInputDelta = -1.2173648755720023E-8
New Minimum: 6.34875495568464 > 6.348754882642752
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=6.348754882642752}, derivative=-0.3549168519249075}, evalInputDelta = -8.521553684914807E-8
New Minimum: 6.348754882642752 > 6.3487543713495524
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=6.3487543713495524}, derivative=-0.35491682796013035}, evalInputDelta = -5.965087366277544E-7
New Minimum: 6.3487543713495524 > 6.348750792298122
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=6.348750792298122}, derivative=-0.3549166602065958}, evalInputDelta = -4.175560166963521E-6
New Minimum: 6.348750792298122 > 6.348725738985474
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=6.348725738985474}, derivative=-0.3549154859272202}, evalInputDelta = -2.92288728154233E-5
New Minimum: 6.348725738985474 > 6.348550368117943
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=6.348550368117943}, derivative=-0.3549072657445447}, evalInputDelta = -2.045997403463673E-4
New Minimum: 6.348550368117943 > 6.347322885789496
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=6.347322885789496}, derivative=-0.354849713340204}, evalInputDelta = -0.0014320820687929725
New Minimum: 6.347322885789496 > 6.338736088078214
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=6.338736088078214}, derivative=-0.35444630124505894}, evalInputDelta = -0.010018879780075096
New Minimum: 6.338736088078214 > 6.278903612441816
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=6.278903612441816}, derivative=-0.3515956633870494}, evalInputDelta = -0.06985135541647303
New Minimum: 6.278903612441816 > 5.8741619445132995
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=5.8741619445132995}, derivative=-0.3303245509448193}, evalInputDelta = -0.47459302334498954
New Minimum: 5.8741619445132995 > 3.90595310338369
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=3.90595310338369}, derivative=-0.13548992296706158}, evalInputDelta = -2.4428018644745992
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=8.986028537551316}, derivative=0.13844171401840513}, evalInputDelta = 2.6372735696930274
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=4.761294257003764}, derivative=-0.24694990493175045}, evalInputDelta = -1.5874607108545247
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=4.91725536824367}, derivative=0.11294295418766542}, evalInputDelta = -1.431499599614619
4.91725536824367 <= 6.348754967858289
F(27.70372281466303) = LineSearchPoint{point=PointSample{avg=4.028151705639581}, derivative=0.08548415789745929}, evalInputDelta = -2.3206032622187083
Right bracket at 27.70372281466303
New Minimum: 3.90595310338369 > 3.6431871145654156
F(22.32628420498971) = LineSearchPoint{point=PointSample{avg=3.6431871145654156}, derivative=0.055183490784729346}, evalInputDelta = -2.7055678532928735
Right bracket at 22.32628420498971
New Minimum: 3.6431871145654156 > 3.513708937420504
F(19.322038272046722) = LineSearchPoint{point=PointSample{avg=3.513708937420504}, derivative=0.02962556859325962}, evalInputDelta = -2.835046030437785
Right bracket at 19.322038272046722
New Minimum: 3.513708937420504 > 3.481331642414274
F(17.833447329406003) = LineSearchPoint{point=PointSample{avg=3.481331642414274}, derivative=0.013416975716040006}, evalInputDelta = -2.867423325444015
Right bracket at 17.833447329406003
New Minimum: 3.481331642414274 > 3.4751769310472356
F(17.183843874057448) = LineSearchPoint{point=PointSample{avg=3.4751769310472356}, derivative=0.005433587196189044}, evalInputDelta = -2.8735780368110535
Right bracket at 17.183843874057448
New Minimum: 3.4751769310472356 > 3.4742012719625373
F(16.92473523179186) = LineSearchPoint{point=PointSample{avg=3.4742012719625373}, derivative=0.0020808770859770077}, evalInputDelta = -2.874553695895752
Right bracket at 16.92473523179186
Converged to right
Fitness changed from 6.348754967858289 to 3.4742012719625373
Iteration 1 complete. Error: 3.4742012719625373 Total: 0.1205; Orientation: 0.0004; Line Search: 0.1117
F(0.0) = LineSearchPoint{point=PointSample{avg=3.4742012719625373}, derivative=-0.01572148571841679}
New Minimum: 3.4742012719625373 > 3.3491735402666505
F(16.92473523179186) = LineSearchPoint{point=PointSample{avg=3.3491735402666505}, derivative=0.0016895080787244813}, evalInputDelta = -0.12502773169588677
3.3491735402666505 <= 3.4742012719625373
New Minimum: 3.3491735402666505 > 3.347883414898824
F(15.282412154916136) = LineSearchPoint{point=PointSample{avg=3.347883414898824}, derivative=-1.2114951034444178E-4}, evalInputDelta = -0.12631785706371312
Left bracket at 15.282412154916136
Converged to left
Fitness changed from 3.4742012719625373 to 3.347883414898824
Iteration 2 complete. Error: 3.347883414898824 Total: 0.0148; Orientation: 0.0004; Line Search: 0.0119
F(0.0) = LineSearchPoint{point=PointSample{avg=3.347883414898824}, derivative=-0.0010295442464504233}
New Minimum: 3.347883414898824 > 3.3379893658957784
F(15.282412154916136) = LineSearchPoint{point=PointSample{avg=3.3379893658957784}, derivative=-2.615953840772406E-4}, evalInputDelta = -0.00989404900304569
F(106.97688508441296) = LineSearchPoint{point=PointSample{avg=3.531526983811127}, derivative=0.00445055047358737}, evalInputDelta = 0.18364356891230305
F(8.228991160339458) = LineSearchPoint{point=PointSample{avg=3.34109669163266}, derivative=-6.187724314483812E-4}, evalInputDelta = -0.006786723266164252
F(57.6029381223762) = LineSearchPoint{point=PointSample{avg=3.3732187694299953}, derivative=0.0019353753249169096}, evalInputDelta = 0.025335354531171195
F(4.430995240182785) = LineSearchPoint{point=PointSample{avg=3.343808882429927}, derivative=-8.092151324851532E-4}, evalInputDelta = -0.004074532468897107
F(31.016966681279495) = LineSearchPoint{point=PointSample{avg=3.3402236887080328}, derivative=5.481092444281608E-4}, evalInputDelta = -0.0076597261907913605
3.3402236887080328 <= 3.347883414898824
New Minimum: 3.3379893658957784 > 3.3373199404805307
F(20.24103503949548) = LineSearchPoint{point=PointSample{avg=3.3373199404805307}, derivative=-8.109316174595728E-6}, evalInputDelta = -0.010563474418293417
Left bracket at 20.24103503949548
Converged to left
Fitness changed from 3.347883414898824 to 3.3373199404805307
Iteration 3 complete. Error: 3.3373199404805307 Total: 0.0327; Orientation: 0.0005; Line Search: 0.0294
F(0.0) = LineSearchPoint{point=PointSample{avg=3.3373199404805307}, derivative=-3.825417326954677E-4}
New Minimum: 3.3373199404805307 > 3.3340238123034296
F(20.24103503949548) = LineSearchPoint{point=PointSample{avg=3.3340238123034296}, derivative=5.3727794807089646E-5}, evalInputDelta = -0.003296128177101121
3.3340238123034296 <= 3.3373199404805307
New Minimum: 3.3340238123034296 > 3.3339554453792486
F(17.748295783763826) = LineSearchPoint{point=PointSample{avg=3.3339554453792486}, derivative=1.068678575022938E-6}, evalInputDelta = -0.0033644951012821167
Right bracket at 17.748295783763826
Converged to right
Fitness changed from 3.3373199404805307 to 3.3339554453792486
Iteration 4 complete. Error: 3.333

...skipping 102225 bytes...

{point=PointSample{avg=3.330398270127069}, derivative=-4.120101987828304E-18}, evalInputDelta = 4.440892098500626E-16
F(0.011411108124883992) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1255680818705625E-18}, evalInputDelta = 4.440892098500626E-16
F(8.777775480679993E-4) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.125988563904415E-18}, evalInputDelta = 0.0
F(0.006144442836475995) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.125778325686807E-18}, evalInputDelta = 0.0
F(0.04301109985533197) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1243066864792334E-18}, evalInputDelta = 4.440892098500626E-16
F(0.003308546142717844) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.125891543582364E-18}, evalInputDelta = 4.440892098500626E-16
F(2.545035494398341E-4) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126013444211246E-18}, evalInputDelta = 0.0
F(0.001781524846078839) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1259524898083195E-18}, evalInputDelta = 4.440892098500626E-16
F(1.370403727752953E-4) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126018137269155E-18}, evalInputDelta = 0.0
F(9.592826094270672E-4) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1259852974077795E-18}, evalInputDelta = 0.0
F(0.00671497826598947) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.125755558374814E-18}, evalInputDelta = 4.440892098500626E-16
F(5.165367896914977E-4) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260029645020246E-18}, evalInputDelta = 0.0
F(0.003615757527840484) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.125879270732236E-18}, evalInputDelta = 4.440892098500626E-16
F(2.78135194449268E-4) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126012497801337E-18}, evalInputDelta = 4.440892098500626E-16
F(2.1395014957635998E-5) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260227555756984E-18}, evalInputDelta = 0.0
F(1.49765104703452E-4) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.12601761513997E-18}, evalInputDelta = 4.440892098500626E-16
F(1.1520392669496307E-5) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1260231376901565E-18}, evalInputDelta = 4.440892098500626E-16
F(8.861840514997159E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1260235543547184E-18}, evalInputDelta = 4.440892098500626E-16
F(6.816800396151661E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(4.771760277306163E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.1260235767620944E-18}, evalInputDelta = 4.440892098500626E-16
F(3.670584828697049E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(2.5694093800879346E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023592824864E-18}, evalInputDelta = 4.440892098500626E-16
F(1.9764687539137958E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(1.3835281277396572E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023588600401E-18}, evalInputDelta = 4.440892098500626E-16
F(1.0642524059535824E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260235978353695E-18}, evalInputDelta = 0.0
F(7.449766841675077E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260235978353695E-18}, evalInputDelta = 0.0
F(5.214836789172553E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023571537091E-18}, evalInputDelta = 4.440892098500626E-16
F(4.0114129147481184E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(2.8079890403236827E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.12602358821264E-18}, evalInputDelta = 4.440892098500626E-16
F(2.159991569479756E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(1.5119940986358292E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023588600401E-18}, evalInputDelta = 4.440892098500626E-16
F(1.1630723835660224E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(8.141506684962157E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(5.699054679473509E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023566924868E-18}, evalInputDelta = 4.440892098500626E-16
F(4.3838882149796223E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(3.068721750485736E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023580375136E-18}, evalInputDelta = 4.440892098500626E-16
F(2.360555192681335E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
F(1.6523886348769345E-7) = LineSearchPoint{point=PointSample{avg=3.330398270127069}, derivative=-4.126023588600401E-18}, evalInputDelta = 4.440892098500626E-16
F(1.271068180674565E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260235978353695E-18}, evalInputDelta = 0.0
Loops = 52
F(2.54213636134913E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 2.54213636134913E-8
F(1.271068180674565E-8) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 1.271068180674565E-8
F(6.355340903372825E-9) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 6.355340903372825E-9
F(3.1776704516864124E-9) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 3.1776704516864124E-9
F(1.5888352258432062E-9) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.1260235978353695E-18}, evalInputDelta = 0.0
Right bracket at 1.5888352258432062E-9
F(3.1776704516864124E-9) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 3.1776704516864124E-9
F(1.5888352258432062E-9) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 1.5888352258432062E-9
F(7.944176129216031E-10) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 7.944176129216031E-10
F(3.9720880646080155E-10) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 3.9720880646080155E-10
F(1.9860440323040077E-10) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 1.9860440323040077E-10
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=3.3303982701270685}, derivative=-4.126023597835369E-18}, evalInputDelta = 0.0
Right bracket at 1.0E-10
Converged to right
Fitness changed from 3.3303982701270685 to 3.3303982701270685
Static Iteration Total: 0.2630; Orientation: 0.0004; Line Search: 0.2604
Iteration 89 failed. Error: 3.3303982701270685
Previous Error: 0.0 -> 3.3303982701270685
Optimization terminated 89
Final threshold in iteration 89: 3.3303982701270685 (> 0.0) after 3.566s (< 30.000s)

Returns

    3.3303982701270685

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -2.1146666666667637, 0.6773333333333583, -1.6946666666665786 ], [ 0.9226666666659031, 2.258666666666078, -0.669333333331971 ] ],
    	[ [ 0.125333333333335, -0.0786666666666651, 0.949333333333335 ], [ 0.6586666666666674, 0.22666666666666743, 0.6906666666666674 ] ]
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.03 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ 0.053091023126708, 0.8661064807259676, 0.08080249614732432 ], [ 0.19970778712299148, 0.7596480498883026, 0.040644162988705976 ] ],
    	[ [ 0.24419652333032474, 0.19913307595063967, 0.5566704007190356 ], [ 0.37289405073862025, 0.24208631424961938, 0.3850196350117604 ] ]
    ]

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 7.51 seconds (0.385 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 556023132867
Reset training subject: 556025854341
Adding measurement 677b5b6 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 6.348754967858289 < 6.348754967858289. Total: 1
th(0)=6.348754967858289;dx=-0.35491685591903666
Adding measurement bba9b7b to history. Total: 1
New Minimum: 6.348754967858289 > 5.625441380892864
END: th(2.154434690031884)=5.625441380892864; dx=-0.3152874878523754 evalInputDelta=0.7233135869654248
Fitness changed from 6.348754967858289 to 5.625441380892864
Iteration 1 complete. Error: 5.625441380892864 Total: 0.0258; Orientation: 0.0072; Line Search: 0.0111
Non-optimal measurement 5.625441380892864 < 5.625441380892864. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 5.625441380892864 < 5.625441380892864. Total: 2
th(0)=5.625441380892864;dx=-0.2809335642048333
Adding measurement 220e4414 to history. Total: 2
New Minimum: 5.625441380892864 > 4.503504226472425
END: th(4.641588833612779)=4.503504226472425; dx=-0.19913884152845376 evalInputDelta=1.121937154420439
Fitness changed from 5.625441380892864 to 4.503504226472425
Iteration 2 complete. Error: 4.503504226472425 Total: 0.1903; Orientation: 0.0116; Line Search: 0.0071
Non-optimal measurement 4.503504226472425 < 4.503504226472425. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 4.503504226472425 < 4.503504226472425. Total: 3
th(0)=4.503504226472425;dx=-0.14367082183960722
Adding measurement 60ad3dc2 to history. Total: 3
New Minimum: 4.503504226472425 > 3.5458644308943468
END: th(10.000000000000002)=3.5458644308943468; dx=-0.04919854107464198 evalInputDelta=0.9576397955780784
Fitness changed from 4.503504226472425 to 3.5458644308943468
Iteration 3 complete. Error: 3.5458644308943468 Total: 0.0195; Orientation: 0.0012; Line Search: 0.0161
Non-optimal measurement 3.5458644308943468 < 3.5458644308943468. Total: 4
Rejected: LBFGS Orientation magnitude: 3.759e+00, gradient 1.442e-01, dot -0.981; [947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.5458644308943468, 4.503504226472425, 5.625441380892864, 6.348754967858289
LBFGS Accumulation History: 3 points
Removed measurement 60ad3dc2 to history. Total: 3
Adding measurement 46a2b0df to history. Total: 3
th(0)=3.5458644308943468;dx=-0.020786174232093904
Adding measurement 548036dd to history. Total: 4
New Minimum: 3.5458644308943468 > 3.357179346132849
WOLF (strong): th(21.544346900318843)=3.357179346132849; dx=0.0032114041979105857 evalInputDelta=0.1886850847614978
Non-optimal measurement 3.385463036189157 < 3.357179346132849. Total: 5
END: th(10.772173450159421)=3.385463036189157; dx=-0.008763586075308994 evalInputDelta=0.16040139470518966
Fitness changed from 3.5458644308943468 to 3.357179346132849
Iteration 4 complete. Error: 3.357179346132849 Total: 0.0801; Orientation: 0.0609; Line Search: 0.0167
Non-optimal measurement 3.357179346132849 < 3.357179346132849. Total: 5
Rejected: LBFGS Orientation magnitude: 8.967e-01, gradient 5.160e-02, dot -0.963; [2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.357179346132849, 3.5458644308943468, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 7.749e-01, gradient 5.160e-02, dot -0.978; [7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.357179346132849, 3.5458644308943468, 4.503504226472425, 5.625441380892864
LBFGS Accumulation History: 3 points
Removed measurement 548036dd to history. Total: 4
Removed measurement 46a2b0df to history. Total: 3
Adding measurement 30a886ab to history. Total: 3
th(0)=3.357179346132849;dx=-0.002662933092976458
Adding measurement 7359b4d8 to history. Total: 4
New Minimum: 3.357179346132849 > 3.3417410728590546
WOLF (strong): th(23.2079441680639)=3.3417410728590546; dx=0.0014291079431895718 evalInputDelta=0.015438273273794323
Adding measurement 57f5521d to history. Total: 5
New Minimum: 3.3417410728590546 > 3.337507554221825
END: th(11.60397208403195)=3.337507554221825; dx=-6.898703762979494E-4 evalInputDelta=0.019671791911024084
Fitness changed from 3.357179346132849 to 3.337507554221825
Iteration 5 complete. Error: 3.337507554221825 Total: 0.0976; Orientation: 0.0847; Line Search: 0.0103
Non-optimal measurement 3.337507554221825 < 3.337507554221825. Total: 6
Rejected: LBFGS Orientation magnitude: 3.782e-01, gradient 1.798e-02, dot -0.995; [5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.337507554221825, 3.3417410728590546, 3.357179346132849, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 3.546e-01, gradient 1.798e-02, dot -0.996; [947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.337507554221825, 3.3417410728590546, 3.357179346132849, 4.503504226472425, 5.625441380892864
Rejected: LBFGS Orientation magnitude: 3.986e-01, gradient 1.798e-02, dot -0.996; [7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.337507554221825, 3.3417410728590546, 3.357179346132849, 4.503504226472425
LBFGS Accumulation History: 3 points
Removed measurement 57f5521d to history. Total: 5
Removed measurement 7359b4d8 to history. Total: 4
Removed measurement 30a886ab to history. Total: 3
Adding measurement 33033be2 to history. Total: 3
th(0)=3.337507554221825;dx=-3.2339531895205815E-4
Adding measurement 4474193 to history. Total: 4
New Minimum: 3.337507554221825 > 3.3327700020265967
END: th(25.000000000000007)=3.3327700020265967; dx=-5.475549619588995E-5 evalInputDelta=0.004737552195228201
Fitness changed from 3.337507554221825 to 3.3327700020265967
Iteration 6 complete. Error: 3.3327700020265967 Total: 0.0953; Orientation: 0.0730; Line Search: 0.0059
Non-optimal measurement 3.3327700020265967 < 3.3327700020265967. Total: 5
Rejected: LBFGS Orientation magnitude: 2.740e-01, gradient 1.037e-02, dot -0.937; [947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.3327700020265967, 3.337507554221825, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 2.410e-01, gradient 1.037e-02, dot -0.925; [0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4

...skipping 107366 bytes...

895E-17 evalInputDelta=1.3322676295501878E-15
Fitness changed from 3.3303982701270725 to 3.330398270127071
Iteration 61 complete. Error: 3.330398270127071 Total: 0.1136; Orientation: 0.0799; Line Search: 0.0313
Non-optimal measurement 3.330398270127071 < 3.330398270127071. Total: 6
Rejected: LBFGS Orientation magnitude: 2.341e-07, gradient 6.112e-09, dot -0.953; [7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.330398270127071, 3.330398270127072, 3.3303982701270725, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 2.149e-07, gradient 6.112e-09, dot -0.951; [5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.330398270127071, 3.330398270127072, 3.3303982701270725, 4.503504226472425, 5.625441380892864
Rejected: LBFGS Orientation magnitude: 2.451e-07, gradient 6.112e-09, dot -0.964; [5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.330398270127071, 3.330398270127072, 3.3303982701270725, 4.503504226472425
LBFGS Accumulation History: 3 points
Removed measurement 15cdd16b to history. Total: 5
Removed measurement 6fcf4f6e to history. Total: 4
Removed measurement 5e860766 to history. Total: 3
Adding measurement 585f1776 to history. Total: 3
th(0)=3.330398270127071;dx=-3.7361546132031014E-17
Adding measurement 2aa971e2 to history. Total: 4
New Minimum: 3.330398270127071 > 3.3303982701270707
END: th(26.151433319651133)=3.3303982701270707; dx=-2.9503450751642663E-17 evalInputDelta=4.440892098500626E-16
Fitness changed from 3.330398270127071 to 3.3303982701270707
Iteration 62 complete. Error: 3.3303982701270707 Total: 0.3452; Orientation: 0.1147; Line Search: 0.2281
Non-optimal measurement 3.3303982701270707 < 3.3303982701270707. Total: 5
Rejected: LBFGS Orientation magnitude: 7.256e-07, gradient 5.027e-09, dot -0.979; [2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.3303982701270707, 3.330398270127071, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 7.083e-07, gradient 5.027e-09, dot -0.984; [0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.3303982701270707, 3.330398270127071, 4.503504226472425, 5.625441380892864
LBFGS Accumulation History: 3 points
Removed measurement 2aa971e2 to history. Total: 4
Removed measurement 585f1776 to history. Total: 3
Adding measurement 74056a7e to history. Total: 3
th(0)=3.3303982701270707;dx=-2.526637440398586E-17
Adding measurement 77e2b886 to history. Total: 4
New Minimum: 3.3303982701270707 > 3.33039827012707
END: th(56.34155513791207)=3.33039827012707; dx=-1.1442698524866358E-17 evalInputDelta=8.881784197001252E-16
Fitness changed from 3.3303982701270707 to 3.33039827012707
Iteration 63 complete. Error: 3.33039827012707 Total: 0.0676; Orientation: 0.0591; Line Search: 0.0067
Non-optimal measurement 3.33039827012707 < 3.33039827012707. Total: 5
Rejected: LBFGS Orientation magnitude: 4.469e-07, gradient 4.700e-09, dot -0.717; [947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.33039827012707, 3.3303982701270707, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 4.170e-07, gradient 4.700e-09, dot -0.705; [0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.33039827012707, 3.3303982701270707, 4.503504226472425, 5.625441380892864
LBFGS Accumulation History: 3 points
Removed measurement 77e2b886 to history. Total: 4
Removed measurement 74056a7e to history. Total: 3
Adding measurement 75ad6818 to history. Total: 3
th(0)=3.33039827012707;dx=-2.208871092829587E-17
Non-optimal measurement 3.3303982701270756 < 3.33039827012707. Total: 4
Armijo: th(121.38420087946187)=3.3303982701270756; dx=1.3206755697437825E-16 evalInputDelta=-5.773159728050814E-15
Non-optimal measurement 3.3303982701270707 < 3.33039827012707. Total: 4
Armijo: th(60.692100439730936)=3.3303982701270707; dx=5.4989422224368006E-17 evalInputDelta=-8.881784197001252E-16
Adding measurement 3c3a61c7 to history. Total: 4
New Minimum: 3.33039827012707 > 3.330398270127069
WOLF (strong): th(20.23070014657698)=3.330398270127069; dx=3.603999979600333E-18 evalInputDelta=8.881784197001252E-16
Non-optimal measurement 3.330398270127069 < 3.330398270127069. Total: 5
END: th(5.057675036644245)=3.330398270127069; dx=-1.5665533170051033E-17 evalInputDelta=8.881784197001252E-16
Fitness changed from 3.33039827012707 to 3.330398270127069
Iteration 64 complete. Error: 3.330398270127069 Total: 0.1245; Orientation: 0.1060; Line Search: 0.0161
Non-optimal measurement 3.330398270127069 < 3.330398270127069. Total: 5
Rejected: LBFGS Orientation magnitude: 5.857e-08, gradient 3.971e-09, dot -0.817; [7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00, 0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.330398270127069, 3.33039827012707, 4.503504226472425, 5.625441380892864, 6.348754967858289
Rejected: LBFGS Orientation magnitude: 5.224e-08, gradient 3.971e-09, dot -0.831; [0a899bf5-7146-4a0b-9709-48c50aa3f5a1 = 1.000/1.000e+00, 7f0370bf-856f-4e4b-acef-73a631b8e7d0 = 1.000/1.000e+00, 947ee811-2fa7-4d30-8a10-204faa75d23e = 1.000/1.000e+00, 2b0c85ea-c281-4ea7-ba1f-860909d1f811 = 1.000/1.000e+00, 5056ffd1-8418-4a29-a24a-a134565e5b81 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.330398270127069, 3.33039827012707, 4.503504226472425, 5.625441380892864
LBFGS Accumulation History: 3 points
Removed measurement 3c3a61c7 to history. Total: 4
Removed measurement 75ad6818 to history. Total: 3
Adding measurement c30e2ad to history. Total: 3
th(0)=3.330398270127069;dx=-1.577209807633986E-17
Non-optimal measurement 3.330398270127069 < 3.330398270127069. Total: 4
END: th(10.89643054985464)=3.330398270127069; dx=-6.038848349577467E-18 evalInputDelta=0.0
Fitness changed from 3.330398270127069 to 3.330398270127069
Static Iteration Total: 0.1024; Orientation: 0.0942; Line Search: 0.0060
Iteration 65 failed. Error: 3.330398270127069
Previous Error: 0.0 -> 3.330398270127069
Optimization terminated 65
Final threshold in iteration 65: 3.330398270127069 (> 0.0) after 7.509s (< 30.000s)

Returns

    3.330398270127069

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -2.114666666667231, 0.6773333333334438, -1.6946666666662125 ], [ 0.9226666666789263, 2.258666666676137, -0.6693333333550516 ] ],
    	[ [ 0.12533333333333554, -0.07866666666666454, 0.9493333333333333 ], [ 0.6586666666666812, 0.22666666666666907, 0.6906666666666557 ] ]
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ 0.053091023126679, 0.8661064807259734, 0.08080249614734755 ], [ 0.19970778712373427, 0.7596480498888759, 0.04064416298738974 ] ],
    	[ [ 0.24419652333032507, 0.1991330759506399, 0.5566704007190351 ], [ 0.372894050738625, 0.24208631424961954, 0.3850196350117555 ] ]
    ]

TrainingTester.java:432 executed in 0.10 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, 0.5224961722978707], [88.0, 0.7501566035417773]; valueStats=DoubleSummaryStatistics{count=216, sum=726.987986, min=3.330398, average=3.365685, max=5.625441}
Plotting 64 points for GD
Plotting 88 points for CjGD
Plotting 64 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, 0.5224961722978707], [7.381, 0.7501566035417773]; valueStats=DoubleSummaryStatistics{count=216, sum=726.987986, min=3.330398, average=3.365685, max=5.625441}
Plotting 64 points for GD
Plotting 88 points for CjGD
Plotting 64 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "NonConverged", "value": 3.330398270127069 }, "CjGD": { "type": "NonConverged", "value": 3.3303982701270685 }, "GD": { "type": "NonConverged", "value": 3.330398270127069 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "NonConverged", "value": 3.330398270127069 }, "CjGD": { "type": "NonConverged", "value": 3.3303982701270685 }, "GD": { "type": "NonConverged", "value": 3.330398270127069 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "13.234",
      "gc_time": "0.757"
    },
    "created_on": 1586735138809,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Basic",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.ImgPixelSoftmaxLayerTest.Basic",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/ImgPixelSoftmaxLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 3.330398270127069
        },
        "CjGD": {
          "type": "NonConverged",
          "value": 3.3303982701270685
        },
        "GD": {
          "type": "NonConverged",
          "value": 3.330398270127069
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/ImgPixelSoftmaxLayer/Basic/trainingTest/202004124538",
    "id": "85fe35e9-c8d8-4df9-bc37-27ea7b500c16",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgPixelSoftmaxLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.ImgPixelSoftmaxLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/ImgPixelSoftmaxLayer.java",
      "javaDoc": ""
    }
  }