Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
This is a network apply the following layout:
LayerTests.java:203 executed in 0.07 seconds (0.000 gc):
return Graphviz.fromGraph((Graph) TestUtil.toGraph(((DAGNetwork) layer).addRef())).height(400).width(600)
.render(Format.PNG).toImage();
executing command [/bin/sh, -c, dot -Tsvg /tmp/GraphvizJava/DotEngine382979879970925908/dotfile.dot -ooutfile.svg]
Returns
Using Seed 7901512952106201088
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.02 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -0.42, -1.672, -1.484 ], [ 0.228, 1.068, -1.608 ], [ 1.792, -1.996, 0.684 ], [ -0.268, 1.256, -0.224 ], [ 0.364, -1.264, -1.396 ], [ -0.184, -0.336, -1.428 ], [ -0.056, -0.804, -1.292 ], [ -0.976, 0.984, 0.468 ], ... ],
[ [ -1.128, -0.628, 0.66 ], [ 0.096, -1.76, 0.644 ], [ -1.724, -0.484, -1.924 ], [ 0.504, 1.196, 0.0 ], [ -1.48, 0.048, 0.096 ], [ -1.472, -0.716, -1.172 ], [ 1.828, -1.492, -1.42 ], [ -0.848, -0.828, -0.78 ], ... ],
[ [ -1.86, 1.636, 1.82 ], [ 0.216, 0.592, -1.244 ], [ -0.1, 1.016, -1.968 ], [ -1.672, 0.764, -0.412 ], [ 0.988, -0.96, 0.084 ], [ -0.9, -0.352, 1.268 ], [ -0.144, -1.152, -0.128 ], [ -0.608, -0.732, -1.132 ], ... ],
[ [ -0.804, -1.74, 0.288 ], [ 0.288, -1.348, -1.196 ], [ 1.828, 1.816, -1.512 ], [ 0.676, -1.548, 0.112 ], [ -0.068, -0.956, 0.852 ], [ -0.692, 1.7, 0.688 ], [ 1.008, 1.808, -1.44 ], [ 1.756, 1.1, -0.276 ], ... ],
[ [ 1.152, -0.124, 0.596 ], [ -0.64, -1.512, 0.108 ], [ -1.844, -1.956, -0.756 ], [ -0.884, -0.856, -1.624 ], [ -1.04, 1.436, -0.64 ], [ 0.124, 0.0, -1.944 ], [ 0.036, -0.804, 0.344 ], [ -1.888, -1.136, 1.336 ], ... ],
[ [ 1.712, -1.468, 1.6 ], [ 0.956, 1.608, -1.868 ], [ 1.22, -0.504, -1.1 ], [ 1.472, -0.508, -1.028 ], [ 1.636, 1.86, 0.84 ], [ 0.404, 1.344, -1.92 ], [ 1.072, -0.432, 1.72 ], [ -1.88, 1.38, -0.68 ], ... ],
[ [ 1.488, 0.804, -0.576 ], [ -0.732, -1.388, 0.672 ], [ -1.136, -0.236, -0.78 ], [ 1.792, -0.056, -0.692 ], [ 0.816, 0.076, -1.624 ], [ -1.68, -1.46, 1.352 ], [ -1.952, -0.756, 0.776 ], [ -0.608, 0.2, -0.952 ], ... ],
[ [ -1.316, -1.42, -0.8 ], [ -1.776, -1.444, -1.452 ], [ -1.064, 0.072, 1.204 ], [ -0.58, 0.112, 0.24 ], [ 1.672, 0.812, -0.54 ], [ 0.812, 1.692, -1.164 ], [ -1.908, -0.304, 1.26 ], [ 0.352, 1.444, 1.06 ], ... ],
...
]
[
[ [ 0.704, 1.104, -0.876 ], [ -1.32, -1.376, -1.992 ], [ 0.236, -1.652, -0.088 ], [ -1.34, 1.568, -1.256 ], [ -1.192, 0.968, 0.796 ], [ 1.012, 0.512, -0.28 ], [ -0.932, -0.976, 1.764 ], [ -0.52, -1.584, -1.76 ], ... ],
[ [ -0.792, 1.932, -1.576 ], [ 1.976, 1.864, 0.804 ], [ -0.42, -0.32, 1.984 ], [ -0.272, -1.88, 1.652 ], [ 1.072, -0.496, -1.2 ], [ 0.636, 1.46, 0.236 ], [ 0.684, 1.744, -1.764 ], [ -1.824, -0.34, 1.592 ], ... ],
[ [ -0.58, 0.136, 0.764 ], [ -1.568, -1.388, -0.708 ], [ 0.228, 1.852, 0.576 ], [ 1.684, -0.688, 1.372 ], [ 1.528, 0.24, -1.808 ], [ -1.916, -1.404, -1.036 ], [ -0.876, 0.62, -0.868 ], [ -1.884, 1.36, 1.86 ], ... ],
[ [ 0.952, -1.936, 0.976 ], [ -0.196, 1.772, -1.22 ], [ -1.888, -0.024, 0.296 ], [ 1.968, 0.048, -0.72 ], [ 0.06, -0.944, 1.356 ], [ 0.512, -0.2, -1.436 ], [ 1.1, -1.96, -0.14 ], [ 0.912, 1.368, -1.128 ], ... ],
[ [ -0.272, -1.988, -0.448 ], [ 0.028, -1.52, -1.796 ], [ 1.46, 0.02, 1.052 ], [ 1.544, -0.436, 1.832 ], [ -1.876, 1.5, -0.968 ], [ 1.076, 0.68, 1.404 ], [ -0.952, 0.376, -0.324 ], [ -0.424, -1.98, 1.428 ], ... ],
[ [ 1.648, 0.324, -0.684 ], [ -1.528, 0.772, 1.884 ], [ 1.516, 1.396, 1.796 ], [ -0.692, -1.828, 0.688 ], [ 0.428, -0.504, -0.204 ], [ 1.768, -0.784, -0.188 ], [ 1.676, -1.8, 1.736 ], [ -0.924, -1.716, 0.068 ], ... ],
[ [ -0.632, 0.216, 0.168 ], [ -1.512, -1.536, 0.156 ], [ 1.216, 1.06, 0.396 ], [ 1.844, -0.932, 1.548 ], [ 0.172, -0.12, -1.688 ], [ 1.572, -0.38, 1.472 ], [ 1.252, 0.448, -1.676 ], [ 1.7, -0.944, 0.188 ], ... ],
[ [ -1.62, 0.36, -1.852 ], [ 1.6, -0.708, 1.232 ], [ 0.52, -1.352, 0.356 ], [ -0.964, -0.624, 1.84 ], [ 0.088, 1.388, 1.192 ], [ 1.376, 0.416, 1.844 ], [ 1.248, 1.692, 1.18 ], [ -1.64, 0.884, -0.16 ], ... ],
...
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 30.25 seconds (0.729 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11461679402933
Reset training subject: 11461847976640
Constructing line search parameters: GD
th(0)=5.793098090202113;dx=-5.044331692212345E-5
New Minimum: 5.793098090202113 > 5.792989413940235
WOLFE (weak): th(2.154434690031884)=5.792989413940235; dx=-5.0442787798107533E-5 evalInputDelta=1.0867626187760493E-4
New Minimum: 5.792989413940235 > 5.792880738818324
WOLFE (weak): th(4.308869380063768)=5.792880738818324; dx=-5.044225867409163E-5 evalInputDelta=2.1735138378886631E-4
New Minimum: 5.792880738818324 > 5.792446049730305
WOLFE (weak): th(12.926608140191302)=5.792446049730305; dx=-5.044014217802799E-5 evalInputDelta=6.520404718077799E-4
New Minimum: 5.792446049730305 > 5.790490174546919
WOLFE (weak): th(51.70643256076521)=5.790490174546919; dx=-5.0430617945741665E-5 evalInputDelta=0.0026079156551936933
New Minimum: 5.790490174546919 > 5.780065078113811
WOLFE (weak): th(258.53216280382605)=5.780065078113811; dx=-5.037982204021453E-5 evalInputDelta=0.013033012088301987
New Minimum: 5.780065078113811 > 5.715146249709504
WOLFE (weak): th(1551.1929768229563)=5.715146249709504; dx=-5.0062347630670014E-5 evalInputDelta=0.07795184049260939
New Minimum: 5.715146249709504 > 5.2598453014288955
WOLFE (weak): th(10858.350837760694)=5.2598453014288955; dx=-4.7776531881949425E-5 evalInputDelta=0.5332527887732175
New Minimum: 5.2598453014288955 > 2.337868632119215
END: th(86866.80670208555)=2.337868632119215; dx=-2.910903660073127E-5 evalInputDelta=3.455229458082898
Fitness changed from 5.793098090202113 to 2.337868632119215
Iteration 1 complete. Error: 2.337868632119215 Total: 2.0768; Orientation: 0.0747; Line Search: 1.7059
th(0)=2.337868632119215;dx=-1.702583117642431E-5
New Minimum: 2.337868632119215 > 0.5353329809104382
END: th(187148.86177126726)=0.5353329809104382; dx=-2.2372905348028732E-6 evalInputDelta=1.802535651208777
Fitness changed from 2.337868632119215 to 0.5353329809104382
Iteration 2 complete. Error: 0.5353329809104382 Total: 0.4497; Orientation: 0.0766; Line Search: 0.3071
th(0)=0.5353329809104382;dx=-1.1232777154297761E-6
New Minimum: 0.5353329809104382 > 0.19243157802883468
END: th(403200.00000000006)=0.19243157802883468; dx=-5.776221004511942E-7 evalInputDelta=0.34290140288160353
Fitness changed from 0.5353329809104382 to 0.19243157802883468
Iteration 3 complete. Error: 0.19243157802883468 Total: 0.4097; Orientation: 0.0800; Line Search: 0.2638
th(0)=0.19243157802883468;dx=-4.342365658319129E-7
New Minimum: 0.19243157802883468 > 0.0680884115429137
WOLF (strong): th(868668.0670208557)=0.0680884115429137; dx=1.4795191647819008E-7 evalInputDelta=0.12434316648592098
New Minimum: 0.0680884115429137 > 0.06704392683960884
END: th(434334.03351042786)=0.06704392683960884; dx=-1.4314232467686142E-7 evalInputDelta=0.12538765118922585
Fitness changed from 0.19243157802883468 to 0.06704392683960884
Iteration 4 complete. Error: 0.06704392683960884 Total: 0.5073; Orientation: 0.0843; Line Search: 0.3574
th(0)=0.06704392683960884;dx=-1.9982572795732415E-7
Armijo: th(935744.3088563365)=0.11046518544708175; dx=2.9263154723191264E-7 evalInputDelta=-0.04342125860747291
New Minimum: 0.06704392683960884 > 0.03115279456669192
WOLF (strong): th(467872.15442816826)=0.03115279456669192; dx=4.640290963729423E-8 evalInputDelta=0.035891132272916924
END: th(155957.38480938942)=0.042279824620491346; dx=-1.1774951542578468E-7 evalInputDelta=0.024764102219117498
Fitness changed from 0.06704392683960884 to 0.03115279456669192
Iteration 5 complete. Error: 0.03115279456669192 Total: 0.6208; Orientation: 0.0752; Line Search: 0.4828
th(0)=0.03115279456669192;dx=-1.8074627189375108E-7
New Minimum: 0.03115279456669192 > 0.014153588881753314
WOLF (strong): th(336000.0000000001)=0.014153588881753314; dx=7.956052376911653E-8 evalInputDelta=0.016999205684938606
New Minimum: 0.014153588881753314 > 0.011720306306382176
END: th(168000.00000000006)=0.011720306306382176; dx=-5.059287406231727E-8 evalInputDelta=0.019432488260309744
Fitness changed from 0.03115279456669192 to 0.011720306306382176
Iteration 6 complete. Error: 0.011720306306382176 Total: 0.5319; Orientation: 0.1007; Line Search: 0.3608
th(0)=0.011720306306382176;dx=-2.6498644438954612E-8
New Minimum: 0.011720306306382176 > 0.004823147778962676
END: th(361945.0279253566)=0.004823147778962676; dx=-1.161299128072098E-8 evalInputDelta=0.0068971585274195
Fitness changed from 0.011720306306382176 to 0.004823147778962676
Iteration 7 complete. Error: 0.004823147778962676 Total: 0.4189; Orientation: 0.0777; Line Search: 0.2757
th(0)=0.004823147778962676;dx=-1.1645727497052016E-8
New Minimum: 0.004823147778962676 > 0.002246522563641574
WOLF (strong): th(779786.9240469472)=0.002246522563641574; dx=5.037190893363142E-9 evalInputDelta=0.0025766252153211025
New Minimum: 0.002246522563641574 > 0.0019086949693286144
END: th(389893.4620234736)=0.0019086949693286144; dx=-3.3042683018444362E-9 evalInputDelta=0.0029144528096340617
Fitness changed from 0.004823147778962676 to 0.0019086949693286144
Iteration 8 complete. Error: 0.0019086949693286144 Total: 0.5129; Orientation: 0.0781; Line Search: 0.3715
th(0)=0.0019086949693286144;dx=-5.614817803816887E-9
Armijo: th(840000.0000000003)=0.0023204652636310595; dx=6.595223266441756E-9 evalInputDelta=-4.1177029430244515E-4
New Minimum: 0.0019086949693286144 > 8.325258041026792E-4
WOLF (strong): th(420000.0000000002)=8.325258041026792E-4; dx=4.902027313124341E-10 evalInputDelta=0.0010761691652259352
END: th(140000.00000000006)=0.0012650709559472678; dx=-3.5798109587737796E-9 evalInputDelta=6.436240133813466E-4
Fitness changed from 0.0019086949693286144 to 8.325258041026792E-4
Iteration 9 complete. Error: 8.325258041026792E-4 Total: 0.6539; Orientation: 0.0841; Line Search: 0.4981
th(0)=8.325258041026792E-4;dx=-3.85544271417963E-9
New Minimum: 8.325258041026792E-4 > 3.4393169430105384E-4
WOLF (strong): th(301620.8566044639)=3.4393169430105384E-4; dx=6.156527652879174E-10 evalInputDelta=4.885941098016253E-4
END: th(150810.42830223194)=4.1965679314219794E-4; dx=-1.6198949744458564E-9 evalInputDelta=4.1286901096048123E-4
Fitness changed from 8.325258041026792E-4 to 3.4393169430105384E-4
Iteration 10 complete. Error: 3.4393169430105384E-4 Total: 0.5064; Orientation: 0.0762; Line Search: 0.3474
th(0)=3.4393169430105384E-4;dx=-1.2715404892636898E-9
New Minimum: 3.4393169430105384E-4 > 1.517630460906424E-4
WOLF (strong): th(324911.2183528947)=1.517630460906424E-4; dx=8.86410548607083E-11 evalInputDelta=1.9216864821041145E-4
END: th(162455.60917644735)=1.926050898605256E-4; dx=-5.914497172014907E-10 evalInputDelta=1.5132660444052822E-4
Fitness changed from 3.4393169430105384E-4 to 1.517630460906424E-4
Iteration 11 complete. Error: 1.517630460906424E-4 Total: 0.4856; Orientation: 0.0782; Line Search: 0.3435
th(0)=1.517630460906424E-4;dx=-5.371432306774484E-10
New Minimum: 1.517630460906424E-4 > 6.806371503572113E-5
WOLF (strong): th(350000.00000000023)=6.806371503572113E-5; dx=5.886133893504145E-11 evalInputDelta=8.369933105492126E-5
END: th(175000.00000000012)=8.383818064263532E-5; dx=-2.3914094587120346E-10 evalInputDelta=6.792486544800706E-5
Fitness changed from 1.517630460906424E-4 to 6.806371503572113E-5
Iteration 12 complete. Error: 6.806371503572113E-5 Total: 0.5430; Orientation: 0.0781; Line Search: 0.4025
th(0)=6.806371503572113E-5;dx=-2.701831857318077E-10
New Minimum: 6.806371503572113E-5 > 3.303177035690477E-5
WOLF (strong): th(377026.0707555799)=3.303177035690477E-5; dx=8.435017631359683E-11 evalInputDelta=3.5031944678816364E-5
END: th(188513.03537778996)=3.383920264084491E-5; dx=-9.291650470910544E-11 evalInputDelta=3.422451239487622E-5
Fitness changed from 6.806371503572113E-5 to 3.303177035690477E-5
Iteration 13 complete. Error: 3.303177035690477E-5 Total: 0.4847; Orientation: 0.0746; Line Search: 0.3477
th(0)=3.303177035690477E-5;dx=-1.7290095764901076E-10
New Minimum: 3.303177035690477E-5 > 2.1232172664052868E-5
WOLF (strong): th(406139.02294111846)=2.1232172664052868E-5; dx=1.1479475742523869E-10 evalInputDelta=1.17995976928519E-5
New Minimum: 2.1232172664052868E-5 > 1.252641443240356E-5
END: th(203069.51147055923)=1.252641443240356E-5; dx=-2.905310011188604
...skipping 16469 bytes...
.1548801340729767E-19 to 1.777187316254973E-20
Iteration 43 complete. Error: 1.777187316254973E-20 Total: 0.6081; Orientation: 0.0791; Line Search: 0.4613
Zero gradient: 2.7516774486698447E-13
th(0)=1.777187316254973E-20;dx=-7.571728781518187E-26
New Minimum: 1.777187316254973E-20 > 7.885384000796102E-21
WOLF (strong): th(332068.2056492404)=7.885384000796102E-21; dx=1.6172356857970776E-26 evalInputDelta=9.886489161753628E-21
END: th(166034.1028246202)=9.014424908415845E-21; dx=-2.977246571318364E-26 evalInputDelta=8.757448254133885E-21
Fitness changed from 1.777187316254973E-20 to 7.885384000796102E-21
Iteration 44 complete. Error: 7.885384000796102E-21 Total: 0.4801; Orientation: 0.0738; Line Search: 0.3415
Zero gradient: 1.8360937137546193E-13
th(0)=7.885384000796102E-21;dx=-3.3712401256892294E-26
New Minimum: 7.885384000796102E-21 > 3.72639388854703E-21
WOLF (strong): th(357709.6308536826)=3.72639388854703E-21; dx=1.0458959182297218E-26 evalInputDelta=4.158990112249072E-21
END: th(178854.8154268413)=3.830823808970466E-21; dx=-1.1626721149606286E-26 evalInputDelta=4.054560191825636E-21
Fitness changed from 7.885384000796102E-21 to 3.72639388854703E-21
Iteration 45 complete. Error: 3.72639388854703E-21 Total: 0.4807; Orientation: 0.0736; Line Search: 0.3429
Zero gradient: 1.3723063922686772E-13
th(0)=3.72639388854703E-21;dx=-1.8832248342614726E-26
New Minimum: 3.72639388854703E-21 > 2.10102217819687E-21
WOLF (strong): th(385331.0188348366)=2.10102217819687E-21; dx=1.0396012312864437E-26 evalInputDelta=1.62537171035016E-21
New Minimum: 2.10102217819687E-21 > 1.505888605041994E-21
END: th(192665.5094174183)=1.505888605041994E-21; dx=-4.218117964645143E-27 evalInputDelta=2.220505283505036E-21
Fitness changed from 3.72639388854703E-21 to 1.505888605041994E-21
Iteration 46 complete. Error: 1.505888605041994E-21 Total: 0.6161; Orientation: 0.0737; Line Search: 0.4783
Zero gradient: 5.5205721229113485E-14
th(0)=1.505888605041994E-21;dx=-3.047671656426591E-27
New Minimum: 1.505888605041994E-21 > 5.10544598895784E-22
END: th(415085.25706155057)=5.10544598895784E-22; dx=-1.7481816960771597E-27 evalInputDelta=9.9534400614621E-22
Fitness changed from 1.505888605041994E-21 to 5.10544598895784E-22
Iteration 47 complete. Error: 5.10544598895784E-22 Total: 0.3830; Orientation: 0.0751; Line Search: 0.2424
Zero gradient: 3.229577414929628E-14
th(0)=5.10544598895784E-22;dx=-1.0430170279023537E-27
New Minimum: 5.10544598895784E-22 > 2.5750838581226822E-23
END: th(894274.0771342065)=2.5750838581226822E-23; dx=-4.1200372209762574E-29 evalInputDelta=4.847937603145572E-22
Fitness changed from 5.10544598895784E-22 to 2.5750838581226822E-23
Iteration 48 complete. Error: 2.5750838581226822E-23 Total: 0.3764; Orientation: 0.0746; Line Search: 0.2372
Zero gradient: 1.480112353551284E-14
th(0)=2.5750838581226822E-23;dx=-2.190732579135121E-28
Armijo: th(1926655.0941741832)=1.5721969780264406E-21; dx=1.8243903131257478E-27 evalInputDelta=-1.5464461394452137E-21
Armijo: th(963327.5470870916)=3.068427183887042E-22; dx=8.026585235537521E-28 evalInputDelta=-2.810918798074774E-22
New Minimum: 2.5750838581226822E-23 > 1.0085648204917351E-23
WOLF (strong): th(321109.18236236385)=1.0085648204917351E-23; dx=1.2150401185039104E-28 evalInputDelta=1.5665190376309472E-23
END: th(80277.29559059096)=1.1581807918554021E-23; dx=-1.3392894913543334E-28 evalInputDelta=1.41690306626728E-23
Fitness changed from 2.5750838581226822E-23 to 1.0085648204917351E-23
Iteration 49 complete. Error: 1.0085648204917351E-23 Total: 0.6999; Orientation: 0.0754; Line Search: 0.5608
Zero gradient: 9.121163898891163E-15
th(0)=1.0085648204917351E-23;dx=-8.319563087043543E-29
New Minimum: 1.0085648204917351E-23 > 1.674510948246178E-24
END: th(172952.19044231274)=1.674510948246178E-24; dx=-1.4069824696802654E-29 evalInputDelta=8.411137256671173E-24
Fitness changed from 1.0085648204917351E-23 to 1.674510948246178E-24
Iteration 50 complete. Error: 1.674510948246178E-24 Total: 0.3768; Orientation: 0.0766; Line Search: 0.2351
Zero gradient: 2.111000633025926E-15
th(0)=1.674510948246178E-24;dx=-4.456323672635861E-30
New Minimum: 1.674510948246178E-24 > 7.026553884111177E-25
END: th(372614.1988059194)=7.026553884111177E-25; dx=-7.600948831818219E-31 evalInputDelta=9.718555598350601E-25
Fitness changed from 1.674510948246178E-24 to 7.026553884111177E-25
Iteration 51 complete. Error: 7.026553884111177E-25 Total: 0.3784; Orientation: 0.0780; Line Search: 0.2348
Zero gradient: 1.4960256966815755E-15
th(0)=7.026553884111177E-25;dx=-2.2380928851315933E-30
Armijo: th(802772.9559059097)=9.627560525422354E-25; dx=2.886098624979983E-30 evalInputDelta=-2.6010066413111764E-25
New Minimum: 7.026553884111177E-25 > 3.1851030181262616E-25
WOLF (strong): th(401386.4779529548)=3.1851030181262616E-25; dx=3.240013786096284E-31 evalInputDelta=3.841450865984916E-25
END: th(133795.49265098493)=4.603415050461045E-25; dx=-1.3840615250817439E-30 evalInputDelta=2.423138833650133E-25
Fitness changed from 7.026553884111177E-25 to 3.1851030181262616E-25
Iteration 52 complete. Error: 3.1851030181262616E-25 Total: 0.6594; Orientation: 0.1195; Line Search: 0.4731
Zero gradient: 1.221077490480506E-15
th(0)=3.1851030181262616E-25;dx=-1.49103023775817E-30
New Minimum: 3.1851030181262616E-25 > 1.2833800612093226E-25
WOLF (strong): th(288253.6507371879)=1.2833800612093226E-25; dx=1.715515380947872E-31 evalInputDelta=1.901722956916939E-25
END: th(144126.82536859394)=1.6351853259719407E-25; dx=-6.59740284562986E-31 evalInputDelta=1.549917692154321E-25
Fitness changed from 3.1851030181262616E-25 to 1.2833800612093226E-25
Iteration 53 complete. Error: 1.2833800612093226E-25 Total: 0.5035; Orientation: 0.0778; Line Search: 0.3568
Zero gradient: 6.551726010919691E-16
th(0)=1.2833800612093226E-25;dx=-4.292511372216163E-31
New Minimum: 1.2833800612093226E-25 > 5.720756846870684E-26
END: th(310511.83233826613)=5.720756846870684E-26; dx=-2.8898292369558277E-32 evalInputDelta=7.113043765222542E-26
Fitness changed from 1.2833800612093226E-25 to 5.720756846870684E-26
Iteration 54 complete. Error: 5.720756846870684E-26 Total: 0.3744; Orientation: 0.0744; Line Search: 0.2354
Zero gradient: 4.108627842883718E-16
th(0)=5.720756846870684E-26;dx=-1.6880822751319313E-31
New Minimum: 5.720756846870684E-26 > 4.2468474934583846E-26
WOLF (strong): th(668977.4632549246)=4.2468474934583846E-26; dx=1.2474350835373241E-31 evalInputDelta=1.4739093534122995E-26
New Minimum: 4.2468474934583846E-26 > 2.529058398782242E-26
END: th(334488.7316274623)=2.529058398782242E-26; dx=-2.2032525001152617E-32 evalInputDelta=3.191698448088442E-26
Fitness changed from 5.720756846870684E-26 to 2.529058398782242E-26
Iteration 55 complete. Error: 2.529058398782242E-26 Total: 0.4972; Orientation: 0.0756; Line Search: 0.3565
Zero gradient: 2.7522243727766006E-16
th(0)=2.529058398782242E-26;dx=-7.574738998105552E-32
New Minimum: 2.529058398782242E-26 > 2.282911171051787E-26
WOLF (strong): th(720634.1268429698)=2.282911171051787E-26; dx=6.891595472498297E-32 evalInputDelta=2.46147227730455E-27
New Minimum: 2.282911171051787E-26 > 1.1028745171699055E-26
END: th(360317.0634214849)=1.1028745171699055E-26; dx=-3.415473375599951E-33 evalInputDelta=1.4261838816123364E-26
Fitness changed from 2.529058398782242E-26 to 1.1028745171699055E-26
Iteration 56 complete. Error: 1.1028745171699055E-26 Total: 0.4974; Orientation: 0.0762; Line Search: 0.3564
Zero gradient: 1.9618531266069542E-16
th(0)=1.1028745171699055E-26;dx=-3.848867690377482E-32
Armijo: th(776279.5808456654)=1.7444877680584787E-26; dx=5.501895407502572E-32 evalInputDelta=-6.416132508885732E-27
New Minimum: 1.1028745171699055E-26 > 5.1632771595686515E-27
WOLF (strong): th(388139.7904228327)=5.1632771595686515E-27; dx=8.265115112782165E-33 evalInputDelta=5.865468012130404E-27
END: th(129379.93014094424)=7.057272749731961E-27; dx=-2.2904029419627433E-32 evalInputDelta=3.971472421967094E-27
Fitness changed from 1.1028745171699055E-26 to 5.1632771595686515E-27
Iteration 57 complete. Error: 5.1632771595686515E-27 Total: 0.5942; Orientation: 0.0770; Line Search: 0.4520
Final threshold in iteration 57: 5.1632771595686515E-27 (> 0.0) after 30.241s (< 30.000s)
Returns
5.1632771595686515E-27
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 30.09 seconds (0.464 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11491944334346
Reset training subject: 11492009285622
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=5.793098090202113}, derivative=-5.044331692212345E-5}
New Minimum: 5.793098090202113 > 5.793098090202108
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=5.793098090202108}, derivative=-5.044331692212342E-5}, evalInputDelta = -5.329070518200751E-15
New Minimum: 5.793098090202108 > 5.7930980902020766
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=5.7930980902020766}, derivative=-5.0443316922123274E-5}, evalInputDelta = -3.6415315207705135E-14
New Minimum: 5.7930980902020766 > 5.793098090201865
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=5.793098090201865}, derivative=-5.0443316922122244E-5}, evalInputDelta = -2.4780177909633494E-13
New Minimum: 5.793098090201865 > 5.793098090200382
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=5.793098090200382}, derivative=-5.044331692211502E-5}, evalInputDelta = -1.731059739995544E-12
New Minimum: 5.793098090200382 > 5.79309809019
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=5.79309809019}, derivative=-5.044331692206447E-5}, evalInputDelta = -1.2112977287870308E-11
New Minimum: 5.79309809019 > 5.793098090117332
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=5.793098090117332}, derivative=-5.044331692171067E-5}, evalInputDelta = -8.478107105247545E-11
New Minimum: 5.793098090117332 > 5.793098089608652
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=5.793098089608652}, derivative=-5.044331691923401E-5}, evalInputDelta = -5.934612801183903E-10
New Minimum: 5.793098089608652 > 5.7930980860478885
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=5.7930980860478885}, derivative=-5.0443316901897415E-5}, evalInputDelta = -4.1542245199366334E-9
New Minimum: 5.7930980860478885 > 5.793098061122544
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=5.793098061122544}, derivative=-5.0443316780541304E-5}, evalInputDelta = -2.9079568975021175E-8
New Minimum: 5.793098061122544 > 5.7930978866451355
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=5.7930978866451355}, derivative=-5.0443315931048485E-5}, evalInputDelta = -2.035569774960777E-7
New Minimum: 5.7930978866451355 > 5.793096665303359
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=5.793096665303359}, derivative=-5.044330998459877E-5}, evalInputDelta = -1.424898753654702E-6
New Minimum: 5.793096665303359 > 5.793088115914957
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=5.793088115914957}, derivative=-5.044326835945074E-5}, evalInputDelta = -9.974287156211403E-6
New Minimum: 5.793088115914957 > 5.793028270393683
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=5.793028270393683}, derivative=-5.0442976983414555E-5}, evalInputDelta = -6.98198084299051E-5
New Minimum: 5.793028270393683 > 5.792609361424003
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=5.792609361424003}, derivative=-5.044093735116123E-5}, evalInputDelta = -4.887287781096816E-4
New Minimum: 5.792609361424003 > 5.789677472919326
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=5.789677472919326}, derivative=-5.042665992538794E-5}, evalInputDelta = -0.003420617282786864
New Minimum: 5.789677472919326 > 5.76917749325758
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=5.76917749325758}, derivative=-5.032671794497496E-5}, evalInputDelta = -0.0239205969445333
New Minimum: 5.76917749325758 > 5.626816389304004
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=5.626816389304004}, derivative=-4.9627124082084096E-5}, evalInputDelta = -0.166281700898109
New Minimum: 5.626816389304004 > 4.686087591882657
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=4.686087591882657}, derivative=-4.47299670418481E-5}, evalInputDelta = -1.1070104983194557
New Minimum: 4.686087591882657 > 0.8351335923640452
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.8351335923640452}, derivative=-1.044986776019606E-5}, evalInputDelta = -4.957964497838068
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=107.85168713484353}, derivative=2.2951082721136824E-4}, evalInputDelta = 102.05858904464142
F(87683.80911825495) = LineSearchPoint{point=PointSample{avg=2.3141684462212124}, derivative=-2.8908382758008694E-5}, evalInputDelta = -3.4789296439809005
F(613786.6638277847) = LineSearchPoint{point=PointSample{avg=21.094156771364734}, derivative=1.0030122222667978E-4}, evalInputDelta = 15.30105868116262
F(47214.35875598344) = LineSearchPoint{point=PointSample{avg=3.6851917955790894}, derivative=-3.884758314144627E-5}, evalInputDelta = -2.1079062946230236
F(330500.5112918841) = LineSearchPoint{point=PointSample{avg=2.5349418577960874}, derivative=3.0726819542616764E-5}, evalInputDelta = -3.2581562324060256
2.5349418577960874 <= 5.793098090202113
New Minimum: 0.8351335923640452 > 0.6128192645782207
F(205390.09493057043) = LineSearchPoint{point=PointSample{avg=0.6128192645782207}, derivative=-8.470329472543003E-22}, evalInputDelta = -5.180278825623892
Left bracket at 205390.09493057043
Converged to left
Fitness changed from 5.793098090202113 to 0.6128192645782207
Iteration 1 complete. Error: 0.6128192645782207 Total: 3.8731; Orientation: 0.0743; Line Search: 3.6057
F(0.0) = LineSearchPoint{point=PointSample{avg=0.6128192645782207}, derivative=-1.2748918280183733E-6}
New Minimum: 0.6128192645782207 > 0.38224941832765036
F(205390.09493057043) = LineSearchPoint{point=PointSample{avg=0.38224941832765036}, derivative=-9.702977107323788E-7}, evalInputDelta = -0.2305698462505703
New Minimum: 0.38224941832765036 > 0.31260324878216617
F(1437730.664513993) = LineSearchPoint{point=PointSample{avg=0.31260324878216617}, derivative=8.572669929835877E-7}, evalInputDelta = -0.3002160157960545
0.31260324878216617 <= 0.6128192645782207
New Minimum: 0.31260324878216617 > 0.06482670329255354
F(859669.109554869) = LineSearchPoint{point=PointSample{avg=0.06482670329255354}, derivative=-1.9852334701272664E-23}, evalInputDelta = -0.5479925612856671
Left bracket at 859669.109554869
Converged to left
Fitness changed from 0.6128192645782207 to 0.06482670329255354
Iteration 2 complete. Error: 0.06482670329255354 Total: 0.7562; Orientation: 0.0786; Line Search: 0.6100
F(0.0) = LineSearchPoint{point=PointSample{avg=0.06482670329255354}, derivative=-5.644775711175043E-7}
F(859669.109554869) = LineSearchPoint{point=PointSample{avg=0.5951093388633303}, derivative=1.7981676727552233E-6}, evalInputDelta = 0.5302826355707767
New Minimum: 0.06482670329255354 > 0.03350785988696427
F(66128.39304268223) = LineSearchPoint{point=PointSample{avg=0.03350785988696427}, derivative=-3.827356292811407E-7}, evalInputDelta = -0.03131884340558927
F(462898.75129877566) = LineSearchPoint{point=PointSample{avg=0.09797915325743103}, derivative=7.077160217370415E-7}, evalInputDelta = 0.0331524499648775
F(35607.59625375197) = LineSearchPoint{point=PointSample{avg=0.04646931214632498}, derivative=-4.6661652551330866E-7}, evalInputDelta = -0.018357391146228558
New Minimum: 0.03350785988696427 > 0.009501493886682153
F(249253.1737762638) = LineSearchPoint{point=PointSample{avg=0.009501493886682153}, derivative=1.205497481118664E-7}, evalInputDelta = -0.055325209405871385
0.009501493886682153 <= 0.06482670329255354
New Minimum: 0.009501493886682153 > 0.006857652333552518
F(205390.09493057043) = LineSearchPoint{point=PointSample{avg=0.006857652333552518}, derivative=4.963083675318166E-24}, evalInputDelta = -0.05796905095900102
Right bracket at 205390.09493057043
Converged to right
Fitness changed from 0.06482670329255354 to 0.006857652333552518
Iteration 3 complete. Error: 0.006857652333552518 Total: 1.0544; Orientation: 0.0756; Line Search: 0.9123
F(0.0) = LineSearchPoint{point=PointSample{avg=0.006857652333552518}, derivative=-1.4266465538505112E-8}
New Minimum: 0.006857652333552518 > 0.004277498713095882
F(205390.09493057043) = LineSearchPoint{point=PointSample{avg=0.004277498713095882}, derivative=-1.0857955591236554E-8}, evalInputDelta = -0.0025801536204566357
New Minimum: 0
...skipping 25817 bytes...
930772048E-22
Iteration 23 complete. Error: 2.1115725930772048E-22 Total: 1.0916; Orientation: 0.0772; Line Search: 0.9488
Zero gradient: 2.0959140497744016E-14
F(0.0) = LineSearchPoint{point=PointSample{avg=2.1115725930772048E-22}, derivative=-4.392855704041733E-28}
New Minimum: 2.1115725930772048E-22 > 1.3171051102271155E-22
F(205390.09430381926) = LineSearchPoint{point=PointSample{avg=1.3171051102271155E-22}, derivative=-3.3433251598271763E-28}, evalInputDelta = -7.944674828500893E-23
New Minimum: 1.3171051102271155E-22 > 1.0771270852980621E-22
F(1437730.6601267348) = LineSearchPoint{point=PointSample{avg=1.0771270852980621E-22}, derivative=2.953858104675566E-28}, evalInputDelta = -1.0344455077791426E-22
1.0771270852980621E-22 <= 2.1115725930772048E-22
New Minimum: 1.0771270852980621E-22 > 2.2337128789488448E-23
F(859669.1657866704) = LineSearchPoint{point=PointSample{avg=2.2337128789488448E-23}, derivative=4.732371707110202E-37}, evalInputDelta = -1.8882013051823202E-22
Right bracket at 859669.1657866704
Converged to right
Fitness changed from 2.1115725930772048E-22 to 2.2337128789488448E-23
Iteration 24 complete. Error: 2.2337128789488448E-23 Total: 0.7848; Orientation: 0.0808; Line Search: 0.6406
Zero gradient: 1.3946334113924342E-14
F(0.0) = LineSearchPoint{point=PointSample{avg=2.2337128789488448E-23}, derivative=-1.9450023521720988E-28}
F(859669.1657866704) = LineSearchPoint{point=PointSample{avg=2.050549658239654E-22}, derivative=6.19588964795697E-28}, evalInputDelta = 1.8271783703447694E-22
New Minimum: 2.2337128789488448E-23 > 1.1545694091050243E-23
F(66128.39736820542) = LineSearchPoint{point=PointSample{avg=1.1545694091050243E-23}, derivative=-1.318779877742428E-28}, evalInputDelta = -1.0791434698438204E-23
F(462898.78157743794) = LineSearchPoint{point=PointSample{avg=3.3760370322830965E-23}, derivative=2.438554820858384E-28}, evalInputDelta = 1.1423241533342517E-23
F(35607.598582879844) = LineSearchPoint{point=PointSample{avg=1.6011777273171812E-23}, derivative=-1.6078054433905873E-28}, evalInputDelta = -6.325351516316636E-24
New Minimum: 1.1545694091050243E-23 > 3.273899747504942E-24
F(249253.1900801589) = LineSearchPoint{point=PointSample{avg=3.273899747504942E-24}, derivative=4.153747474176564E-29}, evalInputDelta = -1.9063229041983507E-23
3.273899747504942E-24 <= 2.2337128789488448E-23
New Minimum: 3.273899747504942E-24 > 2.362918043355352E-24
F(205390.07986332718) = LineSearchPoint{point=PointSample{avg=2.362918043355352E-24}, derivative=-1.3448193460106233E-35}, evalInputDelta = -1.9974210746133095E-23
Left bracket at 205390.07986332718
Converged to left
Fitness changed from 2.2337128789488448E-23 to 2.362918043355352E-24
Iteration 25 complete. Error: 2.362918043355352E-24 Total: 1.0504; Orientation: 0.0766; Line Search: 0.9106
Zero gradient: 2.2171484255159155E-15
F(0.0) = LineSearchPoint{point=PointSample{avg=2.362918043355352E-24}, derivative=-4.9157471407677034E-30}
New Minimum: 2.362918043355352E-24 > 1.4738835695182462E-24
F(205390.07986332718) = LineSearchPoint{point=PointSample{avg=1.4738835695182462E-24}, derivative=-3.741289491443247E-30}, evalInputDelta = -8.890344738371058E-25
New Minimum: 1.4738835695182462E-24 > 1.2053371632694493E-24
F(1437730.5590432903) = LineSearchPoint{point=PointSample{avg=1.2053371632694493E-24}, derivative=3.3054579678369964E-30}, evalInputDelta = -1.1575808800859027E-24
1.2053371632694493E-24 <= 2.362918043355352E-24
New Minimum: 1.2053371632694493E-24 > 2.499590891777099E-25
F(859669.5729454807) = LineSearchPoint{point=PointSample{avg=2.499590891777099E-25}, derivative=3.1690484888258477E-37}, evalInputDelta = -2.112958954177642E-24
Right bracket at 859669.5729454807
Converged to right
Fitness changed from 2.362918043355352E-24 to 2.499590891777099E-25
Iteration 26 complete. Error: 2.499590891777099E-25 Total: 0.8749; Orientation: 0.0763; Line Search: 0.7332
Zero gradient: 1.4753021547586223E-15
F(0.0) = LineSearchPoint{point=PointSample{avg=2.499590891777099E-25}, derivative=-2.1765164478354348E-30}
F(859669.5729454807) = LineSearchPoint{point=PointSample{avg=2.2946307598552187E-24}, derivative=6.933394100857245E-30}, evalInputDelta = 2.0446716706775086E-24
New Minimum: 2.499590891777099E-25 > 1.2919949158165861E-25
F(66128.4286881139) = LineSearchPoint{point=PointSample{avg=1.2919949158165861E-25}, derivative=-1.4757525712698614E-30}, evalInputDelta = -1.207595975960513E-25
F(462899.00081679726) = LineSearchPoint{point=PointSample{avg=3.7778955242553784E-25}, derivative=2.7288200460416828E-30}, evalInputDelta = 1.2783046324782793E-25
F(35607.61544744594) = LineSearchPoint{point=PointSample{avg=1.7917642750189018E-25}, derivative=-1.799182028363946E-30}, evalInputDelta = -7.078266167581973E-26
New Minimum: 1.2919949158165861E-25 > 3.6635946344013145E-26
F(249253.30813212157) = LineSearchPoint{point=PointSample{avg=3.6635946344013145E-26}, derivative=4.648180555688887E-31}, evalInputDelta = -2.1332314283369676E-25
3.6635946344013145E-26 <= 2.499590891777099E-25
New Minimum: 3.6635946344013145E-26 > 2.644168731025155E-26
F(205390.08752119128) = LineSearchPoint{point=PointSample{avg=2.644168731025155E-26}, derivative=2.1406351862085924E-36}, evalInputDelta = -2.235174018674584E-25
Right bracket at 205390.08752119128
Converged to right
Fitness changed from 2.499590891777099E-25 to 2.644168731025155E-26
Iteration 27 complete. Error: 2.644168731025155E-26 Total: 1.0660; Orientation: 0.0806; Line Search: 0.9183
Zero gradient: 2.345395782448231E-16
F(0.0) = LineSearchPoint{point=PointSample{avg=2.644168731025155E-26}, derivative=-5.500881376325948E-32}
New Minimum: 2.644168731025155E-26 > 1.6493146223094636E-26
F(205390.08752119128) = LineSearchPoint{point=PointSample{avg=1.6493146223094636E-26}, derivative=-4.1865843966018514E-32}, evalInputDelta = -9.948541087156914E-27
New Minimum: 1.6493146223094636E-26 > 1.348950643000325E-26
F(1437730.612648339) = LineSearchPoint{point=PointSample{avg=1.348950643000325E-26}, derivative=3.699119211687749E-32}, evalInputDelta = -1.29521808802483E-26
1.348950643000325E-26 <= 2.644168731025155E-26
New Minimum: 1.348950643000325E-26 > 2.797624123365445E-27
F(859650.5484570268) = LineSearchPoint{point=PointSample{avg=2.797624123365445E-27}, derivative=3.624237067687068E-39}, evalInputDelta = -2.3644063186886105E-26
Right bracket at 859650.5484570268
Converged to right
Fitness changed from 2.644168731025155E-26 to 2.797624123365445E-27
Iteration 28 complete. Error: 2.797624123365445E-27 Total: 0.7574; Orientation: 0.0796; Line Search: 0.6099
Zero gradient: 1.560760614940611E-16
F(0.0) = LineSearchPoint{point=PointSample{avg=2.797624123365445E-27}, derivative=-2.435973697149795E-32}
F(859650.5484570268) = LineSearchPoint{point=PointSample{avg=2.568004702784312E-26}, derivative=7.759625860974911E-32}, evalInputDelta = 2.2882422904477675E-26
New Minimum: 2.797624123365445E-27 > 1.4461191496106696E-27
F(66126.96526592514) = LineSearchPoint{point=PointSample{avg=1.4461191496106696E-27}, derivative=-1.651694489430529E-32}, evalInputDelta = -1.3515049737547754E-27
F(462888.756861476) = LineSearchPoint{point=PointSample{avg=4.2279764777625535E-27}, derivative=3.053974194854604E-32}, evalInputDelta = 1.4303523543971084E-27
F(35606.82745088277) = LineSearchPoint{point=PointSample{avg=2.0054509317642624E-27}, derivative=-2.01366882507994E-32}, evalInputDelta = -7.921731916011826E-28
New Minimum: 1.4461191496106696E-27 > 4.1005204532953E-28
F(249247.79215617938) = LineSearchPoint{point=PointSample{avg=4.1005204532953E-28}, derivative=5.201437327996788E-33}, evalInputDelta = -2.387572078035915E-27
4.1005204532953E-28 <= 2.797624123365445E-27
New Minimum: 4.1005204532953E-28 > 2.96001129660126E-28
F(205391.3892640894) = LineSearchPoint{point=PointSample{avg=2.96001129660126E-28}, derivative=1.882611336547917E-37}, evalInputDelta = -2.501622993705319E-27
Right bracket at 205391.3892640894
Converged to right
Fitness changed from 2.797624123365445E-27 to 2.96001129660126E-28
Iteration 29 complete. Error: 2.96001129660126E-28 Total: 1.0795; Orientation: 0.0783; Line Search: 0.9379
Final threshold in iteration 29: 2.96001129660126E-28 (> 0.0) after 30.086s (< 30.000s)
Returns
2.96001129660126E-28
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 31.09 seconds (0.483 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11522055057464
Reset training subject: 11522121308610
Adding measurement 5b61502b to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 5.793098090202113 < 5.793098090202113. Total: 1
th(0)=5.793098090202113;dx=-5.044331692212345E-5
Adding measurement 117ae5f2 to history. Total: 1
New Minimum: 5.793098090202113 > 5.792989413940235
WOLFE (weak): th(2.154434690031884)=5.792989413940235; dx=-5.0442787798107533E-5 evalInputDelta=1.0867626187760493E-4
Adding measurement 6859cd02 to history. Total: 2
New Minimum: 5.792989413940235 > 5.792880738818324
WOLFE (weak): th(4.308869380063768)=5.792880738818324; dx=-5.044225867409163E-5 evalInputDelta=2.1735138378886631E-4
Adding measurement 6f167ce4 to history. Total: 3
New Minimum: 5.792880738818324 > 5.792446049730305
WOLFE (weak): th(12.926608140191302)=5.792446049730305; dx=-5.044014217802799E-5 evalInputDelta=6.520404718077799E-4
Adding measurement 363d139e to history. Total: 4
New Minimum: 5.792446049730305 > 5.790490174546919
WOLFE (weak): th(51.70643256076521)=5.790490174546919; dx=-5.0430617945741665E-5 evalInputDelta=0.0026079156551936933
Adding measurement 7a11aad2 to history. Total: 5
New Minimum: 5.790490174546919 > 5.780065078113811
WOLFE (weak): th(258.53216280382605)=5.780065078113811; dx=-5.037982204021453E-5 evalInputDelta=0.013033012088301987
Adding measurement 606ab4c9 to history. Total: 6
New Minimum: 5.780065078113811 > 5.715146249709504
WOLFE (weak): th(1551.1929768229563)=5.715146249709504; dx=-5.0062347630670014E-5 evalInputDelta=0.07795184049260939
Adding measurement 5f0f7dd to history. Total: 7
New Minimum: 5.715146249709504 > 5.2598453014288955
WOLFE (weak): th(10858.350837760694)=5.2598453014288955; dx=-4.7776531881949425E-5 evalInputDelta=0.5332527887732175
Adding measurement 237ca8d2 to history. Total: 8
New Minimum: 5.2598453014288955 > 2.337868632119215
END: th(86866.80670208555)=2.337868632119215; dx=-2.910903660073127E-5 evalInputDelta=3.455229458082898
Fitness changed from 5.793098090202113 to 2.337868632119215
Iteration 1 complete. Error: 2.337868632119215 Total: 1.8329; Orientation: 0.1071; Line Search: 1.5319
Non-optimal measurement 2.337868632119215 < 2.337868632119215. Total: 9
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00, 94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811, 5.790490174546919, 5.792446049730305, 5.792880738818324, 5.792989413940235, 5.793098090202113
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811, 5.790490174546919, 5.792446049730305, 5.792880738818324, 5.792989413940235
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00, 94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811, 5.790490174546919, 5.792446049730305, 5.792880738818324
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811, 5.790490174546919, 5.792446049730305
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811, 5.790490174546919
Rejected: LBFGS Orientation magnitude: 9.064e+02, gradient 4.126e-03, dot -0.991; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.337868632119215, 5.2598453014288955, 5.715146249709504, 5.780065078113811
LBFGS Accumulation History: 3 points
Removed measurement 237ca8d2 to history. Total: 8
Removed measurement 5f0f7dd to history. Total: 7
Removed measurement 606ab4c9 to history. Total: 6
Removed measurement 7a11aad2 to history. Total: 5
Removed measurement 363d139e to history. Total: 4
Removed measurement 6f167ce4 to history. Total: 3
Adding measurement 730aead4 to history. Total: 3
th(0)=2.337868632119215;dx=-1.702583117642431E-5
Adding measurement 1aa3ec27 to history. Total: 4
New Minimum: 2.337868632119215 > 0.5353329809104382
END: th(187148.86177126726)=0.5353329809104382; dx=-2.2372905348028732E-6 evalInputDelta=1.802535651208777
Fitness changed from 2.337868632119215 to 0.5353329809104382
Iteration 2 complete. Error: 0.5353329809104382 Total: 23.6899; Orientation: 23.2994; Line Search: 0.3272
Non-optimal measurement 0.5353329809104382 < 0.5353329809104382. Total: 5
Rejected: LBFGS Orientation magnitude: 2.526e+02, gradient 1.060e-03, dot -0.996; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5353329809104382, 2.337868632119215, 5.792880738818324, 5.792989413940235, 5.793098090202113
Rejected: LBFGS Orientation magnitude: 2.526e+02, gradient 1.060e-03, dot -0.996; [94fbc1b0-ddb4-45f2-ab6c-f5b940cbfcb3 = 1.000/1.000e+00, 6ce389da-5013-493f-89ea-9fdecb0f3015 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5353329809104382, 2.337868632119215, 5.792880738818324, 5.792989413940235
LBFGS Accumulation History: 3 points
Removed measurement 1aa3ec27 to history. Total: 4
Removed measurement 730aead4 to history. Total: 3
Adding measurement 4cd44b00 to history. Total: 3
th(0)=0.5353329809104382;dx=-1.1232777154297761E-6
Adding measurement 2d29a422 to history. Total: 4
New Minimum: 0.5353329809104382 > 0.19243157802883468
END: th(403200.00000000006)=0.19243157802883468; dx=-5.776221004511942E-7 evalInputDelta=0.34290140288160353
Fitness changed from 0.5353329809104382 to 0.19243157802883468
Iteration 3 complete. Error: 0.19243157802883468 Total: 5.5686; Orientation: 5.1769; Line Search: 0.3278
Final threshold in iteration 3: 0.19243157802883468 (> 0.0) after 31.091s (< 30.000s)
Returns
0.19243157802883468
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -0.7957916224968378, -0.6351230021455619, 0.14103188515205511 ], [ 0.15075613236075755, 1.46514088710901, -1.3551015661474677 ], [ 1.4700613436125134, 0.360416891939466, 0.4189426522319857 ], [ 0.08593158969731571, -0.7730073184452688, -0.693953636884101 ], [ -0.33100135854130064, 1.0423210193781067, 1.2354757088513886 ], [ -0.2502986867891184, -0.49600330405590654, 0.11610465958867178 ], [ -0.7320864918242868, 1.6758488728263843, 1.375292881450221 ], [ -0.6624881028740394, -0.22995218164597836, -0.7612427395266914 ], ... ],
[ [ -1.15611378515066, -0.5493908062095652, 0.2914740235451223 ], [ -0.05706671139153634, -0.9873082617864661, 0.2200083504836359 ], [ -1.9494670768449174, -0.8608373791625625, -0.12389120742463278 ], [ 1.023377367304139, -0.8660353452334741, -1.2115329124647956 ], [ -1.3971814886211076, -0.9568807018260463, 0.7305475508251226 ], [ -1.8029222901064323, 0.2875604531813823, -0.8057877335252741 ], [ 1.7548827389572028, -0.7254287285586789, -0.9131122178560256 ], [ -0.7733372465307407, -0.847706895399168, -1.1202850977834735 ], ... ],
[ [ -1.3261499662845568, -0.9260415475972636, 0.06637293224730006 ], [ 0.38519431179563896, -0.7204640709714685, -1.173849507777788 ], [ 0.1323396014818295, 0.015008064068153693, -1.9766658834505648 ], [ -2.0893028485033676, 1.6241250676196162, 1.084576679251624 ], [ 1.2324519887441305, -1.2788881197664872, -0.8847410715329853 ], [ -0.9656445217036562, 0.2997254704804007, 1.1032479779917215 ], [ -0.387849129382689, -0.1711703407675889, 0.1886749427695166 ], [ -0.8200729739683835, 0.32343502007087827, -0.6273543991561117 ], ... ],
[ [ -1.3173235479549714, -0.02683828834390123, 1.3603624804495063 ], [ -0.17249338382394813, 0.8723814608673127, -0.30007187868100255 ], [ 1.6692713315850927, 2.0430506007805103, 0.9158984141080982 ], [ 0.02615660514332442, 1.1223732499948964, 2.1463016373834995 ], [ -0.14935693962163915, -0.11241523494888506, 0.11630857900104719 ], [ -0.5811328868242749, 0.8785472418394185, 0.7510771451090161 ], [ 1.5170920941926442, -0.49398354337561423, -2.4464671616317095 ], [ 2.1352685214723386, -0.5372540813605493, -1.045115097175774 ], ... ],
[ [ 0.82298475295712, 1.5991232055815665, 1.82317388221716 ], [ -0.8271050909846889, -0.09128183264682732, -0.24147061073950504 ], [ -1.7330958208663012, -2.318977040379762, -1.762792485733447 ], [ -1.0597323926927202, -1.0815825042417693, -0.3560970150837473 ], [ -0.9797698264712409, 1.0425823890566444, -0.7066665450766324 ], [ 0.06305892697267673, -0.6355877962328211, -0.7124294210322109 ], [ 0.4041962704576155, -2.104330956518867, -1.564264211731521 ], [ -1.7499211556508987, -1.1456836138446116, -0.47583498257188883 ], ... ],
[ [ 1.5840231980537536, -0.05362056065843257, 1.1928611578492792 ], [ 0.9007641155387911, 1.5819393142759615, -1.4987091615197616 ], [ 1.113920095827425, -0.02001497765355202, -0.9002703319884104 ], [ 0.8192674755875653, 1.7095049429762201, 1.680077925832188 ], [ 1.9086042983137368, 0.8321771055298828, 0.8867562160248357 ], [ 0.06910890269799945, 2.117626054745868, 0.7447674331763979 ], [ 1.420293514591199, -1.261096207526735, 0.1903174284113995 ], [ -1.5476282842320133, -0.42067369304920355, -0.7390790799751237 ], ... ],
[ [ 1.3775913767251238, 1.7851650378844275, -0.2557924787977182 ], [ -0.6298465444979594, -1.8459349290748805, 0.2523651538792613 ], [ -1.2324542267211263, -0.10569184522436062, -0.14800415460154015 ], [ 1.3940867629946667, 1.4630708379531312, 0.6601718829795078 ], [ 0.7238076116486912, 0.9431739758874184, -1.3112325530018345 ], [ -1.8755864632852488, -0.2760511838856155, 1.2444651212835256 ], [ -1.8975113440805713, -0.9421218582501905, -0.7570713894767425 ], [ -0.751117407766079, 0.48964748285648424, 0.16165056067547573 ], ... ],
[ [ -1.867627860981146, 0.798402898082687, 0.7477109151382892 ], [ -2.5522330214258706, 0.9103531626757012, 1.5483870806137134 ], [ -1.0283993821001995, 0.2283766483682102, 0.3974499395196319 ], [ -0.5663342145635801, 0.4078173840366718, 0.05627381054723732 ], [ 1.6657712355682563, 0.7221327775864987, -0.15281395989756413 ], [ 1.1572443617373083, 0.30116228121842326, -2.1009842830959116 ], [ -2.1252577223280973, 0.4914569318300749, 1.626127528084731 ], [ 0.897605976179197, -0.9241714228973698, -0.16942674946034375 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.02 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.0830729112414291, -1.7187974311486205 ], [ 1.4589519037483887, 0.9926436974904452 ], [ -0.9697476510714802, 2.771954988534449 ], [ -0.013734225259287833, -0.23182830572278842 ], [ -0.032929179376992475, -0.06689517085531343 ], [ -0.16442767027385813, -0.6875445223077344 ], [ 0.44015155602931866, -0.4601795306561553 ], [ 0.751572070083239, -1.2826852955662107 ], ... ],
[ [ 0.25320313517622267, -2.3118825568857084 ], [ -0.5887734144140837, -0.5903705767407573 ], [ 0.8336264967712553, -3.8658352636190654 ], [ -0.31515495977812397, 1.3756841446886983 ], [ -0.06929939392576015, -2.939240974033353 ], [ 1.728725679146043, -3.037724934969781 ], [ -0.8716091261946206, 2.735800502155401 ], [ 0.7308597652250549, -1.7846295229982139 ], ... ],
[ [ 0.3066278290864093, -2.798645148134201 ], [ 0.12215217995529819, 0.32213258680565876 ], [ 1.1287943792147952, 0.24089105679175146 ], [ 1.4164395444443898, -2.8799641912206106 ], [ -0.845736745068554, 1.5397168007404687 ], [ 0.065000931935135, -1.5547331029269709 ], [ 0.036197416440084715, -0.7690663532517875 ], [ 1.0404596128148487, -1.2861849561250729 ], ... ],
[ [ -0.039479461975252206, -2.3370705296111445 ], [ 0.7200208841931971, 0.1284228755247426 ], [ -0.558430107394327, 3.9579477269032366 ], [ -0.7641574794588404, 0.6028373834702929 ], [ -0.03566455327732707, -0.31922359802721845 ], [ 0.3324333229152288, -0.5893589804056693 ], [ 0.320044203488647, 2.4311346166415198 ], [ -0.9292913063271434, 3.500135647522373 ], ... ],
[ [ -0.8156993402175052, 2.244910214184817 ], [ 0.6044182376554835, -1.5042891694898175 ], [ 0.9752874783842848, -4.207393640036517 ], [ 0.32435935782417474, -2.405832862813876 ], [ 1.5424741788732026, -1.2111931089231733 ], [ 0.07956571345671744, -0.20401559975167752 ], [ -0.3384268461388233, -0.3307459333461242 ], [ 0.7850006595725274, -3.6551199910351126 ], ... ],
[ [ -1.7149414864756265, 2.767621123280239 ], [ 1.1481924878373069, 2.3735897996913047 ], [ -0.13982648533028263, 1.9550276201234158 ], [ -0.6716875523470048, 2.2931022786526705 ], [ -1.28681934837503, 3.7795378265682533 ], [ 0.5555057109423169, 1.1722506275132214 ], [ -1.6047551722788402, 1.8798940408056148 ], [ 1.1816639256855341, -2.9386704451376766 ], ... ],
[ [ 0.2033881288508135, 3.3155130473337944 ], [ -0.6860750393249723, -2.026633029315541 ], [ 0.7868955406128983, -2.2264724111673497 ], [ -0.523306121127545, 3.1848521855473453 ], [ 0.8249686563828706, 1.7446109189884509 ], [ 0.2468003887297824, -3.4454559084424443 ], [ 1.1466938603107522, -3.8145024526502223 ], [ 0.6012609945279029, -1.0821059558025474 ], ... ],
[ [ 1.0769173405214698, -2.8984877093217287 ], [ 1.0618735007009392, -4.0506038811080884 ], [ 0.4968920786796174, -1.7008216924341197 ], [ 0.5123941481241256, -0.7967361320079662 ], [ -0.5617001659248774, 3.2965983172253077 ], [ 0.7231703636703687, 2.19075554558895 ], [ 0.5472337962876838, -3.5051919839990466 ], [ -0.9011219956021582, 1.124987916223008 ], ... ],
...
]
TrainingTester.java:432 executed in 0.17 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -27.528706631494362], [57.0, 0.36882010393932985]; valueStats=DoubleSummaryStatistics{count=89, sum=6.934676, min=0.000000, average=0.077918, max=2.337869}
Plotting 57 points for GD
Plotting 29 points for CjGD
Plotting 3 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -27.528706631494362], [29.258, 0.36882010393932985]; valueStats=DoubleSummaryStatistics{count=89, sum=6.934676, min=0.000000, average=0.077918, max=2.337869}
Plotting 57 points for GD
Plotting 29 points for CjGD
Plotting 3 points for LBFGS
Returns
In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:
TrainingTester.java:370 executed in 0.00 seconds (0.000 gc):
RefList<double[]> temp_18_0042 = network_target.state();
assert temp_18_0042 != null;
String temp_18_0041 = temp_18_0042.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
temp_18_0042.freeRef();
return temp_18_0041;
Returns
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 7.96 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11553637718980
Reset training subject: 11553786354689
Constructing line search parameters: GD
th(0)=5.794883666384605;dx=-15.46764810898518
Armijo: th(2.154434690031884)=20.379148622713387; dx=29.00648038253439 evalInputDelta=-14.584264956328783
New Minimum: 5.794883666384605 > 1.1099404906658514
WOLF (strong): th(1.077217345015942)=1.1099404906658514; dx=6.769416136774617 evalInputDelta=4.684943175718754
END: th(0.3590724483386473)=1.5716635736154327; dx=-8.055293360398512 evalInputDelta=4.223220092769172
Fitness changed from 5.794883666384605 to 1.1099404906658514
Iteration 1 complete. Error: 1.1099404906658514 Total: 0.8508; Orientation: 0.0003; Line Search: 0.4390
th(0)=1.1099404906658514;dx=-2.962654378803177
New Minimum: 1.1099404906658514 > 0.001169091239775271
WOLF (strong): th(0.7735981389354633)=0.001169091239775271; dx=0.09612369931667873 evalInputDelta=1.1087713994260762
END: th(0.3867990694677316)=0.2597716623715505; dx=-1.433265339743241 evalInputDelta=0.850168828294301
Fitness changed from 1.1099404906658514 to 0.001169091239775271
Iteration 2 complete. Error: 0.001169091239775271 Total: 0.4011; Orientation: 0.0003; Line Search: 0.3005
th(0)=0.001169091239775271;dx=-0.0031206589927897715
New Minimum: 0.001169091239775271 > 1.4720887859164515E-5
WOLF (strong): th(0.8333333333333335)=1.4720887859164515E-5; dx=3.5017014819111467E-4 evalInputDelta=0.0011543703519161065
END: th(0.41666666666666674)=2.3036136163170792E-4; dx=-0.001385244422299313 evalInputDelta=9.38729878143563E-4
Fitness changed from 0.001169091239775271 to 1.4720887859164515E-5
Iteration 3 complete. Error: 1.4720887859164515E-5 Total: 0.4000; Orientation: 0.0003; Line Search: 0.2997
th(0)=1.4720887859164515E-5;dx=-3.929496919783627E-5
New Minimum: 1.4720887859164515E-5 > 5.777448644046366E-7
WOLF (strong): th(0.8976811208466184)=5.777448644046366E-7; dx=7.784574991435834E-6 evalInputDelta=1.414314299475988E-5
END: th(0.4488405604233092)=2.3665141121877034E-6; dx=-1.575519710319926E-5 evalInputDelta=1.2354373746976812E-5
Fitness changed from 1.4720887859164515E-5 to 5.777448644046366E-7
Iteration 4 complete. Error: 5.777448644046366E-7 Total: 0.4008; Orientation: 0.0002; Line Search: 0.3008
th(0)=5.777448644046366E-7;dx=-1.542204947182815E-6
New Minimum: 5.777448644046366E-7 > 4.8799913487442324E-8
WOLF (strong): th(0.9669976736693292)=4.8799913487442324E-8; dx=4.482106898645187E-7 evalInputDelta=5.289449509171943E-7
END: th(0.4834988368346646)=7.268147761358995E-8; dx=-5.469971286592453E-7 evalInputDelta=5.050633867910467E-7
Fitness changed from 5.777448644046366E-7 to 4.8799913487442324E-8
Iteration 5 complete. Error: 4.8799913487442324E-8 Total: 0.4031; Orientation: 0.0003; Line Search: 0.3028
th(0)=4.8799913487442324E-8;dx=-1.3026486035331717E-7
New Minimum: 4.8799913487442324E-8 > 7.433758724527523E-9
WOLF (strong): th(1.041666666666667)=7.433758724527523E-9; dx=5.08418432085324E-8 evalInputDelta=4.13661547629148E-8
New Minimum: 7.433758724527523E-9 > 4.535234079696355E-9
END: th(0.5208333333333335)=4.535234079696355E-9; dx=-3.9711508572381185E-8 evalInputDelta=4.426467940774597E-8
Fitness changed from 4.8799913487442324E-8 to 4.535234079696355E-9
Iteration 6 complete. Error: 4.535234079696355E-9 Total: 0.3998; Orientation: 0.0003; Line Search: 0.2998
th(0)=4.535234079696355E-9;dx=-1.210617067850544E-8
New Minimum: 4.535234079696355E-9 > 1.123163939422078E-9
WOLF (strong): th(1.122101401058273)=1.123163939422078E-9; dx=6.024598840066745E-9 evalInputDelta=3.412070140274277E-9
New Minimum: 1.123163939422078E-9 > 2.8612877467928904E-10
END: th(0.5610507005291365)=2.8612877467928904E-10; dx=-3.040785919228941E-9 evalInputDelta=4.249105305017066E-9
Fitness changed from 4.535234079696355E-9 to 2.8612877467928904E-10
Iteration 7 complete. Error: 2.8612877467928904E-10 Total: 0.4001; Orientation: 0.0003; Line Search: 0.3001
th(0)=2.8612877467928904E-10;dx=-7.637779650932636E-10
New Minimum: 2.8612877467928904E-10 > 1.0761867649492578E-10
WOLF (strong): th(1.2087470920866614)=1.0761867649492578E-10; dx=4.684141137912024E-10 evalInputDelta=1.7851009818436326E-10
New Minimum: 1.0761867649492578E-10 > 1.0697651556242329E-11
END: th(0.6043735460433307)=1.0697651556242329E-11; dx=-1.4768192564775066E-10 evalInputDelta=2.754311231230467E-10
Fitness changed from 2.8612877467928904E-10 to 1.0697651556242329E-11
Iteration 8 complete. Error: 1.0697651556242329E-11 Total: 0.4000; Orientation: 0.0002; Line Search: 0.2998
Low gradient: 5.343748001600949E-6
th(0)=1.0697651556242329E-11;dx=-2.855564270461413E-11
New Minimum: 1.0697651556242329E-11 > 5.824061484373524E-12
WOLF (strong): th(1.3020833333333337)=5.824061484373524E-12; dx=2.106980835417849E-11 evalInputDelta=4.873590071868805E-12
New Minimum: 5.824061484373524E-12 > 1.837974287113076E-13
END: th(0.6510416666666669)=1.837974287113076E-13; dx=-3.742917174756757E-12 evalInputDelta=1.0513854127531021E-11
Fitness changed from 1.0697651556242329E-11 to 1.837974287113076E-13
Iteration 9 complete. Error: 1.837974287113076E-13 Total: 0.4033; Orientation: 0.0003; Line Search: 0.3002
Low gradient: 7.004381412238413E-7
th(0)=1.837974287113076E-13;dx=-4.906135896811098E-13
New Minimum: 1.837974287113076E-13 > 1.3976587079051695E-13
WOLF (strong): th(1.4026267513228414)=1.3976587079051695E-13; dx=4.278291634180861E-13 evalInputDelta=4.403155792079064E-14
New Minimum: 1.3976587079051695E-13 > 7.526028725225E-16
END: th(0.7013133756614207)=7.526028725225E-16; dx=-3.139221315306185E-14 evalInputDelta=1.830448258387851E-13
Fitness changed from 1.837974287113076E-13 to 7.526028725225E-16
Iteration 10 complete. Error: 7.526028725225E-16 Total: 0.3999; Orientation: 0.0003; Line Search: 0.3001
Low gradient: 4.482076688563152E-8
th(0)=7.526028725225E-16;dx=-2.008901144216123E-15
Armijo: th(1.510933865108327)=7.77716009001484E-16; dx=2.042143018061404E-15 evalInputDelta=-2.511313647898399E-17
New Minimum: 7.526028725225E-16 > 5.196360219395596E-20
WOLF (strong): th(0.7554669325541635)=5.196360219395596E-20; dx=1.662092962065121E-17 evalInputDelta=7.52550908920306E-16
END: th(0.25182231085138784)=3.317286873827754E-16; dx=-1.3337271204274536E-15 evalInputDelta=4.2087418513972463E-16
Fitness changed from 7.526028725225E-16 to 5.196360219395596E-20
Iteration 11 complete. Error: 5.196360219395596E-20 Total: 0.5011; Orientation: 0.0003; Line Search: 0.4007
Low gradient: 3.7245700292807663E-10
th(0)=5.196360219395596E-20;dx=-1.3872421903016525E-19
New Minimum: 5.196360219395596E-20 > 3.953034576041106E-21
END: th(0.5425347222222224)=3.953034576041106E-21; dx=-3.8261905605192944E-20 evalInputDelta=4.8010567617914854E-20
Fitness changed from 5.196360219395596E-20 to 3.953034576041106E-21
Iteration 12 complete. Error: 3.953034576041106E-21 Total: 0.3000; Orientation: 0.0003; Line Search: 0.1997
Low gradient: 1.027285673782417E-10
th(0)=3.953034576041106E-21;dx=-1.0553158555585944E-20
New Minimum: 3.953034576041106E-21 > 1.2406010075984629E-21
WOLF (strong): th(1.168855626102368)=1.2406010075984629E-21; dx=5.911980239890392E-21 evalInputDelta=2.712433568442643E-21
New Minimum: 1.2406010075984629E-21 > 1.9114627954072227E-22
END: th(0.584427813051184)=1.9114627954072227E-22; dx=-2.320586979512567E-21 evalInputDelta=3.7618882965003834E-21
Fitness changed from 3.953034576041106E-21 to 1.9114627954072227E-22
Iteration 13 complete. Error: 1.9114627954072227E-22 Total: 0.3998; Orientation: 0.0003; Line Search: 0.2998
Zero gradient: 2.258957421690842E-11
th(0)=1.9114627954072227E-22;dx=-5.102888633012137E-22
New Minimum: 1.9114627954072227E-22 > 8.856396521144976E-23
WOLF (strong): th(1.2591115542569393)=8.856396521144976E-23; dx=3.4734540565007333E-22 evalInputDelta=1.0258231432927251E-22
New Minimum: 8.856396521144976E-23 > 4.872679934358913E-24
END: th(0.6295557771284697)=4.872679934358913E-24; dx=-8.147282274821982E-23 evalInputDelta=1.8627359960636335E-22
Fitness changed from 1.9114627954072227E-22 to 4.872679934358913E-24
Iteration 14 complete. Error: 4.872679934358913E-24 Total: 0.4006; Orientation: 0.0003; Line Search: 0.3007
Zero gradient: 3.606682099432274E-12
th(0)=4.872679934358913E-24;dx=-1.3008155766365194E-23
New Minimum: 4.872679934358913E-24 > 3.2003240953770058E-24
WOLF (strong): th(1.3563368055555562)=3.2003240953770058E-24; dx=1.0542138355961517E-23 evalInputDelta=1.6723558389819073E-24
New Minimum: 3.2003240953770058E-24 > 4.3779535804865903E-26
END: th(0.6781684027777781)=4.3779535804865903E-26; dx=-1.2329766608418889E-24 evalInputDelta=4.828900398554047E-24
Fitness changed from 4.872679934358913E-24 to 4.3779535804865903E-26
Iteration 15 complete. Error: 4.3779535804865903E-26 Total: 0.4004; Orientation: 0.0003; Line Search: 0.3006
Zero gradient: 3.4186699587563265E-13
th(0)=4.3779535804865903E-26;dx=-1.1687304286902984E-25
New Minimum: 4.3779535804865903E-26 > 3.9500736126316086E-26
WOLF (strong): th(1.4610695326279601)=3.9500736126316086E-26; dx=1.1101491477133008E-25 evalInputDelta=4.278799678549817E-27
New Minimum: 3.9500736126316086E-26 > 2.7479015626345214E-29
END: th(0.7305347663139801)=2.7479015626345214E-29; dx=-2.9245704777175048E-27 evalInputDelta=4.375205678923956E-26
Fitness changed from 4.3779535804865903E-26 to 2.7479015626345214E-29
Iteration 16 complete. Error: 2.7479015626345214E-29 Total: 0.4004; Orientation: 0.0003; Line Search: 0.3003
Zero gradient: 8.559523219074118E-15
th(0)=2.7479015626345214E-29;dx=-7.326543773786894E-29
Armijo: th(1.5738894428211743)=3.2538476702630644E-29; dx=7.972757758108115E-29 evalInputDelta=-5.05946107628543E-30
New Minimum: 2.7479015626345214E-29 > 9.433380570614615E-32
WOLF (strong): th(0.7869447214105871)=9.433380570614615E-32; dx=3.3797241076736836E-30 evalInputDelta=2.7384681820639066E-29
END: th(0.2623149071368624)=1.2080193879892068E-29; dx=-4.8529857792685725E-29 evalInputDelta=1.5398821746453146E-29
Fitness changed from 2.7479015626345214E-29 to 9.433380570614615E-32
Iteration 17 complete. Error: 9.433380570614615E-32 Total: 0.5007; Orientation: 0.0003; Line Search: 0.4005
Zero gradient: 4.055108662279244E-16
th(0)=9.433380570614615E-32;dx=-1.6443906262892162E-31
New Minimum: 9.433380570614615E-32 > 0.0
END: th(0.5651403356481485)=0.0; dx=0.0 evalInputDelta=9.433380570614615E-32
Fitness changed from 9.433380570614615E-32 to 0.0
Iteration 18 complete. Error: 0.0 Total: 0.3019; Orientation: 0.0003; Line Search: 0.2010
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.3004; Orientation: 0.0003; Line Search: 0.1996
Iteration 19 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 19
Final threshold in iteration 19: 0.0 (> 0.0) after 7.965s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 5.41 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11561622177707
Reset training subject: 11561722086494
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=5.794883666384605}, derivative=-15.46764810898518}
New Minimum: 5.794883666384605 > 5.79488366483784
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=5.79488366483784}, derivative=-15.467648106921153}, evalInputDelta = -1.5467644942646075E-9
New Minimum: 5.79488366483784 > 5.794883655557251
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=5.794883655557251}, derivative=-15.4676480945348}, evalInputDelta = -1.0827354124387512E-8
New Minimum: 5.794883655557251 > 5.7948835905931295
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=5.7948835905931295}, derivative=-15.467648007834153}, evalInputDelta = -7.57914753179989E-8
New Minimum: 5.7948835905931295 > 5.7948831358442865
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=5.7948831358442865}, derivative=-15.46764740092809}, evalInputDelta = -5.305403183442081E-7
New Minimum: 5.7948831358442865 > 5.794879952602889
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=5.794879952602889}, derivative=-15.467643152586023}, evalInputDelta = -3.7137817159305087E-6
New Minimum: 5.794879952602889 > 5.794857669937583
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=5.794857669937583}, derivative=-15.467613414191302}, evalInputDelta = -2.5996447021725544E-5
New Minimum: 5.794857669937583 > 5.79470169248
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=5.79470169248}, derivative=-15.46740524542736}, evalInputDelta = -1.8197390460450436E-4
New Minimum: 5.79470169248 > 5.793609909054947
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=5.793609909054947}, derivative=-15.465948064079143}, evalInputDelta = -0.001273757329657954
New Minimum: 5.793609909054947 > 5.785970305203174
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=5.785970305203174}, derivative=-15.455747794643372}, evalInputDelta = -0.008913361181431156
New Minimum: 5.785970305203174 > 5.73263420429702
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=5.73263420429702}, derivative=-15.384345908592737}, evalInputDelta = -0.06224946208758464
New Minimum: 5.73263420429702 > 5.366196674710942
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=5.366196674710942}, derivative=-14.884532706238247}, evalInputDelta = -0.4286869916736631
New Minimum: 5.366196674710942 > 3.139977628701006
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=3.139977628701006}, derivative=-11.385840289756464}, evalInputDelta = -2.654906037683599
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=4.15978370016955}, derivative=13.105006625616586}, evalInputDelta = -1.635099966215055
4.15978370016955 <= 5.794883666384605
New Minimum: 3.139977628701006 > 3.5240854545193343E-6
F(0.7492904029712043) = LineSearchPoint{point=PointSample{avg=3.5240854545193343E-6}, derivative=1.3461887854449017E-14}, evalInputDelta = -5.79488014229915
Right bracket at 0.7492904029712043
Converged to right
Fitness changed from 5.794883666384605 to 3.5240854545193343E-6
Iteration 1 complete. Error: 3.5240854545193343E-6 Total: 1.9062; Orientation: 0.0003; Line Search: 1.6069
F(0.0) = LineSearchPoint{point=PointSample{avg=3.5240854545193343E-6}, derivative=-9.401794323837635E-6}
New Minimum: 3.5240854545193343E-6 > 8.813832084643148E-12
F(0.7492904029712043) = LineSearchPoint{point=PointSample{avg=8.813832084643148E-12}, derivative=-4.643091383369137E-9}, evalInputDelta = -3.5240766406872497E-6
F(5.24503282079843) = LineSearchPoint{point=PointSample{avg=1.2672064887839213E-4}, derivative=5.637826430334068E-5}, evalInputDelta = 1.231965634238728E-4
F(0.40346406313834077) = LineSearchPoint{point=PointSample{avg=7.515643053404604E-7}, derivative=-4.3417898140545635E-6}, evalInputDelta = -2.772521149178874E-6
F(2.8242484419683853) = LineSearchPoint{point=PointSample{avg=2.6988566969590146E-5}, derivative=2.6018237244642897E-5}, evalInputDelta = 2.346448151507081E-5
F(0.21724988015141425) = LineSearchPoint{point=PointSample{avg=1.77750821111932E-6}, derivative=-6.677176510877536E-6}, evalInputDelta = -1.7465772434000143E-6
F(1.5207491610598998) = LineSearchPoint{point=PointSample{avg=3.7284255105230528E-6}, derivative=9.67053036688288E-6}, evalInputDelta = 2.043400560037184E-7
F(0.11698070469691536) = LineSearchPoint{point=PointSample{avg=2.5100682361215012E-6}, derivative=-7.934692424551257E-6}, evalInputDelta = -1.014017218397833E-6
F(0.8188649328784076) = LineSearchPoint{point=PointSample{avg=3.003982059012851E-8}, derivative=8.679189711658006E-7}, evalInputDelta = -3.494045633929206E-6
3.003982059012851E-8 <= 3.5240854545193343E-6
New Minimum: 8.813832084643148E-12 > 7.954346848590729E-12
F(0.7496606240869086) = LineSearchPoint{point=PointSample{avg=7.954346848590729E-12}, derivative=-1.8862204677925519E-19}, evalInputDelta = -3.524077500172486E-6
Left bracket at 0.7496606240869086
Converged to left
Fitness changed from 3.5240854545193343E-6 to 7.954346848590729E-12
Iteration 2 complete. Error: 7.954346848590729E-12 Total: 1.2025; Orientation: 0.0003; Line Search: 1.1025
Low gradient: 4.6054720688420466E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=7.954346848590729E-12}, derivative=-2.121037297688424E-11}
New Minimum: 7.954346848590729E-12 > 2.2947966100716916E-17
F(0.7496606240869086) = LineSearchPoint{point=PointSample{avg=2.2947966100716916E-17}, derivative=-1.0760013308953248E-14}, evalInputDelta = -7.954323900624629E-12
F(5.24762436860836) = LineSearchPoint{point=PointSample{avg=2.860168963225392E-10}, derivative=1.27186917768226E-10}, evalInputDelta = 2.780625494739485E-10
F(0.40366341296987385) = LineSearchPoint{point=PointSample{avg=1.6964397969675069E-12}, derivative=-9.795196765907718E-12}, evalInputDelta = -6.257907051623222E-12
F(2.825643890789117) = LineSearchPoint{point=PointSample{avg=6.0914666256466E-11}, derivative=5.869586050107891E-11}, evalInputDelta = 5.2960319407875264E-11
F(0.2173572223683936) = LineSearchPoint{point=PointSample{avg=4.012126668178568E-12}, derivative=-1.5063739632235113E-11}, evalInputDelta = -3.942220180412161E-12
F(1.5215005565787552) = LineSearchPoint{point=PointSample{avg=8.415123750245717E-12}, derivative=2.1816060434133187E-11}, evalInputDelta = 4.607769016549883E-13
F(0.11703850435221194) = LineSearchPoint{point=PointSample{avg=5.6655991884065184E-12}, derivative=-1.790064732987819E-11}, evalInputDelta = -2.2887476601842107E-12
F(0.8192695304654836) = LineSearchPoint{point=PointSample{avg=6.778535766165049E-14}, derivative=1.9577065525258774E-12}, evalInputDelta = -7.886561490929078E-12
6.778535766165049E-14 <= 7.954346848590729E-12
New Minimum: 2.2947966100716916E-17 > 2.0900897600033122E-17
F(0.7500411196237018) = LineSearchPoint{point=PointSample{avg=2.0900897600033122E-17}, derivative=-3.193168515260613E-22}, evalInputDelta = -7.954325947693129E-12
Left bracket at 0.7500411196237018
Converged to left
Fitness changed from 7.954346848590729E-12 to 2.0900897600033122E-17
Iteration 3 complete. Error: 2.0900897600033122E-17 Total: 1.2029; Orientation: 0.0003; Line Search: 1.1030
Low gradient: 7.467381696916655E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=2.0900897600033122E-17}, derivative=-5.576178940744586E-17}
New Minimum: 2.0900897600033122E-17 > 6.072176758987323E-23
F(0.7500411196237018) = LineSearchPoint{point=PointSample{avg=6.072176758987323E-23}, derivative=2.9280424619088876E-20}, evalInputDelta = -2.090083687826553E-17
6.072176758987323E-23 <= 2.0900897600033122E-17
Converged to right
Fitness changed from 2.0900897600033122E-17 to 6.072176758987323E-23
Iteration 4 complete. Error: 6.072176758987323E-23 Total: 0.2999; Orientation: 0.0003; Line Search: 0.1997
Zero gradient: 1.2730967311233716E-11
F(0.0) = LineSearchPoint{point=PointSample{avg=6.072176758987323E-23}, derivative=-1.6207752867970143E-22}
New Minimum: 6.072176758987323E-23 > 1.914199535956351E-28
F(0.7500411196237018) = LineSearchPoint{point=PointSample{avg=1.914199535956351E-28}, derivative=1.6247622438041993E-25}, evalInputDelta = -6.072157616991964E-23
1.914199535956351E-28 <= 6.072176758987323E-23
Converged to right
Fitness changed from 6.072176758987323E-23 to 1.914199535956351E-28
Iteration 5 complete. Error: 1.914199535956351E-28 Total: 0.3002; Orientation: 0.0003; Line Search: 0.2002
Zero gradient: 2.2605927428311422E-14
F(0.0) = LineSearchPoint{point=PointSample{avg=1.914199535956351E-28}, derivative=-5.110279548940827E-28}
New Minimum: 1.914199535956351E-28 > 0.0
F(0.7500411196237018) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -1.914199535956351E-28
0.0 <= 1.914199535956351E-28
Converged to right
Fitness changed from 1.914199535956351E-28 to 0.0
Iteration 6 complete. Error: 0.0 Total: 0.2996; Orientation: 0.0003; Line Search: 0.1998
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.2003; Orientation: 0.0003; Line Search: 0.1002
Iteration 7 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 7
Final threshold in iteration 7: 0.0 (> 0.0) after 5.411s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 8.21 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11567052879349
Reset training subject: 11567153540705
Adding measurement 6bffd359 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 5.794883666384605 < 5.794883666384605. Total: 1
th(0)=5.794883666384605;dx=-15.46764810898518
Non-optimal measurement 20.379148622713387 < 5.794883666384605. Total: 1
Armijo: th(2.154434690031884)=20.379148622713387; dx=29.00648038253439 evalInputDelta=-14.584264956328783
Adding measurement 77dd6f52 to history. Total: 1
New Minimum: 5.794883666384605 > 1.1099404906658514
WOLF (strong): th(1.077217345015942)=1.1099404906658514; dx=6.769416136774617 evalInputDelta=4.684943175718754
Non-optimal measurement 1.5716635736154327 < 1.1099404906658514. Total: 2
END: th(0.3590724483386473)=1.5716635736154327; dx=-8.055293360398512 evalInputDelta=4.223220092769172
Fitness changed from 5.794883666384605 to 1.1099404906658514
Iteration 1 complete. Error: 1.1099404906658514 Total: 0.7058; Orientation: 0.0009; Line Search: 0.4045
Non-optimal measurement 1.1099404906658514 < 1.1099404906658514. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 1.1099404906658514 < 1.1099404906658514. Total: 2
th(0)=1.1099404906658514;dx=-2.962654378803177
Adding measurement 354025e0 to history. Total: 2
New Minimum: 1.1099404906658514 > 0.001169091239775271
WOLF (strong): th(0.7735981389354633)=0.001169091239775271; dx=0.09612369931667873 evalInputDelta=1.1087713994260762
Non-optimal measurement 0.2597716623715505 < 0.001169091239775271. Total: 3
END: th(0.3867990694677316)=0.2597716623715505; dx=-1.433265339743241 evalInputDelta=0.850168828294301
Fitness changed from 1.1099404906658514 to 0.001169091239775271
Iteration 2 complete. Error: 0.001169091239775271 Total: 0.4020; Orientation: 0.0006; Line Search: 0.3009
Non-optimal measurement 0.001169091239775271 < 0.001169091239775271. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.001169091239775271 < 0.001169091239775271. Total: 3
th(0)=0.001169091239775271;dx=-0.0031206589927897715
Adding measurement 2c1a159a to history. Total: 3
New Minimum: 0.001169091239775271 > 1.4720887859164515E-5
WOLF (strong): th(0.8333333333333335)=1.4720887859164515E-5; dx=3.5017014819111467E-4 evalInputDelta=0.0011543703519161065
Non-optimal measurement 2.3036136163170792E-4 < 1.4720887859164515E-5. Total: 4
END: th(0.41666666666666674)=2.3036136163170792E-4; dx=-0.001385244422299313 evalInputDelta=9.38729878143563E-4
Fitness changed from 0.001169091239775271 to 1.4720887859164515E-5
Iteration 3 complete. Error: 1.4720887859164515E-5 Total: 0.4102; Orientation: 0.0006; Line Search: 0.3095
Non-optimal measurement 1.4720887859164515E-5 < 1.4720887859164515E-5. Total: 4
Rejected: LBFGS Orientation magnitude: 4.697e-03, gradient 6.269e-03, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.4720887859164515E-5, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
LBFGS Accumulation History: 3 points
Removed measurement 2c1a159a to history. Total: 3
Adding measurement 890c8b5 to history. Total: 3
th(0)=1.4720887859164515E-5;dx=-3.929496919783627E-5
Adding measurement 1efd277 to history. Total: 4
New Minimum: 1.4720887859164515E-5 > 5.777448644046366E-7
WOLF (strong): th(0.8976811208466184)=5.777448644046366E-7; dx=7.784574991435834E-6 evalInputDelta=1.414314299475988E-5
Non-optimal measurement 2.3665141121877034E-6 < 5.777448644046366E-7. Total: 5
END: th(0.4488405604233092)=2.3665141121877034E-6; dx=-1.575519710319926E-5 evalInputDelta=1.2354373746976812E-5
Fitness changed from 1.4720887859164515E-5 to 5.777448644046366E-7
Iteration 4 complete. Error: 5.777448644046366E-7 Total: 0.4139; Orientation: 0.0075; Line Search: 0.3036
Non-optimal measurement 5.777448644046366E-7 < 5.777448644046366E-7. Total: 5
Rejected: LBFGS Orientation magnitude: 9.305e-04, gradient 1.242e-03, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 5.777448644046366E-7, 1.4720887859164515E-5, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 9.305e-04, gradient 1.242e-03, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 5.777448644046366E-7, 1.4720887859164515E-5, 0.001169091239775271, 1.1099404906658514
LBFGS Accumulation History: 3 points
Removed measurement 1efd277 to history. Total: 4
Removed measurement 890c8b5 to history. Total: 3
Adding measurement 16f79d6b to history. Total: 3
th(0)=5.777448644046366E-7;dx=-1.542204947182815E-6
Adding measurement 47cda019 to history. Total: 4
New Minimum: 5.777448644046366E-7 > 4.8799913487442324E-8
WOLF (strong): th(0.9669976736693292)=4.8799913487442324E-8; dx=4.482106898645187E-7 evalInputDelta=5.289449509171943E-7
Non-optimal measurement 7.268147761358995E-8 < 4.8799913487442324E-8. Total: 5
END: th(0.4834988368346646)=7.268147761358995E-8; dx=-5.469971286592453E-7 evalInputDelta=5.050633867910467E-7
Fitness changed from 5.777448644046366E-7 to 4.8799913487442324E-8
Iteration 5 complete. Error: 4.8799913487442324E-8 Total: 0.4196; Orientation: 0.0181; Line Search: 0.3020
Non-optimal measurement 4.8799913487442324E-8 < 4.8799913487442324E-8. Total: 5
Rejected: LBFGS Orientation magnitude: 2.704e-04, gradient 3.609e-04, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.8799913487442324E-8, 5.777448644046366E-7, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 2.704e-04, gradient 3.609e-04, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.8799913487442324E-8, 5.777448644046366E-7, 0.001169091239775271, 1.1099404906658514
LBFGS Accumulation History: 3 points
Removed measurement 47cda019 to history. Total: 4
Removed measurement 16f79d6b to history. Total: 3
Adding measurement 3485e664 to history. Total: 3
th(0)=4.8799913487442324E-8;dx=-1.3026486035331717E-7
Adding measurement 4b9a3b1d to history. Total: 4
New Minimum: 4.8799913487442324E-8 > 7.433758724527523E-9
WOLF (strong): th(1.041666666666667)=7.433758724527523E-9; dx=5.08418432085324E-8 evalInputDelta=4.13661547629148E-8
Adding measurement 19b07e80 to history. Total: 5
New Minimum: 7.433758724527523E-9 > 4.535234079696355E-9
END: th(0.5208333333333335)=4.535234079696355E-9; dx=-3.9711508572381185E-8 evalInputDelta=4.426467940774597E-8
Fitness changed from 4.8799913487442324E-8 to 4.535234079696355E-9
Iteration 6 complete. Error: 4.535234079696355E-9 Total: 0.4266; Orientation: 0.0212; Line Search: 0.3051
Non-optimal measurement 4.535234079696355E-9 < 4.535234079696355E-9. Total: 6
Rejected: LBFGS Orientation magnitude: 8.244e-05, gradient 1.100e-04, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.535234079696355E-9, 7.433758724527523E-9, 4.8799913487442324E-8, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 8.244e-05, gradient 1.100e-04, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.535234079696355E-9, 7.433758724527523E-9, 4.8799913487442324E-8, 0.001169091239775271, 1.1099404906658514
Rejected: LBFGS Orientation magnitude: 8.244e-05, gradient 1.100e-04, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.535234079696355E-9, 7.433758724527523E-9, 4.8799913487442324E-8, 0.001169091239775271
LBFGS Accumulation History: 3 points
Removed measurement 19b07e80 to history. Total: 5
Removed measurement 4b9a3b1d to history. Total: 4
Removed measurement 3485e664 to history. Total: 3
Adding measurement 2a0e7d44 to history. Total: 3
th(0)=4.535234079696355E-9;dx=-1.210617067850544E-8
Adding measurement 72eb9393 to history. Total: 4
New Minimum: 4.535234079696355E-9 > 1.123163939422078E-9
WOLF (strong)
...skipping 13012 bytes...
913E-24 < 4.872679934358913E-24. Total: 6
Rejected: LBFGS Orientation magnitude: 2.702e-12, gradient 3.607e-12, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.872679934358913E-24, 8.856396521144976E-23, 1.9114627954072227E-22, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 2.702e-12, gradient 3.607e-12, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.872679934358913E-24, 8.856396521144976E-23, 1.9114627954072227E-22, 0.001169091239775271, 1.1099404906658514
Rejected: LBFGS Orientation magnitude: 2.702e-12, gradient 3.607e-12, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.872679934358913E-24, 8.856396521144976E-23, 1.9114627954072227E-22, 0.001169091239775271
LBFGS Accumulation History: 3 points
Removed measurement 4b0c2366 to history. Total: 5
Removed measurement 1672be55 to history. Total: 4
Removed measurement 5b3f4768 to history. Total: 3
Adding measurement 42707598 to history. Total: 3
th(0)=4.872679934358913E-24;dx=-1.3008155766365194E-23
Adding measurement 1024d749 to history. Total: 4
New Minimum: 4.872679934358913E-24 > 3.2003240953770058E-24
WOLF (strong): th(1.3563368055555562)=3.2003240953770058E-24; dx=1.0542138355961517E-23 evalInputDelta=1.6723558389819073E-24
Adding measurement 53d435e6 to history. Total: 5
New Minimum: 3.2003240953770058E-24 > 4.3779535804865903E-26
END: th(0.6781684027777781)=4.3779535804865903E-26; dx=-1.2329766608418889E-24 evalInputDelta=4.828900398554047E-24
Fitness changed from 4.872679934358913E-24 to 4.3779535804865903E-26
Iteration 15 complete. Error: 4.3779535804865903E-26 Total: 0.4272; Orientation: 0.0262; Line Search: 0.3007
Non-optimal measurement 4.3779535804865903E-26 < 4.3779535804865903E-26. Total: 6
Rejected: LBFGS Orientation magnitude: 2.561e-13, gradient 3.419e-13, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.3779535804865903E-26, 3.2003240953770058E-24, 4.872679934358913E-24, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 2.561e-13, gradient 3.419e-13, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.3779535804865903E-26, 3.2003240953770058E-24, 4.872679934358913E-24, 0.001169091239775271, 1.1099404906658514
Rejected: LBFGS Orientation magnitude: 2.561e-13, gradient 3.419e-13, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.3779535804865903E-26, 3.2003240953770058E-24, 4.872679934358913E-24, 0.001169091239775271
LBFGS Accumulation History: 3 points
Removed measurement 53d435e6 to history. Total: 5
Removed measurement 1024d749 to history. Total: 4
Removed measurement 42707598 to history. Total: 3
Adding measurement c1203f2 to history. Total: 3
th(0)=4.3779535804865903E-26;dx=-1.1687304286902984E-25
Adding measurement 442e0310 to history. Total: 4
New Minimum: 4.3779535804865903E-26 > 3.9500736126316086E-26
WOLF (strong): th(1.4610695326279601)=3.9500736126316086E-26; dx=1.1101491477133008E-25 evalInputDelta=4.278799678549817E-27
Adding measurement 349b58f3 to history. Total: 5
New Minimum: 3.9500736126316086E-26 > 2.7479015626345214E-29
END: th(0.7305347663139801)=2.7479015626345214E-29; dx=-2.9245704777175048E-27 evalInputDelta=4.375205678923956E-26
Fitness changed from 4.3779535804865903E-26 to 2.7479015626345214E-29
Iteration 16 complete. Error: 2.7479015626345214E-29 Total: 0.4272; Orientation: 0.0254; Line Search: 0.3018
Non-optimal measurement 2.7479015626345214E-29 < 2.7479015626345214E-29. Total: 6
Rejected: LBFGS Orientation magnitude: 6.413e-15, gradient 8.560e-15, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.7479015626345214E-29, 3.9500736126316086E-26, 4.3779535804865903E-26, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 6.413e-15, gradient 8.560e-15, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.7479015626345214E-29, 3.9500736126316086E-26, 4.3779535804865903E-26, 0.001169091239775271, 1.1099404906658514
Rejected: LBFGS Orientation magnitude: 6.413e-15, gradient 8.560e-15, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.7479015626345214E-29, 3.9500736126316086E-26, 4.3779535804865903E-26, 0.001169091239775271
LBFGS Accumulation History: 3 points
Removed measurement 349b58f3 to history. Total: 5
Removed measurement 442e0310 to history. Total: 4
Removed measurement c1203f2 to history. Total: 3
Adding measurement 4a98b4f1 to history. Total: 3
th(0)=2.7479015626345214E-29;dx=-7.326543773786894E-29
Non-optimal measurement 3.2538476702630644E-29 < 2.7479015626345214E-29. Total: 4
Armijo: th(1.5738894428211743)=3.2538476702630644E-29; dx=7.972757758108115E-29 evalInputDelta=-5.05946107628543E-30
Adding measurement 790f0984 to history. Total: 4
New Minimum: 2.7479015626345214E-29 > 9.433380570614615E-32
WOLF (strong): th(0.7869447214105871)=9.433380570614615E-32; dx=3.3797241076736836E-30 evalInputDelta=2.7384681820639066E-29
Non-optimal measurement 1.2080193879892068E-29 < 9.433380570614615E-32. Total: 5
END: th(0.2623149071368624)=1.2080193879892068E-29; dx=-4.8529857792685725E-29 evalInputDelta=1.5398821746453146E-29
Fitness changed from 2.7479015626345214E-29 to 9.433380570614615E-32
Iteration 17 complete. Error: 9.433380570614615E-32 Total: 0.5287; Orientation: 0.0282; Line Search: 0.4006
Non-optimal measurement 9.433380570614615E-32 < 9.433380570614615E-32. Total: 5
Rejected: LBFGS Orientation magnitude: 3.039e-16, gradient 4.055e-16, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 9.433380570614615E-32, 2.7479015626345214E-29, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 3.039e-16, gradient 4.055e-16, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 9.433380570614615E-32, 2.7479015626345214E-29, 0.001169091239775271, 1.1099404906658514
LBFGS Accumulation History: 3 points
Removed measurement 790f0984 to history. Total: 4
Removed measurement 4a98b4f1 to history. Total: 3
Adding measurement 773ccf8e to history. Total: 3
th(0)=9.433380570614615E-32;dx=-1.6443906262892162E-31
Adding measurement 3821d5e6 to history. Total: 4
New Minimum: 9.433380570614615E-32 > 0.0
END: th(0.5651403356481485)=0.0; dx=0.0 evalInputDelta=9.433380570614615E-32
Fitness changed from 9.433380570614615E-32 to 0.0
Iteration 18 complete. Error: 0.0 Total: 0.3200; Orientation: 0.0181; Line Search: 0.2015
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 9.433380570614615E-32, 0.001169091239775271, 1.1099404906658514, 5.794883666384605
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 9.433380570614615E-32, 0.001169091239775271, 1.1099404906658514
LBFGS Accumulation History: 3 points
Removed measurement 3821d5e6 to history. Total: 4
Removed measurement 773ccf8e to history. Total: 3
Adding measurement 19ee0c91 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.3151; Orientation: 0.0149; Line Search: 0.2004
Iteration 19 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 19
Final threshold in iteration 19: 0.0 (> 0.0) after 8.212s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.01 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -31.02533264447743], [18.0, 0.04529969476120061]; valueStats=DoubleSummaryStatistics{count=39, sum=2.222253, min=0.000000, average=0.056981, max=1.109940}
Plotting 18 points for GD
Plotting 6 points for CjGD
Plotting 18 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -31.02533264447743], [7.191, 0.04529969476120061]; valueStats=DoubleSummaryStatistics{count=39, sum=2.222253, min=0.000000, average=0.056981, max=1.109940}
Plotting 18 points for GD
Plotting 6 points for CjGD
Plotting 18 points for LBFGS
Returns
In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:
TrainingTester.java:279 executed in 0.00 seconds (0.000 gc):
RefList<double[]> temp_18_0037 = temp_18_0035.state();
assert temp_18_0037 != null;
String temp_18_0036 = temp_18_0037.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
temp_18_0037.freeRef();
return temp_18_0036;
Returns
We simultaneously regress this target input:
TrainingTester.java:287 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(x -> {
RefStream<Tensor> temp_18_0006 = RefArrays.stream(RefUtil.addRef(x));
if (null != x)
RefUtil.freeRef(x);
return temp_18_0006;
}).map(x -> {
String temp_18_0007 = x.prettyPrint();
x.freeRef();
return temp_18_0007;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.676, -0.668, -1.16 ], [ -1.656, 1.68, -0.572 ], [ -1.744, -1.664, -1.728 ], [ -1.848, -1.556, 1.536 ], [ -1.26, -0.532, 1.82 ], [ -0.612, -0.832, -0.38 ], [ -1.236, -1.368, -0.496 ], [ -1.268, -1.312, -0.964 ], ... ],
[ [ -1.912, -0.28, -1.188 ], [ -1.344, 1.52, 1.808 ], [ -0.36, -0.612, -0.756 ], [ -1.716, 0.932, -1.176 ], [ 0.06, 1.72, 0.584 ], [ 0.632, 0.528, -0.808 ], [ 1.188, 0.852, -0.544 ], [ 1.732, 1.924, 0.156 ], ... ],
[ [ -0.528, 0.468, 1.172 ], [ -1.32, 0.768, 1.552 ], [ -0.992, 0.824, -1.272 ], [ -0.06, -1.56, 0.236 ], [ 1.628, 1.696, -1.876 ], [ -0.324, -0.06, 1.64 ], [ 0.776, 1.932, -0.316 ], [ -0.208, -1.588, 0.852 ], ... ],
[ [ 1.712, 0.852, -1.008 ], [ 1.616, -0.388, 1.468 ], [ -0.1, 0.596, 1.688 ], [ 1.748, -0.892, 0.892 ], [ -1.56, -0.528, -1.444 ], [ -0.036, -0.768, 1.776 ], [ -0.68, 0.924, 0.584 ], [ -0.484, 0.8, -1.66 ], ... ],
[ [ 1.272, -0.548, 1.924 ], [ -0.58, -0.612, 0.664 ], [ 0.608, -0.488, 0.788 ], [ 0.652, 1.792, 0.064 ], [ 1.54, -0.976, 0.024 ], [ 1.524, 1.736, -1.092 ], [ 1.868, 1.116, -0.744 ], [ -1.52, -1.352, -1.064 ], ... ],
[ [ -0.376, 0.56, -1.468 ], [ 0.648, 0.936, -0.432 ], [ -0.868, 0.032, 0.164 ], [ 0.06, -1.32, -0.288 ], [ 0.744, 0.44, 0.912 ], [ -0.368, 1.36, 1.672 ], [ -0.936, -1.68, 0.196 ], [ 0.636, -1.54, -0.36 ], ... ],
[ [ -1.676, 0.236, 0.236 ], [ -1.06, 0.256, 0.952 ], [ -0.588, -1.6, -1.368 ], [ -0.804, -0.288, 0.944 ], [ -0.168, 1.608, 0.948 ], [ 0.252, -1.428, 1.076 ], [ 1.268, -1.568, -0.176 ], [ -1.544, -0.052, -0.172 ], ... ],
[ [ -1.2, 0.532, 0.16 ], [ 0.888, -0.808, -1.568 ], [ 1.956, -0.044, -0.14 ], [ -1.804, 0.868, -1.392 ], [ 0.3, -0.352, 0.704 ], [ -1.316, 1.524, 1.876 ], [ -0.6, 0.78, -0.516 ], [ -0.64, 1.116, 1.356 ], ... ],
...
]
[
[ [ -0.58, -0.516, -0.324 ], [ -0.9, 0.928, -1.272 ], [ -1.056, -0.588, -0.46 ], [ -0.092, 1.072, 1.284 ], [ -0.272, -1.444, -0.824 ], [ 1.672, 0.036, -0.104 ], [ -0.9, 0.516, -1.912 ], [ -1.164, -1.864, -1.576 ], ... ],
[ [ -1.96, 0.452, -0.54 ], [ 1.444, 1.968, 0.904 ], [ -1.496, 0.944, 0.772 ], [ 0.196, 1.276, 0.688 ], [ 1.208, -1.6, -1.068 ], [ 0.528, -1.868, -1.664 ], [ -0.452, 0.392, -1.728 ], [ -1.944, 0.3, -1.272 ], ... ],
[ [ 0.488, 0.072, 0.016 ], [ -1.448, 1.096, -0.564 ], [ -1.048, 1.028, 0.664 ], [ -0.944, -0.728, 0.444 ], [ 0.056, -0.668, 1.824 ], [ -0.064, 1.84, 0.948 ], [ 0.392, -1.776, -1.696 ], [ -0.452, 1.4, -1.816 ], ... ],
[ [ -1.248, -1.432, 0.02 ], [ 0.232, -0.884, 1.272 ], [ 0.064, -1.792, -0.956 ], [ 0.608, -1.172, -1.332 ], [ 1.144, 1.108, 1.252 ], [ 1.036, 1.852, -1.204 ], [ 1.952, -0.204, -1.2 ], [ -0.668, -1.42, -0.572 ], ... ],
[ [ -0.568, -1.032, -0.108 ], [ -1.548, -1.784, -0.772 ], [ -0.236, -1.168, 1.096 ], [ 0.344, -0.968, -1.16 ], [ 0.724, 0.208, 0.968 ], [ 1.876, -0.564, -1.792 ], [ -0.32, -0.584, 0.292 ], [ -0.848, -0.476, -1.804 ], ... ],
[ [ -1.352, 0.86, 0.288 ], [ -0.332, 1.936, 1.64 ], [ 1.572, 0.692, -1.54 ], [ 1.508, -0.504, 1.268 ], [ -0.16, -0.208, 1.092 ], [ 0.368, 0.924, -0.472 ], [ -1.724, -0.176, 0.016 ], [ -1.844, 1.736, -1.736 ], ... ],
[ [ -0.98, 1.132, -1.384 ], [ 1.444, -0.18, -0.42 ], [ 1.568, -0.888, -0.196 ], [ 1.14, -1.372, 0.624 ], [ 0.632, -1.46, -0.5 ], [ 1.084, 1.264, 0.616 ], [ 1.74, 1.944, -1.712 ], [ 1.36, -0.664, -1.336 ], ... ],
[ [ 1.44, 1.008, -0.38 ], [ 0.284, 0.616, 0.02 ], [ -1.476, 1.448, -1.444 ], [ -0.34, 0.936, 0.26 ], [ -1.612, -1.352, 1.216 ], [ 1.26, 1.716, -0.404 ], [ 0.24, 0.04, 1.68 ], [ 0.096, -0.884, 1.18 ], ... ],
...
]
Which produces the following output:
TrainingTester.java:308 executed in 0.01 seconds (0.000 gc):
return RefStream.of(RefUtil.addRef(output_target)).map(x -> {
String temp_18_0008 = x.prettyPrint();
x.freeRef();
return temp_18_0008;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -1.487536, -0.464448 ], [ -2.3832479999999996, 1.141248 ], [ -4.056, 0.927232 ], [ 1.615072, 0.9991039999999999 ], [ 2.3965760000000005, 0.7235199999999999 ], [ -0.992224, 0.305536 ], [ -1.56504, 0.642048 ], [ -2.42016, 0.6659839999999999 ], ... ],
[ [ -3.3981919999999994, 1.140096 ], [ 2.053952, 0.938752 ], [ -1.507248, 0.16992 ], [ -3.3949599999999998, 1.117888 ], [ 0.8520159999999999, 0.10112 ], [ -1.050496, -0.342016 ], [ -0.23707200000000023, -0.654144 ], [ 1.241312, -0.899136 ], ... ],
[ [ 1.6379039999999998, 0.35846400000000006 ], [ 1.7154240000000003, 0.8640000000000001 ], [ -3.04368, 0.669056 ], [ 0.573984, -0.08832000000000001 ], [ -2.386752, -0.8541439999999999 ], [ 2.6738399999999998, 0.192192 ], [ -0.26152000000000003, -0.31724800000000003 ], [ 1.560592, -5.760000000000158E-4 ], ... ],
[ [ -0.6887680000000002, -0.972736 ], [ 3.770416, -1.013568 ], [ 2.831344, 0.10848 ], [ 2.911264, -1.134144 ], [ -3.5716319999999997, 0.9062399999999999 ], [ 3.205968, -0.039552000000000004 ], [ 0.4359039999999999, 0.48736 ], [ -3.36944, 0.358272 ], ... ],
[ [ 4.35448, -0.8172159999999999 ], [ 0.8436320000000002, 0.30367999999999995 ], [ 1.878096, -0.408704 ], [ 0.33992, -0.253056 ], [ 1.2452639999999997, -1.0144 ], [ -1.0816960000000002, -0.787712 ], [ -0.147664, -1.046464 ], [ -2.76784, 0.816 ], ... ],
[ [ -2.924432, 0.273408 ], [ -0.428256, -0.319104 ], [ -0.3223999999999999, 0.530304 ], [ -0.29707199999999995, -0.14208 ], [ 2.073248, -0.41715199999999997 ], [ 2.517728, 0.332544 ], [ -0.09441599999999999, 0.434688 ], [ 0.007280000000000021, -0.509888 ], ... ],
[ [ -0.7871039999999998, 1.037888 ], [ 0.9045599999999999, 0.6649600000000001 ], [ -2.619952, 0.22950399999999999 ], [ 1.13928, 0.46579200000000004 ], [ 1.348848, 0.23078400000000002 ], [ 2.257248, -0.26745599999999997 ], [ 0.77784, -0.896384 ], [ -1.3775519999999999, 0.934592 ], ... ],
[ [ -0.625856, 0.77216 ], [ -2.040928, -0.604544 ], [ 1.127872, -1.1927679999999998 ], [ -3.829392, 1.166272 ], [ 1.496912, -0.21055999999999997 ], [ 2.192992, 0.9220480000000001 ], [ -1.430064, 0.42719999999999997 ], [ 1.8011360000000003, 0.47840000000000005 ], ... ],
...
]
[
[ [ -0.911488, 0.31135999999999997 ], [ -2.992592, 0.62144 ], [ -1.475376, 0.5950080000000001 ], [ 2.06336, 0.14169600000000002 ], [ -1.459104, 0.049856 ], [ 0.982336, -1.013696 ], [ -4.068816, 0.58848 ], [ -3.356272, 0.5585919999999999 ], ... ],
[ [ -2.3824159999999996, 1.2278399999999998 ], [ 2.353552, -0.7205119999999999 ], [ 0.1937760000000001, 0.985088 ], [ 1.187504, -0.017087999999999992 ], [ -0.8335520000000001, -0.862464 ], [ -2.3265919999999998, -0.47046400000000005 ], [ -3.414768, 0.306176 ], [ -3.643008, 1.205952 ], ... ],
[ [ 0.360608, -0.290944 ], [ -2.1487839999999996, 0.9680639999999999 ], [ 0.306112, 0.719424 ], [ 0.21560000000000012, 0.515712 ], [ 3.3422400000000003, -0.08748800000000001 ], [ 1.3919519999999999, 0.186112 ], [ -2.490016, -0.380416 ], [ -3.699024, 0.386816 ], ... ],
[ [ -0.6550239999999999, 0.644224 ], [ 2.5193600000000003, -0.21177600000000002 ], [ -1.412208, -0.18227200000000002 ], [ -1.774032, -0.463424 ], [ 2.867504, -0.6069119999999999 ], [ -1.635712, -0.481728 ], [ -0.7242880000000002, -1.2031359999999998 ], [ -1.294848, 0.292544 ], ... ],
[ [ -0.456016, 0.26278399999999996 ], [ -2.217056, 0.7984640000000001 ], [ 1.9176480000000002, 0.05004799999999998 ], [ -1.681536, -0.28659199999999996 ], [ 2.187728, -0.423552 ], [ -1.7756960000000002, -1.185728 ], [ 0.36584, 0.14784 ], [ -3.714928, 0.47750399999999993 ], ... ],
[ [ -0.5484480000000002, 0.890816 ], [ 2.412752, 0.356736 ], [ -1.704736, -0.9004160000000001 ], [ 3.356864, -0.957184 ], [ 1.840912, 0.08064 ], [ -0.69328, -0.14982399999999998 ], [ -1.1560479999999997, 1.034112 ], [ -4.575312, 1.2600319999999998 ], ... ],
[ [ -3.2722719999999996, 0.6863999999999999 ], [ 0.2929599999999999, -0.8923519999999999 ], [ 0.8655199999999998, -1.024384 ], [ 2.0743519999999998, -0.8028799999999999 ], [ -0.25272000000000006, -0.501056 ], [ 1.683632, -0.557952 ], [ -2.0508, -0.9024 ], [ -1.3197120000000002, -0.8800000000000001 ], ... ],
[ [ 0.20865599999999998, -0.7948799999999999 ], [ 0.15523199999999998, -0.12339199999999999 ], [ -3.7657599999999998, 1.013248 ], [ 0.100832, 0.2816 ], [ 1.1896799999999998, 0.871936 ], [ -0.05030400000000011, -0.6288 ], [ 3.1264, -0.14271999999999999 ], [ 2.261872, -0.129088 ], ... ],
...
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 30.74 seconds (0.355 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11575586534018
Reset training subject: 11575910352408
Constructing line search parameters: GD
th(0)=3.570995254823763;dx=-2.418916625194388
Armijo: th(2.154434690031884)=5.8440746635911545; dx=4.529042236211652 evalInputDelta=-2.2730794087673916
New Minimum: 3.570995254823763 > 2.836419509725103
WOLF (strong): th(1.077217345015942)=2.836419509725103; dx=1.0550735974931589 evalInputDelta=0.7345757450986601
END: th(0.3590724483386473)=2.9103318046799687; dx=-1.2609174860679235 evalInputDelta=0.6606634501437942
Fitness changed from 3.570995254823763 to 2.836419509725103
Iteration 1 complete. Error: 2.836419509725103 Total: 1.5166; Orientation: 0.0746; Line Search: 0.8261
th(0)=2.836419509725103;dx=-0.46021533358154804
New Minimum: 2.836419509725103 > 2.6639989716087307
WOLF (strong): th(0.7735981389354633)=2.6639989716087307; dx=0.014452720652775251 evalInputDelta=0.17242053811637215
END: th(0.3867990694677316)=2.704308950245947; dx=-0.222881256500496 evalInputDelta=0.13211055947915584
Fitness changed from 2.836419509725103 to 2.6639989716087307
Iteration 2 complete. Error: 2.6639989716087307 Total: 0.8004; Orientation: 0.0772; Line Search: 0.5835
th(0)=2.6639989716087307;dx=-4.5554386841891744E-4
New Minimum: 2.6639989716087307 > 2.6638302877608986
WOLF (strong): th(0.8333333333333335)=2.6638302877608986; dx=5.070263352827596E-5 evalInputDelta=1.6868384783208512E-4
END: th(0.41666666666666674)=2.663861895674195; dx=-2.0242061737628974E-4 evalInputDelta=1.3707593453560207E-4
Fitness changed from 2.6639989716087307 to 2.6638302877608986
Iteration 3 complete. Error: 2.6638302877608986 Total: 0.8668; Orientation: 0.0760; Line Search: 0.6528
th(0)=2.6638302877608986;dx=-5.645536518584081E-6
New Minimum: 2.6638302877608986 > 2.6638282534635165
WOLF (strong): th(0.8976811208466184)=2.6638282534635165; dx=1.1131979530570547E-6 evalInputDelta=2.034297382103034E-6
END: th(0.4488405604233092)=2.6638285122136653; dx=-2.2661692827712686E-6 evalInputDelta=1.77554723324036E-6
Fitness changed from 2.6638302877608986 to 2.6638282534635165
Iteration 4 complete. Error: 2.6638282534635165 Total: 0.7979; Orientation: 0.0762; Line Search: 0.5804
th(0)=2.6638282534635165;dx=-2.1968768238313168E-7
New Minimum: 2.6638282534635165 > 2.6638281779507134
WOLF (strong): th(0.9669976736693292)=2.6638281779507134; dx=6.350777625113399E-8 evalInputDelta=7.551280312156905E-8
END: th(0.4834988368346646)=2.663828181475946; dx=-7.808995306580874E-8 evalInputDelta=7.198757057125249E-8
Fitness changed from 2.6638282534635165 to 2.6638281779507134
Iteration 5 complete. Error: 2.6638281779507134 Total: 0.8002; Orientation: 0.0760; Line Search: 0.5853
th(0)=2.6638281779507134;dx=-1.853851687157641E-8
New Minimum: 2.6638281779507134 > 2.663828171977059
WOLF (strong): th(1.041666666666667)=2.663828171977059; dx=7.069099144525347E-9 evalInputDelta=5.973654459978661E-9
New Minimum: 2.663828171977059 > 2.663828171629561
END: th(0.5208333333333335)=2.663828171629561; dx=-5.7347088635307926E-9 evalInputDelta=6.321152490329496E-9
Fitness changed from 2.6638281779507134 to 2.663828171629561
Iteration 6 complete. Error: 2.663828171629561 Total: 0.9321; Orientation: 0.0753; Line Search: 0.7172
th(0)=2.663828171629561;dx=-1.8254110561958456E-9
New Minimum: 2.663828171629561 > 2.6638281710240532
WOLF (strong): th(1.122101401058273)=2.6638281710240532; dx=7.461726860981008E-10 evalInputDelta=6.055076440247831E-10
New Minimum: 2.6638281710240532 > 2.66382817096611
END: th(0.5610507005291365)=2.66382817096611; dx=-5.396191850485876E-10 evalInputDelta=6.634510718583897E-10
Fitness changed from 2.663828171629561 to 2.66382817096611
Iteration 7 complete. Error: 2.66382817096611 Total: 0.8124; Orientation: 0.0762; Line Search: 0.5972
th(0)=2.66382817096611;dx=-2.1599122923923087E-10
New Minimum: 2.66382817096611 > 2.6638281708110965
END: th(1.2087470920866614)=2.6638281708110965; dx=-4.049445611463698E-11 evalInputDelta=1.55013335501053E-10
Fitness changed from 2.66382817096611 to 2.6638281708110965
Iteration 8 complete. Error: 2.6638281708110965 Total: 0.6061; Orientation: 0.0759; Line Search: 0.3920
th(0)=2.6638281708110965;dx=-1.4792318264026864E-10
New Minimum: 2.6638281708110965 > 2.6638281706103575
END: th(2.6041666666666674)=2.6638281706103575; dx=-6.244116891026426E-12 evalInputDelta=2.007389809932647E-10
Fitness changed from 2.6638281708110965 to 2.6638281706103575
Iteration 9 complete. Error: 2.6638281706103575 Total: 0.6041; Orientation: 0.0765; Line Search: 0.3887
th(0)=2.6638281706103575;dx=-3.56665516885406E-10
Armijo: th(5.610507005291366)=2.663828173847409; dx=1.5105898961199524E-9 evalInputDelta=-3.2370515157253976E-9
Armijo: th(2.805253502645683)=2.663828170919352; dx=5.769621896165488E-10 evalInputDelta=-3.0899460767841447E-10
New Minimum: 2.6638281706103575 > 2.663828170422349
END: th(0.9350845008818943)=2.663828170422349; dx=-4.54562813851611E-11 evalInputDelta=1.8800827561449296E-10
Fitness changed from 2.6638281706103575 to 2.663828170422349
Iteration 10 complete. Error: 2.663828170422349 Total: 0.9867; Orientation: 0.0769; Line Search: 0.7714
th(0)=2.663828170422349;dx=-1.2238300779771115E-10
New Minimum: 2.663828170422349 > 2.663828170217067
END: th(2.0145784868111027)=2.663828170217067; dx=-8.141342670998712E-11 evalInputDelta=2.0528201361003084E-10
Fitness changed from 2.663828170422349 to 2.663828170217067
Iteration 11 complete. Error: 2.663828170217067 Total: 0.6055; Orientation: 0.0757; Line Search: 0.3901
th(0)=2.663828170217067;dx=-1.5053560013065884E-10
New Minimum: 2.663828170217067 > 2.6638281701089177
WOLF (strong): th(4.3402777777777795)=2.6638281701089177; dx=1.0070019659589929E-10 evalInputDelta=1.081494893639956E-10
New Minimum: 2.6638281701089177 > 2.6638281700266884
END: th(2.1701388888888897)=2.6638281700266884; dx=-2.4917701767347938E-11 evalInputDelta=1.903788238166726E-10
Fitness changed from 2.663828170217067 to 2.6638281700266884
Iteration 12 complete. Error: 2.6638281700266884 Total: 0.8109; Orientation: 0.0761; Line Search: 0.5950
th(0)=2.6638281700266884;dx=-2.62918489019405E-10
Armijo: th(4.675422504409472)=2.6638281710684693; dx=7.085599706353389E-10 evalInputDelta=-1.0417808837814846E-9
New Minimum: 2.6638281700266884 > 2.6638281699798196
WOLF (strong): th(2.337711252204736)=2.6638281699798196; dx=2.2282074080867245E-10 evalInputDelta=4.686873111836576E-11
New Minimum: 2.6638281699798196 > 2.6638281698848965
END: th(0.7792370840682453)=2.6638281698848965; dx=-1.0100541240989193E-10 evalInputDelta=1.4179191154539694E-10
Fitness changed from 2.6638281700266884 to 2.6638281698848965
Iteration 13 complete. Error: 2.6638281698848965 Total: 1.0070; Orientation: 0.0804; Line Search: 0.7873
th(0)=2.6638281698848965;dx=-1.0738342849115292E-10
New Minimum: 2.6638281698848965 > 2.6638281697050745
WOLFE (weak): th(1.678815405675919)=2.6638281697050745; dx=-1.0684159797222579E-10 evalInputDelta=1.798219351201169E-10
New Minimum: 2.6638281697050745 > 2.6638281695261616
WOLFE (weak): th(3.357630811351838)=2.6638281695261616; dx=-1.0629976745327326E-10 evalInputDelta=3.5873481962767073E-10
New Minimum: 2.6638281695261616 > 2.6638281688196086
WOLFE (weak): th(10.072892434055515)=2.6638281688196086; dx=-1.0413244537753119E-10 evalInputDelta=1.065287857926478E-9
New Minimum: 2.6638281688196086 > 2.663828165820224
END: th(40.29156973622206)=2.663828165820224; dx=-9.437949603595379E-11 evalInputDelta=4.064672598502739E-9
Fitness changed from 2.6638281698848965 to 2.663828165820224
Iteration 14 complete. Error: 2.663828165820224 Total: 1.2195; Orientation: 0.0750; Line Search: 1.0049
th(0)=2.663828165820224;dx=-7.815800080444035E-10
Armijo: th(86.8055555555556)=2.663831487853492; dx=7.732122650098475E-8 evalInputDelta=-3.322033268027269E-6
Armijo: th(43.4027777777778)=2.6638289793671692; dx=3.8269823246729975E-8 evalInputDelta=-8.135469453662836E-7
Armijo: th(14.4675925925926)=2.6638282486759417; dx=1.2235554410274169E-8 evalInputDelta=-8.285571784227841E-8
Armijo: th(3.61689814814815)=2.663828168878535; dx=2.4727035965413972E-9 evalInputDelta=-3.0583109378312656E-9
New Minimum: 2.663828165820224 > 2.6638281654902536
END: th(0.72337962962963)=2
...skipping 2643 bytes...
2.663828160988298 Total: 0.6032; Orientation: 0.0756; Line Search: 0.3900
th(0)=2.663828160988298;dx=-2.91869509083826E-10
Armijo: th(6.715261622703678)=2.663828164580999; dx=1.3618802794462452E-9 evalInputDelta=-3.59270124761224E-9
Armijo: th(3.357630811351839)=2.6638281613964785; dx=5.350053851808757E-10 evalInputDelta=-4.081806004307964E-10
New Minimum: 2.663828160988298 > 2.663828160815876
END: th(1.119210270450613)=2.663828160815876; dx=-1.6244544328782257E-11 evalInputDelta=1.724220766163853E-10
Fitness changed from 2.663828160988298 to 2.663828160815876
Iteration 20 complete. Error: 2.663828160815876 Total: 0.9820; Orientation: 0.0773; Line Search: 0.7653
th(0)=2.663828160815876;dx=-1.5207389451689602E-10
New Minimum: 2.663828160815876 > 2.663828160623055
END: th(2.411265432098767)=2.663828160623055; dx=-7.859286361789872E-12 evalInputDelta=1.928208703816381E-10
Fitness changed from 2.663828160815876 to 2.663828160623055
Iteration 21 complete. Error: 2.663828160623055 Total: 0.6045; Orientation: 0.0749; Line Search: 0.3913
th(0)=2.663828160623055;dx=-3.274751486216402E-10
Armijo: th(5.1949138937883035)=2.663828162885766; dx=1.1986006209127672E-9 evalInputDelta=-2.2627109075301632E-9
Armijo: th(2.5974569468941517)=2.6638281607634315; dx=4.355627361449453E-10 evalInputDelta=-1.403765992336048E-10
New Minimum: 2.663828160623055 > 2.66382816044963
END: th(0.8658189822980505)=2.66382816044963; dx=-7.312918703248735E-11 evalInputDelta=1.7342483005222675E-10
Fitness changed from 2.663828160623055 to 2.66382816044963
Iteration 22 complete. Error: 2.66382816044963 Total: 0.9848; Orientation: 0.0747; Line Search: 0.7695
th(0)=2.66382816044963;dx=-1.1252040068691468E-10
New Minimum: 2.66382816044963 > 2.663828160252
END: th(1.8653504507510217)=2.663828160252; dx=-9.93753984561863E-11 evalInputDelta=1.9762991243510442E-10
Fitness changed from 2.66382816044963 to 2.663828160252
Iteration 23 complete. Error: 2.663828160252 Total: 0.6045; Orientation: 0.0754; Line Search: 0.3902
th(0)=2.663828160252;dx=-1.1894090895468175E-10
New Minimum: 2.663828160252 > 2.6638281599000493
END: th(4.018775720164612)=2.6638281599000493; dx=-5.621223908669548E-11 evalInputDelta=3.519509128580012E-10
Fitness changed from 2.663828160252 to 2.6638281599000493
Iteration 24 complete. Error: 2.6638281599000493 Total: 0.6083; Orientation: 0.0758; Line Search: 0.3925
th(0)=2.6638281599000493;dx=-3.297733497295368E-10
Armijo: th(8.658189822980507)=2.663828168171163; dx=2.2403605846523177E-9 evalInputDelta=-8.271113571822752E-9
Armijo: th(4.3290949114902535)=2.663828161254018; dx=9.55293617461725E-10 evalInputDelta=-1.353968492878721E-9
New Minimum: 2.6638281599000493 > 2.663828159733241
WOLF (strong): th(1.4430316371634178)=2.663828159733241; dx=9.858230599999E-11 evalInputDelta=1.6680834491467067E-10
END: th(0.36075790929085444)=2.6638281598003974; dx=-2.2268443579751027E-10 evalInputDelta=9.965184233351465E-11
Fitness changed from 2.6638281599000493 to 2.663828159733241
Iteration 25 complete. Error: 2.663828159733241 Total: 1.1745; Orientation: 0.0746; Line Search: 0.9579
th(0)=2.663828159733241;dx=-2.9762577752722654E-10
New Minimum: 2.663828159733241 > 2.6638281595786255
END: th(0.7772293544795924)=2.6638281595786255; dx=-1.0023817657063172E-10 evalInputDelta=1.5461543156902735E-10
Fitness changed from 2.663828159733241 to 2.6638281595786255
Iteration 26 complete. Error: 2.6638281595786255 Total: 0.6080; Orientation: 0.0753; Line Search: 0.3925
th(0)=2.6638281595786255;dx=-1.07504746756031E-10
New Minimum: 2.6638281595786255 > 2.663828159399093
WOLFE (weak): th(1.6744898834019217)=2.663828159399093; dx=-1.0692748777263577E-10 evalInputDelta=1.7953238895529466E-10
New Minimum: 2.663828159399093 > 2.663828159220527
WOLFE (weak): th(3.3489797668038435)=2.663828159220527; dx=-1.0635022878920681E-10 evalInputDelta=3.580984397899556E-10
New Minimum: 2.663828159220527 > 2.6638281585159307
WOLFE (weak): th(10.04693930041153)=2.6638281585159307; dx=-1.0404119285560666E-10 evalInputDelta=1.0626948210301634E-9
New Minimum: 2.6638281585159307 > 2.6638281555366357
END: th(40.18775720164612)=2.6638281555366357; dx=-9.365053115491402E-11 evalInputDelta=4.041989853931227E-9
Fitness changed from 2.6638281595786255 to 2.6638281555366357
Iteration 27 complete. Error: 2.6638281555366357 Total: 1.2839; Orientation: 0.0755; Line Search: 1.0698
th(0)=2.6638281555366357;dx=-8.238706602986436E-10
Armijo: th(86.58189822980506)=2.663831668013918; dx=8.196039813405435E-8 evalInputDelta=-3.512477282363591E-6
Armijo: th(43.29094911490253)=2.663829015822885; dx=4.056826373655101E-8 evalInputDelta=-8.602862493489738E-7
Armijo: th(14.430316371634177)=2.6638282431981875; dx=1.2973507471916066E-8 evalInputDelta=-8.766155179529278E-8
Armijo: th(3.6075790929085443)=2.663828158786349; dx=2.6254738727479553E-9 evalInputDelta=-3.2497133872766426E-9
New Minimum: 2.6638281555366357 > 2.6638281551910756
END: th(0.7215158185817089)=2.6638281551910756; dx=-1.3400175368995096E-10 evalInputDelta=3.455600250390489E-10
Fitness changed from 2.6638281555366357 to 2.6638281551910756
Iteration 28 complete. Error: 2.6638281551910756 Total: 1.5433; Orientation: 0.0787; Line Search: 1.3260
th(0)=2.6638281551910756;dx=-1.0828921517552442E-10
New Minimum: 2.6638281551910756 > 2.6638281550243534
WOLFE (weak): th(1.5544587089591848)=2.6638281550243534; dx=-1.0621930652865498E-10 evalInputDelta=1.6672219160795976E-10
New Minimum: 2.6638281550243534 > 2.6638281548608482
WOLFE (weak): th(3.1089174179183696)=2.6638281548608482; dx=-1.0414939788170331E-10 evalInputDelta=3.3022740097976566E-10
New Minimum: 2.6638281548608482 > 2.6638281542390057
END: th(9.326752253755108)=2.6638281542390057; dx=-9.586976329406621E-11 evalInputDelta=9.520699784104636E-10
Fitness changed from 2.6638281551910756 to 2.6638281542390057
Iteration 29 complete. Error: 2.6638281542390057 Total: 1.0896; Orientation: 0.0845; Line Search: 0.8596
th(0)=2.6638281542390057;dx=-2.3803688440146433E-10
Armijo: th(20.09387860082306)=2.6638281846650056; dx=3.2664218193709157E-9 evalInputDelta=-3.042599994174111E-8
Armijo: th(10.04693930041153)=2.663828160649735; dx=1.5141924674870375E-9 evalInputDelta=-6.410729280759142E-9
Armijo: th(3.3489797668038435)=2.663828154419855; dx=3.460395662289011E-10 evalInputDelta=-1.8084955755170995E-10
New Minimum: 2.6638281542390057 > 2.6638281541008375
END: th(0.8372449417009609)=2.6638281541008375; dx=-9.201777174409488E-11 evalInputDelta=1.3816814359302043E-10
Fitness changed from 2.6638281542390057 to 2.6638281541008375
Iteration 30 complete. Error: 2.6638281541008375 Total: 1.4073; Orientation: 0.0844; Line Search: 1.1753
th(0)=2.6638281541008375;dx=-1.0908908774065365E-10
New Minimum: 2.6638281541008375 > 2.663828153907942
WOLFE (weak): th(1.8037895464542724)=2.663828153907942; dx=-1.047887740045091E-10 evalInputDelta=1.928954773688929E-10
New Minimum: 2.663828153907942 > 2.6638281537228035
WOLFE (weak): th(3.6075790929085447)=2.6638281537228035; dx=-1.0048846026840992E-10 evalInputDelta=3.7803404850933475E-10
New Minimum: 2.6638281537228035 > 2.6638281530598187
END: th(10.822737278725635)=2.6638281530598187; dx=-8.328720532380222E-11 evalInputDelta=1.0410188266973819E-9
Fitness changed from 2.6638281541008375 to 2.6638281530598187
Iteration 31 complete. Error: 2.6638281530598187 Total: 1.0329; Orientation: 0.0796; Line Search: 0.8098
th(0)=2.6638281530598187;dx=-4.3012391148739436E-10
Armijo: th(23.316880634387775)=2.66382826010451; dx=9.611857401050904E-9 evalInputDelta=-1.0704469133315797E-7
Armijo: th(11.658440317193888)=2.6638281773137047; dx=4.590866744790129E-9 evalInputDelta=-2.4253886010683345E-8
Armijo: th(3.8861467723979626)=2.6638281546403446; dx=1.2435396406064688E-9 evalInputDelta=-1.5805259323542487E-9
New Minimum: 2.6638281530598187 > 2.6638281528451904
END: th(0.9715366930994906)=2.6638281528451904; dx=-1.1708023463631147E-11 evalInputDelta=2.1462831512053526E-10
Fitness changed from 2.6638281530598187 to 2.6638281528451904
Iteration 32 complete. Error: 2.6638281528451904 Total: 1.1768; Orientation: 0.0768; Line Search: 0.9604
Final threshold in iteration 32: 2.6638281528451904 (> 0.0) after 30.744s (< 30.000s)
Returns
2.6638281528451904
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.001702845655286387, 0.0032537769169999333, -0.0011716579616425657, -1.8282238340417006E-4, -0.0012331651374755553, -3.082139839222966E-4, -0.608, 1.764, 0.08]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.8439970339916394, 1.2119972344151486, 0.6600008960522158 ], [ 0.12799745858220385, 1.5879941057655875, -0.5399988846866731 ], [ 1.4439941776373295, -0.34000579543594056, 0.9400017793008507 ], [ -0.3159973860660121, 0.40400072661684255, -0.4200007418022267 ], [ 0.17600286185005812, -1.443996132795783, -0.6880011653140935 ], [ 1.0999976375617688, -1.3320003407152123, -0.6839995674079158 ], [ 0.2799980837441029, 0.6319966561170087, -0.24399927738056607 ], [ 1.3199957637134931, 0.23999578936150737, -0.8119987916509928 ], ... ],
[ [ 0.35599647423385933, 0.04799462267816111, 1.7000013451845046 ], [ 1.6080006424770072, 1.5920000014505735, -0.0400005121921689 ], [ 1.9039959795453245, 0.6159971219265565, 0.3160009619170042 ], [ -1.7560012816231478, 0.39199419057240464, -0.719998869193907 ], [ 1.7039992252206084, -1.3199975678414455, 1.5639997080430932 ], [ -0.6480007885630246, 0.8159980217397124, -1.171999512919993 ], [ 1.1599982041463215, -0.13999948204341175, 1.4880002668844738 ], [ -1.0559976731799827, 0.1200024273101792, 0.16799939799054728 ], ... ],
[ [ -0.567997199145279, -0.5119976130616205, 0.09199914555278514 ], [ 1.8880001300224556, 0.9560004245500734, 1.34799956987834 ], [ 0.15199684723021278, 0.6919948549846072, 1.6120012736976042 ], [ 0.9679991954975804, 0.24800036094458122, -1.232000025912353 ], [ -0.6920022864505948, -1.3040009832352109, -0.8599991608029514 ], [ 1.0160019382868657, 1.6080016779361122, 1.303999160059933 ], [ -0.4639999763900069, -0.5479995433333269, -1.0039999684171248 ], [ -1.1039969418107967, 1.376000680264371, 0.035999329311229865 ], ... ],
[ [ -1.8439988577140738, 0.5279996875388003, 0.8000001169732798 ], [ -0.007996533672652115, 1.2920043113195538, -1.7240013121507136 ], [ 1.2560017630906575, 0.4240031050355764, 0.5959990285652638 ], [ 0.2520025913819125, 0.3800045376093452, 1.0279989200985946 ], [ -0.22000332294434466, 1.2519933165186896, 0.9320014995079955 ], [ 0.06400391059499959, -1.5279940285941287, 1.5679984654498156 ], [ -1.9359971798991, 1.4479985925419412, -1.4400003146334446 ], [ 0.7999957025613751, -1.1280033939373253, 1.8960013264496165 ], ... ],
[ [ -1.339994086236883, 1.236005063087884, -1.6080017811849876 ], [ 0.9320002791093934, -1.919997111689812, 1.2639995232685866 ], [ 1.8239994809014082, -0.539996913908706, -1.4920004950402785 ], [ 0.7359994898605099, 1.9679987727560504, 1.964000146315234 ], [ -0.14799916006152047, 0.6040017745550961, -1.5720003573655539 ], [ -1.3639999818112598, -1.5039991362109701, -1.3279998208314194 ], [ -0.3040005879337947, 1.2319993674190977, -1.3839997342752284 ], [ -0.08400306509916619, 1.2839940854902216, -1.9999987240601214 ], ... ],
[ [ 1.9439944754865972, -1.9560021920539012, -0.26799870966720385 ], [ 0.5959983969741844, 1.6599978926137084, -0.5879995116249584 ], [ 1.7079977688948191, -1.799999246130164, -0.7759998201445085 ], [ 0.9199985233122594, 0.82799891122554, 0.9600003364582084 ], [ -1.4759961116308247, -1.491995409453803, -1.592001174302807 ], [ -0.3279966813502373, 0.1640027438249256, -0.7000010886589926 ], [ 0.09599995470620662, 0.7559986988523356, 0.2360000964120219 ], [ -0.17599994754004905, 0.03200051310964337, 0.44799999968200743 ], ... ],
[ [ -1.983997746670535, 1.607996695976849, 1.996000069490808 ], [ 0.36400086407660204, -1.6999978047916304, -1.0480005271982011 ], [ -1.8520004443490274, 1.1759954388074865, 1.3680008861589479 ], [ -0.9399973142058251, -1.515997382903812, -0.8440007929681881 ], [ 1.223999695527751, 1.5639995947153693, -1.9680001722002745 ], [ -1.991995032803378, -1.231995417960105, -0.4480013439278687 ], [ 0.12400011633575493, 0.4640012410758761, -1.0240001602856228 ], [ 1.9959965297306317, 0.04399724931998119, 1.0520007820803816 ], ... ],
[ [ 0.615998548625362, 1.7959964006385876, -1.275999474227187 ], [ 1.8279948844519218, -0.44800194456939063, -1.6599988287345873 ], [ -1.9879968646763562, -1.0319962965668696, -1.0200007648432663 ], [ 0.3759960013925008, -1.1440047605503314, 1.2920014088539034 ], [ -0.8999972266034183, 0.9120013389061488, 0.7399993328313783 ], [ -1.7079949105200154, 1.8840002714435684, 1.0799989307179008 ], [ 0.23999810650462483, 0.3879972199779472, -0.8959993319346643 ], [ 1.4120005017452422, -0.9639972643580113, -0.5640006632252066 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004607470002664222, -0.0018523157911024904 ], [ 0.00601763247442767, -0.001815224562631591 ], [ 2.5123563413688156E-4, -1.3443285699851656E-4 ], [ 0.0012685306758010817, -3.109781144274227E-4 ], [ -0.003592633533549127, 0.001960560007028539 ], [ -0.0016594922251943014, 0.0016522904251094835 ], [ 0.0028190533476648003, -7.553421709876938E-4 ], [ 0.003980026659919637, -2.8700982967220276E-4 ], ... ],
[ [ -0.0012294492660317748, -6.482336066484969E-4 ], [ 0.007965056682903161, -0.0022448486934012832 ], [ 0.00487628345479776, -0.00120511517391326 ], [ -8.711450967420559E-4, 5.955648955408532E-5 ], [ -0.0032258026493956044, 8.341992016716335E-4 ], [ 0.0029248127603815057, -5.265646250134975E-4 ], [ -2.2365654061568135E-4, -4.980536462399214E-4 ], [ -0.0016045777540080268, -6.7025620425492265E-6 ], ... ],
[ [ -0.0027409291090695737, 7.068647854446512E-4 ], [ 0.004746190504293326, -0.0019395473963628143 ], [ 6.217099302490731E-4, -0.001377973691014172 ], [ 0.003898773713326958, -1.0307768307357969E-4 ], [ -0.004413676522165884, 0.0019996258266104503 ], [ 0.005434332230680581, -0.002570590282287271 ], [ -0.0013968440519883843, 0.0010700503438979853 ], [ 0.0025550839546071478, -0.0015060962125774506 ], ... ],
[ [ -0.0023593787540812723, -5.605577642996189E-4 ], [ 0.006210216805477746, -0.0010604314161371433 ], [ 0.0028200816542622983, -9.361863182072513E-4 ], [ 4.6110839139064277E-4, -8.31523704874238E-4 ], [ 0.0026070882737410056, -0.0017909488736106377 ], [ -0.006699920804362881, 0.0013892885650306059 ], [ 0.003101947843231555, -9.878495309209537E-4 ], [ -0.004529467248555992, 6.603832169552554E-4 ], ... ],
[ [ 0.003623909724973292, -7.836088058167588E-4 ], [ -0.00614116476166577, 0.0018077006610870106 ], [ 0.003097074356358832, 7.922928527406436E-4 ], [ 0.005355586104982392, -0.0031667569675977825 ], [ 0.0035551140395898846, -2.332638792995406E-4 ], [ -0.005660397552292074, 0.0025133571446332445 ], [ 0.005112559330673977, -0.0010371124854024753 ], [ 0.006378100490817682, -9.51591527794584E-4 ], ... ],
[ [ -0.0027400694137316496, 0.002139268958726201 ], [ 0.007105090415337483, -0.001974783704871708 ], [ -0.002039133050168711, 0.0021466100909413683 ], [ 0.00313594719554811, -0.0014851412422178468 ], [ -0.005502732938644601, 0.002600398875576474 ], [ 7.952624670823634E-4, 7.347279318340867E-5 ], [ 0.0023468128294773087, -0.0010225607098488577 ], [ -7.204809815541841E-4, -1.4536505195904013E-4 ], ... ],
[ [ -4.850087838922909E-4, -0.0022354014032666624 ], [ -0.003683678164770855, 0.002352838938762776 ], [ -9.300732267614086E-4, -0.0015332464447984702 ], [ -0.005544507384487179, 0.002301460517312092 ], [ 0.009479011413355803, -0.0015458791434176406 ], [ -0.006875793998329914, 0.0020215153575989598 ], [ 0.0029206875275512674, -2.7924898211235597E-4 ], [ 0.0023094461608886728, -7.434100689723936E-4 ], ... ],
[ [ 0.00838775702663223, -0.0019340975896979672 ], [ 0.0036000456048679423, 7.298968502487312E-4 ], [ -0.005548045534925671, 0.0019504506792664292 ], [ -0.004595856862577518, 9.43793401235643E-4 ], [ 5.678664277923602E-4, -0.0011881867609171122 ], [ 0.001956275536473336, -0.0023438945264236148 ], [ 0.0027209408820504195, -2.461821472706028E-4 ], [ -7.1397259720595E-5, 0.0011044554132786154 ], ... ],
...
]
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 30.46 seconds (0.188 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11606372555460
Reset training subject: 11606511992598
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=3.570995254823763}, derivative=-2.418916625194388}
New Minimum: 3.570995254823763 > 3.570995254581871
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=3.570995254581871}, derivative=-2.418916624871887}, evalInputDelta = -2.418918398916503E-10
New Minimum: 3.570995254581871 > 3.5709952531305214
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=3.5709952531305214}, derivative=-2.4189166229369095}, evalInputDelta = -1.6932415469739226E-9
New Minimum: 3.5709952531305214 > 3.5709952429710716
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=3.5709952429710716}, derivative=-2.4189166093920003}, evalInputDelta = -1.1852691272906668E-8
New Minimum: 3.5709952429710716 > 3.570995171854925
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=3.570995171854925}, derivative=-2.418916514577657}, evalInputDelta = -8.296883802216826E-8
New Minimum: 3.570995171854925 > 3.570994674041974
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=3.570994674041974}, derivative=-2.418915850877433}, evalInputDelta = -5.807817888836553E-7
New Minimum: 3.570994674041974 > 3.570991189355146
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=3.570991189355146}, derivative=-2.4189112049757875}, evalInputDelta = -4.0654686168650755E-6
New Minimum: 3.570991189355146 > 3.5709667967347487
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=3.5709667967347487}, derivative=-2.418878683664098}, evalInputDelta = -2.8458089014193888E-5
New Minimum: 3.5709667967347487 > 3.570796057574605
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=3.570796057574605}, derivative=-2.4186510344823295}, evalInputDelta = -1.991972491577343E-4
New Minimum: 3.570796057574605 > 3.56960133340294
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=3.56960133340294}, derivative=-2.4170574902126885}, evalInputDelta = -0.0013939214208229522
New Minimum: 3.56960133340294 > 3.5612603117185615
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=3.5612603117185615}, derivative=-2.4059026804520727}, evalInputDelta = -0.009734943105201399
New Minimum: 3.5612603117185615 > 3.5039534882162036
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=3.5039534882162036}, derivative=-2.327819018359098}, evalInputDelta = -0.06704176660755934
New Minimum: 3.5039534882162036 > 3.155741789760734
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=3.155741789760734}, derivative=-1.7812336890269704}, evalInputDelta = -0.41525346506302885
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=3.3121202552319557}, derivative=2.0448486557361796}, evalInputDelta = -0.25887499959180715
3.3121202552319557 <= 3.570995254823763
New Minimum: 3.155741789760734 > 2.6638312154083823
F(0.7500600416339407) = LineSearchPoint{point=PointSample{avg=2.6638312154083823}, derivative=4.423115474252951E-6}, evalInputDelta = -0.9071640394153806
Right bracket at 0.7500600416339407
Converged to right
Fitness changed from 3.570995254823763 to 2.6638312154083823
Iteration 1 complete. Error: 2.6638312154083823 Total: 3.7165; Orientation: 0.0769; Line Search: 3.2232
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638312154083823}, derivative=-8.132501825355231E-6}
New Minimum: 2.6638312154083823 > 2.6638281713930683
F(0.7500600416339407) = LineSearchPoint{point=PointSample{avg=2.6638281713930683}, derivative=1.5777444887567946E-8}, evalInputDelta = -3.04401531403542E-6
2.6638281713930683 <= 2.6638312154083823
Converged to right
Fitness changed from 2.6638312154083823 to 2.6638281713930683
Iteration 2 complete. Error: 2.6638281713930683 Total: 0.6050; Orientation: 0.0765; Line Search: 0.3897
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638281713930683}, derivative=-1.747260397959256E-10}
New Minimum: 2.6638281713930683 > 2.6638281712874394
F(0.7500600416339407) = LineSearchPoint{point=PointSample{avg=2.6638281712874394}, derivative=-1.0692977114628293E-10}, evalInputDelta = -1.0562883900888664E-10
F(5.250420291437585) = LineSearchPoint{point=PointSample{avg=2.6638281717215397}, derivative=2.9984784075596817E-10}, evalInputDelta = 3.284714722440185E-10
F(0.4038784839567373) = LineSearchPoint{point=PointSample{avg=2.6638281713298726}, derivative=-1.3822035667674126E-10}, evalInputDelta = -6.319567091850331E-11
New Minimum: 2.6638281712874394 > 2.6638281712603167
F(2.827149387697161) = LineSearchPoint{point=PointSample{avg=2.6638281712603167}, derivative=8.081374203964323E-11}, evalInputDelta = -1.3275158750047922E-10
2.6638281712603167 <= 2.6638281713930683
New Minimum: 2.6638281712603167 > 2.66382817122419
F(1.9330712927572975) = LineSearchPoint{point=PointSample{avg=2.66382817122419}, derivative=-3.3093686850656678E-22}, evalInputDelta = -1.688782447217818E-10
Left bracket at 1.9330712927572975
Converged to left
Fitness changed from 2.6638281713930683 to 2.66382817122419
Iteration 3 complete. Error: 2.66382817122419 Total: 1.6991; Orientation: 0.0745; Line Search: 1.4846
F(0.0) = LineSearchPoint{point=PointSample{avg=2.66382817122419}, derivative=-2.768945668479799E-10}
New Minimum: 2.66382817122419 > 2.6638281711130487
F(1.9330712927572975) = LineSearchPoint{point=PointSample{avg=2.6638281711130487}, derivative=1.6190490321760937E-10}, evalInputDelta = -1.1114131837075547E-10
2.6638281711130487 <= 2.66382817122419
New Minimum: 2.6638281711130487 > 2.663828171055309
F(1.219821296079253) = LineSearchPoint{point=PointSample{avg=2.663828171055309}, derivative=-2.2588493071233226E-22}, evalInputDelta = -1.688809092570409E-10
Left bracket at 1.219821296079253
Converged to left
Fitness changed from 2.66382817122419 to 2.663828171055309
Iteration 4 complete. Error: 2.663828171055309 Total: 1.1298; Orientation: 0.0751; Line Search: 0.7833
F(0.0) = LineSearchPoint{point=PointSample{avg=2.663828171055309}, derivative=-1.7473206675863106E-10}
New Minimum: 2.663828171055309 > 2.663828170909417
F(1.219821296079253) = LineSearchPoint{point=PointSample{avg=2.663828170909417}, derivative=-6.446943577232448E-11}, evalInputDelta = -1.4589218721994257E-10
F(8.53874907255477) = LineSearchPoint{point=PointSample{avg=2.663828172858583}, derivative=5.971063501447015E-10}, evalInputDelta = 1.8032739745876825E-9
F(0.6568268517349823) = LineSearchPoint{point=PointSample{avg=2.6638281709600387}, derivative=-1.1535988084287479E-10}, evalInputDelta = -9.527045818913393E-11
F(4.597787962144876) = LineSearchPoint{point=PointSample{avg=2.6638281712073604}, derivative=2.408732346509159E-10}, evalInputDelta = 1.520512604713531E-10
F(0.3536759970880674) = LineSearchPoint{point=PointSample{avg=2.663828170999164}, derivative=-1.4276242818850452E-10}, evalInputDelta = -5.6145310622923716E-11
New Minimum: 2.663828170909417 > 2.6638281708997376
F(2.475731979616472) = LineSearchPoint{point=PointSample{avg=2.6638281708997376}, derivative=4.905540323124979E-11}, evalInputDelta = -1.5557155563783454E-10
2.6638281708997376 <= 2.663828171055309
New Minimum: 2.6638281708997376 > 2.6638281708864273
F(1.9330383669755233) = LineSearchPoint{point=PointSample{avg=2.6638281708864273}, derivative=-1.5951014890348736E-22}, evalInputDelta = -1.688817974354606E-10
Left bracket at 1.9330383669755233
Converged to left
Fitness changed from 2.663828171055309 to 2.6638281708864273
Iteration 5 complete. Error: 2.6638281708864273 Total: 1.9614; Orientation: 0.0744; Line Search: 1.7471
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638281708864273}, derivative=-2.7690236610985034E-10}
New Minimum: 2.6638281708864273 > 2.6638281707752816
F(1.9330383669755233) = LineSearchPoint{point=PointSample{avg=2.6638281707752816}, derivative=1.6190662426281563E-10}, evalInputDelta = -1.1114575926285397E-10
2.6638281707752816 <= 2.6638281708864273
New Minimum: 2.6638281707752816 > 2.663828170717543
F(1.2198084117238859) = LineSearchPoint{point=PointSample{avg=2.663828170717543}, derivative=-4.986154259590153E-22}, evalInputDelta = -1.688844619707197E-10
Left bracket at 1.2198084117238859
Converged to left
Fitness chan
...skipping 8131 bytes...
-10}, evalInputDelta = 1.520876757865608E-10
F(0.353661811622589) = LineSearchPoint{point=PointSample{avg=2.663828169648058}, derivative=-1.4278129680548899E-10}, evalInputDelta = -5.615063969344192E-11
New Minimum: 2.6638281695583044 > 2.663828169548628
F(2.475632681358123) = LineSearchPoint{point=PointSample{avg=2.663828169548628}, derivative=4.9067132290358196E-11}, evalInputDelta = -1.5558043742203154E-10
2.663828169548628 <= 2.6638281697042085
New Minimum: 2.663828169548628 > 2.663828169535314
F(1.932917654332452) = LineSearchPoint{point=PointSample{avg=2.663828169535314}, derivative=-3.8701066674971225E-22}, evalInputDelta = -1.6889467602254626E-10
Left bracket at 1.932917654332452
Converged to left
Fitness changed from 2.6638281697042085 to 2.663828169535314
Iteration 13 complete. Error: 2.663828169535314 Total: 2.0025; Orientation: 0.0761; Line Search: 1.7869
F(0.0) = LineSearchPoint{point=PointSample{avg=2.663828169535314}, derivative=-2.769339249347475E-10}
New Minimum: 2.663828169535314 > 2.6638281694241526
F(1.932917654332452) = LineSearchPoint{point=PointSample{avg=2.6638281694241526}, derivative=1.619146618321379E-10}, evalInputDelta = -1.1116130238519872E-10
2.6638281694241526 <= 2.663828169535314
New Minimum: 2.6638281694241526 > 2.6638281693664174
F(1.219761185819417) = LineSearchPoint{point=PointSample{avg=2.6638281693664174}, derivative=-4.3235696642381065E-22}, evalInputDelta = -1.6889645237938566E-10
Left bracket at 1.219761185819417
Converged to left
Fitness changed from 2.663828169535314 to 2.6638281693664174
Iteration 14 complete. Error: 2.6638281693664174 Total: 1.0007; Orientation: 0.0759; Line Search: 0.7865
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638281693664174}, derivative=-1.7476199417661322E-10}
New Minimum: 2.6638281693664174 > 2.6638281692205097
F(1.219761185819417) = LineSearchPoint{point=PointSample{avg=2.6638281692205097}, derivative=-6.447745134781625E-11}, evalInputDelta = -1.4590773034228732E-10
F(8.53832830073592) = LineSearchPoint{point=PointSample{avg=2.6638281711700014}, derivative=5.972298056243698E-10}, evalInputDelta = 1.8035839488561578E-9
F(0.6567944846719939) = LineSearchPoint{point=PointSample{avg=2.663828169271136}, derivative=-1.1537800957644861E-10}, evalInputDelta = -9.528156041938018E-11
F(4.597561392703957) = LineSearchPoint{point=PointSample{avg=2.6638281695185135}, derivative=2.4092589802432745E-10}, evalInputDelta = 1.5209611348154795E-10
F(0.35365856866953516) = LineSearchPoint{point=PointSample{avg=2.6638281693102654}, derivative=-1.4278600246854975E-10}, evalInputDelta = -5.615197196107147E-11
New Minimum: 2.6638281692205097 > 2.663828169210835
F(2.475609980686746) = LineSearchPoint{point=PointSample{avg=2.663828169210835}, derivative=4.9069947777670176E-11}, evalInputDelta = -1.5558221377887094E-10
2.663828169210835 <= 2.6638281693664174
New Minimum: 2.663828169210835 > 2.6638281691975196
F(1.9328900658723136) = LineSearchPoint{point=PointSample{avg=2.6638281691975196}, derivative=-3.8104591905138065E-22}, evalInputDelta = -1.6889778464701521E-10
Left bracket at 1.9328900658723136
Converged to left
Fitness changed from 2.6638281693664174 to 2.6638281691975196
Iteration 15 complete. Error: 2.6638281691975196 Total: 2.2005; Orientation: 0.0752; Line Search: 1.9848
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638281691975196}, derivative=-2.769418999718533E-10}
New Minimum: 2.6638281691975196 > 2.6638281690863543
F(1.9328900658723136) = LineSearchPoint{point=PointSample{avg=2.6638281690863543}, derivative=1.619169432205537E-10}, evalInputDelta = -1.1116529918808737E-10
2.6638281690863543 <= 2.6638281691975196
New Minimum: 2.6638281690863543 > 2.663828169028619
F(1.2197503948774495) = LineSearchPoint{point=PointSample{avg=2.663828169028619}, derivative=-1.5178764240348058E-22}, evalInputDelta = -1.6890044918227431E-10
Left bracket at 1.2197503948774495
Converged to left
Fitness changed from 2.6638281691975196 to 2.663828169028619
Iteration 16 complete. Error: 2.663828169028619 Total: 1.0007; Orientation: 0.0760; Line Search: 0.7872
F(0.0) = LineSearchPoint{point=PointSample{avg=2.663828169028619}, derivative=-1.7476794101005327E-10}
New Minimum: 2.663828169028619 > 2.6638281688827083
F(1.2197503948774495) = LineSearchPoint{point=PointSample{avg=2.6638281688827083}, derivative=-6.447910252116516E-11}, evalInputDelta = -1.4591083896675627E-10
F(8.538252764142147) = LineSearchPoint{point=PointSample{avg=2.6638281708322653}, derivative=5.972539284114532E-10}, evalInputDelta = 1.8036461213455368E-9
F(0.6567886741647805) = LineSearchPoint{point=PointSample{avg=2.6638281689333363}, derivative=-1.1538164336206707E-10}, evalInputDelta = -9.528289268700973E-11
F(4.597520719153463) = LineSearchPoint{point=PointSample{avg=2.663828169180724}, derivative=2.409361425247273E-10}, evalInputDelta = 1.5210499526574495E-10
F(0.3536554399348818) = LineSearchPoint{point=PointSample{avg=2.6638281689724663}, derivative=-1.4279070381524112E-10}, evalInputDelta = -5.615286013949117E-11
New Minimum: 2.6638281688827083 > 2.6638281688730343
F(2.4755880795441727) = LineSearchPoint{point=PointSample{avg=2.6638281688730343}, derivative=4.9072719355005554E-11}, evalInputDelta = -1.5558487831413004E-10
2.6638281688730343 <= 2.663828169028619
New Minimum: 2.6638281688730343 > 2.663828168859718
F(1.9328634518204075) = LineSearchPoint{point=PointSample{avg=2.663828168859718}, derivative=-4.326606968049825E-22}, evalInputDelta = -1.6890133736069401E-10
Left bracket at 1.9328634518204075
Converged to left
Fitness changed from 2.663828169028619 to 2.663828168859718
Iteration 17 complete. Error: 2.663828168859718 Total: 1.9605; Orientation: 0.0744; Line Search: 1.7479
F(0.0) = LineSearchPoint{point=PointSample{avg=2.663828168859718}, derivative=-2.769499071475245E-10}
New Minimum: 2.663828168859718 > 2.663828168748549
F(1.9328634518204075) = LineSearchPoint{point=PointSample{avg=2.663828168748549}, derivative=1.6191932702916923E-10}, evalInputDelta = -1.1116885190176617E-10
2.663828168748549 <= 2.663828168859718
New Minimum: 2.663828168748549 > 2.6638281686908147
F(1.219739985908844) = LineSearchPoint{point=PointSample{avg=2.6638281686908147}, derivative=6.876455829729888E-22}, evalInputDelta = -1.6890311371753342E-10
Right bracket at 1.219739985908844
Converged to right
Fitness changed from 2.663828168859718 to 2.6638281686908147
Iteration 18 complete. Error: 2.6638281686908147 Total: 1.0057; Orientation: 0.0760; Line Search: 0.7908
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6638281686908147}, derivative=-1.74773875873246E-10}
New Minimum: 2.6638281686908147 > 2.6638281685449003
F(1.219739985908844) = LineSearchPoint{point=PointSample{avg=2.6638281685449003}, derivative=-6.448076858793533E-11}, evalInputDelta = -1.4591439168043507E-10
F(8.538179901361907) = LineSearchPoint{point=PointSample{avg=2.663828170494522}, derivative=5.972778751201217E-10}, evalInputDelta = 1.8037074056564961E-9
F(0.6567830693355313) = LineSearchPoint{point=PointSample{avg=2.663828168595529}, derivative=-1.1538527964267102E-10}, evalInputDelta = -9.528555722226884E-11
F(4.59748148534872) = LineSearchPoint{point=PointSample{avg=2.6638281688429277}, derivative=2.4094629773905373E-10}, evalInputDelta = 1.5211298887152225E-10
F(0.3536524219499015) = LineSearchPoint{point=PointSample{avg=2.66382816863466}, derivative=-1.4279540097958003E-10}, evalInputDelta = -5.615463649633057E-11
New Minimum: 2.6638281685449003 > 2.663828168535227
F(2.4755669536493103) = LineSearchPoint{point=PointSample{avg=2.663828168535227}, derivative=4.9075448379778765E-11}, evalInputDelta = -1.5558754284938914E-10
2.663828168535227 <= 2.6638281686908147
New Minimum: 2.663828168535227 > 2.66382816852191
F(1.9328377823645269) = LineSearchPoint{point=PointSample{avg=2.66382816852191}, derivative=-1.459521416758604E-22}, evalInputDelta = -1.6890489007437282E-10
Left bracket at 1.9328377823645269
Converged to left
Fitness changed from 2.6638281686908147 to 2.66382816852191
Iteration 19 complete. Error: 2.66382816852191 Total: 2.2430; Orientation: 0.0752; Line Search: 2.0302
Final threshold in iteration 19: 2.66382816852191 (> 0.0) after 30.460s (< 30.000s)
Returns
2.66382816852191
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.001696713369322916, 0.003250608787299474, -0.0011754697585850352, -1.840363392511991E-4, -0.0012351129131373195, -3.091268560210554E-4, -0.608, 1.764, 0.08]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.8439979376657737, 1.211998586565409, 0.660000422381774 ], [ 0.12799846351874325, 1.5879966297830823, -0.5399994576904565 ], [ 1.443996246283932, -0.34000302614725175, 0.9400008168941535 ], [ -0.3159980233915835, 0.4040000843080805, -0.4200002998959414 ], [ 0.17600214306785977, -1.4439980808873243, -0.6880005094382531 ], [ 1.0999985234609377, -1.3320000886222914, -0.6839997985095333 ], [ 0.2799988159579692, 0.6319981217037773, -0.24399965332621174 ], [ 1.3199973167577088, 0.2399977203532882, -0.811999418308264 ], ... ],
[ [ 0.355997764351426, 0.047997104593226814, 1.7000006064575308 ], [ 1.6080007979305813, 1.5919995781651461, -0.0400001264222447 ], [ 1.9039974288808508, 0.6159984505343236, 0.31600048329388625 ], [ -1.7560008672996525, 0.39199684149254366, -0.7199995268792585 ], [ 1.7039996310413126, -1.3199986520883902, 1.5639998897623233 ], [ -0.6480006606930654, 0.8159989878614817, -1.1719997923291723 ], [ 1.1599987342429665, -0.13999957324395282, 1.4880001247161518 ], [ -1.0559986522167168, 0.12000139495838583, 0.16799969370151482 ], ... ],
[ [ -0.5679980470408948, -0.5119988303530265, 0.09199961110339602 ], [ 1.8880004361214715, 0.9559998984953312, 1.3479998958833448 ], [ 0.15199792338099388, 0.6919972440263755, 1.6120005735597727 ], [ 0.967999541844464, 0.24800014187848632, -1.2319999745535766 ], [ -0.6920018195296759, -1.3040001378848531, -0.8599996944791536 ], [ 1.0160014886826965, 1.6080005924343992, 1.3039996938593312 ], [ -0.4640000813607156, -0.547999644355243, -1.004000011204515 ], [ -1.1039979758904572, 1.3760001748316182, 0.03599970564306598 ], ... ],
[ [ -1.8439995796665982, 0.5280000229038962, 0.7999999857664214 ], [ -0.007997737154963415, 1.2920021699659285, -1.724000553552711 ], [ 1.2560013775270347, 0.42400145225580405, 0.5959996225629599 ], [ 0.2520016198922317, 0.3800024555961719, 1.0279995198182876 ], [ -0.22000218023738682, 1.2519963564295729, 0.9320006798949405 ], [ 0.064002697874716, -1.5279968319200463, 1.5679992994198864 ], [ -1.935998124172509, 1.4479990153781086, -1.4400001451274753 ], [ 0.7999971225916838, -1.128001597311156, 1.896000571383747 ], ... ],
[ [ -1.3399961379505305, 1.2360025185456036, -1.608000791648698 ], [ 0.9320003202929922, -1.919998393474101, 1.263999781958387 ], [ 1.8239997717489114, -0.5399983542705777, -1.4920001737798303 ], [ 0.7359996661836975, 1.9679992227352079, 1.9640001073683178 ], [ -0.1479995898081905, 0.604001006740664, -1.5720001588344688 ], [ -1.364000267420307, -1.5039992153658677, -1.327999999001223 ], [ -0.30400060586430133, 1.231999750615754, -1.3839998830679818 ], [ -0.08400196722061085, 1.2839967129524783, -1.9999993990439895 ], ... ],
[ [ 1.9439963930615853, -1.9560009201822044, -0.2679994188252951 ], [ 0.5959989057375567, 1.659998809534749, -0.5879997428423247 ], [ 1.7079987178463756, -1.7999995463162184, -0.7759998949649961 ], [ 0.919999036119855, 0.8279993942595413, 0.960000179822087 ], [ -1.475997510726569, -1.4919974611277995, -1.5920005785701092 ], [ -0.32799765140831283, 0.16400125466935767, -0.7000004666523919 ], [ 0.09600005891085955, 0.7559991688072546, 0.23600006655216285 ], [ -0.17600007888121466, 0.032000369858166997, 0.44799998317045475 ], ... ],
[ [ -1.983998465227221, 1.607997981707928, 1.9960000159718312 ], [ 0.3640007619884326, -1.699998867951187, -1.048000232504034 ], [ -1.8520004731220845, 1.1759975847783442, 1.3680003516805392 ], [ -0.9399981427789498, -1.5159986236863976, -0.8440003854744417 ], [ 1.2239999923690033, 1.5639995164106884, -1.9679999947906177 ], [ -1.9919967980386748, -1.2319975114047408, -0.4480006642721274 ], [ 0.12399994820883331, 0.46400073919797397, -1.024000069186249 ], [ 1.9959979473196872, 0.04399842404773744, 1.052000408228103 ], ... ],
[ [ 0.6159992189671044, 1.795997804766433, -1.2759996983622954 ], [ 1.8279965344435158, -0.4480008086942477, -1.6599994542504792 ], [ -1.9879982333174544, -1.0319977725359795, -1.0200004251399728 ], [ 0.37599743851409145, -1.1440024531602346, 1.2920006138493312 ], [ -0.8999982020638668, 0.9120006118184706, 0.739999696055079 ], [ -1.7079964665164484, 1.883999724044711, 1.0799995390732935 ], [ 0.23999879510180952, 0.38799847019910355, -0.8959996837054908 ], [ 1.4120005857489528, -0.9639986551814942, -0.5640002536476606 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004595945303039424, -0.001856305251298133 ], [ 0.0060138855354473315, -0.0018179831775104699 ], [ 2.3988837851507758E-4, -1.363851521420449E-4 ], [ 0.001270785804277784, -3.1099722936714136E-4 ], [ -0.003586523868667426, 0.0019637893205623987 ], [ -0.0016594079137414444, 0.0016541735155502582 ], [ 0.0028162705959877284, -7.566921525754821E-4 ], [ 0.003974276553735425, -2.8834093025680024E-4 ], ... ],
[ [ -0.0012382533262872387, -6.503142117172485E-4 ], [ 0.007950303208840341, -0.002249864703743735 ], [ 0.004861458857198112, -0.0012089165933842237 ], [ -8.588641004985533E-4, 6.15788005841076E-5 ], [ -0.003238034835238811, 8.332751575964854E-4 ], [ 0.0029306724089928865, -5.262986065283419E-4 ], [ -2.3599762959186505E-4, -5.005474401512224E-4 ], [ -0.0015991280016608397, -5.8063634258788405E-6 ], ... ],
[ [ -0.0027361805378498198, 7.084690976239109E-4 ], [ 0.004726444139774478, -0.0019449314780891416 ], [ 6.12451305887763E-4, -0.0013809805425539063 ], [ 0.003896747917252946, -1.0361099102013143E-4 ], [ -0.0044020194123873995, 0.0020037898923892464 ], [ 0.005418031959514147, -0.002576144816314272 ], [ -0.0013884359499998424, 0.00107259768043246 ], [ 0.0025573535689331064, -0.0015074683142132519 ], ... ],
[ [ -0.0023527930158373134, -5.600781946196751E-4 ], [ 0.0062127342538311515, -0.0010613618187597547 ], [ 0.0028087576432177194, -9.390790539972824E-4 ], [ 4.544314915321412E-4, -8.335056550936431E-4 ], [ 0.0026009311031907354, -0.0017939749111342772 ], [ -0.0067014614535988925, 0.0013907591024454134 ], [ 0.0031147210460768923, -9.87005557000609E-4 ], [ -0.004538017424907855, 6.598761013792827E-4 ], ... ],
[ [ 0.0036343275881707215, -7.829184582850335E-4 ], [ -0.0061456197643235885, 0.0018091566032470072 ], [ 0.003093282486930939, 7.924940225482264E-4 ], [ 0.005337353308183464, -0.003173277115714799 ], [ 0.003560096744567857, -2.3282467348925776E-4 ], [ -0.005642206716824976, 0.0025191549326876823 ], [ 0.005115797331481742, -0.0010378801097477488 ], [ 0.006382182547918785, -9.521679797858288E-4 ], ... ],
[ [ -0.002744763896939513, 0.002140961832685864 ], [ 0.007098421944413993, -0.00197820491041257 ], [ -0.002040945673837705, 0.002148751259621488 ], [ 0.0031240255915904016, -0.0014887478360100174 ], [ -0.005482896231548602, 0.0026065526428621206 ], [ 7.994152988506503E-4, 7.41923631048566E-5 ], [ 0.0023429311834696803, -0.0010243657937188044 ], [ -7.212114354453282E-4, -1.456224861067397E-4 ], ... ],
[ [ -4.855420082776973E-4, -0.0022379484664411422 ], [ -0.0036765337189532065, 0.0023566662033876447 ], [ -9.276489229493295E-4, -0.0015345400631638585 ], [ -0.00553072893432135, 0.0023063264791557104 ], [ 0.009474052201252225, -0.0015486148256593742 ], [ -0.006857978302706102, 0.0020267448706419726 ], [ 0.002922358364189082, -2.7936787927148286E-4 ], [ 0.002293063400304068, -7.468807558050954E-4 ], ... ],
[ [ 0.008383159413846798, -0.0019371805468350522 ], [ 0.0035965899513465804, 7.300642058404826E-4 ], [ -0.005528704555043084, 0.0019558072170493936 ], [ -0.004599452195829974, 9.443829226704987E-4 ], [ 5.676709569164039E-4, -0.001189545137503677 ], [ 0.0019566588212883303, -0.002346475832381015 ], [ 0.0027216609328561516, -2.464128552759354E-4 ], [ -7.485698614868962E-5, 0.0011051353936450666 ], ... ],
...
]
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 38.30 seconds (0.264 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 11636873050413
Reset training subject: 11637013384040
Adding measurement 1f2927f7 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 3.570995254823763 < 3.570995254823763. Total: 1
th(0)=3.570995254823763;dx=-2.418916625194388
Non-optimal measurement 5.8440746635911545 < 3.570995254823763. Total: 1
Armijo: th(2.154434690031884)=5.8440746635911545; dx=4.529042236211652 evalInputDelta=-2.2730794087673916
Adding measurement 5f45d21e to history. Total: 1
New Minimum: 3.570995254823763 > 2.836419509725103
WOLF (strong): th(1.077217345015942)=2.836419509725103; dx=1.0550735974931589 evalInputDelta=0.7345757450986601
Non-optimal measurement 2.9103318046799687 < 2.836419509725103. Total: 2
END: th(0.3590724483386473)=2.9103318046799687; dx=-1.2609174860679235 evalInputDelta=0.6606634501437942
Fitness changed from 3.570995254823763 to 2.836419509725103
Iteration 1 complete. Error: 2.836419509725103 Total: 1.3977; Orientation: 0.1041; Line Search: 0.8767
Non-optimal measurement 2.836419509725103 < 2.836419509725103. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 2.836419509725103 < 2.836419509725103. Total: 2
th(0)=2.836419509725103;dx=-0.46021533358154804
Adding measurement 6d84eafd to history. Total: 2
New Minimum: 2.836419509725103 > 2.6639989716087307
WOLF (strong): th(0.7735981389354633)=2.6639989716087307; dx=0.014452720652775253 evalInputDelta=0.17242053811637215
Non-optimal measurement 2.704308950245947 < 2.6639989716087307. Total: 3
END: th(0.3867990694677316)=2.704308950245947; dx=-0.222881256500496 evalInputDelta=0.13211055947915584
Fitness changed from 2.836419509725103 to 2.6639989716087307
Iteration 2 complete. Error: 2.6639989716087307 Total: 0.8947; Orientation: 0.0779; Line Search: 0.6741
Non-optimal measurement 2.6639989716087307 < 2.6639989716087307. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 2.6639989716087307 < 2.6639989716087307. Total: 3
th(0)=2.6639989716087307;dx=-4.5554386841891744E-4
Adding measurement 465ce0 to history. Total: 3
New Minimum: 2.6639989716087307 > 2.6638302877608986
WOLF (strong): th(0.8333333333333335)=2.6638302877608986; dx=5.070263352827596E-5 evalInputDelta=1.6868384783208512E-4
Non-optimal measurement 2.663861895674195 < 2.6638302877608986. Total: 4
END: th(0.41666666666666674)=2.663861895674195; dx=-2.0242061737628974E-4 evalInputDelta=1.3707593453560207E-4
Fitness changed from 2.6639989716087307 to 2.6638302877608986
Iteration 3 complete. Error: 2.6638302877608986 Total: 1.1655; Orientation: 0.0822; Line Search: 0.9442
Non-optimal measurement 2.6638302877608986 < 2.6638302877608986. Total: 4
Rejected: LBFGS Orientation magnitude: 1.782e-03, gradient 2.376e-03, dot -1.000; [888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.6638302877608986, 2.6639989716087307, 2.836419509725103, 3.570995254823763
LBFGS Accumulation History: 3 points
Removed measurement 465ce0 to history. Total: 3
Adding measurement 289f36f3 to history. Total: 3
th(0)=2.6638302877608986;dx=-5.645536518584081E-6
Adding measurement 2bca062 to history. Total: 4
New Minimum: 2.6638302877608986 > 2.6638282534635165
WOLF (strong): th(0.8976811208466184)=2.6638282534635165; dx=1.1131979530570547E-6 evalInputDelta=2.034297382103034E-6
Non-optimal measurement 2.6638285122136653 < 2.6638282534635165. Total: 5
END: th(0.4488405604233092)=2.6638285122136653; dx=-2.2661692827712686E-6 evalInputDelta=1.77554723324036E-6
Fitness changed from 2.6638302877608986 to 2.6638282534635165
Iteration 4 complete. Error: 2.6638282534635165 Total: 3.2102; Orientation: 2.3783; Line Search: 0.6930
Non-optimal measurement 2.6638282534635165 < 2.6638282534635165. Total: 5
Rejected: LBFGS Orientation magnitude: 3.514e-04, gradient 4.687e-04, dot -1.000; [888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.6638282534635165, 2.6638302877608986, 2.6639989716087307, 2.836419509725103, 3.570995254823763
Rejected: LBFGS Orientation magnitude: 3.514e-04, gradient 4.687e-04, dot -1.000; [1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.6638282534635165, 2.6638302877608986, 2.6639989716087307, 2.836419509725103
LBFGS Accumulation History: 3 points
Removed measurement 2bca062 to history. Total: 4
Removed measurement 289f36f3 to history. Total: 3
Adding measurement 30200c46 to history. Total: 3
th(0)=2.6638282534635165;dx=-2.1968768238313168E-7
Adding measurement 420ef1bd to history. Total: 4
New Minimum: 2.6638282534635165 > 2.6638281779507134
WOLF (strong): th(0.9669976736693292)=2.6638281779507134; dx=6.350777625113399E-8 evalInputDelta=7.551280312156905E-8
Non-optimal measurement 2.663828181475946 < 2.6638281779507134. Total: 5
END: th(0.4834988368346646)=2.663828181475946; dx=-7.808995306580874E-8 evalInputDelta=7.198757057125249E-8
Fitness changed from 2.6638282534635165 to 2.6638281779507134
Iteration 5 complete. Error: 2.6638281779507134 Total: 6.3097; Orientation: 5.3489; Line Search: 0.8195
Non-optimal measurement 2.6638281779507134 < 2.6638281779507134. Total: 5
Rejected: LBFGS Orientation magnitude: 1.021e-04, gradient 1.362e-04, dot -1.000; [1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.6638281779507134, 2.6638282534635165, 2.6639989716087307, 2.836419509725103, 3.570995254823763
Rejected: LBFGS Orientation magnitude: 1.021e-04, gradient 1.362e-04, dot -1.000; [888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.6638281779507134, 2.6638282534635165, 2.6639989716087307, 2.836419509725103
LBFGS Accumulation History: 3 points
Removed measurement 420ef1bd to history. Total: 4
Removed measurement 30200c46 to history. Total: 3
Adding measurement 55888b4d to history. Total: 3
th(0)=2.6638281779507134;dx=-1.8538516871576414E-8
Adding measurement 38b69196 to history. Total: 4
New Minimum: 2.6638281779507134 > 2.663828171977059
WOLF (strong): th(1.041666666666667)=2.663828171977059; dx=7.069099144525347E-9 evalInputDelta=5.973654459978661E-9
Adding measurement 63343968 to history. Total: 5
New Minimum: 2.663828171977059 > 2.663828171629561
END: th(0.5208333333333335)=2.663828171629561; dx=-5.734708863530793E-9 evalInputDelta=6.321152490329496E-9
Fitness changed from 2.6638281779507134 to 2.663828171629561
Iteration 6 complete. Error: 2.663828171629561 Total: 6.0777; Orientation: 5.1621; Line Search: 0.7574
Non-optimal measurement 2.663828171629561 < 2.663828171629561. Total: 6
Rejected: LBFGS Orientation magnitude: 3.206e-05, gradient 4.272e-05, dot -1.000; [1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.663828171629561, 2.663828171977059, 2.6638281779507134, 2.6639989716087307, 2.836419509725103, 3.570995254823763
Rejected: LBFGS Orientation magnitude: 3.206e-05, gradient 4.272e-05, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.663828171629561, 2.663828171977059, 2.6638281779507134, 2.6639989716087307, 2.836419509725103
Rejected: LBFGS Orientation magnitude: 3.203e-05, gradient 4.272e-05, dot -1.000; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.663828171629561, 2.663828171977059, 2.6638281779507134, 2.6639989716087307
LBFGS Accumulation History: 3 points
Removed measurement 63343968 to history. Total: 5
Removed measurement 38b69196 to history. Total: 4
Removed measurement 55888b4d to history. Total: 3
Adding measurement 4cca0eab to history. Total: 3
th(0)=2.663828171629561;dx=-1.8254110561958456E-9
Adding measurement 30ef30e8 to history. Total: 4
New Minimum: 2.663828171629561 > 2.6638281710240532
WOLF (strong): th(1.122101401058273)=2.6638281710240532; dx=7.461726860981008E-10 evalInputDelta=6.055076440247831E-10
Adding measurement 2c19e322 to history. Total: 5
New Minimum: 2.6638281710240532 > 2.66382817096611
END: th(0.5610507005291365)=2.66382817096611; dx=-5.396191850485876E-10 evalInputDelta=6.634510718583897E-10
Fitness changed from 2.663828171629561 to 2.66382817096611
Iteration 7 complete. Error: 2.66382817096611 Total: 9.7848; Orientation: 8.8974; Line Search: 0.7412
Non-optimal measurement 2.66382817096611 < 2.66382817096611. Total: 6
Rejected: LBFGS Orientation magnitude: 1.174e-05, gradient 1.470e-05, dot -0.998; [888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.66382817096611, 2.6638281710240532, 2.663828171629561, 2.6639989716087307, 2.836419509725103, 3.570995254823763
Rejected: LBFGS Orientation magnitude: 1.174e-05, gradient 1.470e-05, dot -0.998; [888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00, ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.66382817096611, 2.6638281710240532, 2.663828171629561, 2.6639989716087307, 2.836419509725103
Rejected: LBFGS Orientation magnitude: 1.171e-05, gradient 1.470e-05, dot -0.998; [ecba8185-41f8-4d57-94e8-8d8d06db1508 = 1.000/1.000e+00, 1f9d3794-1705-4ca1-a2b4-fe37e9356ce6 = 1.000/1.000e+00, 888b0b9f-e13d-4fc0-aabc-99548dad05ba = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.66382817096611, 2.6638281710240532, 2.663828171629561, 2.6639989716087307
LBFGS Accumulation History: 3 points
Removed measurement 2c19e322 to history. Total: 5
Removed measurement 30ef30e8 to history. Total: 4
Removed measurement 4cca0eab to history. Total: 3
Adding measurement 39f32759 to history. Total: 3
th(0)=2.66382817096611;dx=-2.1599122923923087E-10
Adding measurement 15aac0ec to history. Total: 4
New Minimum: 2.66382817096611 > 2.6638281708110965
END: th(1.2087470920866614)=2.6638281708110965; dx=-4.049445611463698E-11 evalInputDelta=1.55013335501053E-10
Fitness changed from 2.66382817096611 to 2.6638281708110965
Iteration 8 complete. Error: 2.6638281708110965 Total: 9.4623; Orientation: 8.8303; Line Search: 0.4899
Final threshold in iteration 8: 2.6638281708110965 (> 0.0) after 38.302s (< 30.000s)
Returns
2.6638281708110965
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.0017069443415921833, 0.0032547804226356006, -0.0011706241234774426, -1.7900253140640466E-4, -0.0012369396420807398, -3.0789287363586627E-4, -0.608, 1.764, 0.08]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.8439976063031105, 1.2119982312046853, 0.6600004563791083 ], [ 0.12799845434538284, 1.5879962430671553, -0.5399994536963272 ], [ 1.4439958254579077, -0.3400024580395808, 0.9400007406781775 ], [ -0.31599798401288537, 0.40399979194954766, -0.4200002305115198 ], [ 0.17600194031702437, -1.4439977426440065, -0.6880004597000101 ], [ 1.0999980432323537, -1.3319995034593124, -0.6839998154989421 ], [ 0.27999873265373865, 0.6319980278115008, -0.24399965845930327 ], [ 1.3199967532219743, 0.23999781608338255, -0.8119994043333686 ], ... ],
[ [ 0.3559978695628071, 0.04799752223475261, 1.7000005014711823 ], [ 1.607999870662104, 1.5919987227987802, -0.03999988734534245 ], [ 1.9039965864798476, 0.6159983159125054, 0.3160005622172694 ], [ -1.7559998884013133, 0.39199708136685685, -0.7199997137648319 ], [ 1.7039988902492913, -1.319998186525784, 1.5639999482307894 ], [ -0.6480003851528383, 0.8159987211667844, -1.1719998243868928 ], [ 1.1599982705465146, -0.1399994905186847, 1.488000155510989 ], [ -1.0559982040132767, 0.12000122798453045, 0.1679996721721886 ], ... ],
[ [ -0.5679978347702094, -0.5119987523998923, 0.09199961816975 ], [ 1.8879994880329856, 0.9559993781951833, 1.3480000979938873 ], [ 0.151998082876541, 0.691997352423432, 1.612000492357084 ], [ 0.9679989656797627, 0.24799990399222446, -1.2319998769210923 ], [ -0.6920013776602015, -1.303999425555107, -0.8599998712043293 ], [ 1.016000888176346, 1.607999713661402, 1.3039999009972603 ], [ -0.4639998866335485, -0.5479994378501061, -1.0040000620978102 ], [ -1.1039975597054204, 1.375999500951845, 0.03599975384987395 ], ... ],
[ [ -1.843998626434975, 0.5279999279063948, 0.7999998602953038 ], [ -0.007998069251173235, 1.2920011705690428, -1.724000359195343 ], [ 1.2560006458798694, 0.4240009950154409, 0.595999807008467 ], [ 0.25200138918313325, 0.38000200825834396, 1.0279996318943914 ], [ -0.22000186711011865, 1.2519962834312055, 0.9320005908199958 ], [ 0.06400263469372282, -1.5279964726466053, 1.567999337773624 ], [ -1.935997330980674, 1.4479984051717167, -1.4400001617204075 ], [ 0.7999970521858335, -1.1280007436421988, 1.8960004353425215 ], ... ],
[ [ -1.3399958462272408, 1.2360015345198154, -1.6080006543496466 ], [ 0.9319999547989867, -1.9199976683189521, 1.2639997764142004 ], [ 1.8239987200067507, -0.5399984511065551, -1.4920000135646305 ], [ 0.7359993466425354, 1.9679984476687156, 1.9640002195541049 ], [ -0.14799969526212448, 0.6040005430935038, -1.5720000917575698 ], [ -1.3639995883373217, -1.5039985485866658, -1.3280001706192657 ], [ -0.3040005713842104, 1.2319991827326133, -1.3839998508207838 ], [ -0.08400193039923813, 1.283996420543471, -1.9999994213175114 ], ... ],
[ [ 1.9439956456885017, -1.9559999021237477, -0.26799948621223846 ], [ 0.5959985562828939, 1.6599981311122647, -0.5879996529627595 ], [ 1.7079979182045235, -1.799998851846813, -0.775999869572531 ], [ 0.9199986364556118, 0.8279990981196481, 0.9600002445927741 ], [ -1.4759969537697657, -1.491997103860648, -1.592000638050186 ], [ -0.32799766558513155, 0.16400093379013578, -0.7000003791185364 ], [ 0.09600001143954957, 0.7559988972915919, 0.23600010127647766 ], [ -0.17599997218800284, 0.032000360602989365, 0.4479999653293927 ], ... ],
[ [ -1.9839973908706279, 1.6079975903499302, 1.9959999297290465 ], [ 0.36400053177257286, -1.6999983020503926, -1.0480002322688586 ], [ -1.8519993958855456, 1.17599748802909, 1.368000189192128 ], [ -0.9399977487295088, -1.5159981164836644, -0.8440004478543739 ], [ 1.2239991741085492, 1.5639986474372922, -1.9679997906524314 ], [ -1.9919959421322007, -1.2319972095372318, -0.44800074640226084 ], [ 0.12399977124277879, 0.46400039693740375, -1.0240000133887004 ], [ 1.9959971164039274, 0.04399856272922905, 1.0520004731835215 ], ... ],
[ [ 0.6159988327848488, 1.7959971032193731, -1.2759995915425097 ], [ 1.8279956698336068, -0.4480005705659977, -1.6599994291249183 ], [ -1.9879973645361078, -1.0319974979185005, -1.0200005416764244 ], [ 0.3759975674400615, -1.1440015300215494, 1.2920004503435571 ], [ -0.8999978288895722, 0.9120001458811163, 0.7399997254902064 ], [ -1.7079957405639723, 1.883998878203721, 1.0799995938935016 ], [ 0.23999869276948863, 0.3879984275358471, -0.89599969164793 ], [ 1.411999807476605, -0.9639984909125487, -0.5640001382986316 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004612832597846922, -0.001853455803453563 ], [ 0.006019201707692819, -0.0018209055683086168 ], [ 2.5779961634836125E-4, -1.2733591861044234E-4 ], [ 0.0012672020445237834, -3.1384384109722167E-4 ], [ -0.0035940801318659213, 0.0019664636967021204 ], [ -0.0016570237866845895, 0.0016612992235403735 ], [ 0.0028205889467341817, -7.567381402432908E-4 ], [ 0.003984847273057751, -2.831367430104945E-4 ], ... ],
[ [ -0.0012261716521721202, -6.465125973812229E-4 ], [ 0.00797319738939683, -0.002244726697461537 ], [ 0.00488503757754716, -0.0012000672663478235 ], [ -8.786806133500211E-4, 5.113447653863991E-5 ], [ -0.003218529060144663, 8.461935310929774E-4 ], [ 0.0029217673389017665, -5.324970629724894E-4 ], [ -2.1750399453243142E-4, -4.92616351012104E-4 ], [ -0.0016086169805167557, -1.113382614841949E-5 ], ... ],
[ [ -0.002743681178202865, 7.066585769848455E-4 ], [ 0.004756276670066732, -0.0019355098401788275 ], [ 6.247050392717087E-4, -0.0013794904629255346 ], [ 0.0039017143655086283, -1.0071119531139024E-4 ], [ -0.0044187030419766266, 0.0020016264127290955 ], [ 0.005441449213634439, -0.0025723575979159194 ], [ -0.001400333130253843, 0.0010700238470443297 ], [ 0.0025519716692877633, -0.0015154940400199097 ], ... ],
[ [ -0.002365578328034463, -5.693378756960114E-4 ], [ 0.006209684266296292, -0.001065888366107325 ], [ 0.0028262615816030305, -9.327950273431114E-4 ], [ 4.635742743594831E-4, -8.316621954273266E-4 ], [ 0.0026084196756119716, -0.0017962192837248587 ], [ -0.006699581920283442, 0.0013958069543830126 ], [ 0.0030939760988267727, -0.0010011724181567933 ], [ -0.004525348143341682, 6.683023161962016E-4 ], ... ],
[ [ 0.003617979625977247, -7.939047048953762E-4 ], [ -0.006137967403482782, 0.0018189143540313655 ], [ 0.003102459115379173, 8.008212743138114E-4 ], [ 0.005362606703891379, -0.003170742713057252 ], [ 0.00355348313005712, -2.3661226988427064E-4 ], [ -0.005668867375149624, 0.0025133965942493036 ], [ 0.0051111183777560925, -0.001043368065125454 ], [ 0.006376987362076481, -9.574039455759856E-4 ], ... ],
[ [ -0.0027343311569820875, 0.0021539888091644623 ], [ 0.007108592360354651, -0.001978961841594306 ], [ -0.002034739474690172, 0.0021596788143374043 ], [ 0.003141542276425648, -0.0014844442268860916 ], [ -0.005511933261226225, 0.002599883205987225 ], [ 7.9335059950762E-4, 7.137828436485428E-5 ], [ 0.002348209675060935, -0.001024971999852389 ], [ -7.207075761704487E-4, -1.4601407075564212E-4 ], ... ],
[ [ -4.894597115642022E-4, -0.002248409562745853 ], [ -0.003684978190677752, 0.002360310077739824 ], [ -9.350603107232582E-4, -0.0015443228412886797 ], [ -0.005550757544102607, 0.0023033218673771247 ], [ 0.009483558673001845, -0.0015477377669218639 ], [ -0.006885666119166236, 0.0020184147408032088 ], [ 0.0029205992340426328, -2.8085445113393336E-4 ], [ 0.002318764512456883, -7.356155517053122E-4 ], ... ],
[ [ 0.008390767836125178, -0.0019389341834478806 ], [ 0.0036053787553621125, 7.380358075748791E-4 ], [ -0.00555828886488266, 0.0019464260742716479 ], [ -0.004594113757900962, 9.499585953151414E-4 ], [ 5.658524887608429E-4, -0.0011948278863632006 ], [ 0.0019522754222681066, -0.0023571815153829567 ], [ 0.0027213929502408894, -2.4701908978571577E-4 ], [ -6.716616643399177E-5, 0.0011133080317438646 ], ... ],
...
]
TrainingTester.java:432 executed in 0.01 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, 0.42550620447787035], [32.0, 0.45277046391756653]; valueStats=DoubleSummaryStatistics{count=59, sum=157.511394, min=2.663828, average=2.669685, max=2.836420}
Plotting 32 points for GD
Plotting 19 points for CjGD
Plotting 8 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, 0.42550620447787035], [36.905, 0.45277046391756653]; valueStats=DoubleSummaryStatistics{count=59, sum=157.511394, min=2.663828, average=2.669685, max=2.836420}
Plotting 32 points for GD
Plotting 19 points for CjGD
Plotting 8 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "NonConverged", "value": 0.19243157802883468 }, "CjGD": { "type": "Converged", "value": 2.96001129660126E-28 }, "GD": { "type": "Converged", "value": 5.1632771595686515E-27 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 2.6638281708110965 }, "CjGD": { "type": "NonConverged", "value": 2.66382816852191 }, "GD": { "type": "NonConverged", "value": 2.6638281528451904 } }}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "NonConverged", "value": 0.19243157802883468 }, "CjGD": { "type": "Converged", "value": 2.96001129660126E-28 }, "GD": { "type": "Converged", "value": 5.1632771595686515E-27 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 2.6638281708110965 }, "CjGD": { "type": "NonConverged", "value": 2.66382816852191 }, "GD": { "type": "NonConverged", "value": 2.6638281528451904 } }} | OK |
{
"result": "OK",
"performance": {
"execution_time": "214.574",
"gc_time": "2.758"
},
"created_on": 1586746092924,
"file_name": "trainingTest",
"report": {
"simpleName": "BandLimit",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayerTest.BandLimit",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "NonConverged",
"value": 0.19243157802883468
},
"CjGD": {
"type": "Converged",
"value": 2.96001129660126E-28
},
"GD": {
"type": "Converged",
"value": 5.1632771595686515E-27
}
},
"model": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
},
"complete": {
"LBFGS": {
"type": "NonConverged",
"value": 2.6638281708110965
},
"CjGD": {
"type": "NonConverged",
"value": 2.66382816852191
},
"GD": {
"type": "NonConverged",
"value": 2.6638281528451904
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer/BandLimit/trainingTest/202004134812",
"id": "d8893d48-0e66-48a9-bba4-abb182cd83f2",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "ConvolutionLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayer",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer.java",
"javaDoc": ""
}
}