Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
This is a network apply the following layout:
LayerTests.java:203 executed in 0.07 seconds (0.000 gc):
return Graphviz.fromGraph((Graph) TestUtil.toGraph(((DAGNetwork) layer).addRef())).height(400).width(600)
.render(Format.PNG).toImage();
executing command [/bin/sh, -c, dot -Tsvg /tmp/GraphvizJava/DotEngine5513938092332764149/dotfile.dot -ooutfile.svg]
Returns
Using Seed 3224499097271563264
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -1.576, -0.1 ], [ 0.248, -1.128 ], [ 1.52, -1.86 ], [ -1.72, 0.98 ], [ -1.232, -0.86 ], [ -0.896, 0.304 ], [ -0.772, -1.584 ], [ -1.156, -0.512 ], ... ],
[ [ -1.376, 0.636 ], [ -1.02, -1.1 ], [ 1.72, -0.772 ], [ -1.632, 1.036 ], [ -1.924, 0.916 ], [ -1.404, 0.732 ], [ -0.256, -1.14 ], [ 1.976, 0.168 ], ... ],
[ [ 0.38, -0.144 ], [ 0.692, -1.992 ], [ -0.428, -0.252 ], [ -1.372, -0.908 ], [ -1.596, -0.384 ], [ 0.136, 1.232 ], [ -1.808, -1.288 ], [ -1.664, -0.992 ], ... ],
[ [ 1.528, 0.888 ], [ 0.456, 1.08 ], [ -0.144, -1.736 ], [ 0.028, 0.024 ], [ -0.304, 0.908 ], [ -0.972, 0.836 ], [ -0.448, -0.84 ], [ -0.5, -1.456 ], ... ],
[ [ -0.516, 1.484 ], [ 1.5, 0.82 ], [ 0.9, 1.26 ], [ -1.552, 0.024 ], [ 0.748, 1.76 ], [ -1.896, 1.856 ], [ 1.84, -0.364 ], [ 1.372, -0.312 ], ... ],
[ [ -1.596, -0.156 ], [ 1.552, -0.472 ], [ -1.144, -0.232 ], [ 0.668, 1.42 ], [ -0.208, -1.956 ], [ 0.128, -1.092 ], [ 0.84, -0.18 ], [ -0.244, 1.712 ], ... ],
[ [ -1.12, -1.992 ], [ -0.552, -0.908 ], [ -0.296, -1.532 ], [ 0.184, -0.388 ], [ 1.88, -1.336 ], [ 1.448, -0.264 ], [ -1.18, -0.528 ], [ -1.44, -1.2 ], ... ],
[ [ -1.852, -1.136 ], [ 0.816, 1.836 ], [ -1.956, -1.828 ], [ -0.696, 0.952 ], [ 0.184, -1.236 ], [ 0.852, -0.836 ], [ -0.7, 1.032 ], [ -0.272, -1.552 ], ... ],
...
]
[
[ [ 0.72, 1.816 ], [ -0.048, -0.272 ], [ 0.428, -0.416 ], [ -0.404, -1.336 ], [ -0.72, -0.136 ], [ 0.924, -1.292 ], [ -0.792, -0.176 ], [ 0.388, -0.052 ], ... ],
[ [ -0.084, 1.296 ], [ 1.536, -0.504 ], [ -1.992, 0.32 ], [ 0.096, -0.772 ], [ -0.132, 0.1 ], [ -1.504, -0.364 ], [ 1.236, 0.652 ], [ -1.812, 0.46 ], ... ],
[ [ -1.384, 0.268 ], [ -1.42, -0.116 ], [ 1.916, -0.192 ], [ -1.2, 1.704 ], [ -1.404, -1.676 ], [ -0.424, 1.484 ], [ 1.42, -0.564 ], [ 1.672, 1.84 ], ... ],
[ [ -1.852, -1.092 ], [ 1.776, -1.032 ], [ 1.44, -0.464 ], [ 1.492, -1.072 ], [ -0.412, 1.568 ], [ -0.072, 1.328 ], [ 0.216, -0.104 ], [ -0.104, -0.836 ], ... ],
[ [ -0.728, -1.136 ], [ 1.948, -0.548 ], [ -1.292, -1.992 ], [ 1.268, 0.136 ], [ -1.252, -0.52 ], [ 1.748, 1.312 ], [ 1.588, -0.108 ], [ -0.184, 1.052 ], ... ],
[ [ 0.172, -1.904 ], [ 0.62, 0.304 ], [ -1.608, -0.98 ], [ -0.212, -0.66 ], [ 0.884, -1.384 ], [ -1.516, 0.064 ], [ -1.088, -1.988 ], [ -1.592, -0.544 ], ... ],
[ [ 1.4, 1.524 ], [ 1.64, -0.824 ], [ -1.672, -1.212 ], [ -1.384, -1.616 ], [ 1.3, 0.76 ], [ 1.716, -1.624 ], [ 0.964, -1.24 ], [ -0.04, 0.14 ], ... ],
[ [ -1.476, -0.844 ], [ 0.68, -1.624 ], [ -0.08, -1.232 ], [ 0.296, 0.02 ], [ -0.328, 1.532 ], [ 1.512, 1.6 ], [ -1.764, 1.892 ], [ -1.848, -0.948 ], ... ],
...
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 24.61 seconds (0.353 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12363793449302
Reset training subject: 12363999245632
Constructing line search parameters: GD
th(0)=1.0139071520497231;dx=-1.1883222856279699E-6
New Minimum: 1.0139071520497231 > 1.0139045918909702
WOLFE (weak): th(2.154434690031884)=1.0139045918909702; dx=-1.1883206641835253E-6 evalInputDelta=2.560158752995534E-6
New Minimum: 1.0139045918909702 > 1.0139020317299265
WOLFE (weak): th(4.308869380063768)=1.0139020317299265; dx=-1.1883190428587548E-6 evalInputDelta=5.120319796603212E-6
New Minimum: 1.0139020317299265 > 1.0138917910621266
WOLFE (weak): th(12.926608140191302)=1.0138917910621266; dx=-1.1883125571386723E-6 evalInputDelta=1.536098759657989E-5
New Minimum: 1.0138917910621266 > 1.0138457091485473
WOLFE (weak): th(51.70643256076521)=1.0138457091485473; dx=-1.1882833717957202E-6 evalInputDelta=6.14429011758233E-5
New Minimum: 1.0138457091485473 > 1.0135999576040664
WOLFE (weak): th(258.53216280382605)=1.0135999576040664; dx=-1.1881277167714766E-6 evalInputDelta=3.0719444565674436E-4
New Minimum: 1.0135999576040664 > 1.0120647402945284
WOLFE (weak): th(1551.1929768229563)=1.0120647402945284; dx=-1.18715487261625E-6 evalInputDelta=0.0018424117551947283
New Minimum: 1.0120647402945284 > 1.001048298416067
WOLFE (weak): th(10858.350837760694)=1.001048298416067; dx=-1.18015039473822E-6 evalInputDelta=0.012858853633656242
New Minimum: 1.001048298416067 > 0.9135208540880031
WOLFE (weak): th(86866.80670208555)=0.9135208540880031; dx=-1.1229471573499283E-6 evalInputDelta=0.10038629796172005
New Minimum: 0.9135208540880031 > 0.31487190135002274
END: th(781801.26031877)=0.31487190135002274; dx=-5.999461303888193E-7 evalInputDelta=0.6990352506997004
Fitness changed from 1.0139071520497231 to 0.31487190135002274
Iteration 1 complete. Error: 0.31487190135002274 Total: 2.3218; Orientation: 0.0606; Line Search: 1.9087
th(0)=0.31487190135002274;dx=-3.1734542774023326E-7
New Minimum: 0.31487190135002274 > 0.034609190739974705
END: th(1684339.7559414052)=0.034609190739974705; dx=-1.5441006362012704E-8 evalInputDelta=0.28026271061004804
Fitness changed from 0.31487190135002274 to 0.034609190739974705
Iteration 2 complete. Error: 0.034609190739974705 Total: 0.3130; Orientation: 0.0497; Line Search: 0.2002
th(0)=0.034609190739974705;dx=-2.7561660237031638E-8
New Minimum: 0.034609190739974705 > 0.013520714918533287
WOLF (strong): th(3628800.0)=0.013520714918533287; dx=1.593882302674083E-8 evalInputDelta=0.021088475821441417
New Minimum: 0.013520714918533287 > 0.004333133622233944
END: th(1814400.0)=0.004333133622233944; dx=-5.811418604019776E-9 evalInputDelta=0.03027605711774076
Fitness changed from 0.034609190739974705 to 0.004333133622233944
Iteration 3 complete. Error: 0.004333133622233944 Total: 0.5588; Orientation: 0.0523; Line Search: 0.4424
th(0)=0.004333133622233944;dx=-3.2893581955536856E-9
New Minimum: 0.004333133622233944 > 0.0016422004733161214
WOLF (strong): th(3909006.30159385)=0.0016422004733161214; dx=1.912571901397968E-9 evalInputDelta=0.002690933148917823
New Minimum: 0.0016422004733161214 > 4.4586985555288694E-4
END: th(1954503.150796925)=4.4586985555288694E-4; dx=-6.883931454026932E-10 evalInputDelta=0.0038872637666810574
Fitness changed from 0.004333133622233944 to 4.4586985555288694E-4
Iteration 4 complete. Error: 4.4586985555288694E-4 Total: 0.5727; Orientation: 0.0772; Line Search: 0.4313
th(0)=4.4586985555288694E-4;dx=-3.480856946174995E-10
New Minimum: 4.4586985555288694E-4 > 2.8398403657389933E-4
WOLF (strong): th(4210849.389853513)=2.8398403657389933E-4; dx=2.711958301766665E-10 evalInputDelta=1.618858189789876E-4
New Minimum: 2.8398403657389933E-4 > 3.896429168275545E-5
END: th(2105424.6949267564)=3.896429168275545E-5; dx=-3.8444932572747574E-11 evalInputDelta=4.069055638701315E-4
Fitness changed from 4.4586985555288694E-4 to 3.896429168275545E-5
Iteration 5 complete. Error: 3.896429168275545E-5 Total: 0.4661; Orientation: 0.0510; Line Search: 0.3532
Low gradient: 5.964730631922537E-6
th(0)=3.896429168275545E-5;dx=-3.557801151139506E-11
Armijo: th(4536000.0)=6.573548250333363E-5; dx=4.738188761876774E-11 evalInputDelta=-2.6771190820578175E-5
New Minimum: 3.896429168275545E-5 > 5.3116245349125E-6
WOLF (strong): th(2268000.0)=5.3116245349125E-6; dx=5.901938018841891E-12 evalInputDelta=3.3652667147842954E-5
END: th(756000.0)=1.729378872316122E-5; dx=-2.1751361875968003E-11 evalInputDelta=2.167050295959423E-5
Fitness changed from 3.896429168275545E-5 to 5.3116245349125E-6
Iteration 6 complete. Error: 5.3116245349125E-6 Total: 0.5279; Orientation: 0.0543; Line Search: 0.4096
Low gradient: 2.580879284306071E-6
th(0)=5.3116245349125E-6;dx=-6.6609378801602246E-12
New Minimum: 5.3116245349125E-6 > 3.0111434574566133E-7
WOLF (strong): th(1628752.6256641042)=3.0111434574566133E-7; dx=5.083642650303343E-13 evalInputDelta=5.0105101891668385E-6
END: th(814376.3128320521)=1.346741915032088E-6; dx=-3.076286710722678E-12 evalInputDelta=3.964882619880412E-6
Fitness changed from 5.3116245349125E-6 to 3.0111434574566133E-7
Iteration 7 complete. Error: 3.0111434574566133E-7 Total: 0.5635; Orientation: 0.0586; Line Search: 0.3593
Low gradient: 5.23093575083203E-7
th(0)=3.0111434574566133E-7;dx=-2.7362688829332693E-13
New Minimum: 3.0111434574566133E-7 > 3.643550978118293E-8
END: th(1754520.5791056307)=3.643550978118293E-8; dx=-2.8083804275155118E-14 evalInputDelta=2.646788359644784E-7
Fitness changed from 3.0111434574566133E-7 to 3.643550978118293E-8
Iteration 8 complete. Error: 3.643550978118293E-8 Total: 0.3776; Orientation: 0.0559; Line Search: 0.2415
Low gradient: 1.70211469630005E-7
th(0)=3.643550978118293E-8;dx=-2.8971944393606136E-14
New Minimum: 3.643550978118293E-8 > 1.6750093122080464E-8
WOLF (strong): th(3780000.000000001)=1.6750093122080464E-8; dx=1.8556376689477212E-14 evalInputDelta=1.9685416659102463E-8
New Minimum: 1.6750093122080464E-8 > 4.135679033751763E-9
END: th(1890000.0000000005)=4.135679033751763E-9; dx=-5.207780727711791E-15 evalInputDelta=3.229983074743116E-8
Fitness changed from 3.643550978118293E-8 to 4.135679033751763E-9
Iteration 9 complete. Error: 4.135679033751763E-9 Total: 0.4941; Orientation: 0.0573; Line Search: 0.3582
Low gradient: 5.7203707472829345E-8
th(0)=4.135679033751763E-9;dx=-3.2722641486370354E-15
New Minimum: 4.135679033751763E-9 > 2.501028913302867E-9
WOLF (strong): th(4071881.5641602613)=2.501028913302867E-9; dx=2.4693671488300264E-15 evalInputDelta=1.634650120448896E-9
New Minimum: 2.501028913302867E-9 > 3.9594966647316537E-10
END: th(2035940.7820801307)=3.9594966647316537E-10; dx=-4.014480864972236E-16 evalInputDelta=3.739729367278598E-9
Fitness changed from 4.135679033751763E-9 to 3.9594966647316537E-10
Iteration 10 complete. Error: 3.9594966647316537E-10 Total: 0.4317; Orientation: 0.0516; Line Search: 0.3096
Low gradient: 1.864249535625106E-8
th(0)=3.9594966647316537E-10;dx=-3.475426331078427E-16
Armijo: th(4386301.447764077)=5.17496736271584E-10; dx=4.02963226408867E-16 evalInputDelta=-1.215470697984186E-10
New Minimum: 3.9594966647316537E-10 > 4.522994737139949E-11
WOLF (strong): th(2193150.7238820386)=4.522994737139949E-11; dx=2.7710920473954607E-17 evalInputDelta=3.5071971910176586E-10
END: th(731050.2412940129)=1.8759859260757366E-10; dx=-2.2245667423048855E-16 evalInputDelta=2.083510738655917E-10
Fitness changed from 3.9594966647316537E-10 to 4.522994737139949E-11
Iteration 11 complete. Error: 4.522994737139949E-11 Total: 0.5093; Orientation: 0.0525; Line Search: 0.3945
Low gradient: 7.237688440268153E-9
th(0)=4.522994737139949E-11;dx=-5.2384133958391304E-17
New Minimum: 4.522994737139949E-11 > 3.5482351597136864E-12
END: th(1575000.0000000005)=3.5482351597136864E-12; dx=-5.466348581758433E-19 evalInputDelta=4.1681712211685805E-11
Fitness changed from 4.522994737139949E-11 to 3.5482351597136864E-12
Iteration 12 complete. Error: 3.5482351597136864E-12 Total: 0.3172; Orientation: 0.0518; Line Search: 0.2025
Low gradient: 1.6518463782525775E-9
th(0)=3.5482351597136864E-12;dx=-2.7285964573461603E-18
New Minimum: 3.5482351597136864E-12 > 7.782800334965205E-13
WOLF (strong): th(3393234.636800218)=7.782800334965205E-13; dx=1.0951936638025622E-18 evalInputDelta=2.769955126217
...skipping 12763 bytes...
647440118E-19
Armijo: th(2289.3416374635663)=9.142802047945594E-18; dx=3.053719722435033E-24 evalInputDelta=-9.785803804823033E-20
New Minimum: 9.044944009897364E-18 > 9.000991965949036E-18
END: th(763.1138791545221)=9.000991965949036E-18; dx=-2.9875167748100585E-24 evalInputDelta=4.395204394832797E-20
Fitness changed from 9.044944009897364E-18 to 9.000991965949036E-18
Iteration 33 complete. Error: 9.000991965949036E-18 Total: 0.5000; Orientation: 0.0527; Line Search: 0.3845
Zero gradient: 3.096887434246625E-12
th(0)=9.000991965949036E-18;dx=-9.590711780394642E-24
Armijo: th(1644.0790136953012)=9.188743599276597E-18; dx=3.087860853713659E-24 evalInputDelta=-1.8775163332756142E-19
Armijo: th(822.0395068476506)=9.035484967672863E-18; dx=1.2329997042935183E-24 evalInputDelta=-3.449300172382713E-20
New Minimum: 9.000991965949036E-18 > 8.961283010850975E-18
END: th(274.0131689492169)=8.961283010850975E-18; dx=-5.5829107439237815E-24 evalInputDelta=3.970895509806049E-20
Fitness changed from 9.000991965949036E-18 to 8.961283010850975E-18
Iteration 34 complete. Error: 8.961283010850975E-18 Total: 0.7449; Orientation: 0.0517; Line Search: 0.6319
Zero gradient: 3.087906496804968E-12
th(0)=8.961283010850975E-18;dx=-9.535166533010328E-24
Armijo: th(590.3434767097604)=9.089482689670118E-18; dx=1.3918908320250742E-24 evalInputDelta=-1.281996788191426E-19
Armijo: th(295.1717383548802)=9.01553637576984E-18; dx=-1.3578848972446281E-24 evalInputDelta=-5.425336491886532E-20
New Minimum: 8.961283010850975E-18 > 8.951798980179108E-18
END: th(98.39057945162672)=8.951798980179108E-18; dx=-6.526266282689444E-24 evalInputDelta=9.484030671867197E-21
Fitness changed from 8.961283010850975E-18 to 8.951798980179108E-18
Iteration 35 complete. Error: 8.951798980179108E-18 Total: 0.6052; Orientation: 0.0562; Line Search: 0.4710
Zero gradient: 3.0872474946881805E-12
th(0)=8.951798980179108E-18;dx=-9.531097093458443E-24
Armijo: th(211.97607754292284)=9.02352962798975E-18; dx=-1.1549261052963655E-24 evalInputDelta=-7.173064781064189E-20
New Minimum: 8.951798980179108E-18 > 8.950429405114438E-18
END: th(105.98803877146142)=8.950429405114438E-18; dx=-3.667487140255662E-24 evalInputDelta=1.3695750646704479E-21
Fitness changed from 8.951798980179108E-18 to 8.950429405114438E-18
Iteration 36 complete. Error: 8.950429405114438E-18 Total: 0.4014; Orientation: 0.0504; Line Search: 0.2871
Zero gradient: 3.0851089598220353E-12
th(0)=8.950429405114438E-18;dx=-9.5178972939742E-24
Armijo: th(228.34430745768077)=9.075354799560785E-18; dx=3.9760082179563436E-25 evalInputDelta=-1.2492539444634705E-19
Armijo: th(114.17215372884039)=9.004583399691301E-18; dx=-1.0827935390756273E-24 evalInputDelta=-5.4153994576863415E-20
New Minimum: 8.950429405114438E-18 > 8.942200461053016E-18
END: th(38.05738457628013)=8.942200461053016E-18; dx=-6.462632552079985E-24 evalInputDelta=8.228944061421176E-21
Fitness changed from 8.950429405114438E-18 to 8.942200461053016E-18
Iteration 37 complete. Error: 8.942200461053016E-18 Total: 0.6018; Orientation: 0.0501; Line Search: 0.4910
Zero gradient: 3.0830128858850436E-12
th(0)=8.942200461053016E-18;dx=-9.504968454533221E-24
Armijo: th(81.99214954302228)=9.033735036163786E-18; dx=-1.0393482180625953E-24 evalInputDelta=-9.153457511076937E-20
Armijo: th(40.99607477151114)=8.948876021657378E-18; dx=-2.927535921367719E-24 evalInputDelta=-6.6755606043612936E-21
New Minimum: 8.942200461053016E-18 > 8.923788562291505E-18
END: th(13.66535825717038)=8.923788562291505E-18; dx=-7.087223136193877E-24 evalInputDelta=1.8411898761511222E-20
Fitness changed from 8.942200461053016E-18 to 8.923788562291505E-18
Iteration 38 complete. Error: 8.923788562291505E-18 Total: 0.6702; Orientation: 0.0509; Line Search: 0.5567
Zero gradient: 3.079505946862472E-12
th(0)=8.923788562291505E-18;dx=-9.483356876761327E-24
Armijo: th(29.441121880961514)=8.965131148594654E-18; dx=-2.744678178019886E-24 evalInputDelta=-4.13425863031487E-20
Armijo: th(14.720560940480757)=8.932531673959382E-18; dx=-4.6159489001076186E-24 evalInputDelta=-8.743111667876818E-21
Armijo: th(4.906853646826919)=8.936701221916138E-18; dx=-7.79602163267971E-24 evalInputDelta=-1.291265962463311E-20
Armijo: th(1.2267134117067298)=8.927277283944188E-18; dx=-9.107429355510109E-24 evalInputDelta=-3.488721652682928E-21
New Minimum: 8.923788562291505E-18 > 8.923778924937986E-18
WOLFE (weak): th(0.24534268234134596)=8.923778924937986E-18; dx=-9.38313747600284E-24 evalInputDelta=9.637353518750926E-24
Armijo: th(0.7360280470240379)=8.924337891389515E-18; dx=-9.257503278695876E-24 evalInputDelta=-5.493290980094042E-22
New Minimum: 8.923778924937986E-18 > 8.923778924937019E-18
WOLFE (weak): th(0.4906853646826919)=8.923778924937019E-18; dx=-9.330822586739862E-24 evalInputDelta=9.63735448633813E-24
New Minimum: 8.923778924937019E-18 > 8.923778924936145E-18
WOLFE (weak): th(0.6133567058533649)=8.923778924936145E-18; dx=-9.288933941977134E-24 evalInputDelta=9.637355359939953E-24
New Minimum: 8.923778924936145E-18 > 8.923730738172923E-18
WOLFE (weak): th(0.6746923764387014)=8.923730738172923E-18; dx=-9.275908702707057E-24 evalInputDelta=5.782411858260398E-23
Armijo: th(0.7053602117313696)=8.924337891389515E-18; dx=-9.263498225783872E-24 evalInputDelta=-5.493290980094042E-22
New Minimum: 8.923730738172923E-18 > 8.923721100820277E-18
WOLFE (weak): th(0.6900262940850355)=8.923721100820277E-18; dx=-9.267307798303715E-24 evalInputDelta=6.746147122775308E-23
WOLFE (weak): th(0.6976932529082025)=8.923721100820277E-18; dx=-9.26528472537137E-24 evalInputDelta=6.746147122775308E-23
WOLFE (weak): th(0.701526732319786)=8.923721100820277E-18; dx=-9.26528472537137E-24 evalInputDelta=6.746147122775308E-23
mu ~= nu (0.701526732319786): th(0.6900262940850355)=8.923721100820277E-18
Fitness changed from 8.923788562291505E-18 to 8.923721100820277E-18
Iteration 39 complete. Error: 8.923721100820277E-18 Total: 1.8455; Orientation: 0.0511; Line Search: 1.7320
Zero gradient: 3.0794943422500595E-12
th(0)=8.923721100820277E-18;dx=-9.483285403950125E-24
Armijo: th(1.5155230186083777)=8.929539050138295E-18; dx=-8.754873159725281E-24 evalInputDelta=-5.8179493180172866E-21
Armijo: th(0.7577615093041888)=8.927190547769468E-18; dx=-9.030532593971051E-24 evalInputDelta=-3.469446949190678E-21
Armijo: th(0.25258716976806295)=8.924369212784687E-18; dx=-9.346733375188207E-24 evalInputDelta=-6.481119644098307E-22
Armijo: th(0.06314679244201574)=8.924347528742158E-18; dx=-9.447645310019211E-24 evalInputDelta=-6.264279218807657E-22
Armijo: th(0.012629358488403148)=8.923730738172923E-18; dx=-9.472903928118342E-24 evalInputDelta=-9.637352645149103E-24
Armijo: th(0.002104893081400525)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
WOLFE (weak): th(3.006990116286464E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
Armijo: th(0.0012027960465145857)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
Armijo: th(7.51747529071616E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
Armijo: th(5.262232703501312E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
Armijo: th(4.1346114098938884E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
WOLFE (weak): th(3.5708007630901766E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
WOLFE (weak): th(3.8527060864920325E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
WOLFE (weak): th(3.9936587481929605E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
Armijo: th(4.0641350790434244E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
WOLFE (weak): th(4.0288969136181927E-4)=8.923721100820277E-18; dx=-9.483285403950125E-24 evalInputDelta=0.0
mu ~= nu (4.0288969136181927E-4): th(0.0)=8.923721100820277E-18
Fitness changed from 8.923721100820277E-18 to 8.923721100820277E-18
Static Iteration Total: 1.9999; Orientation: 0.0513; Line Search: 1.8877
Iteration 40 failed. Error: 8.923721100820277E-18
Previous Error: 0.0 -> 8.923721100820277E-18
Optimization terminated 40
Final threshold in iteration 40: 8.923721100820277E-18 (> 0.0) after 24.591s (< 30.000s)
Returns
8.923721100820277E-18
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 33.44 seconds (0.338 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12388404516695
Reset training subject: 12388465218884
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=1.0139071520497231}, derivative=-1.1883222856279699E-6}, evalInputDelta = 0.0
New Minimum: 1.0139071520497231 > 1.013907152031853
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=1.013907152031853}, derivative=-1.1883222856167365E-6}, evalInputDelta = -1.787014980436652E-11
New Minimum: 1.013907152031853 > 1.0139071514204243
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=1.0139071514204243}, derivative=-1.1883222852320243E-6}, evalInputDelta = -6.292988352640805E-10
New Minimum: 1.0139071514204243 > 1.0139071470970433
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=1.0139071470970433}, derivative=-1.1883222825123318E-6}, evalInputDelta = -4.952679821812467E-9
New Minimum: 1.0139071470970433 > 1.0139071182550006
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=1.0139071182550006}, derivative=-1.18832226433988E-6}, evalInputDelta = -3.379472257591942E-8
New Minimum: 1.0139071182550006 > 1.0139069170863824
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=1.0139069170863824}, derivative=-1.1883221368369301E-6}, evalInputDelta = -2.3496334078387804E-7
New Minimum: 1.0139069170863824 > 1.0139055073134011
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=1.0139055073134011}, derivative=-1.1883212440500314E-6}, evalInputDelta = -1.644736322026219E-6
New Minimum: 1.0139055073134011 > 1.0138956385316573
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=1.0138956385316573}, derivative=-1.188314993951859E-6}, evalInputDelta = -1.1513518065831008E-5
New Minimum: 1.0138956385316573 > 1.0138265590668551
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=1.0138265590668551}, derivative=-1.1882712433060219E-6}, evalInputDelta = -8.059298286799965E-5
New Minimum: 1.0138265590668551 > 1.0133430736167568
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=1.0133430736167568}, derivative=-1.1879649888672024E-6}, evalInputDelta = -5.640784329663528E-4
New Minimum: 1.0133430736167568 > 1.0099621647105994
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=1.0099621647105994}, derivative=-1.1858212071449346E-6}, evalInputDelta = -0.00394498733912374
New Minimum: 1.0099621647105994 > 0.9864667891185712
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.9864667891185712}, derivative=-1.1708147355490477E-6}, evalInputDelta = -0.02744036293115193
New Minimum: 0.9864667891185712 > 0.8303774715660033
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.8303774715660033}, derivative=-1.0657694348840067E-6}, evalInputDelta = -0.18352968048371987
New Minimum: 0.8303774715660033 > 0.14828951923542139
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.14828951923542139}, derivative=-3.3045233013699194E-7}, evalInputDelta = -0.8656176328143017
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=15.490000102255932}, derivative=4.8167674025393655E-6}, evalInputDelta = 14.476092950206208
F(613786.6638277846) = LineSearchPoint{point=PointSample{avg=0.42629401021779356}, derivative=-7.263923095695624E-7}, evalInputDelta = -0.5876131418319296
F(4296506.646794492) = LineSearchPoint{point=PointSample{avg=2.854670796555891}, derivative=2.0451875465313233E-6}, evalInputDelta = 1.840763644506168
F(330500.511291884) = LineSearchPoint{point=PointSample{avg=0.6622689772239726}, derivative=-9.395907601162593E-7}, evalInputDelta = -0.35163817482575055
F(2313503.579043188) = LineSearchPoint{point=PointSample{avg=0.2787637522024436}, derivative=5.527983932070282E-7}, evalInputDelta = -0.7351433998472796
0.2787637522024436 <= 1.0139071520497231
New Minimum: 0.14828951923542139 > 0.07574093468510762
F(1578976.0550639145) = LineSearchPoint{point=PointSample{avg=0.07574093468510762}, derivative=4.391101516627548E-17}, evalInputDelta = -0.9381662173646155
Right bracket at 1578976.0550639145
Converged to right
Fitness changed from 1.0139071520497231 to 0.07574093468510762
Iteration 1 complete. Error: 0.07574093468510762 Total: 3.0875; Orientation: 0.0505; Line Search: 2.8552
F(0.0) = LineSearchPoint{point=PointSample{avg=0.07574093468510762}, derivative=-5.894914121571374E-8}
New Minimum: 0.07574093468510762 > 0.013567011607255238
F(1578976.0550639145) = LineSearchPoint{point=PointSample{avg=0.013567011607255238}, derivative=-1.9803063907201495E-8}, evalInputDelta = -0.06217392307785238
F(1.1052832385447402E7) = LineSearchPoint{point=PointSample{avg=0.9385485663923239}, derivative=2.1507339995353747E-7}, evalInputDelta = 0.8628076317072162
F(850217.8758036463) = LineSearchPoint{point=PointSample{avg=0.03458204655751559}, derivative=-3.787048420211017E-8}, evalInputDelta = -0.04115888812759203
F(5951525.130625524) = LineSearchPoint{point=PointSample{avg=0.16397918860485805}, derivative=8.860145787626483E-8}, evalInputDelta = 0.08823825391975043
F(457809.6254327326) = LineSearchPoint{point=PointSample{avg=0.05135153059474329}, derivative=-4.75990951264443E-8}, evalInputDelta = -0.02438940409036433
F(3204667.378029128) = LineSearchPoint{point=PointSample{avg=0.014134473386893328}, derivative=2.0501181390456597E-8}, evalInputDelta = -0.06160646129821429
0.014134473386893328 <= 0.07574093468510762
New Minimum: 0.013567011607255238 > 0.005658002486672337
F(2377742.2623348627) = LineSearchPoint{point=PointSample{avg=0.005658002486672337}, derivative=-7.654026285045048E-19}, evalInputDelta = -0.07008293219843528
Left bracket at 2377742.2623348627
Converged to left
Fitness changed from 0.07574093468510762 to 0.005658002486672337
Iteration 2 complete. Error: 0.005658002486672337 Total: 1.4351; Orientation: 0.0561; Line Search: 1.3032
F(0.0) = LineSearchPoint{point=PointSample{avg=0.005658002486672337}, derivative=-6.631307840006529E-9}
New Minimum: 0.005658002486672337 > 0.0017624428789870168
F(2377742.2623348627) = LineSearchPoint{point=PointSample{avg=0.0017624428789870168}, derivative=3.3546199751622566E-9}, evalInputDelta = -0.00389555960768532
0.0017624428789870168 <= 0.005658002486672337
New Minimum: 0.0017624428789870168 > 4.226643402843828E-4
F(1578976.0548624122) = LineSearchPoint{point=PointSample{avg=4.226643402843828E-4}, derivative=-4.364124623611568E-18}, evalInputDelta = -0.005235338146387954
Left bracket at 1578976.0548624122
Converged to left
Fitness changed from 0.005658002486672337 to 4.226643402843828E-4
Iteration 3 complete. Error: 4.226643402843828E-4 Total: 0.5173; Orientation: 0.0498; Line Search: 0.4042
F(0.0) = LineSearchPoint{point=PointSample{avg=4.226643402843828E-4}, derivative=-3.289594979903706E-10}
New Minimum: 4.226643402843828E-4 > 7.570928524081875E-5
F(1578976.0548624122) = LineSearchPoint{point=PointSample{avg=7.570928524081875E-5}, derivative=-1.1050892103683986E-10}, evalInputDelta = -3.469550550435641E-4
F(1.1052832384036886E7) = LineSearchPoint{point=PointSample{avg=0.005237471787543089}, derivative=1.2001945409018026E-9}, evalInputDelta = 0.004814807447258707
F(850217.875695145) = LineSearchPoint{point=PointSample{avg=1.9298148307321435E-4}, derivative=-2.113322645569097E-10}, evalInputDelta = -2.296828
...skipping 34612 bytes...
rivative=-9.810753141933237E-24}, evalInputDelta = -2.0522709487390385E-29
F(26.636959088286105) = LineSearchPoint{point=PointSample{avg=9.249626961231461E-18}, derivative=-9.607013358402797E-24}, evalInputDelta = 1.8178455050382957E-21
F(2.048996852945085) = LineSearchPoint{point=PointSample{avg=9.247809115715126E-18}, derivative=-9.814348019937189E-24}, evalInputDelta = -1.129673468179777E-29
New Minimum: 9.247327248027457E-18 > 9.24732724802129E-18
F(14.342977970615596) = LineSearchPoint{point=PointSample{avg=9.24732724802129E-18}, derivative=-9.718224467018525E-24}, evalInputDelta = -4.818677051331035E-22
F(100.40084579430918) = LineSearchPoint{point=PointSample{avg=9.251718153435805E-18}, derivative=-9.129001849717681E-24}, evalInputDelta = 3.90903770938232E-21
F(7.723141984177629) = LineSearchPoint{point=PointSample{avg=9.247327248054146E-18}, derivative=-9.772720846073577E-24}, evalInputDelta = -4.818676722767387E-22
F(54.0619938892434) = LineSearchPoint{point=PointSample{avg=9.250947918382544E-18}, derivative=-9.436220712004413E-24}, evalInputDelta = 3.1388026561206454E-21
F(4.158614914557185) = LineSearchPoint{point=PointSample{avg=9.247346522777938E-18}, derivative=-9.809252627587628E-24}, evalInputDelta = -4.62592948485187E-22
F(29.110304401900294) = LineSearchPoint{point=PointSample{avg=9.249655873276075E-18}, derivative=-9.588545638747503E-24}, evalInputDelta = 1.8467575496524708E-21
F(2.239254184761561) = LineSearchPoint{point=PointSample{avg=9.247809115715126E-18}, derivative=-9.814348019937189E-24}, evalInputDelta = -1.129673468179777E-29
New Minimum: 9.24732724802129E-18 > 9.247326495095962E-18
F(15.674779293330927) = LineSearchPoint{point=PointSample{avg=9.247326495095962E-18}, derivative=-9.699716258826237E-24}, evalInputDelta = -4.826206304612411E-22
F(109.72345505331648) = LineSearchPoint{point=PointSample{avg=9.249087156115659E-18}, derivative=-9.0691699650455E-24}, evalInputDelta = 1.2780403892362482E-21
F(8.440265773332037) = LineSearchPoint{point=PointSample{avg=9.247327248051039E-18}, derivative=-9.770446409582567E-24}, evalInputDelta = -4.818676753844192E-22
F(59.08186041332426) = LineSearchPoint{point=PointSample{avg=9.250485325428926E-18}, derivative=-9.399337665356777E-24}, evalInputDelta = 2.6762097025032403E-21
F(4.544758493332635) = LineSearchPoint{point=PointSample{avg=9.247346522775864E-18}, derivative=-9.808578269942545E-24}, evalInputDelta = -4.625929505590284E-22
F(31.813309453328444) = LineSearchPoint{point=PointSample{avg=9.249617323852127E-18}, derivative=-9.579016224791992E-24}, evalInputDelta = 1.80820812570438E-21
F(2.4471776502560343) = LineSearchPoint{point=PointSample{avg=9.247809115713105E-18}, derivative=-9.81322409052985E-24}, evalInputDelta = -1.3318190751426613E-29
F(17.13024355179224) = LineSearchPoint{point=PointSample{avg=9.247362635160188E-18}, derivative=-9.681283063698632E-24}, evalInputDelta = -4.464805662344527E-22
F(119.91170486254569) = LineSearchPoint{point=PointSample{avg=9.248971507830598E-18}, derivative=-9.005829944817287E-24}, evalInputDelta = 1.1623921041754737E-21
F(9.2239772971189) = LineSearchPoint{point=PointSample{avg=9.24732724804798E-18}, derivative=-9.767846294770153E-24}, evalInputDelta = -4.81867678442796E-22
F(64.5678410798323) = LineSearchPoint{point=PointSample{avg=9.249868986581883E-18}, derivative=-9.367211399392458E-24}, evalInputDelta = 2.0598708554599218E-21
F(4.966757006140947) = LineSearchPoint{point=PointSample{avg=9.247327248068552E-18}, derivative=-9.803764391040887E-24}, evalInputDelta = -4.818676578707827E-22
F(34.76729904298663) = LineSearchPoint{point=PointSample{avg=9.250986881997623E-18}, derivative=-9.547780984725166E-24}, evalInputDelta = 3.1777662712004443E-21
F(2.674407618691279) = LineSearchPoint{point=PointSample{avg=9.247809115712068E-18}, derivative=-9.813224090529365E-24}, evalInputDelta = -1.43551114334847E-29
F(18.720853330838953) = LineSearchPoint{point=PointSample{avg=9.2473626351519E-18}, derivative=-9.663220695083403E-24}, evalInputDelta = -4.464805745236552E-22
F(131.04597331587266) = LineSearchPoint{point=PointSample{avg=9.249383504599608E-18}, derivative=-8.911908459256479E-24}, evalInputDelta = 1.5743888731852576E-21
F(10.080459485836359) = LineSearchPoint{point=PointSample{avg=9.24732724804285E-18}, derivative=-9.76016759504469E-24}, evalInputDelta = -4.818676835734733E-22
F(70.56321640085451) = LineSearchPoint{point=PointSample{avg=9.249842483832314E-18}, derivative=-9.306084663086865E-24}, evalInputDelta = 2.0333681058912403E-21
F(5.427939723142655) = LineSearchPoint{point=PointSample{avg=9.247327248066432E-18}, derivative=-9.802640461633542E-24}, evalInputDelta = -4.818676599908464E-22
F(37.995578061998586) = LineSearchPoint{point=PointSample{avg=9.250986881981194E-18}, derivative=-9.516114958056018E-24}, evalInputDelta = 3.1777662547714915E-21
F(2.9227367739998913) = LineSearchPoint{point=PointSample{avg=9.247809115710982E-18}, derivative=-9.811523119243877E-24}, evalInputDelta = -1.5441335922119102E-29
New Minimum: 9.247326495095962E-18 > 9.247246535160086E-18
F(20.45915741799924) = LineSearchPoint{point=PointSample{avg=9.247246535160086E-18}, derivative=-9.647945029167059E-24}, evalInputDelta = -5.625805663369327E-22
F(143.21410192599467) = LineSearchPoint{point=PointSample{avg=9.249499152766764E-18}, derivative=-8.801498624899218E-24}, evalInputDelta = 1.690037040340601E-21
F(11.016469378922666) = LineSearchPoint{point=PointSample{avg=9.247327248038705E-18}, derivative=-9.74935868333416E-24}, evalInputDelta = -4.818676877180746E-22
F(77.11528565245867) = LineSearchPoint{point=PointSample{avg=9.249842483797435E-18}, derivative=-9.274442965352465E-24}, evalInputDelta = 2.0333680710118787E-21
F(5.931945050189128) = LineSearchPoint{point=PointSample{avg=9.247327248064409E-18}, derivative=-9.80129174634435E-24}, evalInputDelta = -4.818676620138432E-22
F(41.5236153513239) = LineSearchPoint{point=PointSample{avg=9.250936474090906E-18}, derivative=-9.49238502373009E-24}, evalInputDelta = 3.127358364482673E-21
F(3.194124257794146) = LineSearchPoint{point=PointSample{avg=9.247809115709996E-18}, derivative=-9.811298333362505E-24}, evalInputDelta = -1.6427412053645367E-29
New Minimum: 9.247246535160086E-18 > 9.247246535149825E-18
F(22.35886980455902) = LineSearchPoint{point=PointSample{avg=9.247246535149825E-18}, derivative=-9.631006589958692E-24}, evalInputDelta = -5.625805765982875E-22
F(156.51208863191314) = LineSearchPoint{point=PointSample{avg=9.249503971375351E-18}, derivative=-8.712017400062112E-24}, evalInputDelta = 1.6948556489282193E-21
F(12.039391433224088) = LineSearchPoint{point=PointSample{avg=9.247327248032588E-18}, derivative=-9.748459539806267E-24}, evalInputDelta = -4.818676938348281E-22
F(84.27574003256862) = LineSearchPoint{point=PointSample{avg=9.249842483760483E-18}, derivative=-9.225355347000565E-24}, evalInputDelta = 2.0333680340602164E-21
F(6.482749233274509) = LineSearchPoint{point=PointSample{avg=9.24732724806135E-18}, derivative=-9.78767733164426E-24}, evalInputDelta = -4.818676650722199E-22
F(45.37924463292156) = LineSearchPoint{point=PointSample{avg=9.250943099751312E-18}, derivative=-9.460322749653765E-24}, evalInputDelta = 3.133984024888713E-21
F(3.490711125609351) = LineSearchPoint{point=PointSample{avg=9.247809115707923E-18}, derivative=-9.810977927815087E-24}, evalInputDelta = -1.8499712673806033E-29
F(24.434977879265457) = LineSearchPoint{point=PointSample{avg=9.24962696124271E-18}, derivative=-9.617694800000814E-24}, evalInputDelta = 1.8178455162872673E-21
F(1.8796136830204198) = LineSearchPoint{point=PointSample{avg=9.247809115716162E-18}, derivative=-9.814348019937668E-24}, evalInputDelta = -1.0261354743695193E-29
Loops = 52
F(1948.4036786288548) = LineSearchPoint{point=PointSample{avg=9.268163052549349E-18}, derivative=4.765116545743052E-24}, evalInputDelta = 2.035393682292619E-20
9.268163052549349E-18 > 9.247809115716162E-18
Fitness changed from 9.247809115726423E-18 to 9.247246535149825E-18
Iteration 29 complete. Error: 9.247246535149825E-18 Total: 5.7490; Orientation: 0.0508; Line Search: 5.6382
Final threshold in iteration 29: 9.247246535149825E-18 (> 0.0) after 33.441s (< 30.000s)
Returns
9.247246535149825E-18
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 31.82 seconds (0.297 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12421865859846
Reset training subject: 12421928450161
Adding measurement 107c4f44 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 1.0139071520497231 < 1.0139071520497231. Total: 1
th(0)=1.0139071520497231;dx=-1.1883222856279699E-6
Adding measurement 5267670b to history. Total: 1
New Minimum: 1.0139071520497231 > 1.0139045918909702
WOLFE (weak): th(2.154434690031884)=1.0139045918909702; dx=-1.1883206641835253E-6 evalInputDelta=2.560158752995534E-6
Adding measurement 6f732e98 to history. Total: 2
New Minimum: 1.0139045918909702 > 1.0139020317299265
WOLFE (weak): th(4.308869380063768)=1.0139020317299265; dx=-1.1883190428587548E-6 evalInputDelta=5.120319796603212E-6
Adding measurement 77ac0e1c to history. Total: 3
New Minimum: 1.0139020317299265 > 1.0138917910621266
WOLFE (weak): th(12.926608140191302)=1.0138917910621266; dx=-1.1883125571386723E-6 evalInputDelta=1.536098759657989E-5
Adding measurement 2a280dd8 to history. Total: 4
New Minimum: 1.0138917910621266 > 1.0138457091485473
WOLFE (weak): th(51.70643256076521)=1.0138457091485473; dx=-1.1882833717957202E-6 evalInputDelta=6.14429011758233E-5
Adding measurement 7f594685 to history. Total: 5
New Minimum: 1.0138457091485473 > 1.0135999576040664
WOLFE (weak): th(258.53216280382605)=1.0135999576040664; dx=-1.1881277167714766E-6 evalInputDelta=3.0719444565674436E-4
Adding measurement 460ee45b to history. Total: 6
New Minimum: 1.0135999576040664 > 1.0120647402945284
WOLFE (weak): th(1551.1929768229563)=1.0120647402945284; dx=-1.18715487261625E-6 evalInputDelta=0.0018424117551947283
Adding measurement 4d1236ae to history. Total: 7
New Minimum: 1.0120647402945284 > 1.001048298416067
WOLFE (weak): th(10858.350837760694)=1.001048298416067; dx=-1.18015039473822E-6 evalInputDelta=0.012858853633656242
Adding measurement 5e71597c to history. Total: 8
New Minimum: 1.001048298416067 > 0.9135208540880031
WOLFE (weak): th(86866.80670208555)=0.9135208540880031; dx=-1.1229471573499283E-6 evalInputDelta=0.10038629796172005
Adding measurement 2af85ce1 to history. Total: 9
New Minimum: 0.9135208540880031 > 0.31487190135002274
END: th(781801.26031877)=0.31487190135002274; dx=-5.999461303888193E-7 evalInputDelta=0.6990352506997004
Fitness changed from 1.0139071520497231 to 0.31487190135002274
Iteration 1 complete. Error: 0.31487190135002274 Total: 1.6132; Orientation: 0.0736; Line Search: 1.3559
Non-optimal measurement 0.31487190135002274 < 0.31487190135002274. Total: 10
Rejected: LBFGS Orientation magnitude: 1.020e+03, gradient 5.633e-04, dot -0.988; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664, 1.0138457091485473, 1.0138917910621266, 1.0139020317299265, 1.0139045918909702, 1.0139071520497231
Rejected: LBFGS Orientation magnitude: 1.020e+03, gradient 5.633e-04, dot -0.988; [61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00, 34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664, 1.0138457091485473, 1.0138917910621266, 1.0139020317299265, 1.0139045918909702
Rejected: LBFGS Orientation magnitude: 1.018e+03, gradient 5.633e-04, dot -0.988; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664, 1.0138457091485473, 1.0138917910621266, 1.0139020317299265
Rejected: LBFGS Orientation magnitude: 1.018e+03, gradient 5.633e-04, dot -0.988; [61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00, 34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664, 1.0138457091485473, 1.0138917910621266
Rejected: LBFGS Orientation magnitude: 1.018e+03, gradient 5.633e-04, dot -0.988; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664, 1.0138457091485473
Rejected: LBFGS Orientation magnitude: 1.018e+03, gradient 5.633e-04, dot -0.988; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284, 1.0135999576040664
Rejected: LBFGS Orientation magnitude: 1.018e+03, gradient 5.633e-04, dot -0.988; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.31487190135002274, 0.9135208540880031, 1.001048298416067, 1.0120647402945284
LBFGS Accumulation History: 3 points
Removed measurement 2af85ce1 to history. Total: 9
Removed measurement 5e71597c to history. Total: 8
Removed measurement 4d1236ae to history. Total: 7
Removed measurement 460ee45b to history. Total: 6
Removed measurement 7f594685 to history. Total: 5
Removed measurement 2a280dd8 to history. Total: 4
Removed measurement 77ac0e1c to history. Total: 3
Adding measurement 7fd4ec66 to history. Total: 3
th(0)=0.31487190135002274;dx=-3.1734542774023326E-7
Adding measurement 2c5274af to history. Total: 4
New Minimum: 0.31487190135002274 > 0.034609190739974705
END: th(1684339.7559414052)=0.034609190739974705; dx=-1.5441006362012704E-8 evalInputDelta=0.28026271061004804
Fitness changed from 0.31487190135002274 to 0.034609190739974705
Iteration 2 complete. Error: 0.034609190739974705 Total: 19.6645; Orientation: 19.3409; Line Search: 0.2547
Non-optimal measurement 0.034609190739974705 < 0.034609190739974705. Total: 5
Rejected: LBFGS Orientation magnitude: 2.850e+02, gradient 1.660e-04, dot -0.975; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.034609190739974705, 0.31487190135002274, 1.0139020317299265, 1.0139045918909702, 1.0139071520497231
Rejected: LBFGS Orientation magnitude: 2.850e+02, gradient 1.660e-04, dot -0.975; [61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00, 34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.034609190739974705, 0.31487190135002274, 1.0139020317299265, 1.0139045918909702
LBFGS Accumulation History: 3 points
Removed measurement 2c5274af to history. Total: 4
Removed measurement 7fd4ec66 to history. Total: 3
Adding measurement 34992953 to history. Total: 3
th(0)=0.034609190739974705;dx=-2.7561660237031638E-8
Adding measurement 7ed63624 to history. Total: 4
New Minimum: 0.034609190739974705 > 0.013520714918533287
WOLF (strong): th(3628800.0)=0.013520714918533287; dx=1.593882302674083E-8 evalInputDelta=0.021088475821441417
Adding measurement 1657e77b to history. Total: 5
New Minimum: 0.013520714918533287 > 0.004333133622233944
END: th(1814400.0)=0.004333133622233944; dx=-5.811418604019776E-9 evalInputDelta=0.03027605711774076
Fitness changed from 0.034609190739974705 to 0.004333133622233944
Iteration 3 complete. Error: 0.004333133622233944 Total: 4.0568; Orientation: 3.5397; Line Search: 0.4540
Non-optimal measurement 0.004333133622233944 < 0.004333133622233944. Total: 6
Rejected: LBFGS Orientation magnitude: 1.531e+02, gradient 5.735e-05, dot -0.982; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.004333133622233944, 0.013520714918533287, 0.034609190739974705, 1.0139020317299265, 1.0139045918909702, 1.0139071520497231
Rejected: LBFGS Orientation magnitude: 1.532e+02, gradient 5.735e-05, dot -0.982; [34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00, 61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.004333133622233944, 0.013520714918533287, 0.034609190739974705, 1.0139020317299265, 1.0139045918909702
Rejected: LBFGS Orientation magnitude: 1.533e+02, gradient 5.735e-05, dot -0.982; [61bae47b-a16f-475c-9cbc-138207c68a84 = 1.000/1.000e+00, 34942ed2-0fa6-490f-b8f7-952e2a996660 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.004333133622233944, 0.013520714918533287, 0.034609190739974705, 1.0139020317299265
LBFGS Accumulation History: 3 points
Removed measurement 1657e77b to history. Total: 5
Removed measurement 7ed63624 to history. Total: 4
Removed measurement 34992953 to history. Total: 3
Adding measurement 415667ff to history. Total: 3
th(0)=0.004333133622233944;dx=-3.2893581955536856E-9
Adding measurement 6fe9fbdf to history. Total: 4
New Minimum: 0.004333133622233944 > 0.0016422004733161214
WOLF (strong): th(3909006.30159385)=0.0016422004733161214; dx=1.912571901397968E-9 evalInputDelta=0.002690933148917823
Adding measurement 2981ef7 to history. Total: 5
New Minimum: 0.0016422004733161214 > 4.4586985555288694E-4
END: th(1954503.150796925)=4.4586985555288694E-4; dx=-6.883931454026932E-10 evalInputDelta=0.0038872637666810574
Fitness changed from 0.004333133622233944 to 4.4586985555288694E-4
Iteration 4 complete. Error: 4.4586985555288694E-4 Total: 6.4830; Orientation: 6.0026; Line Search: 0.4190
Final threshold in iteration 4: 4.4586985555288694E-4 (> 0.0) after 31.818s (< 30.000s)
Returns
4.4586985555288694E-4
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -1.5163656506623213, -0.09880303997967241 ], [ 0.2809677812384661, -1.1268524691346227 ], [ 1.4845334595779687, -1.864736526349465 ], [ -1.7045289803263384, 1.005775521997929 ], [ -1.1474319126569932, -0.879763536196307 ], [ -0.8348357162365071, 0.30601509094797946 ], [ -0.7674520120516002, -1.5943916485776215 ], [ -1.0621555819881912, -0.5068745567834622 ], ... ],
[ [ -1.3696675957989297, 0.6387610083627637 ], [ -1.0344965713241752, -1.1117870834152304 ], [ 1.6039396715626513, -0.8000339942803276 ], [ -1.5854413435543098, 1.0307248285934762 ], [ -1.8494671532203868, 0.9328073163539875 ], [ -1.4131212381489602, 0.7332559226261096 ], [ -0.22910318816219205, -1.145273846531554 ], [ 1.9089471155517646, 0.1695759410651377 ], ... ],
[ [ 0.41464874898441684, -0.1440021525755319 ], [ 0.6779990841405873, -2.005292082449435 ], [ -0.45680382507965694, -0.2618569676414685 ], [ -1.3140461106590233, -0.9047184352933318 ], [ -1.5959632361265403, -0.38143666662973097 ], [ 0.07432798814927719, 1.2475192536204778 ], [ -1.743273131140681, -1.2821488722298937 ], [ -1.5821142200852214, -1.0075349037057466 ], ... ],
[ [ 1.4310527925769547, 0.8768397034176921 ], [ 0.37786762946839114, 1.0764930446682954 ], [ -0.19662754069902785, -1.7531066525795347 ], [ -0.02812050490933185, 0.020405332310547086 ], [ -0.32309275745554344, 0.9313226218234157 ], [ -0.9549962615553398, 0.8278682402201468 ], [ -0.3682525020387779, -0.8475123750611008 ], [ -0.4800257319007215, -1.4590711045482203 ], ... ],
[ [ -0.4376007570316171, 1.5044466917344523 ], [ 1.5050197160577268, 0.8110798429042216 ], [ 0.821472390264471, 1.276074322566569 ], [ -1.5011462896284424, 0.039137013489738276 ], [ 0.72425898825789, 1.7859554208376431 ], [ -1.8614208540926007, 1.8686666458210934 ], [ 1.7999772127760343, -0.374236418406058 ], [ 1.296349130311224, -0.3258710372670848 ], ... ],
[ [ -1.5318240148926192, -0.13794781843121115 ], [ 1.5558489879927844, -0.4626139622674626 ], [ -1.0598534508621198, -0.2141801145050727 ], [ 0.6008811000823862, 1.431405940825016 ], [ -0.1382234824097717, -1.9602103679727403 ], [ 0.10089074476244553, -1.0862146740524137 ], [ 0.8681775053186653, -0.18650751234532556 ], [ -0.1804006096016165, 1.7217910795261633 ], ... ],
[ [ -1.1078174829432699, -2.0188321713833077 ], [ -0.5405497091719764, -0.9007569150251644 ], [ -0.2169301341831252, -1.5574020045332582 ], [ 0.21208109081543502, -0.3750884113726343 ], [ 1.8106583887778729, -1.343198511174065 ], [ 1.4432184117751565, -0.2548286913325795 ], [ -1.140319200242713, -0.5426344753706434 ], [ -1.4091741307988361, -1.2076907845184826 ], ... ],
[ [ -1.7689224032569426, -1.1587268581077361 ], [ 0.7683074500943463, 1.8424450580303562 ], [ -1.9423085831791007, -1.8557461161068651 ], [ -0.7067916218973288, 0.9749712452102824 ], [ 0.1607380680450646, -1.2639672031964289 ], [ 0.8339760641396146, -0.8321644962616641 ], [ -0.6097491619424729, 1.0419436826352788 ], [ -0.21161116351311085, -1.5709595712300677 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.02 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ -0.7394705414772034, -0.19047138094902039 ], [ 0.2835971415042877, -0.7663193345069885 ], [ 0.9750148057937622, -1.186552882194519 ], [ -0.9741856455802917, 0.5676805973052979 ], [ -0.4565165042877197, -0.7076290249824524 ], [ -0.45324844121932983, 0.14742371439933777 ], [ -0.17657403647899628, -1.177470326423645 ], [ -0.4619492292404175, -0.4397846460342407 ], ... ],
[ [ -0.7611165642738342, 0.3375592827796936 ], [ -0.37080153822898865, -0.8610106706619263 ], [ 0.8979583978652954, -0.43170860409736633 ], [ -0.9183117151260376, 0.5946720838546753 ], [ -1.0367350578308105, 0.5050077438354492 ], [ -0.7947648763656616, 0.40022945404052734 ], [ 0.032959870994091034, -0.8200199007987976 ], [ 0.9251319766044617, 0.2714189291000366 ], ... ],
[ [ 0.22409804165363312, -0.0676296055316925 ], [ 0.5929649472236633, -1.3494645357131958 ], [ -0.19305700063705444, -0.21984417736530304 ], [ -0.5359629392623901, -0.7384266257286072 ], [ -0.7427738904953003, -0.39468273520469666 ], [ -0.12281578779220581, 0.8792096972465515 ], [ -0.7005484104156494, -1.0369659662246704 ], [ -0.6557641625404358, -0.8318435549736023 ], ... ],
[ [ 0.5975666642189026, 0.728272020816803 ], [ 0.04963123798370361, 0.7837744951248169 ], [ 0.12687039375305176, -1.242904782295227 ], [ -0.016559652984142303, 0.012034091167151928 ], [ -0.2794632911682129, 0.6260784268379211 ], [ -0.5796452760696411, 0.503108024597168 ], [ -0.07417166233062744, -0.622718870639801 ], [ -0.051331646740436554, -1.0597518682479858 ], ... ],
[ [ -0.40961915254592896, 1.0181046724319458 ], [ 0.642671525478363, 0.6881574392318726 ], [ 0.2441128045320511, 0.95896977186203 ], [ -0.7495781183242798, -0.09269578754901886 ], [ 0.13063012063503265, 1.3081095218658447 ], [ -1.1624541282653809, 1.1591529846191406 ], [ 0.9406909346580505, -0.11796730011701584 ], [ 0.6847006678581238, -0.12440180033445358 ], ... ],
[ [ -0.7421273589134216, -0.21910938620567322 ], [ 0.8309156894683838, -0.1993618607521057 ], [ -0.49827221035957336, -0.2347143441438675 ], [ 0.1148170605301857, 1.0500545501708984 ], [ 0.1823480874300003, -1.3832050561904907 ], [ 0.18907728791236877, -0.7522789835929871 ], [ 0.4544890224933624, -0.06110105291008949 ], [ -0.3098679780960083, 1.190821647644043 ], ... ],
[ [ -0.2910669147968292, -1.5018079280853271 ], [ -0.1528157740831375, -0.6737737655639648 ], [ 0.0917501226067543, -1.1075358390808105 ], [ 0.153203547000885, -0.24559541046619415 ], [ 1.0700160264968872, -0.7953863143920898 ], [ 0.7484543919563293, -0.06292260438203812 ], [ -0.49614110589027405, -0.47106966376304626 ], [ -0.5443659424781799, -0.9581174850463867 ], ... ],
[ [ -0.7290684580802917, -0.9526225328445435 ], [ 0.14524750411510468, 1.3511760234832764 ], [ -0.7258495092391968, -1.454406976699829 ], [ -0.4753649830818176, 0.6259365081787109 ], [ 0.24151387810707092, -0.8719179630279541 ], [ 0.520169198513031, -0.5157970786094666 ], [ -0.4358043670654297, 0.6805806159973145 ], [ 0.09612369537353516, -1.116600513458252 ], ... ],
...
]
TrainingTester.java:432 executed in 0.13 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -17.049454011479586], [39.0, -0.5018660933587232]; valueStats=DoubleSummaryStatistics{count=72, sum=0.790421, min=0.000000, average=0.010978, max=0.314872}
Plotting 39 points for GD
Plotting 29 points for CjGD
Plotting 4 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -17.049454011479586], [30.353, -0.5018660933587232]; valueStats=DoubleSummaryStatistics{count=72, sum=0.790421, min=0.000000, average=0.010978, max=0.314872}
Plotting 39 points for GD
Plotting 29 points for CjGD
Plotting 4 points for LBFGS
Returns
In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:
TrainingTester.java:370 executed in 0.00 seconds (0.000 gc):
RefList<double[]> temp_18_0042 = network_target.state();
assert temp_18_0042 != null;
String temp_18_0041 = temp_18_0042.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
temp_18_0042.freeRef();
return temp_18_0041;
Returns
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 1.84 seconds (0.047 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12454077051436
Reset training subject: 12454114865564
Constructing line search parameters: GD
th(0)=0.9153342337458942;dx=-2.444156129895479
Armijo: th(2.154434690031884)=3.222855249884618; dx=4.5862694419041965 evalInputDelta=-2.3075210161387236
New Minimum: 0.9153342337458942 > 0.17577085502984013
WOLF (strong): th(1.077217345015942)=0.17577085502984013; dx=1.0710557661779188 evalInputDelta=0.739563378716054
END: th(0.3590724483386473)=0.24807447387610962; dx=-1.2724184467200685 evalInputDelta=0.6672597598697846
Fitness changed from 0.9153342337458942 to 0.17577085502984013
Iteration 1 complete. Error: 0.17577085502984013 Total: 0.3583; Orientation: 0.0004; Line Search: 0.2387
th(0)=0.17577085502984013;dx=-0.46934932023621284
New Minimum: 0.17577085502984013 > 1.8965043312499561E-4
WOLF (strong): th(0.7735981389354633)=1.8965043312499561E-4; dx=0.015415622449723162 evalInputDelta=0.17558120459671514
END: th(0.3867990694677316)=0.04110357127095593; dx=-0.22696693296556447 evalInputDelta=0.1346672837588842
Fitness changed from 0.17577085502984013 to 1.8965043312499561E-4
Iteration 2 complete. Error: 1.8965043312499561E-4 Total: 0.1666; Orientation: 0.0002; Line Search: 0.1246
th(0)=1.8965043312499561E-4;dx=-5.064179690029449E-4
New Minimum: 1.8965043312499561E-4 > 2.40511220341685E-6
WOLF (strong): th(0.8333333333333335)=2.40511220341685E-6; dx=5.702909189655309E-5 evalInputDelta=1.8724532092157875E-4
END: th(0.41666666666666674)=3.73353812778957E-5; dx=-2.246943572378066E-4 evalInputDelta=1.5231505184709992E-4
Fitness changed from 1.8965043312499561E-4 to 2.40511220341685E-6
Iteration 3 complete. Error: 2.40511220341685E-6 Total: 0.1559; Orientation: 0.0002; Line Search: 0.1158
th(0)=2.40511220341685E-6;dx=-6.422321701776301E-6
New Minimum: 2.40511220341685E-6 > 9.480299783643673E-8
WOLF (strong): th(0.8976811208466184)=9.480299783643673E-8; dx=1.275070240344006E-6 evalInputDelta=2.310309205580413E-6
END: th(0.4488405604233092)=3.8619953914748265E-7; dx=-2.5735349146771607E-6 evalInputDelta=2.018912664269367E-6
Fitness changed from 2.40511220341685E-6 to 9.480299783643673E-8
Iteration 4 complete. Error: 9.480299783643673E-8 Total: 0.1558; Orientation: 0.0002; Line Search: 0.1163
th(0)=9.480299783643673E-8;dx=-2.5315088715731445E-7
New Minimum: 9.480299783643673E-8 > 8.030361924513083E-9
WOLF (strong): th(0.9669976736693292)=8.030361924513083E-9; dx=7.367763746119349E-8 evalInputDelta=8.677263591192364E-8
END: th(0.4834988368346646)=1.1912499937883845E-8; dx=-8.97365969736164E-8 evalInputDelta=8.289049789855289E-8
Fitness changed from 9.480299783643673E-8 to 8.030361924513083E-9
Iteration 5 complete. Error: 8.030361924513083E-9 Total: 0.1499; Orientation: 0.0002; Line Search: 0.1109
th(0)=8.030361924513083E-9;dx=-2.144339814654125E-8
New Minimum: 8.030361924513083E-9 > 1.2261337607580438E-9
WOLF (strong): th(1.041666666666667)=1.2261337607580438E-9; dx=8.379041516827722E-9 evalInputDelta=6.804228163755039E-9
New Minimum: 1.2261337607580438E-9 > 7.451719636206965E-10
END: th(0.5208333333333335)=7.451719636206965E-10; dx=-6.532113376999092E-9 evalInputDelta=7.285189960892387E-9
Fitness changed from 8.030361924513083E-9 to 7.451719636206965E-10
Iteration 6 complete. Error: 7.451719636206965E-10 Total: 0.1521; Orientation: 0.0002; Line Search: 0.1151
th(0)=7.451719636206965E-10;dx=-1.989822967657326E-9
New Minimum: 7.451719636206965E-10 > 1.848838904248273E-10
WOLF (strong): th(1.122101401058273)=1.848838904248273E-10; dx=9.911380578661056E-10 evalInputDelta=5.602880731958692E-10
New Minimum: 1.848838904248273E-10 > 4.711184935983177E-11
END: th(0.5610507005291365)=4.711184935983177E-11; dx=-5.003170438213985E-10 evalInputDelta=6.980601142608647E-10
Fitness changed from 7.451719636206965E-10 to 4.711184935983177E-11
Iteration 7 complete. Error: 4.711184935983177E-11 Total: 0.1507; Orientation: 0.0002; Line Search: 0.1123
th(0)=4.711184935983177E-11;dx=-1.258000343165394E-10
New Minimum: 4.711184935983177E-11 > 1.78142227538419E-11
WOLF (strong): th(1.2087470920866614)=1.78142227538419E-11; dx=7.735442597151764E-11 evalInputDelta=2.929762660598987E-11
New Minimum: 1.78142227538419E-11 > 1.7131727868545399E-12
END: th(0.6043735460433307)=1.7131727868545399E-12; dx=-2.3977290907420386E-11 evalInputDelta=4.539867657297723E-11
Fitness changed from 4.711184935983177E-11 to 1.7131727868545399E-12
Iteration 8 complete. Error: 1.7131727868545399E-12 Total: 0.1479; Orientation: 0.0002; Line Search: 0.1110
Low gradient: 2.138429893945248E-6
th(0)=1.7131727868545399E-12;dx=-4.572881895519149E-12
New Minimum: 1.7131727868545399E-12 > 9.302442384833739E-13
WOLF (strong): th(1.3020833333333337)=9.302442384833739E-13; dx=3.3690970667080816E-12 evalInputDelta=7.82928548371166E-13
New Minimum: 9.302442384833739E-13 > 3.684091300078942E-14
END: th(0.6510416666666669)=3.684091300078942E-14; dx=-6.618474953267549E-13 evalInputDelta=1.6763318738537505E-12
Fitness changed from 1.7131727868545399E-12 to 3.684091300078942E-14
Iteration 9 complete. Error: 3.684091300078942E-14 Total: 0.1481; Orientation: 0.0003; Line Search: 0.1113
Low gradient: 3.1079468046709063E-7
th(0)=3.684091300078942E-14;dx=-9.659333340669501E-14
New Minimum: 3.684091300078942E-14 > 3.401904140394287E-14
WOLF (strong): th(1.4026267513228414)=3.401904140394287E-14; dx=9.260698356219791E-14 evalInputDelta=2.821871596846549E-15
New Minimum: 3.401904140394287E-14 > 0.0
END: th(0.7013133756614207)=0.0; dx=0.0 evalInputDelta=3.684091300078942E-14
Fitness changed from 3.684091300078942E-14 to 0.0
Iteration 10 complete. Error: 0.0 Total: 0.1471; Orientation: 0.0003; Line Search: 0.1102
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.1094; Orientation: 0.0003; Line Search: 0.0722
Iteration 11 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 11
Final threshold in iteration 11: 0.0 (> 0.0) after 1.841s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 1.33 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12455935289772
Reset training subject: 12455972267560
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.9153342337458942}, derivative=-2.4441570522103215}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.9153342337458942}, derivative=-2.444157184160053}, evalInputDelta = 0.0
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.9153342337458942}, derivative=-2.444156920482886}, evalInputDelta = 0.0
New Minimum: 0.9153342337458942 > 0.9153342171058041
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.9153342171058041}, derivative=-2.44415718416}, evalInputDelta = -1.664009008361944E-8
New Minimum: 0.9153342171058041 > 0.9153341840602545
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.9153341840602545}, derivative=-2.444157579120657}, evalInputDelta = -4.968563971363693E-8
New Minimum: 0.9153341840602545 > 0.9153336725746687
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.9153336725746687}, derivative=-2.4441561293403335}, evalInputDelta = -5.611712254882661E-7
New Minimum: 0.9153336725746687 > 0.9153301294603834
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.9153301294603834}, derivative=-2.4441517807766666}, evalInputDelta = -4.104285510786454E-6
New Minimum: 0.9153301294603834 > 0.9153054698477707
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.9153054698477707}, derivative=-2.444117649572033}, evalInputDelta = -2.8763898123451703E-5
New Minimum: 0.9153054698477707 > 0.9151329274549244
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.9151329274549244}, derivative=-2.443887555995582}, evalInputDelta = -2.01306290969816E-4
New Minimum: 0.9151329274549244 > 0.913925780618822
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.913925780618822}, derivative=-2.442274530303838}, evalInputDelta = -0.0014084531270721623
New Minimum: 0.913925780618822 > 0.9054977782176464
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.9054977782176464}, derivative=-2.430988887439855}, evalInputDelta = -0.009836455528247745
New Minimum: 0.9054977782176464 > 0.8475947656268332
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.8475947656268332}, derivative=-2.3519791088828734}, evalInputDelta = -0.06773946811906095
New Minimum: 0.8475947656268332 > 0.49583800461692856
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.49583800461692856}, derivative=-1.7989076292048596}, evalInputDelta = -0.4194962291289656
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.6581799492445035}, derivative=2.072581857068589}, evalInputDelta = -0.25715428450139066
0.6581799492445035 <= 0.9153342337458942
New Minimum: 0.49583800461692856 > 1.7761410132709113E-7
F(0.7489979032105168) = LineSearchPoint{point=PointSample{avg=1.7761410132709113E-7}, derivative=-3.050982919176057E-7}, evalInputDelta = -0.9153340561317929
Left bracket at 0.7489979032105168
Converged to left
Fitness changed from 0.9153342337458942 to 1.7761410132709113E-7
Iteration 1 complete. Error: 1.7761410132709113E-7 Total: 0.7022; Orientation: 0.0002; Line Search: 0.5920
F(0.0) = LineSearchPoint{point=PointSample{avg=1.7761410132709113E-7}, derivative=-4.7427055064352074E-7}
New Minimum: 1.7761410132709113E-7 > 2.7934946580901485E-14
F(0.7489979032105168) = LineSearchPoint{point=PointSample{avg=2.7934946580901485E-14}, derivative=-2.7825725515077616E-11}, evalInputDelta = -1.7761407339214456E-7
F(5.242985322473618) = LineSearchPoint{point=PointSample{avg=6.394104249959779E-6}, derivative=2.8456229913090175E-6}, evalInputDelta = 6.216490148632688E-6
F(0.4033065632672014) = LineSearchPoint{point=PointSample{avg=3.783827042373342E-8}, derivative=-2.1890363341595065E-7}, evalInputDelta = -1.397758309033577E-7
F(2.8231459428704095) = LineSearchPoint{point=PointSample{avg=1.3619985113137309E-6}, derivative=1.3133364005067558E-6}, evalInputDelta = 1.1843844099866398E-6
F(0.21716507252849304) = LineSearchPoint{point=PointSample{avg=8.955369468089483E-8}, derivative=-3.367667602481884E-7}, evalInputDelta = -8.80604066461963E-8
F(1.5201555076994513) = LineSearchPoint{point=PointSample{avg=1.8826559625890233E-7}, derivative=4.882842931500899E-7}, evalInputDelta = 1.0651494931811204E-8
F(0.11693503905380395) = LineSearchPoint{point=PointSample{avg=1.2650831757172042E-7}, derivative=-4.0026447692368985E-7}, evalInputDelta = -5.110578375537071E-8
F(0.8185452733766276) = LineSearchPoint{point=PointSample{avg=1.5299933205448758E-9}, derivative=4.401777934303348E-8}, evalInputDelta = -1.7608410800654627E-7
1.5299933205448758E-9 <= 1.7761410132709113E-7
F(0.749026931671519) = LineSearchPoint{point=PointSample{avg=3.688043519581106E-14}, derivative=1.9903028720964606E-11}, evalInputDelta = -1.7761406444665594E-7
Right bracket at 0.749026931671519
Converged to right
Fitness changed from 1.7761410132709113E-7 to 2.7934946580901485E-14
Iteration 2 complete. Error: 2.7934946580901485E-14 Total: 0.4457; Orientation: 0.0003; Line Search: 0.4081
Low gradient: 2.704006930955363E-7
F(0.0) = LineSearchPoint{point=PointSample{avg=2.7934946580901485E-14}, derivative=-7.311654951570284E-14}
New Minimum: 2.7934946580901485E-14 > 0.0
F(0.749026931671519) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -2.7934946580901485E-14
0.0 <= 2.7934946580901485E-14
Converged to right
Fitness changed from 2.7934946580901485E-14 to 0.0
Iteration 3 complete. Error: 0.0 Total: 0.1086; Orientation: 0.0003; Line Search: 0.0723
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0725; Orientation: 0.0003; Line Search: 0.0365
Iteration 4 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 4
Final threshold in iteration 4: 0.0 (> 0.0) after 1.329s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 2.02 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12457279985551
Reset training subject: 12457316471620
Adding measurement de4dba9 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.9153342337458942 < 0.9153342337458942. Total: 1
th(0)=0.9153342337458942;dx=-2.4441563933507746
Non-optimal measurement 3.2228557450827635 < 0.9153342337458942. Total: 1
Armijo: th(2.154434690031884)=3.2228557450827635; dx=4.586271007612906 evalInputDelta=-2.3075215113368692
Adding measurement 2741cccd to history. Total: 1
New Minimum: 0.9153342337458942 > 0.17577091285943436
WOLF (strong): th(1.077217345015942)=0.17577091285943436; dx=1.071056450014562 evalInputDelta=0.7395633208864598
Non-optimal measurement 0.24807443958449465 < 0.17577091285943436. Total: 2
END: th(0.3590724483386473)=0.24807443958449465; dx=-1.2724185810808701 evalInputDelta=0.6672597941613996
Fitness changed from 0.9153342337458942 to 0.17577091285943436
Iteration 1 complete. Error: 0.17577091285943436 Total: 0.2555; Orientation: 0.0007; Line Search: 0.1457
Non-optimal measurement 0.17577091285943436 < 0.17577091285943436. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.17577091285943436 < 0.17577091285943436. Total: 2
th(0)=0.17577091285943436;dx=-0.4693494070157871
Adding measurement 3f65e07 to history. Total: 2
New Minimum: 0.17577091285943436 > 1.8965017536418457E-4
WOLF (strong): th(0.7735981389354633)=1.8965017536418457E-4; dx=0.015415608481841293 evalInputDelta=0.17558126268407018
Non-optimal measurement 0.041103592265850704 < 1.8965017536418457E-4. Total: 3
END: th(0.3867990694677316)=0.041103592265850704; dx=-0.22696700422293026 evalInputDelta=0.13466732059358366
Fitness changed from 0.17577091285943436 to 1.8965017536418457E-4
Iteration 2 complete. Error: 1.8965017536418457E-4 Total: 0.1489; Orientation: 0.0005; Line Search: 0.1120
Non-optimal measurement 1.8965017536418457E-4 < 1.8965017536418457E-4. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 1.8965017536418457E-4 < 1.8965017536418457E-4. Total: 3
th(0)=1.8965017536418457E-4;dx=-5.064170494568283E-4
Adding measurement 7635f38b to history. Total: 3
New Minimum: 1.8965017536418457E-4 > 2.40511220341685E-6
WOLF (strong): th(0.8333333333333335)=2.40511220341685E-6; dx=5.7029041477240187E-5 evalInputDelta=1.872450631607677E-4
Non-optimal measurement 3.7335537350805705E-5 < 2.40511220341685E-6. Total: 4
END: th(0.41666666666666674)=3.7335537350805705E-5; dx=-2.2469462201483672E-4 evalInputDelta=1.5231463801337886E-4
Fitness changed from 1.8965017536418457E-4 to 2.40511220341685E-6
Iteration 3 complete. Error: 2.40511220341685E-6 Total: 0.1476; Orientation: 0.0006; Line Search: 0.1105
Non-optimal measurement 2.40511220341685E-6 < 2.40511220341685E-6. Total: 4
Rejected: LBFGS Orientation magnitude: 1.898e-03, gradient 2.534e-03, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 2.40511220341685E-6, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
LBFGS Accumulation History: 3 points
Removed measurement 7635f38b to history. Total: 3
Adding measurement 38686932 to history. Total: 3
th(0)=2.40511220341685E-6;dx=-6.4223210961172545E-6
Adding measurement 265aaa to history. Total: 4
New Minimum: 2.40511220341685E-6 > 9.48081682148249E-8
WOLF (strong): th(0.8976811208466184)=9.48081682148249E-8; dx=1.2751056050961297E-6 evalInputDelta=2.310304035202025E-6
Non-optimal measurement 3.8619953914748265E-7 < 9.48081682148249E-8. Total: 5
END: th(0.4488405604233092)=3.8619953914748265E-7; dx=-2.5735339011392353E-6 evalInputDelta=2.018912664269367E-6
Fitness changed from 2.40511220341685E-6 to 9.48081682148249E-8
Iteration 4 complete. Error: 9.48081682148249E-8 Total: 0.1557; Orientation: 0.0064; Line Search: 0.1125
Non-optimal measurement 9.48081682148249E-8 < 9.48081682148249E-8. Total: 5
Rejected: LBFGS Orientation magnitude: 3.769e-04, gradient 5.032e-04, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 9.48081682148249E-8, 2.40511220341685E-6, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 3.769e-04, gradient 5.032e-04, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 9.48081682148249E-8, 2.40511220341685E-6, 1.8965017536418457E-4, 0.17577091285943436
LBFGS Accumulation History: 3 points
Removed measurement 265aaa to history. Total: 4
Removed measurement 38686932 to history. Total: 3
Adding measurement 310c133a to history. Total: 3
th(0)=9.48081682148249E-8;dx=-2.5316498540993774E-7
Adding measurement 17ba4164 to history. Total: 4
New Minimum: 9.48081682148249E-8 > 8.031867964989193E-9
WOLF (strong): th(0.9669976736693292)=8.031867964989193E-9; dx=7.368661810155605E-8 evalInputDelta=8.677630024983571E-8
Non-optimal measurement 1.1912499937883845E-8 < 8.031867964989193E-9. Total: 5
END: th(0.4834988368346646)=1.1912499937883845E-8; dx=-8.973909407028267E-8 evalInputDelta=8.289566827694107E-8
Fitness changed from 9.48081682148249E-8 to 8.031867964989193E-9
Iteration 5 complete. Error: 8.031867964989193E-9 Total: 0.1623; Orientation: 0.0121; Line Search: 0.1131
Non-optimal measurement 8.031867964989193E-9 < 8.031867964989193E-9. Total: 5
Rejected: LBFGS Orientation magnitude: 1.097e-04, gradient 1.464e-04, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 8.031867964989193E-9, 9.48081682148249E-8, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 1.097e-04, gradient 1.464e-04, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 8.031867964989193E-9, 9.48081682148249E-8, 1.8965017536418457E-4, 0.17577091285943436
LBFGS Accumulation History: 3 points
Removed measurement 17ba4164 to history. Total: 4
Removed measurement 310c133a to history. Total: 3
Adding measurement 79eebeab to history. Total: 3
th(0)=8.031867964989193E-9;dx=-2.1447426685032632E-8
Adding measurement 47492f4f to history. Total: 4
New Minimum: 8.031867964989193E-9 > 1.2267237989004582E-9
WOLF (strong): th(1.041666666666667)=1.2267237989004582E-9; dx=8.381846984867154E-9 evalInputDelta=6.805144166088735E-9
Adding measurement 3c38837f to history. Total: 5
New Minimum: 1.2267237989004582E-9 > 7.451719636206965E-10
END: th(0.5208333333333335)=7.451719636206965E-10; dx=-6.532729221041792E-9 evalInputDelta=7.286696001368497E-9
Fitness changed from 8.031867964989193E-9 to 7.451719636206965E-10
Iteration 6 complete. Error: 7.451719636206965E-10 Total: 0.1612; Orientation: 0.0121; Line Search: 0.1117
Non-optimal measurement 7.451719636206965E-10 < 7.451719636206965E-10. Total: 6
Rejected: LBFGS Orientation magnitude: 3.341e-05, gradient 4.461e-05, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 7.451719636206965E-10, 1.2267237989004582E-9, 8.031867964989193E-9, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 3.341e-05, gradient 4.461e-05, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 7.451719636206965E-10, 1.2267237989004582E-9, 8.031867964989193E-9, 1.8965017536418457E-4, 0.17577091285943436
Rejected: LBFGS Orientation magnitude: 3.341e-05, gradient 4.461e-05, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 7.451719636206965E-10, 1.2267237989004582E-9, 8.031867964989193E-9, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 3c38837f to history. Total: 5
Removed measurement 47492f4f to history. Total: 4
Removed measurement 79eebeab to history. Total: 3
Adding measurement 470b3cc9 to history. Total: 3
th(0)=7.451719636206965E-10;dx=-1.9898236301769156E-9
Adding measurement 5f890df7 to history. Total: 4
New Minimum: 7.451719636206965E-10 >
...skipping 1359 bytes...
cfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 4.722757620569172E-11, 1.848838904248273E-10, 7.451719636206965E-10, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 298fd05c to history. Total: 5
Removed measurement 5f890df7 to history. Total: 4
Removed measurement 470b3cc9 to history. Total: 3
Adding measurement 2b81ece7 to history. Total: 3
th(0)=4.722757620569172E-11;dx=-1.2610906425459523E-10
Adding measurement 4b9d449b to history. Total: 4
New Minimum: 4.722757620569172E-11 > 1.7884989179534926E-11
WOLF (strong): th(1.2087470920866614)=1.7884989179534926E-11; dx=7.760326173798611E-11 evalInputDelta=2.9342587026156795E-11
Adding measurement 6f9a3e65 to history. Total: 5
New Minimum: 1.7884989179534926E-11 > 1.6907670528612767E-12
END: th(0.6043735460433307)=1.6907670528612767E-12; dx=-2.3852996385819945E-11 evalInputDelta=4.5536809152830444E-11
Fitness changed from 4.722757620569172E-11 to 1.6907670528612767E-12
Iteration 8 complete. Error: 1.6907670528612767E-12 Total: 0.1755; Orientation: 0.0222; Line Search: 0.1172
Non-optimal measurement 1.6907670528612767E-12 < 1.6907670528612767E-12. Total: 6
Rejected: LBFGS Orientation magnitude: 1.591e-06, gradient 2.124e-06, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.6907670528612767E-12, 1.7884989179534926E-11, 4.722757620569172E-11, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 1.591e-06, gradient 2.124e-06, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.6907670528612767E-12, 1.7884989179534926E-11, 4.722757620569172E-11, 1.8965017536418457E-4, 0.17577091285943436
Rejected: LBFGS Orientation magnitude: 1.591e-06, gradient 2.124e-06, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.6907670528612767E-12, 1.7884989179534926E-11, 4.722757620569172E-11, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 6f9a3e65 to history. Total: 5
Removed measurement 4b9d449b to history. Total: 4
Removed measurement 2b81ece7 to history. Total: 3
Adding measurement 380ff9df to history. Total: 3
th(0)=1.6907670528612767E-12;dx=-4.513075759094734E-12
Adding measurement 4cb1402e to history. Total: 4
New Minimum: 1.6907670528612767E-12 > 9.1375376042068E-13
WOLF (strong): th(1.3020833333333337)=9.1375376042068E-13; dx=3.3171723552593834E-12 evalInputDelta=7.770132924405967E-13
Adding measurement 26bff119 to history. Total: 5
New Minimum: 9.1375376042068E-13 > 3.684091300078942E-14
END: th(0.6510416666666669)=3.684091300078942E-14; dx=-6.578888988641293E-13 evalInputDelta=1.6539261398604873E-12
Fitness changed from 1.6907670528612767E-12 to 3.684091300078942E-14
Iteration 9 complete. Error: 3.684091300078942E-14 Total: 0.1783; Orientation: 0.0252; Line Search: 0.1165
Non-optimal measurement 3.684091300078942E-14 < 3.684091300078942E-14. Total: 6
Rejected: LBFGS Orientation magnitude: 2.328e-07, gradient 3.108e-07, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.684091300078942E-14, 9.1375376042068E-13, 1.6907670528612767E-12, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 2.328e-07, gradient 3.108e-07, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.684091300078942E-14, 9.1375376042068E-13, 1.6907670528612767E-12, 1.8965017536418457E-4, 0.17577091285943436
Rejected: LBFGS Orientation magnitude: 2.328e-07, gradient 3.108e-07, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.684091300078942E-14, 9.1375376042068E-13, 1.6907670528612767E-12, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 26bff119 to history. Total: 5
Removed measurement 4cb1402e to history. Total: 4
Removed measurement 380ff9df to history. Total: 3
Adding measurement 4d156210 to history. Total: 3
th(0)=3.684091300078942E-14;dx=-9.659331868531366E-14
Adding measurement 5a2805aa to history. Total: 4
New Minimum: 3.684091300078942E-14 > 3.401904140394287E-14
WOLF (strong): th(1.4026267513228414)=3.401904140394287E-14; dx=9.260695216404916E-14 evalInputDelta=2.821871596846549E-15
Adding measurement 70a2ed92 to history. Total: 5
New Minimum: 3.401904140394287E-14 > 1.702869631582857E-16
END: th(0.7013133756614207)=1.702869631582857E-16; dx=-3.963785094340455E-15 evalInputDelta=3.6670626037631137E-14
Fitness changed from 3.684091300078942E-14 to 1.702869631582857E-16
Iteration 10 complete. Error: 1.702869631582857E-16 Total: 0.1698; Orientation: 0.0194; Line Search: 0.1134
Non-optimal measurement 1.702869631582857E-16 < 1.702869631582857E-16. Total: 6
Rejected: LBFGS Orientation magnitude: 1.491e-08, gradient 1.994e-08, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.702869631582857E-16, 3.401904140394287E-14, 3.684091300078942E-14, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 1.491e-08, gradient 1.994e-08, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.702869631582857E-16, 3.401904140394287E-14, 3.684091300078942E-14, 1.8965017536418457E-4, 0.17577091285943436
Rejected: LBFGS Orientation magnitude: 1.491e-08, gradient 1.994e-08, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.702869631582857E-16, 3.401904140394287E-14, 3.684091300078942E-14, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 70a2ed92 to history. Total: 5
Removed measurement 5a2805aa to history. Total: 4
Removed measurement 4d156210 to history. Total: 3
Adding measurement 6cec9d2d to history. Total: 3
th(0)=1.702869631582857E-16;dx=-3.9762564393877526E-16
Adding measurement 1eabd506 to history. Total: 4
New Minimum: 1.702869631582857E-16 > 1.6944693840175674E-16
WOLF (strong): th(1.510933865108327)=1.6944693840175674E-16; dx=3.9731803174739596E-16 evalInputDelta=8.400247565289534E-19
Adding measurement 2120e9d4 to history. Total: 5
New Minimum: 1.6944693840175674E-16 > 0.0
END: th(0.7554669325541635)=0.0; dx=0.0 evalInputDelta=1.702869631582857E-16
Fitness changed from 1.702869631582857E-16 to 0.0
Iteration 11 complete. Error: 0.0 Total: 0.1670; Orientation: 0.0195; Line Search: 0.1113
Non-optimal measurement 0.0 < 0.0. Total: 6
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.6944693840175674E-16, 1.702869631582857E-16, 1.8965017536418457E-4, 0.17577091285943436, 0.9153342337458942
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.6944693840175674E-16, 1.702869631582857E-16, 1.8965017536418457E-4, 0.17577091285943436
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 1.6944693840175674E-16, 1.702869631582857E-16, 1.8965017536418457E-4
LBFGS Accumulation History: 3 points
Removed measurement 2120e9d4 to history. Total: 5
Removed measurement 1eabd506 to history. Total: 4
Removed measurement 6cec9d2d to history. Total: 3
Adding measurement 7febec7a to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.1309; Orientation: 0.0187; Line Search: 0.0760
Iteration 12 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 12
Final threshold in iteration 12: 0.0 (> 0.0) after 2.023s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.01 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -15.768818599513276], [11.0, -0.7550529917753773]; valueStats=DoubleSummaryStatistics{count=21, sum=0.351926, min=0.000000, average=0.016758, max=0.175771}
Plotting 10 points for GD
Plotting 3 points for CjGD
Plotting 11 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -15.768818599513276], [1.636, -0.7550529917753773]; valueStats=DoubleSummaryStatistics{count=21, sum=0.351926, min=0.000000, average=0.016758, max=0.175771}
Plotting 10 points for GD
Plotting 3 points for CjGD
Plotting 11 points for LBFGS
Returns
In this apply, attempt to train a network to emulate a randomized network given an example input/output. The target state is:
TrainingTester.java:279 executed in 0.00 seconds (0.000 gc):
RefList<double[]> temp_18_0037 = temp_18_0035.state();
assert temp_18_0037 != null;
String temp_18_0036 = temp_18_0037.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b).orElse("");
temp_18_0037.freeRef();
return temp_18_0036;
Returns
We simultaneously regress this target input:
TrainingTester.java:287 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(x -> {
RefStream<Tensor> temp_18_0006 = RefArrays.stream(RefUtil.addRef(x));
if (null != x)
RefUtil.freeRef(x);
return temp_18_0006;
}).map(x -> {
String temp_18_0007 = x.prettyPrint();
x.freeRef();
return temp_18_0007;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -0.848, 1.776 ], [ -0.396, 0.596 ], [ -0.128, 1.144 ], [ 1.04, -0.192 ], [ -0.984, -0.952 ], [ 1.94, 0.8 ], [ 0.484, 1.5 ], [ -1.892, 0.888 ], ... ],
[ [ -0.02, -1.88 ], [ 1.736, -0.144 ], [ 1.8, 0.628 ], [ 1.356, 1.592 ], [ -0.492, 1.064 ], [ -1.28, 1.036 ], [ -1.776, 1.972 ], [ -0.172, -0.052 ], ... ],
[ [ -0.108, -0.992 ], [ 0.532, 0.352 ], [ -0.128, -1.456 ], [ -0.772, 0.86 ], [ 0.552, 1.024 ], [ 1.804, 0.552 ], [ 1.132, -0.232 ], [ 0.852, 1.6 ], ... ],
[ [ 0.588, 1.448 ], [ 0.524, 0.04 ], [ 0.716, 1.744 ], [ -0.096, 0.768 ], [ -0.672, -1.356 ], [ 1.54, -0.76 ], [ 1.636, 0.112 ], [ 1.136, 1.32 ], ... ],
[ [ 1.644, -1.42 ], [ -0.472, 0.556 ], [ 1.248, -1.288 ], [ 1.708, -0.48 ], [ 1.964, 1.776 ], [ 0.04, 1.06 ], [ 0.784, -0.044 ], [ -1.028, -0.316 ], ... ],
[ [ 1.928, 1.892 ], [ 0.392, -0.848 ], [ -0.416, 0.176 ], [ 0.468, -1.72 ], [ 1.284, -0.644 ], [ -0.896, 0.436 ], [ -0.016, -0.368 ], [ -0.148, -0.444 ], ... ],
[ [ -1.536, 1.328 ], [ 0.456, 0.692 ], [ 1.636, 0.0 ], [ -1.108, -0.696 ], [ -1.64, 1.588 ], [ -1.168, -1.864 ], [ -1.576, 0.9 ], [ 1.532, 0.092 ], ... ],
[ [ 1.696, -0.596 ], [ -1.464, 0.064 ], [ 1.052, 1.42 ], [ 0.452, 0.096 ], [ -0.108, -0.588 ], [ 1.06, 0.42 ], [ 0.52, 1.524 ], [ 0.028, 1.228 ], ... ],
...
]
[
[ [ -1.9, -1.804 ], [ 1.436, 0.244 ], [ -1.892, -0.912 ], [ -0.84, -1.872 ], [ 1.248, 1.024 ], [ -0.684, 0.204 ], [ 0.704, -0.176 ], [ 1.056, 0.56 ], ... ],
[ [ 0.876, 0.656 ], [ -1.864, 1.248 ], [ -0.252, -1.412 ], [ -0.628, 0.308 ], [ 0.436, -0.364 ], [ -1.124, 0.412 ], [ -0.536, 0.492 ], [ 0.672, 1.712 ], ... ],
[ [ 0.532, 0.576 ], [ -0.524, -0.68 ], [ -1.516, -1.292 ], [ -1.312, -0.572 ], [ -0.26, -1.352 ], [ 0.56, -0.288 ], [ -0.736, -1.332 ], [ 1.956, -1.652 ], ... ],
[ [ 0.66, -0.048 ], [ 0.192, -0.392 ], [ 0.82, 0.956 ], [ 0.4, -0.528 ], [ -0.528, 0.468 ], [ 1.036, 1.612 ], [ -0.512, -1.636 ], [ -0.9, -1.928 ], ... ],
[ [ 0.848, 0.704 ], [ 1.664, 1.16 ], [ -1.936, 0.572 ], [ -0.076, -0.6 ], [ -1.344, -1.12 ], [ -0.244, 0.572 ], [ -1.668, -1.52 ], [ 0.932, 1.124 ], ... ],
[ [ -0.176, 0.724 ], [ -0.18, -0.496 ], [ -1.608, 1.612 ], [ 1.644, -1.524 ], [ -0.864, -0.916 ], [ 0.332, -0.844 ], [ 1.708, -1.656 ], [ 1.572, -0.44 ], ... ],
[ [ -0.916, 1.324 ], [ 0.552, 0.58 ], [ 0.14, 1.468 ], [ 0.396, -1.796 ], [ 0.076, 0.836 ], [ -1.204, -0.316 ], [ -1.212, 0.552 ], [ -0.008, -1.2 ], ... ],
[ [ -0.644, -0.132 ], [ 1.128, -1.096 ], [ -0.816, 1.5 ], [ -0.368, 1.42 ], [ -1.86, 1.32 ], [ 1.964, 1.612 ], [ 0.044, 0.928 ], [ 0.088, 0.932 ], ... ],
...
]
Which produces the following output:
TrainingTester.java:308 executed in 0.01 seconds (0.000 gc):
return RefStream.of(RefUtil.addRef(output_target)).map(x -> {
String temp_18_0008 = x.prettyPrint();
x.freeRef();
return temp_18_0008;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.1753599643707275, -0.6479359865188599 ], [ 0.38552001118659973, -0.27270400524139404 ], [ 0.7905600070953369, -0.2099200189113617 ], [ -0.051200006157159805, 0.5404160022735596 ], [ -0.7451199889183044, -0.3662079870700836 ], [ 0.7152000069618225, 0.8598400354385376 ], [ 1.0887199640274048, 0.048063986003398895 ], [ 0.4702399969100952, -1.0520960092544556 ], ... ],
[ [ -1.3176000118255615, 0.23072001338005066 ], [ 0.03808000683784485, 0.8794879913330078 ], [ 0.5835999846458435, 0.8124159574508667 ], [ 1.2228800058364868, 0.4687999486923218 ], [ 0.7054399847984314, -0.38022398948669434 ], [ 0.6227999925613403, -0.7674879431724548 ], [ 1.238319993019104, -1.1333119869232178 ], [ -0.050160001963377, -0.07865600287914276 ], ... ],
[ [ -0.7030400037765503, 0.07340800017118454 ], [ 0.28895998001098633, 0.21881599724292755 ], [ -1.029439926147461, 0.1228800043463707 ], [ 0.5402399897575378, -0.49299201369285583 ], [ 0.760960042476654, 0.14271998405456543 ], [ 0.5307199954986572, 0.8241279721260071 ], [ -0.07183999568223953, 0.5911679863929749 ], [ 1.1881599426269531, 0.21779197454452515 ], ... ],
[ [ 1.0606399774551392, 0.10630399733781815 ], [ 0.06991999596357346, 0.2547839879989624 ], [ 1.2780799865722656, 0.1319040060043335 ], [ 0.5299199819564819, -0.1459200084209442 ], [ -1.002959966659546, -0.1597440093755722 ], [ -0.40880000591278076, 0.8611199855804443 ], [ 0.20927999913692474, 0.7971200346946716 ], [ 1.0148800611495972, 0.39449599385261536 ], ... ],
[ [ -0.8624799251556396, 0.9971840381622314 ], [ 0.3514399826526642, -0.30527999997138977 ], [ -0.8017599582672119, 0.7838720083236694 ], [ -0.1993599832057953, 0.9086079597473145 ], [ 1.4003199338912964, 0.7468159794807434 ], [ 0.7451999187469482, -0.11584000289440155 ], [ 0.031919993460178375, 0.39449599385261536 ], [ -0.30344000458717346, -0.4694399833679199 ], ... ],
[ [ 1.4786399602890015, 0.7141119837760925 ], [ -0.5622400045394897, 0.30297598242759705 ], [ 0.08991999924182892, -0.22886401414871216 ], [ -1.1665600538253784, 0.4522880017757416 ], [ -0.34807997941970825, 0.7192960381507874 ], [ 0.23351998627185822, -0.5002239942550659 ], [ -0.25887998938560486, 0.03916800394654274 ], [ -0.32264000177383423, -0.016575995832681656 ], ... ],
[ [ 0.806719958782196, -0.9318400025367737 ], [ 0.5208799839019775, 0.13759998977184296 ], [ 0.13087999820709229, 0.811456024646759 ], [ -0.5758399963378906, -0.4604800045490265 ], [ 0.980400025844574, -1.0167039632797241 ], [ -1.3982399702072144, -0.3407360017299652 ], [ 0.5039199590682983, -0.8968959450721741 ], [ 0.18695999681949615, 0.7480959296226501 ], ... ],
[ [ -0.2815200090408325, 0.9175039529800415 ], [ -0.07231999933719635, -0.7343359589576721 ], [ 1.0781599283218384, 0.3400319814682007 ], [ 0.10335999727249146, 0.21190398931503296 ], [ -0.42023998498916626, 0.021696001291275024 ], [ 0.3787999749183655, 0.4719999432563782 ], [ 1.1083999872207642, 0.06284797936677933 ], [ 0.861840009689331, -0.143296018242836 ], ... ],
...
]
[
[ [ -1.4147999286651611, -0.7114879488945007 ], [ 0.28567999601364136, 0.6810239553451538 ], [ -0.7897599935531616, -0.8216959834098816 ], [ -1.3775999546051025, -0.1770239770412445 ], [ 0.8166400194168091, 0.48793599009513855 ], [ 0.08807999640703201, -0.36537599563598633 ], [ -0.066880002617836, 0.37171199917793274 ], [ 0.47647997736930847, 0.45209598541259766 ], ... ],
[ [ 0.5292800068855286, 0.35052797198295593 ], [ 0.7244800329208374, -1.0842880010604858 ], [ -1.0085599422454834, 0.055744003504514694 ], [ 0.1653600037097931, -0.3509120047092438 ], [ -0.21991999447345734, 0.26284798979759216 ], [ 0.1984800100326538, -0.6102399230003357 ], [ 0.3015200197696686, -0.3288320004940033 ], [ 1.2521599531173706, 0.11417599022388458 ], ... ],
[ [ 0.44575998187065125, 0.19014400243759155 ], [ -0.5179200172424316, -0.17286399006843567 ], [ -1.0256800651550293, -0.5865600109100342 ], [ -0.5053600072860718, -0.5775359869003296 ], [ -0.967199981212616, 0.04409600794315338 ], [ -0.15679998695850372, 0.31462401151657104 ], [ -0.9912800192832947, -0.1945599913597107 ], [ -0.9999199509620667, 1.1816320419311523 ], ... ],
[ [ 0.019200000911951065, 0.33350399136543274 ], [ -0.2590399980545044, 0.1454080045223236 ], [ 0.7347999811172485, 0.28435197472572327 ], [ -0.3375999927520752, 0.2659839987754822 ], [ 0.2853599786758423, -0.32179200649261475 ], [ 1.2112799882888794, 0.30751997232437134 ], [ -1.1861599683761597, -0.04454398155212402 ], [ -1.4215999841690063, -0.1996159851551056 ], ... ],
[ [ 0.5606399774551392, 0.3304959833621979 ], [ 0.9451199769973755, 0.6768640279769897 ], [ 0.24552001059055328, -1.033471941947937 ], [ -0.42608001828193665, 0.03910401090979576 ], [ -0.8915199637413025, -0.5232639908790588 ], [ 0.3808799982070923, -0.194240003824234 ], [ -1.1974400281906128, -0.6327679753303528 ], [ 0.8613599538803101, 0.31839999556541443 ], ... ],
[ [ 0.49271997809410095, -0.179967999458313 ], [ -0.36159998178482056, -0.025791998952627182 ], [ 0.9997599720954895, -1.0039039850234985 ], [ -0.9352800250053406, 1.0104960203170776 ], [ -0.710319995880127, -0.3112960159778595 ], [ -0.5642399787902832, 0.27270397543907166 ], [ -1.0225600004196167, 1.0591360330581665 ], [ -0.18223999440670013, 0.8360320329666138 ], ... ],
[ [ 0.8535199761390686, -0.623807966709137 ], [ 0.4501599669456482, 0.1995519995689392 ], [ 1.0388000011444092, -0.11846401542425156 ], [ -1.2255200147628784, 0.42630401253700256 ], [ 0.5912799835205078, -0.06931201368570328 ], [ -0.31751999258995056, -0.5567359924316406 ], [ 0.28943997621536255, -0.6718080043792725 ], [ -0.8406400084495544, 0.14963200688362122 ], ... ],
[ [ -0.14391998946666718, -0.3025279939174652 ], [ -0.6769599318504333, 0.6997759938240051 ], [ 0.98471999168396, -0.5967360138893127 ], [ 0.9645599722862244, -0.36428800225257874 ], [ 0.7752000093460083, -1.0915199518203735 ], [ 1.2855199575424194, 0.7678079605102539 ], [ 0.6531199812889099, -0.09696000069379807 ], [ 0.6594399809837341, -0.0756480023264885 ], ... ],
...
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 30.67 seconds (0.350 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12459565760137
Reset training subject: 12459632585878
Constructing line search parameters: GD
th(0)=1.01268213507908;dx=-1.3505963364356053
Armijo: th(2.154434690031884)=2.28509395096315; dx=2.5317894451509138 evalInputDelta=-1.2724118158840698
New Minimum: 1.01268213507908 > 0.6033446102593674
WOLF (strong): th(1.077217345015942)=0.6033446102593674; dx=0.5906034596646821 evalInputDelta=0.40933752481971264
END: th(0.3590724483386473)=0.6438924676531846; dx=-0.7035282019674599 evalInputDelta=0.36878966742589536
Fitness changed from 1.01268213507908 to 0.6033446102593674
Iteration 1 complete. Error: 0.6033446102593674 Total: 0.7385; Orientation: 0.0514; Line Search: 0.4848
th(0)=0.6033446102593674;dx=-0.25826657598721153
New Minimum: 0.6033446102593674 > 0.5066623961053072
WOLF (strong): th(0.7735981389354633)=0.5066623961053072; dx=0.008311911785004155 evalInputDelta=0.0966822141540602
END: th(0.3867990694677316)=0.5292254249157562; dx=-0.1249773197202271 evalInputDelta=0.07411918534361117
Fitness changed from 0.6033446102593674 to 0.5066623961053072
Iteration 2 complete. Error: 0.5066623961053072 Total: 0.4324; Orientation: 0.0515; Line Search: 0.3152
th(0)=0.5066623961053072;dx=-2.6757868611710045E-4
New Minimum: 0.5066623961053072 > 0.506563381477714
WOLF (strong): th(0.8333333333333335)=0.506563381477714; dx=2.9943444338592028E-5 evalInputDelta=9.9014627593208E-5
END: th(0.41666666666666674)=0.5065818969174025; dx=-1.1881759511406808E-4 evalInputDelta=8.049918790464794E-5
Fitness changed from 0.5066623961053072 to 0.506563381477714
Iteration 3 complete. Error: 0.506563381477714 Total: 0.5860; Orientation: 0.0515; Line Search: 0.4714
th(0)=0.506563381477714;dx=-3.3509283801882E-6
New Minimum: 0.506563381477714 > 0.5065621748953211
WOLF (strong): th(0.8976811208466184)=0.5065621748953211; dx=6.627054494270349E-7 evalInputDelta=1.206582392887512E-6
END: th(0.4488405604233092)=0.5065623278161431; dx=-1.3441109407933747E-6 evalInputDelta=1.0536615708822694E-6
Fitness changed from 0.506563381477714 to 0.5065621748953211
Iteration 4 complete. Error: 0.5065621748953211 Total: 0.4601; Orientation: 0.0496; Line Search: 0.3471
th(0)=0.5065621748953211;dx=-1.3107027656984561E-7
New Minimum: 0.5065621748953211 > 0.5065621299143073
WOLF (strong): th(0.9669976736693292)=0.5065621299143073; dx=3.803846803452897E-8 evalInputDelta=4.498101380256969E-8
END: th(0.4834988368346646)=0.5065621319639372; dx=-4.6516009222402566E-8 evalInputDelta=4.2931383892330643E-8
Fitness changed from 0.5065621748953211 to 0.5065621299143073
Iteration 5 complete. Error: 0.5065621299143073 Total: 0.4373; Orientation: 0.0505; Line Search: 0.3164
th(0)=0.5065621299143073;dx=-1.1047394854781976E-8
New Minimum: 0.5065621299143073 > 0.5065621264004925
WOLF (strong): th(1.041666666666667)=0.5065621264004925; dx=4.300545385874502E-9 evalInputDelta=3.5138147946867093E-9
New Minimum: 0.5065621264004925 > 0.506562126158796
END: th(0.5208333333333335)=0.506562126158796; dx=-3.3734896584702538E-9 evalInputDelta=3.755511235326026E-9
Fitness changed from 0.5065621299143073 to 0.506562126158796
Iteration 6 complete. Error: 0.506562126158796 Total: 0.4503; Orientation: 0.0515; Line Search: 0.3289
th(0)=0.506562126158796;dx=-1.0324657798858183E-9
New Minimum: 0.506562126158796 > 0.5065621258635435
WOLF (strong): th(1.122101401058273)=0.5065621258635435; dx=5.060615084564884E-10 evalInputDelta=2.952524891242092E-10
New Minimum: 0.5065621258635435 > 0.5065621257955248
END: th(0.5610507005291365)=0.5065621257955248; dx=-2.632371801438588E-10 evalInputDelta=3.632711909062891E-10
Fitness changed from 0.506562126158796 to 0.5065621257955248
Iteration 7 complete. Error: 0.5065621257955248 Total: 0.5100; Orientation: 0.0553; Line Search: 0.3790
Low gradient: 8.35637205043974E-6
th(0)=0.5065621257955248;dx=-6.983260762543898E-11
New Minimum: 0.5065621257955248 > 0.506562125774229
WOLF (strong): th(1.2087470920866614)=0.506562125774229; dx=3.492794283087965E-11 evalInputDelta=2.1295853969149903E-11
New Minimum: 0.506562125774229 > 0.5065621257691666
END: th(0.6043735460433307)=0.5065621257691666; dx=-1.744353340496342E-11 evalInputDelta=2.635824891683569E-11
Fitness changed from 0.5065621257955248 to 0.5065621257691666
Iteration 8 complete. Error: 0.5065621257691666 Total: 0.4850; Orientation: 0.0561; Line Search: 0.3496
Low gradient: 2.7041059492691083E-6
th(0)=0.5065621257691666;dx=-7.312253832644451E-12
New Minimum: 0.5065621257691666 > 0.5065621257622357
END: th(1.3020833333333337)=0.5065621257622357; dx=-3.086101751662514E-12 evalInputDelta=6.930900298129927E-12
Fitness changed from 0.5065621257691666 to 0.5065621257622357
Iteration 9 complete. Error: 0.5065621257622357 Total: 0.3174; Orientation: 0.0508; Line Search: 0.2032
Low gradient: 2.490352197807762E-6
th(0)=0.5065621257622357;dx=-6.2022087926574075E-12
New Minimum: 0.5065621257622357 > 0.506562125751856
END: th(2.805253502645683)=0.506562125751856; dx=-1.2517090755694424E-12 evalInputDelta=1.0379697101825514E-11
Fitness changed from 0.5065621257622357 to 0.506562125751856
Iteration 10 complete. Error: 0.506562125751856 Total: 0.5021; Orientation: 0.0612; Line Search: 0.2311
Low gradient: 3.851065097662257E-6
th(0)=0.506562125751856;dx=-1.48305525845708E-11
Armijo: th(6.043735460433308)=0.5065621259047154; dx=6.541846607309574E-11 evalInputDelta=-1.5285939181097774E-10
Armijo: th(3.021867730216654)=0.5065621257679442; dx=2.529475750347697E-11 evalInputDelta=-1.6088241849843143E-11
New Minimum: 0.506562125751856 > 0.5065621257438195
END: th(1.0072892434055514)=0.5065621257438195; dx=-1.4538026533370554E-12 evalInputDelta=8.036460386051658E-12
Fitness changed from 0.506562125751856 to 0.5065621257438195
Iteration 11 complete. Error: 0.5065621257438195 Total: 0.5515; Orientation: 0.0534; Line Search: 0.4278
Low gradient: 2.4615284457262382E-6
th(0)=0.5065621257438195;dx=-6.058777152563303E-12
New Minimum: 0.5065621257438195 > 0.5065621257343198
END: th(2.1701388888888897)=0.5065621257343198; dx=-2.644216179949451E-12 evalInputDelta=9.499734332507614E-12
Fitness changed from 0.5065621257438195 to 0.5065621257343198
Iteration 12 complete. Error: 0.5065621257343198 Total: 0.4315; Orientation: 0.1629; Line Search: 0.2047
Low gradient: 3.019611546318762E-6
th(0)=0.5065621257343198;dx=-9.117803546827536E-12
Armijo: th(4.675422504409472)=0.5065621257534276; dx=1.7321653413209224E-11 evalInputDelta=-1.9107826432218644E-11
New Minimum: 0.5065621257343198 > 0.5065621257285685
WOLF (strong): th(2.337711252204736)=0.5065621257285685; dx=4.103458378920351E-12 evalInputDelta=5.7512883344656984E-12
END: th(0.7792370840682453)=0.5065621257289135; dx=-4.7122044647038424E-12 evalInputDelta=5.406342040714662E-12
Fitness changed from 0.5065621257343198 to 0.5065621257285685
Iteration 13 complete. Error: 0.5065621257285685 Total: 0.5352; Orientation: 0.0508; Line Search: 0.4205
Low gradient: 4.889858664394105E-6
th(0)=0.5065621257285685;dx=-2.3913776430614074E-11
New Minimum: 0.5065621257285685 > 0.506562125724131
WOLF (strong): th(1.678815405675919)=0.506562125724131; dx=1.8720457500546007E-11 evalInputDelta=4.437561429426751E-12
New Minimum: 0.506562125724131 > 0.5065621257173045
END: th(0.8394077028379595)=0.5065621257173045; dx=-2.59614665117181E-12 evalInputDelta=1.126398974093945E-11
Fitness changed from 0.5065621257285685 to 0.5065621257173045
Iteration 14 complete. Error: 0.5065621257173045 Total: 0.4303; Orientation: 0.0511; Line Search: 0.3157
Low gradient: 2.2702222285417767E-6
th(0)=0.5065621257173045;dx=-5.1538388101986465E-12
New Minimum: 0.5065621257173045 > 0.5065621257085589
END: th(1.8084490740740748)=0.5065621257085589; dx=-4.492245508790231E-12 evalInputDelta=8.745670854182208E-12
Fitness changed from 0.5065621257173045 to 0.5065621257085589
Iteration 15 complete. Error: 0.5065621257085589 Total: 0.3318; Orientation: 0.0518; Line Search: 0.2126
Low gradient: 2.3294083951553354E-6
th(0)=0.5065621257085589;dx=-5.426343692803531E-12
New Minimum: 0.5065621257085589 > 0.5065621256929351
END: th(3.8961854203412267)=0.5065621256929351; dx=-2.5857942851824654E-12 evalInputDelta=1.5623724536339978E-11
Fitness changed from 0.50
...skipping 19793 bytes...
te. Error: 0.5065620853280468 Total: 0.4317; Orientation: 0.0518; Line Search: 0.3172
Low gradient: 2.247525293170575E-6
th(0)=0.5065620853280468;dx=-5.05137938597949E-12
New Minimum: 0.5065620853280468 > 0.50656208531939
WOLFE (weak): th(1.7697816488225515)=0.50656208531939; dx=-4.7690000950478745E-12 evalInputDelta=8.656853012212196E-12
New Minimum: 0.50656208531939 > 0.5065620853111332
END: th(3.539563297645103)=0.5065620853111332; dx=-4.486836719286766E-12 evalInputDelta=1.6913581646349485E-11
Fitness changed from 0.5065620853280468 to 0.5065620853111332
Iteration 49 complete. Error: 0.5065620853111332 Total: 0.5738; Orientation: 0.0505; Line Search: 0.4549
Low gradient: 2.5668207540979118E-6
th(0)=0.5065620853111332;dx=-6.58888874399431E-12
Armijo: th(7.62575795601026)=0.506562085325291; dx=1.027064959925644E-11 evalInputDelta=-1.4157786054624921E-11
New Minimum: 0.5065620853111332 > 0.5065620853022413
WOLF (strong): th(3.81287897800513)=0.5065620853022413; dx=1.8418589163047733E-12 evalInputDelta=8.891887226525341E-12
END: th(1.2709596593350434)=0.5065620853048077; dx=-3.778514199069348E-12 evalInputDelta=6.325495682801829E-12
Fitness changed from 0.5065620853111332 to 0.5065620853022413
Iteration 50 complete. Error: 0.5065620853022413 Total: 0.5347; Orientation: 0.0528; Line Search: 0.4103
Low gradient: 5.7125086642059264E-6
th(0)=0.5065620853022413;dx=-3.2630551260147246E-11
Armijo: th(2.7381995797025227)=0.5065620853514186; dx=6.86017022353196E-11 evalInputDelta=-4.9177217853468846E-11
New Minimum: 0.5065620853022413 > 0.5065620852922256
WOLF (strong): th(1.3690997898512614)=0.5065620852922256; dx=1.79832596147924E-11 evalInputDelta=1.0015765994353387E-11
New Minimum: 0.5065620852922256 > 0.5065620852910847
END: th(0.4563665966170871)=0.5065620852910847; dx=-1.5758445207156207E-11 evalInputDelta=1.1156631174458198E-11
Fitness changed from 0.5065620853022413 to 0.5065620852910847
Iteration 51 complete. Error: 0.5065620852910847 Total: 0.6778; Orientation: 0.0504; Line Search: 0.5565
Low gradient: 3.027035620880531E-6
th(0)=0.5065620852910847;dx=-9.163211971444352E-12
New Minimum: 0.5065620852910847 > 0.5065620852848292
END: th(0.9832120271236399)=0.5065620852848292; dx=-3.609467517629265E-12 evalInputDelta=6.2555516322504445E-12
Fitness changed from 0.5065620852910847 to 0.5065620852848292
Iteration 52 complete. Error: 0.5065620852848292 Total: 0.3795; Orientation: 0.0553; Line Search: 0.2451
Low gradient: 2.312049424270087E-6
th(0)=0.5065620852848292;dx=-5.345284481348216E-12
New Minimum: 0.5065620852848292 > 0.5065620852749907
END: th(2.118266098891739)=0.5065620852749907; dx=-4.175820049209764E-12 evalInputDelta=9.83846337732075E-12
Fitness changed from 0.5065620852848292 to 0.5065620852749907
Iteration 53 complete. Error: 0.5065620852749907 Total: 0.3797; Orientation: 0.0557; Line Search: 0.2457
Low gradient: 2.5129178066229852E-6
th(0)=0.5065620852749907;dx=-6.314780491869366E-12
New Minimum: 0.5065620852749907 > 0.5065620852651788
WOLF (strong): th(4.563665966170872)=0.5065620852651788; dx=2.1074999226267413E-12 evalInputDelta=9.811929047032208E-12
END: th(2.281832983085436)=0.5065620852651908; dx=-2.10381049320901E-12 evalInputDelta=9.799938638366257E-12
Fitness changed from 0.5065620852749907 to 0.5065620852651788
Iteration 54 complete. Error: 0.5065620852651788 Total: 0.4910; Orientation: 0.0518; Line Search: 0.3594
Low gradient: 6.384358745984209E-6
th(0)=0.5065620852651788;dx=-4.0762860027504576E-11
Armijo: th(4.9160601356182)=0.5065620856423251; dx=1.9430319016008875E-10 evalInputDelta=-3.771463141788445E-10
Armijo: th(2.4580300678091)=0.50656208530931; dx=7.676785462040354E-11 evalInputDelta=-4.413125420654751E-11
New Minimum: 0.5065620852651788 > 0.5065620852478154
END: th(0.8193433559363666)=0.5065620852478154; dx=-1.585883075118025E-12 evalInputDelta=1.7363332993625136E-11
Fitness changed from 0.5065620852651788 to 0.5065620852478154
Iteration 55 complete. Error: 0.5065620852478154 Total: 0.7545; Orientation: 0.0554; Line Search: 0.6196
Low gradient: 2.2901953062235767E-6
th(0)=0.5065620852478154;dx=-5.244806394089545E-12
New Minimum: 0.5065620852478154 > 0.5065620852392914
END: th(1.7652217490764495)=0.5065620852392914; dx=-4.507170270402115E-12 evalInputDelta=8.524070338467027E-12
Fitness changed from 0.5065620852478154 to 0.5065620852392914
Iteration 56 complete. Error: 0.5065620852392914 Total: 0.3498; Orientation: 0.0527; Line Search: 0.2194
Low gradient: 2.3470389348750923E-6
th(0)=0.5065620852392914;dx=-5.508372115139937E-12
New Minimum: 0.5065620852392914 > 0.5065620852238226
END: th(3.8030549718090603)=0.5065620852238226; dx=-2.5816043361593724E-12 evalInputDelta=1.5468737402102306E-11
Fitness changed from 0.5065620852392914 to 0.5065620852238226
Iteration 57 complete. Error: 0.5065620852238226 Total: 0.3213; Orientation: 0.0504; Line Search: 0.2072
Low gradient: 3.8096111136290773E-6
th(0)=0.5065620852238226;dx=-1.4511910598681023E-11
Armijo: th(8.193433559363667)=0.5065620855340781; dx=9.025703742170204E-11 evalInputDelta=-3.1025548796748126E-10
Armijo: th(4.096716779681834)=0.5065620852715984; dx=3.787387355160231E-11 evalInputDelta=-4.777578332948451E-11
New Minimum: 0.5065620852238226 > 0.5065620852158434
WOLF (strong): th(1.3655722598939446)=0.5065620852158434; dx=2.9479768287502183E-12 evalInputDelta=7.979172877981E-12
END: th(0.34139306497348615)=0.5065620852196209; dx=-1.014493049669086E-11 evalInputDelta=4.2017500589963674E-12
Fitness changed from 0.5065620852238226 to 0.5065620852158434
Iteration 58 complete. Error: 0.5065620852158434 Total: 0.6240; Orientation: 0.0516; Line Search: 0.5018
Low gradient: 3.378973392376182E-6
th(0)=0.5065620852158434;dx=-1.1417326840076974E-11
New Minimum: 0.5065620852158434 > 0.5065620852098078
END: th(0.7355090621151874)=0.5065620852098078; dx=-5.050105011548482E-12 evalInputDelta=6.035616451072201E-12
Fitness changed from 0.5065620852158434 to 0.5065620852098078
Iteration 59 complete. Error: 0.5065620852098078 Total: 0.3223; Orientation: 0.0514; Line Search: 0.2068
Low gradient: 2.221296748200708E-6
th(0)=0.5065620852098078;dx=-4.934172764914454E-12
New Minimum: 0.5065620852098078 > 0.5065620852019261
WOLFE (weak): th(1.5846062382537753)=0.5065620852019261; dx=-4.9295665112732435E-12 evalInputDelta=7.881695296418911E-12
New Minimum: 0.5065620852019261 > 0.5065620851941521
WOLFE (weak): th(3.1692124765075507)=0.5065620851941521; dx=-4.92499251191528E-12 evalInputDelta=1.5655698959449182E-11
New Minimum: 0.5065620851941521 > 0.5065620851628513
WOLFE (weak): th(9.507637429522653)=0.5065620851628513; dx=-4.906736905280937E-12 evalInputDelta=4.695654975961361E-11
New Minimum: 0.5065620851628513 > 0.5065620850242735
WOLFE (weak): th(38.03054971809061)=0.5065620850242735; dx=-4.824516837650736E-12 evalInputDelta=1.8553436564872072E-10
New Minimum: 0.5065620850242735 > 0.5065620843235517
END: th(190.15274859045306)=0.5065620843235517; dx=-4.3859393159626946E-12 evalInputDelta=8.862561795552892E-10
Fitness changed from 0.5065620852098078 to 0.5065620843235517
Iteration 60 complete. Error: 0.5065620843235517 Total: 0.7521; Orientation: 0.0509; Line Search: 0.6374
th(0)=0.5065620843235517;dx=-1.430995482118203E-10
Armijo: th(409.6716779681835)=0.5065774983918985; dx=7.539394075497538E-8 evalInputDelta=-1.5414068346819576E-5
Armijo: th(204.83583898409174)=0.50656592318447; dx=3.762539822571122E-8 evalInputDelta=-3.838860918370379E-6
Armijo: th(68.27861299469724)=0.5065625043496024; dx=1.2446395277997821E-8 evalInputDelta=-4.2002605071811416E-7
Armijo: th(17.06965324867431)=0.5065621087431889; dx=3.004268024808379E-9 evalInputDelta=-2.4419637201233968E-8
Armijo: th(3.413930649734862)=0.5065620849096033; dx=4.863647958974798E-10 evalInputDelta=-5.86051651652042E-10
New Minimum: 0.5065620843235517 > 0.5065620842719571
END: th(0.568988441622477)=0.5065620842719571; dx=-3.820459796515198E-11 evalInputDelta=5.15945064449852E-11
Fitness changed from 0.5065620843235517 to 0.5065620842719571
Iteration 61 complete. Error: 0.5065620842719571 Total: 0.8882; Orientation: 0.0506; Line Search: 0.7747
Final threshold in iteration 61: 0.5065620842719571 (> 0.0) after 30.673s (< 30.000s)
Returns
0.5065620842719571
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.0019842819821582307, 9.999409759194576E-4, 8.437734885137693E-4, -6.825456430607815E-4]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.9640179806046163, 0.8760171792484027 ], [ -1.5639927051818745, 0.2600074017979024 ], [ -1.2839824780889804, -0.4959881203001367 ], [ 0.988003722223316, -0.10400520063247598 ], [ 1.5999783585838852, -1.620006254967323 ], [ -0.9679732305997799, -0.3239973102377673 ], [ 0.5400240666263459, 1.9240112992132323 ], [ 1.9039976676169499, 0.4640121200885676 ], ... ],
[ [ 0.07597146489341212, 0.7479823484375354 ], [ 0.7040090249058867, 0.9639929944239989 ], [ 0.9200203504267972, 1.6119995145323345 ], [ -1.103964850258046, -1.4639877948436866 ], [ -1.175986539553363, -0.18798804597554017 ], [ 1.5440047005042918, 0.21601210405922305 ], [ -0.7639835814922986, 1.472022911358957 ], [ 1.1239968701610932, 0.1759992577102372 ], ... ],
[ [ -1.6760148348976949, 1.503991473367445 ], [ -1.4239894978005614, -0.31599725505483933 ], [ -0.2960224968268685, 0.4439869124258666 ], [ -1.9639900301351132, -1.559988068430092 ], [ -1.507980086039646, 0.9960083458149187 ], [ -1.8719777961037551, 0.9400007504210214 ], [ 1.0360047742759506, -1.856005070101688 ], [ 0.25202992516233236, -0.5999875046088501 ], ... ],
[ [ -1.5279719821391569, -1.615986666373551 ], [ -0.32799583883397154, 0.7919987494551263 ], [ 1.0400303323692268, -0.5359862922849203 ], [ -1.5919872447632508, -1.0039912660474304 ], [ 1.4879729496254495, 1.3359878969607961 ], [ 1.0439976618444875, 1.8439872681793135 ], [ 0.5200123008279907, 0.9239957587250718 ], [ 1.3840271003970563, -1.1759912859532444 ], ... ],
[ [ 0.947989875408914, -0.14401815869828646 ], [ -1.0239949809193136, 1.356006435925972 ], [ 1.3079890679477604, -0.800015758811804 ], [ -1.1799938315658034, -0.05600845699392248 ], [ 0.5680392164449517, 1.0200101115261102 ], [ 0.8400148639553005, 0.6320088804568675 ], [ -0.5559937485762628, -1.368001647875343 ], [ -1.5800104376796567, -0.21999902085319656 ], ... ],
[ [ 0.4080424382316181, -1.7119874028252617 ], [ -1.5240081656048856, -0.376007839892076 ], [ 1.7599968967733208, 1.928000879554758 ], [ -0.760022029556193, 0.9359828739280805 ], [ 0.9279988788733765, -0.21600996034360306 ], [ -1.7439971487381325, -1.9919915678379123 ], [ -1.1360033544881762, -1.9960017844718734 ], [ -0.29200626582380235, -1.8760026690889189 ], ... ],
[ [ -0.5319897843314959, -1.6639824266168806 ], [ -0.30398558339014475, -1.0599942376710774 ], [ 0.18001096044957174, 0.9079949341433865 ], [ -1.4760171705566343, 0.511997310638672 ], [ -1.1639875705347966, 0.8440195103488292 ], [ -1.416034102391952, -0.968012661202961 ], [ 1.6360012469401244, -1.0679878055648693 ], [ 0.06801281633576185, -1.0080030054191909 ], ... ],
[ [ 0.984002454839843, -0.18001071380702324 ], [ -0.24401017549736706, 1.5480041162960114 ], [ -1.6199700417947807, 0.14001105296338412 ], [ -0.6719942132562602, -0.8919995778965544 ], [ -0.03200834489778616, -1.892004253876229 ], [ -1.3599841757399251, -1.3479976740989048 ], [ -1.7959728404223705, 1.3440130575943547 ], [ 1.0840177544973504, -0.5679891449212758 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004773131106048822, 0.0010592646431177855 ], [ -0.002843410475179553, -0.0014971224591135979 ], [ -0.0030437421519309282, -7.448559044860303E-4 ], [ 0.0018564789788797498, 9.046396589837968E-4 ], [ 0.0015548974042758346, 0.00245574745349586 ], [ -0.0022447099909186363, -5.956072127446532E-4 ], [ 0.0029954577330499887, -8.57567647472024E-4 ], [ 0.004242052789777517, 0.0012898333370685577 ], ... ],
[ [ 8.986869943328202E-4, -4.4642938883043826E-4 ], [ 0.002360888523980975, -6.394510273821652E-5 ], [ 0.00343748414888978, -3.2397452741861343E-4 ], [ -0.0036544788163155317, 6.774231587769464E-5 ], [ -0.002521465765312314, -8.639558218419552E-4 ], [ 0.0032797399908304214, 0.00115535210352391 ], [ -4.4022846850566566E-5, -0.0016493519069626927 ], [ 0.0024063154123723507, 8.282712078653276E-4 ], ... ],
[ [ -0.0018217830220237374, -0.0024407196324318647 ], [ -0.0031415752600878477, -9.858420817181468E-4 ], [ -1.4343138900585473E-4, -5.528173060156405E-4 ], [ -0.005457005929201841, -5.923996213823557E-4 ], [ -0.001996308099478483, -0.0019522147485986352 ], [ -0.0027745864354074, -0.0022211186587810516 ], [ 1.998299267143011E-4, 0.0021409615874290466 ], [ -9.985367069020867E-5, 6.22175051830709E-4 ], ... ],
[ [ -0.004647818394005299, -1.8627750978339463E-4 ], [ 1.4111583004705608E-4, -8.173295063897967E-4 ], [ 0.0015277587808668613, 0.0012433851370587945 ], [ -0.004162883851677179, -6.58006698358804E-4 ], [ 0.004288467112928629, 3.436393744777888E-4 ], [ 0.003915464039891958, -3.7770794006064534E-4 ], [ 0.0019557923078536987, -1.9189665908925235E-4 ], [ 0.0015703780809417367, 0.00197047321125865 ], ... ],
[ [ 0.0017370694549754262, 8.981876890175045E-4 ], [ -6.75968243740499E-4, -0.0017895560013130307 ], [ 0.001795450458303094, 0.0016496938187628984 ], [ -0.002397445496171713, -9.574191644787788E-4 ], [ 0.0021470999345183372, -2.1690703579224646E-4 ], [ 0.002298797946423292, 2.7740735094994307E-4 ], [ -0.0024711694568395615, 4.645908484235406E-4 ], [ -0.0033551722299307585, -0.001183011569082737 ], ... ],
[ [ -9.022151934914291E-4, 0.001512804883532226 ], [ -0.003400047542527318, -0.0010292752413079143 ], [ 0.005420217290520668, 1.690900098765269E-4 ], [ -5.72170305531472E-4, -0.0012801374541595578 ], [ 0.001625414239242673, 9.304575505666435E-4 ], [ -0.0054524559527635574, -1.1191329394932836E-4 ], [ -0.004250035155564547, 4.0383284795098007E-4 ], [ -0.002455314854159951, 0.0010340703884139657 ], ... ],
[ [ -0.0027195019647479057, 6.868650671094656E-4 ], [ -0.0016631248872727156, 4.669994523283094E-4 ], [ 0.0012651338474825025, -4.678595287259668E-4 ], [ -0.0024168670643121004, -0.0015948857180774212 ], [ -0.001465709530748427, -0.0015582237392663956 ], [ -0.003777766600251198, -5.340991774573922E-4 ], [ 0.0021783628035336733, 0.002109364839270711 ], [ -8.729869732633233E-4, 7.453954895026982E-4 ], ... ],
[ [ 0.0017725381767377257, 9.531407267786562E-4 ], [ 0.0010637278901413083, -0.0012624728260561824 ], [ -0.0030744746327400208, -0.001462451764382422 ], [ -0.002225372940301895, 4.181955591775477E-5 ], [ -0.0019554062746465206, 0.0012643714435398579 ], [ -0.004046509973704815, -2.2744857415091246E-4 ], [ -0.002219782443717122, -0.0024327444843947887 ], [ 0.0015830411575734615, 0.0013023439096286893 ], ... ],
...
]
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 30.50 seconds (0.282 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12490278232923
Reset training subject: 12490340636745
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=1.01268213507908}, derivative=-1.3505962386365507}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=1.01268213507908}, derivative=-1.350596225490999}, evalInputDelta = 0.0
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=1.01268213507908}, derivative=-1.350596091419794}, evalInputDelta = 0.0
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=1.01268213507908}, derivative=-1.3505959641978065}, evalInputDelta = 0.0
New Minimum: 1.01268213507908 > 1.0126821324878816
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=1.0126821324878816}, derivative=-1.35059589290028}, evalInputDelta = -2.5911983669857364E-9
New Minimum: 1.0126821324878816 > 1.012681801125833
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=1.012681801125833}, derivative=-1.3505953934272052}, evalInputDelta = -3.339532470558737E-7
New Minimum: 1.012681801125833 > 1.0126798853019694
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=1.0126798853019694}, derivative=-1.350592501440919}, evalInputDelta = -2.249777110607809E-6
New Minimum: 1.0126798853019694 > 1.0126662408782092
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=1.0126662408782092}, derivative=-1.3505732768556384}, evalInputDelta = -1.589420087078075E-5
New Minimum: 1.0126662408782092 > 1.01257094433499
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=1.01257094433499}, derivative=-1.3504463242248195}, evalInputDelta = -1.1119074409005059E-4
New Minimum: 1.01257094433499 > 1.0119038516622667
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=1.0119038516622667}, derivative=-1.3495563483290847}, evalInputDelta = -7.782834168132968E-4
New Minimum: 1.0119038516622667 > 1.0072466541030478
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=1.0072466541030478}, derivative=-1.343323557025964}, evalInputDelta = -0.005435480976032192
New Minimum: 1.0072466541030478 > 0.9752501094127728
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.9752501094127728}, derivative=-1.299692219226127}, evalInputDelta = -0.03743202566630721
New Minimum: 0.9752501094127728 > 0.7808537926007908
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.7808537926007908}, derivative=-0.9942707982897337}, evalInputDelta = -0.2318283424782892
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.869478440264113}, derivative=1.1436701712854278}, evalInputDelta = -0.14320369481496698
0.869478440264113 <= 1.01268213507908
New Minimum: 0.7808537926007908 > 0.5065622580493978
F(0.7494784982548671) = LineSearchPoint{point=PointSample{avg=0.5065622580493978}, derivative=2.5623025065083663E-6}, evalInputDelta = -0.5061198770296822
Right bracket at 0.7494784982548671
Converged to right
Fitness changed from 1.01268213507908 to 0.5065622580493978
Iteration 1 complete. Error: 0.5065622580493978 Total: 1.9403; Orientation: 0.0511; Line Search: 1.6901
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065622580493978}, derivative=-3.5338059853777895E-7}
New Minimum: 0.5065622580493978 > 0.5065621257917756
F(0.7494784982548671) = LineSearchPoint{point=PointSample{avg=0.5065621257917756}, derivative=4.489863496265715E-10}, evalInputDelta = -1.3225762218738168E-7
0.5065621257917756 <= 0.5065622580493978
Converged to right
Fitness changed from 0.5065622580493978 to 0.5065621257917756
Iteration 2 complete. Error: 0.5065621257917756 Total: 0.3309; Orientation: 0.0507; Line Search: 0.2178
Low gradient: 2.356588017091791E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621257917756}, derivative=-5.553332890430253E-12}
New Minimum: 0.5065621257917756 > 0.5065621257877788
F(0.7494784982548671) = LineSearchPoint{point=PointSample{avg=0.5065621257877788}, derivative=-4.878011938018355E-12}, evalInputDelta = -3.9968028886505635E-12
New Minimum: 0.5065621257877788 > 0.5065621257747124
F(5.2463494877840695) = LineSearchPoint{point=PointSample{avg=0.5065621257747124}, derivative=-8.281453956089154E-13}, evalInputDelta = -1.7063239710068956E-11
F(36.72444641448848) = LineSearchPoint{point=PointSample{avg=0.5065621261951422}, derivative=2.7519362729091767E-11}, evalInputDelta = 4.0336656237371926E-10
F(2.8249574164991142) = LineSearchPoint{point=PointSample{avg=0.5065621257796608}, derivative=-3.009113796575298E-12}, evalInputDelta = -1.211486466701217E-11
F(19.7747019154938) = LineSearchPoint{point=PointSample{avg=0.5065621258577834}, derivative=1.225545058290665E-11}, evalInputDelta = 6.600775481757637E-11
F(1.5211309165764462) = LineSearchPoint{point=PointSample{avg=0.5065621257841543}, derivative=-4.1840732255201466E-12}, evalInputDelta = -7.621347997144312E-12
F(10.647916416035123) = LineSearchPoint{point=PointSample{avg=0.5065621257834861}, derivative=4.035234294007143E-12}, evalInputDelta = -8.289480213363731E-12
0.5065621257834861 <= 0.5065621257917756
New Minimum: 0.5065621257747124 > 0.5065621257744582
F(6.166867615392276) = LineSearchPoint{point=PointSample{avg=0.5065621257744582}, derivative=6.589847032982498E-16}, evalInputDelta = -1.7317369760405654E-11
Right bracket at 6.166867615392276
Converged to right
Fitness changed from 0.5065621257917756 to 0.5065621257744582
Iteration 3 complete. Error: 0.5065621257744582 Total: 1.1654; Orientation: 0.0522; Line Search: 1.0509
Low gradient: 6.3415165747199885E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621257744582}, derivative=-4.021320583312546E-11}
F(6.166867615392276) = LineSearchPoint{point=PointSample{avg=0.5065621264241771}, derivative=2.509509839547103E-10}, evalInputDelta = 6.497188342891036E-10
New Minimum: 0.5065621257744582 > 0.5065621257607309
F(0.47437443195325196) = LineSearchPoint{point=PointSample{avg=0.5065621257607309}, derivative=-1.781675954219771E-11}, evalInputDelta = -1.3727352587977748E-11
F(3.320621023672764) = LineSearchPoint{point=PointSample{avg=0.5065621259011944}, derivative=1.1657097436185282E-10}, evalInputDelta = 1.267361771084552E-10
F(0.25543238643636645) = LineSearchPoint{point=PointSample{avg=0.5065621257657912}, derivative=-2.815679207537187E-11}, evalInputDelta = -8.667067064038747E-12
F(1.7880267050545653) = LineSearchPoint{point=PointSample{avg=0.5065621257783226}, derivative=4.4209048118212906E-11}, evalInputDelta = 3.864353281812782E-12
F(0.1375405157734281) = LineSearchPoint{point=PointSample{avg=0.5065621257693529}, derivative=-3.371808661445126E-11}, evalInputDelta = -5.105360578738782E-12
New Minimum: 0.5065621257607309 > 0.5065621257576013
F(0.9627836104139967) = LineSearchPoint{point=PointSample{avg=0.5065621257576013}, derivative=5.242853602304797E-12}, evalInputDelta = -1.6856960272093602E-11
0.5065621257576013 <= 0.5065621257744582
New Minimum: 0.5065621257576013 > 0.5065621257572497
F(0.851737171659901) = LineSearchPoint{point=PointSample{avg=0.5065621257572497}, derivative=1.7626428317542902E-15}, evalInputDelta = -1.720856790399239E-11
Right bracket at 0.851737171659901
Converged to right
Fitness changed from 0.5065621257744582 to 0.5065621257572497
Iteration 4 complete. Error: 0.5065621257572497 Total: 1.2037; Orientation: 0.0508; Line Search: 1.0879
Low gradient: 2.356532603029713E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621257572497}, derivative=-5.553360638112633E-12}
New Minimum: 0.5065621257572497 > 0.506562125752889
F(0.851737171659901) = LineSearchPoint{point=PointSample{avg=0.506562125752889}, derivative=-4.787165698976788E-12}, evalInputDelta = -4.36073399612269E-12
New Minimum: 0.506562125752889 > 0.5065621257400736
F(5.962160201619307) = LineSearchPoint{point=PointSample{avg=0.5065621257400736}, derivative=-1.8670623400316178E-13}, evalInputDelta = -1.7176038369370872E-11
F(41.73512141133515) = LineSearchPoint{point=PointSample{avg=0.5065621263098367}, derivative=3.201378989442523E-11}, evalInputDelta = 5.525869761768831E-10
F(3.2103939547180884) = LineSearchPoint{point=PointSample{avg=0.5065621257442263}, derivative=-2.6638094533977363E-12}, evalInputDelta = -1.3023360168062936E-11
F(22.472757683026618) = L
...skipping 27498 bytes...
68566 > 0.5065621255564827
F(2.4961272809056956) = LineSearchPoint{point=PointSample{avg=0.5065621255564827}, derivative=2.4546603021771407E-12}, evalInputDelta = -6.9072525477054114E-12
0.5065621255564827 <= 0.50656212556339
New Minimum: 0.5065621255564827 > 0.5065621255557912
F(1.9111871222360695) = LineSearchPoint{point=PointSample{avg=0.5065621255557912}, derivative=-7.08226834302179E-16}, evalInputDelta = -7.598810469744421E-12
Left bracket at 1.9111871222360695
Converged to left
Fitness changed from 0.50656212556339 to 0.5065621255557912
Iteration 26 complete. Error: 0.5065621255557912 Total: 1.4509; Orientation: 0.0510; Line Search: 1.3296
Low gradient: 3.5299772914542447E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621255557912}, derivative=-1.2460805827526204E-11}
New Minimum: 0.5065621255557912 > 0.5065621255505524
F(1.9111871222360695) = LineSearchPoint{point=PointSample{avg=0.5065621255505524}, derivative=6.897086253477646E-12}, evalInputDelta = -5.238809386298726E-12
0.5065621255505524 <= 0.5065621255557912
New Minimum: 0.5065621255505524 > 0.5065621255480827
F(1.2302440539805546) = LineSearchPoint{point=PointSample{avg=0.5065621255480827}, derivative=-1.5918682983285527E-16}, evalInputDelta = -7.708500504577387E-12
Left bracket at 1.2302440539805546
Converged to left
Fitness changed from 0.5065621255557912 to 0.5065621255480827
Iteration 27 complete. Error: 0.5065621255480827 Total: 0.5343; Orientation: 0.0518; Line Search: 0.4118
Low gradient: 2.8321088712784693E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621255480827}, derivative=-8.020529525033289E-12}
New Minimum: 0.5065621255480827 > 0.5065621255414181
F(1.2302440539805546) = LineSearchPoint{point=PointSample{avg=0.5065621255414181}, derivative=-2.856796439370571E-12}, evalInputDelta = -6.664557794522352E-12
F(8.611708377863883) = LineSearchPoint{point=PointSample{avg=0.5065621256349807}, derivative=2.8122134245895127E-11}, evalInputDelta = 8.68980443158307E-11
F(0.6624391059895294) = LineSearchPoint{point=PointSample{avg=0.5065621255437225}, derivative=-5.239105114397219E-12}, evalInputDelta = -4.360178884610377E-12
F(4.637073741926706) = LineSearchPoint{point=PointSample{avg=0.5065621255561812}, derivative=1.1441175477158094E-11}, evalInputDelta = 8.098521853128204E-12
F(0.35669798014820814) = LineSearchPoint{point=PointSample{avg=0.5065621255456201}, derivative=-6.524293139653488E-12}, evalInputDelta = -2.4625856909210597E-12
New Minimum: 0.5065621255414181 > 0.5065621255412274
F(2.496885861037457) = LineSearchPoint{point=PointSample{avg=0.5065621255412274}, derivative=2.458937071477107E-12}, evalInputDelta = -6.855294110152954E-12
0.5065621255412274 <= 0.5065621255480827
New Minimum: 0.5065621255412274 > 0.5065621255406056
F(1.911008216368355) = LineSearchPoint{point=PointSample{avg=0.5065621255406056}, derivative=-8.81701008867183E-16}, evalInputDelta = -7.477019003943042E-12
Left bracket at 1.911008216368355
Converged to left
Fitness changed from 0.5065621255480827 to 0.5065621255406056
Iteration 28 complete. Error: 0.5065621255406056 Total: 1.0751; Orientation: 0.0510; Line Search: 0.9529
Low gradient: 3.529422886503968E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621255406056}, derivative=-1.2457548857199318E-11}
New Minimum: 0.5065621255406056 > 0.506562125535275
F(1.911008216368355) = LineSearchPoint{point=PointSample{avg=0.506562125535275}, derivative=6.888543273563465E-12}, evalInputDelta = -5.330624830435227E-12
0.506562125535275 <= 0.5065621255406056
New Minimum: 0.506562125535275 > 0.5065621255327601
F(1.2305574718143069) = LineSearchPoint{point=PointSample{avg=0.5065621255327601}, derivative=1.7801153505303096E-15}, evalInputDelta = -7.845502025816131E-12
Right bracket at 1.2305574718143069
Converged to right
Fitness changed from 0.5065621255406056 to 0.5065621255327601
Iteration 29 complete. Error: 0.5065621255327601 Total: 0.5537; Orientation: 0.0533; Line Search: 0.4367
Low gradient: 2.832569043226917E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621255327601}, derivative=-8.022944911879363E-12}
New Minimum: 0.5065621255327601 > 0.5065621255261867
F(1.2305574718143069) = LineSearchPoint{point=PointSample{avg=0.5065621255261867}, derivative=-2.8539454754320676E-12}, evalInputDelta = -6.573408484200627E-12
F(8.613902302700147) = LineSearchPoint{point=PointSample{avg=0.5065621256196616}, derivative=2.8157590118232577E-11}, evalInputDelta = 8.690148600720704E-11
F(0.6626078694384728) = LineSearchPoint{point=PointSample{avg=0.5065621255284252}, derivative=-5.239832363875914E-12}, evalInputDelta = -4.334976821951386E-12
F(4.63825508606931) = LineSearchPoint{point=PointSample{avg=0.5065621255408511}, derivative=1.1458312653386101E-11}, evalInputDelta = 8.090972336560753E-12
F(0.3567888527745623) = LineSearchPoint{point=PointSample{avg=0.5065621255303863}, derivative=-6.524473142893014E-12}, evalInputDelta = -2.3738788712535097E-12
New Minimum: 0.5065621255261867 > 0.5065621255259027
F(2.497521969421936) = LineSearchPoint{point=PointSample{avg=0.5065621255259027}, derivative=2.467812031236547E-12}, evalInputDelta = -6.857403533899742E-12
0.5065621255259027 <= 0.5065621255327601
New Minimum: 0.5065621255259027 > 0.5065621255251627
F(1.9100129080799408) = LineSearchPoint{point=PointSample{avg=0.5065621255251627}, derivative=-9.101086060205158E-16}, evalInputDelta = -7.597478202114871E-12
Left bracket at 1.9100129080799408
Converged to left
Fitness changed from 0.5065621255327601 to 0.5065621255251627
Iteration 30 complete. Error: 0.5065621255251627 Total: 1.0911; Orientation: 0.0515; Line Search: 0.9756
Low gradient: 3.5289448917333612E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.5065621255251627}, derivative=-1.245197327242514E-11}
New Minimum: 0.5065621255251627 > 0.5065621255196828
F(1.9100129080799408) = LineSearchPoint{point=PointSample{avg=0.5065621255196828}, derivative=6.875458418357771E-12}, evalInputDelta = -5.479838804944848E-12
0.5065621255196828 <= 0.5065621255251627
New Minimum: 0.5065621255196828 > 0.506562125517351
F(1.230553032700178) = LineSearchPoint{point=PointSample{avg=0.506562125517351}, derivative=9.853418186749535E-16}, evalInputDelta = -7.811640223565064E-12
Right bracket at 1.230553032700178
Converged to right
Fitness changed from 0.5065621255251627 to 0.506562125517351
Iteration 31 complete. Error: 0.506562125517351 Total: 0.5968; Orientation: 0.0509; Line Search: 0.4818
Low gradient: 2.8327145826035416E-6
F(0.0) = LineSearchPoint{point=PointSample{avg=0.506562125517351}, derivative=-8.02404217650844E-12}
New Minimum: 0.506562125517351 > 0.5065621255108925
F(1.230553032700178) = LineSearchPoint{point=PointSample{avg=0.5065621255108925}, derivative=-2.8530428510944595E-12}, evalInputDelta = -6.458500401151923E-12
F(8.613871228901246) = LineSearchPoint{point=PointSample{avg=0.5065621256045634}, derivative=2.816612488483571E-11}, evalInputDelta = 8.721234845410208E-11
F(0.6626054791462497) = LineSearchPoint{point=PointSample{avg=0.5065621255133268}, derivative=-5.2385561374600805E-12}, evalInputDelta = -4.024225397358805E-12
F(4.6382383540237475) = LineSearchPoint{point=PointSample{avg=0.50656212552576}, derivative=1.14637176730698E-11}, evalInputDelta = 8.408940210813398E-12
F(0.35678756569413445) = LineSearchPoint{point=PointSample{avg=0.5065621255149084}, derivative=-6.525058705684653E-12}, evalInputDelta = -2.442601676477807E-12
New Minimum: 0.5065621255108925 > 0.5065621255106147
F(2.497512959858941) = LineSearchPoint{point=PointSample{avg=0.5065621255106147}, derivative=2.47018001903056E-12}, evalInputDelta = -6.736278201913137E-12
0.5065621255106147 <= 0.506562125517351
New Minimum: 0.5065621255106147 > 0.5065621255098751
F(1.9096364602231752) = LineSearchPoint{point=PointSample{avg=0.5065621255098751}, derivative=-6.243858061152125E-16}, evalInputDelta = -7.475908780918417E-12
Left bracket at 1.9096364602231752
Converged to left
Fitness changed from 0.506562125517351 to 0.5065621255098751
Iteration 32 complete. Error: 0.5065621255098751 Total: 1.1108; Orientation: 0.0502; Line Search: 0.9971
Final threshold in iteration 32: 0.5065621255098751 (> 0.0) after 30.498s (< 30.000s)
Returns
0.5065621255098751
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.0019735542811842206, 9.942393141630617E-4, 8.403732856232923E-4, -6.794488269538835E-4]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.963998173242986, 0.8759989758408454 ], [ -1.5639994727931603, 0.26000041835379567 ], [ -1.2839992935828732, -0.4959993856359608 ], [ 0.9879999450260063, -0.10400007907574937 ], [ 1.599999410976403, -1.6200004266306605 ], [ -0.9679986692822479, -0.3239990618820557 ], [ 0.5399991095029696, 1.923999505994918 ], [ 1.9039980068729088, 0.46399880322312675 ], ... ],
[ [ 0.07599984392920864, 0.7479996661080492 ], [ 0.7039999931774094, 0.9639999105346774 ], [ 0.9199996139336928, 1.6119997232903105 ], [ -1.1039984950466604, -1.463998810905526 ], [ -1.1759995969234962, -0.18799959686081277 ], [ 1.5439985152102031, 0.2159991507854571 ], [ -0.7640009823218953, 1.4719995633572958 ], [ 1.1239992811574469, 0.17599951491359966 ], ... ],
[ [ -1.6759996051250947, 1.5040001108185899 ], [ -1.4239989551473426, -0.31599926907208997 ], [ -0.29599993099226596, 0.4439998743283281 ], [ -1.9639987945511421, -1.5599990387858642 ], [ -1.5079994184175671, 0.9960004581514631 ], [ -1.8719986696831101, 0.9400008762541948 ], [ 1.0360005743308789, -1.8559996154123373 ], [ 0.2520002852669591, -0.5999996350138802 ], ... ],
[ [ -1.5279985118987798, -1.6159988156707166 ], [ -0.3279998995993123, 0.7920000356313988 ], [ 1.0399997811880892, -0.5359999530496314 ], [ -1.5919989092513138, -1.0039991574604503 ], [ 1.4879985902520056, 1.3359988882082234 ], [ 1.043999477655206, 1.8439994779443185 ], [ 0.5200000434417139, 0.9239999758731994 ], [ 1.3840000262781773, -1.1759998279826245 ], ... ],
[ [ 0.9480003214291316, -0.1439999778152866 ], [ -1.0240001691552207, 1.3559999201833786 ], [ 1.3080001932469412, -0.800000019849784 ], [ -1.179998638681237, -0.055999185964314815 ], [ 0.5679999782993463, 1.0200001065912512 ], [ 0.839999264004418, 0.6319996084293396 ], [ -0.5559989110718335, -1.3679992585363838 ], [ -1.5799994725630355, -0.21999965372423771 ], ... ],
[ [ 0.4080009974151588, -1.7119991212735604 ], [ -1.5239988384953504, -0.37599931952143995 ], [ 1.7599982151061457, 1.9279987851416764 ], [ -0.7599995999636067, 0.9360000368154082 ], [ 0.9280001502156721, -0.2159999951130371 ], [ -1.743998773549083, -1.9919990562046825 ], [ -1.1359986765717576, -1.9959991012917069 ], [ -0.29199921017542424, -1.8759994651087106 ], ... ],
[ [ -0.5319998559539242, -1.6639996719835983 ], [ -0.30399934307197396, -1.0599994656869505 ], [ 0.18000023760374062, 0.9080000930732379 ], [ -1.4759997848848634, 0.5120000766740906 ], [ -1.1640004653064338, 0.8439998800117561 ], [ -1.4159992251862061, -0.9679996355277775 ], [ 1.6359988105573682, -1.068000623073021 ], [ 0.06800091999769306, -1.0079993838998038 ], ... ],
[ [ 0.9840002680811418, -0.1799999220715396 ], [ -0.24400100237748856, 1.5479993288944585 ], [ -1.6199988925191267, 0.14000086544000565 ], [ -0.6719991605347689, -0.8919994152267006 ], [ -0.03199931396127929, -1.891999550574791 ], [ -1.3599984222476802, -1.3479988889660794 ], [ -1.7959994424093584, 1.3440004881292416 ], [ 1.083999537762221, -0.5680001551837647 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004747009836137295, 0.0010552951134741306 ], [ -0.0028281351551413536, -0.0014910002937540412 ], [ -0.0030271844007074833, -7.420324836857617E-4 ], [ 0.0018464704044163227, 9.009514469653368E-4 ], [ 0.001547017483972013, 0.0024453040678054094 ], [ -0.0022325303871184587, -5.933394422754645E-4 ], [ 0.0029786336235702038, -8.534584194421768E-4 ], [ 0.004218969028443098, 0.001284805708564818 ], ... ],
[ [ 8.936804952099919E-4, -4.4435926247388124E-4 ], [ 0.002347828820347786, -6.336584192467853E-5 ], [ 0.0034183827228844166, -3.221282677259296E-4 ], [ -0.0036343662068247795, 6.694140756735578E-5 ], [ -0.0025078158359974623, -8.605425246059895E-4 ], [ 0.0032619195990264416, 0.001150774653069675 ], [ -4.427754174685106E-5, -0.0016421943437308073 ], [ 0.0023932592011988163, 8.249962120316923E-4 ], ... ],
[ [ -0.0018123401096090674, -0.002430356340482831 ], [ -0.0031245180871337652, -9.819853585213423E-4 ], [ -1.427298120688647E-4, -5.50425669644028E-4 ], [ -0.005427070427685976, -5.905525758862495E-4 ], [ -0.00198585563339293, -0.001944013754837215 ], [ -0.0027599050663411617, -0.0022118601482361555 ], [ 1.992956385947764E-4, 0.002131683984771371 ], [ -9.920701268129051E-5, 6.19443308096379E-4 ], ... ],
[ [ -0.004622277803719044, -1.8610063125379384E-4 ], [ 1.401120243826881E-4, -8.137658587656915E-4 ], [ 0.0015195837477222085, 0.0012381725246086717 ], [ -0.004140111617743969, -6.557073793374002E-4 ], [ 0.004264948423951864, 3.4273136407136917E-4 ], [ 0.003893766552209854, -3.755540819838643E-4 ], [ 0.0019449254032224417, -1.9081655773334205E-4 ], [ 0.0015621738275513053, 0.001962108537554741 ], ... ],
[ [ 0.0017277596052736044, 8.945147274062037E-4 ], [ -6.727314321324229E-4, -0.0017818749183788896 ], [ 0.0017860179068520665, 0.0016427674563601613 ], [ -0.0023844679817557335, -9.535907302051783E-4 ], [ 0.0021351028699427843, -2.157058916054666E-4 ], [ 0.0022861429024487734, 2.7650155243463814E-4 ], [ -0.0024574128910899162, 4.6223890967667103E-4 ], [ -0.0033369469456374645, -0.0011783107183873653 ], ... ],
[ [ -8.969248156063259E-4, 0.001506089000031352 ], [ -0.003381527727469802, -0.0010252556530758739 ], [ 0.005390344187617302, 1.690789795247838E-4 ], [ -5.692922859452665E-4, -0.0012746474239975214 ], [ 0.0016167029971256852, 9.266274864785373E-4 ], [ -0.005422400310635567, -1.1214858386665583E-4 ], [ -0.004226455930620432, 4.0151627035811543E-4 ], [ -0.002441468881443143, 0.001029257313348353 ], ... ],
[ [ -0.002704344457015395, 6.835241802036762E-4 ], [ -0.0016538522904738784, 4.647424793802202E-4 ], [ 0.0012580096954479814, -4.65672230347991E-4 ], [ -0.002403915161266923, -0.0015882686711847782 ], [ -0.0014580803690478206, -0.0015516496496275067 ], [ -0.0037569745909422636, -5.32261619810015E-4 ], [ 0.002166884019970894, 0.0021005014423280954 ], [ -8.679892052896321E-4, 7.420301553793252E-4 ], ... ],
[ [ 0.0017630149377509952, 9.492282988503575E-4 ], [ 0.0010575327323749661, -0.0012568383244797587 ], [ -0.0030579615850001574, -0.0014565272722393274 ], [ -0.0022130876313894987, 4.133781840209849E-5 ], [ -0.0019442528719082475, 0.0012586255325004458 ], [ -0.00402426440268755, -2.2701012494508177E-4 ], [ -0.0022082440555095673, -0.002422489458695054 ], [ 0.0015746037242934108, 0.0012968912487849593 ], ... ],
...
]
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 31.47 seconds (0.255 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 12520813506317
Reset training subject: 12520883921281
Adding measurement 325ffc5 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 1.01268213507908 < 1.01268213507908. Total: 1
th(0)=1.01268213507908;dx=-1.3505968460951956
Non-optimal measurement 2.2850947482225794 < 1.01268213507908. Total: 1
Armijo: th(2.154434690031884)=2.2850947482225794; dx=2.5317901964567944 evalInputDelta=-1.2724126131434994
Adding measurement 6b794991 to history. Total: 1
New Minimum: 1.01268213507908 > 0.6033446911614605
WOLF (strong): th(1.077217345015942)=0.6033446911614605; dx=0.590603777505727 evalInputDelta=0.4093374439176195
Non-optimal measurement 0.6438924390454142 < 0.6033446911614605. Total: 2
END: th(0.3590724483386473)=0.6438924390454142; dx=-0.7035279785009576 evalInputDelta=0.36878969603366585
Fitness changed from 1.01268213507908 to 0.6033446911614605
Iteration 1 complete. Error: 0.6033446911614605 Total: 0.7838; Orientation: 0.0638; Line Search: 0.5094
Non-optimal measurement 0.6033446911614605 < 0.6033446911614605. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.6033446911614605 < 0.6033446911614605. Total: 2
th(0)=0.6033446911614605;dx=-0.258266895688609
Adding measurement 130fd30e to history. Total: 2
New Minimum: 0.6033446911614605 > 0.5066623967475152
WOLF (strong): th(0.7735981389354633)=0.5066623967475152; dx=0.008311940058690149 evalInputDelta=0.09668229441394527
Non-optimal measurement 0.5292254416086739 < 0.5066623967475152. Total: 3
END: th(0.3867990694677316)=0.5292254416086739; dx=-0.12497738839211243 evalInputDelta=0.07411924955278659
Fitness changed from 0.6033446911614605 to 0.5066623967475152
Iteration 2 complete. Error: 0.5066623967475152 Total: 0.5028; Orientation: 0.0654; Line Search: 0.3731
Non-optimal measurement 0.5066623967475152 < 0.5066623967475152. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.5066623967475152 < 0.5066623967475152. Total: 3
th(0)=0.5066623967475152;dx=-2.675803594319463E-4
Adding measurement 3810c3a6 to history. Total: 3
New Minimum: 0.5066623967475152 > 0.5065633814830979
WOLF (strong): th(0.8333333333333335)=0.5065633814830979; dx=2.9943612388233745E-5 evalInputDelta=9.901526441735697E-5
Non-optimal measurement 0.5065818970512852 < 0.5065633814830979. Total: 4
END: th(0.41666666666666674)=0.5065818970512852; dx=-1.1881842055503741E-4 evalInputDelta=8.049969623002973E-5
Fitness changed from 0.5066623967475152 to 0.5065633814830979
Iteration 3 complete. Error: 0.5065633814830979 Total: 0.5018; Orientation: 0.0531; Line Search: 0.3804
Non-optimal measurement 0.5065633814830979 < 0.5065633814830979. Total: 4
Rejected: LBFGS Orientation magnitude: 1.372e-03, gradient 1.831e-03, dot -1.000; [1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065633814830979, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
LBFGS Accumulation History: 3 points
Removed measurement 3810c3a6 to history. Total: 3
Adding measurement 6b38504a to history. Total: 3
th(0)=0.5065633814830979;dx=-3.35094173698539E-6
Adding measurement 1602b3bf to history. Total: 4
New Minimum: 0.5065633814830979 > 0.506562174895493
WOLF (strong): th(0.8976811208466184)=0.506562174895493; dx=6.627073171635194E-7 evalInputDelta=1.206587604829501E-6
Non-optimal measurement 0.5065623278171341 < 0.506562174895493. Total: 5
END: th(0.4488405604233092)=0.5065623278171341; dx=-1.3441162107374E-6 evalInputDelta=1.0536659638127333E-6
Fitness changed from 0.5065633814830979 to 0.506562174895493
Iteration 4 complete. Error: 0.506562174895493 Total: 2.0264; Orientation: 1.5860; Line Search: 0.3759
Non-optimal measurement 0.506562174895493 < 0.506562174895493. Total: 5
Rejected: LBFGS Orientation magnitude: 2.713e-04, gradient 3.620e-04, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.506562174895493, 0.5065633814830979, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
Rejected: LBFGS Orientation magnitude: 2.713e-04, gradient 3.620e-04, dot -1.000; [1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.506562174895493, 0.5065633814830979, 0.5066623967475152, 0.6033446911614605
LBFGS Accumulation History: 3 points
Removed measurement 1602b3bf to history. Total: 4
Removed measurement 6b38504a to history. Total: 3
Adding measurement 14d4fc0 to history. Total: 3
th(0)=0.506562174895493;dx=-1.3107018580070163E-7
Adding measurement 243489b6 to history. Total: 4
New Minimum: 0.506562174895493 > 0.5065621299142029
WOLF (strong): th(0.9669976736693292)=0.5065621299142029; dx=3.803806618187606E-8 evalInputDelta=4.498129013708052E-8
Non-optimal measurement 0.5065621319638351 < 0.5065621299142029. Total: 5
END: th(0.4834988368346646)=0.5065621319638351; dx=-4.6516163184869547E-8 evalInputDelta=4.293165789537312E-8
Fitness changed from 0.506562174895493 to 0.5065621299142029
Iteration 5 complete. Error: 0.5065621299142029 Total: 3.9517; Orientation: 3.4988; Line Search: 0.3817
Non-optimal measurement 0.5065621299142029 < 0.5065621299142029. Total: 5
Rejected: LBFGS Orientation magnitude: 7.877e-05, gradient 1.051e-04, dot -1.000; [fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621299142029, 0.506562174895493, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
Rejected: LBFGS Orientation magnitude: 7.877e-05, gradient 1.051e-04, dot -1.000; [1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621299142029, 0.506562174895493, 0.5066623967475152, 0.6033446911614605
LBFGS Accumulation History: 3 points
Removed measurement 243489b6 to history. Total: 4
Removed measurement 14d4fc0 to history. Total: 3
Adding measurement 1aaa17ef to history. Total: 3
th(0)=0.5065621299142029;dx=-1.1047347566814448E-8
Adding measurement 3f4d2638 to history. Total: 4
New Minimum: 0.5065621299142029 > 0.5065621264003084
WOLF (strong): th(1.041666666666667)=0.5065621264003084; dx=4.300604318197414E-9 evalInputDelta=3.5138945086998774E-9
Adding measurement 5a4e214b to history. Total: 5
New Minimum: 0.5065621264003084 > 0.5065621261588826
END: th(0.5208333333333335)=0.5065621261588826; dx=-3.3734402267029944E-9 evalInputDelta=3.75532027696579E-9
Fitness changed from 0.5065621299142029 to 0.5065621261588826
Iteration 6 complete. Error: 0.5065621261588826 Total: 4.2734; Orientation: 3.7937; Line Search: 0.4129
Non-optimal measurement 0.5065621261588826 < 0.5065621261588826. Total: 6
Rejected: LBFGS Orientation magnitude: 2.408e-05, gradient 3.213e-05, dot -1.000; [fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621261588826, 0.5065621264003084, 0.5065621299142029, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
Rejected: LBFGS Orientation magnitude: 2.408e-05, gradient 3.213e-05, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621261588826, 0.5065621264003084, 0.5065621299142029, 0.5066623967475152, 0.6033446911614605
Rejected: LBFGS Orientation magnitude: 2.408e-05, gradient 3.213e-05, dot -1.000; [1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621261588826, 0.5065621264003084, 0.5065621299142029, 0.5066623967475152
LBFGS Accumulation History: 3 points
Removed measurement 5a4e214b to history. Total: 5
Removed measurement 3f4d2638 to history. Total: 4
Removed measurement 1aaa17ef to history. Total: 3
Adding measurement 69a4a70d to history. Total: 3
th(0)=0.5065621261588826;dx=-1.0324271079501779E-9
Adding measurement 468d5c65 to history. Total: 4
New Minimum: 0.5065621261588826 > 0.5065621258636575
WOLF (strong): th(1.122101401058273)=0.5065621258636575; dx=5.060370331802811E-10 evalInputDelta=2.9522517763780343E-10
Adding measurement 5c0c24ab to history. Total: 5
New Minimum: 0.5065621258636575 > 0.5065621257952866
END: th(0.5610507005291365)=0.5065621257952866; dx=-2.632139244352334E-10 evalInputDelta=3.6359604216329444E-10
Fitness changed from 0.5065621261588826 to 0.5065621257952866
Iteration 7 complete. Error: 0.5065621257952866 Total: 6.3706; Orientation: 5.9108; Line Search: 0.3898
Non-optimal measurement 0.5065621257952866 < 0.5065621257952866. Total: 6
Rejected: LBFGS Orientation magnitude: 6.267e-06, gradient 8.357e-06, dot -1.000; [fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257952866, 0.5065621258636575, 0.5065621261588826, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
Rejected: LBFGS Orientation magnitude: 6.267e-06, gradient 8.357e-06, dot -1.000; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257952866, 0.5065621258636575, 0.5065621261588826, 0.5066623967475152, 0.6033446911614605
Rejected: LBFGS Orientation magnitude: 6.266e-06, gradient 8.357e-06, dot -1.000; [fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257952866, 0.5065621258636575, 0.5065621261588826, 0.5066623967475152
LBFGS Accumulation History: 3 points
Removed measurement 5c0c24ab to history. Total: 5
Removed measurement 468d5c65 to history. Total: 4
Removed measurement 69a4a70d to history. Total: 3
Adding measurement 4f981135 to history. Total: 3
th(0)=0.5065621257952866;dx=-6.983598958107681E-11
Adding measurement 611c1032 to history. Total: 4
New Minimum: 0.5065621257952866 > 0.5065621257744792
WOLF (strong): th(1.2087470920866614)=0.5065621257744792; dx=3.493652458004666E-11 evalInputDelta=2.0807355838314834E-11
Adding measurement 44bdedf to history. Total: 5
New Minimum: 0.5065621257744792 > 0.5065621257690438
END: th(0.6043735460433307)=0.5065621257690438; dx=-1.7447236370154602E-11 evalInputDelta=2.6242785722274675E-11
Fitness changed from 0.5065621257952866 to 0.5065621257690438
Iteration 8 complete. Error: 0.5065621257690438 Total: 6.6020; Orientation: 6.1291; Line Search: 0.4071
Non-optimal measurement 0.5065621257690438 < 0.5065621257690438. Total: 6
Rejected: LBFGS Orientation magnitude: 2.235e-06, gradient 2.704e-06, dot -0.998; [1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257690438, 0.5065621257744792, 0.5065621257952866, 0.5066623967475152, 0.6033446911614605, 1.01268213507908
Rejected: LBFGS Orientation magnitude: 2.235e-06, gradient 2.704e-06, dot -0.998; [fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00, 3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257690438, 0.5065621257744792, 0.5065621257952866, 0.5066623967475152, 0.6033446911614605
Rejected: LBFGS Orientation magnitude: 2.233e-06, gradient 2.704e-06, dot -0.998; [3ed969f5-7f15-4f23-bfb4-706668ecfb99 = 1.000/1.000e+00, 1180c78c-39d4-463b-b17c-104d19af5420 = 1.000/1.000e+00, fa0834fd-70c9-4537-abfb-3f68448cae9f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.5065621257690438, 0.5065621257744792, 0.5065621257952866, 0.5066623967475152
LBFGS Accumulation History: 3 points
Removed measurement 44bdedf to history. Total: 5
Removed measurement 611c1032 to history. Total: 4
Removed measurement 4f981135 to history. Total: 3
Adding measurement bd29297 to history. Total: 3
th(0)=0.5065621257690438;dx=-7.309137777681762E-12
Adding measurement 2caee7dc to history. Total: 4
New Minimum: 0.5065621257690438 > 0.5065621257623028
END: th(1.3020833333333337)=0.5065621257623028; dx=-3.0880484135473947E-12 evalInputDelta=6.7410521609190255E-12
Fitness changed from 0.5065621257690438 to 0.5065621257623028
Iteration 9 complete. Error: 0.5065621257623028 Total: 6.4563; Orientation: 6.1163; Line Search: 0.2748
Final threshold in iteration 9: 0.5065621257623028 (> 0.0) after 31.469s (< 30.000s)
Returns
0.5065621257623028
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
[0.001973623307313436, 9.93499420269031E-4, 8.415484271241693E-4, -6.807414394651847E-4]
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.963997194175255, 0.8759982752929791 ], [ -1.5639990474121253, 0.2600006818546423 ], [ -1.2839988550971282, -0.49599908408599475 ], [ 0.9879996129102189, -0.1040002512796733 ], [ 1.5999993668422556, -1.6200004762880804 ], [ -0.9679984289783837, -0.3239988011861514 ], [ 0.5399983421575547, 1.923998987824376 ], [ 1.9039972651401378, 0.46399821579140516 ], ... ],
[ [ 0.07599982655286568, 0.7479996303746113 ], [ 0.7039994845564762, 0.9639996332182442 ], [ 0.91999881496296, 1.6119992468576914 ], [ -1.1039980043538038, -1.463998385108489 ], [ -1.1759992404820336, -0.18799937316631665 ], [ 1.5439979096172083, 0.21599868807131184 ], [ -0.7640011633149768, 1.4719993420105864 ], [ 1.123998878924296, 0.17599923492470196 ], ... ],
[ [ -1.6759992846572764, 1.5040002866694553 ], [ -1.4239984643763515, -0.3159989091426872 ], [ -0.2959997912328685, 0.44399994619438243 ], [ -1.9639978391771211, -1.5599984037769683 ], [ -1.5079992546757168, 0.9960005795318828 ], [ -1.871998380621952, 0.9400011383053328 ], [ 1.0360006161645505, -1.8559995012124313 ], [ 0.252000151428677, -0.5999996655886476 ], ... ],
[ [ -1.527997800564548, -1.6159982762572114 ], [ -0.3279999920575688, 0.7919999837193441 ], [ 1.0399993625506767, -0.5360001805110102 ], [ -1.5919982214837065, -1.0039986804060346 ], [ 1.4879979496118754, 1.3359983977474954 ], [ 1.0439987247375317, 1.8439990089922613 ], [ 0.5199995856359801, 0.9239997298150364 ], [ 1.3839996570307171, -1.1759999930369178 ], ... ],
[ [ 0.948000099838015, -0.144000053406069 ], [ -1.0240001611180716, 1.355999880939088 ], [ 1.308000001926077, -0.800000080768457 ], [ -1.179998259380264, -0.055998854600461064 ], [ 0.5679993156836259, 1.0199997454926315 ], [ 0.8399987434120889, 0.6319992562531093 ], [ -0.555998449587318, -1.3679988886882863 ], [ -1.579998824349191, -0.21999926644470777 ], ... ],
[ [ 0.4080009872816022, -1.7119989969601204 ], [ -1.5239981842902677, -0.375998861806334 ], [ 1.7599971996761299, 1.9279980529289358 ], [ -0.7599994154022702, 0.9360001558422865 ], [ 0.9279998946440704, -0.21600010499422415 ], [ -1.7439977510422369, -1.9919983761710844 ], [ -1.135997818444017, -1.9959984938442334 ], [ -0.29199864784057045, -1.8759990638243138 ], ... ],
[ [ -0.5319993665897012, -1.6639993793093322 ], [ -0.3039990885709713, -1.059999251394951 ], [ 0.17999990714922867, 0.9079999317536537 ], [ -1.475999295914442, 0.5120003384056724 ], [ -1.164000343199973, 0.8439998771328402 ], [ -1.415998316276823, -0.9679990743974088 ], [ 1.6359984850762044, -1.068000889296949 ], [ 0.06800104937033793, -1.0079992069184933 ], ... ],
[ [ 0.9839999641943427, -0.18000004544698295 ], [ -0.2440012105146693, 1.5479990931843253 ], [ -1.6199985582465255, 0.1400011418574424 ], [ -0.671998765425348, -0.8919991133491281 ], [ -0.031998823175963934, -1.8919991955837263 ], [ -1.3599977513420693, -1.3479983646043783 ], [ -1.7959993049254388, 1.3440005864415039 ], [ 1.0839991879333195, -0.5680003706978888 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.01 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 0.004746494814753532, 0.0010564705589786172 ], [ -0.0028284345753490925, -0.0014931741170585155 ], [ -0.0030269050039350986, -7.429000688716769E-4 ], [ 0.0018466149922460318, 9.022468002513051E-4 ], [ 0.0015483265742659569, 0.002449278486892581 ], [ -0.0022323569282889366, -5.940581322647631E-4 ], [ 0.0029772452544420958, -8.553110528737307E-4 ], [ 0.004218755755573511, 0.0012864430900663137 ], ... ],
[ [ 8.931321790441871E-4, -4.452368011698127E-4 ], [ 0.002347162924706936, -6.378487887559459E-5 ], [ 0.0034172513987869024, -3.2313112751580775E-4 ], [ -0.0036333578173071146, 6.75366900395602E-5 ], [ -0.002507756929844618, -8.616814156994224E-4 ], [ 0.0032618651166558266, 0.001152309705503285 ], [ -4.5420165406540036E-5, -0.0016449949471279979 ], [ 0.002393205650150776, 8.260895265266299E-4 ], ... ],
[ [ -0.001813568058423698, -0.002434269990772009 ], [ -0.0031243814155459404, -9.83250210992992E-4 ], [ -1.4307841775007546E-4, -5.513473297469318E-4 ], [ -0.0054260496981441975, -5.908437306061387E-4 ], [ -0.001986696617677808, -0.0019470733823254704 ], [ -0.00276072951965034, -0.002215275075286627 ], [ 2.0074055646546185E-4, 0.0021353005431592464 ], [ -9.87459352472797E-5, 6.205149693414569E-4 ], ... ],
[ [ -0.004621185828000307, -1.858071773312986E-4 ], [ 1.3950312859378755E-4, -8.151750662364066E-4 ], [ 0.0015200512716546655, 0.001240087440237403 ], [ -0.0041394769214093685, -6.562801427207887E-4 ], [ 0.004264061339199543, 3.4275284269824624E-4 ], [ 0.0038924722466617823, -3.767110756598413E-4 ], [ 0.0019442765042185783, -1.9140008953399956E-4 ], [ 0.0015631387941539288, 0.0019652547780424356 ], ... ],
[ [ 0.0017279312014579773, 8.95814795512706E-4 ], [ -6.73805654514581E-4, -0.001784831052646041 ], [ 0.001786699634976685, 0.0016453384887427092 ], [ -0.0023845070973038673, -9.549050009809434E-4 ], [ 0.0021343857515603304, -2.163571771234274E-4 ], [ 0.0022857319563627243, 2.766715479083359E-4 ], [ -0.00245643756352365, 4.633538774214685E-4 ], [ -0.0033368917647749186, -0.001179882907308638 ], ... ],
[ [ -8.956296369433403E-4, 0.0015087812207639217 ], [ -0.0033813531044870615, -0.0010265602031722665 ], [ 0.005389036610722542, 1.6865473298821598E-4 ], [ -5.700369947589934E-4, -0.0012767503503710032 ], [ 0.001616926398128271, 9.279971127398312E-4 ], [ -0.005421043839305639, -1.1162266309838742E-4 ], [ -0.00422505522146821, 4.0276176878251135E-4 ], [ -0.002440099371597171, 0.0010313392849639058 ], ... ],
[ [ -0.0027031488716602325, 6.850501522421837E-4 ], [ -0.00165308837313205, 4.657554381992668E-4 ], [ 0.001257349387742579, -4.666345485020429E-4 ], [ -0.00240439479239285, -0.0015906647313386202 ], [ -0.0014587851474061608, -0.0015541084576398134 ], [ -0.0037563538644462824, -5.326740792952478E-4 ], [ 0.0021677864715456963, 0.0021038043778389692 ], [ -8.672381518408656E-4, 7.434129947796464E-4 ], ... ],
[ [ 0.0017632154049351811, 9.50617075432092E-4 ], [ 0.001056369743309915, -0.0012591260019689798 ], [ -0.0030581760220229626, -0.0014586118049919605 ], [ -0.002212473191320896, 4.1701277950778604E-5 ], [ -0.0019428537925705314, 0.0012610336998477578 ], [ -0.004023359157145023, -2.268655807711184E-4 ], [ -0.0022093625739216805, -0.002426337217912078 ], [ 0.001575098023749888, 0.0012988991802558303 ], ... ],
...
]
TrainingTester.java:432 executed in 0.01 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -0.2953673198903306], [61.0, -0.21943450427757108]; valueStats=DoubleSummaryStatistics{count=102, sum=51.863103, min=0.506562, average=0.508462, max=0.603345}
Plotting 61 points for GD
Plotting 32 points for CjGD
Plotting 9 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -0.2953673198903306], [30.685, -0.21943450427757108]; valueStats=DoubleSummaryStatistics{count=102, sum=51.863103, min=0.506562, average=0.508462, max=0.603345}
Plotting 61 points for GD
Plotting 32 points for CjGD
Plotting 9 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "NonConverged", "value": 4.4586985555288694E-4 }, "CjGD": { "type": "Converged", "value": 9.247246535149825E-18 }, "GD": { "type": "Converged", "value": 8.923721100820277E-18 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 0.5065621257623028 }, "CjGD": { "type": "NonConverged", "value": 0.5065621255098751 }, "GD": { "type": "NonConverged", "value": 0.5065620842719571 } }}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "NonConverged", "value": 4.4586985555288694E-4 }, "CjGD": { "type": "Converged", "value": 9.247246535149825E-18 }, "GD": { "type": "Converged", "value": 8.923721100820277E-18 } }, "model":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "complete":{ "LBFGS": { "type": "NonConverged", "value": 0.5065621257623028 }, "CjGD": { "type": "NonConverged", "value": 0.5065621255098751 }, "GD": { "type": "NonConverged", "value": 0.5065620842719571 } }} | OK |
{
"result": "OK",
"performance": {
"execution_time": "189.533",
"gc_time": "2.156"
},
"created_on": 1586746995050,
"file_name": "trainingTest",
"report": {
"simpleName": "Float",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayerTest.Float",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "NonConverged",
"value": 4.4586985555288694E-4
},
"CjGD": {
"type": "Converged",
"value": 9.247246535149825E-18
},
"GD": {
"type": "Converged",
"value": 8.923721100820277E-18
}
},
"model": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
},
"complete": {
"LBFGS": {
"type": "NonConverged",
"value": 0.5065621257623028
},
"CjGD": {
"type": "NonConverged",
"value": 0.5065621255098751
},
"GD": {
"type": "NonConverged",
"value": 0.5065620842719571
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer/Float/trainingTest/202004130315",
"id": "6a919727-afaa-4519-ab07-c2e311b6a602",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "ConvolutionLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayer",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer.java",
"javaDoc": ""
}
}