Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
This is a network apply the following layout:
LayerTests.java:203 executed in 0.07 seconds (0.000 gc):
return Graphviz.fromGraph((Graph) TestUtil.toGraph(((DAGNetwork) layer).addRef())).height(400).width(600)
.render(Format.PNG).toImage();
executing command [/bin/sh, -c, dot -Tsvg /tmp/GraphvizJava/DotEngine5069896863839555129/dotfile.dot -ooutfile.svg]
Returns
Using Seed 7167703419518375936
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.04 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.4 ], [ 0.436 ], [ -0.828 ], [ 0.052 ], [ 0.12 ], [ 1.636 ], [ 1.84 ], [ 0.436 ], ... ],
[ [ 1.84 ], [ 1.948 ], [ -0.672 ], [ -1.952 ], [ -1.716 ], [ -1.252 ], [ 1.876 ], [ -0.4 ], ... ],
[ [ 0.668 ], [ 1.528 ], [ 1.552 ], [ 1.172 ], [ -0.352 ], [ -1.016 ], [ -0.296 ], [ -1.612 ], ... ],
[ [ -1.464 ], [ 0.888 ], [ 1.488 ], [ -1.028 ], [ 0.972 ], [ -0.416 ], [ -0.808 ], [ -1.66 ], ... ],
[ [ -0.764 ], [ -0.324 ], [ -1.06 ], [ -0.704 ], [ -0.648 ], [ -1.176 ], [ -1.66 ], [ 0.856 ], ... ],
[ [ 0.384 ], [ -1.48 ], [ 0.296 ], [ -0.24 ], [ 0.876 ], [ 0.26 ], [ 0.864 ], [ 0.236 ], ... ],
[ [ -1.108 ], [ -1.872 ], [ -1.052 ], [ 0.432 ], [ -1.376 ], [ 1.04 ], [ -0.268 ], [ 0.324 ], ... ],
[ [ -1.652 ], [ 0.396 ], [ -1.42 ], [ 1.048 ], [ 0.724 ], [ 1.384 ], [ 1.744 ], [ 0.528 ], ... ],
...
]
[
[ [ -1.124 ], [ 0.8 ], [ 0.38 ], [ 1.2 ], [ -0.744 ], [ 0.56 ], [ 0.332 ], [ 0.64 ], ... ],
[ [ -1.852 ], [ 1.96 ], [ -1.884 ], [ 1.212 ], [ 1.904 ], [ 0.012 ], [ 1.34 ], [ -1.564 ], ... ],
[ [ -1.22 ], [ 0.42 ], [ -0.34 ], [ 0.472 ], [ -1.94 ], [ -0.468 ], [ 1.956 ], [ -0.948 ], ... ],
[ [ -1.244 ], [ -1.272 ], [ -0.968 ], [ -0.864 ], [ -1.164 ], [ 0.964 ], [ -0.672 ], [ 1.764 ], ... ],
[ [ 1.22 ], [ 0.392 ], [ -0.244 ], [ 0.392 ], [ -0.928 ], [ 1.1 ], [ -1.076 ], [ 0.756 ], ... ],
[ [ 1.244 ], [ 1.016 ], [ -1.1 ], [ -1.328 ], [ -1.312 ], [ -1.504 ], [ -1.128 ], [ 0.804 ], ... ],
[ [ 1.752 ], [ 0.716 ], [ -0.02 ], [ 0.448 ], [ 1.588 ], [ -0.664 ], [ 1.716 ], [ 1.532 ], ... ],
[ [ 1.084 ], [ 1.328 ], [ -0.396 ], [ 1.16 ], [ -0.408 ], [ -1.376 ], [ 1.64 ], [ 1.924 ], ... ],
...
]
[
[ [ 1.468 ], [ -0.176 ], [ 0.14 ], [ 1.868 ], [ 0.136 ], [ 1.288 ], [ 1.784 ], [ -0.692 ], ... ],
[ [ 0.592 ], [ 0.308 ], [ 0.708 ], [ 0.424 ], [ -0.176 ], [ -0.276 ], [ -0.468 ], [ 0.388 ], ... ],
[ [ -1.88 ], [ 1.632 ], [ 0.668 ], [ -1.556 ], [ -0.78 ], [ 1.7 ], [ 0.156 ], [ 1.5 ], ... ],
[ [ -1.496 ], [ -0.88 ], [ -0.384 ], [ -0.768 ], [ -0.316 ], [ -1.12 ], [ 0.724 ], [ -0.8 ], ... ],
[ [ 0.392 ], [ -0.7 ], [ 0.14 ], [ 0.404 ], [ 1.54 ], [ -1.532 ], [ -0.736 ], [ 1.416 ], ... ],
[ [ 1.4 ], [ 0.288 ], [ 1.988 ], [ -0.516 ], [ 0.24 ], [ 0.98 ], [ 0.568 ], [ -1.524 ], ... ],
[ [ -0.284 ], [ 1.38 ], [ -0.1 ], [ -0.52 ], [ -0.092 ], [ 1.84 ], [ 1.06 ], [ 1.612 ], ... ],
[ [ 0.252 ], [ 0.172 ], [ -1.908 ], [ -1.98 ], [ -0.492 ], [ 0.752 ], [ -1.268 ], [ -0.148 ], ... ],
...
]
[
[ [ 1.392 ], [ 1.096 ], [ -1.096 ], [ -0.952 ], [ -0.096 ], [ -0.772 ], [ -1.252 ], [ -1.124 ], ... ],
[ [ 1.828 ], [ -1.904 ], [ 0.676 ], [ 0.596 ], [ 1.124 ], [ 0.712 ], [ -1.744 ], [ -0.688 ], ... ],
[ [ -0.848 ], [ -1.884 ], [ -0.188 ], [ -1.076 ], [ 0.08 ], [ -0.292 ], [ 0.656 ], [ 1.36 ], ... ],
[ [ -1.288 ], [ 0.18 ], [ -1.872 ], [ -1.172 ], [ 0.052 ], [ 0.236 ], [ 1.132 ], [ 0.988 ], ... ],
[ [ 0.428 ], [ -1.44 ], [ 0.536 ], [ 1.948 ], [ -0.704 ], [ 0.88 ], [ -0.492 ], [ 1.004 ], ... ],
[ [ 0.06 ], [ 1.34 ], [ 0.748 ], [ 1.26 ], [ 0.212 ], [ 1.752 ], [ 0.056 ], [ -1.496 ], ... ],
[ [ 1.464 ], [ 0.18 ], [ 0.78 ], [ -1.132 ], [ 0.072 ], [ -1.188 ], [ -0.092 ], [ 0.348 ], ... ],
[ [ 1.508 ], [ 0.312 ], [ -1.576 ], [ 0.108 ], [ -1.336 ], [ -1.304 ], [ -0.752 ], [ 0.036 ], ... ],
...
]
[
[ [ -1.356 ], [ 1.052 ], [ -0.26 ], [ 0.832 ], [ 1.676 ], [ 1.272 ], [ -0.644 ], [ 0.456 ], ... ],
[ [ 0.868 ], [ 1.92 ], [ -1.844 ], [ 1.568 ], [ 1.696 ], [ 0.664 ], [ -1.812 ], [ 0.02 ], ... ],
[ [ 1.216 ], [ -0.976 ], [ 1.396 ], [ -1.976 ], [ 0.86 ], [ 1.516 ], [ 1.688 ], [ 1.852 ], ... ],
[ [ -1.124 ], [ 0.328 ], [ 0.284 ], [ 1.964 ], [ 1.084 ], [ -0.776 ], [ -1.144 ], [ -0.744 ], ... ],
[ [ -1.484 ], [ 1.036 ], [ -1.48 ], [ 1.168 ], [ -1.984 ], [ -1.14 ], [ 0.82 ], [ 0.7 ], ... ],
[ [ 0.372 ], [ -0.096 ], [ 1.2 ], [ -0.384 ], [ 0.18 ], [ -0.088 ], [ -1.604 ], [ -0.268 ], ... ],
[ [ -0.308 ], [ -1.952 ], [ -0.8 ], [ -1.708 ], [ -0.848 ], [ 0.864 ], [ -0.732 ], [ -1.624 ], ... ],
[ [ 1.912 ], [ 0.744 ], [ -0.252 ], [ 0.4 ], [ 0.868 ], [ 1.504 ], [ 0.372 ], [ 1.848 ], ... ],
...
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 30.43 seconds (0.936 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 7564892144616
Reset training subject: 7565370595542
Constructing line search parameters: GD
th(0)=10.679692229546669;dx=-8.899743524622212E-6
New Minimum: 10.679692229546669 > 10.679653881748727
WOLFE (weak): th(2.154434690031884)=10.679653881748727; dx=-8.899727546358732E-6 evalInputDelta=3.834779794154031E-5
New Minimum: 10.679653881748727 > 10.679615534019636
WOLFE (weak): th(4.308869380063768)=10.679615534019636; dx=-8.899711568095257E-6 evalInputDelta=7.669552703326588E-5
New Minimum: 10.679615534019636 > 10.679462143791753
WOLFE (weak): th(12.926608140191302)=10.679462143791753; dx=-8.899647655041324E-6 evalInputDelta=2.3008575491623162E-4
New Minimum: 10.679462143791753 > 10.678771901398235
WOLFE (weak): th(51.70643256076521)=10.678771901398235; dx=-8.899360046298579E-6 evalInputDelta=9.20328148433569E-4
New Minimum: 10.678771901398235 > 10.675090985370428
WOLFE (weak): th(258.53216280382605)=10.675090985370428; dx=-8.897826133004047E-6 evalInputDelta=0.004601244176241082
New Minimum: 10.675090985370428 > 10.6520996357113
WOLFE (weak): th(1551.1929768229563)=10.6520996357113; dx=-8.88823917491313E-6 evalInputDelta=0.027592593835368717
New Minimum: 10.6520996357113 > 10.487293582290906
WOLFE (weak): th(10858.350837760694)=10.487293582290906; dx=-8.81921307665855E-6 evalInputDelta=0.1923986472557626
New Minimum: 10.487293582290906 > 9.189471011498833
WOLFE (weak): th(86866.80670208555)=9.189471011498833; dx=-8.255499940912634E-6 evalInputDelta=1.4902212180478358
New Minimum: 9.189471011498833 > 1.297064832687612
END: th(781801.26031877)=1.297064832687612; dx=-3.1015512712360054E-6 evalInputDelta=9.382627396859057
Fitness changed from 10.679692229546669 to 1.297064832687612
Iteration 1 complete. Error: 1.297064832687612 Total: 4.8380; Orientation: 0.1705; Line Search: 3.9005
th(0)=1.297064832687612;dx=-1.080887360573012E-6
New Minimum: 1.297064832687612 > 0.21129996716625601
WOLF (strong): th(1684339.7559414052)=0.21129996716625601; dx=4.3626393368340174E-7 evalInputDelta=1.0857648655213559
New Minimum: 0.21129996716625601 > 0.11533283975342627
END: th(842169.8779707026)=0.11533283975342627; dx=-3.2231171344480405E-7 evalInputDelta=1.1817319929341856
Fitness changed from 1.297064832687612 to 0.11533283975342627
Iteration 2 complete. Error: 0.11533283975342627 Total: 1.1118; Orientation: 0.1602; Line Search: 0.8201
th(0)=0.11533283975342627;dx=-9.611069979452184E-8
New Minimum: 0.11533283975342627 > 0.030233811944322193
WOLF (strong): th(1814400.0)=0.030233811944322193; dx=4.920867829479529E-8 evalInputDelta=0.08509902780910408
New Minimum: 0.030233811944322193 > 0.006866455947559982
END: th(907200.0)=0.006866455947559982; dx=-2.345101074986334E-8 evalInputDelta=0.10846638380586629
Fitness changed from 0.11533283975342627 to 0.006866455947559982
Iteration 3 complete. Error: 0.006866455947559982 Total: 1.2255; Orientation: 0.1553; Line Search: 0.9418
th(0)=0.006866455947559982;dx=-5.722046622966654E-9
New Minimum: 0.006866455947559982 > 0.0027145150978300898
WOLF (strong): th(1954503.150796925)=0.0027145150978300898; dx=3.5977518383627025E-9 evalInputDelta=0.004151940849729892
New Minimum: 0.0027145150978300898 > 2.365916583298959E-4
END: th(977251.5753984625)=2.365916583298959E-4; dx=-1.0621473923019773E-9 evalInputDelta=0.006629864289230086
Fitness changed from 0.006866455947559982 to 2.365916583298959E-4
Iteration 4 complete. Error: 2.365916583298959E-4 Total: 1.0522; Orientation: 0.1573; Line Search: 0.7615
th(0)=2.365916583298959E-4;dx=-1.9715971527491342E-10
New Minimum: 2.365916583298959E-4 > 1.3469193967226558E-4
WOLF (strong): th(2105424.6949267564)=1.3469193967226558E-4; dx=1.4876106254552853E-10 evalInputDelta=1.0189971865763033E-4
New Minimum: 1.3469193967226558E-4 > 3.5642619732231063E-6
END: th(1052712.3474633782)=3.5642619732231063E-6; dx=-2.4199326364692258E-11 evalInputDelta=2.3302739635667281E-4
Fitness changed from 2.365916583298959E-4 to 3.5642619732231063E-6
Iteration 5 complete. Error: 3.5642619732231063E-6 Total: 1.0677; Orientation: 0.1558; Line Search: 0.7799
Low gradient: 1.7234321312483575E-6
th(0)=3.5642619732231063E-6;dx=-2.9702183110192515E-12
New Minimum: 3.5642619732231063E-6 > 2.8232519089900263E-6
WOLF (strong): th(2268000.0)=2.8232519089900263E-6; dx=2.643494296807138E-12 evalInputDelta=7.4101006423308E-7
New Minimum: 2.8232519089900263E-6 > 1.0781892468999791E-8
END: th(1134000.0)=1.0781892468999791E-8; dx=-1.6336200710605832E-13 evalInputDelta=3.5534800807541063E-6
Fitness changed from 3.5642619732231063E-6 to 1.0781892468999791E-8
Iteration 6 complete. Error: 1.0781892468999791E-8 Total: 1.0285; Orientation: 0.1537; Line Search: 0.7491
Low gradient: 9.47887672186592E-8
th(0)=1.0781892468999791E-8;dx=-8.984910390833166E-15
Armijo: th(2443128.938496156)=1.1570839164079003E-8; dx=9.30783509719958E-15 evalInputDelta=-7.889466950792126E-10
New Minimum: 1.0781892468999791E-8 > 3.481849949942262E-12
WOLF (strong): th(1221564.469248078)=3.481849949942262E-12; dx=1.6146235318322684E-16 evalInputDelta=1.0778410619049848E-8
END: th(407188.156416026)=4.706225825629965E-9; dx=-5.9361194761610344E-15 evalInputDelta=6.075666643369826E-9
Fitness changed from 1.0781892468999791E-8 to 3.481849949942262E-12
Iteration 7 complete. Error: 3.481849949942262E-12 Total: 1.2236; Orientation: 0.1539; Line Search: 0.9422
Low gradient: 1.703391213125125E-9
th(0)=3.481849949942262E-12;dx=-2.9015416249518843E-18
New Minimum: 3.481849949942262E-12 > 2.5185603925162607E-13
END: th(877260.2895528153)=2.5185603925162607E-13; dx=-7.803689199048249E-19 evalInputDelta=3.229993910690636E-12
Fitness changed from 3.481849949942262E-12 to 2.5185603925162607E-13
Iteration 8 complete. Error: 2.5185603925162607E-13 Total: 0.8608; Orientation: 0.1715; Line Search: 0.5599
Low gradient: 4.581266557511017E-10
th(0)=2.5185603925162607E-13;dx=-2.098800327096889E-19
New Minimum: 2.5185603925162607E-13 > 8.326990297931028E-14
WOLF (strong): th(1890000.0000000002)=8.326990297931028E-14; dx=1.2068101880933287E-19 evalInputDelta=1.6858613627231577E-13
New Minimum: 8.326990297931028E-14 > 1.1372874271982204E-14
END: th(945000.0000000001)=1.1372874271982204E-14; dx=-4.459950694987939E-20 evalInputDelta=2.404831649796439E-13
Fitness changed from 2.5185603925162607E-13 to 1.1372874271982204E-14
Iteration 9 complete. Error: 1.1372874271982204E-14 Total: 1.2204; Orientation: 0.1712; Line Search: 0.9058
Zero gradient: 9.735191434508023E-11
th(0)=1.1372874271982204E-14;dx=-9.477395226651848E-21
New Minimum: 1.1372874271982204E-14 > 5.518979390262958E-15
WOLF (strong): th(2035940.7820801304)=5.518979390262958E-15; dx=6.602117648228988E-21 evalInputDelta=5.853894881719246E-15
New Minimum: 5.518979390262958E-15 > 2.616928266282238E-16
END: th(1017970.3910400652)=2.616928266282238E-16; dx=-1.4376387892233093E-21 evalInputDelta=1.111118144535398E-14
Fitness changed from 1.1372874271982204E-14 to 2.616928266282238E-16
Iteration 10 complete. Error: 2.616928266282238E-16 Total: 1.0256; Orientation: 0.1555; Line Search: 0.7418
Zero gradient: 1.4767442416461962E-11
th(0)=2.616928266282238E-16;dx=-2.180773555235196E-22
New Minimum: 2.616928266282238E-16 > 1.7925020172715508E-16
WOLF (strong): th(2193150.7238820386)=1.7925020172715508E-16; dx=1.804864029267293E-22 evalInputDelta=8.244262490106872E-17
New Minimum: 1.7925020172715508E-16 > 1.9439153335183064E-18
END: th(1096575.3619410193)=1.9439153335183064E-18; dx=-1.879547630183237E-23 evalInputDelta=2.597489112947055E-16
Fitness changed from 2.616928266282238E-16 to 1.9439153335183064E-18
Iteration 11 complete. Error: 1.9439153335183064E-18 Total: 1.0250; Orientation: 0.1533; Line Search: 0.7431
Zero gradient: 1.2727644890546677E-12
th(0)=1.9439153335183064E-18;dx=-1.6199294445985826E-24
New Minimum: 1.9439153335183064E-18 > 1.8243189913761914E-18
WOLF (strong): th(2362500.000000001)=1.8243189913761914E-18; dx=1.5693066543516066E-24 evalInputDelta=1.1959634214211497E-19
New Minimum: 1.8243189913761914E-18 > 4.745886056512597E-22
END: th(1181250.0000000005)=4.745886056512597E-22; dx=-2.531139493637484E-26 evalInputDelta=1.943440744912655E-18
Fitness changed from 1.9439153335183064E-18 to 4.745886056512597E-22
Iteration 12 complete. Error: 4.745886056512597E-22 Total: 1.0245; Orientation: 0.1538; Line Search: 0.7444
Zero gradient: 1.9886943071004735E-14
th(0)=4.745886056512597E-22;dx=-3.9549050470938297E-28
Armijo: th(2544925.977600164)=5.961445497368376E-22; dx=4.432545439243053E-28 evalInputDelta=-1.215559440855779E-22
New Minimum: 4.745886056512597E-22 > 1.7305630569939882E-24
WOLF (strong): th(1272462.988800082)=1.7305630569939882E-24; dx=2.3882023454313914E-29 evalInputDelta=4.728580425942658E-22
END: th(424154.3296000273)=1.9838346255757013E-22; dx=-2.556996541014933E-28 evalInputDelta=2.7620514309368956E-22
Fitness changed from 4.745886056512597E-22 to 1.7305630569939882E-24
Iteration 13 complete. Error: 1.7305630569939882E-24 Total: 1.2227; Orientation: 0.1533; Line Search: 0.9418
Zero gradient: 1.2008896205848078E-15
th(0)=1.7305630569939882E-24;dx=-1.4421358808283231E-30
New Minimum: 1.7305630569939882E-24 > 9.84288840805049E-26
END: th(913812.8016175163)=9.84288840805049E-26; dx=-3.4393290511840637E-31 evalInputDelta=1.6321341729134834E-24
Fitness changed from 1.7305630569939882E-24 to 9.84288840805049E-26
Iteration 14 complete. Error: 9.84288840805049E-26 Total: 0.8445; Orientation: 0.1552; Line Search: 0.4996
Zero gradient: 2.8639844634195807E-16
th(0)=9.84288840805049E-26;dx=-8.202407006708741E-32
New Minimum: 9.84288840805049E-26 > 4.039490407951888E-26
WOLF (strong): th(1968750.0000000012)=4.039490407951888E-26; dx=5.254643721516994E-32 evalInputDelta=5.803398000098602E-26
New Minimum: 4.039490407951888E-26 > 3.177957656258276E-27
END: th(984375.0000000006)=3.177957656258276E-27; dx=-1.4738505409770514E-32 evalInputDelta=9.525092642424662E-26
Fitness changed from 9.84288840805049E-26 to 3.177957656258276E-27
Iteration 15 complete. Error: 3.177957656258276E-27 Total: 1.0415; Orientation: 0.1558; Line Search: 0.7576
Zero gradient: 5.146161721984548E-17
th(0)=3.177957656258276E-27;dx=-2.6482980468818924E-33
New Minimum: 3.177957656258276E-27 > 1.8711020509027683E-27
WOLF (strong): th(2120771.648000137)=1.8711020509027683E-27; dx=2.0320783307928545E-33 evalInputDelta=1.3068556053555078E-27
New Minimum: 1.8711020509027683E-27 > 4.3014415284526367E-29
END: th(1060385.8240000685)=4.3014415284526367E-29; dx=-3.080723565793875E-34 evalInputDelta=3.13494324097375E-27
Fitness changed from 3.177957656258276E-27 to 4.3014415284526367E-29
Iteration 16 complete. Error: 4.3014415284526367E-29 Total: 1.0345; Orientation: 0.1539; Line Search: 0.7510
Zero gradient: 5.98709830138429E-18
th(0)=4.3014415284526367E-29;dx=-3.58453460704387E-35
New Minimum: 4.3014415284526367E-29 > 3.5078421075593005E-29
WOLF (strong): th(2284532.0040437914)=3.5078421075593005E-29; dx=3.236590079699119E-35 evalInputDelta=7.935994208933362E-30
New Minimum: 3.5078421075593005E-29 > 9.952661699924018E-32
END: th(1142266.0020218957)=9.952661699924018E-32; dx=-1.6480117601299946E-36 evalInputDelta=4.291488866752713E-29
Fitness changed from 4.3014415284526367E-29 to 9.952661699924018E-32
Iteration 17 complete. Error: 9.952661699924018E-32 Total: 1.0494; Orientation: 0.1560; Line Search: 0.7652
Zero gradient: 2.8799105454747517E-19
th(0)=9.952661699924018E-32;dx=-8.293884749936695E-38
Armijo: th(2460937.500000002)=1.0036006601091495E-31; dx=8.321800381998691E-38 evalInputDelta=-8.334490116747678E-34
New Minimum: 9.952661699924018E-32 > 1.3964120784203847E-36
WOLF (strong): th(1230468.750000001)=1.3964120784203847E-36; dx=3.3359672886856766E-41 evalInputDelta=9.952522058716175E-32
END: th(410156.2500000003)=7.270841287883376E-32; dx=-6.841123190969081E-38 evalInputDelta=2.681820412040642E-32
Fitness changed from 9.952661699924018E-32 to 1.3964120784203847E-36
Iteration 18 complete. Error: 1.3964120784203847E-36 Total: 1.3849; Orientation: 0.1619; Line Search: 1.0780
Zero gradient: 1.0787384910241164E-21
th(0)=1.3964120784203847E-36;dx=-1.1636767320169874E-42
New Minimum: 1.3964120784203847E-36 > 2.994752655572536E-38
END: th(883654.8533333904)=2.994752655572536E-38; dx=-6.397853206284595E-44 evalInputDelta=1.3664645518646593E-36
Fitness changed from 1.3964120784203847E-36 to 2.994752655572536E-38
Iteration 19 complete. Error: 2.994752655572536E-38 Total: 0.7804; Orientation: 0.1550; Line Search: 0.4993
Zero gradient: 1.5797554282157457E-22
th(0)=2.994752655572536E-38;dx=-2.495627212977114E-44
New Minimum: 2.994752655572536E-38 > 2.796526274043396E-38
WOLF (strong): th(1903776.6700364929)=2.796526274043396E-38; dx=2.3900122160165615E-44 evalInputDelta=1.9822638152913984E-39
New Minimum: 2.796526274043396E-38 > 9.706009195173802E-41
END: th(951888.3350182464)=9.706009195173802E-41; dx=-3.911951981736771E-46 evalInputDelta=2.985046646377362E-38
Fitness changed from 2.994752655572536E-38 to 9.706009195173802E-41
Iteration 20 complete. Error: 9.706009195173802E-41 Total: 1.2472; Orientation: 0.1717; Line Search: 0.9301
Zero gradient: 8.993520443062422E-24
th(0)=9.706009195173802E-41;dx=-8.088340995978168E-47
New Minimum: 9.706009195173802E-41 > 4.878811361842582E-41
WOLF (strong): th(2050781.2500000016)=4.878811361842582E-41; dx=5.734507385820472E-47 evalInputDelta=4.827197833331219E-41
New Minimum: 4.878811361842582E-41 > 2.055007077618134E-42
END: th(1025390.6250000008)=2.055007077618134E-42; dx=-1.1769168050788483E-47 evalInputDelta=9.500508487411989E-41
Fitness changed from 9.706009195173802E-41 to 2.055007077618134E-42
Iteration 21 complete. Error: 2.055007077618134E-42 Total: 1.0855; Orientation: 0.1539; Line Search: 0.8027
Zero gradient: 1.3086274863440367E-24
th(0)=2.055007077618134E-42;dx=-1.712505898015112E-48
New Minimum: 2.055007077618134E-42 > 1.4532863831634078E-42
WOLF (strong): th(2209137.1333334763)=1.4532863831634078E-42; dx=1.4401277439497008E-48 evalInputDelta=6.017206944547261E-43
New Minimum: 1.4532863831634078E-42 > 1.2996718825565098E-44
END: th(1104568.5666667381)=1.2996718825565098E-44; dx=-1.3618907703270565E-49 evalInputDelta=2.0420103587925688E-42
Fitness changed from 2.055007077618134E-42 to 1.2996718825565098E-44
Iteration 22 complete. Error: 1.2996718825565098E-44 Total: 1.0345; Orientation: 0.1534; Line Search: 0.7554
Zero gradient: 1.040701639342624E-25
th(0)=1.2996718825565098E-44;dx=-1.083059902130425E-50
New Minimum: 1.2996718825565098E-44 > 1.2561159557768684E-44
WOLF (strong): th(2379720.837545616)=1.2561159557768684E-44; dx=1.0647569457111486E-50 evalInputDelta=4.355592677964144E-46
New Minimum: 1.2561159557768684E-44 > 9.279215665550305E-49
END: th(1189860.418772808)=9.279215665550305E-49; dx=-9.15147820963824E-53 evalInputDelta=1.2995790903998544E-44
Fitness changed from 1.2996718825565098E-44 to 9.279215665550305E-49
Iteration 23 complete. Error: 9.279215665550305E-49 Total: 1.0173; Orientation: 0.1516; Line Search: 0.7374
Zero gradient: 8.793565671155202E-28
th(0)=9.279215665550305E-49;dx=-7.732679721291922E-55
Armijo: th(2563476.5625000023)=1.1979650021706304E-48; dx=8.786106304417158E-55 evalInputDelta=-2.7004343561559986E-49
New Minimum: 9.279215665550305E-49 > 4.305263916385914E-51
WOLF (strong): th(1281738.2812500012)=4.305263916385914E-51; dx=5.267132915626174E-56 evalInputDelta=9.236163026386446E-49
END: th(427246.0937500004)=3.847965722429391E-49; dx=-4.979548717007075E-55 evalInputDelta=5.431249943120914E-49
Fitness changed from 9.279215665550305E-49 to 4.305263916385914E-51
Iteration 24 complete. Error: 4.305263916385914E-51 Total: 1.2088; Orientation: 0.1523; Line Search: 0.9325
Zero gradient: 5.989757866827002E-29
th(0)=4.305263916385914E-51;dx=-3.587719930321596E-57
New Minimum: 4.305263916385914E-51 > 2.3360509519582303E-52
END: th(920473.8055556153)=2.3360509519582303E-52; dx=-8.357180823792235E-58 evalInputDelta=4.071658821190091E-51
Fitness changed from 4.305263916385914E-51 to 2.3360509519582303E-52
Iteration 25 complete. Error: 2.3360509519582303E-52 Total: 0.7656; Orientation: 0.1547; Line Search: 0.4823
Final threshold in iteration 25: 2.3360509519582303E-52 (> 0.0) after 30.421s (< 30.000s)
Returns
2.3360509519582303E-52
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 30.57 seconds (0.602 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 7595360136248
Reset training subject: 7595484915941
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=10.679692229546669}, derivative=-8.899743524622212E-6}
New Minimum: 10.679692229546669 > 10.679692229546665
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=10.679692229546665}, derivative=-8.899743524622212E-6}, evalInputDelta = -3.552713678800501E-15
New Minimum: 10.679692229546665 > 10.679692229546655
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=10.679692229546655}, derivative=-8.899743524622209E-6}, evalInputDelta = -1.4210854715202004E-14
New Minimum: 10.679692229546655 > 10.67969222954658
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=10.67969222954658}, derivative=-8.899743524622173E-6}, evalInputDelta = -8.881784197001252E-14
New Minimum: 10.67969222954658 > 10.679692229546056
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=10.679692229546056}, derivative=-8.899743524621948E-6}, evalInputDelta = -6.128431095930864E-13
New Minimum: 10.679692229546056 > 10.679692229542393
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=10.679692229542393}, derivative=-8.899743524620444E-6}, evalInputDelta = -4.275690912436403E-12
New Minimum: 10.679692229542393 > 10.679692229516752
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=10.679692229516752}, derivative=-8.899743524609749E-6}, evalInputDelta = -2.991740188917902E-11
New Minimum: 10.679692229516752 > 10.679692229337258
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=10.679692229337258}, derivative=-8.899743524534995E-6}, evalInputDelta = -2.0941115508321673E-10
New Minimum: 10.679692229337258 > 10.679692228080803
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=10.679692228080803}, derivative=-8.899743524011435E-6}, evalInputDelta = -1.4658656510846413E-9
New Minimum: 10.679692228080803 > 10.679692219285617
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=10.679692219285617}, derivative=-8.89974352034677E-6}, evalInputDelta = -1.0261052452165131E-8
New Minimum: 10.679692219285617 > 10.679692157719318
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=10.679692157719318}, derivative=-8.899743494694153E-6}, evalInputDelta = -7.182735117794437E-8
New Minimum: 10.679692157719318 > 10.67969172675522
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=10.67969172675522}, derivative=-8.899743315125804E-6}, evalInputDelta = -5.027914493638264E-7
New Minimum: 10.67969172675522 > 10.679688710006783
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=10.679688710006783}, derivative=-8.899742058147144E-6}, evalInputDelta = -3.519539886198686E-6
New Minimum: 10.679688710006783 > 10.679667592779648
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=10.679667592779648}, derivative=-8.899733259296718E-6}, evalInputDelta = -2.4636767021135597E-5
New Minimum: 10.679667592779648 > 10.67951977277429
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=10.67951977277429}, derivative=-8.899671667343611E-6}, evalInputDelta = -1.7245677237909263E-4
New Minimum: 10.67951977277429 > 10.678485061381185
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=10.678485061381185}, derivative=-8.899240523672138E-6}, evalInputDelta = -0.0012071681654841626
New Minimum: 10.678485061381185 > 10.67124348520506
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=10.67124348520506}, derivative=-8.896222517971738E-6}, evalInputDelta = -0.00844874434160836
New Minimum: 10.67124348520506 > 10.620621227177148
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=10.620621227177148}, derivative=-8.875096478068812E-6}, evalInputDelta = -0.0590710023695209
New Minimum: 10.620621227177148 > 10.269635406024824
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=10.269635406024824}, derivative=-8.72721419874827E-6}, evalInputDelta = -0.41005682352184536
New Minimum: 10.269635406024824 > 7.977863925068613
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=7.977863925068613}, derivative=-7.692038243504694E-6}, evalInputDelta = -2.701828304478056
New Minimum: 7.977863925068613 > 0.026797646768442977
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.026797646768442977}, derivative=-4.458065567995528E-7}, evalInputDelta = -10.652894582778226
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=340.8447040299391}, derivative=5.027781525013656E-5}, evalInputDelta = 330.16501180039245
F(613786.6638277846) = LineSearchPoint{point=PointSample{avg=2.5486349460398126}, derivative=-4.34762361887156E-6}, evalInputDelta = -8.131057283506856
F(4296506.646794492) = LineSearchPoint{point=PointSample{avg=71.11157183737832}, derivative=2.296509581563249E-5}, evalInputDelta = 60.431879607831654
F(330500.511291884) = LineSearchPoint{point=PointSample{avg=5.607056173975641}, derivative=-6.448602036910316E-6}, evalInputDelta = -5.072636055571028
F(2313503.579043188) = LineSearchPoint{point=PointSample{avg=9.19558746792583}, derivative=8.258246889361082E-6}, evalInputDelta = -1.4841047616208396
9.19558746792583 <= 10.679692229546669
New Minimum: 0.026797646768442977 > 1.68644777550843E-29
F(1199999.9999999984) = LineSearchPoint{point=PointSample{avg=1.68644777550843E-29}, derivative=-1.1149317595833339E-20}, evalInputDelta = -10.679692229546669
Left bracket at 1199999.9999999984
Converged to left
Fitness changed from 10.679692229546669 to 1.68644777550843E-29
Iteration 1 complete. Error: 1.68644777550843E-29 Total: 8.2409; Orientation: 0.1536; Line Search: 7.7098
Zero gradient: 3.748830679367935E-18
F(0.0) = LineSearchPoint{point=PointSample{avg=1.68644777550843E-29}, derivative=-1.405373146257024E-35}
New Minimum: 1.68644777550843E-29 > 1.186034365526415E-62
F(1199999.9999999984) = LineSearchPoint{point=PointSample{avg=1.186034365526415E-62}, derivative=-8.342919840825525E-54}, evalInputDelta = -1.68644777550843E-29
F(8399999.999999989) = LineSearchPoint{point=PointSample{avg=6.071211991830347E-28}, derivative=8.432238877542259E-35}, evalInputDelta = 5.902567214279505E-28
F(646153.8461538453) = LineSearchPoint{point=PointSample{avg=3.5823020724910224E-30}, derivative=-6.46868108915124E-36}, evalInputDelta = -1.3282175682593278E-29
F(4523076.923076917) = LineSearchPoint{point=PointSample{avg=1.2935690171130212E-28}, derivative=3.892099386249695E-35}, evalInputDelta = 1.1249242395621783E-28
F(347928.9940828398) = LineSearchPoint{point=PointSample{avg=8.505422737797256E-30}, derivative=-9.97501918722642E-36}, evalInputDelta = -8.359055017287043E-30
F(2435502.958579879) = LineSearchPoint{point=PointSample{avg=1.7732608998827666E-29}, derivative=1.4407072350973412E-35}, evalInputDelta = 8.681312437433671E-31
F(187346.38142922145) = LineSearchPoint{point=PointSample{avg=1.2081097116406582E-29}, derivative=-1.1890107703747157E-35}, evalInputDelta = -4.783380638677717E-30
F(1311424.6700045501) = LineSearchPoint{point=PointSample{avg=1.5118122146800976E-31}, derivative=1.2890860018413923E-36}, evalInputDelta = -1.671329653361629E-29
1.5118122146800976E-31 <= 1.68644777550843E-29
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=1.901225982955848E-37}, derivative=1.1326527577595857E-40}, evalInputDelta = -1.68644775649617E-29
Right bracket at 1201240.26687953
Converged to right
Fitness changed from 1.68644777550843E-29 to 1.186034365526415E-62
Iteration 2 complete. Error: 1.186034365526415E-62 Total: 2.8035; Orientation: 0.1536; Line Search: 2.5241
Zero gradient: 9.941639559308178E-35
F(0.0) = LineSearchPoint{point=PointSample{avg=1.186034365526415E-62}, derivative=-9.88361971272013E-69}
New Minimum: 1.186034365526415E-62 > 1.2669663299150763E-68
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=1.2669663299150763E-68}, derivative=1.0215271816297616E-71}, evalInputDelta = -1.1860330985600852E-62
1.2669663299150763E-68 <= 1.186034365526415E-62
Converged to right
Fitness changed from 1.186034365526415E-62 to 1.2669663299150763E-68
Iteration 3 complete. Error: 1.2669663299150763E-68 Total: 0.8571; Orientation: 0.1559; Line Search: 0.5683
Zero gradient: 1.027523
...skipping 6681 bytes...
5405584956E-134 Total: 0.7580; Orientation: 0.1509; Line Search: 0.4839
Zero gradient: 1.4772867980858513E-70
F(0.0) = LineSearchPoint{point=PointSample{avg=2.6188515405584956E-134}, derivative=-2.1823762837987464E-140}
New Minimum: 2.6188515405584956E-134 > 2.797555299724618E-140
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=2.797555299724618E-140}, derivative=2.255607519556262E-143}, evalInputDelta = -2.618848743003196E-134
2.797555299724618E-140 <= 2.6188515405584956E-134
Converged to right
Fitness changed from 2.6188515405584956E-134 to 2.797555299724618E-140
Iteration 15 complete. Error: 2.797555299724618E-140 Total: 0.7609; Orientation: 0.1510; Line Search: 0.4853
Zero gradient: 1.526858239360763E-73
F(0.0) = LineSearchPoint{point=PointSample{avg=2.797555299724618E-140}, derivative=-2.331296083103849E-146}
New Minimum: 2.797555299724618E-140 > 2.988453348274954E-146
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=2.988453348274954E-146}, derivative=2.409524431876545E-149}, evalInputDelta = -2.79755231127127E-140
2.988453348274954E-146 <= 2.797555299724618E-140
Converged to right
Fitness changed from 2.797555299724618E-140 to 2.988453348274954E-146
Iteration 16 complete. Error: 2.988453348274954E-146 Total: 0.7626; Orientation: 0.1534; Line Search: 0.4851
Zero gradient: 1.578093086680608E-76
F(0.0) = LineSearchPoint{point=PointSample{avg=2.988453348274954E-146}, derivative=-2.490377790229129E-152}
New Minimum: 2.988453348274954E-146 > 3.1923777934599684E-152
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=3.1923777934599684E-152}, derivative=2.5739442422822253E-155}, evalInputDelta = -2.9884501558971607E-146
3.1923777934599684E-152 <= 2.988453348274954E-146
Converged to right
Fitness changed from 2.988453348274954E-146 to 3.1923777934599684E-152
Iteration 17 complete. Error: 3.1923777934599684E-152 Total: 0.7530; Orientation: 0.1524; Line Search: 0.4783
Zero gradient: 1.631047156854549E-79
F(0.0) = LineSearchPoint{point=PointSample{avg=3.1923777934599684E-152}, derivative=-2.6603148278833078E-158}
New Minimum: 3.1923777934599684E-152 > 3.4102175234082143E-158
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=3.4102175234082143E-158}, derivative=2.7495836417886927E-161}, evalInputDelta = -3.192374383242445E-152
3.4102175234082143E-158 <= 3.1923777934599684E-152
Converged to right
Fitness changed from 3.1923777934599684E-152 to 3.4102175234082143E-158
Iteration 18 complete. Error: 3.4102175234082143E-158 Total: 0.7705; Orientation: 0.1537; Line Search: 0.4938
Zero gradient: 1.6857781396653333E-82
F(0.0) = LineSearchPoint{point=PointSample{avg=3.4102175234082143E-158}, derivative=-2.841847936173512E-164}
New Minimum: 3.4102175234082143E-158 > 3.642922081711156E-164
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=3.642922081711156E-164}, derivative=2.9372082265806656E-167}, evalInputDelta = -3.4102138804861326E-158
3.642922081711156E-164 <= 3.4102175234082143E-158
Converged to right
Fitness changed from 3.4102175234082143E-158 to 3.642922081711156E-164
Iteration 19 complete. Error: 3.642922081711156E-164 Total: 0.8504; Orientation: 0.1590; Line Search: 0.5134
Zero gradient: 1.7423456607188952E-85
F(0.0) = LineSearchPoint{point=PointSample{avg=3.642922081711156E-164}, derivative=-3.035768401425964E-170}
New Minimum: 3.642922081711156E-164 > 3.891505806396398E-170
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=3.891505806396398E-170}, derivative=3.137635835176948E-173}, evalInputDelta = -3.64291819020535E-164
3.891505806396398E-170 <= 3.642922081711156E-164
Converged to right
Fitness changed from 3.642922081711156E-164 to 3.891505806396398E-170
Iteration 20 complete. Error: 3.891505806396398E-170 Total: 0.7544; Orientation: 0.1512; Line Search: 0.4788
Zero gradient: 1.800811346402041E-88
F(0.0) = LineSearchPoint{point=PointSample{avg=3.891505806396398E-170}, derivative=-3.242921505330332E-176}
New Minimum: 3.891505806396398E-170 > 4.157052251336177E-176
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=4.157052251336177E-176}, derivative=3.351740113314272E-179}, evalInputDelta = -3.891501649344147E-170
4.157052251336177E-176 <= 3.891505806396398E-170
Converged to right
Fitness changed from 3.891505806396398E-170 to 4.157052251336177E-176
Iteration 21 complete. Error: 4.157052251336177E-176 Total: 0.7561; Orientation: 0.1517; Line Search: 0.4822
Zero gradient: 1.8612388910203908E-91
F(0.0) = LineSearchPoint{point=PointSample{avg=4.157052251336177E-176}, derivative=-3.464210209446814E-182}
New Minimum: 4.157052251336177E-176 > 4.440718909357144E-182
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=4.440718909357144E-182}, derivative=3.580454322088951E-185}, evalInputDelta = -4.157047810617268E-176
4.440718909357144E-182 <= 4.157052251336177E-176
Converged to right
Fitness changed from 4.157052251336177E-176 to 4.440718909357144E-182
Iteration 22 complete. Error: 4.440718909357144E-182 Total: 0.7554; Orientation: 0.1533; Line Search: 0.4778
Zero gradient: 1.9236941261881923E-94
F(0.0) = LineSearchPoint{point=PointSample{avg=4.440718909357144E-182}, derivative=-3.700599091130953E-188}
New Minimum: 4.440718909357144E-182 > 4.7437422576498624E-188
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=4.7437422576498624E-188}, derivative=3.824775405957255E-191}, evalInputDelta = -4.4407141656148866E-182
4.7437422576498624E-188 <= 4.440718909357144E-182
Converged to right
Fitness changed from 4.440718909357144E-182 to 4.7437422576498624E-188
Iteration 23 complete. Error: 4.7437422576498624E-188 Total: 0.7599; Orientation: 0.1518; Line Search: 0.4845
Zero gradient: 1.9882450925480875E-97
F(0.0) = LineSearchPoint{point=PointSample{avg=4.7437422576498624E-188}, derivative=-3.953118548041553E-194}
New Minimum: 4.7437422576498624E-188 > 5.067443147459071E-194
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=5.067443147459071E-194}, derivative=4.0857683383265995E-197}, evalInputDelta = -4.743737190206715E-188
5.067443147459071E-194 <= 4.7437422576498624E-188
Converged to right
Fitness changed from 4.7437422576498624E-188 to 5.067443147459071E-194
Iteration 24 complete. Error: 5.067443147459071E-194 Total: 0.7593; Orientation: 0.1509; Line Search: 0.4826
Zero gradient: 2.0549621138963188E-100
F(0.0) = LineSearchPoint{point=PointSample{avg=5.067443147459071E-194}, derivative=-4.222869289549227E-200}
New Minimum: 5.067443147459071E-194 > 5.41323256155402E-200
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=5.41323256155402E-200}, derivative=4.364570763677147E-203}, evalInputDelta = -5.0674377342265096E-194
5.41323256155402E-200 <= 5.067443147459071E-194
Converged to right
Fitness changed from 5.067443147459071E-194 to 5.41323256155402E-200
Iteration 25 complete. Error: 5.41323256155402E-200 Total: 0.7543; Orientation: 0.1506; Line Search: 0.4803
Zero gradient: 2.1239178737955833E-103
F(0.0) = LineSearchPoint{point=PointSample{avg=5.41323256155402E-200}, derivative=-4.5110271346283516E-206}
New Minimum: 5.41323256155402E-200 > 5.78261776457466E-206
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=5.78261776457466E-206}, derivative=4.6623979564508284E-209}, evalInputDelta = -5.413226778936255E-200
5.78261776457466E-206 <= 5.41323256155402E-200
Converged to right
Fitness changed from 5.41323256155402E-200 to 5.78261776457466E-206
Iteration 26 complete. Error: 5.78261776457466E-206 Total: 0.7519; Orientation: 0.1503; Line Search: 0.4791
Zero gradient: 2.1951874947588305E-106
F(0.0) = LineSearchPoint{point=PointSample{avg=5.78261776457466E-206}, derivative=-4.818848137145552E-212}
New Minimum: 5.78261776457466E-206 > 6.177208873061326E-212
F(1201240.26687953) = LineSearchPoint{point=PointSample{avg=6.177208873061326E-212}, derivative=4.980548118322466E-215}, evalInputDelta = -5.782611587365787E-206
6.177208873061326E-212 <= 5.78261776457466E-206
Converged to right
Fitness changed from 5.78261776457466E-206 to 6.177208873061326E-212
Iteration 27 complete. Error: 6.177208873061326E-212 Total: 0.7631; Orientation: 0.1548; Line Search: 0.4848
Final threshold in iteration 27: 6.177208873061326E-212 (> 0.0) after 30.567s (< 30.000s)
Returns
6.177208873061326E-212
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 63.44 seconds (0.795 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 7625972073938
Reset training subject: 7626094584974
Adding measurement 2b8b29a to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 10.679692229546669 < 10.679692229546669. Total: 1
th(0)=10.679692229546669;dx=-8.899743524622212E-6
Adding measurement 314aca03 to history. Total: 1
New Minimum: 10.679692229546669 > 10.679653881748727
WOLFE (weak): th(2.154434690031884)=10.679653881748727; dx=-8.899727546358732E-6 evalInputDelta=3.834779794154031E-5
Adding measurement 1a9fa5f5 to history. Total: 2
New Minimum: 10.679653881748727 > 10.679615534019636
WOLFE (weak): th(4.308869380063768)=10.679615534019636; dx=-8.899711568095257E-6 evalInputDelta=7.669552703326588E-5
Adding measurement 54441a12 to history. Total: 3
New Minimum: 10.679615534019636 > 10.679462143791753
WOLFE (weak): th(12.926608140191302)=10.679462143791753; dx=-8.899647655041324E-6 evalInputDelta=2.3008575491623162E-4
Adding measurement 5cfdd10e to history. Total: 4
New Minimum: 10.679462143791753 > 10.678771901398235
WOLFE (weak): th(51.70643256076521)=10.678771901398235; dx=-8.899360046298579E-6 evalInputDelta=9.20328148433569E-4
Adding measurement 5db4e1d1 to history. Total: 5
New Minimum: 10.678771901398235 > 10.675090985370428
WOLFE (weak): th(258.53216280382605)=10.675090985370428; dx=-8.897826133004047E-6 evalInputDelta=0.004601244176241082
Adding measurement 5ced92c6 to history. Total: 6
New Minimum: 10.675090985370428 > 10.6520996357113
WOLFE (weak): th(1551.1929768229563)=10.6520996357113; dx=-8.88823917491313E-6 evalInputDelta=0.027592593835368717
Adding measurement 2b43feb8 to history. Total: 7
New Minimum: 10.6520996357113 > 10.487293582290906
WOLFE (weak): th(10858.350837760694)=10.487293582290906; dx=-8.81921307665855E-6 evalInputDelta=0.1923986472557626
Adding measurement 7a0b6b63 to history. Total: 8
New Minimum: 10.487293582290906 > 9.189471011498833
WOLFE (weak): th(86866.80670208555)=9.189471011498833; dx=-8.255499940912634E-6 evalInputDelta=1.4902212180478358
Adding measurement 4cb33b5f to history. Total: 9
New Minimum: 9.189471011498833 > 1.297064832687612
END: th(781801.26031877)=1.297064832687612; dx=-3.1015512712360054E-6 evalInputDelta=9.382627396859057
Fitness changed from 10.679692229546669 to 1.297064832687612
Iteration 1 complete. Error: 1.297064832687612 Total: 4.3162; Orientation: 0.2060; Line Search: 3.7435
Non-optimal measurement 1.297064832687612 < 1.297064832687612. Total: 10
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428, 10.678771901398235, 10.679462143791753, 10.679615534019636, 10.679653881748727, 10.679692229546669
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428, 10.678771901398235, 10.679462143791753, 10.679615534019636, 10.679653881748727
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00, dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, 99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428, 10.678771901398235, 10.679462143791753, 10.679615534019636
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, 99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00, dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428, 10.678771901398235, 10.679462143791753
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, 99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428, 10.678771901398235
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, 99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113, 10.675090985370428
Rejected: LBFGS Orientation magnitude: 1.248e+03, gradient 1.040e-03, dot -1.000; [dcba3d61-4f56-4272-883c-55a40ab74539 = 1.000/1.000e+00, 99ceb83c-f5ee-43b5-a59a-37c56fd28158 = 1.000/1.000e+00, c5db6cea-6ab4-429d-a6e7-caf193e38f0c = 1.000/1.000e+00, 4c21e54a-ee26-479f-962b-68cc952884c8 = 1.000/1.000e+00, d3ebde28-ace1-4fbc-94bd-e4e888387d67 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.297064832687612, 9.189471011498833, 10.487293582290906, 10.6520996357113
LBFGS Accumulation History: 3 points
Removed measurement 4cb33b5f to history. Total: 9
Removed measurement 7a0b6b63 to history. Total: 8
Removed measurement 2b43feb8 to history. Total: 7
Removed measurement 5ced92c6 to history. Total: 6
Removed measurement 5db4e1d1 to history. Total: 5
Removed measurement 5cfdd10e to history. Total: 4
Removed measurement 54441a12 to history. Total: 3
Adding measurement 424514a2 to history. Total: 3
th(0)=1.297064832687612;dx=-1.080887360573012E-6
Adding measurement 72878de5 to history. Total: 4
New Minimum: 1.297064832687612 > 0.21129996716625601
WOLF (strong): th(1684339.7559414052)=0.21129996716625601; dx=4.3626393368340174E-7 evalInputDelta=1.0857648655213559
Adding measurement 498cc61a to history. Total: 5
New Minimum: 0.21129996716625601 > 0.11533283975342627
END: th(842169.8779707026)=0.11533283975342627; dx=-3.2231171344480405E-7 evalInputDelta=1.1817319929341856
Fitness changed from 1.297064832687612 to 0.11533283975342627
Iteration 2 complete. Error: 0.11533283975342627 Total: 59.1211; Orientation: 57.9631; Line Search: 1.0296
Final threshold in iteration 2: 0.11533283975342627 (> 0.0) after 63.437s (< 30.000s)
Returns
0.11533283975342627
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 1.0687045763223653 ], [ 0.4609406843421055 ], [ -0.8945084915789479 ], [ 0.1663114699013168 ], [ 0.03935845396052562 ], [ 1.3574956915131553 ], [ 1.5997380741710507 ], [ 0.4763207730197372 ], ... ],
[ [ 1.537386363315787 ], [ 1.6956834100723661 ], [ -0.4595885050197351 ], [ -1.6759897599473659 ], [ -1.718909746506579 ], [ -1.1526529407039465 ], [ 1.8768313561447367 ], [ -0.34928727517105224 ], ... ],
[ [ 0.4551728269473666 ], [ 1.4589974399868415 ], [ 1.5112635489078945 ], [ 0.8448613570460496 ], [ -0.4289004433881585 ], [ -0.8638618255131566 ], [ -0.4568674140065804 ], [ -1.3081393290986818 ], ... ],
[ [ -1.3737978582960517 ], [ 0.8609809252960524 ], [ 1.3354461474407882 ], [ -0.9003868317828937 ], [ 0.8044817368355248 ], [ -0.3490758303486836 ], [ -0.7435698987828943 ], [ -1.4313770601973665 ], ... ],
[ [ -0.8899504559276328 ], [ -0.3697245879605267 ], [ -0.8230634987499981 ], [ -0.7056627122894736 ], [ -0.570683878539473 ], [ -1.0924487074539466 ], [ -1.3923033213947344 ], [ 0.8414512674671051 ], ... ],
[ [ 0.39896441060526333 ], [ -1.1919350958486818 ], [ 0.09772155948026151 ], [ -0.0936813185263145 ], [ 0.6419732452565768 ], [ 0.1735389609473677 ], [ 0.6274791768223664 ], [ 0.3798246130394749 ], ... ],
[ [ -1.1158978833750002 ], [ -1.6225931565789455 ], [ -0.8661919016513142 ], [ 0.5637699489407906 ], [ -1.4030190747039475 ], [ 0.8338236761052614 ], [ -0.39104070942105373 ], [ 0.15897580526973548 ], ... ],
[ [ -1.6856699238618422 ], [ 0.46708095037500064 ], [ -1.1751656153749979 ], [ 0.7815503556118398 ], [ 0.7950809503750006 ], [ 1.2484889484078934 ], [ 1.5378236761052615 ], [ 0.514282623611842 ], ... ],
...
]
To produce the following output:
TrainingTester.java:633 executed in 0.03 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[
[ [ 2.1374091526447305 ], [ 0.921881368684211 ], [ -1.7890169831578957 ], [ 0.3326229398026336 ], [ 0.07871690792105124 ], [ 2.7149913830263106 ], [ 3.1994761483421015 ], [ 0.9526415460394744 ], ... ],
[ [ 3.074772726631574 ], [ 3.3913668201447322 ], [ -0.9191770100394702 ], [ -3.3519795198947318 ], [ -3.437819493013158 ], [ -2.305305881407893 ], [ 3.7536627122894735 ], [ -0.6985745503421045 ], ... ],
[ [ 0.9103456538947332 ], [ 2.917994879973683 ], [ 3.022527097815789 ], [ 1.6897227140920992 ], [ -0.857800886776317 ], [ -1.7277236510263132 ], [ -0.9137348280131607 ], [ -2.6162786581973636 ], ... ],
[ [ -2.7475957165921034 ], [ 1.7219618505921048 ], [ 2.6708922948815763 ], [ -1.8007736635657874 ], [ 1.6089634736710496 ], [ -0.6981516606973672 ], [ -1.4871397975657885 ], [ -2.862754120394733 ], ... ],
[ [ -1.7799009118552656 ], [ -0.7394491759210534 ], [ -1.6461269974999961 ], [ -1.4113254245789473 ], [ -1.141367757078946 ], [ -2.1848974149078932 ], [ -2.784606642789469 ], [ 1.6829025349342102 ], ... ],
[ [ 0.7979288212105267 ], [ -2.3838701916973637 ], [ 0.19544311896052302 ], [ -0.187362637052629 ], [ 1.2839464905131537 ], [ 0.3470779218947354 ], [ 1.2549583536447328 ], [ 0.7596492260789498 ], ... ],
[ [ -2.2317957667500004 ], [ -3.245186313157891 ], [ -1.7323838033026284 ], [ 1.1275398978815812 ], [ -2.806038149407895 ], [ 1.6676473522105228 ], [ -0.7820814188421075 ], [ 0.31795161053947096 ], ... ],
[ [ -3.3713398477236844 ], [ 0.9341619007500013 ], [ -2.3503312307499957 ], [ 1.5631007112236797 ], [ 1.5901619007500012 ], [ 2.4969778968157867 ], [ 3.075647352210523 ], [ 1.028565247223684 ], ... ],
...
]
TrainingTester.java:432 executed in 0.14 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -211.20920771339692], [27.0, 0.11296168446864156]; valueStats=DoubleSummaryStatistics{count=54, sum=2.831902, min=0.000000, average=0.052443, max=1.297065}
Plotting 25 points for GD
Plotting 27 points for CjGD
Plotting 2 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -211.20920771339692], [59.121, 0.11296168446864156]; valueStats=DoubleSummaryStatistics{count=54, sum=2.831902, min=0.000000, average=0.052443, max=1.297065}
Plotting 25 points for GD
Plotting 27 points for CjGD
Plotting 2 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "NonConverged", "value": 0.11533283975342627 }, "CjGD": { "type": "Converged", "value": 6.177208873061326E-212 }, "GD": { "type": "Converged", "value": 2.3360509519582303E-52 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "NonConverged", "value": 0.11533283975342627 }, "CjGD": { "type": "Converged", "value": 6.177208873061326E-212 }, "GD": { "type": "Converged", "value": 2.3360509519582303E-52 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "125.892",
"gc_time": "2.554"
},
"created_on": 1586742195844,
"file_name": "trainingTest",
"report": {
"simpleName": "OnePlusOne",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.SumInputsLayerTest.OnePlusOne",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/SumInputsLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "NonConverged",
"value": 0.11533283975342627
},
"CjGD": {
"type": "Converged",
"value": 6.177208873061326E-212
},
"GD": {
"type": "Converged",
"value": 2.3360509519582303E-52
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/SumInputsLayer/OnePlusOne/trainingTest/202004134315",
"id": "dd183a18-c132-4f40-9d4d-cf9ca92cc9bb",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "SumInputsLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.java.SumInputsLayer",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/SumInputsLayer.java",
"javaDoc": ""
}
}