1. Test Modules
  2. Network Diagram
  3. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  4. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Network Diagram

This is a network apply the following layout:

LayerTests.java:203 executed in 0.07 seconds (0.000 gc):

    return Graphviz.fromGraph((Graph) TestUtil.toGraph(((DAGNetwork) layer).addRef())).height(400).width(600)
        .render(Format.PNG).toImage();
Logging
executing command [/bin/sh, -c, dot -Tsvg /tmp/GraphvizJava/DotEngine4446810918610906439/dotfile.dot -ooutfile.svg]

Returns

Result

Using Seed 4165348181060136960

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -1.44, 1.932, -0.304, -1.756 ], [ -0.996, -0.5, -0.496, 1.22 ], [ -1.204, 0.872, 0.228, 1.516 ], [ -1.836, -1.112, 0.648, -1.308 ], [ 1.78, -0.832, -0.256, 1.156 ], [ -0.376, -1.676, -0.2, -1.004 ], [ -0.236, 1.388, 0.412, 0.516 ], [ 0.372, 1.908, 1.88, -1.612 ], ... ],
    	[ [ -0.76, -0.588, -1.576, -0.288 ], [ -0.828, 0.308, 0.04, -0.456 ], [ -1.784, -0.124, 0.604, -1.672 ], [ 1.448, -0.052, -1.404, -0.3 ], [ 1.988, 1.376, 0.836, 1.924 ], [ -1.092, -1.34, 0.112, 1.16 ], [ -0.244, -0.5, 1.644, -0.636 ], [ 1.2, 1.512, -1.396, 0.196 ], ... ],
    	[ [ 0.756, 0.52, -0.2, -1.708 ], [ -0.784, -1.504, 0.192, 0.872 ], [ 1.348, -0.912, 1.556, -1.868 ], [ 0.94, -1.308, -1.408, -0.552 ], [ 1.092, -0.44, -1.248, 0.196 ], [ 1.54, 1.64, 1.58, 1.068 ], [ -0.384, -0.864, 0.148, 1.876 ], [ 1.776, -0.152, -0.692, 0.828 ], ... ],
    	[ [ -1.972, -1.328, 0.312, 0.592 ], [ -1.32, -0.04, 0.376, -0.196 ], [ -1.96, -0.264, 0.296, -1.68 ], [ -1.4, -0.316, -0.316, 1.048 ], [ 0.408, -0.404, 1.9, -1.028 ], [ -1.592, 0.708, 0.36, 0.372 ], [ 1.512, 0.836, 0.976, 0.132 ], [ -1.708, 0.676, -0.324, -0.656 ], ... ],
    	[ [ -0.408, 1.708, -0.876, -1.692 ], [ -0.488, 1.704, 1.508, 0.208 ], [ 1.156, 1.44, 0.652, 1.484 ], [ 1.024, 1.712, 1.08, 0.216 ], [ 0.008, 0.036, 0.84, -0.972 ], [ -1.412, 0.984, -0.432, 1.8 ], [ 0.968, -0.636, 0.432, 1.564 ], [ -0.344, 0.108, 1.868, -0.436 ], ... ],
    	[ [ 0.452, -1.34, 0.372, 1.208 ], [ -1.716, 0.476, -0.716, 1.86 ], [ 1.32, 0.768, 0.64, -1.508 ], [ -0.076, 0.076, 0.152, 1.356 ], [ 0.72, -0.032, -1.504, -1.576 ], [ 1.644, -1.492, -1.772, -0.916 ], [ 0.484, -0.68, -1.228, 1.628 ], [ -1.036, -1.5, 1.12, 1.756 ], ... ],
    	[ [ 0.196, 0.528, 0.96, -1.82 ], [ -1.964, 0.988, 1.364, 1.884 ], [ 0.864, 0.16, 0.972, -0.38 ], [ -0.932, 1.588, -0.944, 1.588 ], [ 0.236, -1.652, -0.228, 1.972 ], [ -0.516, -1.436, 0.376, 1.168 ], [ -1.716, -1.332, -0.46, 0.448 ], [ 0.772, 1.064, -0.468, 1.728 ], ... ],
    	[ [ 0.496, 0.44, 1.224, 1.312 ], [ 1.332, 0.012, 0.848, -0.912 ], [ 1.324, 1.596, -1.628, 1.676 ], [ -0.888, 0.932, -0.648, -0.772 ], [ -1.94, 1.052, 0.088, 1.116 ], [ -1.068, -1.588, 0.508, 0.464 ], [ -1.624, 1.756, 0.432, -0.16 ], [ -0.116, 1.284, 1.456, -1.616 ], ... ],
    	...
    ]
    [
    	[ [ -1.096, 1.996, 1.224, 0.888 ], [ 1.296, 0.476, 0.208, -0.94 ], [ -0.24, -0.384, 0.412, -1.868 ], [ 0.812, -0.752, 0.916, -0.484 ], [ -1.888, -1.62, 0.728, 1.112 ], [ -0.444, 1.04, -1.076, -1.584 ], [ 1.088, -1.916, 0.088, -0.048 ], [ -1.444, -0.904, 0.508, 1.048 ], ... ],
    	[ [ -0.204, -0.708, 1.52, -1.556 ], [ -0.58, 0.664, -1.96, 1.76 ], [ 0.932, 0.6, 0.824, 1.564 ], [ -0.064, 0.676, -0.016, 1.512 ], [ -1.684, 0.588, -0.988, 1.744 ], [ 1.004, -0.3, 0.712, 0.68 ], [ -1.332, 0.64, 0.184, -1.484 ], [ -1.568, 1.132, 0.776, 0.668 ], ... ],
    	[ [ 0.196, 1.352, 0.392, -0.276 ], [ 0.252, -0.608, -0.616, -1.944 ], [ 0.864, 1.088, 0.82, -1.788 ], [ -0.696, -1.348, 1.944, 0.688 ], [ -1.816, -1.684, -1.42, -1.888 ], [ 0.78, 1.824, 0.92, 1.036 ], [ -1.072, -0.136, -0.552, 0.904 ], [ 1.804, 0.952, 0.532, -1.984 ], ... ],
    	[ [ 0.972, 0.832, -1.784, 0.396 ], [ -0.412, -0.728, 1.156, 1.9 ], [ -0.264, 1.912, -0.736, -1.72 ], [ 1.92, 0.936, 0.376, 1.716 ], [ -1.164, -0.348, -0.072, 0.288 ], [ 1.46, -0.916, 0.8, -0.436 ], [ 1.452, 1.936, 0.668, 1.22 ], [ -1.544, 1.156, 0.496, -0.896 ], ... ],
    	[ [ -0.568, 0.528, -0.416, -1.832 ], [ 0.8, 1.492, 0.064, 0.596 ], [ -1.324, 0.832, -0.464, 0.628 ], [ 0.704, 0.576, 0.268, -0.524 ], [ 0.78, -0.536, -1.884, -0.144 ], [ -1.276, 0.436, -1.06, -1.52 ], [ 0.54, -1.38, -1.364, 1.108 ], [ 1.896, -0.6, 0.196, 0.516 ], ... ],
    	[ [ -1.72, -0.808, -1.696, 0.764 ], [ 0.164, 1.048, 0.612, 1.5 ], [ 0.536, -0.336, 0.9, 1.328 ], [ -1.172, 0.756, 0.936, -0.104 ], [ -1.38, 1.732, 0.192, 1.0 ], [ -0.384, 0.44, 1.296, -1.16 ], [ 1.304, 0.516, 1.94, -0.22 ], [ 1.02, 0.5, 0.3, 0.168 ], ... ],
    	[ [ 0.384, -0.304, 0.992, 1.176 ], [ 0.76, -1.392, 1.692, -1.756 ], [ -0.172, 0.328, 0.392, -1.048 ], [ 1.24, -0.992, -1.232, 0.692 ], [ 1.512, -0.084, -1.584, -0.624 ], [ -0.156, -1.82, 0.568, 1.776 ], [ -1.752, 1.236, -1.912, -0.216 ], [ 1.648, -0.136, 0.196, -0.548 ], ... ],
    	[ [ 1.784, 0.62, 0.032, -1.88 ], [ 0.408, 0.948, -0.048, -0.864 ], [ -0.848, -0.552, 0.96, -1.8 ], [ 1.932, 1.36, 1.604, -1.564 ], [ -0.736, -1.208, 1.952, 1.236 ], [ -0.996, 1.276, -0.448, -0.224 ], [ 1.676, 0.4, 0.48, 1.4 ], [ -1.86, -0.372, 0.312, -1.764 ], ... ],
    	...
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 30.57 seconds (0.664 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 13736631217428
Reset training subject: 13736925844923
Constructing line search parameters: GD
th(0)=140.96797785740105;dx=-0.019421814301975097
New Minimum: 140.96797785740105 > 140.92613869113967
WOLFE (weak): th(2.154434690031884)=140.92613869113967; dx=-0.019418227083319552 evalInputDelta=0.04183916626138284
New Minimum: 140.92613869113967 > 140.8843072533066
WOLFE (weak): th(4.308869380063768)=140.8843072533066; dx=-0.019414639864664004 evalInputDelta=0.0836706040944648
New Minimum: 140.8843072533066 > 140.71705878625747
WOLFE (weak): th(12.926608140191302)=140.71705878625747; dx=-0.019400290990041834 evalInputDelta=0.2509190711435849
New Minimum: 140.71705878625747 > 139.96597091334218
WOLFE (weak): th(51.70643256076521)=139.96597091334218; dx=-0.01933572105424204 evalInputDelta=1.002006944058877
New Minimum: 139.96597091334218 > 136.00245888418522
WOLFE (weak): th(258.53216280382605)=136.00245888418522; dx=-0.018991348063309835 evalInputDelta=4.965518973215836
New Minimum: 136.00245888418522 > 112.84420453354849
END: th(1551.1929768229563)=112.84420453354849; dx=-0.016839016869983514 evalInputDelta=28.12377332385256
Fitness changed from 140.96797785740105 to 112.84420453354849
Iteration 1 complete. Error: 112.84420453354849 Total: 2.6862; Orientation: 0.0974; Line Search: 2.0090
th(0)=112.84420453354849;dx=-0.014644730686661295
New Minimum: 112.84420453354849 > 70.63150053536918
END: th(3341.943960201201)=70.63150053536918; dx=-0.010617634272706773 evalInputDelta=42.21270399817931
Fitness changed from 112.84420453354849 to 70.63150053536918
Iteration 2 complete. Error: 70.63150053536918 Total: 0.7161; Orientation: 0.1005; Line Search: 0.4652
th(0)=70.63150053536918;dx=-0.00785595654856587
New Minimum: 70.63150053536918 > 29.043442710341342
END: th(7200.000000000001)=29.043442710341342; dx=-0.0036962817361640855 evalInputDelta=41.588057825027846
Fitness changed from 70.63150053536918 to 29.043442710341342
Iteration 3 complete. Error: 29.043442710341342 Total: 0.7494; Orientation: 0.1595; Line Search: 0.4395
th(0)=29.043442710341342;dx=-0.002114758224201549
New Minimum: 29.043442710341342 > 9.25488866904522
END: th(15511.929768229566)=9.25488866904522; dx=-4.3663987222687065E-4 evalInputDelta=19.788554041296123
Fitness changed from 29.043442710341342 to 9.25488866904522
Iteration 4 complete. Error: 9.25488866904522 Total: 0.6764; Orientation: 0.0939; Line Search: 0.4323
th(0)=9.25488866904522;dx=-3.651572828603568E-4
New Minimum: 9.25488866904522 > 4.864409299925368
WOLF (strong): th(33419.43960201202)=4.864409299925368; dx=1.0240725345199276E-4 evalInputDelta=4.390479369119852
END: th(16709.71980100601)=5.10643088681863; dx=-1.31375014704182E-4 evalInputDelta=4.148457782226591
Fitness changed from 9.25488866904522 to 4.864409299925368
Iteration 5 complete. Error: 4.864409299925368 Total: 1.0173; Orientation: 0.1625; Line Search: 0.7041
th(0)=4.864409299925368;dx=-4.0609072874391776E-4
Armijo: th(36000.000000000015)=13.690351182923525; dx=8.964208333549263E-4 evalInputDelta=-8.825941882998157
New Minimum: 4.864409299925368 > 3.4160782119796473
WOLF (strong): th(18000.000000000007)=3.4160782119796473; dx=2.451650523055042E-4 evalInputDelta=1.448331087945721
New Minimum: 3.4160782119796473 > 3.079120708511284
END: th(6000.000000000003)=3.079120708511284; dx=-1.890054683941104E-4 evalInputDelta=1.7852885914140844
Fitness changed from 4.864409299925368 to 3.079120708511284
Iteration 6 complete. Error: 3.079120708511284 Total: 1.3450; Orientation: 0.0899; Line Search: 1.1137
th(0)=3.079120708511284;dx=-1.0333607235165527E-4
New Minimum: 3.079120708511284 > 2.2533567038806774
END: th(12926.608140191309)=2.2533567038806774; dx=-2.4425827084771186E-5 evalInputDelta=0.8257640046306065
Fitness changed from 3.079120708511284 to 2.2533567038806774
Iteration 7 complete. Error: 2.2533567038806774 Total: 0.6610; Orientation: 0.0946; Line Search: 0.4208
th(0)=2.2533567038806774;dx=-3.577742164896476E-5
New Minimum: 2.2533567038806774 > 1.5965877255029368
END: th(27849.53300167669)=1.5965877255029368; dx=-1.13881073628287E-5 evalInputDelta=0.6567689783777406
Fitness changed from 2.2533567038806774 to 1.5965877255029368
Iteration 8 complete. Error: 1.5965877255029368 Total: 0.6462; Orientation: 0.0909; Line Search: 0.4162
th(0)=1.5965877255029368;dx=-3.504739329907441E-5
Armijo: th(60000.00000000004)=3.5679671306895724; dx=1.0076004013862886E-4 evalInputDelta=-1.9713794051866356
New Minimum: 1.5965877255029368 > 1.5637216773134792
WOLF (strong): th(30000.00000000002)=1.5637216773134792; dx=3.285632341977723E-5 evalInputDelta=0.0328660481894576
New Minimum: 1.5637216773134792 > 1.3592866537102788
END: th(10000.000000000005)=1.3592866537102788; dx=-1.2412821059457195E-5 evalInputDelta=0.23730107179265802
Fitness changed from 1.5965877255029368 to 1.3592866537102788
Iteration 9 complete. Error: 1.3592866537102788 Total: 1.2819; Orientation: 0.1004; Line Search: 1.0448
th(0)=1.3592866537102788;dx=-1.3869676748453093E-5
New Minimum: 1.3592866537102788 > 1.0994389706711885
END: th(21544.34690031885)=1.0994389706711885; dx=-1.0252445332234752E-5 evalInputDelta=0.25984768303909034
Fitness changed from 1.3592866537102788 to 1.0994389706711885
Iteration 10 complete. Error: 1.0994389706711885 Total: 0.6629; Orientation: 0.0931; Line Search: 0.4248
th(0)=1.0994389706711885;dx=-1.0913680273127197E-5
New Minimum: 1.0994389706711885 > 0.877949770915285
WOLF (strong): th(46415.88833612782)=0.877949770915285; dx=1.369999964696805E-6 evalInputDelta=0.22148919975590342
END: th(23207.94416806391)=0.9174246295087946; dx=-4.771840154215196E-6 evalInputDelta=0.1820143411623939
Fitness changed from 1.0994389706711885 to 0.877949770915285
Iteration 11 complete. Error: 0.877949770915285 Total: 0.8475; Orientation: 0.0915; Line Search: 0.6177
th(0)=0.877949770915285;dx=-4.4909067590582816E-5
Armijo: th(50000.00000000004)=5.402202771933401; dx=2.2587918763130734E-4 evalInputDelta=-4.524253001018115
Armijo: th(25000.00000000002)=1.4476496762875284; dx=9.048506002036227E-5 evalInputDelta=-0.5696999053722434
New Minimum: 0.877949770915285 > 0.6917549404534074
WOLF (strong): th(8333.33333333334)=0.6917549404534074; dx=2.223082797322055E-7 evalInputDelta=0.18619483046187768
END: th(2083.333333333335)=0.796142175901132; dx=-3.3626223623004066E-5 evalInputDelta=0.08180759501415302
Fitness changed from 0.877949770915285 to 0.6917549404534074
Iteration 12 complete. Error: 0.6917549404534074 Total: 1.2407; Orientation: 0.0895; Line Search: 1.0104
th(0)=0.6917549404534074;dx=-5.3237453846445E-6
New Minimum: 0.6917549404534074 > 0.6695165838241214
END: th(4488.405604233095)=0.6695165838241214; dx=-4.585500164961178E-6 evalInputDelta=0.02223835662928597
Fitness changed from 0.6917549404534074 to 0.6695165838241214
Iteration 13 complete. Error: 0.6695165838241214 Total: 0.6669; Orientation: 0.0899; Line Search: 0.4345
th(0)=0.6695165838241214;dx=-4.267346452414352E-6
New Minimum: 0.6695165838241214 > 0.6300169149660868
WOLFE (weak): th(9669.976736693297)=0.6300169149660868; dx=-3.902201403507755E-6 evalInputDelta=0.039499668858034576
New Minimum: 0.6300169149660868 > 0.5940481902364976
END: th(19339.953473386595)=0.5940481902364976; dx=-3.53705635460116E-6 evalInputDelta=0.0754683935876238
Fitness changed from 0.6695165838241214 to 0.5940481902364976
Iteration 14 complete. Error: 0.5940481902364976 Total: 0.9373; Orientation: 0.0960; Line Search: 0.6995
th(0)=0.5940481902364976;dx=-3.8082407547346377E-6
New Minimum: 0.5940481902364976 > 0.4976184491681904
END: th(41666.6666666667)=0.4976184491681904; dx=-8.203868165440989E-7 evalInputDelta=0.09642974106830715
Fitness changed from 0.5940481902364976 to 0.4976184491681904
Iteration 15 complete. Error: 0.4976184491681904 Total: 0.7174; Orientation: 0.0988; Line Search: 0.4689
th(0)=0.4976184491681904;dx=-1.2149924580073532E-5
Armijo: th(89768.1120846619)=5.200634716358735; dx=1.169313699725097E-4 evalInputDelta=-4.703016267190544
Armijo: th(44884.05604233095)=1.4007035680847695; dx=5.239072269621809E-5 evalInputDelta=-0.903085118916579
New Minimum: 0.4976184491681904 > 0.4767750411006737
WOLF (strong): th(14961.352014110

...skipping 375 bytes...

Orientation: 0.0988; Line Search: 1.2911
th(0)=0.46223211767883843;dx=-4.297502180222883E-6
New Minimum: 0.46223211767883843 > 0.43757230769038014
END: th(8058.313947244415)=0.43757230769038014; dx=-1.8228376699987254E-6 evalInputDelta=0.024659809988458292
Fitness changed from 0.46223211767883843 to 0.43757230769038014
Iteration 17 complete. Error: 0.43757230769038014 Total: 0.6925; Orientation: 0.0934; Line Search: 0.4451
th(0)=0.43757230769038014;dx=-2.1597437089860742E-6
New Minimum: 0.43757230769038014 > 0.4026019444631226
END: th(17361.111111111128)=0.4026019444631226; dx=-1.868842134793993E-6 evalInputDelta=0.03497036322725755
Fitness changed from 0.43757230769038014 to 0.4026019444631226
Iteration 18 complete. Error: 0.4026019444631226 Total: 0.6782; Orientation: 0.0890; Line Search: 0.4404
th(0)=0.4026019444631226;dx=-1.982591836862526E-6
New Minimum: 0.4026019444631226 > 0.34774522653619205
END: th(37403.3800352758)=0.34774522653619205; dx=-9.506573975753352E-7 evalInputDelta=0.05485671792693053
Fitness changed from 0.4026019444631226 to 0.34774522653619205
Iteration 19 complete. Error: 0.34774522653619205 Total: 0.6926; Orientation: 0.0904; Line Search: 0.4474
th(0)=0.34774522653619205;dx=-4.269483206554338E-6
Armijo: th(80583.13947244416)=1.384390245019794; dx=2.9998066760638582E-5 evalInputDelta=-1.036645018483602
Armijo: th(40291.56973622208)=0.5208943909798359; dx=1.2864291777042124E-5 evalInputDelta=-0.17314916444364387
New Minimum: 0.34774522653619205 > 0.32875642695114954
WOLF (strong): th(13430.523245407361)=0.32875642695114954; dx=1.4417751213111484E-6 evalInputDelta=0.018988799585042515
END: th(3357.6308113518403)=0.3358069152899698; dx=-2.841668624587967E-6 evalInputDelta=0.011938311246222244
Fitness changed from 0.34774522653619205 to 0.32875642695114954
Iteration 20 complete. Error: 0.32875642695114954 Total: 1.3293; Orientation: 0.0930; Line Search: 1.0816
th(0)=0.32875642695114954;dx=-3.9570559010662056E-6
New Minimum: 0.32875642695114954 > 0.3105018275455489
END: th(7233.796296296305)=0.3105018275455489; dx=-1.0899757425942648E-6 evalInputDelta=0.018254599405600647
Fitness changed from 0.32875642695114954 to 0.3105018275455489
Iteration 21 complete. Error: 0.3105018275455489 Total: 0.7013; Orientation: 0.0885; Line Search: 0.4586
th(0)=0.3105018275455489;dx=-1.230313053958841E-6
New Minimum: 0.3105018275455489 > 0.2919538984806679
WOLFE (weak): th(15584.741681364918)=0.2919538984806679; dx=-1.1499547033257402E-6 evalInputDelta=0.018547929064881008
New Minimum: 0.2919538984806679 > 0.27465833355234426
END: th(31169.483362729836)=0.27465833355234426; dx=-1.0695963526926402E-6 evalInputDelta=0.035843493993204634
Fitness changed from 0.3105018275455489 to 0.27465833355234426
Iteration 22 complete. Error: 0.27465833355234426 Total: 0.9341; Orientation: 0.0923; Line Search: 0.6818
th(0)=0.27465833355234426;dx=-1.1278499595639354E-6
New Minimum: 0.27465833355234426 > 0.2465903184037766
WOLF (strong): th(67152.61622703681)=0.2465903184037766; dx=2.91902926505021E-7 evalInputDelta=0.02806801514856766
END: th(33576.308113518404)=0.24870681089113336; dx=-4.179735165294573E-7 evalInputDelta=0.025951522661210896
Fitness changed from 0.27465833355234426 to 0.2465903184037766
Iteration 23 complete. Error: 0.2465903184037766 Total: 0.9031; Orientation: 0.0898; Line Search: 0.6607
th(0)=0.2465903184037766;dx=-1.0454847991530265E-5
Armijo: th(72337.96296296305)=3.25459769174767; dx=9.362023584974214E-5 evalInputDelta=-3.0080073733438937
Armijo: th(36168.981481481525)=0.8095215600410688; dx=4.158269392910593E-5 evalInputDelta=-0.5629312416372922
New Minimum: 0.2465903184037766 > 0.22510685560850627
WOLF (strong): th(12056.327160493842)=0.22510685560850627; dx=6.890999315348468E-6 evalInputDelta=0.02148346279527033
New Minimum: 0.22510685560850627 > 0.22161377676673705
END: th(3014.0817901234605)=0.22161377676673705; dx=-6.118386164810584E-6 evalInputDelta=0.02497654163703955
Fitness changed from 0.2465903184037766 to 0.22161377676673705
Iteration 24 complete. Error: 0.22161377676673705 Total: 1.5043; Orientation: 0.0891; Line Search: 1.2625
th(0)=0.22161377676673705;dx=-3.6997533477053764E-6
New Minimum: 0.22161377676673705 > 0.20703194962536456
END: th(6493.642367235383)=0.20703194962536456; dx=-7.913554373879406E-7 evalInputDelta=0.014581827141372489
Fitness changed from 0.22161377676673705 to 0.20703194962536456
Iteration 25 complete. Error: 0.20703194962536456 Total: 0.6675; Orientation: 0.0930; Line Search: 0.4267
th(0)=0.20703194962536456;dx=-6.47189195720563E-7
New Minimum: 0.20703194962536456 > 0.19822360990776874
WOLFE (weak): th(13990.12838063267)=0.19822360990776874; dx=-6.120329469139102E-7 evalInputDelta=0.008808339717595814
New Minimum: 0.19822360990776874 > 0.18990711062435947
END: th(27980.25676126534)=0.18990711062435947; dx=-5.768766981072575E-7 evalInputDelta=0.017124839001005088
Fitness changed from 0.20703194962536456 to 0.18990711062435947
Iteration 26 complete. Error: 0.18990711062435947 Total: 0.8764; Orientation: 0.0886; Line Search: 0.6447
th(0)=0.18990711062435947;dx=-6.23835487696233E-7
New Minimum: 0.18990711062435947 > 0.17172993761579688
WOLF (strong): th(60281.635802469216)=0.17172993761579688; dx=2.0760512487347738E-8 evalInputDelta=0.018177173008562586
END: th(30140.817901234608)=0.1759613614547288; dx=-3.015374876044426E-7 evalInputDelta=0.013945749169630678
Fitness changed from 0.18990711062435947 to 0.17172993761579688
Iteration 27 complete. Error: 0.17172993761579688 Total: 0.8551; Orientation: 0.0883; Line Search: 0.6229
th(0)=0.17172993761579688;dx=-4.357924314198765E-6
Armijo: th(64936.423672353834)=1.1047487664752984; dx=3.309430294715779E-5 evalInputDelta=-0.9330188288595015
Armijo: th(32468.211836176917)=0.33423763993095656; dx=1.4368189316479511E-5 evalInputDelta=-0.16250770231515968
New Minimum: 0.17172993761579688 > 0.1583432356953854
WOLF (strong): th(10822.737278725639)=0.1583432356953854; dx=1.8841135626939937E-6 evalInputDelta=0.013386701920411498
END: th(2705.6843196814098)=0.1620498931333067; dx=-2.797414844975574E-6 evalInputDelta=0.009680044482490185
Fitness changed from 0.17172993761579688 to 0.1583432356953854
Iteration 28 complete. Error: 0.1583432356953854 Total: 1.2603; Orientation: 0.0903; Line Search: 1.0261
th(0)=0.1583432356953854;dx=-1.747921226750031E-6
New Minimum: 0.1583432356953854 > 0.15151751421696025
END: th(5829.220158596946)=0.15151751421696025; dx=-5.939774467315564E-7 evalInputDelta=0.00682572147842514
Fitness changed from 0.1583432356953854 to 0.15151751421696025
Iteration 29 complete. Error: 0.15151751421696025 Total: 0.6634; Orientation: 0.0904; Line Search: 0.4285
th(0)=0.15151751421696025;dx=-4.223984976686232E-7
New Minimum: 0.15151751421696025 > 0.14660347696908946
END: th(12558.67412551442)=0.14660347696908946; dx=-3.6017412086717157E-7 evalInputDelta=0.0049140372478707894
Fitness changed from 0.15151751421696025 to 0.14660347696908946
Iteration 30 complete. Error: 0.14660347696908946 Total: 0.6534; Orientation: 0.0894; Line Search: 0.4229
th(0)=0.14660347696908946;dx=-3.94752576520176E-7
New Minimum: 0.14660347696908946 > 0.137264351459402
END: th(27056.8431968141)=0.137264351459402; dx=-2.9558113623069805E-7 evalInputDelta=0.009339125509687463
Fitness changed from 0.14660347696908946 to 0.137264351459402
Iteration 31 complete. Error: 0.137264351459402 Total: 0.6546; Orientation: 0.0873; Line Search: 0.4246
th(0)=0.137264351459402;dx=-5.132529871905996E-7
Armijo: th(58292.20158596947)=0.1523346854314887; dx=1.030314740291859E-6 evalInputDelta=-0.015070333972086708
New Minimum: 0.137264351459402 > 0.1335522733039448
WOLF (strong): th(29146.100792984736)=0.1335522733039448; dx=2.5853087655062957E-7 evalInputDelta=0.0037120781554571824
New Minimum: 0.1335522733039448 > 0.13352760426502727
END: th(9715.366930994913)=0.13352760426502727; dx=-2.5599169927685656E-7 evalInputDelta=0.0037367471943747232
Fitness changed from 0.137264351459402 to 0.13352760426502727
Iteration 32 complete. Error: 0.13352760426502727 Total: 1.0894; Orientation: 0.0919; Line Search: 0.8541
Final threshold in iteration 32: 0.13352760426502727 (> 0.0) after 30.557s (< 30.000s)

Returns

    0.13352760426502727

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -1.4397630246468553, 2.085641145013716, -0.45053195314724814, -1.418281205247531 ], [ -0.6808223614058794, -0.2695074144536193, -0.541109157675915, 1.3854784740100925 ], [ -1.055677476005819, 0.8377416696230036, 0.6128033973750445, 1.1413918575253605 ], [ -1.39105310835348, -0.8166898384079571, 0.5709961606780543, -1.33875273800899 ], [ 2.3137040344239526, 0.02648393626398978, -0.5862225814891998, 1.4846062307019485 ], [ -0.10482985015495372, -1.3768441431834362, -0.6171944315563143, -0.20891898683974858 ], [ -1.0678118173423348, 0.15209196623691576, 0.6210222328697372, 0.5338898435250424 ], [ -0.40644745753962047, 0.9191983738737419, 2.3493858585683425, -2.3791763028254485 ], ... ],
    	[ [ -0.5725122418918996, -0.4201708462836197, -1.3978419910737592, 0.13069388715822342 ], [ -0.7178285771730302, 0.24373539713782788, -0.06491712560539954, -0.1382926873088928 ], [ -1.6355578180911343, 0.020261158655257225, 0.559701766274918, -1.6178466694204443 ], [ 1.4472030364332236, 0.44966449723762875, -1.4997633025675552, 0.3378007733779561 ], [ 1.6867435711277388, 1.5707944840707562, 0.615771919017072, 1.8758279113378378 ], [ -1.097438399553061, -1.2439267064523338, -0.06303534681306741, 0.216995502612234 ], [ 0.09800071035234932, -1.0622038984465882, 1.8784706370392759, -1.0779715657004427 ], [ 1.2792949664073932, 0.8976689423765628, -1.1165999244698377, 0.39352338188504454 ], ... ],
    	[ [ 0.9849175441322003, 0.37090922897215617, 0.03230527141135758, -1.435432774728848 ], [ -0.8202697952701047, -1.6368545392988916, 0.22531426639622112, 0.8510179163141428 ], [ 1.708757639361888, -0.990568259384367, 1.559323968575898, -1.9968880973172087 ], [ 0.8198641680093819, -1.250613720678864, -1.1956995326853672, -0.4339855503243902 ], [ 0.604903182878653, -0.8842555362963179, -1.229406456801794, 0.6804463640177416 ], [ 1.6165533037378936, 1.3370892919353166, 1.39928507726431, 1.1600004231382701 ], [ 0.17528189986255197, -0.7061057716940884, 0.2733408979493073, 1.43975205173321 ], [ 2.0463565719416446, 0.28958092901119964, -0.612476983376016, 0.601297791162572 ], ... ],
    	[ [ -2.1574890970887872, -1.7417815934832035, 0.773130553501822, 0.49594237125147617 ], [ -1.2365934391071491, -0.029503218855537264, 0.29265056651597215, 0.13836183481246814 ], [ -1.799755802452663, -0.3749302877656891, 0.18993271119769103, -1.7636893920483112 ], [ -1.244186863132481, -0.3209871788011414, -0.06696339866479228, 0.27684535050287257 ], [ 0.8771255151928966, -0.02494396301129225, 2.0130878648105193, -1.2808034782223858 ], [ -1.2323577378376165, 0.5645358363054737, 0.44493654078612865, 0.9440989337282931 ], [ 1.5293588933868816, 0.5093413550690868, 1.0178656507491324, 0.28213447832850846 ], [ -2.005738921266441, 0.7067285076520798, 0.03059273359398976, -0.26782922178200025 ], ... ],
    	[ [ -0.5928960996512204, 1.2889483436402274, -0.3464706714339511, -2.608879896181939 ], [ -0.47449333496396484, 1.7164238770223936, 1.7893775775136787, 0.5104201160325036 ], [ 0.9840165067452266, 1.261721437507309, 0.4508245420058037, 1.6645140051178664 ], [ 0.945289711284158, 1.5570689418635857, 1.2301329043637166, 0.34131715306185 ], [ 0.4041054551983949, 0.38379690468694405, 0.44514819226456165, -0.9665227532303862 ], [ -1.814731207937434, 1.1406423385597217, -0.580953672996438, 1.1174859112349629 ], [ 0.9382782696102526, -0.4846876843910342, 0.27738468034687647, 0.9199186325614859 ], [ -0.48503877382828564, 0.13322243239440446, 2.111758741419089, -0.6835844826268039 ], ... ],
    	[ [ 0.8829788350182104, -1.8787657538914244, 0.02693847464562194, 0.23995742657086858 ], [ -2.132008223294772, 0.4682003818163932, -0.18453641006068514, 0.9956848184848478 ], [ 1.6393289873363461, 0.6852250267337919, 0.48555348066646953, -1.5236558752299583 ], [ -0.2817083599503833, -0.7663489111822657, 0.5248914757908493, 1.2984622714584682 ], [ 0.6952388243287743, -0.5203257832592448, -1.8352557894345725, -1.3834774758711603 ], [ 1.2893175081528474, -1.8254012976158533, -1.8625994439436067, -0.5277911869709592 ], [ 0.3530609653875068, -0.9699441281241804, -1.127886114156107, 1.7430787460719317 ], [ -1.1493000998184737, -1.58486866047939, 1.303599986957453, 1.3399000345741272 ], ... ],
    	[ [ 0.3770098224335296, -0.2704022497795756, 0.12085575623594152, -1.7292031909553212 ], [ -2.647416529877358, 0.7740260704847789, 1.2771270405988309, 0.9895658393786317 ], [ 0.8602219327513925, 0.4930792509706479, 0.3739252852420932, -0.3071174768653008 ], [ -1.719944409706924, 0.7927667091182645, -0.9559975292468204, 0.8723862279316396 ], [ 0.01400650044492687, -1.4900276784285078, -0.5401161076880102, 1.1461339931818824 ], [ 0.022683694473861194, -0.8755993668907376, 0.22175992984248727, 1.304633150372796 ], [ -1.5112386405132254, -1.6751442367862794, -0.5300158595797615, 0.41642707008581736 ], [ 0.7980071034664151, 0.8187221034080978, -0.24956586836815278, 1.5555950598314292 ], ... ],
    	[ [ -0.21917652617609512, -0.4376999131281624, 0.7764093699861749, 0.8333769582840009 ], [ 1.5199580355549311, -0.7273423847392473, 0.623234717216964, -1.0775236986723098 ], [ 1.0444714505989623, 1.0447254745887005, -1.8118490471236408, 0.8467681606545437 ], [ -0.9696006719779828, 0.617262223150946, -0.46232691019136823, -0.8548227247737888 ], [ -2.0505320371897513, 0.6384022605352199, 0.46959744378044393, 1.7554596834026754 ], [ -0.8593096503313968, -1.6092775443398506, 0.4825715404071457, 0.47873661189033706 ], [ -1.2916542693347806, 1.913799756507716, 0.3538059120687914, -0.42363222074431783 ], [ -0.09108879700759302, 1.348688809380831, 1.4714358460108516, -1.6220656514011798 ], ... ],
    	...
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.06 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ -3.531063266070487, 6.331604297163652, 1.2460766937170833, 2.5341030054657714 ], [ -2.0850147101873358, -9.250344133832018, -0.5246442050453182, -11.323054894287523 ], [ 3.4425650684482836, 6.2230231140604815, 6.005243861869746, 1.0966695113432283 ], [ 4.2041066581144255, -5.137950551815655, -13.055906803893613, 3.797635830623029 ], [ 0.3414511962146416, -4.897362846520481, 3.7090713151633223, 14.581525908225679 ], [ -11.484740884225644, -0.8027200081482353, -4.161921418167515, -3.9155749547639793 ], [ 3.924479010063451, 0.6281705418262925, 10.450742555275157, -8.874709877292648 ], [ 16.966194450423654, 3.9515945054069697, 6.096016435419726, -1.5248161380577052 ], ... ],
    	[ [ -11.072928112561065, 5.036477313791801, 1.962597471362306, -1.9623037789345719 ], [ 6.782305332157086, 12.789948101391811, -6.308061888585583, -15.006031246390776 ], [ 6.5224742922765, 2.461365752215706, 13.230960836045973, 3.8137288751063583 ], [ -2.71091445282674, 10.947594534924622, -15.33132320827165, -7.330284097193526 ], [ -3.251841187696617, -2.003000101289956, -0.7256770124468792, -3.3602299462638574 ], [ -12.524491753328075, -6.607676256912973, 5.0593858883376255, 5.4007736343218715 ], [ 9.889520794457203, 0.04587722697990748, -7.978223401544909, 10.079783022800285 ], [ 6.135888613026329, -3.3069583628023, 12.69254114736325, 6.088486217839723 ], ... ],
    	[ [ -6.954499946844814, 9.089860582682572, -5.848231089366669, -6.706807510283166 ], [ -6.941916696678566, -8.587345106485715, 4.656263285772567, -21.47112728667389 ], [ 0.39917017270492305, 9.620843196646646, -2.574735951864062, 3.3458198279549363 ], [ 5.860119497180736, -2.7176588134058046, -16.86316966441897, -8.182112592631581 ], [ -6.292958771281473, -1.0966010695600232, 2.1665555968082457, -4.5427246559804715 ], [ -1.8804398708260122, 14.23497062388422, -14.505033280784136, -9.327736069828548 ], [ 1.1062145518981579, -14.989629485843341, 9.817187932842147, -0.1037839053508649 ], [ 5.902430586084556, -3.4590132181447695, -10.361280779524868, 5.0130849894717615 ], ... ],
    	[ [ -8.248404478089611, -5.400687392384521, 3.0343638270805173, -6.403908851373212 ], [ 14.757284622018908, 0.22184058213684466, -3.8449194475898643, 14.948865565543503 ], [ 12.096561231409037, -18.82622832614192, 8.243269797692815, 12.137036139271995 ], [ 5.280836174352964, -25.800735663997152, 0.20958217670631718, 6.818685617899394 ], [ 9.534136980862716, 13.08724663897318, -1.0842104497856377, 11.356598102715429 ], [ -13.559020790076826, -7.39783604078346, -2.1336741314001224, 0.3803978034041552 ], [ 8.778707044715798, -7.0757665695434735, 3.5073847716450977, 8.722573198722735 ], [ -6.598350069291145, -4.327398387887889, 25.92180948737891, 0.2397231422065622 ], ... ],
    	[ [ -5.71593621095499, 12.67299833940595, 3.798332188541985, -4.476627091595984 ], [ -4.637054434996413, -6.447088258358226, -3.0993044557318705, -23.666862855754836 ], [ -5.707352105223286, -2.408354497236586, 9.784562344146542, -8.679335062025292 ], [ 11.01083611136379, 9.608366844106524, -3.1803449883821027, -6.4525961603058 ], [ 4.427683316513028, 2.0522517477202054, 11.906570987466282, 6.589478341488976 ], [ -7.650765967230871, -0.6486867548743899, -11.490575015628709, 6.471892218661558 ], [ -10.703735322176628, -0.12770370233584427, 1.8385247052619462, -7.05979204562299 ], [ -5.62304608061283, -4.045188683064386, 2.1178364119513535, 0.6150095770210675 ], ... ],
    	[ [ -4.745818785579385, 7.95954068100499, -5.512293338487006, -8.959038860341886 ], [ -1.19441093004197, 2.4533459051295843, -11.699426013905203, -13.831086296798947 ], [ 23.62766003800953, 7.905954894175267, -4.422225106993873, 9.925126926642212 ], [ -0.6567514995879122, -3.493720294616262, -2.0444203267675096, 3.5055720662181384 ], [ -2.451911890049316, -1.6792974334946822, -4.720072046490974, 11.933552938789024 ], [ -7.846853162985881, 4.173829060095126, -15.452857956284177, -0.3451996459339468 ], [ -7.198205900206462, 13.24332691979654, -5.634782760587037, -9.216886612215568 ], [ -10.642325462504012, -12.799367986751491, 9.600422617251544, 0.9635369980729234 ], ... ],
    	[ [ 4.028372611233608, 0.8906943950288103, 3.751276223581732, -1.4826123315835567 ], [ 2.3404152673364926, -5.975712563227303, 4.801232098646972, 2.147144434985567 ], [ 5.554822291235661, 6.149551346576857, 7.492561346128602, 2.4154563415965207 ], [ -13.154163279400592, -9.68727476085474, -1.2798730442922546, 11.913125734009085 ], [ -8.64837378865445, -7.688538851062107, -2.6856209059759752, -4.477656744913623 ], [ -6.432271624558503, -4.182081743499366, 9.073901600531894, -1.7026486460315695 ], [ -14.735437336284669, -3.9677382248382624, -3.464956511050224, -3.7094994032248074 ], [ 0.5577497894015367, 11.04382980059357, 13.678205823113942, 13.458222759199643 ], ... ],
    	[ [ -0.8117515457321876, -7.5609391541041555, -3.4678585425426682, -7.572589539427322 ], [ 6.303779378108064, -3.678252164959388, -10.551503500240763, 11.573479241161353 ], [ 9.35454486436349, 5.711835892045498, 6.842019984898448, -7.1468852992504335 ], [ -0.6968458131749968, 8.498032707565567, -7.270888362748828, -0.6124586424124754 ], [ 5.62270681166983, 1.5850213386633374, 6.242456633010423, -2.5838437340069476 ], [ -2.84761480766392, -1.2686810482968953, 16.138945947093, 13.610080126036062 ], [ -5.000311580962024, 16.749205204927158, 8.921540172272646, -0.4034617517362902 ], [ 9.20896599220532, 4.35887155444043, 7.004444150256308, 10.313343989472664 ], ... ],
    	...
    ]

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 30.05 seconds (0.606 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 13767295998234
Reset training subject: 13767441576138
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=140.96797785740105}, derivative=-0.019421814301975097}
New Minimum: 140.96797785740105 > 140.9679778573991
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=140.9679778573991}, derivative=-0.01942181430197493}, evalInputDelta = -1.9610979506978765E-12
New Minimum: 140.9679778573991 > 140.96797785738744
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=140.96797785738744}, derivative=-0.019421814301973928}, evalInputDelta = -1.361399881716352E-11
New Minimum: 140.96797785738744 > 140.96797785730587
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=140.96797785730587}, derivative=-0.019421814301966937}, evalInputDelta = -9.518430488242302E-11
New Minimum: 140.96797785730587 > 140.96797785673488
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=140.96797785673488}, derivative=-0.019421814301917986}, evalInputDelta = -6.661764473392395E-10
New Minimum: 140.96797785673488 > 140.96797785273787
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=140.96797785273787}, derivative=-0.01942181430157532}, evalInputDelta = -4.663178287955816E-9
New Minimum: 140.96797785273787 > 140.9679778247588
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=140.9679778247588}, derivative=-0.019421814299176665}, evalInputDelta = -3.264224801569071E-8
New Minimum: 140.9679778247588 > 140.96797762890532
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=140.96797762890532}, derivative=-0.019421814282386072}, evalInputDelta = -2.2849573610983498E-7
New Minimum: 140.96797762890532 > 140.96797625793113
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=140.96797625793113}, derivative=-0.019421814164851947}, evalInputDelta = -1.5994699253951694E-6
New Minimum: 140.96797625793113 > 140.96796666111186
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=140.96796666111186}, derivative=-0.019421813342113015}, evalInputDelta = -1.1196289193549092E-5
New Minimum: 140.96796666111186 > 140.96789948338846
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=140.96789948338846}, derivative=-0.019421807582940515}, evalInputDelta = -7.837401258825594E-5
New Minimum: 140.96789948338846 > 140.96742923988225
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=140.96742923988225}, derivative=-0.019421767268733032}, evalInputDelta = -5.4861751880253E-4
New Minimum: 140.96742923988225 > 140.9641375626694
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=140.9641375626694}, derivative=-0.019421485069280625}, evalInputDelta = -0.0038402947316455993
New Minimum: 140.9641375626694 > 140.94109716138072
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=140.94109716138072}, derivative=-0.019419509673113802}, evalInputDelta = -0.026880696020327832
New Minimum: 140.94109716138072 > 140.77987997322177
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=140.77987997322177}, derivative=-0.019405681899946056}, evalInputDelta = -0.1880978841792853
New Minimum: 140.77987997322177 > 139.65457507832903
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=139.65457507832903}, derivative=-0.01930888748777177}, evalInputDelta = -1.313402779072021
New Minimum: 139.65457507832903 > 131.9349965028626
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=131.9349965028626}, derivative=-0.018631326602551845}, evalInputDelta = -9.032981354538464
New Minimum: 131.9349965028626 > 85.61817522494863
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=85.61817522494863}, derivative=-0.01388840040601233}, evalInputDelta = -55.34980263245242
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=139.69163504676573}, derivative=0.019312082969764267}, evalInputDelta = -1.2763428106353274
139.69163504676573 <= 140.96797785740105
New Minimum: 85.61817522494863 > 27.6953221089507
F(11664.477271511252) = LineSearchPoint{point=PointSample{avg=27.6953221089507}, derivative=5.421010862427522E-20}, evalInputDelta = -113.27265574845035
Right bracket at 11664.477271511252
Converged to right
Fitness changed from 140.96797785740105 to 27.6953221089507
Iteration 1 complete. Error: 27.6953221089507 Total: 5.4272; Orientation: 0.0937; Line Search: 4.9073
F(0.0) = LineSearchPoint{point=PointSample{avg=27.6953221089507}, derivative=-0.0025467917222940844}
New Minimum: 27.6953221089507 > 12.32749005816508
F(11664.477271511252) = LineSearchPoint{point=PointSample{avg=12.32749005816508}, derivative=-8.818825890391719E-5}, evalInputDelta = -15.367832050785617
F(81651.34090057877) = LineSearchPoint{point=PointSample{avg=522.3653063392939}, derivative=0.014663432521437089}, evalInputDelta = 494.6699842303432
F(6280.872376967598) = LineSearchPoint{point=PointSample{avg=15.85675687103748}, derivative=-0.001222928318930148}, evalInputDelta = -11.838565237913219
F(43966.106638773184) = LineSearchPoint{point=PointSample{avg=119.44072414596222}, derivative=0.0067202521012534695}, evalInputDelta = 91.74540203701153
F(3382.0082029825526) = LineSearchPoint{point=PointSample{avg=20.28748308319622}, derivative=-0.0018339421974058112}, evalInputDelta = -7.407839025754477
F(23674.057420877867) = LineSearchPoint{point=PointSample{avg=26.468570682620815}, derivative=0.002443154951923829}, evalInputDelta = -1.2267514263298835
26.468570682620815 <= 27.6953221089507
New Minimum: 12.32749005816508 > 12.309041249219838
F(12082.873306868843) = LineSearchPoint{point=PointSample{avg=12.309041249219838}, derivative=-3.4220131069073734E-19}, evalInputDelta = -15.38628085973086
Left bracket at 12082.873306868843
Converged to left
Fitness changed from 27.6953221089507 to 12.309041249219838
Iteration 2 complete. Error: 12.309041249219838 Total: 2.0483; Orientation: 0.0899; Line Search: 1.8203
F(0.0) = LineSearchPoint{point=PointSample{avg=12.309041249219838}, derivative=-8.448392903838122E-4}
New Minimum: 12.309041249219838 > 7.30165409274095
F(12082.873306868843) = LineSearchPoint{point=PointSample{avg=7.30165409274095}, derivative=1.599882681091372E-5}, evalInputDelta = -5.007387156478888
7.30165409274095 <= 12.309041249219838
New Minimum: 7.30165409274095 > 7.299857725976068
F(11858.310995380169) = LineSearchPoint{point=PointSample{avg=7.299857725976068}, derivative=4.573977915173222E-20}, evalInputDelta = -5.00918352324377
Right bracket at 11858.310995380169
Converged to right
Fitness changed from 12.309041249219838 to 7.299857725976068
Iteration 3 complete. Error: 7.299857725976068 Total: 1.0571; Orientation: 0.0897; Line Search: 0.8263
F(0.0) = LineSearchPoint{point=PointSample{avg=7.299857725976068}, derivative=-3.6484050108957506E-4}
New Minimum: 7.299857725976068 > 4.997763270900839
F(11858.310995380169) = LineSearchPoint{point=PointSample{avg=4.997763270900839}, derivative=-2.3426336569193247E-5}, evalInputDelta = -2.3020944550752294
F(83008.17696766119) = LineSearchPoint{point=PointSample{avg=76.20569870377162}, derivative=0.002025058650553098}, evalInputDelta = 68.90584097779555
F(6385.244382127784) = LineSearchPoint{point=PointSample{avg=5.557188509554955}, derivative=-1.810021048093694E-4}, evalInputDelta = -1.7426692164211133
F(44696.710674894486) = LineSearchPoint{point=PointSample{avg=19.752088039447127}, derivative=9.220282728718641E-4}, evalInputDelta = 12.452230313471059
F(3438.208513453422) = LineSearchPoint{point=PointSample{avg=6.215633977320801}, derivative=-2.6585059540023355E-4}, evalInputDelta = -1.0842237486552673
F(24067.459594173954) = LineSearchPoint{point=PointSample{avg=6.8575981516171485}, derivative=3.280888387358153E-4}, evalInputDelta = -0.4422595743589195
6.8575981516171485 <= 7.299857725976068
New Minimum: 4.997763270900839 > 4.988232674856081
F(12671.976078403859) = LineSearchPoint{point=PointSample{avg=4.988232674856081}, derivative=-2.879912020664621E-20}, evalInputDelta = -2.3116250511199867
Left bracket at 12671.976078403859
Converged to left
Fitness changed from 7.299857725976068 to 4.988232674856081
Iteration 4 complete. Error: 4.988232674856081 Total: 2.2428; Orientation: 0.0895; Li

...skipping 6022 bytes...

7903403E-4}, evalInputDelta = 7.070188421753862
F(6829.638314130548) = LineSearchPoint{point=PointSample{avg=1.436630809071405}, derivative=-1.659929725786819E-5}, evalInputDelta = -0.17311518239154267
F(47807.46819891383) = LineSearchPoint{point=PointSample{avg=2.9073551204878196}, derivative=8.838076131058287E-5}, evalInputDelta = 1.297609129024872
F(3677.4975537626024) = LineSearchPoint{point=PointSample{avg=1.5016815121529121}, derivative=-2.4674686378518273E-5}, evalInputDelta = -0.1080644793100356
F(25742.482876338217) = LineSearchPoint{point=PointSample{avg=1.5808766178460578}, derivative=3.1853037466032306E-5}, evalInputDelta = -0.028869373616889904
1.5808766178460578 <= 1.6097459914629477
New Minimum: 1.3833554057240007 > 1.3828544303861734
F(13308.994379609912) = LineSearchPoint{point=PointSample{avg=1.3828544303861734}, derivative=8.470329472543003E-22}, evalInputDelta = -0.22689156107677433
Right bracket at 13308.994379609912
Converged to right
Fitness changed from 1.6097459914629477 to 1.3828544303861734
Iteration 10 complete. Error: 1.3828544303861734 Total: 2.0218; Orientation: 0.0900; Line Search: 1.7972
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3828544303861734}, derivative=-2.7853705142167746E-5}
New Minimum: 1.3828544303861734 > 1.2052485563107362
F(13308.994379609912) = LineSearchPoint{point=PointSample{avg=1.2052485563107362}, derivative=1.1641042587923728E-6}, evalInputDelta = -0.17760587407543715
1.2052485563107362 <= 1.3828544303861734
New Minimum: 1.2052485563107362 > 1.204937789680666
F(12775.078920194323) = LineSearchPoint{point=PointSample{avg=1.204937789680666}, derivative=8.470329472543003E-22}, evalInputDelta = -0.1779166407055075
Right bracket at 12775.078920194323
Converged to right
Fitness changed from 1.3828544303861734 to 1.204937789680666
Iteration 11 complete. Error: 1.204937789680666 Total: 1.0436; Orientation: 0.0877; Line Search: 0.8194
F(0.0) = LineSearchPoint{point=PointSample{avg=1.204937789680666}, derivative=-2.128998801923794E-5}
New Minimum: 1.204937789680666 > 1.0626470653556483
F(12775.078920194323) = LineSearchPoint{point=PointSample{avg=1.0626470653556483}, derivative=-9.863087009475126E-7}, evalInputDelta = -0.14229072432501755
F(89425.55244136025) = LineSearchPoint{point=PointSample{avg=5.65590593829657}, derivative=1.2083576720879507E-4}, evalInputDelta = 4.450968148615904
F(6878.888649335404) = LineSearchPoint{point=PointSample{avg=1.0960889190743486}, derivative=-1.0357237617081555E-5}, evalInputDelta = -0.10884887060631732
F(48152.220545347824) = LineSearchPoint{point=PointSample{avg=2.0223043210320677}, derivative=5.523926479585675E-5}, evalInputDelta = 0.8173665313514018
F(3704.0169650267558) = LineSearchPoint{point=PointSample{avg=1.1369818379009589}, derivative=-1.5403122418076816E-5}, evalInputDelta = -0.06795595177970704
F(25928.11875518729) = LineSearchPoint{point=PointSample{avg=1.1871521784309147}, derivative=1.9918071188889962E-5}, evalInputDelta = -0.0177856112497512
1.1871521784309147 <= 1.204937789680666
New Minimum: 1.0626470653556483 > 1.0623410208554518
F(13395.664543950115) = LineSearchPoint{point=PointSample{avg=1.0623410208554518}, derivative=2.541098841762901E-21}, evalInputDelta = -0.1425967688252141
Right bracket at 13395.664543950115
Converged to right
Fitness changed from 1.204937789680666 to 1.0623410208554518
Iteration 12 complete. Error: 1.0623410208554518 Total: 2.3503; Orientation: 0.0890; Line Search: 2.1257
F(0.0) = LineSearchPoint{point=PointSample{avg=1.0623410208554518}, derivative=-1.8121991827682624E-5}
New Minimum: 1.0623410208554518 > 0.9461785191653418
F(13395.664543950115) = LineSearchPoint{point=PointSample{avg=0.9461785191653418}, derivative=7.78693731647159E-7}, evalInputDelta = -0.11616250169011
0.9461785191653418 <= 1.0623410208554518
New Minimum: 0.9461785191653418 > 0.9459636420980098
F(12843.773450958886) = LineSearchPoint{point=PointSample{avg=0.9459636420980098}, derivative=-5.293955920339377E-22}, evalInputDelta = -0.11637737875744203
Left bracket at 12843.773450958886
Converged to left
Fitness changed from 1.0623410208554518 to 0.9459636420980098
Iteration 13 complete. Error: 0.9459636420980098 Total: 1.1077; Orientation: 0.0958; Line Search: 0.8537
F(0.0) = LineSearchPoint{point=PointSample{avg=0.9459636420980098}, derivative=-1.4341020440543173E-5}
New Minimum: 0.9459636420980098 > 0.8496469380466884
F(12843.773450958886) = LineSearchPoint{point=PointSample{avg=0.8496469380466884}, derivative=-6.571737302097413E-7}, evalInputDelta = -0.09631670405132131
F(89906.4141567122) = LineSearchPoint{point=PointSample{avg=3.962543482527364}, derivative=8.144590653179085E-5}, evalInputDelta = 3.0165798404293542
F(6915.878012054784) = LineSearchPoint{point=PointSample{avg=0.8722617673201467}, derivative=-6.97279528882517E-6}, evalInputDelta = -0.07370187477786305
F(48411.146084383494) = LineSearchPoint{point=PointSample{avg=1.5001631912603157}, derivative=3.723655562148284E-5}, evalInputDelta = 0.554199549162306
F(3723.9343141833456) = LineSearchPoint{point=PointSample{avg=0.8999459895690578}, derivative=-1.0373514589618095E-5}, evalInputDelta = -0.04601765252895196
F(26067.54019928342) = LineSearchPoint{point=PointSample{avg=0.934109429174991}, derivative=1.3431520515932376E-5}, evalInputDelta = -0.011854212923018803
0.934109429174991 <= 0.9459636420980098
New Minimum: 0.8496469380466884 > 0.8494442562106808
F(13460.602233640404) = LineSearchPoint{point=PointSample{avg=0.8494442562106808}, derivative=-1.0587911840678754E-21}, evalInputDelta = -0.096519385887329
Left bracket at 13460.602233640404
Converged to left
Fitness changed from 0.9459636420980098 to 0.8494442562106808
Iteration 14 complete. Error: 0.8494442562106808 Total: 2.0194; Orientation: 0.0888; Line Search: 1.7934
F(0.0) = LineSearchPoint{point=PointSample{avg=0.8494442562106808}, derivative=-1.2583745910739772E-5}
New Minimum: 0.8494442562106808 > 0.7684658108678184
F(13460.602233640404) = LineSearchPoint{point=PointSample{avg=0.7684658108678184}, derivative=5.518258023685412E-7}, evalInputDelta = -0.08097844534286236
0.7684658108678184 <= 0.8494442562106808
New Minimum: 0.7684658108678184 > 0.7683097875451861
F(12895.121888347941) = LineSearchPoint{point=PointSample{avg=0.7683097875451861}, derivative=-4.764560328305439E-22}, evalInputDelta = -0.08113446866549467
Left bracket at 12895.121888347941
Converged to left
Fitness changed from 0.8494442562106808 to 0.7683097875451861
Iteration 15 complete. Error: 0.7683097875451861 Total: 1.0464; Orientation: 0.0878; Line Search: 0.8223
F(0.0) = LineSearchPoint{point=PointSample{avg=0.7683097875451861}, derivative=-1.0219578347653576E-5}
New Minimum: 0.7683097875451861 > 0.699423132582375
F(12895.121888347941) = LineSearchPoint{point=PointSample{avg=0.699423132582375}, derivative=-4.6456338582738117E-7}, evalInputDelta = -0.0688866549628111
F(90265.85321843559) = LineSearchPoint{point=PointSample{avg=2.927737448869048}, derivative=5.80655263851298E-5}, evalInputDelta = 2.1594276613238623
F(6943.527170648891) = LineSearchPoint{point=PointSample{avg=0.7155860014642432}, derivative=-4.966877983593318E-6}, evalInputDelta = -0.05272378608094286
F(48604.690194542236) = LineSearchPoint{point=PointSample{avg=1.1651609066183104}, derivative=2.6549324200768234E-5}, evalInputDelta = 0.3968511190731243
F(3738.822322657095) = LineSearchPoint{point=PointSample{avg=0.735387999645745}, derivative=-7.391201228544208E-6}, evalInputDelta = -0.03292178789944111
F(26171.756258599664) = LineSearchPoint{point=PointSample{avg=0.7599280619552914}, derivative=9.579061486112011E-6}, evalInputDelta = -0.008381725589894695
0.7599280619552914 <= 0.7683097875451861
New Minimum: 0.699423132582375 > 0.6992804872804479
F(13509.226685577922) = LineSearchPoint{point=PointSample{avg=0.6992804872804479}, derivative=0.0}, evalInputDelta = -0.06902930026473819
Right bracket at 13509.226685577922
Converged to right
Fitness changed from 0.7683097875451861 to 0.6992804872804479
Iteration 16 complete. Error: 0.6992804872804479 Total: 2.0139; Orientation: 0.0914; Line Search: 1.7856
Final threshold in iteration 16: 0.6992804872804479 (> 0.0) after 30.047s (< 30.000s)

Returns

    0.6992804872804479

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -1.5260592242698772, 1.8096253905688864, -0.28342007475490205, -1.055497252183793 ], [ -0.3966740616328737, -0.2445833456679226, -0.7672139561228838, 1.7536032407397735 ], [ -0.7362513937427871, 1.232946155685005, 0.3507887859948535, 0.9085959383895552 ], [ -0.8033198371840335, -0.47379344720647637, 0.5062036662685951, -0.9262776548340849 ], [ 2.3407175641817837, -0.15202358766202015, -0.35223722528369045, 1.3687158211038923 ], [ 0.29015893253004355, -0.7263999680941047, -0.963238949660335, 0.1614321289940567 ], [ -0.7125967686527542, 0.5454248875592344, 0.6282804649405315, 1.0815488727470033 ], [ -0.2644612303354863, 1.137683038994579, 2.264095530020432, -2.2400160138470575 ], ... ],
    	[ [ -0.46659772560509033, -0.8606126422055195, -1.2167982888764834, 0.1286955217673938 ], [ -0.6311022360069771, 0.12433508027152246, -0.024337325240818487, -0.14904372987897868 ], [ -1.0628694691803608, 0.6816521383817786, 0.3603694570826768, -1.0857426440038478 ], [ 1.5170769208719355, 0.655921179878174, -1.493351140052327, 0.8456158334642687 ], [ 1.498589349712615, 1.523526464160883, 0.7523200405254171, 1.8145994316309502 ], [ -0.7761716489513274, -0.5536453288206192, -0.2602835299669907, 0.2236807508161464 ], [ 0.22220825739040173, -0.5818468609244087, 1.8307158289508003, -0.9588215861761002 ], [ 1.1670727596981418, 0.672016863492997, -0.969613977210941, 0.47348621602269475 ], ... ],
    	[ [ 1.024628912304856, -7.473580867626076E-5, 0.10290805367228682, -1.3884106975753265 ], [ -1.0079346583429434, -1.4457948728581331, 0.2520597738135698, 0.8099080713174446 ], [ 1.9316159991872914, -0.3780672617268508, 1.7565610318213094, -1.9485557040789097 ], [ 0.9821653652678416, -1.1537428596646575, -0.8197771750840117, -0.12910922169160335 ], [ 0.3786301010541581, -1.5340465871565616, -0.8977849597542787, 0.90926029419794 ], [ 1.9136580841724515, 1.3226900268692354, 1.1756728824233125, 1.3042745273426708 ], [ 0.32566851807944014, -0.5165079730779788, 0.36654508174527434, 1.4375477041233233 ], [ 1.9906803282930794, 0.0043422751193291425, -0.42268928069708367, 0.4225598348436469 ], ... ],
    	[ [ -2.2144494441026286, -1.909486132823456, 0.5349499074316879, 0.18348894645816113 ], [ -1.3217763770722788, 0.23705591646028507, 0.3604251674754195, 0.22660784591548383 ], [ -1.8806393802553139, -0.2846815027150198, 0.1860682560803283, -1.9746561954674 ], [ -1.2408065308118197, -0.3331764816953269, 0.09183487448397197, 0.1053097444815568 ], [ 0.8603521098443724, -0.10687613997165861, 1.939165561683751, -1.3220396735628033 ], [ -1.324166652451414, 0.7518065416336456, 0.1876404809693741, 1.2790149663041095 ], [ 1.4309989412281525, 0.7203118201307841, 0.935646862100591, 0.3194277397242842 ], [ -2.0204830942331378, 0.5883359850123914, 0.2304101602756842, -0.2479714186409261 ], ... ],
    	[ [ -0.787215517163921, 1.2398392429818619, -0.263529878043129, -2.0709139731738073 ], [ -0.7146526334180632, 1.625656121601401, 1.6355909984061945, 0.7822612845755152 ], [ 0.9027812600969324, 1.1918952565215497, 0.3169017783821131, 1.9265277508865277 ], [ 0.656586285970611, 0.9507503551047066, 1.4068171103000076, -0.09051992034633963 ], [ 0.4715914900950864, 0.33885622811366123, 0.34929890697126426, -1.0658348596855056 ], [ -2.124040829920571, 1.1614996667463182, -0.577480049557636, 0.860074881968412 ], [ 1.0033939008102404, -0.6017550010075684, 0.3465441477176905, 0.8212394177589549 ], [ -0.3371360133267875, 0.02848864471953704, 2.340458049605975, -0.6976380638108352 ], ... ],
    	[ [ 0.8226727564365474, -1.5129429125654483, 0.024147931529229304, -0.38709637757865334 ], [ -1.9585249665812265, 0.3124161022728282, -0.0436208979979002, 0.9961661640406037 ], [ 1.7023481283527104, 0.36181037315444786, 0.43123157385405586, -1.7996860212328236 ], [ 0.02837604548837149, -1.2362653143355822, 0.6558612710095805, 1.3585823560822983 ], [ 0.505459034250872, -0.9186502371027719, -1.7321247729099254, -1.2679311596645202 ], [ 1.6298384154726213, -1.596537204368514, -1.9210441421382736, -0.31245190859066163 ], [ 0.6240435943129681, -0.9292785133183536, -1.0759069235895895, 2.0013663770453305 ], [ -1.2190272235419204, -1.774369104741052, 1.546115133988097, 0.9115547316786946 ], ... ],
    	[ [ 0.6346819065346293, -0.26847690819553816, 0.12234971577011045, -1.3896152862352615 ], [ -2.7835403191180528, 0.8570078432001476, 1.1983403330100815, 0.7794220056405519 ], [ 0.8311874836384477, 0.5306555336224338, 0.4593001499632962, -0.24256690436241068 ], [ -1.5648469108850198, 0.7518764832705371, -1.064943134742919, 0.2993588151751369 ], [ -0.1256029379903704, -1.1720970814619487, -0.31123660258174907, 1.0667774721880003 ], [ 0.3340539922164959, -0.8701333469204338, 0.12346466043615237, 1.6501391851911054 ], [ -1.3690733647066113, -1.773013573437846, -0.6221034016433714, 0.2573664443928867 ], [ 0.7757877265770723, 0.9601495504882245, 0.008202930976170142, 1.470155090685636 ], ... ],
    	[ [ 1.6650881783831714E-4, -0.2825962310767399, 0.5291627855637066, 0.44113052493932153 ], [ 1.7555333774734443, -0.6958034211708943, 0.6523146482603649, -1.0815511413616854 ], [ 0.6728626042973214, 0.5828519906604328, -1.5485219387323597, 0.47820232821919895 ], [ -0.9709225580178228, 0.586788705884405, -0.4934637194900548, -0.7734246614522177 ], [ -2.176373795567125, 0.2458513859011208, 0.9248434278065931, 1.684097163610449 ], [ -0.9793317697441477, -1.6640943025226496, 0.4845242186403023, 0.3716252298540469 ], [ -1.2589023589074768, 1.6512725203527674, 0.32958379946280253, -0.27021339863705923 ], [ -0.07326047891646699, 1.2137334063803653, 1.5946134463179464, -1.67119196682067 ], ... ],
    	...
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.05 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ -4.325286709583894, 4.888954629618872, 0.8758275860376266, 1.1792072516760723 ], [ -2.5904761600232744, -8.66127380886709, -1.7334659770508278, -11.569752810171334 ], [ 3.4753252845412854, 7.696472809988889, 4.2501297839105225, 2.5412083288372145 ], [ 3.501366501230704, -6.190264552310519, -13.504691676834478, 3.913804313963879 ], [ 0.8805871988083158, -5.5666386434406165, 2.8439752600954495, 12.958652302999146 ], [ -11.568372825856176, -0.38700253060144396, -4.873425719922647, -3.727539794425596 ], [ 3.8617357434504007, 1.514274756714636, 10.135880172266159, -9.018454362144887 ], [ 16.561725099460713, 4.896924088384191, 5.863726766022753, -1.627012022292404 ], ... ],
    	[ [ -11.221093490743119, 4.363717780310713, 1.156118472886295, -1.9004666541323623 ], [ 6.1592968920215245, 12.056079877298465, -7.65965110945951, -15.82126656003468 ], [ 5.942111878602694, 3.6388805364663823, 12.112271315548355, 4.262387103558272 ], [ -2.188066191326255, 10.558119895565873, -14.619768914257893, -7.2278655227831266 ], [ -2.875532677743292, -2.061866298063266, -1.4204094187260812, -4.264445149799515 ], [ -11.753139969264302, -6.333297094617565, 3.914385799648022, 4.749836194321281 ], [ 9.980203404737214, 0.9230125231383133, -7.932875387414453, 10.196545837603662 ], [ 6.1795270442774655, -3.0996919700185757, 12.4733644172611, 5.457024583806417 ], ... ],
    	[ [ -6.897851707921295, 8.933997655367582, -5.874004626449258, -6.513567712232246 ], [ -6.987819881747376, -8.428973625824884, 4.550435782686385, -21.472178013128143 ], [ 0.8970898174687545, 9.97324376110463, -3.327437725625349, 3.155281551042575 ], [ 5.336802692062858, -2.089324940207771, -17.209470910628244, -8.708843796132804 ], [ -6.827782382778586, -1.1379091726532131, 1.3402522552783884, -4.554232514056973 ], [ -2.832107798969687, 14.600509019131673, -15.377491394332282, -8.583777735443222 ], [ 0.017807796310029822, -13.777446953452747, 9.714147247378703, -0.03241223414515115 ], [ 6.5247848565974635, -3.3449204195403968, -10.240701588266898, 5.203798956056519 ], ... ],
    	[ [ -8.182949259825055, -4.913412857997377, 2.853000885532318, -6.224016136640204 ], [ 14.724894579509131, 0.7345243526611634, -3.966222709101517, 15.15675472371028 ], [ 12.490887476451276, -19.366513187862104, 8.509668761885447, 11.692944466985555 ], [ 5.648784357938643, -25.55523014084004, 0.3731822094342405, 6.457783656324636 ], [ 9.932504485762523, 13.189647444621466, -1.0731080389239538, 10.850653740627408 ], [ -14.159136890353562, -6.947211299711436, -1.566373552852577, 0.37907137039838845 ], [ 8.529253731275126, -7.011548008165657, 3.001358730806215, 8.363579136582539 ], [ -6.680321125921148, -4.4889078300344085, 25.64722398042056, -0.09682089714772468 ], ... ],
    	[ [ -6.457507515888516, 13.372260026313095, 4.184484131862022, -5.142290899566657 ], [ -4.112149695869967, -6.491791736963254, -3.2887515594924928, -22.800142982225584 ], [ -6.225660829582635, -1.267161533988796, 9.427171877810892, -8.440955699027235 ], [ 11.217219078198042, 9.11768574179299, -2.955816540280176, -6.549187326227434 ], [ 3.440506766134831, 2.136200107115803, 11.410764018515241, 6.159543937907865 ], [ -7.389562287254389, -0.8541643960238348, -11.587139613016578, 6.882250952984414 ], [ -10.8829542585194, -0.5019829933228994, 1.8580006092587336, -7.2238587288263 ], [ -6.244284377427487, -3.57401503504604, 1.6240472808253763, 0.05404620975264818 ], ... ],
    	[ [ -4.30762620063172, 8.990341635764034, -4.910222560746448, -7.656412724144115 ], [ -2.192335207396786, 2.792596473113095, -11.796373085149611, -14.295962313775554 ], [ 23.444807660242166, 7.586275226554118, -3.991821905575106, 10.221880495217237 ], [ -1.4392443360324747, -3.1371752039272818, -1.9543090141789081, 3.5561521452899063 ], [ -3.4232190037144323, -2.368999128610299, -5.433216242386846, 11.613442571494216 ], [ -7.931113684588162, 4.273358370224756, -16.174219443237483, 0.3414216316266405 ], [ -7.539986923760662, 12.958218745542213, -5.468588387144569, -10.064471138401833 ], [ -10.271256940871044, -12.838758526969745, 9.085387434547881, 0.4941613226864857 ], ... ],
    	[ [ 3.9409945008673772, 0.7848585945830786, 3.5929424846715077, -1.7774455810398258 ], [ 3.0253798887025845, -6.284544950348453, 5.703552463912082, 2.644332273855545 ], [ 5.46994265040334, 5.8687854661489105, 7.522390371654565, 1.8773293286023658 ], [ -13.136148285348465, -10.083105212938607, -1.2089310946531535, 11.46025780965323 ], [ -8.876528183684659, -8.602930219177129, -3.0253216176798197, -4.904680631413993 ], [ -6.330180444790635, -4.902282510056492, 8.25551697947073, -2.0969908041893817 ], [ -14.89234311393923, -3.637092072727394, -3.7128934752840754, -3.7670658704476168 ], [ 1.0118273681082772, 11.20737237210907, 13.480076484495573, 12.493762332062802 ], ... ],
    	[ [ -0.22563075821317558, -7.003637972433941, -4.305540602762848, -6.228533079306557 ], [ 6.1114723661243495, -3.200628607647745, -10.130340951900125, 10.86726958772393 ], [ 10.52329171415795, 4.603591476492018, 6.68651570336082, -7.200191180705589 ], [ -0.4301143946415533, 8.88863308917681, -6.877264631408269, -0.42410762373775235 ], [ 6.022312123156944, 0.8756923908182432, 5.914842479193972, -3.062472627685626 ], [ -2.826900192734163, -1.7833474095724826, 16.605696782625365, 13.555710869313813 ], [ -5.673772607032941, 17.328914115468237, 7.949947997035785, -1.0659304287808875 ], [ 9.7293780767739, 5.168212859732475, 6.128203377757222, 10.962105666391386 ], ... ],
    	...
    ]

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 33.31 seconds (0.655 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 13797434474059
Reset training subject: 13797579536756
Adding measurement 145be8ea to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 140.96797785740105 < 140.96797785740105. Total: 1
th(0)=140.96797785740105;dx=-0.019421814301975097
Adding measurement 7a8fa4b4 to history. Total: 1
New Minimum: 140.96797785740105 > 140.92613869113967
WOLFE (weak): th(2.154434690031884)=140.92613869113967; dx=-0.019418227083319552 evalInputDelta=0.04183916626138284
Adding measurement 6baf1ea2 to history. Total: 2
New Minimum: 140.92613869113967 > 140.8843072533066
WOLFE (weak): th(4.308869380063768)=140.8843072533066; dx=-0.019414639864664004 evalInputDelta=0.0836706040944648
Adding measurement 6798d2ca to history. Total: 3
New Minimum: 140.8843072533066 > 140.71705878625747
WOLFE (weak): th(12.926608140191302)=140.71705878625747; dx=-0.019400290990041834 evalInputDelta=0.2509190711435849
Adding measurement 517f5cb to history. Total: 4
New Minimum: 140.71705878625747 > 139.96597091334218
WOLFE (weak): th(51.70643256076521)=139.96597091334218; dx=-0.01933572105424204 evalInputDelta=1.002006944058877
Adding measurement 285e086a to history. Total: 5
New Minimum: 139.96597091334218 > 136.00245888418522
WOLFE (weak): th(258.53216280382605)=136.00245888418522; dx=-0.018991348063309835 evalInputDelta=4.965518973215836
Adding measurement 19052e7d to history. Total: 6
New Minimum: 136.00245888418522 > 112.84420453354849
END: th(1551.1929768229563)=112.84420453354849; dx=-0.016839016869983514 evalInputDelta=28.12377332385256
Fitness changed from 140.96797785740105 to 112.84420453354849
Iteration 1 complete. Error: 112.84420453354849 Total: 2.4803; Orientation: 0.1212; Line Search: 1.9416
Non-optimal measurement 112.84420453354849 < 112.84420453354849. Total: 7
Rejected: LBFGS Orientation magnitude: 1.680e+03, gradient 1.210e-01, dot -0.967; [ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00, ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 112.84420453354849, 136.00245888418522, 139.96597091334218, 140.71705878625747, 140.8843072533066, 140.92613869113967, 140.96797785740105
Rejected: LBFGS Orientation magnitude: 1.680e+03, gradient 1.210e-01, dot -0.967; [ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00, ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 112.84420453354849, 136.00245888418522, 139.96597091334218, 140.71705878625747, 140.8843072533066, 140.92613869113967
Rejected: LBFGS Orientation magnitude: 1.680e+03, gradient 1.210e-01, dot -0.967; [ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00, ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 112.84420453354849, 136.00245888418522, 139.96597091334218, 140.71705878625747, 140.8843072533066
Rejected: LBFGS Orientation magnitude: 1.680e+03, gradient 1.210e-01, dot -0.967; [ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00, ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 112.84420453354849, 136.00245888418522, 139.96597091334218, 140.71705878625747
LBFGS Accumulation History: 3 points
Removed measurement 19052e7d to history. Total: 6
Removed measurement 285e086a to history. Total: 5
Removed measurement 517f5cb to history. Total: 4
Removed measurement 6798d2ca to history. Total: 3
Adding measurement 29c91e86 to history. Total: 3
th(0)=112.84420453354849;dx=-0.014644730686661295
Adding measurement 53df118 to history. Total: 4
New Minimum: 112.84420453354849 > 70.63150053536918
END: th(3341.943960201201)=70.63150053536918; dx=-0.010617634272706773 evalInputDelta=42.21270399817931
Fitness changed from 112.84420453354849 to 70.63150053536918
Iteration 2 complete. Error: 70.63150053536918 Total: 16.6924; Orientation: 15.9896; Line Search: 0.5676
Non-optimal measurement 70.63150053536918 < 70.63150053536918. Total: 5
Rejected: LBFGS Orientation magnitude: 1.314e+03, gradient 8.863e-02, dot -0.977; [ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00, ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 70.63150053536918, 112.84420453354849, 140.8843072533066, 140.92613869113967, 140.96797785740105
Rejected: LBFGS Orientation magnitude: 1.314e+03, gradient 8.863e-02, dot -0.977; [ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00, ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 70.63150053536918, 112.84420453354849, 140.8843072533066, 140.92613869113967
LBFGS Accumulation History: 3 points
Removed measurement 53df118 to history. Total: 4
Removed measurement 29c91e86 to history. Total: 3
Adding measurement 3734fd52 to history. Total: 3
th(0)=70.63150053536918;dx=-0.00785595654856587
Adding measurement 27c8805f to history. Total: 4
New Minimum: 70.63150053536918 > 29.043442710341342
END: th(7200.000000000001)=29.043442710341342; dx=-0.0036962817361640855 evalInputDelta=41.588057825027846
Fitness changed from 70.63150053536918 to 29.043442710341342
Iteration 3 complete. Error: 29.043442710341342 Total: 7.1235; Orientation: 6.3930; Line Search: 0.5721
Non-optimal measurement 29.043442710341342 < 29.043442710341342. Total: 5
Rejected: LBFGS Orientation magnitude: 7.811e+02, gradient 4.599e-02, dot -0.998; [ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00, ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 29.043442710341342, 70.63150053536918, 140.8843072533066, 140.92613869113967, 140.96797785740105
Rejected: LBFGS Orientation magnitude: 7.811e+02, gradient 4.599e-02, dot -0.998; [ee3430f2-7eea-40e0-b3c6-d97a1936ae47 = 1.000/1.000e+00, ab44c6d5-c0e6-4dac-949d-820aaa0f6651 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 29.043442710341342, 70.63150053536918, 140.8843072533066, 140.92613869113967
LBFGS Accumulation History: 3 points
Removed measurement 27c8805f to history. Total: 4
Removed measurement 3734fd52 to history. Total: 3
Adding measurement 78cec141 to history. Total: 3
th(0)=29.043442710341342;dx=-0.002114758224201549
Adding measurement b9acad7 to history. Total: 4
New Minimum: 29.043442710341342 > 9.25488866904522
END: th(15511.929768229566)=9.25488866904522; dx=-4.3663987222687065E-4 evalInputDelta=19.788554041296123
Fitness changed from 29.043442710341342 to 9.25488866904522
Iteration 4 complete. Error: 9.25488866904522 Total: 7.0109; Orientation: 6.3067; Line Search: 0.5498
Final threshold in iteration 4: 9.25488866904522 (> 0.0) after 33.307s (< 30.000s)

Returns

    9.25488866904522

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ -0.9416702235905013, 0.6957075606828759, -0.3435949102992497, -0.5284587737587598 ], [ 0.39329008964104994, 0.7632278854504253, -1.880219276658273, 1.609341871936365 ], [ -0.22423042507173152, 1.6983060567362145, 0.8730225556216353, 0.10842625499394615 ], [ 0.12789362547060187, 0.02979521848345753, -0.39946407710015974, -0.11980880342557426 ], [ 1.9662909138035043, 0.4893935169018767, -0.6109780057261351, 1.3673379490408055 ], [ 0.15461747469442988, 0.7857207672478589, -1.2829033243362988, 0.7464803585950047 ], [ -0.015979799591144872, 0.5944054635456384, 0.3353054791908896, 2.0229043895108063 ], [ 0.7710181356970169, 0.5279684142113852, 0.693006263176725, -1.4929158325571923 ], ... ],
    	[ [ -0.7086015526471079, -0.968251264401166, -1.358978946739192, 0.3760582612961151 ], [ -0.5561736718654329, 0.05837520652294281, 0.4720939272710832, -0.5611437197483016 ], [ 0.06148055628409993, 1.6091434045849433, 0.2336572558898145, 0.12459600718197028 ], [ 0.862602024383554, 0.3689638940735022, -1.242003620124048, 1.342059377407433 ], [ 0.559082051949729, 1.708884001201911, 1.0208527441376598, 1.510775626661987 ], [ -0.9163361682102683, 0.03643176337355358, -0.6031145457335845, 0.30914301355719004 ], [ 0.052100631511348305, 0.5235990499664406, 1.1120056496167898, -0.8147187458105273 ], [ 0.22743760523931472, 0.137007245928926, -0.5030939409115099, 0.8209333282295848 ], ... ],
    	[ [ 0.27784871243433634, -1.3415380683278961, -0.5812583711102477, -1.0117336167185875 ], [ -1.108519631969237, -1.4288058765739, 0.11511570093395401, 0.38657016840931613 ], [ 1.8403898471892193, 1.3562107601773585, 2.034576735123105, -2.5944758838200164 ], [ 1.2058145081427778, -0.37026524202468625, -0.34451191633971007, 0.5210784117653222 ], [ 0.2873077998541646, -2.0382414357136747, 0.37484398719336587, 1.1942738535970125 ], [ 1.9841153995044667, 1.1383731701926962, 1.3656498868724818, 1.2643465111444543 ], [ 0.3457136590025621, -0.4698783397366848, 0.7346816929266105, 0.9434474283371315 ], [ 2.200563822576366, 0.18382010505500357, -0.47602596740093184, -0.4807071348205482 ], ... ],
    	[ [ -1.848284693407373, -0.524393946163781, -0.7820897552024728, -0.8184339805014746 ], [ -1.5147658173258027, 0.36136720002053, 0.5597006913651219, 0.44288598769159765 ], [ -2.1373386322523236, -0.8308232136086561, 0.5248696609322343, -2.0694191294765893 ], [ -1.094910969801669, -0.04954226488056879, -0.5043916416545317, -0.15321704656793483 ], [ 1.5866143269919195, -0.19456527434021917, 1.3283656874919314, -1.4377956835204628 ], [ -1.4831451980516757, 1.5179317050116463, -0.7540443898759875, 1.275095079420273 ], [ 0.9868887870807821, 0.9593285494783395, 0.39693226855340014, 0.780438262477629 ], [ -1.3976666987568485, -0.12519182369448545, 0.537464318382255, -0.3212889613189902 ], ... ],
    	[ [ -1.2914431524547811, 0.5749229526326627, -0.510633719768733, -0.7535586125099603 ], [ -1.1768098513123606, 1.5887113503339503, 0.5529298957984156, 1.0710021419416975 ], [ 1.468034786555677, 0.3016099685324046, -0.7900270358966873, 1.8851271468702497 ], [ 1.28112554194279, -0.10007201811710542, 1.3854954488066304, -0.7520773003461753 ], [ 0.6303451431879045, -0.0383658334052374, 0.4743401399634582, -0.9323149220155849 ], [ -2.455275502422708, 0.14341307046795743, -0.44793614117926156, 0.8607756798178348 ], [ 0.19078875632344017, -0.8315603724349285, 0.12510799324029082, 0.926049316597436 ], [ 0.19124961104945387, -1.0693251880234502, 1.7331228570035524, -0.38286603885904386 ], ... ],
    	[ [ 0.5646729696446618, -0.5610650962914632, -0.5637371745273579, -1.440971189996636 ], [ -1.309580602817007, -1.1079553569317067, -0.3496874251226111, 0.6985991320064886 ], [ 1.4269859489749284, 0.5124041797074325, 0.0473225608885508, -1.3148030049073713 ], [ 0.27607175779885523, -1.806055248639924, 1.118913887263251, 0.759447549625184 ], [ 0.1918039787315396, -1.4159829215984088, -1.426946257100662, -1.3353370185690576 ], [ 1.9317387229990957, -1.1774785452585799, -1.5812764939625454, -0.3989556963497374 ], [ 1.2008940450845234, -0.6524258840037901, -0.1171396899002825, 2.669996038623184 ], [ -1.486370585006791, -1.7912916519307056, 1.5458801519317975, 0.3756656658000811 ], ... ],
    	[ [ 0.9939185573354373, -0.2222700353367512, 0.7619454154384504, -0.7484352444636435 ], [ -2.928778018261519, 0.9418411338433694, 1.016605669139909, 0.07170103383230797 ], [ 0.6830165498946117, 1.0375715657338347, 0.2852404740764277, -0.44517932846490105 ], [ -0.6365806911172704, -0.3566945830921612, -1.012038745766356, -0.87881587989957 ], [ -0.3009852502572171, -0.5535284122369227, 0.25172413960589, 1.093185020889904 ], [ 0.7289711614247616, -1.3105616818620884, 0.14104105826380253, 1.2619369637478077 ], [ -0.650279760833257, -0.589711087642398, -0.7201765176913946, 0.023952419929192725 ], [ 0.8952446814091033, 1.135853313497249, 1.2226788824212287, 1.5675534413126337 ], ... ],
    	[ [ 0.6184860889938835, 0.4799884509546375, -0.834774756386568, -0.7957914582348526 ], [ 2.010644772714286, -0.5042219015025564, 1.1042560208953767, -0.7954516766322438 ], [ 0.5317724043105144, -0.1517145401489636, -0.7333873804115737, 0.5903240227518086 ], [ -1.5371716446359411, 0.8838527573118564, -1.4425157940998439, -1.0961885685537813 ], [ -1.9128432766235615, -0.9116066649274257, 1.4926327933036834, 1.550259654201531 ], [ -1.1878504585566771, -1.2602469939004834, 0.28069878439660245, 0.2552289121941085 ], [ -1.1393609669901765, 0.7816523545803682, 0.3595796894960938, 0.17231403264279432 ], [ 0.2318628421412402, 1.9516824241141573, 1.3445114306805763, -1.2685270856279756 ], ... ],
    	...
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.06 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ -4.385119844635036, 1.9085287359713594, -2.2594736971848373, -1.5931623901881409 ], [ -3.7491670095337972, -3.2453957421102477, -2.710971537624274, -9.589003026578371 ], [ 8.84145355997481, 10.06850752356486, 1.759240988108168, 0.9604845967120031 ], [ -1.0346244195631185, -8.37871739733193, -12.753384614352239, 2.078979260524367 ], [ 0.9661637613945238, -1.0011347125895664, 4.05596427329349, 5.891794290597163 ], [ -8.672006574381133, 0.933950178746624, -6.498190612851068, -6.175345703752576 ], [ 2.4716277088530227, 0.4641628946320555, 5.2684371530699705, -6.108110948211381 ], [ 11.93976053362847, 6.859495376266722, -0.012612510126111642, 1.064579798931037 ], ... ],
    	[ [ -11.774839791746963, 1.678646695364513, 1.0705700090570505, -3.264157713921284 ], [ 4.680058695281183, 7.891228590885585, -11.214231148772623, -20.013395531185946 ], [ 4.804627187741658, 3.465594455947904, 10.226195209915197, 5.232525125150073 ], [ 1.480868599439714, 7.28314824265636, -15.072923271793792, -8.231616433503127 ], [ -2.9793442684001965, 0.10675073040829969, 1.4251553118860356, -4.089332936652546 ], [ -9.331864181908205, -4.333942389547838, 2.947956516278481, -0.6485246399960252 ], [ 12.470613546711931, 3.116242664147742, -4.141096807201901, 8.70832603936244 ], [ 6.322204392972662, -4.210488769532955, 16.2430152749008, 5.381106343920726 ], ... ],
    	[ [ -7.88034664928386, 8.322098710067863, -8.316865629857244, -1.8080739185444685 ], [ -4.34663110133906, -0.7989334470220208, 5.605825431282263, -17.82088558184316 ], [ 4.733495843979375, 11.932296471500475, -0.995564768339094, 1.4052955161647627 ], [ 3.8145802749400386, -2.2694264537629767, -19.683106788639595, -13.317650454875883 ], [ -7.212669958889597, -1.1130798955472825, 4.378040755849849, -5.281758599312133 ], [ -6.606278091998305, 16.8448819133254, -13.301284152708558, -6.013286835169206 ], [ -2.9052367548691773, -10.14682272949937, 10.446682307050832, -0.37759162789301204 ], [ 7.95276239893772, 1.5221663487943538, -8.49502265674351, 9.733615737876727 ], ... ],
    	[ [ -9.25634124573884, 2.704692624514175, 2.284404237545022, -4.8760334942881505 ], [ 16.726547602867534, 0.24666391005981958, -3.5794504589539424, 12.784954315260306 ], [ 12.053110544811094, -22.126923901192914, 7.255480965355867, 13.702180642011298 ], [ 5.834150155106654, -21.303146989315564, -2.920402598126172, 8.936084975443421 ], [ 11.892194794766901, 17.348553912792692, -2.613827470234104, 10.486331275875145 ], [ -15.267499680016067, -4.006328465761803, -0.6835554519510381, -1.4570347502416925 ], [ 7.575233389293677, -9.49115133966614, -1.4542074306099741, 5.2688724274523 ], [ -7.204811618342295, -7.962883681176139, 20.60448229875204, -1.1033970069034251 ], ... ],
    	[ [ -9.01751687687315, 13.784875553389625, 5.305741564733376, -5.4294448252791145 ], [ -4.3488930234190155, 0.07075304609581679, -5.601925877233112, -17.330618254851398 ], [ -8.387979697711593, -2.305671472566811, 1.664520771418541, -9.373661394203541 ], [ 11.345545574823078, 9.421771689403691, -3.4204737870990884, -5.379612808908854 ], [ 1.934633099481701, 2.1414994103870795, 10.626816150796019, 1.9091690608583138 ], [ -12.20226196377802, -2.741657987582399, -13.367656268171155, 5.688188338867107 ], [ -9.778431869782963, 0.9129525573845716, 3.1873684380152185, -4.346515826525721 ], [ -6.269744304840477, -3.8220935779562355, -3.265507157812303, 1.089250533386542 ], ... ],
    	[ [ -2.538064459696554, 9.299720408335695, -2.7224595782262546, -3.033856934057022 ], [ -3.289175094580425, 2.3238892983133956, -13.525178828902554, -16.808104444072097 ], [ 22.463392404660244, 10.98369013192283, 1.9072352696668462, 10.929561105489089 ], [ -0.5071990934158634, -3.8181325879574066, -0.2862783303286447, 7.716642340760133 ], [ -7.689443754566365, -6.856644677319988, -6.02637474448987, 10.91592140550838 ], [ -8.302058690996, 3.392405691360149, -16.68597019058036, -1.376401194589274 ], [ -12.140634208166782, 10.43324945676005, -3.0495296136505132, -11.009859609958863 ], [ -8.512533290859075, -9.952747425906551, 13.137706789465362, 1.5192134484604984 ], ... ],
    	[ [ 0.2819551540611611, 0.19708298427247786, 5.403356762248593, -1.526263617037948 ], [ 4.260092150097937, -3.6529456287046846, 8.555359723139748, 5.165710088917039 ], [ 6.639340749258417, 2.900762153180472, 7.795292008478545, 4.772922502886454 ], [ -13.73377088091679, -8.011226180319404, -5.217694977247856, 8.895616925618183 ], [ -7.23817139649162, -9.985035086660227, -0.3650496979609411, -4.5294175201617835 ], [ -5.758723618819182, -4.066138460037192, 3.1936932053660283, -3.0180856742085838 ], [ -13.042673437350203, -4.159030297025315, -4.5093585246761245, -3.5237732980151506 ], [ 1.3945201544771761, 9.122916507882682, 10.950080465784712, 7.8566334636662 ], ... ],
    	[ [ 1.630680801807512, -6.88850941455385, -7.768524263135425, -2.840125055360004 ], [ 9.67712873666462, -3.306235155281363, -10.814454805692424, 4.433185600829994 ], [ 10.030159187128916, 1.3803142022505606, 8.228575002416502, -6.9862939332550855 ], [ -1.3420745299206782, 9.093502924642996, -4.02420578795559, 3.6664330063807737 ], [ 8.533666538516748, -4.873105851705367, 0.9893644914234832, -2.7836119480425996 ], [ -0.3549132836774449, -0.997875554436281, 15.876113746023865, 11.998945573698098 ], [ -2.8675788725730156, 16.11851181522863, 2.9732775440165526, -2.6683230143220107 ], [ 8.368975638553655, 11.17804335463114, 2.6967022879474416, 12.617043915613374 ], ... ],
    	...
    ]

TrainingTester.java:432 executed in 0.13 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, -0.8744289429832307], [32.0, 2.0524792594374275]; valueStats=DoubleSummaryStatistics{count=52, sum=537.042329, min=0.133528, average=10.327737, max=112.844205}
Plotting 32 points for GD
Plotting 16 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, -0.8744289429832307], [30.827, 2.0524792594374275]; valueStats=DoubleSummaryStatistics{count=52, sum=537.042329, min=0.133528, average=10.327737, max=112.844205}
Plotting 32 points for GD
Plotting 16 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "NonConverged", "value": 9.25488866904522 }, "CjGD": { "type": "NonConverged", "value": 0.6992804872804479 }, "GD": { "type": "NonConverged", "value": 0.13352760426502727 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "NonConverged", "value": 9.25488866904522 }, "CjGD": { "type": "NonConverged", "value": 0.6992804872804479 }, "GD": { "type": "NonConverged", "value": 0.13352760426502727 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "95.365",
      "gc_time": "2.191"
    },
    "created_on": 1586748367761,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "SqGrid",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayerTest.SqGrid",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 9.25488866904522
        },
        "CjGD": {
          "type": "NonConverged",
          "value": 0.6992804872804479
        },
        "GD": {
          "type": "NonConverged",
          "value": 0.13352760426502727
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer/SqGrid/trainingTest/202004132607",
    "id": "33c20f02-a8aa-4589-9e78-9fa0f2d5eaae",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ConvolutionLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.conv.ConvolutionLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/conv/ConvolutionLayer.java",
      "javaDoc": ""
    }
  }