1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 5856045075146573824

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 1.56, -1.324, -1.996, 1.98, -0.344 ], [ -0.488, -0.224, -1.78, 1.684, 1.732 ], [ 0.612, -1.36, -0.716, -0.552, 0.936 ], [ 0.384, 1.348, -1.484, 0.116, -1.26 ], [ 0.564, -1.972, 0.924, 0.02, -0.084 ], [ 1.068, 1.164, 0.672, 1.86, -0.852 ], [ 0.128, 0.632, 1.996, -1.88, 0.72 ], [ -1.188, 1.028, -1.06, -0.992, 1.768 ], ... ],
    	[ [ -1.52, 0.228, 1.388, -1.672, 1.328 ], [ -0.888, -0.092, -0.224, -1.856, -1.304 ], [ 1.984, -1.384, -1.996, -1.36, 1.384 ], [ -1.16, -0.964, -0.524, -0.468, 1.98 ], [ -0.868, 0.576, -1.404, 0.836, -0.2 ], [ -0.244, -1.58, 0.012, 0.86, -1.14 ], [ 1.136, 1.924, -1.104, -0.34, -0.3 ], [ 0.42, -1.256, 1.268, -0.252, -1.856 ], ... ],
    	[ [ -0.092, 0.304, -0.324, 1.62, -0.896 ], [ -1.252, 0.168, 0.584, -0.204, -1.208 ], [ -0.108, -0.508, -1.808, -0.376, -0.904 ], [ 0.86, -0.772, 1.024, 0.808, 0.488 ], [ 1.868, -0.932, -1.568, -0.028, -1.508 ], [ 1.78, 1.152, -1.368, -1.22, -1.732 ], [ -0.196, 1.588, 1.812, -0.132, 0.18 ], [ -1.684, 0.792, -1.824, -0.248, -1.084 ], ... ],
    	[ [ -0.58, -1.0, 0.668, 1.016, -1.988 ], [ -0.448, 1.712, -0.456, -0.364, -1.188 ], [ -1.112, -0.62, -1.164, 0.42, 1.336 ], [ -1.828, 1.008, 1.368, 0.916, 1.576 ], [ 1.212, 1.612, -1.14, 0.584, 0.768 ], [ 1.996, 1.224, 0.168, 1.232, -1.336 ], [ 0.392, 1.232, -0.064, 1.156, -0.312 ], [ 0.728, -0.64, -0.528, -0.12, -1.984 ], ... ],
    	[ [ -0.748, -1.084, 0.64, 0.452, 0.476 ], [ 0.8, -0.524, 0.972, -1.104, -0.512 ], [ -0.864, -1.928, -1.624, -0.836, -0.004 ], [ 1.708, -1.432, 0.992, -0.056, 0.56 ], [ -0.288, -0.648, 1.588, 0.548, 0.392 ], [ 1.98, -0.932, 0.524, -1.432, -0.432 ], [ 0.34, -0.284, -1.392, -1.104, -1.828 ], [ 1.356, -1.392, 0.88, 0.704, -0.472 ], ... ],
    	[ [ 0.884, -1.164, 1.744, 0.532, 1.136 ], [ -1.28, -0.428, 0.056, -1.048, 1.436 ], [ -0.328, 1.228, 0.884, 1.292, 0.38 ], [ 1.792, 1.832, -1.336, 1.628, 0.128 ], [ 1.38, 1.68, -1.7, 1.016, -1.848 ], [ -1.656, -1.5, 0.984, 0.536, -1.892 ], [ -1.352, 1.22, -0.516, 0.86, -1.804 ], [ 0.14, 0.94, 1.62, 1.208, 0.5 ], ... ],
    	[ [ 0.052, -1.052, -1.064, 0.204, 0.56 ], [ -0.196, 1.7, -0.856, 0.26, -1.236 ], [ -1.972, 0.168, -1.488, -1.748, 1.66 ], [ 0.168, 0.668, -1.736, 0.784, -1.904 ], [ -1.26, -1.924, -0.012, 1.032, -1.332 ], [ -1.776, 0.588, -1.32, 0.384, -1.74 ], [ 0.78, -0.688, 0.048, -1.268, -0.556 ], [ -0.064, 1.264, 1.512, 1.448, -0.392 ], ... ],
    	[ [ -1.66, 0.68, -0.312, 0.896, 0.22 ], [ 0.656, 0.352, 0.008, 0.596, 0.38 ], [ -0.04, 0.336, -1.148, -0.732, -0.508 ], [ 0.908, 1.684, -1.052, -0.504, -1.976 ], [ -1.656, 1.756, 1.816, -1.436, 0.844 ], [ 1.188, 1.404, 0.488, 1.96, 0.968 ], [ -1.048, -1.66, 1.624, 0.2, 0.744 ], [ 0.156, -0.124, -0.56, -0.968, 1.948 ], ... ],
    	...
    ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 30.99 seconds (0.996 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 11154266370031
Reset training subject: 11154409163760
Constructing line search parameters: GD
th(0)=2.668021517728;dx=-5.336043035456059E-6
New Minimum: 2.668021517728 > 2.6680100215841605
WOLFE (weak): th(2.154434690031884)=2.6680100215841605; dx=-5.336031539299718E-6 evalInputDelta=1.149614383955111E-5
New Minimum: 2.6680100215841605 > 2.6679985254650886
WOLFE (weak): th(4.308869380063768)=2.6679985254650886; dx=-5.336020043143578E-6 evalInputDelta=2.2992262911358807E-5
New Minimum: 2.6679985254650886 > 2.6679525412364806
WOLFE (weak): th(12.926608140191302)=2.6679525412364806; dx=-5.335974058518603E-6 evalInputDelta=6.897649151937912E-5
New Minimum: 2.6679525412364806 > 2.6677456171117484
WOLFE (weak): th(51.70643256076521)=2.6677456171117484; dx=-5.335767127706579E-6 evalInputDelta=2.759006162516009E-4
New Minimum: 2.6677456171117484 > 2.666642157308797
WOLFE (weak): th(258.53216280382605)=2.666642157308797; dx=-5.334663496709241E-6 evalInputDelta=0.0013793604192029463
New Minimum: 2.666642157308797 > 2.6597507050398215
WOLFE (weak): th(1551.1929768229563)=2.6597507050398215; dx=-5.327765802975305E-6 evalInputDelta=0.00827081268817853
New Minimum: 2.6597507050398215 > 2.61039546019347
WOLFE (weak): th(10858.350837760694)=2.61039546019347; dx=-5.278102408091672E-6 evalInputDelta=0.057626057534529984
New Minimum: 2.61039546019347 > 2.2246289679228672
WOLFE (weak): th(86866.80670208555)=2.2246289679228672; dx=-4.872518016541041E-6 evalInputDelta=0.44339254980513276
New Minimum: 2.2246289679228672 > 0.12702634538981447
END: th(781801.26031877)=0.12702634538981447; dx=-1.164317865221298E-6 evalInputDelta=2.5409951723381856
Fitness changed from 2.668021517728 to 0.12702634538981447
Iteration 1 complete. Error: 0.12702634538981447 Total: 3.6449; Orientation: 0.3985; Line Search: 2.9346
th(0)=0.12702634538981447;dx=-2.540526907796274E-7
New Minimum: 0.12702634538981447 > 0.05948909259507654
WOLF (strong): th(1684339.7559414052)=0.05948909259507654; dx=1.7385835640438898E-7 evalInputDelta=0.06753725279473793
New Minimum: 0.05948909259507654 > 0.003164270395125619
END: th(842169.8779707026)=0.003164270395125619; dx=-4.00971671876198E-8 evalInputDelta=0.12386207499468885
Fitness changed from 0.12702634538981447 to 0.003164270395125619
Iteration 2 complete. Error: 0.003164270395125619 Total: 1.0559; Orientation: 0.1767; Line Search: 0.8017
th(0)=0.003164270395125619;dx=-6.328540790251274E-9
New Minimum: 0.003164270395125619 > 0.002098693985893224
WOLF (strong): th(1814400.0)=0.002098693985893224; dx=5.153963619580644E-9 evalInputDelta=0.0010655764092323953
New Minimum: 0.002098693985893224 > 2.7250190359558633E-5
END: th(907200.0)=2.7250190359558633E-5; dx=-5.872885853353086E-10 evalInputDelta=0.0031370202047660604
Fitness changed from 0.003164270395125619 to 2.7250190359558633E-5
Iteration 3 complete. Error: 2.7250190359558633E-5 Total: 0.8732; Orientation: 0.1706; Line Search: 0.6246
Low gradient: 7.382437315623971E-6
th(0)=2.7250190359558633E-5;dx=-5.4500380719116935E-11
New Minimum: 2.7250190359558633E-5 > 2.4827001650089917E-5
WOLF (strong): th(1954503.150796925)=2.4827001650089917E-5; dx=5.202078511602913E-11 evalInputDelta=2.4231887094687162E-6
New Minimum: 2.4827001650089917E-5 > 1.4101723404787075E-8
END: th(977251.5753984625)=1.4101723404787075E-8; dx=-1.2397978015439386E-12 evalInputDelta=2.7236088636153845E-5
Fitness changed from 2.7250190359558633E-5 to 1.4101723404787075E-8
Iteration 4 complete. Error: 1.4101723404787075E-8 Total: 0.8787; Orientation: 0.1684; Line Search: 0.6339
Low gradient: 1.679388186500493E-7
th(0)=1.4101723404787075E-8;dx=-2.8203446809574212E-14
Armijo: th(2105424.6949267564)=1.723179489995717E-8; dx=3.11767865853566E-14 evalInputDelta=-3.1300714951700943E-9
New Minimum: 1.4101723404787075E-8 > 3.9182929846928456E-11
WOLF (strong): th(1052712.3474633782)=3.9182929846928456E-11; dx=1.4866698878911358E-15 evalInputDelta=1.4062540474940147E-8
END: th(350904.1158211261)=5.941415197023724E-9; dx=-1.8306741243752674E-14 evalInputDelta=8.160308207763352E-9
Fitness changed from 1.4101723404787075E-8 to 3.9182929846928456E-11
Iteration 5 complete. Error: 3.9182929846928456E-11 Total: 1.0014; Orientation: 0.1699; Line Search: 0.7553
Low gradient: 8.852449361270411E-9
th(0)=3.9182929846928456E-11;dx=-7.836585969385711E-17
New Minimum: 3.9182929846928456E-11 > 2.3327949113675088E-12
END: th(756000.0)=2.3327949113675088E-12; dx=-1.9121269765304226E-17 evalInputDelta=3.6850134935560945E-11
Fitness changed from 3.9182929846928456E-11 to 2.3327949113675088E-12
Iteration 6 complete. Error: 2.3327949113675088E-12 Total: 0.6614; Orientation: 0.1681; Line Search: 0.4182
Low gradient: 2.1599976441503397E-9
th(0)=2.3327949113675088E-12;dx=-4.6655898227350185E-18
New Minimum: 2.3327949113675088E-12 > 9.222234957029703E-13
WOLF (strong): th(1628752.6256641042)=9.222234957029703E-13; dx=2.9335018513165593E-18 evalInputDelta=1.4105714156645385E-12
New Minimum: 9.222234957029703E-13 > 8.03791389385249E-14
END: th(814376.3128320521)=8.03791389385249E-14; dx=-8.660439857094827E-19 evalInputDelta=2.252415772428984E-12
Fitness changed from 2.3327949113675088E-12 to 8.03791389385249E-14
Iteration 7 complete. Error: 8.03791389385249E-14 Total: 0.8934; Orientation: 0.1707; Line Search: 0.6497
Low gradient: 4.009467269813407E-10
th(0)=8.03791389385249E-14;dx=-1.6075827787705042E-19
New Minimum: 8.03791389385249E-14 > 4.5759948635616605E-14
WOLF (strong): th(1754520.5791056307)=4.5759948635616605E-14; dx=1.2129542891967565E-19 evalInputDelta=3.461919030290829E-14
New Minimum: 4.5759948635616605E-14 > 1.2109146636042715E-15
END: th(877260.2895528154)=1.2109146636042715E-15; dx=-1.9731424478586823E-20 evalInputDelta=7.916822427492062E-14
Fitness changed from 8.03791389385249E-14 to 1.2109146636042715E-15
Iteration 8 complete. Error: 1.2109146636042715E-15 Total: 0.9172; Orientation: 0.1832; Line Search: 0.6602
Zero gradient: 4.921208517436081E-11
th(0)=1.2109146636042715E-15;dx=-2.421829327208528E-21
New Minimum: 1.2109146636042715E-15 > 9.591655051001711E-16
WOLF (strong): th(1890000.0000000005)=9.591655051001711E-16; dx=2.1554281012821523E-21 evalInputDelta=2.5174915850410037E-16
New Minimum: 9.591655051001711E-16 > 3.663016855764622E-18
END: th(945000.0000000002)=3.663016855764622E-18; dx=-1.3320061296668196E-22 evalInputDelta=1.2072516467485068E-15
Fitness changed from 1.2109146636042715E-15 to 3.663016855764622E-18
Iteration 9 complete. Error: 3.663016855764622E-18 Total: 1.2115; Orientation: 0.1843; Line Search: 0.9474
Zero gradient: 2.7066646839845607E-12
th(0)=3.663016855764622E-18;dx=-7.326033711529278E-24
Armijo: th(2035940.7820801307)=3.9310519020873195E-18; dx=7.589337093114674E-24 evalInputDelta=-2.6803504632269765E-19
New Minimum: 3.663016855764622E-18 > 1.1829161791346326E-21
WOLF (strong): th(1017970.3910400653)=1.1829161791346326E-21; dx=1.316516905036662E-25 evalInputDelta=3.661833939585487E-18
END: th(339323.46368002176)=1.5988829951795089E-18; dx=-4.840138577246704E-24 evalInputDelta=2.064133860585113E-18
Fitness changed from 3.663016855764622E-18 to 1.1829161791346326E-21
Iteration 10 complete. Error: 1.1829161791346326E-21 Total: 0.9994; Orientation: 0.1730; Line Search: 0.7414
Zero gradient: 4.8639822761491075E-14
th(0)=1.1829161791346326E-21;dx=-2.3658323582692806E-27
New Minimum: 1.1829161791346326E-21 > 8.556502341971563E-23
END: th(731050.2412940129)=8.556502341971563E-23; dx=-6.3629002999713525E-28 evalInputDelta=1.097351155714917E-21
Fitness changed from 1.1829161791346326E-21 to 8.556502341971563E-23
Iteration 11 complete. Error: 8.556502341971563E-23 Total: 0.6380; Orientation: 0.1671; Line Search: 0.3965
Zero gradient: 1.3081668350765939E-14
th(0)=8.556502341971563E-23;dx=-1.7113004683942983E-28
New Minimum: 8.556502341971563E-23 > 2.8289953707807467E-23
WOLF (strong): th(1575000.0000000005)=2.8289953707807467E-23; dx=9.839980795404738E-29 evalInputDelta=5.727506971190817E-23
New Minimum: 2.8289953707807467E-23 > 3.863791910633402E-24
END: th(787500.0000000002)=3.863791910633402E-24; dx=-3.63651176333153E-29 evalInputDelta=8.170123150908223E-23
Fitness changed from 8.556502341971563E-23 to 3

...skipping 4218 bytes...

d from 6.342152344906919E-39 to 7.837898215138737E-42
Iteration 19 complete. Error: 7.837898215138737E-42 Total: 0.8610; Orientation: 0.1729; Line Search: 0.6130
Zero gradient: 3.959267158234901E-24
th(0)=7.837898215138737E-42;dx=-1.567579643027747E-47
New Minimum: 7.837898215138737E-42 > 6.402091427667534E-42
WOLF (strong): th(1903776.6700364926)=6.402091427667534E-42; dx=1.416741909792611E-47 evalInputDelta=1.435806787471203E-42
New Minimum: 6.402091427667534E-42 > 1.8142636220040031E-44
END: th(951888.3350182463)=1.8142636220040031E-44; dx=-7.541886661756808E-49 evalInputDelta=7.819755578918697E-42
Fitness changed from 7.837898215138737E-42 to 1.8142636220040031E-44
Iteration 20 complete. Error: 1.8142636220040031E-44 Total: 0.8498; Orientation: 0.1658; Line Search: 0.6087
Zero gradient: 1.9048693509025773E-25
th(0)=1.8142636220040031E-44;dx=-3.6285272440080065E-50
Armijo: th(2050781.2500000014)=2.0032032768529054E-44; dx=3.812788393117792E-50 evalInputDelta=-1.8893965484890223E-45
New Minimum: 1.8142636220040031E-44 > 1.169626434778947E-47
WOLF (strong): th(1025390.6250000007)=1.169626434778947E-47; dx=9.213057455489314E-52 evalInputDelta=1.8130939955692243E-44
END: th(341796.87500000023)=7.859958850378896E-45; dx=-2.3883079711537065E-50 evalInputDelta=1.0282677369661136E-44
Fitness changed from 1.8142636220040031E-44 to 1.169626434778947E-47
Iteration 21 complete. Error: 1.169626434778947E-47 Total: 0.9877; Orientation: 0.1662; Line Search: 0.7501
Zero gradient: 4.836582336276199E-27
th(0)=1.169626434778947E-47;dx=-2.339252869557894E-53
New Minimum: 1.169626434778947E-47 > 8.128436831169041E-49
END: th(736379.0444444921)=8.128436831169041E-49; dx=-6.16676076758816E-54 evalInputDelta=1.0883420664672566E-47
Fitness changed from 1.169626434778947E-47 to 8.128436831169041E-49
Iteration 22 complete. Error: 8.128436831169041E-49 Total: 0.6691; Orientation: 0.1697; Line Search: 0.4270
Zero gradient: 1.2750244571120228E-27
th(0)=8.128436831169041E-49;dx=-1.625687366233808E-54
New Minimum: 8.128436831169041E-49 > 2.7958526239191484E-49
WOLF (strong): th(1586480.558363744)=2.7958526239191484E-49; dx=9.534340342736882E-55 evalInputDelta=5.332584207249893E-49
New Minimum: 2.7958526239191484E-49 > 3.4748727808782693E-50
END: th(793240.279181872)=3.4748727808782693E-50; dx=-3.3612666598006005E-55 evalInputDelta=7.780949553081215E-49
Fitness changed from 8.128436831169041E-49 to 3.4748727808782693E-50
Iteration 23 complete. Error: 3.4748727808782693E-50 Total: 0.9161; Orientation: 0.1832; Line Search: 0.6608
Zero gradient: 2.6362370078876703E-28
th(0)=3.4748727808782693E-50;dx=-6.949745561756537E-56
New Minimum: 3.4748727808782693E-50 > 1.7466755350629818E-50
WOLF (strong): th(1708984.3750000014)=1.7466755350629818E-50; dx=4.927261013510992E-56 evalInputDelta=1.7281972458152875E-50
New Minimum: 1.7466755350629818E-50 > 7.357182560756468E-52
END: th(854492.1875000007)=7.357182560756468E-52; dx=-1.0112422741227729E-56 evalInputDelta=3.4013009552707044E-50
Fitness changed from 3.4748727808782693E-50 to 7.357182560756468E-52
Iteration 24 complete. Error: 7.357182560756468E-52 Total: 0.9952; Orientation: 0.1828; Line Search: 0.7399
Zero gradient: 3.835930802492784E-29
th(0)=7.357182560756468E-52;dx=-1.4714365121512939E-57
New Minimum: 7.357182560756468E-52 > 5.202947157918009E-52
WOLF (strong): th(1840947.6111112304)=5.202947157918009E-52; dx=1.2374010197954714E-57 evalInputDelta=2.1542354028384594E-52
New Minimum: 5.202947157918009E-52 > 4.652988017994117E-54
END: th(920473.8055556152)=4.652988017994117E-54; dx=-1.1701774617791118E-58 evalInputDelta=7.310652680576527E-52
Fitness changed from 7.357182560756468E-52 to 4.652988017994117E-54
Iteration 25 complete. Error: 4.652988017994117E-54 Total: 0.8452; Orientation: 0.1639; Line Search: 0.6097
Zero gradient: 3.0505697887424628E-30
th(0)=4.652988017994117E-54;dx=-9.305976035988234E-60
New Minimum: 4.652988017994117E-54 > 4.497052348277504E-54
WOLF (strong): th(1983100.6979546803)=4.497052348277504E-54; dx=9.148711536129562E-60 evalInputDelta=1.5593566971661333E-55
New Minimum: 4.497052348277504E-54 > 3.3220753551473194E-58
END: th(991550.3489773402)=3.3220753551473194E-58; dx=-7.863224992933634E-62 evalInputDelta=4.652655810458602E-54
Fitness changed from 4.652988017994117E-54 to 3.3220753551473194E-58
Iteration 26 complete. Error: 3.3220753551473194E-58 Total: 0.8462; Orientation: 0.1664; Line Search: 0.6067
Zero gradient: 2.5776250135143086E-32
th(0)=3.3220753551473194E-58;dx=-6.64415071029464E-64
Armijo: th(2136230.468750002)=4.288864655678886E-58; dx=7.549286476003736E-64 evalInputDelta=-9.667893005315669E-59
New Minimum: 3.3220753551473194E-58 > 1.541338370561774E-60
WOLF (strong): th(1068115.234375001)=1.541338370561774E-60; dx=4.52567882854548E-65 evalInputDelta=3.3066619714417016E-58
END: th(356038.41145833366)=1.3776198931762002E-58; dx=-4.2785778459115776E-64 evalInputDelta=1.9444554619711193E-58
Fitness changed from 3.3220753551473194E-58 to 1.541338370561774E-60
Iteration 27 complete. Error: 1.541338370561774E-60 Total: 0.9797; Orientation: 0.1663; Line Search: 0.7407
Zero gradient: 1.7557553192639195E-33
th(0)=1.541338370561774E-60;dx=-3.082676741123548E-66
New Minimum: 1.541338370561774E-60 > 8.36335480883405E-62
END: th(767061.5046296794)=8.36335480883405E-62; dx=-7.180740817904027E-67 evalInputDelta=1.4577048224734336E-60
Fitness changed from 1.541338370561774E-60 to 8.36335480883405E-62
Iteration 28 complete. Error: 8.36335480883405E-62 Total: 0.6422; Orientation: 0.1680; Line Search: 0.4000
Zero gradient: 4.089830023077743E-34
th(0)=8.36335480883405E-62;dx=-1.6726709617668097E-67
New Minimum: 8.36335480883405E-62 > 3.5616665025578855E-62
WOLF (strong): th(1652583.9149622337)=3.5616665025578855E-62; dx=1.0915581646734292E-67 evalInputDelta=4.801688306276164E-62
New Minimum: 3.5616665025578855E-62 > 2.5235991616441014E-63
END: th(826291.9574811169)=2.5235991616441014E-63; dx=-2.9055639854669016E-68 evalInputDelta=8.11099489266964E-62
Fitness changed from 8.36335480883405E-62 to 2.5235991616441014E-63
Iteration 29 complete. Error: 2.5235991616441014E-63 Total: 0.8535; Orientation: 0.1656; Line Search: 0.6156
Zero gradient: 7.104363675437936E-35
th(0)=2.5235991616441014E-63;dx=-5.047198323288202E-69
New Minimum: 2.5235991616441014E-63 > 1.536113916997336E-63
WOLF (strong): th(1780192.0572916686)=1.536113916997336E-63; dx=3.9377840434052823E-69 evalInputDelta=9.874852446467655E-64
New Minimum: 1.536113916997336E-63 > 3.04822588090387E-65
END: th(890096.0286458343)=3.04822588090387E-65; dx=-5.5470713994145984E-70 evalInputDelta=2.4931169028350626E-63
Fitness changed from 2.5235991616441014E-63 to 3.04822588090387E-65
Iteration 30 complete. Error: 3.04822588090387E-65 Total: 0.8517; Orientation: 0.1734; Line Search: 0.6068
Zero gradient: 7.807977818749065E-36
th(0)=3.04822588090387E-65;dx=-6.0964517618077405E-71
New Minimum: 3.04822588090387E-65 > 2.566875734542963E-65
WOLF (strong): th(1917653.7615741987)=2.566875734542963E-65; dx=5.594431891478524E-71 evalInputDelta=4.813501463609068E-66
New Minimum: 2.566875734542963E-65 > 5.167430992077446E-68
END: th(958826.8807870994)=5.167430992077446E-68; dx=-2.510099351646085E-72 evalInputDelta=3.0430584499117925E-65
Fitness changed from 3.04822588090387E-65 to 5.167430992077446E-68
Iteration 31 complete. Error: 5.167430992077446E-68 Total: 0.8512; Orientation: 0.1682; Line Search: 0.6093
Zero gradient: 3.2147880154303945E-37
th(0)=5.167430992077446E-68;dx=-1.0334861984154893E-73
Armijo: th(2065729.8937027925)=5.869065838386879E-68; dx=1.1014171363806423E-73 evalInputDelta=-7.016348463094335E-69
New Minimum: 5.167430992077446E-68 > 5.581366664475621E-71
WOLF (strong): th(1032864.9468513962)=5.581366664475621E-71; dx=3.3965468982576547E-75 evalInputDelta=5.16184962541297E-68
END: th(344288.31561713206)=2.2217773283691926E-68; dx=-6.776689759494674E-74 evalInputDelta=2.9456536637082533E-68
Fitness changed from 5.167430992077446E-68 to 5.581366664475621E-71
Iteration 32 complete. Error: 5.581366664475621E-71 Total: 1.1084; Orientation: 0.1676; Line Search: 0.8678
Final threshold in iteration 32: 5.581366664475621E-71 (> 0.0) after 30.978s (< 30.000s)

Returns

    5.581366664475621E-71

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 12.44 seconds (0.101 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 11185286073885
Reset training subject: 11185358480094
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=2.668021517728}, derivative=-5.336043035456059E-6}
New Minimum: 2.668021517728 > 2.6680215177279996
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=2.6680215177279996}, derivative=-5.336043035456058E-6}, evalInputDelta = -4.440892098500626E-16
New Minimum: 2.6680215177279996 > 2.668021517727996
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=2.668021517727996}, derivative=-5.336043035456056E-6}, evalInputDelta = -3.9968028886505635E-15
New Minimum: 2.668021517727996 > 2.668021517727974
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=2.668021517727974}, derivative=-5.336043035456025E-6}, evalInputDelta = -2.6201263381153694E-14
New Minimum: 2.668021517727974 > 2.668021517727817
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=2.668021517727817}, derivative=-5.336043035455841E-6}, evalInputDelta = -1.829647544582258E-13
New Minimum: 2.668021517727817 > 2.668021517726719
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=2.668021517726719}, derivative=-5.336043035454775E-6}, evalInputDelta = -1.2811973704174306E-12
New Minimum: 2.668021517726719 > 2.668021517719031
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=2.668021517719031}, derivative=-5.336043035447002E-6}, evalInputDelta = -8.968825682131865E-12
New Minimum: 2.668021517719031 > 2.6680215176652218
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=2.6680215176652218}, derivative=-5.3360430353932585E-6}, evalInputDelta = -6.277822706124425E-11
New Minimum: 2.6680215176652218 > 2.6680215172885533
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=2.6680215172885533}, derivative=-5.336043035016536E-6}, evalInputDelta = -4.3944670125029006E-10
New Minimum: 2.6680215172885533 > 2.668021514651877
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=2.668021514651877}, derivative=-5.336043032379854E-6}, evalInputDelta = -3.0761229119491418E-9
New Minimum: 2.668021514651877 > 2.6680214961951414
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=2.6680214961951414}, derivative=-5.336043013923168E-6}, evalInputDelta = -2.1532858607287153E-8
New Minimum: 2.6680214961951414 > 2.6680213669979933
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=2.6680213669979933}, derivative=-5.3360428847260186E-6}, evalInputDelta = -1.507300066982964E-7
New Minimum: 2.6680213669979933 > 2.668020462618045
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=2.668020462618045}, derivative=-5.336041980345912E-6}, evalInputDelta = -1.0551099549616083E-6
New Minimum: 2.668020462618045 > 2.668014131962694
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=2.668014131962694}, derivative=-5.33603564968557E-6}, evalInputDelta = -7.385765306011649E-6
New Minimum: 2.668014131962694 > 2.667969817585541
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=2.667969817585541}, derivative=-5.335991335063149E-6}, evalInputDelta = -5.170014245914345E-5
New Minimum: 2.667969817585541 > 2.6676596272501034
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=2.6676596272501034}, derivative=-5.335681132705545E-6}, evalInputDelta = -3.61890477896587E-4
New Minimum: 2.6676596272501034 > 2.665488799829396
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=2.665488799829396}, derivative=-5.333509716202944E-6}, evalInputDelta = -0.0025327178986040977
New Minimum: 2.665488799829396 > 2.6503177493246253
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=2.6503177493246253}, derivative=-5.318309800684596E-6}, evalInputDelta = -0.01770376840337473
New Minimum: 2.6503177493246253 > 2.545332726360233
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=2.545332726360233}, derivative=-5.211910392056349E-6}, evalInputDelta = -0.122688791367767
New Minimum: 2.545332726360233 > 1.8698417634905762
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=1.8698417634905762}, derivative=-4.467114531658854E-6}, evalInputDelta = -0.7981797542374238
New Minimum: 1.8698417634905762 > 0.05221071957621772
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.05221071957621772}, derivative=7.464564911243343E-7}, evalInputDelta = -2.615810798151782
0.05221071957621772 <= 2.668021517728
New Minimum: 0.05221071957621772 > 7.953183269855387E-31
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=7.953183269855387E-31}, derivative=2.8758649011223443E-21}, evalInputDelta = -2.668021517728
Right bracket at 1000000.0000000006
Converged to right
Fitness changed from 2.668021517728 to 7.953183269855387E-31
Iteration 1 complete. Error: 7.953183269855387E-31 Total: 5.2620; Orientation: 0.1651; Line Search: 4.8791
Zero gradient: 1.2612044457466351E-18
F(0.0) = LineSearchPoint{point=PointSample{avg=7.953183269855387E-31}, derivative=-1.5906366539710603E-36}
New Minimum: 7.953183269855387E-31 > 1.0714455854319734E-64
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=1.0714455854319734E-64}, derivative=4.214246796875461E-55}, evalInputDelta = -7.953183269855387E-31
1.0714455854319734E-64 <= 7.953183269855387E-31
Converged to right
Fitness changed from 7.953183269855387E-31 to 1.0714455854319734E-64
Iteration 2 complete. Error: 1.0714455854319734E-64 Total: 0.6494; Orientation: 0.1694; Line Search: 0.4062
Zero gradient: 1.4638617321536713E-35
F(0.0) = LineSearchPoint{point=PointSample{avg=1.0714455854319734E-64}, derivative=-2.142891170863947E-70}
New Minimum: 1.0714455854319734E-64 > 2.500966302773853E-95
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=2.500966302773853E-95}, derivative=1.0265063800323044E-85}, evalInputDelta = -1.0714455854319734E-64
2.500966302773853E-95 <= 1.0714455854319734E-64
Converged to right
Fitness changed from 1.0714455854319734E-64 to 2.500966302773853E-95
Iteration 3 complete. Error: 2.500966302773853E-95 Total: 0.6994; Orientation: 0.1902; Line Search: 0.4230
Zero gradient: 7.072434238328205E-51
F(0.0) = LineSearchPoint{point=PointSample{avg=2.500966302773853E-95}, derivative=-5.001932605547706E-101}
New Minimum: 2.500966302773853E-95 > 4.932286353833572E-126
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=4.932286353833572E-126}, derivative=2.2213042985209458E-116}, evalInputDelta = -2.500966302773853E-95
4.932286353833572E-126 <= 2.500966302773853E-95
Converged to right
Fitness changed from 2.500966302773853E-95 to 4.932286353833572E-126
Iteration 4 complete. Error: 4.932286353833572E-126 Total: 0.6778; Orientation: 0.1690; Line Search: 0.4344
Zero gradient: 3.140791732615702E-66
F(0.0) = LineSearchPoint{point=PointSample{avg=4.932286353833572E-126}, derivative=-9.864572707667144E-132}
New Minimum: 4.932286353833572E-126 > 9.727219694735989E-157
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=9.727219694735989E-157}, derivative=4.380750299256395E-147}, evalInputDelta = -4.932286353833572E-126
9.727219694735989E-157 <= 4.932286353833572E-126
Converged to right
Fitness changed from 4.932286353833572E-126 to 9.727219694735989E-157
Iteration 5 complete. Error: 9.727219694735989E-157 Total: 0.6896; Orientation: 0.1847; Line Search: 0.4322
Zero gradient: 1.3947917188409162E-81
F(0.0) = LineSearchPoint{point=PointSample{avg=9.727219694735989E-157}, derivative=-1.9454439389471977E-162}
New Minimum: 9.727219694735989E-157 > 1.9183558334182716E-187
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=1.9183558334182716E-187}, derivative=8.639506616546545E-178}, evalInputDelta = -9.727219694735989E-157
1.9183558334182716E-187 <= 9.727219694735989E-157
Converged to right
Fitness changed from 9.727219694735989E-157 to 1.9183558334182716E-187
Iteration 6 complete. Error: 1.9183558334182716E-187 Total: 0.7610; Orientation: 0.1833; Line Search: 0.5056
Zero gradient: 6.194119523254732E-97
F(0.0) = LineSearchPoint{point=PointSample{avg=1.9183558334182716E-187}, derivative=-3.836711666836543E-193}
New Minimum: 1.9183558334182716E-187 > 3.7832897982158656E-218
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=3.7832897982158656E-218}, derivative=1.703842252547957E-208}, evalInputDelta = -1.9183558334182716E-187
3.7832897982158656E-218 <= 1.9183558334182716E-187
Converged to right
Fitness changed from 1.9183558334182716E-187 to 3.7832897982158656E-218
Iteration 7 complete. Error: 3.7832897982158656E-218 Total: 0.6331; Orientation: 0.1628; Line Search: 0.3977
Zero gradient: 2.7507416447990404E-112
F(0.0) = LineSearchPoint{point=PointSample{avg=3.7832897982158656E-218}, derivative=-7.566579596431731E-224}
New Minimum: 3.7832897982158656E-218 > 7.461223537334967E-249
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=7.461223537334967E-249}, derivative=3.360236354246973E-239}, evalInputDelta = -3.7832897982158656E-218
7.461223537334967E-249 <= 3.7832897982158656E-218
Converged to right
Fitness changed from 3.7832897982158656E-218 to 7.461223537334967E-249
Iteration 8 complete. Error: 7.461223537334967E-249 Total: 0.6331; Orientation: 0.1649; Line Search: 0.3940
Zero gradient: 1.2215746835404674E-127
F(0.0) = LineSearchPoint{point=PointSample{avg=7.461223537334967E-249}, derivative=-1.4922447074669934E-254}
New Minimum: 7.461223537334967E-249 > 1.4714668884295955E-279
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=1.4714668884295955E-279}, derivative=6.626897730419549E-270}, evalInputDelta = -7.461223537334967E-249
1.4714668884295955E-279 <= 7.461223537334967E-249
Converged to right
Fitness changed from 7.461223537334967E-249 to 1.4714668884295955E-279
Iteration 9 complete. Error: 1.4714668884295955E-279 Total: 0.7223; Orientation: 0.1683; Line Search: 0.4827
Zero gradient: 5.424881359863265E-143
F(0.0) = LineSearchPoint{point=PointSample{avg=1.4714668884295955E-279}, derivative=-2.942933776859191E-285}
New Minimum: 1.4714668884295955E-279 > 2.9019567540233E-310
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=2.9019567540233E-310}, derivative=1.3069251356064585E-300}, evalInputDelta = -1.4714668884295955E-279
2.9019567540233E-310 <= 1.4714668884295955E-279
Converged to right
Fitness changed from 1.4714668884295955E-279 to 2.9019567540233E-310
Iteration 10 complete. Error: 2.9019567540233E-310 Total: 0.6371; Orientation: 0.1664; Line Search: 0.3998
Zero gradient: 2.40913316129019E-158
F(0.0) = LineSearchPoint{point=PointSample{avg=2.9019567540233E-310}, derivative=-5.8039226E-316}
New Minimum: 2.9019567540233E-310 > 0.0
F(1000000.0000000006) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -2.9019567540233E-310
0.0 <= 2.9019567540233E-310
Converged to right
Fitness changed from 2.9019567540233E-310 to 0.0
Iteration 11 complete. Error: 0.0 Total: 0.6399; Orientation: 0.1686; Line Search: 0.3992
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.4311; Orientation: 0.1708; Line Search: 0.1872
Iteration 12 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 12
Final threshold in iteration 12: 0.0 (> 0.0) after 12.436s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 65.06 seconds (1.355 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging
Reset training subject: 11197763811662
Reset training subject: 11197836666457
Adding measurement 7fdfc12e to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 2.668021517728 < 2.668021517728. Total: 1
th(0)=2.668021517728;dx=-5.336043035456059E-6
Adding measurement c6d3cc1 to history. Total: 1
New Minimum: 2.668021517728 > 2.6680100215841605
WOLFE (weak): th(2.154434690031884)=2.6680100215841605; dx=-5.336031539299718E-6 evalInputDelta=1.149614383955111E-5
Adding measurement 7d411ee5 to history. Total: 2
New Minimum: 2.6680100215841605 > 2.6679985254650886
WOLFE (weak): th(4.308869380063768)=2.6679985254650886; dx=-5.336020043143578E-6 evalInputDelta=2.2992262911358807E-5
Adding measurement 2ea60097 to history. Total: 3
New Minimum: 2.6679985254650886 > 2.6679525412364806
WOLFE (weak): th(12.926608140191302)=2.6679525412364806; dx=-5.335974058518603E-6 evalInputDelta=6.897649151937912E-5
Adding measurement 41025210 to history. Total: 4
New Minimum: 2.6679525412364806 > 2.6677456171117484
WOLFE (weak): th(51.70643256076521)=2.6677456171117484; dx=-5.335767127706579E-6 evalInputDelta=2.759006162516009E-4
Adding measurement 258a516a to history. Total: 5
New Minimum: 2.6677456171117484 > 2.666642157308797
WOLFE (weak): th(258.53216280382605)=2.666642157308797; dx=-5.334663496709241E-6 evalInputDelta=0.0013793604192029463
Adding measurement 5352e77a to history. Total: 6
New Minimum: 2.666642157308797 > 2.6597507050398215
WOLFE (weak): th(1551.1929768229563)=2.6597507050398215; dx=-5.327765802975305E-6 evalInputDelta=0.00827081268817853
Adding measurement 5d326f36 to history. Total: 7
New Minimum: 2.6597507050398215 > 2.61039546019347
WOLFE (weak): th(10858.350837760694)=2.61039546019347; dx=-5.278102408091672E-6 evalInputDelta=0.057626057534529984
Adding measurement 5f818b84 to history. Total: 8
New Minimum: 2.61039546019347 > 2.2246289679228672
WOLFE (weak): th(86866.80670208555)=2.2246289679228672; dx=-4.872518016541041E-6 evalInputDelta=0.44339254980513276
Adding measurement 52751d2d to history. Total: 9
New Minimum: 2.2246289679228672 > 0.12702634538981447
END: th(781801.26031877)=0.12702634538981447; dx=-1.164317865221298E-6 evalInputDelta=2.5409951723381856
Fitness changed from 2.668021517728 to 0.12702634538981447
Iteration 1 complete. Error: 0.12702634538981447 Total: 4.0794; Orientation: 0.2356; Line Search: 3.6288
Non-optimal measurement 0.12702634538981447 < 0.12702634538981447. Total: 10
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797, 2.6677456171117484, 2.6679525412364806, 2.6679985254650886, 2.6680100215841605, 2.668021517728
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797, 2.6677456171117484, 2.6679525412364806, 2.6679985254650886, 2.6680100215841605
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797, 2.6677456171117484, 2.6679525412364806, 2.6679985254650886
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797, 2.6677456171117484, 2.6679525412364806
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797, 2.6677456171117484
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215, 2.666642157308797
Rejected: LBFGS Orientation magnitude: 5.040e+02, gradient 5.040e-04, dot -1.000; [9abefa27-f729-4e99-80e4-32ce5318e226 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.12702634538981447, 2.2246289679228672, 2.61039546019347, 2.6597507050398215
LBFGS Accumulation History: 3 points
Removed measurement 52751d2d to history. Total: 9
Removed measurement 5f818b84 to history. Total: 8
Removed measurement 5d326f36 to history. Total: 7
Removed measurement 5352e77a to history. Total: 6
Removed measurement 258a516a to history. Total: 5
Removed measurement 41025210 to history. Total: 4
Removed measurement 2ea60097 to history. Total: 3
Adding measurement 7534ea24 to history. Total: 3
th(0)=0.12702634538981447;dx=-2.540526907796274E-7
Adding measurement 70bf7f82 to history. Total: 4
New Minimum: 0.12702634538981447 > 0.05948909259507654
WOLF (strong): th(1684339.7559414052)=0.05948909259507654; dx=1.7385835640438898E-7 evalInputDelta=0.06753725279473793
Adding measurement 2ea4f96a to history. Total: 5
New Minimum: 0.05948909259507654 > 0.003164270395125619
END: th(842169.8779707026)=0.003164270395125619; dx=-4.00971671876198E-8 evalInputDelta=0.12386207499468885
Fitness changed from 0.12702634538981447 to 0.003164270395125619
Iteration 2 complete. Error: 0.003164270395125619 Total: 60.9784; Orientation: 59.8790; Line Search: 1.0270
Final threshold in iteration 2: 0.003164270395125619 (> 0.0) after 65.058s (< 30.000s)

Returns

    0.003164270395125619

This training apply resulted in the following configuration:

TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):

    RefList<double[]> state = network.state();
    assert state != null;
    String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
        .orElse("");
    state.freeRef();
    return description;

Returns

    

And regressed input:

TrainingTester.java:622 executed in 0.01 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
      return RefArrays.stream(x);
    }).limit(1).map(x -> {
      String temp_18_0015 = x.prettyPrint();
      x.freeRef();
      return temp_18_0015;
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [
    	[ [ 1.356, 0.296, -1.948612852814314, 1.9018938591445236, 0.68 ], [ 0.408, -0.64, -1.7835815867058948, 1.6757347999094734, 1.64 ], [ 1.924, -0.056, -0.6469855792441029, -0.5697701801946322, -0.232 ], [ 1.292, -0.192, -1.3932205523390495, 0.11269391996378937, -0.276 ], [ -1.952, -0.052, 0.945351766900527, 0.021377533348421098, -1.932 ], [ 0.448, -1.428, 0.6675918932850525, 1.7641236789498915, 1.68 ], [ 0.416, -1.476, 1.910592932397892, -1.8447351462804198, -1.456 ], [ 0.596, -1.504, -1.0291432529953675, -0.9446128528143142, 0.152 ], ... ],
    	[ [ -1.908, 0.468, 1.2972205523390494, -1.5560116920629434, 1.984 ], [ -1.016, 1.164, -0.19961765973294657, -1.770592932397892, 0.924 ], [ -0.264, -1.352, -1.9610106529501041, -1.2634349122756812, 0.416 ], [ -1.472, -0.824, -0.46793439271926135, -0.47970903346157934, -1.58 ], [ 1.656, 1.244, -1.2915932787688384, 0.7454960590087338, -1.884 ], [ 1.032, -1.428, 0.033765026905053355, 0.8588979733212632, -1.384 ], [ -1.684, 1.704, -1.0402202059681032, -0.3189237397691572, 0.72 ], [ -1.628, -1.204, 1.168817598913681, -0.29291274044810667, 1.556 ], ... ],
    	[ [ -0.68, 0.684, -0.3781370605929492, 1.5636588860495773, 0.272 ], [ 0.512, 0.172, 0.6115506669684219, -0.17355651299989372, 1.268 ], [ 0.46, -0.056, -1.6939402387507332, -0.3686990732533682, 1.084 ], [ 1.564, 1.004, 1.0532037069865272, 0.8307293002489482, 1.54 ], [ -0.304, 0.548, -1.551469599818947, 0.039636887407475896, 0.068 ], [ 1.676, -1.984, -1.2748787456467339, -1.1575977393165242, 0.42 ], [ -0.716, 0.152, 1.686368958623996, -0.08406183947494578, 1.296 ], [ 0.028, -1.336, -1.8069185864795785, -0.21879629301347273, 0.032 ], ... ],
    	[ [ 0.288, 1.012, 0.6547756798551575, 1.0288110601403162, 1.892 ], [ 0.94, 1.704, -0.37376125909926045, -0.38769357359284284, 1.556 ], [ 1.408, -0.728, -1.1098629394070507, 0.3687557594387351, -0.876 ], [ -0.408, -0.892, 1.2579350854611544, 0.8176441189227337, -0.932 ], [ 0.564, -0.944, -1.1220920664705256, 0.4970776457146287, -1.164 ], [ 1.24, -0.656, 0.2202085139051596, 1.1715262860043139, 1.688 ], [ 1.432, -0.648, -0.12350944065179148, 1.1520051532895788, 0.872 ], [ -0.988, -0.268, -0.529239780013579, -0.17854516730789666, -0.336 ], ... ],
    	[ [ -1.396, 1.344, 0.6289797332126312, 0.40957197286863023, -1.716 ], [ -1.64, 1.384, 0.9350821062623146, -1.0460058460314718, -1.532 ], [ 0.436, 1.964, -1.580056686185367, -0.7937097262034722, -1.944 ], [ -1.604, -1.352, 0.9987499134072634, -0.002138446076735079, 1.428 ], [ -0.88, 0.536, 1.5068632857779973, 0.586846440425475, -1.04 ], [ -0.58, -1.16, 0.46724562604505077, -1.3614702925608397, -0.904 ], [ 0.048, 0.512, -1.2940573789272598, -1.061158712864104, -1.976 ], [ -1.848, -0.304, 0.8958416335068426, 0.6916021998642101, -0.676 ], ... ],
    	[ [ 1.92, -1.02, 1.6741590592350504, 0.4746946127056823, -1.172 ], [ 0.064, 1.772, 0.0077863328052615866, -1.0624641001584216, 1.044 ], [ 0.056, -1.624, 0.917887320371159, 1.2552198595971567, -0.84 ], [ -1.956, 1.084, -1.2825517060812615, 1.5872250128867353, -0.804 ], [ -1.78, 0.6, -1.6053634589634704, 1.039831326927685, 0.14 ], [ 1.276, 0.424, 0.904929585800629, 0.5362755066696843, 0.984 ], [ -1.228, 0.46, -0.47164342618084065, 0.8922342803530536, -0.004 ], [ 1.46, 0.392, 1.5453376925155766, 1.157857786117472, -0.2 ], ... ],
    	[ [ -1.664, 1.748, -0.9744603323526286, 0.2279690802625271, 0.628 ], [ -1.072, -1.356, -0.8307911397238938, 0.24553589984157848, -0.68 ], [ 0.748, -1.344, -1.4656839597555782, -1.6660367657689448, -0.664 ], [ 0.456, 0.072, -1.699357612931999, 0.8076935735928429, 0.624 ], [ -0.508, -1.264, -0.0048368265882102884, 1.0273163866153683, 1.672 ], [ 0.276, -1.62, -1.317382686638, 0.3239395460088402, 0.256 ], [ -0.708, 0.784, 0.11095327402284416, -1.1747409923118917, -0.172 ], [ -1.86, 0.412, 1.4186032389770495, 1.3843579593029451, 1.088 ], ... ],
    	[ [ 0.484, 0.664, -0.23348059913999736, 0.8031542523164181, 1.864 ], [ 0.452, 1.532, -0.012525246891474356, 0.5669340463483148, -0.812 ], [ 0.376, -0.468, -1.150755066696842, -0.670562012660419, 0.236 ], [ -0.628, -1.308, -0.9470319588503124, -0.4818217130904203, -0.876 ], [ -0.06, 1.472, 1.7683373461446301, -1.4269082799004207, -0.24 ], [ 0.456, 0.784, 0.4261487526558927, 1.909857786117472, 0.144 ], [ 1.968, -1.088, 1.562424259325577, 0.1564699461898933, 1.912 ], [ 0.128, -1.312, -0.6061473671721068, -0.8780470723481023, 1.792 ], ... ],
    	...
    ]

To produce the following output:

TrainingTester.java:633 executed in 0.02 seconds (0.000 gc):

    Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
    @Nullable
    Result eval = layer.eval(array);
    assert eval != null;
    TensorList tensorList = Result.getData(eval);
    String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
      String temp_18_0017 = x.prettyPrint();
      x.freeRef();
      return temp_18_0017;
    }).reduce((a, b) -> a + "\n" + b).orElse("");
    tensorList.freeRef();
    return temp_18_0016;

Returns

    [
    	[ [ -1.948612852814314, 1.9018938591445236 ], [ -1.7835815867058948, 1.6757347999094734 ], [ -0.6469855792441029, -0.5697701801946322 ], [ -1.3932205523390495, 0.11269391996378937 ], [ 0.945351766900527, 0.021377533348421098 ], [ 0.6675918932850525, 1.7641236789498915 ], [ 1.910592932397892, -1.8447351462804198 ], [ -1.0291432529953675, -0.9446128528143142 ], ... ],
    	[ [ 1.2972205523390494, -1.5560116920629434 ], [ -0.19961765973294657, -1.770592932397892 ], [ -1.9610106529501041, -1.2634349122756812 ], [ -0.46793439271926135, -0.47970903346157934 ], [ -1.2915932787688384, 0.7454960590087338 ], [ 0.033765026905053355, 0.8588979733212632 ], [ -1.0402202059681032, -0.3189237397691572 ], [ 1.168817598913681, -0.29291274044810667 ], ... ],
    	[ [ -0.3781370605929492, 1.5636588860495773 ], [ 0.6115506669684219, -0.17355651299989372 ], [ -1.6939402387507332, -0.3686990732533682 ], [ 1.0532037069865272, 0.8307293002489482 ], [ -1.551469599818947, 0.039636887407475896 ], [ -1.2748787456467339, -1.1575977393165242 ], [ 1.686368958623996, -0.08406183947494578 ], [ -1.8069185864795785, -0.21879629301347273 ], ... ],
    	[ [ 0.6547756798551575, 1.0288110601403162 ], [ -0.37376125909926045, -0.38769357359284284 ], [ -1.1098629394070507, 0.3687557594387351 ], [ 1.2579350854611544, 0.8176441189227337 ], [ -1.1220920664705256, 0.4970776457146287 ], [ 0.2202085139051596, 1.1715262860043139 ], [ -0.12350944065179148, 1.1520051532895788 ], [ -0.529239780013579, -0.17854516730789666 ], ... ],
    	[ [ 0.6289797332126312, 0.40957197286863023 ], [ 0.9350821062623146, -1.0460058460314718 ], [ -1.580056686185367, -0.7937097262034722 ], [ 0.9987499134072634, -0.002138446076735079 ], [ 1.5068632857779973, 0.586846440425475 ], [ 0.46724562604505077, -1.3614702925608397 ], [ -1.2940573789272598, -1.061158712864104 ], [ 0.8958416335068426, 0.6916021998642101 ], ... ],
    	[ [ 1.6741590592350504, 0.4746946127056823 ], [ 0.0077863328052615866, -1.0624641001584216 ], [ 0.917887320371159, 1.2552198595971567 ], [ -1.2825517060812615, 1.5872250128867353 ], [ -1.6053634589634704, 1.039831326927685 ], [ 0.904929585800629, 0.5362755066696843 ], [ -0.47164342618084065, 0.8922342803530536 ], [ 1.5453376925155766, 1.157857786117472 ], ... ],
    	[ [ -0.9744603323526286, 0.2279690802625271 ], [ -0.8307911397238938, 0.24553589984157848 ], [ -1.4656839597555782, -1.6660367657689448 ], [ -1.699357612931999, 0.8076935735928429 ], [ -0.0048368265882102884, 1.0273163866153683 ], [ -1.317382686638, 0.3239395460088402 ], [ 0.11095327402284416, -1.1747409923118917 ], [ 1.4186032389770495, 1.3843579593029451 ], ... ],
    	[ [ -0.23348059913999736, 0.8031542523164181 ], [ -0.012525246891474356, 0.5669340463483148 ], [ -1.150755066696842, -0.670562012660419 ], [ -0.9470319588503124, -0.4818217130904203 ], [ 1.7683373461446301, -1.4269082799004207 ], [ 0.4261487526558927, 1.909857786117472 ], [ 1.562424259325577, 0.1564699461898933 ], [ -0.6061473671721068, -0.8780470723481023 ], ... ],
    	...
    ]

TrainingTester.java:432 executed in 0.16 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging
Plotting range=[1.0, -309.53730906385863], [32.0, -0.8961061965965436]; valueStats=DoubleSummaryStatistics{count=44, sum=0.260408, min=0.000000, average=0.005918, max=0.127026}
Plotting 32 points for GD
Plotting 11 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.02 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging
Plotting range=[0.0, -309.53730906385863], [60.978, -0.8961061965965436]; valueStats=DoubleSummaryStatistics{count=44, sum=0.260408, min=0.000000, average=0.005918, max=0.127026}
Plotting 32 points for GD
Plotting 11 points for CjGD
Plotting 2 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "NonConverged", "value": 0.003164270395125619 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 5.581366664475621E-71 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "NonConverged", "value": 0.003164270395125619 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 5.581366664475621E-71 } }, "model":null, "complete":null}OK
  {
    "result": "OK",
    "performance": {
      "execution_time": "110.682",
      "gc_time": "2.695"
    },
    "created_on": 1586745784806,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Double",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgBandSelectLayerTest.Double",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "NonConverged",
          "value": 0.003164270395125619
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 5.581366664475621E-71
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayer/Double/trainingTest/202004134304",
    "id": "d30b5252-eff6-46f2-89fd-2bad1e2ff4bb",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "ImgBandSelectLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgBandSelectLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgBandSelectLayer.java",
      "javaDoc": ""
    }
  }