1. Test Modules
  2. Training Characteristics
    1. Input Learning
      1. Gradient Descent
      2. Conjugate Gradient Descent
      3. Limited-Memory BFGS
    2. Results
  3. Results

Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase

Test Modules

Using Seed 7415343329776930816

Training Characteristics

Input Learning

In this apply, we use a network to learn this target input, given it's pre-evaluated output:

TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):

    return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
      try {
        return x.prettyPrint();
      } finally {
        x.freeRef();
      }
    }).reduce((a, b) -> a + "\n" + b).orElse("");

Returns

    [ 0.08 ]
    [ 0.08 ]
    [ 0.08 ]
    [ 0.08 ]
    [ 0.08 ]

Gradient Descent

First, we train using basic gradient descent method apply weak line search conditions.

TrainingTester.java:480 executed in 0.20 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 254674561344
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 254703480438
Constructing line search parameters: GD
th(0)=0.20205829154466043;dx=-0.20000000000000004
New Minimum: 0.20205829154466043 > 0.05372855756854631
END: th(2.154434690031884)=0.05372855756854631; dx=-0.031318083923688594 evalInputDelta=0.14832973397611413
Fitness changed from 0.20205829154466043 to 0.05372855756854631
Iteration 1 complete. Error: 0.05372855756854631 Total: 0.0524; Orientation: 0.0029; Line Search: 0.0126
th(0)=0.05372855756854631;dx=-0.004904111903256011
New Minimum: 0.05372855756854631 > 0.03369675944530693
END: th(4.641588833612779)=0.03369675944530693; dx=-0.003817808399627478 evalInputDelta=0.02003179812323938
Fitness changed from 0.05372855756854631 to 0.03369675944530693
Iteration 2 complete. Error: 0.03369675944530693 Total: 0.0138; Orientation: 0.0008; Line Search: 0.0092
th(0)=0.03369675944530693;dx=-0.002972130584252949
New Minimum: 0.03369675944530693 > 0.008423404812076674
END: th(10.000000000000002)=0.008423404812076674; dx=-0.002167039891791671 evalInputDelta=0.025273354633230258
Fitness changed from 0.03369675944530693 to 0.008423404812076674
Iteration 3 complete. Error: 0.008423404812076674 Total: 0.0108; Orientation: 0.0011; Line Search: 0.0067
th(0)=0.008423404812076674;dx=-0.001580032155214614
New Minimum: 0.008423404812076674 > 0.0
WOLFE (weak): th(21.544346900318843)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(43.088693800637685)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(129.26608140191306)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(517.0643256076522)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(2585.321628038261)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(15511.929768229567)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(108583.50837760697)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(868668.0670208557)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(7818012.603187702)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(4343340.335104278)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(6080676.46914599)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(5212008.402125134)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(5646342.435635562)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(5429175.418880348)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
WOLFE (weak): th(5320591.910502741)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(5374883.664691545)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Armijo: th(5347737.787597143)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
mu ~= nu (5320591.910502741): th(21.544346900318843)=0.0
Fitness changed from 0.008423404812076674 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0601; Orientation: 0.0010; Line Search: 0.0562
th(0)=0.0;dx=-0.0012799999999999999
Armijo: th(1.1492109793141883E7)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(5746054.896570941)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(1915351.632190314)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(478837.9080475785)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(95767.58160951569)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(15961.263601585948)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(2280.1805145122785)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(285.0225643140348)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(31.669173812670536)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(3.1669173812670537)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(0.2879015801151867)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(0.023991798342932222)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(0.0018455229494563248)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(1.3182306781830893E-4)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(8.788204521220595E-6)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(5.492627825762872E-7)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(3.230957544566395E-8)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(1.7949764136479975E-9)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(9.447244282357881E-11)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(4.723622141178941E-12)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(2.2493438767518765E-13)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Armijo: th(1.0224290348872166E-14)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
MIN ALPHA (4.44534362994442E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0529; Orientation: 0.0009; Line Search: 0.0497
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.190s (< 30.000s)

Returns

    0.0

Training Converged

Conjugate Gradient Descent

First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.

TrainingTester.java:452 executed in 0.11 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
      iterativeTrainer.setOrientation(new GradientDescent());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 254870756473
Reset training subject: 254872402893
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.20205829154466043}, derivative=-0.20000000000000004}
New Minimum: 0.20205829154466043 > 0.20205829152466043
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.20205829152466043}, derivative=-0.19999999995000003}, evalInputDelta = -2.000000165480742E-11
New Minimum: 0.20205829152466043 > 0.20205829140466042
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.20205829140466042}, derivative=-0.19999999965000004}, evalInputDelta = -1.4000001158365194E-10
New Minimum: 0.20205829140466042 > 0.20205829056466046
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.20205829056466046}, derivative=-0.19999999755000009}, evalInputDelta = -9.799999700632611E-10
New Minimum: 0.20205829056466046 > 0.20205828468466075
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.20205828468466075}, derivative=-0.1999999828500015}, evalInputDelta = -6.859999679420525E-9
New Minimum: 0.20205828468466075 > 0.20205824352467486
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.20205824352467486}, derivative=-0.1999998799500721}, evalInputDelta = -4.801998557124598E-8
New Minimum: 0.20205824352467486 > 0.20205795540536667
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.20205795540536667}, derivative=-0.19999915965353096}, evalInputDelta = -3.3613929376064E-7
New Minimum: 0.20205795540536667 > 0.202055938599263
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.202055938599263}, derivative=-0.19999411772301104}, evalInputDelta = -2.352945397438022E-6
New Minimum: 0.202055938599263 > 0.20204182237998541
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.20204182237998541}, derivative=-0.19995883132604333}, evalInputDelta = -1.6469164675014802E-5
New Minimum: 0.20204182237998541 > 0.20194307852724752
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.20194307852724752}, derivative=-0.1997121747638024}, evalInputDelta = -1.1521301741290735E-4
New Minimum: 0.20194307852724752 > 0.20125526326401263
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.20125526326401263}, derivative=-0.19800247151981518}, evalInputDelta = -8.03028280647794E-4
New Minimum: 0.20125526326401263 > 0.19659934670373755
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.19659934670373755}, derivative=-0.1868078514141577}, evalInputDelta = -0.00545894484092288
New Minimum: 0.19659934670373755 > 0.16992396562862108
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.16992396562862108}, derivative=-0.13383909469845756}, evalInputDelta = -0.03213432591603935
New Minimum: 0.16992396562862108 > 0.08244061826853832
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.08244061826853832}, derivative=-0.04483981402166769}, evalInputDelta = -0.1196176732761221
New Minimum: 0.08244061826853832 > 0.0
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
0.0 <= 0.20205829154466043
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.016}, evalInputDelta = -0.20205829154466043
Right bracket at 1.0E10
Converged to right
Fitness changed from 0.20205829154466043 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0650; Orientation: 0.0006; Line Search: 0.0599
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-0.0012799999999999999}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0435; Orientation: 0.0006; Line Search: 0.0412
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.109s (< 30.000s)

Returns

    0.0

Training Converged

Limited-Memory BFGS

Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.

TrainingTester.java:509 executed in 0.14 seconds (0.000 gc):

    IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
    try {
      iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
      iterativeTrainer.setOrientation(new LBFGS());
      iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
      iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
      iterativeTrainer.setIterationsPerSample(100);
      iterativeTrainer.setMaxIterations(250);
      iterativeTrainer.setTerminateThreshold(0);
      return iterativeTrainer.run();
    } finally {
      iterativeTrainer.freeRef();
    }
Logging 
Reset training subject: 254985063389
Reset training subject: 254986622645
Adding measurement 1fa9b5c3 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.20205829154466043 < 0.20205829154466043. Total: 1
th(0)=0.20205829154466043;dx=-0.20000000000000004
Adding measurement 6ebf9875 to history. Total: 1
New Minimum: 0.20205829154466043 > 0.05372855756854631
END: th(2.154434690031884)=0.05372855756854631; dx=-0.031318083923688594 evalInputDelta=0.14832973397611413
Fitness changed from 0.20205829154466043 to 0.05372855756854631
Iteration 1 complete. Error: 0.05372855756854631 Total: 0.0134; Orientation: 0.0034; Line Search: 0.0056
Non-optimal measurement 0.05372855756854631 < 0.05372855756854631. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.05372855756854631 < 0.05372855756854631. Total: 2
th(0)=0.05372855756854631;dx=-0.004904111903256011
Adding measurement 10bab77f to history. Total: 2
New Minimum: 0.05372855756854631 > 0.03369675944530693
END: th(4.641588833612779)=0.03369675944530693; dx=-0.003817808399627478 evalInputDelta=0.02003179812323938
Fitness changed from 0.05372855756854631 to 0.03369675944530693
Iteration 2 complete. Error: 0.03369675944530693 Total: 0.0075; Orientation: 0.0013; Line Search: 0.0045
Non-optimal measurement 0.03369675944530693 < 0.03369675944530693. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.03369675944530693 < 0.03369675944530693. Total: 3
th(0)=0.03369675944530693;dx=-0.002972130584252949
Adding measurement 24b8fa93 to history. Total: 3
New Minimum: 0.03369675944530693 > 0.008423404812076674
END: th(10.000000000000002)=0.008423404812076674; dx=-0.002167039891791671 evalInputDelta=0.025273354633230258
Fitness changed from 0.03369675944530693 to 0.008423404812076674
Iteration 3 complete. Error: 0.008423404812076674 Total: 0.0083; Orientation: 0.0016; Line Search: 0.0052
Non-optimal measurement 0.008423404812076674 < 0.008423404812076674. Total: 4
Rejected: LBFGS Orientation magnitude: 1.015e-01, gradient 3.975e-02, dot -1.000; [ad343a35-2949-4b9a-8f9e-eddab197f08d = 1.000/1.000e+00, 7c5b8ef6-aed0-4f49-a834-883fc0f45d78 = 1.000/1.000e+00, c94886c3-8bb3-4278-bacb-50b55afa37b9 = 1.000/1.000e+00, b4bcf8a3-bceb-409d-97db-5989d5ece2c1 = 1.000/1.000e+00, 879a99c1-5dcf-47c8-939a-b8099ae056c1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.008423404812076674, 0.03369675944530693, 0.05372855756854631, 0.20205829154466043
LBFGS Accumulation History: 3 points
Removed measurement 24b8fa93 to history. Total: 3
Adding measurement 79d2b89a to history. Total: 3
th(0)=0.008423404812076674;dx=-0.001580032155214614
Adding measurement 71088e32 to history. Total: 4
New Minimum: 0.008423404812076674 > 0.0
WOLFE (weak): th(21.544346900318843)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(43.088693800637685)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(129.26608140191306)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(517.0643256076522)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(2585.321628038261)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(15511.929768229567)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(108583.50837760697)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(868668.0670208557)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(7818012.603187702)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(4343340.335104278)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(6080676.46914599)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(5212008.402125134)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(5646342.435635562)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(5429175.418880348)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
WOLFE (weak): th(5320591.910502741)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(5374883.664691545)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
Armijo: th(5347737.787597143)=0.0; dx=-0.0014221255776740343 evalInputDelta=0.008423404812076674
Non-optimal measurement 0.0 < 0.0. Total: 5
mu ~= nu (5320591.910502741): th(21.544346900318843)=0.0
Fitness changed from 0.008423404812076674 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0553; Orientation: 0.0155; Line Search: 0.0384
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 9.139e-02, gradient 3.578e-02, dot -1.000; [ad343a35-2949-4b9a-8f9e-eddab197f08d = 1.000/1.000e+00, c94886c3-8bb3-4278-bacb-50b55afa37b9 = 1.000/1.000e+00, b4bcf8a3-bceb-409d-97db-5989d5ece2c1 = 1.000/1.000e+00, 7c5b8ef6-aed0-4f49-a834-883fc0f45d78 = 1.000/1.000e+00, 879a99c1-5dcf-47c8-939a-b8099ae056c1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.008423404812076674, 0.03369675944530693, 0.05372855756854631, 0.20205829154466043
Rejected: LBFGS Orientation magnitude: 7.497e-01, gradient 3.578e-02, dot -1.000; [7c5b8ef6-aed0-4f49-a834-883fc0f45d78 = 1.000/1.000e+00, c94886c3-8bb3-4278-bacb-50b55afa37b9 = 1.000/1.000e+00, ad343a35-2949-4b9a-8f9e-eddab197f08d = 1.000/1.000e+00, 879a99c1-5dcf-47c8-939a-b8099ae056c1 = 1.000/1.000e+00, b4bcf8a3-bceb-409d-97db-5989d5ece2c1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.0, 0.008423404812076674, 0.03369675944530693, 0.05372855756854631
LBFGS Accumulation History: 3 points
Removed measurement 71088e32 to history. Total: 4
Removed measurement 79d2b89a to history. Total: 3
Adding measurement 6d861a02 to history. Total: 3
th(0)=0.0;dx=-0.0012799999999999999
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.1492109793141883E7)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(5746054.896570941)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1915351.632190314)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(478837.9080475785)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(95767.58160951569)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(15961.263601585948)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2280.1805145122785)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(285.0225643140348)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(31.669173812670536)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.1669173812670537)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.2879015801151867)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.023991798342932222)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(0.0018455229494563248)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.3182306781830893E-4)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(8.788204521220595E-6)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(5.492627825762872E-7)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(3.230957544566395E-8)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.7949764136479975E-9)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(9.447244282357881E-11)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(4.723622141178941E-12)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(2.2493438767518765E-13)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
Armijo: th(1.0224290348872166E-14)=0.0; dx=-0.0012799999999999999 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 4
MIN ALPHA (4.44534362994442E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0591; Orientation: 0.0176; Line Search: 0.0403
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.144s (< 30.000s)

Returns

    0.0

Training Converged

TrainingTester.java:432 executed in 0.39 seconds (0.000 gc):

    return TestUtil.compare(title + " vs Iteration", runs);
Logging 
Plotting range=[1.0, -2.0745123274781694], [4.0, -1.2697948186241161]; valueStats=DoubleSummaryStatistics{count=6, sum=0.191697, min=0.008423, average=0.031950, max=0.053729}
Plotting 4 points for GD
Only 1 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):

    return TestUtil.compareTime(title + " vs Time", runs);
Logging 
Plotting range=[0.0, -2.0745123274781694], [0.084, -1.2697948186241161]; valueStats=DoubleSummaryStatistics{count=6, sum=0.191697, min=0.008423, average=0.031950, max=0.053729}
Plotting 4 points for GD
Only 1 points for CjGD
Plotting 4 points for LBFGS

Returns

Result

Results

TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):

    return grid(inputLearning, modelLearning, completeLearning);

Returns

Result

TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):

    return new ComponentResult(null == inputLearning ? null : inputLearning.value,
        null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);

Returns

    {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}

LayerTests.java:425 executed in 0.00 seconds (0.000 gc):

    throwException(exceptions.addRef());

Results

detailsresult
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}OK
 
  {
    "result": "OK",
    "performance": {
      "execution_time": "1.873",
      "gc_time": "0.225"
    },
    "created_on": 1586734842481,
    "file_name": "trainingTest",
    "report": {
      "simpleName": "Basic",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.StaticScalarLossLayerTest.Basic",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/StaticScalarLossLayerTest.java",
      "javaDoc": ""
    },
    "training_analysis": {
      "input": {
        "LBFGS": {
          "type": "Converged",
          "value": 0.0
        },
        "CjGD": {
          "type": "Converged",
          "value": 0.0
        },
        "GD": {
          "type": "Converged",
          "value": 0.0
        }
      }
    },
    "archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/StaticScalarLossLayer/Basic/trainingTest/202004124042",
    "id": "ff111322-3643-4985-ae94-b35d33a01f36",
    "report_type": "Components",
    "display_name": "Comparative Training",
    "target": {
      "simpleName": "StaticScalarLossLayer",
      "canonicalName": "com.simiacryptus.mindseye.layers.java.StaticScalarLossLayer",
      "link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/StaticScalarLossLayer.java",
      "javaDoc": ""
    }
  }