Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ -0.128, 0.08, 0.7 ]
[ 0.496, 1.764, -0.608 ]
[ 0.7, -0.128, 0.08 ]
[ 1.764, -0.608, 0.496 ]
[ 0.08, -0.128, 0.7 ]
[ 0.496, -0.608, 1.764 ]
[ 0.08, 0.7, -0.128 ]
[ 1.764, 0.496, -0.608 ]
[ -0.128, 0.08, 0.7 ]
[ 1.764, 0.496, -0.608 ]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.21 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1715591002445
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 1715622484663
Constructing line search parameters: GD
th(0)=52.206048210555274;dx=-4.28146944E24
New Minimum: 52.206048210555274 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-4.281469440000603E12 evalInputDelta=52.206048210555274
Armijo: th(1.077217345015942)=0.0; dx=-4.281469440000603E12 evalInputDelta=52.206048210555274
Armijo: th(0.3590724483386473)=0.01667345354575365; dx=-4.28146944000062E12 evalInputDelta=52.18937475700952
Armijo: th(0.08976811208466183)=0.09415279262501829; dx=-4.281469440000718E12 evalInputDelta=52.111895417930256
Armijo: th(0.017953622416932366)=0.12012254562659819; dx=-4.2814694400007583E12 evalInputDelta=52.08592566492867
Armijo: th(0.002992270402822061)=0.12592199567876436; dx=-4.2814694400007686E12 evalInputDelta=52.08012621487651
Armijo: th(4.2746720040315154E-4)=0.12693101149771216; dx=-4.28146944000077E12 evalInputDelta=52.07911719905756
Armijo: th(5.343340005039394E-5)=0.12707852953535054; dx=-4.28146944000077E12 evalInputDelta=52.07896968101992
Armijo: th(5.9370444500437714E-6)=0.12709726874480118; dx=-4.281469440000771E12 evalInputDelta=52.078950941810476
Armijo: th(5.937044450043771E-7)=0.12709937700124432; dx=-4.2814694400007705E12 evalInputDelta=52.07894883355403
Armijo: th(5.397313136403428E-8)=0.1270995899575133; dx=-4.281469440000771E12 evalInputDelta=52.078948620597764
Armijo: th(4.4977609470028565E-9)=0.12709960947851448; dx=-4.2814694400007705E12 evalInputDelta=52.07894860107676
Armijo: th(3.4598161130791205E-10)=0.12709961111664056; dx=-4.28146944000077E12 evalInputDelta=52.07894859943863
Armijo: th(2.4712972236279432E-11)=0.12709961124340027; dx=-4.281469440000771E12 evalInputDelta=52.07894859931187
Armijo: th(1.6475314824186289E-12)=0.12709961125250097; dx=-4.281469440000771E12 evalInputDelta=52.07894859930277
Armijo: th(1.029707176511643E-13)=52.20604821055524; dx=-4.28146944E24 evalInputDelta=3.552713678800501E-14
Armijo: th(6.057101038303783E-15)=52.206048210555274; dx=-4.28146944E24 evalInputDelta=0.0
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 52.206048210555274 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.1793; Orientation: 0.0041; Line Search: 0.1323
th(0)=0.0;dx=-4.76056704
Armijo: th(2.154434690031884E-15)=0.0; dx=-4.76056704 evalInputDelta=0.0
Armijo: th(1.077217345015942E-15)=0.0; dx=-4.76056704 evalInputDelta=0.0
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0203; Orientation: 0.0011; Line Search: 0.0160
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.201s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.19 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1715796925911
Reset training subject: 1715799484550
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=52.206048210555274}, derivative=-4.28146944E24}
New Minimum: 52.206048210555274 > 0.12709961121369506
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.12709961121369506}, derivative=-4.281469440000771E12}, evalInputDelta = -52.07894859934158
New Minimum: 0.12709961121369506 > 0.12709961097695904
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.12709961097695904}, derivative=-4.2814694400007705E12}, evalInputDelta = -52.07894859957832
New Minimum: 0.12709961097695904 > 0.12709960931980704
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.12709960931980704}, derivative=-4.28146944000077E12}, evalInputDelta = -52.07894860123547
New Minimum: 0.12709960931980704 > 0.12709959771974347
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.12709959771974347}, derivative=-4.281469440000771E12}, evalInputDelta = -52.07894861283553
New Minimum: 0.12709959771974347 > 0.12709951651931453
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.12709951651931453}, derivative=-4.2814694400007705E12}, evalInputDelta = -52.07894869403596
New Minimum: 0.12709951651931453 > 0.12709894811711334
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.12709894811711334}, derivative=-4.281469440000771E12}, evalInputDelta = -52.07894926243816
New Minimum: 0.12709894811711334 > 0.12709496934096806
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.12709496934096806}, derivative=-4.281469440000771E12}, evalInputDelta = -52.07895324121431
New Minimum: 0.12709496934096806 > 0.12706711983165003
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.12706711983165003}, derivative=-4.2814694400007705E12}, evalInputDelta = -52.07898109072362
New Minimum: 0.12706711983165003 > 0.12687226746585353
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.12687226746585353}, derivative=-4.2814694400007705E12}, evalInputDelta = -52.07917594308942
New Minimum: 0.12687226746585353 > 0.12551289558561574
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.12551289558561574}, derivative=-4.2814694400007676E12}, evalInputDelta = -52.08053531496966
New Minimum: 0.12551289558561574 > 0.11621544979692693
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.11621544979692693}, derivative=-4.2814694400007524E12}, evalInputDelta = -52.08983276075835
New Minimum: 0.11621544979692693 > 0.05990641318630145
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.05990641318630145}, derivative=-4.281469440000671E12}, evalInputDelta = -52.14614179736897
New Minimum: 0.05990641318630145 > 0.0
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.2814694400006025E12}, evalInputDelta = -52.206048210555274
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
0.0 <= 52.206048210555274
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.281469440000603E12}, evalInputDelta = -52.206048210555274
Right bracket at 1.0E10
Converged to right
Fitness changed from 52.206048210555274 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.1149; Orientation: 0.0013; Line Search: 0.1070
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.76056704}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0757; Orientation: 0.0008; Line Search: 0.0729
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.191s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.08 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1715992892924
Reset training subject: 1715994639653
Adding measurement 7f501f19 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 52.206048210555274 < 52.206048210555274. Total: 1
th(0)=52.206048210555274;dx=-4.2814694399999994E24
Adding measurement 392f1107 to history. Total: 1
New Minimum: 52.206048210555274 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-4.281469440000603E12 evalInputDelta=52.206048210555274
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942)=0.0; dx=-4.281469440000603E12 evalInputDelta=52.206048210555274
Non-optimal measurement 0.01667345354575365 < 0.0. Total: 2
Armijo: th(0.3590724483386473)=0.01667345354575365; dx=-4.28146944000062E12 evalInputDelta=52.18937475700952
Non-optimal measurement 0.09415279262501829 < 0.0. Total: 2
Armijo: th(0.08976811208466183)=0.09415279262501829; dx=-4.281469440000718E12 evalInputDelta=52.111895417930256
Non-optimal measurement 0.12012254562659819 < 0.0. Total: 2
Armijo: th(0.017953622416932366)=0.12012254562659819; dx=-4.2814694400007583E12 evalInputDelta=52.08592566492867
Non-optimal measurement 0.12592199567876436 < 0.0. Total: 2
Armijo: th(0.002992270402822061)=0.12592199567876436; dx=-4.2814694400007686E12 evalInputDelta=52.08012621487651
Non-optimal measurement 0.12693101149771216 < 0.0. Total: 2
Armijo: th(4.2746720040315154E-4)=0.12693101149771216; dx=-4.28146944000077E12 evalInputDelta=52.07911719905756
Non-optimal measurement 0.12707852953535054 < 0.0. Total: 2
Armijo: th(5.343340005039394E-5)=0.12707852953535054; dx=-4.28146944000077E12 evalInputDelta=52.07896968101992
Non-optimal measurement 0.12709726874480118 < 0.0. Total: 2
Armijo: th(5.9370444500437714E-6)=0.12709726874480118; dx=-4.281469440000771E12 evalInputDelta=52.078950941810476
Non-optimal measurement 0.12709937700124432 < 0.0. Total: 2
Armijo: th(5.937044450043771E-7)=0.12709937700124432; dx=-4.2814694400007705E12 evalInputDelta=52.07894883355403
Non-optimal measurement 0.1270995899575133 < 0.0. Total: 2
Armijo: th(5.397313136403428E-8)=0.1270995899575133; dx=-4.281469440000771E12 evalInputDelta=52.078948620597764
Non-optimal measurement 0.12709960947851448 < 0.0. Total: 2
Armijo: th(4.4977609470028565E-9)=0.12709960947851448; dx=-4.2814694400007705E12 evalInputDelta=52.07894860107676
Non-optimal measurement 0.12709961111664056 < 0.0. Total: 2
Armijo: th(3.4598161130791205E-10)=0.12709961111664056; dx=-4.2814694400007705E12 evalInputDelta=52.07894859943863
Non-optimal measurement 0.12709961124340027 < 0.0. Total: 2
Armijo: th(2.4712972236279432E-11)=0.12709961124340027; dx=-4.281469440000771E12 evalInputDelta=52.07894859931187
Non-optimal measurement 0.12709961125250097 < 0.0. Total: 2
Armijo: th(1.6475314824186289E-12)=0.12709961125250097; dx=-4.281469440000771E12 evalInputDelta=52.07894859930277
Non-optimal measurement 52.20604821055524 < 0.0. Total: 2
Armijo: th(1.029707176511643E-13)=52.20604821055524; dx=-4.28146944E24 evalInputDelta=3.552713678800501E-14
Non-optimal measurement 52.206048210555274 < 0.0. Total: 2
Armijo: th(6.057101038303783E-15)=52.206048210555274; dx=-4.28146944E24 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 52.206048210555274 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.0673; Orientation: 0.0041; Line Search: 0.0574
Non-optimal measurement 0.0 < 0.0. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.0 < 0.0. Total: 2
th(0)=0.0;dx=-4.76056704
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(2.154434690031884E-15)=0.0; dx=-4.76056704 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942E-15)=0.0; dx=-4.76056704 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0101; Orientation: 0.0013; Line Search: 0.0070
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.078s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.09 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[0.0, 0.0], [2.0, 1.0]; valueStats=DoubleSummaryStatistics{count=0, sum=0.000000, min=Infinity, average=0.000000, max=-Infinity}
Only 0 points for GD
Only 0 points for CjGD
Only 0 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.00 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
No Data
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
details | result |
---|---|
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "1.029",
"gc_time": "0.224"
},
"created_on": 1586736303398,
"file_name": "trainingTest",
"report": {
"simpleName": "Basic",
"canonicalName": "com.simiacryptus.mindseye.layers.java.BiasMetaLayerTest.Basic",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/BiasMetaLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/BiasMetaLayer/Basic/trainingTest/202004130503",
"id": "08030049-a1c5-4999-bea3-1e0c2f08bfb9",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "BiasMetaLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.java.BiasMetaLayer",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/BiasMetaLayer.java",
"javaDoc": ""
}
}