Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
Using Seed 6970473746721338368
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.048, -1.028, -0.384 ], [ -0.128, -0.608, 0.496 ] ],
[ [ 1.208, -1.72, 0.7 ], [ 1.764, 0.08, 1.524 ] ]
]
[
[ [ 0.7, -0.608, 0.048 ], [ 1.524, 1.208, -0.128 ] ],
[ [ 1.764, 0.08, -1.028 ], [ 0.496, -1.72, -0.384 ] ]
]
[
[ [ -0.384, 1.764, 0.048 ], [ 1.208, -1.72, -0.608 ] ],
[ [ 0.496, 0.08, 1.524 ], [ -0.128, 0.7, -1.028 ] ]
]
[
[ [ -1.028, 0.08, -0.384 ], [ 0.7, 1.524, -0.128 ] ],
[ [ 0.496, 0.048, -1.72 ], [ 1.764, 1.208, -0.608 ] ]
]
[
[ [ -1.028, 1.208, 0.048 ], [ -0.608, 1.764, 0.7 ] ],
[ [ 0.08, -1.72, 1.524 ], [ -0.128, 0.496, -0.384 ] ]
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.34 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1148272996880
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 1148313413425
Constructing line search parameters: GD
th(0)=49.733527300149476;dx=-2.2900454400000005E24
New Minimum: 49.733527300149476 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Armijo: th(1.077217345015942)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Armijo: th(0.3590724483386473)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Armijo: th(0.08976811208466183)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Armijo: th(0.017953622416932366)=0.055267308354657055; dx=-2.2900454400026807E12 evalInputDelta=49.67825999179482
Armijo: th(0.002992270402822061)=0.0808692883827264; dx=-2.290045440002746E12 evalInputDelta=49.652658011766746
Armijo: th(4.2746720040315154E-4)=0.08535759222045643; dx=-2.290045440002758E12 evalInputDelta=49.64816970792902
Armijo: th(5.343340005039394E-5)=0.08601464239726922; dx=-2.2900454400027593E12 evalInputDelta=49.64751265775221
Armijo: th(5.9370444500437714E-6)=0.08609812321886968; dx=-2.29004544000276E12 evalInputDelta=49.64742917693061
Armijo: th(5.937044450043771E-7)=0.08610751545876398; dx=-2.29004544000276E12 evalInputDelta=49.64741978469071
Armijo: th(5.397313136403428E-8)=0.08610846417714692; dx=-2.29004544000276E12 evalInputDelta=49.64741883597233
Armijo: th(4.4977609470028565E-9)=0.08610855114306537; dx=-2.29004544000276E12 evalInputDelta=49.64741874900641
Armijo: th(3.4598161130791205E-10)=0.08610855844090518; dx=-2.29004544000276E12 evalInputDelta=49.64741874170857
Armijo: th(2.4712972236279432E-11)=0.10357571452032807; dx=-2.2920824031440854E12 evalInputDelta=49.62995158562915
Armijo: th(1.6475314824186289E-12)=2.397945369023384; dx=-6.593280003490795E21 evalInputDelta=47.33558193112609
Armijo: th(1.029707176511643E-13)=40.33658873817501; dx=-1.87667712000419E24 evalInputDelta=9.396938561974466
Armijo: th(6.057101038303783E-15)=49.73352730014946; dx=-2.2900454400000005E24 evalInputDelta=1.4210854715202004E-14
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 49.733527300149476 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.3053; Orientation: 0.0035; Line Search: 0.2470
th(0)=0.0;dx=-4.652718720000001
Armijo: th(2.154434690031884E-15)=0.0; dx=-4.652718720000001 evalInputDelta=0.0
Armijo: th(1.077217345015942E-15)=0.0; dx=-4.652718720000001 evalInputDelta=0.0
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0268; Orientation: 0.0014; Line Search: 0.0206
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.333s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.37 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1148611146220
Reset training subject: 1148614205855
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=49.733527300149476}, derivative=-2.2900454400000002E24}
New Minimum: 49.733527300149476 > 0.08610855887328239
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.08610855887328239}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64741874127619
New Minimum: 0.08610855887328239 > 0.08610855781862503
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.08610855781862503}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64741874233085
New Minimum: 0.08610855781862503 > 0.08610855043602486
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.08610855043602486}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64741874971345
New Minimum: 0.08610855043602486 > 0.08610849875782511
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.08610849875782511}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64741880139165
New Minimum: 0.08610849875782511 > 0.08610813701053867
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.08610813701053867}, derivative=-2.29004544000276E12}, evalInputDelta = -49.647419163138935
New Minimum: 0.08610813701053867 > 0.08610560478497138
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.08610560478497138}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64742169536451
New Minimum: 0.08610560478497138 > 0.08608787947253478
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.08608787947253478}, derivative=-2.29004544000276E12}, evalInputDelta = -49.64743942067694
New Minimum: 0.08608787947253478 > 0.08596381534377859
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.08596381534377859}, derivative=-2.2900454400027593E12}, evalInputDelta = -49.647563484805694
New Minimum: 0.08596381534377859 > 0.08509600566004769
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.08509600566004769}, derivative=-2.290045440002757E12}, evalInputDelta = -49.64843129448943
New Minimum: 0.08509600566004769 > 0.07905244204006437
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.07905244204006437}, derivative=-2.290045440002741E12}, evalInputDelta = -49.65447485810941
New Minimum: 0.07905244204006437 > 0.03820089140301722
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.03820089140301722}, derivative=-2.2900454400026387E12}, evalInputDelta = -49.69532640874646
New Minimum: 0.03820089140301722 > 0.0
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(474.7561509943) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(3323.2930569601003) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(23263.0513987207) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(162841.3597910449) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(1139889.5185373144) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(7979226.6297612) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(5.58545864083284E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(3.909821048582988E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(2.7368747340080914E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
F(1.915812313805664E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
0.0 <= 49.733527300149476
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-2.290045440002549E12}, evalInputDelta = -49.733527300149476
Right bracket at 1.0E10
Converged to right
Fitness changed from 49.733527300149476 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.1886; Orientation: 0.0011; Line Search: 0.1756
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}
F(1.0E10) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
0.0 <= 0.0
F(5.0E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 5.0E9
F(2.5E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 2.5E9
F(1.25E9) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 1.25E9
F(6.25E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 6.25E8
F(3.125E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 3.125E8
F(1.5625E8) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 1.5625E8
F(7.8125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 7.8125E7
F(3.90625E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 3.90625E7
F(1.953125E7) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 1.953125E7
F(9765625.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 9765625.0
F(4882812.5) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Right bracket at 4882812.5
F(2441406.25) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=-4.652718720000001}, evalInputDelta = 0.0
Loops = 12
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.1817; Orientation: 0.0009; Line Search: 0.1765
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.371s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.17 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 1148985795616
Reset training subject: 1148988338635
Adding measurement 64145c2b to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 49.733527300149476 < 49.733527300149476. Total: 1
th(0)=49.733527300149476;dx=-2.2900454400000005E24
Adding measurement 57cb644f to history. Total: 1
New Minimum: 49.733527300149476 > 0.0
Armijo: th(2.154434690031884)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.3590724483386473)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(0.08976811208466183)=0.0; dx=-2.290045440002549E12 evalInputDelta=49.733527300149476
Non-optimal measurement 0.055267308354657055 < 0.0. Total: 2
Armijo: th(0.017953622416932366)=0.055267308354657055; dx=-2.2900454400026807E12 evalInputDelta=49.67825999179482
Non-optimal measurement 0.0808692883827264 < 0.0. Total: 2
Armijo: th(0.002992270402822061)=0.0808692883827264; dx=-2.290045440002746E12 evalInputDelta=49.652658011766746
Non-optimal measurement 0.08535759222045643 < 0.0. Total: 2
Armijo: th(4.2746720040315154E-4)=0.08535759222045643; dx=-2.290045440002758E12 evalInputDelta=49.64816970792902
Non-optimal measurement 0.08601464239726922 < 0.0. Total: 2
Armijo: th(5.343340005039394E-5)=0.08601464239726922; dx=-2.2900454400027593E12 evalInputDelta=49.64751265775221
Non-optimal measurement 0.08609812321886968 < 0.0. Total: 2
Armijo: th(5.9370444500437714E-6)=0.08609812321886968; dx=-2.29004544000276E12 evalInputDelta=49.64742917693061
Non-optimal measurement 0.08610751545876398 < 0.0. Total: 2
Armijo: th(5.937044450043771E-7)=0.08610751545876398; dx=-2.29004544000276E12 evalInputDelta=49.64741978469071
Non-optimal measurement 0.08610846417714692 < 0.0. Total: 2
Armijo: th(5.397313136403428E-8)=0.08610846417714692; dx=-2.29004544000276E12 evalInputDelta=49.64741883597233
Non-optimal measurement 0.08610855114306537 < 0.0. Total: 2
Armijo: th(4.4977609470028565E-9)=0.08610855114306537; dx=-2.29004544000276E12 evalInputDelta=49.64741874900641
Non-optimal measurement 0.08610855844090518 < 0.0. Total: 2
Armijo: th(3.4598161130791205E-10)=0.08610855844090518; dx=-2.29004544000276E12 evalInputDelta=49.64741874170857
Non-optimal measurement 0.10357571452032807 < 0.0. Total: 2
Armijo: th(2.4712972236279432E-11)=0.10357571452032807; dx=-2.2920824031440854E12 evalInputDelta=49.62995158562915
Non-optimal measurement 2.397945369023384 < 0.0. Total: 2
Armijo: th(1.6475314824186289E-12)=2.397945369023384; dx=-6.593280003490795E21 evalInputDelta=47.33558193112609
Non-optimal measurement 40.33658873817501 < 0.0. Total: 2
Armijo: th(1.029707176511643E-13)=40.33658873817501; dx=-1.87667712000419E24 evalInputDelta=9.396938561974466
Non-optimal measurement 49.73352730014946 < 0.0. Total: 2
Armijo: th(6.057101038303783E-15)=49.73352730014946; dx=-2.2900454400000002E24 evalInputDelta=1.4210854715202004E-14
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.3650561323909904E-16): th(2.154434690031884)=0.0
Fitness changed from 49.733527300149476 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.1275; Orientation: 0.0045; Line Search: 0.1148
Non-optimal measurement 0.0 < 0.0. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.0 < 0.0. Total: 2
th(0)=0.0;dx=-4.652718720000001
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(2.154434690031884E-15)=0.0; dx=-4.652718720000001 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
Armijo: th(1.077217345015942E-15)=0.0; dx=-4.652718720000001 evalInputDelta=0.0
Non-optimal measurement 0.0 < 0.0. Total: 2
MIN ALPHA (3.5907244833864734E-16): th(0.0)=0.0
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0392; Orientation: 0.0011; Line Search: 0.0144
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.167s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.11 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[0.0, 0.0], [2.0, 1.0]; valueStats=DoubleSummaryStatistics{count=0, sum=0.000000, min=Infinity, average=0.000000, max=-Infinity}
Only 0 points for GD
Only 0 points for CjGD
Only 0 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.00 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
No Data
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
details | result |
---|---|
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "1.420",
"gc_time": "0.214"
},
"created_on": 1586735736077,
"file_name": "trainingTest",
"report": {
"simpleName": "Basic",
"canonicalName": "com.simiacryptus.mindseye.layers.java.ImgPixelSumLayerTest.Basic",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/ImgPixelSumLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/ImgPixelSumLayer/Basic/trainingTest/202004125536",
"id": "304ccadf-0130-4fe6-9250-0772f0a925a5",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "ImgPixelSumLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.java.ImgPixelSumLayer",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/ImgPixelSumLayer.java",
"javaDoc": ""
}
}