Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ 0.7, -0.128, 0.08 ] ]
]
[
[ [ -0.128, 0.08, 0.7 ] ]
]
[
[ [ 0.7, -0.128, 0.08 ] ]
]
[
[ [ -0.128, 0.7, 0.08 ] ]
]
[
[ [ 0.08, -0.128, 0.7 ] ]
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.39 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 4507184176040
Reset training subject: 4507241558271
Constructing line search parameters: GD
th(0)=0.1497648;dx=-0.05990592
New Minimum: 0.1497648 > 0.048507272747655206
END: th(2.154434690031884)=0.048507272747655206; dx=-0.03409324156274503 evalInputDelta=0.1012575272523448
Fitness changed from 0.1497648 to 0.048507272747655206
Iteration 1 complete. Error: 0.048507272747655206 Total: 0.1577; Orientation: 0.0077; Line Search: 0.0553
th(0)=0.048507272747655206;dx=-0.019402909099062084
New Minimum: 0.048507272747655206 > 2.4924698439949313E-4
END: th(4.641588833612779)=2.4924698439949313E-4; dx=-0.0013908438563000106 evalInputDelta=0.04825802576325571
Fitness changed from 0.048507272747655206 to 2.4924698439949313E-4
Iteration 2 complete. Error: 2.4924698439949313E-4 Total: 0.0678; Orientation: 0.0017; Line Search: 0.0472
th(0)=2.4924698439949313E-4;dx=-9.969879375979725E-5
Armijo: th(10.000000000000002)=2.4924698439949313E-4; dx=9.969879375979725E-5 evalInputDelta=0.0
New Minimum: 2.4924698439949313E-4 > 0.0
END: th(5.000000000000001)=0.0; dx=0.0 evalInputDelta=2.4924698439949313E-4
Fitness changed from 2.4924698439949313E-4 to 0.0
Iteration 3 complete. Error: 0.0 Total: 0.0939; Orientation: 0.0015; Line Search: 0.0702
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0552; Orientation: 0.0017; Line Search: 0.0348
Iteration 4 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 4
Final threshold in iteration 4: 0.0 (> 0.0) after 0.375s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.34 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 4507566894280
Reset training subject: 4507581733242
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=0.1497648}, derivative=-0.05990592}
New Minimum: 0.1497648 > 0.14976479999400938
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=0.14976479999400938}, derivative=-0.059905919998801876}, evalInputDelta = -5.9906246629992665E-12
New Minimum: 0.14976479999400938 > 0.14976479995806585
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=0.14976479995806585}, derivative=-0.05990591999161317}, evalInputDelta = -4.193415059638994E-11
New Minimum: 0.14976479995806585 > 0.14976479970646095
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=0.14976479970646095}, derivative=-0.0599059199412922}, evalInputDelta = -2.935390541747296E-10
New Minimum: 0.14976479970646095 > 0.14976479794522693
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=0.14976479794522693}, derivative=-0.05990591958904539}, evalInputDelta = -2.0547730739117753E-9
New Minimum: 0.14976479794522693 > 0.14976478561658893
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=0.14976478561658893}, derivative=-0.05990591712331772}, evalInputDelta = -1.4383411073293217E-8
New Minimum: 0.14976478561658893 > 0.14976469931613715
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=0.14976469931613715}, derivative=-0.05990589986322405}, evalInputDelta = -1.006838628581086E-7
New Minimum: 0.14976469931613715 > 0.14976409521367096
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=0.14976409521367096}, derivative=-0.05990577904256836}, evalInputDelta = -7.047863290476908E-7
New Minimum: 0.14976409521367096 > 0.1497598665305221
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=0.1497598665305221}, derivative=-0.05990493329797851}, evalInputDelta = -4.93346947791351E-6
New Minimum: 0.1497598665305221 > 0.14973026742009704
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=0.14973026742009704}, derivative=-0.05989901308584955}, evalInputDelta = -3.453257990296277E-5
New Minimum: 0.14973026742009704 > 0.14952315555634937
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=0.14952315555634937}, derivative=-0.05985757160094693}, evalInputDelta = -2.4164444365062931E-4
New Minimum: 0.14952315555634937 > 0.14807738606226403
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=0.14807738606226403}, derivative=-0.05956748120662852}, evalInputDelta = -0.001687413937735971
New Minimum: 0.14807738606226403 > 0.13815366365894885
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=0.13815366365894885}, derivative=-0.05753684844639963}, evalInputDelta = -0.011611136341051154
New Minimum: 0.13815366365894885 > 0.0783241455445727
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=0.0783241455445727}, derivative=-0.043322419124797396}, evalInputDelta = -0.0714406544554273
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.13170791547660848}, derivative=0.05617858612641821}, evalInputDelta = -0.01805688452339152
0.13170791547660848 <= 0.1497648
New Minimum: 0.0783241455445727 > 0.0
F(5.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -0.1497648
Right bracket at 5.0
Converged to right
Fitness changed from 0.1497648 to 0.0
Iteration 1 complete. Error: 0.0 Total: 0.3091; Orientation: 0.0012; Line Search: 0.2670
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0278; Orientation: 0.0010; Line Search: 0.0141
Iteration 2 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 2
Final threshold in iteration 2: 0.0 (> 0.0) after 0.338s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.24 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 4507912087613
Reset training subject: 4507926010835
Adding measurement 72997067 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 0.1497648 < 0.1497648. Total: 1
th(0)=0.1497648;dx=-0.05990592
Adding measurement 71c98c48 to history. Total: 1
New Minimum: 0.1497648 > 0.048507272747655206
END: th(2.154434690031884)=0.048507272747655206; dx=-0.034093241562745034 evalInputDelta=0.1012575272523448
Fitness changed from 0.1497648 to 0.048507272747655206
Iteration 1 complete. Error: 0.048507272747655206 Total: 0.0705; Orientation: 0.0052; Line Search: 0.0282
Non-optimal measurement 0.048507272747655206 < 0.048507272747655206. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 0.048507272747655206 < 0.048507272747655206. Total: 2
th(0)=0.048507272747655206;dx=-0.01940290909906208
Adding measurement 71c5b288 to history. Total: 2
New Minimum: 0.048507272747655206 > 2.4924698439949313E-4
END: th(4.641588833612779)=2.4924698439949313E-4; dx=-0.0013908438563000106 evalInputDelta=0.04825802576325571
Fitness changed from 0.048507272747655206 to 2.4924698439949313E-4
Iteration 2 complete. Error: 2.4924698439949313E-4 Total: 0.0434; Orientation: 0.0071; Line Search: 0.0254
Non-optimal measurement 2.4924698439949313E-4 < 2.4924698439949313E-4. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 2.4924698439949313E-4 < 2.4924698439949313E-4. Total: 3
th(0)=2.4924698439949313E-4;dx=-9.969879375979725E-5
Non-optimal measurement 2.4924698439949313E-4 < 2.4924698439949313E-4. Total: 3
Armijo: th(10.000000000000002)=2.4924698439949313E-4; dx=9.969879375979725E-5 evalInputDelta=0.0
Adding measurement 4c77c7b7 to history. Total: 3
New Minimum: 2.4924698439949313E-4 > 0.0
END: th(5.000000000000001)=0.0; dx=0.0 evalInputDelta=2.4924698439949313E-4
Fitness changed from 2.4924698439949313E-4 to 0.0
Iteration 3 complete. Error: 0.0 Total: 0.0486; Orientation: 0.0016; Line Search: 0.0369
Non-optimal measurement 0.0 < 0.0. Total: 4
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [11d39ac0-e923-4aab-bea1-62d72f43223c = 0.000e+00, 0ce00b83-6114-403d-b662-a96c293ed026 = 0.000e+00, 0c422ccb-858b-4a61-a741-0458313c5eda = 0.000e+00, 85c4e679-13e3-4b9f-bff0-4c7c49b7f816 = 0.000e+00, e24f30fa-f910-45c2-a75c-4eca7e1aac16 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 2.4924698439949313E-4, 0.048507272747655206, 0.1497648
LBFGS Accumulation History: 3 points
Removed measurement 4c77c7b7 to history. Total: 3
Adding measurement 4473a384 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0766; Orientation: 0.0389; Line Search: 0.0249
Iteration 4 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 4
Final threshold in iteration 4: 0.0 (> 0.0) after 0.239s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.15 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -3.603370087445925], [3.0, -1.3141931422764295]; valueStats=DoubleSummaryStatistics{count=4, sum=0.097513, min=0.000249, average=0.024378, max=0.048507}
Plotting 3 points for GD
Only 1 points for CjGD
Plotting 3 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -3.603370087445925], [0.161, -1.3141931422764295]; valueStats=DoubleSummaryStatistics{count=4, sum=0.097513, min=0.000249, average=0.024378, max=0.048507}
Plotting 3 points for GD
Only 1 points for CjGD
Plotting 3 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "1.842",
"gc_time": "0.246"
},
"created_on": 1586739138807,
"file_name": "trainingTest",
"report": {
"simpleName": "BandLimitTest",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgConcatLayerTest.BandLimitTest",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgConcatLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgConcatLayer/BandLimitTest/trainingTest/202004135218",
"id": "55d36118-730d-4d96-9c94-aa560f9399c4",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "ImgConcatLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgConcatLayer",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgConcatLayer.java",
"javaDoc": ""
}
}