Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.01 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[
[ [ -0.608 ], [ 1.524 ], [ -0.128 ], [ -1.72 ] ],
[ [ 0.7 ], [ -0.852 ], [ 1.912 ], [ -0.384 ] ],
[ [ -1.688 ], [ 0.496 ], [ 1.764 ], [ 0.08 ] ],
[ [ -1.028 ], [ 1.208 ], [ 0.048 ], [ -0.804 ] ]
]
[
[ [ -0.852 ], [ 0.048 ], [ 1.524 ], [ 1.208 ] ],
[ [ -1.028 ], [ 1.764 ], [ -1.72 ], [ -0.128 ] ],
[ [ 1.912 ], [ -0.608 ], [ -1.688 ], [ 0.08 ] ],
[ [ 0.496 ], [ 0.7 ], [ -0.384 ], [ -0.804 ] ]
]
[
[ [ -0.384 ], [ -1.72 ], [ -0.852 ], [ 0.496 ] ],
[ [ 0.048 ], [ 1.524 ], [ 1.912 ], [ 0.08 ] ],
[ [ 1.208 ], [ -1.688 ], [ -0.608 ], [ -0.128 ] ],
[ [ -0.804 ], [ 0.7 ], [ -1.028 ], [ 1.764 ] ]
]
[
[ [ -0.128 ], [ 0.048 ], [ 0.7 ], [ 0.496 ] ],
[ [ -1.688 ], [ 0.08 ], [ -1.028 ], [ 1.764 ] ],
[ [ -1.72 ], [ -0.804 ], [ 1.912 ], [ -0.608 ] ],
[ [ 1.208 ], [ -0.852 ], [ -0.384 ], [ 1.524 ] ]
]
[
[ [ -0.804 ], [ -1.688 ], [ 1.764 ], [ -0.852 ] ],
[ [ -1.028 ], [ 0.08 ], [ 0.7 ], [ -0.128 ] ],
[ [ 0.496 ], [ 1.912 ], [ -0.384 ], [ 1.208 ] ],
[ [ 1.524 ], [ -0.608 ], [ -1.72 ], [ 0.048 ] ]
]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.43 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2315140653493
Reset training subject: 2315198879239
Constructing line search parameters: GD
th(0)=4.2951576;dx=-0.85903152
New Minimum: 4.2951576 > 2.6437938490288526
END: th(2.154434690031884)=2.6437938490288526; dx=-0.6739587893481183 evalInputDelta=1.651363750971147
Fitness changed from 4.2951576 to 2.6437938490288526
Iteration 1 complete. Error: 2.6437938490288526 Total: 0.1469; Orientation: 0.0049; Line Search: 0.0498
th(0)=2.6437938490288526;dx=-0.5287587698057706
New Minimum: 2.6437938490288526 > 0.7591011655876259
END: th(4.641588833612779)=0.7591011655876259; dx=-0.2833306896452411 evalInputDelta=1.8846926834412268
Fitness changed from 2.6437938490288526 to 0.7591011655876259
Iteration 2 complete. Error: 0.7591011655876259 Total: 0.0710; Orientation: 0.0016; Line Search: 0.0517
th(0)=0.7591011655876259;dx=-0.15182023311752518
New Minimum: 0.7591011655876259 > 6.190901806232769E-32
WOLF (strong): th(10.000000000000002)=6.190901806232769E-32; dx=3.9775636039460573E-17 evalInputDelta=0.7591011655876259
END: th(5.000000000000001)=0.18977529139690635; dx=-0.07591011655876258 evalInputDelta=0.5693258741907194
Fitness changed from 0.7591011655876259 to 6.190901806232769E-32
Iteration 3 complete. Error: 6.190901806232769E-32 Total: 0.0751; Orientation: 0.0020; Line Search: 0.0562
Zero gradient: 1.112735530683978E-16
th(0)=6.190901806232769E-32;dx=-1.2381803612465543E-32
New Minimum: 6.190901806232769E-32 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=6.190901806232769E-32
Fitness changed from 6.190901806232769E-32 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.0637; Orientation: 0.0018; Line Search: 0.0407
Zero gradient: 0.0
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0635; Orientation: 0.0014; Line Search: 0.0437
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.422s (< 30.000s)
Returns
0.0
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.48 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2315568362424
Reset training subject: 2315584995806
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=4.2951576}, derivative=-0.85903152}
New Minimum: 4.2951576 > 4.295157599914097
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=4.295157599914097}, derivative=-0.8590315199914098}, evalInputDelta = -8.590284039655671E-11
New Minimum: 4.295157599914097 > 4.295157599398678
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=4.295157599398678}, derivative=-0.8590315199398678}, evalInputDelta = -6.013216591327364E-10
New Minimum: 4.295157599398678 > 4.295157595790745
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=4.295157595790745}, derivative=-0.8590315195790746}, evalInputDelta = -4.209254278464414E-9
New Minimum: 4.295157595790745 > 4.295157570535219
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=4.295157570535219}, derivative=-0.8590315170535219}, evalInputDelta = -2.9464780837429316E-8
New Minimum: 4.295157570535219 > 4.295157393746534
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=4.295157393746534}, derivative=-0.8590314993746533}, evalInputDelta = -2.0625346586200521E-7
New Minimum: 4.295157393746534 > 4.295156156225846
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=4.295156156225846}, derivative=-0.8590313756225725}, evalInputDelta = -1.4437741535644477E-6
New Minimum: 4.295156156225846 > 4.295147493586015
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=4.295147493586015}, derivative=-0.8590305093580071}, evalInputDelta = -1.010641398480061E-5
New Minimum: 4.295147493586015 > 4.2950868553518
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=4.2950868553518}, derivative=-0.8590244455060493}, evalInputDelta = -7.074464820000514E-5
New Minimum: 4.2950868553518 > 4.294662399697514
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=4.294662399697514}, derivative=-0.8589819985423448}, evalInputDelta = -4.952003024856211E-4
New Minimum: 4.294662399697514 > 4.291691797393435
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=4.291691797393435}, derivative=-0.8586848697964131}, evalInputDelta = -0.003465802606564594
New Minimum: 4.291691797393435 > 4.270926357784816
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=4.270926357784816}, derivative=-0.8566049685748915}, evalInputDelta = -0.024231242215183357
New Minimum: 4.270926357784816 > 4.126978330001552
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=4.126978330001552}, derivative=-0.8420456600242406}, evalInputDelta = -0.1681792699984479
New Minimum: 4.126978330001552 > 3.1884345598950024
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=3.1884345598950024}, derivative=-0.7401305001696843}, evalInputDelta = -1.1067230401049972
New Minimum: 3.1884345598950024 > 0.004156963587728905
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=0.004156963587728905}, derivative=-0.02672438118778961}, evalInputDelta = -4.291000636412271
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=143.60512471692707}, derivative=4.967118451685473}, evalInputDelta = 139.30996711692706
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=0.9825657095542624}, derivative=-0.41086613756265594}, evalInputDelta = -3.312591890445737
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=30.207615591843364}, derivative=2.2781261570614086}, evalInputDelta = 25.912457991843365
F(2.809207994049113) = LineSearchPoint{point=PointSample{avg=2.2209181724525107}, derivative=-0.617711698687584}, evalInputDelta = -2.074239427547489
F(19.66445595834379) = LineSearchPoint{point=PointSample{avg=4.011750601387772}, derivative=0.8302072291869124}, evalInputDelta = -0.28340699861222785
4.011750601387772 <= 4.2951576
New Minimum: 0.004156963587728905 > 2.952450604745633E-32
F(10.0) = LineSearchPoint{point=PointSample{avg=2.952450604745633E-32}, derivative=3.609335053056387E-18}, evalInputDelta = -4.2951576
Right bracket at 10.0
Converged to right
Fitness changed from 4.2951576 to 2.952450604745633E-32
Iteration 1 complete. Error: 2.952450604745633E-32 Total: 0.4175; Orientation: 0.0012; Line Search: 0.3743
Zero gradient: 7.684335501194145E-17
F(0.0) = LineSearchPoint{point=PointSample{avg=2.952450604745633E-32}, derivative=-5.9049012094912665E-33}
New Minimum: 2.952450604745633E-32 > 0.0
F(10.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}, evalInputDelta = -2.952450604745633E-32
0.0 <= 2.952450604745633E-32
Converged to right
Fitness changed from 2.952450604745633E-32 to 0.0
Iteration 2 complete. Error: 0.0 Total: 0.0411; Orientation: 0.0009; Line Search: 0.0321
Zero gradient: 0.0
F(0.0) = LineSearchPoint{point=PointSample{avg=0.0}, derivative=0.0}
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.0209; Orientation: 0.0013; Line Search: 0.0100
Iteration 3 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 3
Final threshold in iteration 3: 0.0 (> 0.0) after 0.480s (< 30.000s)
Returns
0.0
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.43 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2316053421339
Reset training subject: 2316065457456
Adding measurement 6a8d33a to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 4.2951576 < 4.2951576. Total: 1
th(0)=4.2951576;dx=-0.85903152
Adding measurement 58e239c2 to history. Total: 1
New Minimum: 4.2951576 > 2.6437938490288526
END: th(2.154434690031884)=2.6437938490288526; dx=-0.6739587893481183 evalInputDelta=1.651363750971147
Fitness changed from 4.2951576 to 2.6437938490288526
Iteration 1 complete. Error: 2.6437938490288526 Total: 0.0743; Orientation: 0.0083; Line Search: 0.0304
Non-optimal measurement 2.6437938490288526 < 2.6437938490288526. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 2.6437938490288526 < 2.6437938490288526. Total: 2
th(0)=2.6437938490288526;dx=-0.5287587698057706
Adding measurement 59c63737 to history. Total: 2
New Minimum: 2.6437938490288526 > 0.7591011655876259
END: th(4.641588833612779)=0.7591011655876259; dx=-0.2833306896452411 evalInputDelta=1.8846926834412268
Fitness changed from 2.6437938490288526 to 0.7591011655876259
Iteration 2 complete. Error: 0.7591011655876259 Total: 0.0336; Orientation: 0.0026; Line Search: 0.0218
Non-optimal measurement 0.7591011655876259 < 0.7591011655876259. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 0.7591011655876259 < 0.7591011655876259. Total: 3
th(0)=0.7591011655876259;dx=-0.15182023311752518
Adding measurement 6659d4dc to history. Total: 3
New Minimum: 0.7591011655876259 > 6.190901806232769E-32
WOLF (strong): th(10.000000000000002)=6.190901806232769E-32; dx=3.9775636039460573E-17 evalInputDelta=0.7591011655876259
Non-optimal measurement 0.18977529139690635 < 6.190901806232769E-32. Total: 4
END: th(5.000000000000001)=0.18977529139690635; dx=-0.07591011655876256 evalInputDelta=0.5693258741907194
Fitness changed from 0.7591011655876259 to 6.190901806232769E-32
Iteration 3 complete. Error: 6.190901806232769E-32 Total: 0.0576; Orientation: 0.0031; Line Search: 0.0466
Non-optimal measurement 6.190901806232769E-32 < 6.190901806232769E-32. Total: 4
Rejected: LBFGS Orientation magnitude: 1.113e-15, gradient 1.113e-16, dot -1.000; [aee0485a-f877-4bec-b307-e5263d14c1a4 = 1.000/1.000e+00, 61932f06-defe-4946-9136-1d725c675eb8 = 1.000/1.000e+00, f45ab9ae-5245-4f84-bc69-6bc5c06e4941 = 1.000/1.000e+00, 0235b079-83f8-416f-a5cf-ee53a089eb6e = 1.000/1.000e+00, b02299dc-f13c-4706-ad21-a5ed53a65e4b = 1.000/1.000e+00]
Orientation rejected. Popping history element from 6.190901806232769E-32, 0.7591011655876259, 2.6437938490288526, 4.2951576
LBFGS Accumulation History: 3 points
Removed measurement 6659d4dc to history. Total: 3
Adding measurement 3eb5898 to history. Total: 3
th(0)=6.190901806232769E-32;dx=-1.2381803612465543E-32
Adding measurement 6467ca17 to history. Total: 4
New Minimum: 6.190901806232769E-32 > 0.0
END: th(10.772173450159421)=0.0; dx=0.0 evalInputDelta=6.190901806232769E-32
Fitness changed from 6.190901806232769E-32 to 0.0
Iteration 4 complete. Error: 0.0 Total: 0.1252; Orientation: 0.0923; Line Search: 0.0248
Non-optimal measurement 0.0 < 0.0. Total: 5
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [61932f06-defe-4946-9136-1d725c675eb8 = 0.000e+00, aee0485a-f877-4bec-b307-e5263d14c1a4 = 0.000e+00, f45ab9ae-5245-4f84-bc69-6bc5c06e4941 = 0.000e+00, 0235b079-83f8-416f-a5cf-ee53a089eb6e = 0.000e+00, b02299dc-f13c-4706-ad21-a5ed53a65e4b = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 6.190901806232769E-32, 0.7591011655876259, 2.6437938490288526, 4.2951576
Rejected: LBFGS Orientation magnitude: 0.000e+00, gradient 0.000e+00, dot NaN; [f45ab9ae-5245-4f84-bc69-6bc5c06e4941 = 0.000e+00, 0235b079-83f8-416f-a5cf-ee53a089eb6e = 0.000e+00, b02299dc-f13c-4706-ad21-a5ed53a65e4b = 0.000e+00, 61932f06-defe-4946-9136-1d725c675eb8 = 0.000e+00, aee0485a-f877-4bec-b307-e5263d14c1a4 = 0.000e+00]
Orientation rejected. Popping history element from 0.0, 6.190901806232769E-32, 0.7591011655876259, 2.6437938490288526
LBFGS Accumulation History: 3 points
Removed measurement 6467ca17 to history. Total: 4
Removed measurement 3eb5898 to history. Total: 3
Adding measurement 5eeeee62 to history. Total: 3
th(0)=0.0;dx=0.0 (ERROR: Starting derivative negative)
Non-optimal measurement 0.0 < 0.0. Total: 4
Fitness changed from 0.0 to 0.0
Static Iteration Total: 0.1405; Orientation: 0.1150; Line Search: 0.0183
Iteration 5 failed. Error: 0.0
Previous Error: 0.0 -> 0.0
Optimization terminated 5
Final threshold in iteration 5: 0.0 (> 0.0) after 0.432s (< 30.000s)
Returns
0.0
TrainingTester.java:432 executed in 0.14 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, -31.52981735951152], [4.0, 0.4222275878333845]; valueStats=DoubleSummaryStatistics{count=7, sum=6.805790, min=0.000000, average=0.972256, max=2.643794}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, -31.52981735951152], [0.217, 0.4222275878333845]; valueStats=DoubleSummaryStatistics{count=7, sum=6.805790, min=0.000000, average=0.972256, max=2.643794}
Plotting 4 points for GD
Plotting 2 points for CjGD
Plotting 4 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "Converged", "value": 0.0 }, "CjGD": { "type": "Converged", "value": 0.0 }, "GD": { "type": "Converged", "value": 0.0 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "2.278",
"gc_time": "0.265"
},
"created_on": 1586736946754,
"file_name": "trainingTest",
"report": {
"simpleName": "Center",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgCropLayerTest.Center",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/test/java/com/simiacryptus/mindseye/layers/cudnn/ImgCropLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "Converged",
"value": 0.0
},
"CjGD": {
"type": "Converged",
"value": 0.0
},
"GD": {
"type": "Converged",
"value": 0.0
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/cudnn/ImgCropLayer/Center/trainingTest/202004131546",
"id": "dc1484eb-1d0f-4e69-9da1-3fc4748b6010",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "ImgCropLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.cudnn.ImgCropLayer",
"link": "https://github.com/SimiaCryptus/mindseye-cudnn/tree/59d5b3318556370acb2d83ee6ec123ce0fc6974f/src/main/java/com/simiacryptus/mindseye/layers/cudnn/ImgCropLayer.java",
"javaDoc": ""
}
}