Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
Using Seed 1311434486519769088
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ 0.08, 0.496, 0.7, -0.128 ]
[ 0.08, 0.7, -0.128, 0.496 ]
[ 0.496, -0.128, 0.08, 0.7 ]
[ 0.496, -0.128, 0.7, 0.08 ]
[ 0.7, 0.08, 0.496, -0.128 ]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.16 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 3186509605206
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 3186537426990
Constructing line search parameters: GD
th(0)=9.567103787899327;dx=-7.411983653137287E21
New Minimum: 9.567103787899327 > 9.261034085383539
Armijo: th(2.154434690031884)=9.261034085383539; dx=4.148201762402486E32 evalInputDelta=0.3060697025157886
Armijo: th(1.077217345015942)=9.261034085383539; dx=2.074100881164183E32 evalInputDelta=0.3060697025157886
Armijo: th(0.3590724483386473)=9.261034085383539; dx=6.913669603386479E31 evalInputDelta=0.3060697025157886
Armijo: th(0.08976811208466183)=9.261034085383539; dx=1.7284174002907208E31 evalInputDelta=0.3060697025157886
Armijo: th(0.017953622416932366)=9.261034085383539; dx=3.4568347946518544E30 evalInputDelta=0.3060697025157886
Armijo: th(0.002992270402822061)=9.261034085383539; dx=5.761391262653227E29 evalInputDelta=0.3060697025157886
Armijo: th(4.2746720040315154E-4)=9.261034085383539; dx=8.230558311334581E28 evalInputDelta=0.3060697025157886
Armijo: th(5.343340005039394E-5)=9.261034085383539; dx=1.0288191403682532E28 evalInputDelta=0.3060697025157886
Armijo: th(5.9370444500437714E-6)=9.261034085383539; dx=1.143125789757034E27 evalInputDelta=0.3060697025157886
Armijo: th(5.937044450043771E-7)=9.261034085383539; dx=1.1430590819041556E26 evalInputDelta=0.3060697025157886
Armijo: th(5.397313136403428E-8)=9.261034085383539; dx=1.0384708032171289E25 evalInputDelta=0.3060697025157886
Armijo: th(4.4977609470028565E-9)=9.261034085383539; dx=8.585980176655625E23 evalInputDelta=0.3060697025157886
Armijo: th(3.4598161130791205E-10)=11.202114236791562; dx=5.8627343090358115E22 evalInputDelta=-1.6350104488922348
New Minimum: 9.261034085383539 > 2.826056851623167
Armijo: th(2.4712972236279432E-11)=2.826056851623167; dx=2.2910921155719784E20 evalInputDelta=6.7410469362761605
Armijo: th(1.6475314824186289E-12)=9.571598384662197; dx=-7.094763872434233E21 evalInputDelta=-0.004494596762869563
Armijo: th(1.029707176511643E-13)=9.5673740853955; dx=-7.392157416843349E21 evalInputDelta=-2.7029749617213383E-4
Armijo: th(6.057101038303783E-15)=9.567119652759008; dx=-7.410817403943527E21 evalInputDelta=-1.5864859680192467E-5
MIN ALPHA (3.3650561323909904E-16): th(2.4712972236279432E-11)=2.826056851623167
Fitness changed from 9.567103787899327 to 2.826056851623167
Iteration 1 complete. Error: 2.826056851623167 Total: 0.1309; Orientation: 0.0030; Line Search: 0.0911
th(0)=2.826056851623167;dx=-2.866265820009901E20
Armijo: th(2.154434690031884E-15)=2.826058506325309; dx=-2.8658591927528997E20 evalInputDelta=-1.6547021419732744E-6
Armijo: th(1.077217345015942E-15)=2.826057678945749; dx=-2.8660625063814E20 evalInputDelta=-8.273225819976915E-7
MIN ALPHA (3.5907244833864734E-16): th(0.0)=2.826056851623167
Fitness changed from 2.826056851623167 to 2.826056851623167
Static Iteration Total: 0.0200; Orientation: 0.0010; Line Search: 0.0162
Iteration 2 failed. Error: 2.826056851623167
Previous Error: 0.0 -> 2.826056851623167
Optimization terminated 2
Final threshold in iteration 2: 2.826056851623167 (> 0.0) after 0.151s (< 30.000s)
Returns
2.826056851623167
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ 0.4605714830024652, -0.13965680538471564, 0.146571006733959, 0.7 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ -0.06432194196742515, 0.06750642593662382, 0.32240003810172563, -0.020469638562486358, -0.09775976376930094, 0.1025997047137713 ]
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.01 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 3186674147182
Reset training subject: 3186675783659
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=9.567103787899327}, derivative=-7.411983653137287E21}
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=10.865010509797788}, derivative=1.2237318338731724E22}, evalInputDelta = 1.2979067218984603
10.865010509797788 <= 9.567103787899327
Converged to right
Fitness changed from 9.567103787899327 to 9.567103787899327
Static Iteration Total: 0.0124; Orientation: 0.0006; Line Search: 0.0072
Iteration 1 failed. Error: 9.567103787899327
Previous Error: 0.0 -> 9.567103787899327
Optimization terminated 1
Final threshold in iteration 1: 9.567103787899327 (> 0.0) after 0.013s (< 30.000s)
Returns
9.567103787899327
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ 0.496, 0.08, -0.128, 0.7 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ 0.03968, -0.063488, 0.34719999999999995, -0.01024, 0.055999999999999994, -0.0896 ]
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.09 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 3186695776185
Reset training subject: 3186697961690
Adding measurement 1e010560 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 9.567103787899327 < 9.567103787899327. Total: 1
th(0)=9.567103787899327;dx=-7.411983653137287E21
Adding measurement 250c60e4 to history. Total: 1
New Minimum: 9.567103787899327 > 9.261034085383539
Armijo: th(2.154434690031884)=9.261034085383539; dx=4.148201762402486E32 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(1.077217345015942)=9.261034085383539; dx=2.074100881164183E32 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(0.3590724483386473)=9.261034085383539; dx=6.913669603386479E31 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(0.08976811208466183)=9.261034085383539; dx=1.7284174002907208E31 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(0.017953622416932366)=9.261034085383539; dx=3.4568347946518544E30 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(0.002992270402822061)=9.261034085383539; dx=5.761391262653227E29 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(4.2746720040315154E-4)=9.261034085383539; dx=8.230558311334581E28 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(5.343340005039394E-5)=9.261034085383539; dx=1.0288191403682532E28 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(5.9370444500437714E-6)=9.261034085383539; dx=1.143125789757034E27 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(5.937044450043771E-7)=9.261034085383539; dx=1.1430590819041556E26 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(5.397313136403428E-8)=9.261034085383539; dx=1.0384708032171286E25 evalInputDelta=0.3060697025157886
Non-optimal measurement 9.261034085383539 < 9.261034085383539. Total: 2
Armijo: th(4.4977609470028565E-9)=9.261034085383539; dx=8.585980176655625E23 evalInputDelta=0.3060697025157886
Non-optimal measurement 11.202114236791562 < 9.261034085383539. Total: 2
Armijo: th(3.4598161130791205E-10)=11.202114236791562; dx=5.8627343090358115E22 evalInputDelta=-1.6350104488922348
Adding measurement 5f9e8f18 to history. Total: 2
New Minimum: 9.261034085383539 > 2.826056851623167
Armijo: th(2.4712972236279432E-11)=2.826056851623167; dx=2.2910921155719784E20 evalInputDelta=6.7410469362761605
Non-optimal measurement 9.571598384662197 < 2.826056851623167. Total: 3
Armijo: th(1.6475314824186289E-12)=9.571598384662197; dx=-7.094763872434233E21 evalInputDelta=-0.004494596762869563
Non-optimal measurement 9.5673740853955 < 2.826056851623167. Total: 3
Armijo: th(1.029707176511643E-13)=9.5673740853955; dx=-7.392157416843349E21 evalInputDelta=-2.7029749617213383E-4
Non-optimal measurement 9.567119652759008 < 2.826056851623167. Total: 3
Armijo: th(6.057101038303783E-15)=9.567119652759008; dx=-7.410817403943527E21 evalInputDelta=-1.5864859680192467E-5
Non-optimal measurement 2.826056851623167 < 2.826056851623167. Total: 3
MIN ALPHA (3.3650561323909904E-16): th(2.4712972236279432E-11)=2.826056851623167
Fitness changed from 9.567103787899327 to 2.826056851623167
Iteration 1 complete. Error: 2.826056851623167 Total: 0.0748; Orientation: 0.0037; Line Search: 0.0649
Non-optimal measurement 2.826056851623167 < 2.826056851623167. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 2.826056851623167 < 2.826056851623167. Total: 3
th(0)=2.826056851623167;dx=-2.866265820009901E20
Non-optimal measurement 2.826058506325309 < 2.826056851623167. Total: 3
Armijo: th(2.154434690031884E-15)=2.826058506325309; dx=-2.8658591927528997E20 evalInputDelta=-1.6547021419732744E-6
Non-optimal measurement 2.826057678945749 < 2.826056851623167. Total: 3
Armijo: th(1.077217345015942E-15)=2.826057678945749; dx=-2.8660625063814E20 evalInputDelta=-8.273225819976915E-7
Non-optimal measurement 2.826056851623167 < 2.826056851623167. Total: 3
MIN ALPHA (3.5907244833864734E-16): th(0.0)=2.826056851623167
Fitness changed from 2.826056851623167 to 2.826056851623167
Static Iteration Total: 0.0138; Orientation: 0.0014; Line Search: 0.0106
Iteration 2 failed. Error: 2.826056851623167
Previous Error: 0.0 -> 2.826056851623167
Optimization terminated 2
Final threshold in iteration 2: 2.826056851623167 (> 0.0) after 0.089s (< 30.000s)
Returns
2.826056851623167
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ 0.4605714830024652, -0.13965680538471564, 0.146571006733959, 0.7 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ -0.06432194196742515, 0.06750642593662382, 0.32240003810172563, -0.020469638562486358, -0.09775976376930094, 0.1025997047137713 ]
TrainingTester.java:432 executed in 0.12 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[0.0, -0.5488191057224638], [2.0, 1.4511808942775362]; valueStats=DoubleSummaryStatistics{count=2, sum=5.652114, min=2.826057, average=2.826057, max=2.826057}
Only 1 points for GD
Only 1 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[-1.0, -0.5488191057224638], [1.0, 1.4511808942775362]; valueStats=DoubleSummaryStatistics{count=2, sum=5.652114, min=2.826057, average=2.826057, max=2.826057}
Only 1 points for GD
Only 0 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "NonConverged", "value": 2.826056851623167 }, "CjGD": { "type": "NonConverged", "value": NaN }, "GD": { "type": "NonConverged", "value": 2.826056851623167 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
details | result |
---|---|
{"input":{ "LBFGS": { "type": "NonConverged", "value": 2.826056851623167 }, "CjGD": { "type": "NonConverged", "value": NaN }, "GD": { "type": "NonConverged", "value": 2.826056851623167 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "0.869",
"gc_time": "0.197"
},
"created_on": 1586737774314,
"file_name": "trainingTest",
"report": {
"simpleName": "Basic",
"canonicalName": "com.simiacryptus.mindseye.layers.java.CrossProductLayerTest.Basic",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/CrossProductLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "NonConverged",
"value": 2.826056851623167
},
"CjGD": {
"type": "NonConverged",
"value": "NaN"
},
"GD": {
"type": "NonConverged",
"value": 2.826056851623167
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/CrossProductLayer/Basic/trainingTest/202004132934",
"id": "eadf93b3-2578-4ff3-a884-b461fd2212e3",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "CrossProductLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.java.CrossProductLayer",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/CrossProductLayer.java",
"javaDoc": ""
}
}