Subreport: Logs for com.simiacryptus.ref.lang.ReferenceCountingBase
Using Seed 8562658766145657856
In this apply, we use a network to learn this target input, given it's pre-evaluated output:
TrainingTester.java:332 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(input_target)).flatMap(RefArrays::stream).map(x -> {
try {
return x.prettyPrint();
} finally {
x.freeRef();
}
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ -0.128, 0.08, 0.7, 0.496 ]
[ 0.7, 0.08, 0.496, -0.128 ]
[ -0.128, 0.496, 0.08, 0.7 ]
[ 0.496, -0.128, 0.7, 0.08 ]
[ -0.128, 0.496, 0.7, 0.08 ]
First, we train using basic gradient descent method apply weak line search conditions.
TrainingTester.java:480 executed in 0.37 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2607955219966
BACKPROP_AGG_SIZE = 3
THREADS = 64
SINGLE_THREADED = false
Initialized CoreSettings = {
"backpropAggregationSize" : 3,
"jvmThreads" : 64,
"singleThreaded" : false
}
Reset training subject: 2607992655438
Constructing line search parameters: GD
th(0)=1.4410043780930528;dx=-0.010947934512708877
New Minimum: 1.4410043780930528 > 1.418813461440417
END: th(2.154434690031884)=1.418813461440417; dx=-0.009654902395848403 evalInputDelta=0.022190916652635728
Fitness changed from 1.4410043780930528 to 1.418813461440417
Iteration 1 complete. Error: 1.418813461440417 Total: 0.0735; Orientation: 0.0030; Line Search: 0.0170
th(0)=1.418813461440417;dx=-0.00851982260347121
New Minimum: 1.418813461440417 > 1.3841826791867486
END: th(4.641588833612779)=1.3841826791867486; dx=-0.006412407922864998 evalInputDelta=0.034630782253668446
Fitness changed from 1.418813461440417 to 1.3841826791867486
Iteration 2 complete. Error: 1.3841826791867486 Total: 0.0187; Orientation: 0.0012; Line Search: 0.0125
th(0)=1.3841826791867486;dx=-0.004836597402970378
New Minimum: 1.3841826791867486 > 1.348158768808661
END: th(10.000000000000002)=1.348158768808661; dx=-0.002392378802878564 evalInputDelta=0.03602391037808772
Fitness changed from 1.3841826791867486 to 1.348158768808661
Iteration 3 complete. Error: 1.348158768808661 Total: 0.0189; Orientation: 0.0013; Line Search: 0.0109
th(0)=1.348158768808661;dx=-0.0011942631090930134
New Minimum: 1.348158768808661 > 1.335374593042268
END: th(21.544346900318843)=1.335374593042268; dx=-1.0570420357761426E-5 evalInputDelta=0.012784175766392902
Fitness changed from 1.348158768808661 to 1.335374593042268
Iteration 4 complete. Error: 1.335374593042268 Total: 0.0204; Orientation: 0.0010; Line Search: 0.0159
th(0)=1.335374593042268;dx=-4.061904484286916E-5
Armijo: th(46.4158883361278)=1.3358287301773089; dx=5.979188543035564E-5 evalInputDelta=-4.54137135040833E-4
New Minimum: 1.335374593042268 > 1.3350189306891795
WOLF (strong): th(23.2079441680639)=1.3350189306891795; dx=9.882988931102869E-6 evalInputDelta=3.556623530884817E-4
END: th(7.735981389354634)=1.3351257298933505; dx=-2.372866580140759E-5 evalInputDelta=2.488631489174775E-4
Fitness changed from 1.335374593042268 to 1.3350189306891795
Iteration 5 complete. Error: 1.3350189306891795 Total: 0.0331; Orientation: 0.0010; Line Search: 0.0284
th(0)=1.3350189306891795;dx=-7.248526456593706E-6
New Minimum: 1.3350189306891795 > 1.3349608846598973
WOLF (strong): th(16.66666666666667)=1.3349608846598973; dx=2.873733471412577E-7 evalInputDelta=5.804602928227709E-5
END: th(8.333333333333336)=1.334974207753929; dx=-3.4838543875697025E-6 evalInputDelta=4.472293525048521E-5
Fitness changed from 1.3350189306891795 to 1.3349608846598973
Iteration 6 complete. Error: 1.3349608846598973 Total: 0.0235; Orientation: 0.0008; Line Search: 0.0207
th(0)=1.3349608846598973;dx=-1.6571597496874236E-7
New Minimum: 1.3349608846598973 > 1.3349590545784682
END: th(17.95362241693237)=1.3349590545784682; dx=-3.8166997039698756E-8 evalInputDelta=1.830081429066155E-6
Fitness changed from 1.3349608846598973 to 1.3349590545784682
Iteration 7 complete. Error: 1.3349590545784682 Total: 0.0180; Orientation: 0.0009; Line Search: 0.0143
th(0)=1.3349590545784682;dx=-1.9450977404921913E-8
New Minimum: 1.3349590545784682 > 1.334958827097895
WOLF (strong): th(38.679906946773166)=1.334958827097895; dx=7.683460614246735E-9 evalInputDelta=2.274805732671581E-7
New Minimum: 1.334958827097895 > 1.33495880964349
END: th(19.339953473386583)=1.33495880964349; dx=-5.879777496794782E-9 evalInputDelta=2.4493497829958244E-7
Fitness changed from 1.3349590545784682 to 1.33495880964349
Iteration 8 complete. Error: 1.33495880964349 Total: 0.0168; Orientation: 0.0009; Line Search: 0.0120
th(0)=1.33495880964349;dx=-2.245710880293941E-9
New Minimum: 1.33495880964349 > 1.3349587845809432
WOLF (strong): th(41.66666666666668)=1.3349587845809432; dx=1.0424685435424332E-9 evalInputDelta=2.5062546704646138E-8
New Minimum: 1.3349587845809432 > 1.334958779986282
END: th(20.83333333333334)=1.334958779986282; dx=-6.014410972108617E-10 evalInputDelta=2.9657207800681817E-8
Fitness changed from 1.33495880964349 to 1.334958779986282
Iteration 9 complete. Error: 1.334958779986282 Total: 0.0153; Orientation: 0.0007; Line Search: 0.0118
th(0)=1.334958779986282;dx=-1.9916990149287806E-10
New Minimum: 1.334958779986282 > 1.3349587782124557
WOLF (strong): th(44.884056042330926)=1.3349587782124557; dx=1.2012198871405612E-10 evalInputDelta=1.7738264190825248E-9
New Minimum: 1.3349587782124557 > 1.3349587773079796
END: th(22.442028021165463)=1.3349587773079796; dx=-3.9518312514211326E-11 evalInputDelta=2.6783024686949375E-9
Fitness changed from 1.334958779986282 to 1.3349587773079796
Iteration 10 complete. Error: 1.3349587773079796 Total: 0.0216; Orientation: 0.0008; Line Search: 0.0181
Low gradient: 3.6605039131792775E-6
th(0)=1.3349587773079796;dx=-1.3399288898400806E-11
New Minimum: 1.3349587773079796 > 1.3349587772838742
WOLF (strong): th(48.349883683466466)=1.3349587772838742; dx=1.2402065067679963E-11 evalInputDelta=2.4105384355266324E-11
New Minimum: 1.3349587772838742 > 1.3349587771399904
END: th(24.174941841733233)=1.3349587771399904; dx=-4.985305092068398E-13 evalInputDelta=1.6798917812366199E-10
Fitness changed from 1.3349587773079796 to 1.3349587771399904
Iteration 11 complete. Error: 1.3349587771399904 Total: 0.0255; Orientation: 0.0010; Line Search: 0.0219
Low gradient: 1.0991801852582167E-6
th(0)=1.3349587771399904;dx=-1.2081970796642875E-12
Armijo: th(52.08333333333336)=1.3349587771686173; dx=2.3074795374593114E-12 evalInputDelta=-2.8626878645354736E-11
New Minimum: 1.3349587771399904 > 1.3349587771314155
WOLF (strong): th(26.04166666666668)=1.3349587771314155; dx=5.496404347464484E-13 evalInputDelta=8.574918552994859E-12
END: th(8.680555555555559)=1.3349587771320457; dx=-6.222514180212087E-13 evalInputDelta=7.94475596421762E-12
Fitness changed from 1.3349587771399904 to 1.3349587771314155
Iteration 12 complete. Error: 1.3349587771314155 Total: 0.0135; Orientation: 0.0008; Line Search: 0.0105
Low gradient: 6.467417357749068E-7
th(0)=1.3349587771314155;dx=-4.1827487279313936E-13
New Minimum: 1.3349587771314155 > 1.3349587771283333
WOLF (strong): th(18.701690017637887)=1.3349587771283333; dx=8.867688458119496E-14 evalInputDelta=3.0822011609643596E-12
END: th(9.350845008818943)=1.3349587771286893; dx=-1.6479895145698165E-13 evalInputDelta=2.7262636592695344E-12
Fitness changed from 1.3349587771314155 to 1.3349587771283333
Iteration 13 complete. Error: 1.3349587771283333 Total: 0.0110; Orientation: 0.0008; Line Search: 0.0076
Low gradient: 1.4325899307229908E-7
th(0)=1.3349587771283333;dx=-2.052313909608904E-14
New Minimum: 1.3349587771283333 > 1.334958777128187
WOLF (strong): th(20.14578486811103)=1.334958777128187; dx=5.996850334780661E-15 evalInputDelta=1.4632739464559563E-13
END: th(10.072892434055515)=1.3349587771281937; dx=-7.263145012032452E-15 evalInputDelta=1.396660564978447E-13
Fitness changed from 1.3349587771283333 to 1.334958777128187
Iteration 14 complete. Error: 1.334958777128187 Total: 0.0105; Orientation: 0.0009; Line Search: 0.0077
Low gradient: 4.4901161316379076E-8
th(0)=1.334958777128187;dx=-2.0161142875594966E-15
New Minimum: 1.334958777128187 > 1.334958777128174
WOLF (strong): th(21.7013888888889)=1.334958777128174; dx=7.888262005057928E-16 evalInputDelta=1.3100631690576847E-14
New Minimum: 1.334958777128174 > 1.334958777128173
END: th(10.85069444444445)=1.334958777128173; dx=-6.136440256676807E-16 evalInputDelta=1.3988810110276972E-14
Fitness changed from 1.334958777128187 to 1.334958777128173
Iteration 15 complete. Error: 1.334958777128173 Total: 0.0114; Orientation: 0.0008; Line Search: 0.0084
Low gradient: 1.3947166222576362E-8
th(0)=1.334958777128173;dx=-1.9452344564017502E-16
New Minimum: 1.334958777128173 > 1.3349587771281715
WOLF (strong): th(23.377112522047362)=1.3349587771281715; dx=8.155581398415972E-17 evalInputDelta=1.5543122344752192E-15
END: th(11.688556261023681)=1.3349587771281715; dx=-5.648381545808909E-17 evalInputDelta=1.5543122344752192E-15
Fitness changed from 1.334958777128173 to 1.3349587771281715
Iteration 16 complete. Error: 1.3349587771281715 Total: 0.0091; Orientation: 0.0005; Line Search: 0.0072
Low gradient: 6.854302105566327E-9
th(0)=1.3349587771281715;dx=-4.698145735437098E-17
New Minimum: 1.3349587771281715 > 1.3349587771281712
WOLF (strong): th(25.18223108513879)=1.3349587771281712; dx=2.9183875447026415E-17 evalInputDelta=2.220446049250313E-16
END: th(12.591115542569394)=1.3349587771281712; dx=-8.898791038253689E-18 evalInputDelta=2.220446049250313E-16
Fitness changed from 1.3349587771281715 to 1.3349587771281712
Iteration 17 complete. Error: 1.3349587771281712 Total: 0.0131; Orientation: 0.0006; Line Search: 0.0102
Low gradient: 4.343408227886446E-9
th(0)=1.3349587771281712;dx=-1.886519503407168E-17
WOLF (strong): th(27.12673611111113)=1.3349587771281712; dx=1.447204312409627E-17 evalInputDelta=0.0
END: th(13.563368055555564)=1.3349587771281712; dx=-2.1965759502070755E-18 evalInputDelta=0.0
Fitness changed from 1.3349587771281712 to 1.3349587771281712
Static Iteration Total: 0.0088; Orientation: 0.0006; Line Search: 0.0064
Iteration 18 failed. Error: 1.3349587771281712
Previous Error: 0.0 -> 1.3349587771281712
Optimization terminated 18
Final threshold in iteration 18: 1.3349587771281712 (> 0.0) after 0.364s (< 30.000s)
Returns
1.3349587771281712
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ -0.12799999769777096, 0.07999999900021168, 0.6999999969320329, 0.49600000176552533 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ 0.1565844917827319, 0.19278888779342374, 0.358380668934368, 0.2922459514894764 ]
First, we use a conjugate gradient descent method, which converges the fastest for purely linear functions.
TrainingTester.java:452 executed in 0.31 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new QuadraticSearch());
iterativeTrainer.setOrientation(new GradientDescent());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2608333780963
Reset training subject: 2608335763859
Constructing line search parameters: GD
F(0.0) = LineSearchPoint{point=PointSample{avg=1.4410043780930528}, derivative=-0.010947934512708877}
New Minimum: 1.4410043780930528 > 1.441004378091958
F(1.0E-10) = LineSearchPoint{point=PointSample{avg=1.441004378091958}, derivative=-0.010947934512648515}, evalInputDelta = -1.0949019468853294E-12
New Minimum: 1.441004378091958 > 1.4410043780853892
F(7.000000000000001E-10) = LineSearchPoint{point=PointSample{avg=1.4410043780853892}, derivative=-0.010947934512286352}, evalInputDelta = -7.66364749438253E-12
New Minimum: 1.4410043780853892 > 1.441004378039408
F(4.900000000000001E-9) = LineSearchPoint{point=PointSample{avg=1.441004378039408}, derivative=-0.010947934509751215}, evalInputDelta = -5.364486632686294E-11
New Minimum: 1.441004378039408 > 1.4410043777175385
F(3.430000000000001E-8) = LineSearchPoint{point=PointSample{avg=1.4410043777175385}, derivative=-0.010947934492005247}, evalInputDelta = -3.755142863326455E-10
New Minimum: 1.4410043777175385 > 1.4410043754644537
F(2.4010000000000004E-7) = LineSearchPoint{point=PointSample{avg=1.4410043754644537}, derivative=-0.010947934367783466}, evalInputDelta = -2.6285991161500988E-9
New Minimum: 1.4410043754644537 > 1.4410043596928603
F(1.6807000000000003E-6) = LineSearchPoint{point=PointSample{avg=1.4410043596928603}, derivative=-0.010947933498231012}, evalInputDelta = -1.8400192480783062E-8
New Minimum: 1.4410043596928603 > 1.44100424929174
F(1.1764900000000001E-5) = LineSearchPoint{point=PointSample{avg=1.44100424929174}, derivative=-0.010947927411363984}, evalInputDelta = -1.2880131272652307E-7
New Minimum: 1.44100424929174 > 1.4410034764856166
F(8.235430000000001E-5) = LineSearchPoint{point=PointSample{avg=1.4410034764856166}, derivative=-0.010947884803302754}, evalInputDelta = -9.016074362655502E-7
New Minimum: 1.4410034764856166 > 1.440998066926968
F(5.764801000000001E-4) = LineSearchPoint{point=PointSample{avg=1.440998066926968}, derivative=-0.010947586547263962}, evalInputDelta = -6.311166084849162E-6
New Minimum: 1.440998066926968 > 1.4409602041429312
F(0.004035360700000001) = LineSearchPoint{point=PointSample{avg=1.4409602041429312}, derivative=-0.010945498774098308}, evalInputDelta = -4.417395012157321E-5
New Minimum: 1.4409602041429312 > 1.4406953668445959
F(0.028247524900000005) = LineSearchPoint{point=PointSample{avg=1.4406953668445959}, derivative=-0.0109308852998755}, evalInputDelta = -3.090112484569474E-4
New Minimum: 1.4406953668445959 > 1.4388514100140006
F(0.19773267430000002) = LineSearchPoint{point=PointSample{avg=1.4388514100140006}, derivative=-0.01082863753773983}, evalInputDelta = -0.0021529680790521866
New Minimum: 1.4388514100140006 > 1.426427904850544
F(1.3841287201) = LineSearchPoint{point=PointSample{avg=1.426427904850544}, derivative=-0.010115387528677291}, evalInputDelta = -0.01457647324250888
New Minimum: 1.426427904850544 > 1.3626492927831282
F(9.688901040700001) = LineSearchPoint{point=PointSample{avg=1.3626492927831282}, derivative=-0.005307466850348752}, evalInputDelta = -0.07835508530992463
F(67.8223072849) = LineSearchPoint{point=PointSample{avg=1.720440208785614}, derivative=0.013529049133231327}, evalInputDelta = 0.2794358306925613
F(5.217100560376924) = LineSearchPoint{point=PointSample{avg=1.392019727288731}, derivative=-0.007849452578931564}, evalInputDelta = -0.048984650804321905
F(36.51970392263847) = LineSearchPoint{point=PointSample{avg=1.3926963478020185}, derivative=0.006495954093395335}, evalInputDelta = -0.048308030291034365
1.3926963478020185 <= 1.4410043780930528
New Minimum: 1.3626492927831282 > 1.3378249508533755
F(22.920080264021777) = LineSearchPoint{point=PointSample{avg=1.3378249508533755}, derivative=0.0013061700814243028}, evalInputDelta = -0.10317942723967732
Right bracket at 22.920080264021777
New Minimum: 1.3378249508533755 > 1.3359743273068017
F(20.477019420633688) = LineSearchPoint{point=PointSample{avg=1.3359743273068017}, derivative=1.998197689669216E-4}, evalInputDelta = -0.10503005078625116
Right bracket at 20.477019420633688
New Minimum: 1.3359743273068017 > 1.3359323082896668
F(20.10997569268859) = LineSearchPoint{point=PointSample{avg=1.3359323082896668}, derivative=2.8937008286021098E-5}, evalInputDelta = -0.10507206980338601
Right bracket at 20.10997569268859
Converged to right
Fitness changed from 1.4410043780930528 to 1.3359323082896668
Iteration 1 complete. Error: 1.3359323082896668 Total: 0.0633; Orientation: 0.0007; Line Search: 0.0576
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3359323082896668}, derivative=-1.0386898005989995E-4}
New Minimum: 1.3359323082896668 > 1.3350378192592491
F(20.10997569268859) = LineSearchPoint{point=PointSample{avg=1.3350378192592491}, derivative=1.4942657285881674E-5}, evalInputDelta = -8.944890304176667E-4
1.3350378192592491 <= 1.3359323082896668
New Minimum: 1.3350378192592491 > 1.335018931716966
F(17.580791839016822) = LineSearchPoint{point=PointSample{avg=1.335018931716966}, derivative=-7.106624802392191E-9}, evalInputDelta = -9.133765727007415E-4
Left bracket at 17.580791839016822
Converged to left
Fitness changed from 1.3359323082896668 to 1.335018931716966
Iteration 2 complete. Error: 1.335018931716966 Total: 0.0101; Orientation: 0.0004; Line Search: 0.0084
F(0.0) = LineSearchPoint{point=PointSample{avg=1.335018931716966}, derivative=-5.0382891658547775E-6}
New Minimum: 1.335018931716966 > 1.3349657174842497
F(17.580791839016822) = LineSearchPoint{point=PointSample{avg=1.3349657174842497}, derivative=-1.0163527371166264E-6}, evalInputDelta = -5.3214232716314314E-5
F(123.06554287311775) = LineSearchPoint{point=PointSample{avg=1.3361262880318387}, derivative=2.2979617096482676E-5}, evalInputDelta = 0.0011073563148726873
F(9.466580221009057) = LineSearchPoint{point=PointSample{avg=1.3349814917428966}, derivative=-2.8719124697582645E-6}, evalInputDelta = -3.74399740694642E-5
F(66.2660615470634) = LineSearchPoint{point=PointSample{avg=1.3351867767135168}, derivative=1.0089361838853284E-5}, evalInputDelta = 1.678449965507589E-4
F(5.097389349774108) = LineSearchPoint{point=PointSample{avg=1.334996223401094}, derivative=-3.871572985175836E-6}, evalInputDelta = -2.270831587214417E-5
F(35.68172544841876) = LineSearchPoint{point=PointSample{avg=1.3349847514685138}, derivative=3.1183692357242292E-6}, evalInputDelta = -3.4180248452253537E-5
1.3349847514685138 <= 1.335018931716966
New Minimum: 1.3349657174842497 > 1.3349634579034482
F(22.040257406264704) = LineSearchPoint{point=PointSample{avg=1.3349634579034482}, derivative=2.9022711997941997E-9}, evalInputDelta = -5.547381351789227E-5
Right bracket at 22.040257406264704
Converged to right
Fitness changed from 1.335018931716966 to 1.3349634579034482
Iteration 3 complete. Error: 1.3349634579034482 Total: 0.0230; Orientation: 0.0006; Line Search: 0.0207
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349634579034482}, derivative=-4.5712873027853087E-7}
New Minimum: 1.3349634579034482 > 1.3349593592709355
F(22.040257406264704) = LineSearchPoint{point=PointSample{avg=1.3349593592709355}, derivative=8.515481804988097E-8}, evalInputDelta = -4.098632512716804E-6
1.3349593592709355 <= 1.3349634579034482
New Minimum: 1.3349593592709355 > 1.3349592118734441
F(18.57927446664142) = LineSearchPoint{point=PointSample{avg=1.3349592118734441}, derivative=2.0554857303955376E-11}, evalInputDelta = -4.246030004040335E-6
Right bracket at 18.57927446664142
Converged to right
Fitness changed from 1.3349634579034482 to 1.3349592118734441
Iteration 4 complete. Error: 1.3349592118734441 Total: 0.0114; Orientation: 0.0005; Line Search: 0.0091
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349592118734441}, derivative=-3.567852298142671E-8}
New Minimum: 1.3349592118734441 > 1.3349588276558406
F(18.57927446664142) = LineSearchPoint{point=PointSample{avg=1.3349588276558406}, derivative=-5.683168499147133E-9}, evalInputDelta = -3.842176035107059E-7
F(130.05492126648994) = LineSearchPoint{point=PointSample{avg=1.3349682159046772}, derivative=1.7405092704997804E-7}, evalInputDelta = 9.004031233050824E-6
F(10.004224712806918) = LineSearchPoi
...skipping 4841 bytes...
.4814455315408842E-12}, evalInputDelta = -2.0948132117837304E-11
F(70.0460187751001) = LineSearchPoint{point=PointSample{avg=1.3349587772717473}, derivative=5.86264247662876E-12}, evalInputDelta = 1.1057421644977694E-10
F(5.388155290392315) = LineSearchPoint{point=PointSample{avg=1.3349587771483713}, derivative=-2.046378782111063E-12}, evalInputDelta = -1.2801759652347755E-11
F(37.71708703274621) = LineSearchPoint{point=PointSample{avg=1.3349587771461366}, derivative=1.908143760535028E-12}, evalInputDelta = -1.503641655631327E-11
1.3349587771461366 <= 1.334958777161173
New Minimum: 1.3349587771320173 > 1.3349587771312539
F(22.117677053194473) = LineSearchPoint{point=PointSample{avg=1.3349587771312539}, derivative=3.932696612595334E-18}, evalInputDelta = -2.991917824601842E-11
Right bracket at 22.117677053194473
Converged to right
Fitness changed from 1.334958777161173 to 1.3349587771312539
Iteration 9 complete. Error: 1.3349587771312539 Total: 0.0248; Orientation: 0.0005; Line Search: 0.0226
Low gradient: 5.484213606133697E-7
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349587771312539}, derivative=-3.0076598877701964E-13}
New Minimum: 1.3349587771312539 > 1.3349587771285603
F(22.117677053194473) = LineSearchPoint{point=PointSample{avg=1.3349587771285603}, derivative=5.719596876958802E-14}, evalInputDelta = -2.693623102345555E-12
1.3349587771285603 <= 1.3349587771312539
New Minimum: 1.3349587771285603 > 1.334958777128459
F(18.58366473897912) = LineSearchPoint{point=PointSample{avg=1.334958777128459}, derivative=1.2060673279567256E-20}, evalInputDelta = -2.794875442191369E-12
Right bracket at 18.58366473897912
Converged to right
Fitness changed from 1.3349587771312539 to 1.334958777128459
Iteration 10 complete. Error: 1.334958777128459 Total: 0.0141; Orientation: 0.0006; Line Search: 0.0113
Low gradient: 1.5363909513765158E-7
F(0.0) = LineSearchPoint{point=PointSample{avg=1.334958777128459}, derivative=-2.360497155471635E-14}
New Minimum: 1.334958777128459 > 1.3349587771282045
F(18.58366473897912) = LineSearchPoint{point=PointSample{avg=1.3349587771282045}, derivative=-3.7717543634974336E-15}, evalInputDelta = -2.544631172440859E-13
F(130.08565317285385) = LineSearchPoint{point=PointSample{avg=1.3349587771344185}, derivative=1.1522741398730338E-13}, evalInputDelta = 5.959455151582915E-12
F(10.006588705604143) = LineSearchPoint{point=PointSample{avg=1.3349587771282763}, derivative=-1.2925546115384994E-14}, evalInputDelta = -1.8274270985330077E-13
F(70.046120939229) = LineSearchPoint{point=PointSample{avg=1.3349587771294238}, derivative=5.115096743380197E-14}, evalInputDelta = 9.64783808399261E-13
F(5.388163149171461) = LineSearchPoint{point=PointSample{avg=1.3349587771283473}, derivative=-1.7854511470180594E-14}, evalInputDelta = -1.1168843627729075E-13
F(37.71714204420023) = LineSearchPoint{point=PointSample{avg=1.3349587771283278}, derivative=1.6648237694018092E-14}, evalInputDelta = -1.312283615106935E-13
1.3349587771283278 <= 1.334958777128459
New Minimum: 1.3349587771282045 > 1.334958777128198
F(22.117790896549053) = LineSearchPoint{point=PointSample{avg=1.334958777128198}, derivative=3.208076070817797E-21}, evalInputDelta = -2.609024107869118E-13
Right bracket at 22.117790896549053
Converged to right
Fitness changed from 1.334958777128459 to 1.334958777128198
Iteration 11 complete. Error: 1.334958777128198 Total: 0.0279; Orientation: 0.0005; Line Search: 0.0261
Low gradient: 5.1227133452970395E-8
F(0.0) = LineSearchPoint{point=PointSample{avg=1.334958777128198}, derivative=-2.6242192018084388E-15}
New Minimum: 1.334958777128198 > 1.3349587771281746
F(22.117790896549053) = LineSearchPoint{point=PointSample{avg=1.3349587771281746}, derivative=4.990573156669673E-16}, evalInputDelta = -2.353672812205332E-14
1.3349587771281746 <= 1.334958777128198
New Minimum: 1.3349587771281746 > 1.3349587771281737
F(18.58366726338535) = LineSearchPoint{point=PointSample{avg=1.3349587771281737}, derivative=9.806836262895936E-24}, evalInputDelta = -2.4424906541753444E-14
Right bracket at 18.58366726338535
Converged to right
Fitness changed from 1.334958777128198 to 1.3349587771281737
Iteration 12 complete. Error: 1.3349587771281737 Total: 0.0190; Orientation: 0.0005; Line Search: 0.0173
Low gradient: 1.4351201548977958E-8
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349587771281737}, derivative=-2.0595698589938737E-16}
New Minimum: 1.3349587771281737 > 1.3349587771281715
F(18.58366726338535) = LineSearchPoint{point=PointSample{avg=1.3349587771281715}, derivative=-3.290921934524792E-17}, evalInputDelta = -2.220446049250313E-15
F(130.08567084369747) = LineSearchPoint{point=PointSample{avg=1.3349587771282256}, derivative=1.0053772697255319E-15}, evalInputDelta = 5.1958437552457326E-14
F(10.006590064899806) = LineSearchPoint{point=PointSample{avg=1.3349587771281721}, derivative=-1.1277741867983147E-16}, evalInputDelta = -1.5543122344752192E-15
F(70.04613045429863) = LineSearchPoint{point=PointSample{avg=1.3349587771281821}, derivative=4.462999527521886E-16}, evalInputDelta = 8.43769498715119E-15
F(5.388163881099895) = LineSearchPoint{point=PointSample{avg=1.3349587771281726}, derivative=-1.5578337248544825E-16}, evalInputDelta = -1.1102230246251565E-15
F(37.71714716769927) = LineSearchPoint{point=PointSample{avg=1.3349587771281723}, derivative=1.4525829815008097E-16}, evalInputDelta = -1.3322676295501878E-15
1.3349587771281723 <= 1.3349587771281737
F(22.117801531350285) = LineSearchPoint{point=PointSample{avg=1.3349587771281715}, derivative=2.528557821269251E-24}, evalInputDelta = -2.220446049250313E-15
Right bracket at 22.117801531350285
Converged to right
Fitness changed from 1.3349587771281737 to 1.3349587771281715
Iteration 13 complete. Error: 1.3349587771281715 Total: 0.0200; Orientation: 0.0047; Line Search: 0.0139
Low gradient: 4.6284572979124976E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349587771281715}, derivative=-2.1422616958599456E-17}
New Minimum: 1.3349587771281715 > 1.3349587771281712
F(22.117801531350285) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=-3.074006286115902E-18}, evalInputDelta = -2.220446049250313E-16
F(154.82461071945198) = LineSearchPoint{point=PointSample{avg=1.3349587771281781}, derivative=1.0701765007332974E-16}, evalInputDelta = 6.661338147750939E-15
F(11.909585439957844) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=-1.1542595756208684E-17}, evalInputDelta = -2.220446049250313E-16
F(83.3670980797049) = LineSearchPoint{point=PointSample{avg=1.3349587771281723}, derivative=4.773752906100965E-17}, evalInputDelta = 8.881784197001252E-16
F(6.4128536984388385) = LineSearchPoint{point=PointSample{avg=1.3349587771281715}, derivative=-1.6102605516446938E-17}, evalInputDelta = 0.0
F(44.88997588907187) = LineSearchPoint{point=PointSample{avg=1.3349587771281715}, derivative=1.5817462338700376E-17}, evalInputDelta = 0.0
1.3349587771281715 <= 1.3349587771281715
F(25.82327365833721) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=1.7454540893791252E-25}, evalInputDelta = -2.220446049250313E-16
Right bracket at 25.82327365833721
Converged to right
Fitness changed from 1.3349587771281715 to 1.3349587771281712
Iteration 14 complete. Error: 1.3349587771281712 Total: 0.0219; Orientation: 0.0005; Line Search: 0.0204
Low gradient: 1.3405263347875513E-9
F(0.0) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=-1.797010854258946E-18}
F(25.82327365833721) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=3.010592697439953E-19}, evalInputDelta = 0.0
1.3349587771281712 <= 1.3349587771281712
F(22.117803654719996) = LineSearchPoint{point=PointSample{avg=1.3349587771281712}, derivative=7.818790239866192E-27}, evalInputDelta = 0.0
Right bracket at 22.117803654719996
Converged to right
Fitness changed from 1.3349587771281712 to 1.3349587771281712
Static Iteration Total: 0.0081; Orientation: 0.0006; Line Search: 0.0063
Iteration 15 failed. Error: 1.3349587771281712
Previous Error: 0.0 -> 1.3349587771281712
Optimization terminated 15
Final threshold in iteration 15: 1.3349587771281712 (> 0.0) after 0.311s (< 30.000s)
Returns
1.3349587771281712
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ -0.12799998976079824, 0.07999999423799242, 0.699999993335781, 0.49600000218702606 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ 0.15658449315721712, 0.19278888703744493, 0.358380667946918, 0.29224595185842 ]
Next, we apply the same optimization using L-BFGS, which is nearly ideal for purely second-order or quadratic functions.
TrainingTester.java:509 executed in 0.69 seconds (0.000 gc):
IterativeTrainer iterativeTrainer = new IterativeTrainer(trainable.addRef());
try {
iterativeTrainer.setLineSearchFactory(label -> new ArmijoWolfeSearch());
iterativeTrainer.setOrientation(new LBFGS());
iterativeTrainer.setMonitor(TrainingTester.getMonitor(history));
iterativeTrainer.setTimeout(30, TimeUnit.SECONDS);
iterativeTrainer.setIterationsPerSample(100);
iterativeTrainer.setMaxIterations(250);
iterativeTrainer.setTerminateThreshold(0);
return iterativeTrainer.run();
} finally {
iterativeTrainer.freeRef();
}
Reset training subject: 2608654524913
Reset training subject: 2608655802914
Adding measurement 7ee05728 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD
Non-optimal measurement 1.4410043780930528 < 1.4410043780930528. Total: 1
th(0)=1.4410043780930528;dx=-0.010947934512708877
Adding measurement 70df88d3 to history. Total: 1
New Minimum: 1.4410043780930528 > 1.418813461440417
END: th(2.154434690031884)=1.418813461440417; dx=-0.009654902395848403 evalInputDelta=0.022190916652635728
Fitness changed from 1.4410043780930528 to 1.418813461440417
Iteration 1 complete. Error: 1.418813461440417 Total: 0.0107; Orientation: 0.0022; Line Search: 0.0049
Non-optimal measurement 1.418813461440417 < 1.418813461440417. Total: 2
LBFGS Accumulation History: 2 points
Non-optimal measurement 1.418813461440417 < 1.418813461440417. Total: 2
th(0)=1.418813461440417;dx=-0.00851982260347121
Adding measurement 26749fd2 to history. Total: 2
New Minimum: 1.418813461440417 > 1.3841826791867486
END: th(4.641588833612779)=1.3841826791867486; dx=-0.006412407922864998 evalInputDelta=0.034630782253668446
Fitness changed from 1.418813461440417 to 1.3841826791867486
Iteration 2 complete. Error: 1.3841826791867486 Total: 0.0058; Orientation: 0.0010; Line Search: 0.0037
Non-optimal measurement 1.3841826791867486 < 1.3841826791867486. Total: 3
LBFGS Accumulation History: 3 points
Non-optimal measurement 1.3841826791867486 < 1.3841826791867486. Total: 3
th(0)=1.3841826791867486;dx=-0.004836597402970378
Adding measurement 4595e4ba to history. Total: 3
New Minimum: 1.3841826791867486 > 1.348158768808661
END: th(10.000000000000002)=1.348158768808661; dx=-0.002392378802878564 evalInputDelta=0.03602391037808772
Fitness changed from 1.3841826791867486 to 1.348158768808661
Iteration 3 complete. Error: 1.348158768808661 Total: 0.0051; Orientation: 0.0009; Line Search: 0.0031
Non-optimal measurement 1.348158768808661 < 1.348158768808661. Total: 4
Rejected: LBFGS Orientation magnitude: 6.834e-01, gradient 3.456e-02, dot -0.988; [b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.348158768808661, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
LBFGS Accumulation History: 3 points
Removed measurement 4595e4ba to history. Total: 3
Adding measurement 44a5682e to history. Total: 3
th(0)=1.348158768808661;dx=-0.0011942631090930134
Adding measurement 333170b5 to history. Total: 4
New Minimum: 1.348158768808661 > 1.335374593042268
END: th(21.544346900318843)=1.335374593042268; dx=-1.0570420357761427E-5 evalInputDelta=0.012784175766392902
Fitness changed from 1.348158768808661 to 1.335374593042268
Iteration 4 complete. Error: 1.335374593042268 Total: 0.0207; Orientation: 0.0156; Line Search: 0.0040
Non-optimal measurement 1.335374593042268 < 1.335374593042268. Total: 5
Rejected: LBFGS Orientation magnitude: 1.140e-01, gradient 6.373e-03, dot -0.993; [70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.335374593042268, 1.348158768808661, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 1.189e-01, gradient 6.373e-03, dot -0.998; [f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.335374593042268, 1.348158768808661, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 333170b5 to history. Total: 4
Removed measurement 44a5682e to history. Total: 3
Adding measurement 675e2334 to history. Total: 3
th(0)=1.335374593042268;dx=-4.061904484286917E-5
Non-optimal measurement 1.3358287301773089 < 1.335374593042268. Total: 4
Armijo: th(46.4158883361278)=1.3358287301773089; dx=5.979188543035564E-5 evalInputDelta=-4.54137135040833E-4
Adding measurement 31106833 to history. Total: 4
New Minimum: 1.335374593042268 > 1.3350189306891795
WOLF (strong): th(23.2079441680639)=1.3350189306891795; dx=9.882988931102869E-6 evalInputDelta=3.556623530884817E-4
Non-optimal measurement 1.3351257298933505 < 1.3350189306891795. Total: 5
END: th(7.735981389354634)=1.3351257298933505; dx=-2.372866580140759E-5 evalInputDelta=2.488631489174775E-4
Fitness changed from 1.335374593042268 to 1.3350189306891795
Iteration 5 complete. Error: 1.3350189306891795 Total: 0.0559; Orientation: 0.0473; Line Search: 0.0077
Non-optimal measurement 1.3350189306891795 < 1.3350189306891795. Total: 5
Rejected: LBFGS Orientation magnitude: 5.184e-02, gradient 2.692e-03, dot -0.987; [dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3350189306891795, 1.335374593042268, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 5.022e-02, gradient 2.692e-03, dot -0.991; [70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3350189306891795, 1.335374593042268, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 31106833 to history. Total: 4
Removed measurement 675e2334 to history. Total: 3
Adding measurement 27b6baa5 to history. Total: 3
th(0)=1.3350189306891795;dx=-7.248526456593706E-6
Adding measurement 2350841a to history. Total: 4
New Minimum: 1.3350189306891795 > 1.3349608846598973
WOLF (strong): th(16.66666666666667)=1.3349608846598973; dx=2.873733471412577E-7 evalInputDelta=5.804602928227709E-5
Non-optimal measurement 1.334974207753929 < 1.3349608846598973. Total: 5
END: th(8.333333333333336)=1.334974207753929; dx=-3.4838543875697025E-6 evalInputDelta=4.472293525048521E-5
Fitness changed from 1.3350189306891795 to 1.3349608846598973
Iteration 6 complete. Error: 1.3349608846598973 Total: 0.0362; Orientation: 0.0302; Line Search: 0.0050
Non-optimal measurement 1.3349608846598973 < 1.3349608846598973. Total: 5
Rejected: LBFGS Orientation magnitude: 6.017e-03, gradient 4.071e-04, dot -0.992; [dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349608846598973, 1.3350189306891795, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 6.389e-03, gradient 4.071e-04, dot -0.991; [f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349608846598973, 1.3350189306891795, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 2350841a to history. Total: 4
Removed measurement 27b6baa5 to history. Total: 3
Adding measuremen
...skipping 16304 bytes...
77128187. Total: 5
Rejected: LBFGS Orientation magnitude: 7.463e-07, gradient 4.490e-08, dot -0.987; [70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.334958777128187, 1.3349587771283333, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 7.329e-07, gradient 4.490e-08, dot -0.992; [dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.334958777128187, 1.3349587771283333, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 77a0b4e7 to history. Total: 4
Removed measurement 5e92a4cb to history. Total: 3
Adding measurement 2b7369eb to history. Total: 3
th(0)=1.334958777128187;dx=-2.0161142875594966E-15
Adding measurement 4b79f81c to history. Total: 4
New Minimum: 1.334958777128187 > 1.334958777128174
WOLF (strong): th(21.7013888888889)=1.334958777128174; dx=7.888262005057927E-16 evalInputDelta=1.3100631690576847E-14
Adding measurement 1af95fbc to history. Total: 5
New Minimum: 1.334958777128174 > 1.334958777128173
END: th(10.85069444444445)=1.334958777128173; dx=-6.136440256676807E-16 evalInputDelta=1.3988810110276972E-14
Fitness changed from 1.334958777128187 to 1.334958777128173
Iteration 15 complete. Error: 1.334958777128173 Total: 0.0349; Orientation: 0.0288; Line Search: 0.0049
Non-optimal measurement 1.334958777128173 < 1.334958777128173. Total: 6
Rejected: LBFGS Orientation magnitude: 2.437e-07, gradient 1.395e-08, dot -0.989; [70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.334958777128173, 1.334958777128174, 1.334958777128187, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 2.413e-07, gradient 1.395e-08, dot -0.994; [dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.334958777128173, 1.334958777128174, 1.334958777128187, 1.3841826791867486, 1.418813461440417
Rejected: LBFGS Orientation magnitude: 2.359e-07, gradient 1.395e-08, dot -0.993; [dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.334958777128173, 1.334958777128174, 1.334958777128187, 1.3841826791867486
LBFGS Accumulation History: 3 points
Removed measurement 1af95fbc to history. Total: 5
Removed measurement 4b79f81c to history. Total: 4
Removed measurement 2b7369eb to history. Total: 3
Adding measurement 2eb85a1c to history. Total: 3
th(0)=1.334958777128173;dx=-1.9452344564017502E-16
Adding measurement 5785d6f7 to history. Total: 4
New Minimum: 1.334958777128173 > 1.3349587771281715
WOLF (strong): th(23.377112522047362)=1.3349587771281715; dx=8.155581398415972E-17 evalInputDelta=1.5543122344752192E-15
Non-optimal measurement 1.3349587771281715 < 1.3349587771281715. Total: 5
END: th(11.688556261023681)=1.3349587771281715; dx=-5.648381545808909E-17 evalInputDelta=1.5543122344752192E-15
Fitness changed from 1.334958777128173 to 1.3349587771281715
Iteration 16 complete. Error: 1.3349587771281715 Total: 0.0493; Orientation: 0.0428; Line Search: 0.0054
Non-optimal measurement 1.3349587771281715 < 1.3349587771281715. Total: 5
Rejected: LBFGS Orientation magnitude: 1.320e-07, gradient 6.854e-09, dot -0.989; [dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349587771281715, 1.334958777128173, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 1.272e-07, gradient 6.854e-09, dot -0.992; [dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349587771281715, 1.334958777128173, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 5785d6f7 to history. Total: 4
Removed measurement 2eb85a1c to history. Total: 3
Adding measurement 2af76f8d to history. Total: 3
th(0)=1.3349587771281715;dx=-4.698145735437098E-17
Adding measurement 54f515ed to history. Total: 4
New Minimum: 1.3349587771281715 > 1.3349587771281712
WOLF (strong): th(25.18223108513879)=1.3349587771281712; dx=2.9183875447026415E-17 evalInputDelta=2.220446049250313E-16
Non-optimal measurement 1.3349587771281712 < 1.3349587771281712. Total: 5
END: th(12.591115542569394)=1.3349587771281712; dx=-8.898791038253689E-18 evalInputDelta=2.220446049250313E-16
Fitness changed from 1.3349587771281715 to 1.3349587771281712
Iteration 17 complete. Error: 1.3349587771281712 Total: 0.0339; Orientation: 0.0273; Line Search: 0.0056
Non-optimal measurement 1.3349587771281712 < 1.3349587771281712. Total: 5
Rejected: LBFGS Orientation magnitude: 7.555e-08, gradient 4.343e-09, dot -0.987; [dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349587771281712, 1.3349587771281715, 1.3841826791867486, 1.418813461440417, 1.4410043780930528
Rejected: LBFGS Orientation magnitude: 7.369e-08, gradient 4.343e-09, dot -0.991; [b5531317-dd91-4b1b-9a4d-aead00fe81b1 = 1.000/1.000e+00, f2996fcc-49ff-4c65-bd59-0cc404576fa3 = 1.000/1.000e+00, dd33d23d-003c-49d7-8f63-004a952737d7 = 1.000/1.000e+00, 70bcc19d-fa88-4620-9574-af61f1fca2e9 = 1.000/1.000e+00, dc3f0777-6f51-4824-a329-4be874db2387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.3349587771281712, 1.3349587771281715, 1.3841826791867486, 1.418813461440417
LBFGS Accumulation History: 3 points
Removed measurement 54f515ed to history. Total: 4
Removed measurement 2af76f8d to history. Total: 3
Adding measurement 3c2572c to history. Total: 3
th(0)=1.3349587771281712;dx=-1.886519503407168E-17
Non-optimal measurement 1.3349587771281712 < 1.3349587771281712. Total: 4
WOLF (strong): th(27.12673611111113)=1.3349587771281712; dx=1.4472043124096273E-17 evalInputDelta=0.0
Non-optimal measurement 1.3349587771281712 < 1.3349587771281712. Total: 4
END: th(13.563368055555564)=1.3349587771281712; dx=-2.1965759502070755E-18 evalInputDelta=0.0
Fitness changed from 1.3349587771281712 to 1.3349587771281712
Static Iteration Total: 0.0290; Orientation: 0.0237; Line Search: 0.0044
Iteration 18 failed. Error: 1.3349587771281712
Previous Error: 0.0 -> 1.3349587771281712
Optimization terminated 18
Final threshold in iteration 18: 1.3349587771281712 (> 0.0) after 0.685s (< 30.000s)
Returns
1.3349587771281712
This training apply resulted in the following configuration:
TrainingTester.java:610 executed in 0.00 seconds (0.000 gc):
RefList<double[]> state = network.state();
assert state != null;
String description = state.stream().map(RefArrays::toString).reduce((a, b) -> a + "\n" + b)
.orElse("");
state.freeRef();
return description;
Returns
TrainingTester.java:622 executed in 0.00 seconds (0.000 gc):
return RefArrays.stream(RefUtil.addRef(data)).flatMap(x -> {
return RefArrays.stream(x);
}).limit(1).map(x -> {
String temp_18_0015 = x.prettyPrint();
x.freeRef();
return temp_18_0015;
}).reduce((a, b) -> a + "\n" + b).orElse("");
Returns
[ -0.12799999769777096, 0.07999999900021168, 0.6999999969320329, 0.49600000176552533 ]
To produce the following output:
TrainingTester.java:633 executed in 0.00 seconds (0.000 gc):
Result[] array = ConstantResult.batchResultArray(pop(RefUtil.addRef(data)));
@Nullable
Result eval = layer.eval(array);
assert eval != null;
TensorList tensorList = Result.getData(eval);
String temp_18_0016 = tensorList.stream().limit(1).map(x -> {
String temp_18_0017 = x.prettyPrint();
x.freeRef();
return temp_18_0017;
}).reduce((a, b) -> a + "\n" + b).orElse("");
tensorList.freeRef();
return temp_18_0016;
Returns
[ 0.1565844917827319, 0.19278888779342374, 0.358380668934368, 0.2922459514894764 ]
TrainingTester.java:432 executed in 0.10 seconds (0.000 gc):
return TestUtil.compare(title + " vs Iteration", runs);
Plotting range=[1.0, 0.12546785510720343], [17.0, 0.15192530032538973]; valueStats=DoubleSummaryStatistics{count=48, sum=64.372574, min=1.334959, average=1.341095, max=1.418813}
Plotting 17 points for GD
Plotting 14 points for CjGD
Plotting 17 points for LBFGS
Returns
TrainingTester.java:435 executed in 0.01 seconds (0.000 gc):
return TestUtil.compareTime(title + " vs Time", runs);
Plotting range=[0.0, 0.12546785510720343], [0.644, 0.15192530032538973]; valueStats=DoubleSummaryStatistics{count=48, sum=64.372574, min=1.334959, average=1.341095, max=1.418813}
Plotting 17 points for GD
Plotting 14 points for CjGD
Plotting 17 points for LBFGS
Returns
TrainingTester.java:255 executed in 0.00 seconds (0.000 gc):
return grid(inputLearning, modelLearning, completeLearning);
Returns
TrainingTester.java:258 executed in 0.00 seconds (0.000 gc):
return new ComponentResult(null == inputLearning ? null : inputLearning.value,
null == modelLearning ? null : modelLearning.value, null == completeLearning ? null : completeLearning.value);
Returns
{"input":{ "LBFGS": { "type": "NonConverged", "value": 1.3349587771281712 }, "CjGD": { "type": "NonConverged", "value": 1.3349587771281712 }, "GD": { "type": "NonConverged", "value": 1.3349587771281712 } }, "model":null, "complete":null}
LayerTests.java:425 executed in 0.00 seconds (0.000 gc):
throwException(exceptions.addRef());
| details | result |
|---|---|
| {"input":{ "LBFGS": { "type": "NonConverged", "value": 1.3349587771281712 }, "CjGD": { "type": "NonConverged", "value": 1.3349587771281712 }, "GD": { "type": "NonConverged", "value": 1.3349587771281712 } }, "model":null, "complete":null} | OK |
{
"result": "OK",
"performance": {
"execution_time": "2.181",
"gc_time": "0.354"
},
"created_on": 1586737195758,
"file_name": "trainingTest",
"report": {
"simpleName": "Basic",
"canonicalName": "com.simiacryptus.mindseye.layers.java.SoftmaxLayerTest.Basic",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/test/java/com/simiacryptus/mindseye/layers/java/SoftmaxLayerTest.java",
"javaDoc": ""
},
"training_analysis": {
"input": {
"LBFGS": {
"type": "NonConverged",
"value": 1.3349587771281712
},
"CjGD": {
"type": "NonConverged",
"value": 1.3349587771281712
},
"GD": {
"type": "NonConverged",
"value": 1.3349587771281712
}
}
},
"archive": "s3://code.simiacrypt.us/tests/com/simiacryptus/mindseye/layers/java/SoftmaxLayer/Basic/trainingTest/202004131955",
"id": "cc6be786-2999-4f86-84cd-ef971b9171ab",
"report_type": "Components",
"display_name": "Comparative Training",
"target": {
"simpleName": "SoftmaxLayer",
"canonicalName": "com.simiacryptus.mindseye.layers.java.SoftmaxLayer",
"link": "https://github.com/SimiaCryptus/mindseye-java/tree/93db34cedee48c0202777a2b25deddf1dfaf5731/src/main/java/com/simiacryptus/mindseye/layers/java/SoftmaxLayer.java",
"javaDoc": ""
}
}