-
Notifications
You must be signed in to change notification settings - Fork 4
/
ChangeLog
1354 lines (1211 loc) · 62.7 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2014.09.25 Attila Krasznahorkay <[email protected]>
* sframe_main.cxx now instantiates a TApplication object to make
it possible to nicely (auto-)load libraries at runtime. The
system should be much more flexible in this respect now.
* Added a new configuration option for the XML files: Macro.
It can be used to execute a ROOT macro at initialisation time.
(Which can potentially load libraries.)
* Updated the JobConfig.dtd file to include the description of the
new Macro node.
* Tagging as SFrame-04-00-01
2014.05.23 Attila Krasznahorkay <[email protected]>
* Hiding all the auto-generated .pcm file from SVN.
* Hiding the Cintex code from the compiler when ROOT 6 (or newer)
is used.
* Commented the Cintex parts of the example configuration,
as very soon those lines should not be used anymore.
(Once ROOT 6 becomes production ready.)
* Taught Makefile.common how to install the Pre-Compiled Module
files generated by ROOT 6 in the library directory. Only tested
with ROOT 6 so far, but in principle the code should work with
ROOT 5 as well.
* All of these changes make the code work correctly with the ROOT
master branch (pre-ROOT-6) in LOCAL mode, but PROOF-Lite still
has some issues.
* Tagging as SFrame-04-00-00
2014.03.10 Attila Krasznahorkay <[email protected]>
* Trying to fix yet another issue with missing input files. This time the
issue was reported by Christoph Wasicki.
* SInputData::ValidateInputFiles() now tries to keep the iterators and the
summed luminosity correct when non-existent input files are removed.
* Tagging as SFrame-03-06-28
2013.12.02 Attila Krasznahorkay <[email protected]>
* Making the PROOF files aware of the ROOTPROOFLIBDIR environment variable,
which can be used to specify a common library directory that should be
searched for libraries on a PROOF cluster with a shared filesystem.
* Suggestion received from Max Bellomo.
* Tagging as SFrame-03-06-27
2013.11.26 Attila Krasznahorkay <[email protected]>
* Fixing SSummedVar after the changes made in ISCycleBaseHist.
* Tagging as SFrame-03-06-26
2013.11.25 Attila Krasznahorkay <[email protected]>
* In ROOT v5-34-12 TSelectorList now inherits from THashList, and
not TList.
* For this reason the SCycleBaseHist::Book(...) and
SCycleBaseHist::WriteObj(...) functions have been updated to take this
into account.
* At the same time, changed the type of the output list from TList to
TSelectorList in SCycleBaseNTuple and SCycleBaseHist. This should make
it easier to react to such changes in the future.
* Tagging as SFrame-03-06-25
2013.11.13 Attila Krasznahorkay <[email protected]>
* DeclareVariable<...>(...) no longer uses cxxabi.h to declare branches
to ROOT. The type names reported by clang 5.0 on MacOS X Mavericks are
not recognised by ROOT.
* Put explicit "::" in front of the message level enumeration values in the
header/icc files to avoid clashes with the same sort of enumeration values
coming from the RooFit code. (RooFit::MsgLevel)
* Still seeing some issues with clang 5.0 and PROOF-Lite mode that will need
following up.
* Tagging as SFrame-03-06-24
2013.10.17 Attila Krasznahorkay <[email protected]>
* Fixed the compilation of SCycleConfig in C++11 mode with GCC.
Apparently in this mode the language is more strict about automatic
type conversions.
* Removed libpcre.so from the list of libraries that sframe_main is
linked against. It was left here from an earlier version of the code,
back when the core library was using TPRegexp. It's not doing that
these days.
* Thanks to Jochen Ott for pointint both of these issues out.
* Tagging as SFrame-03-06-23
2013.08.26 Attila Krasznahorkay <[email protected]>
* Adding $CPPEXPFLAGS to the compilation commands. This should
make the compilation against SLC5 ROOT on an SL(C)6 system
succeed.
* Tagging as SFrame-03-06-22
2013.06.19 Attila Krasznahorkay <[email protected]>
* SCycleController.cxx now opens the output file for writing with
TFile::Open(...) rather than just creating a TFile on the stack. This makes
it possible to use output directories on XRootD/EOS. So far the code was
silently failing when trying to use such an output directory in the XML
configuration...
* Tagging as SFrame-03-06-21
2013.04.15 Attila Krasznahorkay <[email protected]>
* Adding a fix proposed by Mike Hance. With this fix the cache files made
will finally store the full names of the metadata trees. (Including
possible directory names.) Previously just the tree's name was saved,
making the cache fail on reading.
* Tagging as SFrame-03-06-20
2013.02.02 Attila Krasznahorkay <[email protected]>
* Made the print macros in SLogger.h a bit more robust. Had some problems
with using output operators designed for std::ostream instead of SLogger
when using these macros. (clang 4.2 started complaining about these.)
* Tagging as SFrame-03-06-19
2013.01.21 Attila Krasznahorkay <[email protected]>
* Found a bug in the code that makes absolute path names from relative ones.
DCache file names usually start with "dcache:/". (Only a single slash.)
The code now looks for just ":/" in a file name to decide whether it should
make it an absolute path name.
* Tagging as SFrame-03-06-18
2013.01.21 Attila Krasznahorkay <[email protected]>
* Added a new property to SInputData, which controls whether files belonging
to this input data object should be "looked up" during dataset validation.
Turning this look-up off makes the code not try to check where a given file
is located in an XRootD storage. (Most notably EOS.) This can come in very
handy with some storage systems.
* Updated all the code to print/respect this new configuration option.
* Updated SCycleBaseConfig to read this new configuration option from the XML
files.
* Added the new SkipLookup property of InputData blocks to the JobConfig.dtd
file.
* Added an example usage of SkipLookup to FirstCycle_config.xml, together
with a short description of this property left in the comments.
* Tagging as SFrame-03-06-17
2013.01.20 Attila Krasznahorkay <[email protected]>
* Cycles no longer continue when they detect a TTree write failure. This can
happen sometimes on the grid when the worker node runs out of free space.
Until now the job would print ERROR messages in this case, but would
continue to run, usually ending up with a quite empty output file, and
the grid job marked successful.
* Tagging as SFrame-03-06-16
2013.01.16 Attila Krasznahorkay <[email protected]>
* PROOF-Lite (at least in ROOT 5.34.04) doesn't seem to like relative path
names to the input files anymore. In order not to have to always use
absolute path names for the input files just to use PROOF-Lite, the code
now makes sure that all path names are absolute when passing them to ROOT.
* Tagging as SFrame-03-06-15
2013.01.10 Attila Krasznahorkay <[email protected]>
* Found another way in which the code can crash if none of the input files
can be used. It is fixed now.
* Tagging as SFrame-03-06-14
2013.01.07 Attila Krasznahorkay <[email protected]>
* Discovered that if all the input files from an InputData block would be
unaccessible, the code crashed. This is now fixed, the code just reports
the appropriate ERROR and WARNING messages, and finishes gracefully.
(Stating that it processed 0 events.)
* Tagging as SFrame-03-06-13
2012.12.19 Attila Krasznahorkay <[email protected]>
* SFileMerger now also respects the $SFRAME_TEMP_DIR environment variable if
it is set. This avoids problems in case the user has a too small default
temporary directory. (Usually /tmp on linux.)
* Tagging as SFrame-03-06-12
2012.12.13 Attila Krasznahorkay <[email protected]>
* Updated the documentation of practically all the files.
* Modified the source text formatting a bit for it to fit into 80 column
editors.
* There should be no functionality changes at this point.
* Tagging as SFrame-03-06-11
2012.12.03 Attila Krasznahorkay <[email protected]>
* Enabled keyword expansion for the Cintex source files. Should've been done
ages ago...
* Not tagging for this
2012.12.03 Attila Krasznahorkay <[email protected]>
* File merging on PROOF with TProofOutputFile is now configured to always
download the files-to-be-merged before the merging starts. This should
provide better performance on clusters with slower networks. Should not
be an issue on quick networks either, so I didn't make this a configurable
option.
* Added the new extra "inFile" option to SToolBase::Book(...). It was just
an oversight now to put the option in so far.
* Tagging as SFrame-03-06-10
2012.11.29 Attila Krasznahorkay <[email protected]>
* The new code seems to work fine with the patched version of ROOT 5.34.03 on
the NYU cluster, so let's tag it as SFrame-03-06-09
2012.11.29 Attila Krasznahorkay <[email protected]>
* The code seems to be working now.
* The output file opening and closing is now handled by
SCycleBaseNTuple, and not SCycleBaseExec.
* SCycleBaseHist can request an output file to be created on
demand, it can not only work with in-file histogram merging
in case the user also writes an output TTree.
* Empty output trees are no longer written to the output file
on the PROOF workers. This avoids the problem of TFileMerger
creating empty merged trees when it receives an empty TTree
as its first input.
* Have to run some larger tests first, will tag after that.
2012.11.29 Attila Krasznahorkay <[email protected]>
* Intermediate check-in. I have to test the code's behaviour on Linux as
well.
* Re-arranged a bit how output files are handled during the event processing.
This will make it possible to handle in-file histogram merging, and empty
TTrees more elegantly.
* Made some documentation updates here and there.
* Not tagging yet.
2012.11.21 Attila Krasznahorkay <[email protected]>
* Made a number of changes to make it possible to merge histograms
and other objects in-file on PROOF, and not only in-memory.
* Updated SFileMerger to handle regular TObject objects as well,
not just TTree-s. Tried to use TFileMerger once more, but gave
up after some failed attempts. Should re-visit this later on
again.
* In any case, SFileMerger now has pretty much the same interface
as TFileMerger. It just does slightly different things...
* The Book(...) and WriteObj(...) functions now have one more
optional parameter. This controls whether the object created
should be merged in memory, or in file when running on PROOF.
The default choice is in-memory merging (the previous
implementation).
* Updated the code to use the in-file merging correctly in
SCycleBaseExec and SCycleController.
* Moved some DEBUG messages to VERBOSE in SCycleBaseNTuple,
which were printed too often now that DeclareVariable(...)
handles metadata trees as well.
* Updated the example cycles to make use of the new in-file
histogram merging functionality.
* Tagging as SFrame-03-06-08
2012.11.21 Attila Krasznahorkay <[email protected]>
* SCycleBaseNTuple::DeclareVariable(...) will now add variables
to metadata output trees as well if the user asks explicitly
for such a tree by name.
* The ConnectVariable(...) functions still don't support
metadata trees, as this would mean a big-big development
effort, and nobody really asked for this functionality anyway...
* The name of the cycle is written into the metadata of the
output file in a more robust way now.
* Updated the FirstCycle example to make use of the new feature
that allows us to create branches in output metadata trees
using DeclareVariable(...).
* SecondCycle now prints messages with their proper source string
when running.
* Tagging as SFrame-03-06-07
2012.11.20 Attila Krasznahorkay <[email protected]>
* All SFrame cycles will now add a TObjString description to their
output files, which describes the full configuration that was used
to produce this output file.
* Only configuration that the cycle knows about is written at the
moment. But it wouldn't be terribly difficult to add some info
about the loaded libraries/packages later on as well.
* Had to touch quite a few classes to implement this new feature in
a way that I liked...
* Added a new script, sframe_print_config.py, which can nicely print
the configuration of the cycle that was used to create an output
ROOT file.
* Fixed sframe_new_package.sh to make the BUILD.sh file correctly
for PROOF-Lite builds.
* Tagging as SFrame-03-06-06
2012.10.11 Attila Krasznahorkay <[email protected]>
* Cycles running in LOCAL mode will now either create a temporary directory
in the running directory as they did so far, or, if the
SFRAME_TEMP_DIR environment variable is set, create a temporary directory
inside that directory. (A unique directory is still made, the files are
not just put blindly into the specified directory.)
* Tagging as SFrame-03-06-05
2012.09.01 Attila Krasznahorkay <[email protected]>
* Updated the Makefile-s to handle parallel builds a little
better. The code is no longer built on 2 cores by default,
but now the user can freely specify the number of build
processes with -jX after the make command.
* Tagging as SFrame-03-06-04
2012.08.17 Attila Krasznahorkay <[email protected]>
* Added a new script (sframe_reset_proof.py) that can be used
to conveniently kill a PROOF job. This is sometimes needed,
and SFrame didn't really help with it so far.
* Tagging as SFrame-03-06-03
2012.08.17 Attila Krasznahorkay <[email protected]>
* Added two new variables that the user can set in their package's
Makefile: USERCXXFLAGS and USERLDFLAGS. They can be used
to fine-tune the compilation of user libraries. It is also still
possible to modify the INCLUDES variable in the user Makefile.
* Also extended the PAR file maker script to take these new
variables into account.
* Tagging as SFrame-03-06-02
2012.07.17 Attila Krasznahorkay <[email protected]>
* Disabled the code setting up TTreeCache in LOCAL mode for now.
It seems to cause serious problems.
* Tagging as SFrame-03-06-01
2012.06.22 Attila Krasznahorkay <[email protected]>
* Added some non-siginificant updates to make the code fit more on
my screen.
* Added an implementation to TObject::ExecuteEvent(...) in SCycleBase to
avoid a very verbose warning from the compiler. This doesn't however
change the fact that the user code would still fire this warning.
* Tweaked the cache handling a bit, to no effect.
* Added the ability to start the memory profiling of the PROOF master or
worker processes by specifying ";MemProfMaster" or ";MemProfWorkers"
after the PROOF server name in the ProofServer XML property of the
cycles.
* SProofManager still doesn't download the full Valgrind logs correctly.
This will have to be fixed. For now one has to look at the full Valgrind
logs directly on the PROOF workers.
* Tagging as SFrame-03-06-00
2012.06.07 Attila Krasznahorkay <[email protected]>
* "Stole" a header from the ATLAS offline code, and added it under
plug-ins/include/FPCompare.h. It holds function and functors that can be
used to implement safe floating point comparisons. (Sidestepping an x87
"bug".)
* Tagging as SFrame-03-05-17
2012.05.25 Attila Krasznahorkay <[email protected]>
* Added the possibility in sframe_clear_server.py to not only be able to
purge all packages from a cluster, but also only remove a single
package. The suggestion and code came from George Lewis.
* Tagging as SFrame-03-05-16
2012.04.21 Attila Krasznahorkay <[email protected]>
* Updated SH1.icc not to use <cmath> anymore. There is no good reason for
this, just that Xcode seems to go crazy over <cmath>, so my IDE is happier
if I use TMath.h instead.
* Tagging as SFrame-03-05-15
2012.04.16 Attila Krasznahorkay <[email protected]>
* Xcode managed to mess up the indentation of sframe_clear_server.py.
Fixing it now.
* Tagging as SFrame-03-05-14
2012.04.13 Attila Krasznahorkay <[email protected]>
* Added the long missing feature to the PROOF handling convenience scripts
that they wouldn't open X11 for no reason, when running on a Mac.
* Tagging as SFrame-03-05-13
2012.04.12 Attila Krasznahorkay <[email protected]>
* Updated all BUILD.sh scripts not to compile the code when running
on PROOF-Lite
* Updated sframe_new_package.sh to add the same protection for new packages.
* Tagging as SFrame-03-05-12
2012.04.11 Attila Krasznahorkay <[email protected]>
* SProofManager now keeps track of which PROOF servers have been
"configured" already. This way the code no longer needs to go through
the exercise of loading all packages on all nodes in a job that runs
multiple cycle instances. (The package configuration is job-wide,
not specific for a given cycle.)
* Updated SCycleController to take advantage of the new feature.
* Tagging as SFrame-03-05-11
2012.03.29 Attila Krasznahorkay <[email protected]>
* Adding support for the Makefile.arch location used by
MacPorts installed 5.32.XX. (Thanks to Martin Zeman for
pointing this out.)
* Tagging as SFrame-03-05-10
2012.03.28 Attila Krasznahorkay <[email protected]>
* Trying to enable TTree caching in LOCAL mode once more.
* Code seems to be reasonably stable (no crashes at file changes
anymore), but the D3PDReader code still reports inconsistent
cache sizes...
* Updated FirstCycle example to use TTree caching explicitly.
* Tagging as SFrame-03-05-09
2012.03.19 Attila Krasznahorkay <[email protected]>
* Exchanged the simple Add(...) calls with TList::AddLast(...) in
the Book(...) and WriteObj(...) functions on the suggestion
of Mike Hance. Previously both SFrame and ROOT were
checking for duplicates in the output object list, leading to
measurable slowdowns when handling a lot of histograms.
* Tagging as SFrame-03-05-08
2012.03.07 Attila Krasznahorkay <[email protected]>
* Fixed the == operator as well... :-P
* Tagging as SFrame-03-05-07
2012.03.07 Attila Krasznahorkay <[email protected]>
* Fixed the assignment operator of SInputData. The operator
forgot about the dataset and current-entry variables. This
lead to weird problems when the InputData objects were
re-ordered.
* Tagging as SFrame-03-05-06
2012.01.16 Attila Krasznahorkay <[email protected]>
* In ROOT 5.32 TProof::GetMissingFiles() doesn't play nicely
with PROOF-Lite. So for the moment I just disabled the
missing file list request for PROOF-Lite until I understand
what's going wrong.
* Updated python/PARHelpers.py to make .par packages that
play nicely with ROOT 5.32
* Tagging as SFrame-03-05-05
2011.12.06 Attila Krasznahorkay <[email protected]>
* Updated Makefile.common to find Makefile.arch in
ROOT 5.32 and up.
* Tagging as SFrame-03-05-04
2011.11.17 Attila Krasznahorkay <[email protected]>
* Added a new cycle property (ProcessOnlyLocal) for forcing
the PROOF workers to only process files that are on their
local disks. In some cases enabling this speeded up long
jobs on the NYU PROOF cluster. (By default it is left off.)
* Introduced some more rigorous testing for the user
properties. If the same property is defined multiple times
in the XML, the code will now print a warning.
* The DeclareProperty(...) functions also do a more verbose
checking to decide whether a given property name was used
already or not.
* Added some extra documentation to the SCycleConfig
members.
* Added the new property (ProcessOnlyLocal) to the common
DTD (JobConfig.dtd)
* Tagging as SFrame-03-05-03
2011.11.15 Attila Krasznahorkay <[email protected]>
* DecodeEnvVar(...) now only does anything if a property value
begins with ":exp:". This is needed to be able to give actual
regular expressions, and unexpanded environment variable
names to the code.
* Returned to printing an ERROR in this function when the
environment variable expansion fails, even though the user
explicitly requested it.
* Added some documentation to FirstCycle_config.xml about
how to use the environment variable expansion in property
values.
* Tagging as SFrame-03-05-02
2011.11.14 Attila Krasznahorkay <[email protected]>
* Fixing how the learning period of TTreeCache is handled on
PROOF. Now setting a positive number for the learning
period actually works...
* Added the possibility for letting the user set up the caching
details when the new files are opened. Will be useful with
the new D3PDReader features.
* DecodeEnvVar(...) still waiting for yet more updates... :-(
* Tagging as SFrame-03-05-01
2011.11.10 Attila Krasznahorkay <[email protected]>
* Changed the message printed by DecodeEnvVar from ERROR
to DEBUG when something unexpected happens. Apparently
unexpected things can happen more often than I thought.
* Added 3 new parameters to a cycle, which control how
TTreeCache should be used by the SFrame job.
* At the moment TTreeCache is only used in PROOF mode,
but I plan on adding this feature for LOCAL running as well.
* The default configuration still has TTreeCache disabled, as
it doesn't play nicely with MC10b D3PDs.
* Added some documentation about the new TTreeCache features
to FirstCycle_config.xml
* Tagging as SFrame-03-05-00
2011.10.21 Attila Krasznahorkay <[email protected]>
* Fixed the DecodeEnvVar(...) function after reading the ROOT
documentation a little more carefully.
* Changed "lite" to "lite://" as the PROOF server name in the
example configuration. This is the new ROOT naming scheme.
* Tagging as SFrame-03-04-26
2011.10.11 Attila Krasznahorkay <[email protected]>
* Small correction to the SCycleBaseConfig::DecodeEnvVar(...)
function.
* Tagging as SFrame-03-04-25
2011.10.11 Attila Krasznahorkay <[email protected]>
* Removed the automatic disabling of branches in
SCycleBaseNTuple::LoadInputTrees(...). That function call
was just giving us problems, and now actually caused
very measurable performance problems.
* Added the patches of Massimiliano to SCycleBaseConfig,
needed for his extension to SFrame.
* Made his code a bit more general in the DecodeEnvVar(...)
function.
* Modified SToolBase to make it possible to generate
dictionaries for tools inheriting from it. The modifications
came from Massimiliano.
* Tagging as SFrame-03-04-24
2011.09.06 Attila Krasznahorkay <[email protected]>
* Updated python/PARHelpers.py to include header files from the
source directory as well. (On the request of Mark Owen.)
* Added some additional "pass" statements to the module
while I was at it...
* Tagging as SFrame-03-04-23
2011.08.05 Attila Krasznahorkay <[email protected]>
* Removed the explicit optimization flags from Makefile.common
and the PAR Makefile generator. This way the code is only
built in optimized mode when the ROOT version used is in
optimized mode. (Using a debug build of ROOT builds the
SFrame libraries in debug mode automatically as well.)
* Disabled some messages from being printed on PROOF.
This reduces the logs considerably when creating many histograms
in a PROOF session.
* Tagging as SFrame-03-04-22
2011.07.30 Attila Krasznahorkay <[email protected]>
* Fixed hopefully all the file merging issues on MacOS X
Lion, while keeping all the previous features of the code.
* The usage of TFileMerger has once again been dropped, as
it was producing unreliable results somehow... (Don't know
why, since PROOF-Lite uses the same code, and that makes
usable output files...)
* Fixed the .par files for being used on a 64-bit Mac in
PROOF(-Lite) mode. Very surprised that nobody complained
about this so far... :-/
* Tagging as SFrame-03-04-21
2011.07.08 Attila Krasznahorkay <[email protected]>
* Found out that the last change only works with ROOT versions
>= 5.28. So now that code is protected by pre-processor
macros.
* Tagging as SFrame-03-04-20
2011.07.06 Attila Krasznahorkay <[email protected]>
* At the end of processing an InputData block using PROOF,
the cycle controller can now print which files were not
available/had problems during event processing.
* This is because we had some faulty file replications on the
NYU Tier3.
* Tagging as SFrame-03-04-19
2011.06.27 Attila Krasznahorkay <[email protected]>
* Added the missing virtual destructor to SToolBaseT.
(Didn't get a complaint on MacOS X... :-P)
* Tagging as SFrame-03-04-18
2011.06.27 Attila Krasznahorkay <[email protected]>
* Made the Retrieve<...>(...) function in SCycleBaseHist a bit
more general by making it possible to retrieve output objects
only. (This is needed for the ATLAS lumiblock merger code.)
* Added a new function, RetrieveAll<...>(...) to SCycleBaseHist.
It can be used to retrieve all versions of a given object from
the input file. (Was needed for the ATLAS lumiblock merger
code.)
* Followed the developments of SCycleBaseHist with
SToolBaseT.
* Added some functions to SToolBaseT that were missing so
far. (Coming from SCycleBaseNTuple.)
* Tagging as SFrame-03-04-17
2011.06.26 Attila Krasznahorkay <[email protected]>
* Made it possible to write out metadata trees into
sub-directories. This required changes in SCycleBaseNTuple
and SFileMerger.
* For the moment event-level trees are still only supported in
the main directory. But on request it should be pretty simple
to add this capability for those trees as well.
* Tagging as SFrame-03-04-16
2011.06.24 Attila Krasznahorkay <[email protected]>
* Made output ntuple management a bit more robust.
Now the code can efficiently use distant PROOF clusters
and retrieve the merged ntuple files from the PROOF
work directory using TFileMerger. Then it uses
SFileMerger to merge the file into the InputData's
output file.
* Tagging as SFrame-03-04-15
2011.06.03 Attila Krasznahorkay <[email protected]>
* Disabling the usage of TTreeCache when running in PROOF mode.
This is a temporary measure to avoid crashes on MC10b D3PDs.
But obviously this will cause a significant performance
decrease, so a proper fix will have to be found quickly!
* Tagging as SFrame-03-04-14
2011.05.24 Attila Krasznahorkay <[email protected]>
* Fixing a crash on the NYU Tier3. The previous cleanup
method for PROOF doesn't seem to work nicely with
ROOT 5.28c installed on the US Tier3s...
* Tagging as SFrame-03-04-13
2011.03.23 Attila Krasznahorkay <[email protected]>
* Fixing a compilation warning on MacOS X that was introduced
with the last update.
* Tagging as SFrame-03-04-12
2011.03.18 Attila Krasznahorkay <[email protected]>
* Made the temporary file creation in LOCAL running more a
little smarter. Now the code always creates unique file
names, making it possible to run multiple jobs in the
same directory.
* Added a protection to the DeclareProperty(...) functions
which will print an ERROR message if the user tries to
use the same property name in multiple places by
mistake.
* Tried to get rid of some memory leaks, but apparently
there are still a few of them... :-(
* Tagging as SFrame-03-04-11
2011.02.21 Attila Krasznahorkay <[email protected]>
* Modified Makefile.common so that it wouldn't re-create the
dependency files of the modified source files when the
user calls "make clean" or "make distclean". This was not
a problem per se, be was useless.
* Modified SLogWriter so the ALWAYS messages would be shown
with the default terminal colors from now on. Beforehand
they were always shown with black characters. But this
created a problem when people were using a terminal with
black background.
* Added a few lines of output to FirstCycle::BeginCycle() that
demonstrate how messages of different message levels look
like on the terminal.
* Tagging as SFrame-03-04-10
2011.02.18 Attila Krasznahorkay <[email protected]>
* Added a little hack to SCycleBaseExec so that it would
only call BeginInputFile(...) once at the beginning of the
job when running in LOCAL mode. Previously this function
was always called twice for the first file in this mode.
* Tried to check in all running scenarios that the modification
is not disruptive.
* Tagging as SFrame-03-04-09
2011.02.14 Attila Krasznahorkay <[email protected]>
* Fixed a previous limitation of the code, that it would
only process a maximum of 1E8 events if "EvMax" was set
to "-1" in the configuration. Now the code uses the
maximal number that can be expressed with Long64_t.
* Tagging as SFrame-03-04-08
2011.02.14 Attila Krasznahorkay <[email protected]>
* Made the type checking in the "primitive version" of
ConnectVariable(...) more permissive. The previous
implementation was very much based on the conventions
of ATLAS ntuples. With the new one it's now even possible
to use branches which have multiple leaves for multiple
primitives. While this is not used in ATLAS, it's one of the
common ways of creating a TTree.
(See: http://root.cern.ch/root/html528/TTree.html)
* Tagging as SFrame-03-04-07
2011.02.10 Attila Krasznahorkay <[email protected]>
* Added a cleanup function to SCycleBaseNTuple. Justin
Griffiths reported that under certain conditions dangling
TTree pointers can cause problems. Now the code should
always clean up after itself.
* Modified JobConfig.dtd to allow the user to run the same
cycle with different configurations one after the other.
(There's no reason for not allowing this.)
* Tagging as SFrame-03-04-06
2011.02.09 Attila Krasznahorkay <[email protected]>
* Extended SInputData with the current entry number
This can be used to feed the D3PDReader objects with
the information.
* Updated SCycleBaseExec so that it fills this information.
* Fixed a problem in SCycleBaseHist::Hist(...) in which it
could get into an improper state when the user asked for
a non-existent histogram.
* Tagging as SFrame-03-04-05
2011.01.03 Attila Krasznahorkay <[email protected]>
* Modified PARHelpers.py according to the suggestions of
Simon Heisterkamp, slightly improving on his recipe.
* sframe_parMaker.py now creates a unique temporary directory
when making the PAR packages, so there should be no more clashes
between "make processes" compiling the same package
on the same node at the same time.
* Tagging as SFrame-03-04-04
2010.12.03 Attila Krasznahorkay <[email protected]>
* Re-introduced the ClassDef(...) and ClassImp(...) macros
for the SCycleBase component classes. These macros are
necessary if you want to hide some member variables from
the dictionary generation. This is unfortunately a needed
feature for making the code work without WARNING/ERROR
messages in the ROOT trunk. (To become 5.28 soon.)
* Hid some of the SCycleBase variables from the dictionary
generator.
* Tagging as SFrame-03-04-03
2010.12.01 Attila Krasznahorkay <[email protected]>
* Tried to fix the ERROR messages coming from the SCycleBaseConfig
class in ROOT 5.28, but didn't succeed for now. Issue is
followed up on: http://savannah.cern.ch/bugs/?75727
* Added the long missing functionality to be able to retrieve
histograms and other kinds of objects from the input files.
Now the Retrieve<...>(...) function of SCycleBaseHist looks
first in the output object list for the requested object,
and if it doesn't find it there, it looks in the input file.
* Extended the documentation for SSummedVar a bit, stating that
users should always mark such variables as transient when
using new ROOT versions.
* Marked the SSummedVar variables as transient in the FirstCycle
class.
* Added some code to SecodCycle to show off how to use the
Retrieve<...>(...) function for accessing objects from
the input file.
* Now the code now works out of the box with the trunk version of
ROOT, but it still produces some (mostly harmless) ERROR
messages. So for the time being you should stick to ROOT 5.26.
* Tagging as SFrame-03-04-02
2010.11.25 Attila Krasznahorkay <[email protected]>
* Created a new SFrame sub-package as MacOS X wasn't playing nicely with
SFrameCore enabling Cintex at loading time. Somehow the
ROOT::Cintex::Cintex::Enable() command messed up the memory when
sframe_main was starting up.
* This package makes it possible to have sframe_main load the library after
the ROOT services are already up and running in the background.
* Removed the Cintex specific code from the core library.
* Modified the DTD to enable the declaration of some of the XML
elements in random order inside the configuration XMLs.
* Extended the FirstCycle_config.xml example to explicitly
load the SFrameCintex library.
* Tagging as SFrame-03-04-01
2010.11.05 Attila Krasznahorkay <[email protected]>
* Linked the SFrameCore library against the Cintex ROOT library.
It turns out that this ROOT library has been with us since
a long time (it's already in ROOT 5.20). So explicitly
relying on it is not a problem.
* This should solve the issues seen by people using a ROOT
version coupled to an ATLAS offline software release.
* While code-wise the change is small, logically it's quite a big
step. So let's up the "middle tag number".
* Tagging as SFrame-03-04-00
2010.10.27 Attila Krasznahorkay <[email protected]>
* Moved some of the functions of SCycleBaseConfig,
SCycleBaseHist and SCycleBaseNTuple into public scope.
Having them as protected function and declaring all
SToolBase incarnations as friends of these classes was just
not scaling...
* Tagging as SFrame-03-03-20
2010.09.27 Attila Krasznahorkay <[email protected]>
* Re-wrote how ConnectVariable(...) checks whether the user
tries to connect the correct type of primitive to a branch.
The previous implementation used a convention of the ATLAS
D3PDs which is not followed anymore...
* The example SecondCycle configuration uses LOCAL mode once
again. Unfortunately PROOF(-Lite) doesn't play nicely with
relative file paths.
* Tagging as SFrame-03-03-19
2010.09.20 Attila Krasznahorkay <[email protected]>
* Migrated back the change from the development branch that
enables the user to connect polymorphic object pointers to
input ntuple branches.
* Unfortunately the code to write out poymorphic objects in
a smart way to the output is still not fully solved. (I can't
get the sub-branch names to inherit the name of the main branch
as prefix correctly...)
* Tagging as SFrame-03-03-18
2010.09.06 Attila Krasznahorkay <[email protected]>
* sframe_input.py should now be able to handle any number of
input files. (Fixed SFrameHelpers.py so that each file gets
closed after it's not needed anymore.)
* Tagging as SFrame-03-03-17
2010.08.20 Attila Krasznahorkay <[email protected]>
* Changed the order of the TTree::SetBranchAddress(...) and
TTree::SetBranchStatus(...) calls in the ConnectVariable(...)
functions. Apparently putting them in the wrong order can
disable the reading of the branches completely... :-/
* As it turns out, the previous two tags are broken because of
this, don't use them!
* Fixed the description of the x-section parameter in
sframe_input.py.
* Tagging as SFrame-03-03-16
2010.08.16 Attila Krasznahorkay <[email protected]>
* Put the function calls operation on the cache inside #if
statements, as these function calls are not available in old ROOT
releases. So SFrameARA would not be able to compile with them.
* Tagging as SFrame-03-03-15
2010.08.16 Attila Krasznahorkay <[email protected]>
* Made the GetInputTree(...) and GetOutputTree(...) functions
of SCycleBaseNTuple protected. This should help users in some
special circumstances.
* Started playing with TTreeCache, but not much effect on the
performance so far. Will talk with Gerri next week (when he's
back from vacation) to see how caching could be used effectively
in SFrame.
* Put a security check into FirstCycle::EndMasterInputData(...).
This way when the worker nodes die, the master process is
still able to collect the messages from the workers.
* Tagging as SFrame-03-03-14
2010.08.16 Attila Krasznahorkay <[email protected]>
* Added the ability to sframe_input.py to ignore a hyphen
("-") in the argument list. One needs to give "-" as the
first argument to the script on some GRID sites, to ignore
a nasty PyROOT bug.
* Not tagging yet, as more updates should also come soon.
2010.08.06 Attila Krasznahorkay <[email protected]>
* Modified sframe_input.py a bit. Now it should be able to receive
any kind of input filenames. (So the filenames don't have to point
to locally accessible files.)
* Added the possibility to sframe_input.py to receive a comma separated
list of input filenames. This is a nice feature when running an
SFrame job with prun.
* Pimped up the python scripts so that they now return proper exit
codes. So an automated script that calls them can know when
something has gone wrong.
* Modified SToolBase to be able to extend it more easily in SFrameARA.
Also modified the friend relations in some core classes because
of this change.
* Tagging as SFrame-03-03-13
2010.08.04 Attila Krasznahorkay <[email protected]>
* Added a few non-const Write(...) functions to various classes.
This is to get rid of some of the overloaded virtual warnings
of GCC. However I still couldn't get rid of all these warnings.
TObject has a function called ExecuteEvent(...) and even if I
remove the warning from the compilation of SCycleBaseExec, the
users would still have to modify their cycles as well.
So for now I leave the flag that turns the overloaded virtual
warning off active in the Makefile.
* Not tagging at the moment, as the added modifications really
don't change or fix anything.
2010.08.02 Attila Krasznahorkay <[email protected]>
* Modified the friend-relations of SCycleNTupleBase to make
it possible to use SOutputVariables-derived classes with this
template parameter.
* Created the new SOutputVariables class, which works in an
analogous way to SInputVariables.
* Fixed the friend relations of SToolBase to make it possible
to use SInputVariable and SOutputVariable classes with such
a parent as well.
* Tested with some simple code that the new class seems to work.
* Tagging as SFrame-03-03-12
2010.06.30 Attila Krasznahorkay <[email protected]>
* Re-designed the SInputData class in some details. Now the
types of input and output trees can be extended by the
libraries built on top of SFrame. (Like SFrameARA.)
* Modifed all the code handling the SInputData class (basically
all SFrame classes...) to use the new "formalism".
* Added some macros to SLogger.h to help with printing
verbose and error messages with more information on
where they're coming from in the code.
* Added a new helper class, STreeTypeDecoder that is used
by the framework so that it doesn't have to know about all
the possible tree types in the base library.
* The plugin helper classes are now using the message macros from
SLogger.h.
* Updated FirstCycle to show off some of the new features of
the framework.
* Modified JobConfig.dtd to adhere to the new design of
SInputData.
* Tagging as SFrame-03-03-11
2010.06.21 Attila Krasznahorkay <[email protected]>
* Fixed the long-standing double-event-issue in the output
TTree-s...
* SFileMerger was modified to take it into account that a single
object might be present in the input files with multiple
versions/cycles. In this case, since the code always asks for
the latest version of the object for the merging, it has to
process that object only once. Previously it processed every
object (TTree) as many times as they appeared in the list of
keys in the input file. (Most of the time twice in some older
versions of ROOT.)
* Tagging as SFrame-03-03-10
2010.05.20 Attila Krasznahorkay <[email protected]>
* Added a new property to SInputData: SkipValid
When this property is set to kTRUE, the validation of the files
(datasets) in that SInputData is skipped. This can be very
useful when running on a lot of individually declared files
on a remote PROOF farm. (Since the input files are remote,
reading from all of them can take a *long* time.)
* The property can only be used when all the events from the
InputData are to be processed. If the configuration specifies
that only a subset of events should be processed, the
validation is forced. (So that the event weigthing can be
correct.)
* Added the SkipValid property to the InputData definitions
in the DTD
* Added some explanation about this property to
FirstCycle_config.xml
* Tagging as SFrame-03-03-09
2010.05.12 Attila Krasznahorkay <[email protected]>
* Fixed yet another typo in CycleCreators.py. Thanks to
Clemens Lange for spotting it.
* Tagging as SFrame-03-03-08
2010.05.12 Attila Krasznahorkay <[email protected]>
* Made the code aware that ROOT version "5.26/00-proof" can also
handle multiple datasets per query (per InputData).
* Tagging as SFrame-03-03-07
2010.05.05 Attila Krasznahorkay <[email protected]>
* Changed the SH1::Fill(...) function to throw an exception
when it receives a NaN value. (Instead of plainly crashing.)
This should make debugging a little easier, as the code now
reports which histogram received the NaN value.
* Tagging as SFrame-03-03-06
2010.05.03 Attila Krasznahorkay <[email protected]>
* Added some fixes after trying to run some "real" jobs on
PROOF (PQ2) datasets.
* SCycleController now automatically specifies to PROOF which
TTree the cycle should run on. If it's not specified for a
D3PD file for which an event selection was applied in Athena
(CollectionTree has more entries than the D3PD tree), then the
job crashes.
* Disabled the checking of the metadata trees when running on a
dataset. The current implementation of PROOF doesn't catalog
the TTree-s inside subdirectories of a file, so the PROOF
dataset doesn't know about these metadata trees. This cataloging
should be added in a future version of PROOF.
* Tagging as SFrame-03-03-05
2010.04.30 Attila Krasznahorkay <[email protected]>
* Added first implementation to support PROOF (PQ2) datasets in
the InputData.
* Created a new singleton class (SProofManager) to handle the
PROOF connections. Moved some of the functionality that was
previously in SCycleController, to this new class.
* Added a new class to SInputData (SDataSet), and updated all the
classes to know about this new member of the input data.
* Removed the dictionary generation for SCycleController. This
was left in the code by mistake, there is no need to have such
a dictionary.
* Revived the <DataSet> node in the DTD which I actually left in
it the last time I tried to implement the support for
PROOF (PQ2) datasets.
* Created a small script (sframe_clear_server.py) that can be used
to clear a PROOF server of all the uploaded PAR packages (of the
given user). By default packages are only updated on the server
if they change locally. But this mechanism doesn't like it when the
user removes a source file locally from a package. That source
file will remain on the server after the next update. If this happens,
this new script can be used in an easy way to "reset" the server.
* Made sframe_dset_maker.py more usable, by making sure that files
hosted on an xrootd server can indeed be added to datasets.
* Tagging as SFrame-03-03-04
2010.04.30 Attila Krasznahorkay <[email protected]>
* Updated the Doxygen documentation a bit.
* Not tagging for the moment...
2010.04.27 Attila Krasznahorkay <[email protected]>
* Added a check in SCycleBaseNTuple::ConnectVariable(...) that makes
sure that the user tries to connect the correct type of variable
to a branch. Previously it was possible to try to connect a
primitive variable (bool, int, etc.) to a branch holding let's
say a vector<double> variable. In this case ROOT crashed in a
quite undecipherable way in the ConnectVariable(...) function.
This should not happen anymore, the code will exit with an error
message if the user makes this mistake.
* Tagging as SFrame-03-03-03
2010.04.13 Attila Krasznahorkay <[email protected]>
* Added a new configuration option to the XML files: <MetadataTree>
Using this the user can access additional TTree-s in the input
files which don't hold event-level data. The used case for this
right now is to be able to read the trigger configuration
metadata from the D3PD ntuples.
* Added a new function in SCycleBaseNTuple to give access to the
metadata trees.
* The metadata trees can also be in subdirectories inside the file,
just like it is done with the trigger configuration metadata
right now.
* Updated the DTD in the example to know about the new <MetadataTree>
input type. Removed the EVInputTree parameter from the DTD, which
is not supported by the code since a while.
* Modified the source code created by sframe_create_cycle.py a little
bit once again.
* Tagging as SFrame-03-03-02
2010.04.09 Attila Krasznahorkay <[email protected]>
* Fixed the way SCycleBaseConfig handles the case when it receives
an empty string as a vector property. (Now the std::vector remains
empty.)
* Tagging as SFrame-03-03-01
2010.04.08 Attila Krasznahorkay <[email protected]>
* Created a non-const accessor to the SCycleConfig object
of the cycle. This can apparently be used to modify the total
luminosity of the cycle for the re-weighting in a nicer fasion,
but that doesn't work with PROOF yet...
* Added a new feature to SCycleBaseConfig: It is now possible to
add and retrieve TObject based "configuration objects" to/from
the cycle. The model is that the user creates/initializes a
configuration object in BeginCycle(), and adds it as a configuration
object with AddConfigObject(...). Then the user can retrieve
this same object inside the function which are executed
on the worker nodes, with GetConfigObject(...). The first example
is sending a TGoodRunsList object to the workers, to select
the good luminosity blocks out of ATLAS data.
* This feature needed some modification in SCycleBaseExec, and
SCycleController as well.
* Added some "friend ..." lines to the cycle base classes to make