Skip to content
Snippets Groups Projects
Commit 5d25eb52 authored by Gerardo Ganis's avatar Gerardo Ganis
Browse files

Update PROOF release notes 

git-svn-id: http://root.cern.ch/svn/root/trunk@31700 27541ba8-7e3a-0410-8455-c3a389f83636
parent 353d003a
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
proof/doc/v526/index.html
+ 198
− 5
View file @ 5d25eb52
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
</head><body>
<title>PROOF</title></head>
<body>
<br>
<hr> <a name="proof"></a>
<h3>Proof</h3>
......@@ -29,6 +30,94 @@ created on the worker nodes by worker processes. The implementation is
an extension of the functionality of the class TProofOutputFile used
for merging via file. See
http://root.cern.ch/drupal/content/handling-large-outputs-root-files</li>
<li>Add the possibility to enable/disable the tree cache and
to change its size on per-query base; two new parameters are available:</li>
<ul>
<li>PROOF_UseTreeCache&nbsp;&nbsp;&nbsp;
Int_t&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Enable (0) or Disable (1) the tree cache (default 1)</li>
<li>PROOF_CacheSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Long64_t&nbsp;&nbsp;&nbsp;&nbsp; Cache size in bytes
(default 10000000)</li>
</ul>
Examples:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
a) to disable the cache for the next run enter:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
proof-&gt;SetParameter("PROOF_UseTreeCache", 0)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
b) to set the cache size to 20M<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
proof-&gt;SetParameter("PROOF_CacheSize", 20000000)<br>
<li> &nbsp;Add the parameter
PROOF_UseParallelUnzip&nbsp;to toggle the use of the parallel unzip
(default off for now); to enable it add the following call<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;proof-&gt;SetParameter("PROOF_UseParallelUnzip", 1)
</li>
<li>&nbsp;Add the possibility to give indications about
the number of workers at startup.<br>
&nbsp;E.g.<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1. To
start max 5 workers<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
TProof::Open("&lt;master&gt;","workers=5")<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2. To
start max 2 workers per physical machine<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
TProof::Open("&lt;master&gt;","workers=2x")<br>
&nbsp;&nbsp;&nbsp;&nbsp; This is useful in general when
running tests (equivalent but quicker then full startup<br>
&nbsp;&nbsp;&nbsp;&nbsp; followed by
TProof::SetParallel(n) or TProof::DeactivateWorker(...)).</li>
<li>Add support for the worker SysInfo_t in TSlaveInfo
(obtained via TProof::GetListOfSlaveInfos())</li>
<li>Add new submerger functionality to speed up the merging
phase. At the end of the query, a set of workers are promoted
submergers and assigned a sub-set of workers to merge. Once each
sub-merger has merged its sub-set of workers, it sends its result to
the master, which merges the partial results into the final<br>
set of results.<br>
The determination of the sub-mergers is always done dynamically, based
on the recent performance of workers. An optimal (i.e. giving the
highest speed-up) number can be calculated analytically under simple
assumptions.<br>
Merging via submergers is by default disabled. To enable it, with the
optimal number of sub-mergers, one should set the integer parameter
'PROOF_UseMergers' to 0, i.e.<br>
<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
proof-&gt;SetParameter("PROOF_UseMergers", 0)<br>
<br>
To force S sub-mergers (regardless of the optimal number) do<br>
<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
proof-&gt;SetParameter("PROOF_UseMergers", S)</li>
<br>
The new functionality can be tested in tutorials by adding the argument
'submergers' to runProof, e.g. <br>
<br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;root [0] .L
tutorials/proof/runProof.C+ <br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;root [1]
runProof("simple(nhist=10000,submergers)") <br>
<br>
(see the top of tutorials/proof/runProof.C for additional options).
<br>
A test for the submerger functionality has also been added to
test/stressProof.cxx .
<li>In PROOF-Lite, add the possibility for the administrator
to control the number of workers. This is done using
the&nbsp;rootrc variable ProofLite.MaxWorkers, which is read out of
/etc/system.rootrc and cannot be overwritten by users. Setting the
value to 0 disables PROOF-Lite.</li>
</ul>
<li>Improvements</li>
<ul>
......@@ -47,7 +136,7 @@ hadd).</li>
</ul>
<li>TProofOutputFile</li>
<ul>
<li>Add support for the placeholded &lt;file&gt;
<li>Add support for the placeholder &lt;file&gt;
the definition of the outputfile. This allows to have complete URL and
to pass options to TFile::Open.</li>
</ul>
......@@ -58,6 +147,57 @@ session rootrc file if the ROOT version is &lt; 5.24/00 ; this acts
as a workaround for the wrong TTreeCache initialization at the
transition between local and remote files fixed in 5.24/00 .</li>
</ul>
<li>Enable mass storage domain settings when working with
TChain's
in multi-master mode. The Mass Storage Domain must be specified as
option in the URL<br>
<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
chain.AddFile("root:// .....?msd=CERN")<br>
<br>
&nbsp;and the string must match the value specified in defining the
submaster node.</li>
<li>Improved performance monitoring: the 'Rate plot' button
in the dialog box has been renamed 'Performance Plot' and now shows up
to 4 plots as a function of the processing time:</li>
<ul>
<li>Instantaneous processing rate, which is now better
estimated by a better estimation of the normalizing times</li>
<li>Average read chunck size, defined as
TFile::GetFileBytesRead() / TFile::GetFileReadCalls() during the last
unit of time; this allows to monitor the usage of the cache; this plot
is present only if some I/O is done, i.e. not for pure CPU tasks.</li>
<li>The number of active workers</li>
<li>The number of total and effecive sessions running
concurrently on the cluster (started by the same daemon); this plot is
present only is the number is at least onec different from 1.</li>
</ul>
See also http://root.cern.ch/drupal/content/progress-dialog .
<li>If enabled, send monitoring information from the master
at each GetNextPacket (at each call of TPerfStat::PacketEvent) to allow
extrnal real-time progress monitoring.</li>
<li>Save the status of a 'proofserv' session into a new file
in the 'activesessions' area. The full path of the new file is<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&lt;admin_path&gt;/.xproofd.&lt;port&gt;/activesessions/&lt;user&gt;.&lt;group&gt;.&lt;pid&gt;.status<br>
<br>
The status indicates whether the session is idle, running or queued.<br>
The status is updated every 'checkfq' secs (see xpd.proofservmgr;
default 30 s). The status is dumped by the reader thread of TXProofServ
and therefore its r/w access is protected.<br>
</li>
<li>Enable the use of the tree cache also for local files,
adapting the default settings for the cache to the recent changes</li>
<li>In the XrdProofd plug-in</li>
<ul>
<li>Improve synchronization between parent and child during
fork</li>
<li>Optimize loops over directory entries</li>
<li>Improve error and notification messages</li>
</ul>
<li>Improved handling of Ctrl-C; this follows from a fix in
TMonitor and an improved handling of non-finished query state in the
workers (results are not send to master if the query was aborted) </li>
</ul>
<li>Fixes</li>
<ul>
......@@ -69,6 +209,10 @@ non-mergeable objects (fixes bug #52886)</li>
MergeRecursive</li>
<li>Do not remove the first file in the list when returning
from MergeRecursive (fixes bug #54591)</li>
<li>Fix a major leak when merging files with collections
written using kSingleKey option. &nbsp;The merger was reading each
key in memory and deleted the object at the end, but the container is
not owner by default, so all objects inside leaked.</li>
</ul>
<li>PROOF-Lite</li>
<ul>
......@@ -87,9 +231,16 @@ outputfile just after having open it</li>
</ul>
<li>XrdProofd plugin</li>
<ul>
<li>&nbsp;Make sure that the limit on the number of old
<li>Make sure that the limit on the number of old
sessions is applied whenever a new session is started and not only when
the daemon is started.</li>
<li>Fix the behaviour of the xpd.allowedusers directive: if
at least one of these directives is present, users in the password file
are not allowed by default but must be explicitly appear in one
xpd.allowedusers directive &nbsp;</li>
<li>Fix a source for memory leak in
XrdProofdProtocol::SendMsg</li>
<li>Optimize the usage of strings in a few places</li>
</ul>
<li>DataSet manager</li>
<ul>
......@@ -107,7 +258,49 @@ histogram in PolyMarker3D</li>
creation of one pad per object</li>
<li>Add wrapper to handle the feedback default canvas</li>
</ul>
<li>TEventIter</li>
<ul>
<li>Fix a problem with changing the tree cache size: the
size was reset to the default value after the first file.</li>
</ul>
<li>TDataSetManagerFile</li>
<ul>
<li>Solve a consistency problem in checking URLs for
duplication when adding them to the relevant TFileInfo</li>
<li>During dataset validation, do not fail on duplications
but notify and add them to the bad file list</li>
</ul>
<li>TPacketizerAdaptive, TPacketizer</li>
<ul>
<li>Improve data node / worker matching by always using the
host FQDN</li>
</ul>
<li>TPacketizerUnit, TEventIter</li>
<ul>
<li>Make sure that the entry
number passed to TSelector::Process is unique and in increasing order
for non-data driven processing (packetizer TPacketizerUnit). This
allows to give a meaning to this variable, for example to related it to
one dimension of an integration.</li>
</ul>
<li>Fixes in PROOF-Lite:</li>
<ul>
<li>Make sure that with envs settings via TProof::AddEnvVar
are effective; this enables, for example, the automatic valgrind setup
introduced in 5.24/00 or the experiment specific settings via the
script defined by the env PROOF_INIT</li>
<li>Fix a problem with TProof::Load so that now it can be
also be used for PROOF-Lite</li>
</ul><li>TProofPlayerRemote</li><ul><li>In SendSelector, add misisng
option kCpBin when sending the selector source; the binary files were
&nbsp;never retrieved, even if present and valid</li></ul><li>TProofPlayerSlave</li><ul><li>In
Process, fix a problem with&nbsp;cache directory locking while building
the selector; &nbsp;the net effect was that each worker process was
re-buidling its own selector binary. </li></ul><li>TProofServ</li><ul><li>Fix
the order in which the log file is sent in asynchronous processing; the
wrong order was screwing up an immediate synchronous query submission
after an asynchronous run; this case occured, for example, in
'stressProof' .</li></ul>
</ul>
</ul>
</body>
</html>
</body></html>
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment