Table of Contents generated with DocToc

Run VBFInv rel20.7 code
myRel21Tempalte
Run VBFInv code in rel21
- Every login in lxplus
- First time
Tips
- cross sections

Ntuple production from AOD to MiniNtuple

Typical Production

echo "kinit othrif@CERN.CH"
export vNEW="vXX"
export vOLD="vYY"
export AB="AB21p2pXX"
cd /nfs/dust/atlas/user/othrif/vbf/versions_myVBF
mkdir ${vNEW} && cd ${vNEW}
git clone https://:@gitlab.cern.ch:8443/VBFInv/STAnalysisCode.git
source STAnalysisCode/VBFInvAnalysis/scripts/setupRelease.sh
cd STAnalysisCode/VBFInvAnalysis/data/
for i in $(ls v*); do new=`echo $i | sed -e "s/${vOLD}/${vNEW}/g"`; mv $i $new; done
cd ../arguments/
for i in $(ls v*); do new=`echo $i | sed -e "s/${vOLD}/${vNEW}/g"`; mv $i $new; sed -i 's/${vOLD}/${vNEW}/g' ${new}; done
cd ../../../../run/
runVBF.sh ${vNEW} pilot
nohup runVBF.sh ${vNEW} mc16a_EXOT5  > nohup_mc16a.out &
nohup runVBF.sh ${vNEW} data15_EXOT5 > nohup_data15.out &
nohup runVBF.sh ${vNEW} data16_EXOT5 > nohup_data16.out &
nohup runVBF.sh ${vNEW} data17_EXOT5 > nohup_data17.out &
cd ../STAnalysisCode
git add .
git commit -m "Preparing ${vNEW} submission"
git tag -a ${vNEW}-${AB} -m "Tag version ${vNEW}"
git push origin  ${vNEW}-${AB}

Step1: Preparation

Produce list of DAODs from AODs

To configure the script starting at Line 57 to:

Select which DSIDs to list in the OrderedDict inside listSamples.py

Determine the p-tags for data and MC you want to select Then run:

lsetup rucio
python $TestArea/../STAnalysisCode/VBFInvAnalysis/scripts/listSamples.py -t mc16_13TeV,data15_13TeV,data16_13TeV -c mc16a -d DAOD_EXOT5 -v v05 -s

Clean DESY-HH_LOCALGROUPDISK

Step2: Run Ntuple maker algorithm on DAODs (nominal, systematics, truth)

Prepare arguments

Modify file in data/arguments/v05_args.txt, section called TO MODIFY

Run with a pilot

runVBF.sh v05 pilot

Full production for data years and mc campaigns of interest

runVBF.sh v05 mc16a_EXOT5
runVBF.sh v05 data15_EXOT5
runVBF.sh v05 data16_EXOT5

Step3: Babysit jobs in the grid (finished, failed, aborted, broken)

finished and failed

To retry finished: pandamon | awk '$1 ~ /finished/ {print $2}' | panda-resub-taskid
To retry failed: pandamon | awk '$1 ~ /failed/ {print $2}' | panda-resub-taskid
aborted and broken

Make a list of DAODs to replicate

pandamon user.othrif  | egrep 'aborted|pending' | awk -F"." '{print $4}'> dsid_rep.list
for i in $(cat dsid_rep.list ); do echo $i; cat listProduction/v04/TRUTH3.list | grep $i; done > dup.list
gridDuplicate.sh dup.list

Replicate from pattern

ruciolist.sh user.rzou v08.*MiniNtuple.root | gridDuplicate.sh

Re-proccess with

Step4: Run post-processing code on MiniNtuples at DESY

Duplicate files to the NAF

python ~/bin/createMyGridLinksToDESY.py  *.v05.*MiniNtuple.root -s user.othrif -u othrif -o /nfs/dust/atlas/user/othrif/samples/MiniNtuples/v05
python ~/bin/createMyGridLinksToDESY.py  *.v05.*_hist.root -s user.othrif -u othrif -o /nfs/dust/atlas/user/othrif/samples/MiniNtuples/v05

Condor

Step5: Run plotting code on MicroNtuples at DESY

Run in condor

condor
cd vbf/myPostProcessing/build
ls -d /nfs/dust/atlas/user/othrif/samples/MiniNtuples/v04/*MiniNtuple > list
submitVBFAnalysisCondor.py -l list -n
condor_q

Merge output

mkidr -p merged
hadd merged/data.root data*root
hadd merged/W.root W*root
hadd merged/Z.root Z*root
hadd merged/ttbar.root ttbar*root
hadd merged/signal.root *H125*root

Procedure to submit jobs in the grid

For production version: vXX
Prepare the list of Rucio datasets you want to run over and put it in: /nfs/dust/atlas/user/othrif/myBookKeeping/ntupleProduction/listProduction/vXX with the NAME list_vXX_<type=TRUTH3,EXOT5>_<description>.txt
Modify the arguments list using the template args_${version}.txt in /nfs/dust/atlas/user/othrif/myBookKeeping/ntupleProduction/argsParser/vXX with the NAME args_vXX_<type=TRUTH3,EXOT5>_<description>.txt
Run the runVBF.py with the log redirected to /nfs/dust/atlas/user/othrif/myBookKeeping/ntupleProduction/logProduction with the name log_vXX_<type=TRUTH3,EXOT5>_<description>.txt runVBF.py @./argsParser/args_vXX_Truth3_Znunu.txt --inputRucioLists=./listProduction/list_vXX_Truth3_Znunu.txt -s submitDir_vXX_Truth3_Znunu 2>&1 | tee log_vXX_Truth3_Znunu.txt
Once everyting is ready run with this wrapper script that ensures all the above is true:
```
runVBF.sh vXX <type=TRUTH3,EXOT5>_<description>
```

In general, I would need the following:

version=Vxx
runVBF.sh ${version} EXOT5_mc16a
runVBF.sh ${version} EXOT5_mc16c
runVBF.sh ${version} EXOT5_mc16d
runVBF.sh ${version} EXOT5_data15
runVBF.sh ${version} EXOT5_data16
runVBF.sh ${version} EXOT5_data17

VBF eos area

/eos/atlas/atlascerngroupdisk/phys-exotics/jdm/vbfinv/

Run VBFInv rel20.7 code

To run on MC:

setupATLAS
git clone --recursive https://:@gitlab.cern.ch:8443/VBFInv/AnalysisCode.git
cd AnalysisCode
rcSetup Base,2.4.37
cp /nfs/dust/atlas/user/othrif/vbf/framework_rel20p7/needed/prw_mc15c_170410_withNNPDFsignals.root VBFInv/data/.
cp /nfs/dust/atlas/user/othrif/vbf/framework_rel20p7/needed/vbf20p7.py VBFInv/scripts/.
rc find_packages
rc clean
rc compile
xAH_run.py --file $ASG_TEST_FILE_MC --config RootCoreBin/user_scripts/VBFInv/vbf20p7.py --mode "athena" --nevents 10 direct

To run on Data: change the flags isMC and m_runOverSkimAOD to:

isMC = False
m_runOverSkimAOD = False

myRel21Tempalte

Code located in [https://gitlab.cern.ch/othrif/myRel21Template][https://gitlab.cern.ch/othrif/myRel21Template]

Every login, do:

cd workarea/myRel21tutorial/build
setupATLAS
asetup --restore
source */setuplsh
make
make && rm -rf ../run/* && AnalysisJob_run.py -o ../run/txx

First time:

# git clone http://:@gitlab.cern.ch:8443/othrif/myRel21Template.git # in naf
ln -s ~/cernbox/myRel21Template myRel21Template # in lxplus
mkdir build run
cd build
setupATLAS
asetup 21.2.21,AnalysisBase
cmake ../myRel21Template/
make
source */setup.sh
AnalysisJob_run.py -i DAOD.pool.root -o output

Run VBFInv code in rel21

cd /afs/cern.ch/user/o/othrif/workarea/vbf_rel21/build
setupATLAS
asetup --restore
source */setup.sh
cd ../run
itr=0 # once
itr=$((itr+1)) && xAH_run.py --file $ASG_TEST_FILE_MC --config ${myVBFrel21_DIR}/data/VBFInv/vbf21.py --nevents=10 --submitDir=test${itr} direct

First time

setupATLAS
git clone --recursive https://:@gitlab.cern.ch:8443/othrif/myVBFrel21.git
#ln -s ~/cernbox/myVBFrel21 myVBFrel21
mkdir build run
cd build/
asetup 21.2.21,AnalysisBase
cmake ../myVBFrel21
make
source */setup.sh
cd ../run
itr=0 # once
itr=$((itr+1)) && xAH_run.py --file $ASG_TEST_FILE_MC --config ${myVBFrel21_DIR}/data/VBFInv/vbf21.py --nevents=10 --submitDir=test${itr} direct

Submit to the grid

xAH_run.py --files samples.list --inputRucio --inputList  --config ${AnalysisCode_DIR}/data/VBFInv/truth.py --submitDir=submitDir_micro prun --optGridOutputSampleName=user.%nickname%.%in:name[2]%.%in:name[3]%.%in:name[6]%.%in:name[7]%_xAH_mar14 --optRemoveSubmitDir=1

Code in Analysis repo:

xAH_run.py --file $ASG_TEST_FILE_MC --config ${AnalysisCode_DIR}/data/VBFInv/vbf21.py --nevents=10 --submitDir=test direct

SUSYTools

Compile SUSYTools with:

kinit othrif@CERN.CH
setupATLAS
lsetup git
asetup 21.2,AnalysisBase,latest # asetup AnalysisBase,21.2.22
git atlas init-workdir https://:@gitlab.cern.ch:8443/atlas/athena.git
cd athena
git atlas addpkg SUSYTools
git fetch upstream
git checkout -b my21.2-2018-YY-XX-test upstream/21.2
cd ..
mkdir build
cd build
cmake ../athena/Projects/WorkDir
gmake
source x86_64-slc6-gcc62-opt/setup.sh

Run VBF Statistical framework for VBF+MET 36/fb

Code used by the analysis team is located here: /nfs/dust/atlas/user/othrif/vbf/myStats36ifb All the inputs of the analysis live in data/ and the fit model lives in workspaces/templateAll. The steering file is vbfhi_125.xml that defines what gets included in the fit model and luminosity uncertainty. The individual files like hvbfhi_MET_S1_125_2016.xml determines details of the fit model for each region 3x(SR, We x(n,p)x(lowSig,highSig), Wmx(n,p), Zee, Zmm) and the correlaction model. For example, <HistoSys Name="JER" has this name in all samples and regions, so JER is correlated everywhere. While <HistoSys Name="QCD_W-ckkw2" is specific for each background and each region. This means that it not correlated across samples nor across regions. Also, the normalization factor for the backgrounds are seperate across processes (Z,W) and across bins (1,2,3). So we find <NormFactor Name="kZ2" for the normalization of Z in the second Mjj bin, etc. The names of the up and down histograms HistoNameLow="QCD_W-pdf.down" HistoNameHigh="QCD_W-pdf.up" correspond to the names of the input root files in data/.

Generate Workspaces

For all systematics, copy the folder allSystTemplate and for statistical only copy noSystTemplate
Change the path to the input root files in hvbfhi_* using the script:
```
changePath.sh
```
Where the input folder is one in L2 <Channel Name="ATLAS_vbfhi_WnEV_Low_S2_125_2016" InputFile="/nfs/dust/atlas/user/othrif/scratch/vbfStat/myVBFStats/data/WnEV_Low_S2.root" > and the output is where the root files are located in absolute path.
setupATLAS and ROOT, then run:
```
hist2workspace vbfhi_125.xml
```
This step creates .root files for each of the regions in the form of a RooStats::HistFactory::Measurement.

Run Asymptotic fit

The command to run is:

root -b -q 'runAsymptoticsCLs.C+(\"hvbfhi_125_2016_combined_allsys_model.root\",\"combined\",\"ModelConfig\",\"obsData\",\"asimovData_0\",\"FOLDERNAME\",\"125\",0.95)'

A script to make life easy is given, called commandToRun.sh and takes a folder name as argument to output a histogrmam with expected and observed limits with bands, just copy and paste to see what you are running. The output looks something like:

Correct bands
+2sigma:  0.369979
+1sigma:  0.279449
-1sigma:  0.146516
-2sigma:  0.109136
Median:   0.203337
Observed: 0.310368

Make Pull Plots

Copy folder myStats36ifb/pullRanking/templatePull into a new directory inside myStats36ifb/pullRanking/

Setup and compile:

lsetup "lcgenv -p LCG_86 x86_64-slc6-gcc62-opt ROOT";
export CC=/cvmfs/sft.cern.ch/lcg/releases/LCG_86/gcc/6.2.0/x86_64-slc6/bin/gcc; export CXX=/cvmfs/sft.cern.ch/lcg/releases/LCG_86/gcc/6.2.0/x86_64-slc6/bin/g++;
mkdir build;
cd build;
cmake .. -DBOOST_ROOT=/cvmfs/sft.cern.ch/lcg/releases/LCG_86/Boost/1.62.0/x86_64-slc6-gcc62-opt -DBOOST_INCLUDEDIR=/cvmfs/sft.cern.ch/lcg/releases/LCG_86/Boost/1.62.0/x86_64-slc6-gcc62-opt/include/boost-1_62 -DBoost_LIBRARY_DIRS=/cvmfs/sft.cern.ch/lcg/releases/LCG_86/Boost/1.62.0/x86_64-slc6-gcc62-opt/lib;
make;

OR run source setup.sh; ./Compile

Copy the fit model file like hvbfhi_125_2016_combined_allsys_model.root to the same area where the code runs.
Run a quick fit (parameters that minimize the likelihood):
```
./bin/fit.exe --input input.root --workspace combined --data asimovData --poi mu
```
like ./bin/fit.exe --input hvbfhi_125_2016_combined_allsys_model.root --workspace combined --data obsData --poi mu
For nuisance impact (correlation), run ./bin/pulls.exe --input hvbfhi_125_2016_combined_allsys_model.root --workspace combined --data obsData --poi mu --parameter alpha_sys where alpha_sys is one of the uncertainties, like ./bin/pulls.exe --input hvbfhi_125_2016_combined_allsys_model.root --workspace combined --data obsData --poi mu --parameter alpha_MET_trigger
Now currently this part i can only run in lxplus since it uses lxbatch to compute the nuisance parameters and correlations: python submit_pulls.py input.root --workspaceName combined --data asimovData --poi mu --folder ranking like python submit_pulls.py hvbfhi_125_2016_combined_allsys_model.root --workspaceName combined --data obsData --poi mu --folder ranking
Then plot ranking plot with ./bin/plot_pulls.exe --input root-files/rankingNew/pulls --poi mu
You can remove the nuisance parameters that are too small, so that you can actually read the text. Also remove k* norm_fakeel* since they are not nuisance parameters but fit parameters. Et voila!!

Tips

cross sections

/cvmfs/atlas.cern.ch/repo/sw/database/GroupData/dev/SUSYTools/xsdb/mc15_13TeV/

Histfitter

 /afs/cern.ch/work/m/mperego/public/histfitter_useful_VBFHinv/configManager_to_change/

Analysis Log

Ntuple production from AOD to MiniNtuple

Typical Production

Step1: Preparation

Produce list of DAODs from AODs

Clean DESY-HH_LOCALGROUPDISK

Step2: Run Ntuple maker algorithm on DAODs (nominal, systematics, truth)

Prepare arguments

Run with a pilot

Full production for data years and mc campaigns of interest

Step3: Babysit jobs in the grid (finished, failed, aborted, broken)

finished and failed

aborted and broken

Step4: Run post-processing code on MiniNtuples at DESY

Duplicate files to the NAF

Condor

Step5: Run plotting code on MicroNtuples at DESY

Run in condor

Merge output

Procedure to submit jobs in the grid

VBF eos area

Run VBFInv rel20.7 code

myRel21Tempalte

Run VBFInv code in rel21

First time

SUSYTools

Run VBF Statistical framework for VBF+MET 36/fb

Generate Workspaces

Run Asymptotic fit

Make Pull Plots

Tips

cross sections

Histfitter

results matching ""

No results matching ""

Ntuple production from AOD to MiniNtuple

Typical Production

Step1: Preparation

Produce list of DAODs from AODs

Clean DESY-HH_LOCALGROUPDISK

Step2: Run Ntuple maker algorithm on DAODs (nominal, systematics, truth)

Prepare arguments

Run with a pilot

Full production for data years and mc campaigns of interest

Step3: Babysit jobs in the grid (finished, failed, aborted, broken)

finished and failed

aborted and broken

Step4: Run post-processing code on MiniNtuples at DESY

Duplicate files to the NAF

Condor

Step5: Run plotting code on MicroNtuples at DESY

Run in condor

Merge output

Procedure to submit jobs in the grid

VBF eos area

Run VBFInv rel20.7 code

myRel21Tempalte

Run VBFInv code in rel21

Every login in lxplus

First time

SUSYTools

Run VBF Statistical framework for VBF+MET 36/fb

Generate Workspaces

Run Asymptotic fit

Make Pull Plots

Tips

cross sections

Histfitter

results matching ""

No results matching ""