Recent Changes - Search:

Background

Workflow

Cluster Operations

https://www.jetphotos.com/photographer/598301 https://www.jetphotos.com/photographer/598304 https://www.jetphotos.com/photographer/598305 https://www.jetphotos.com/photographer/598307 https://www.jetphotos.com/photographer/598310 https://www.jetphotos.com/photographer/598312 https://www.jetphotos.com/photographer/598317 https://www.jetphotos.com/photographer/598318 https://www.jetphotos.com/photographer/598320 https://www.jetphotos.com/photographer/598321 https://www.jetphotos.com/photographer/598322 https://www.jetphotos.com/photographer/598324 https://www.jetphotos.com/photographer/598328 https://www.jetphotos.com/photographer/598340 https://www.jetphotos.com/photographer/598341 https://www.jetphotos.com/photographer/598346 https://www.jetphotos.com/photographer/598349 https://www.jetphotos.com/photographer/598357 https://www.jetphotos.com/photographer/598366 https://www.jetphotos.com/photographer/598372 https://www.jetphotos.com/photographer/598374 https://www.jetphotos.com/photographer/598378 https://www.jetphotos.com/photographer/600028 https://www.jetphotos.com/photographer/600031 https://www.jetphotos.com/photographer/600032 https://www.jetphotos.com/photographer/600034 https://www.jetphotos.com/photographer/600036 https://www.jetphotos.com/photographer/600037 https://www.jetphotos.com/photographer/600039 https://www.jetphotos.com/photographer/600041 https://www.jetphotos.com/photographer/600042 https://www.jetphotos.com/photographer/600045 https://www.jetphotos.com/photographer/600046 https://www.jetphotos.com/photographer/600047 https://www.jetphotos.com/photographer/600048 https://www.jetphotos.com/photographer/600050 https://www.jetphotos.com/photographer/600051 https://www.jetphotos.com/photographer/600052 https://www.jetphotos.com/photographer/600053 https://www.jetphotos.com/photographer/600055 https://www.jetphotos.com/photographer/600057 https://www.jetphotos.com/photographer/600641 https://www.jetphotos.com/photographer/600644 https://www.jetphotos.com/photographer/600645 https://www.jetphotos.com/photographer/600646 https://www.jetphotos.com/photographer/602231 https://www.jetphotos.com/photographer/602240 https://www.jetphotos.com/photographer/602244 https://www.jetphotos.com/photographer/602247 https://www.jetphotos.com/photographer/602261 https://www.jetphotos.com/photographer/602265 https://www.jetphotos.com/photographer/602279 https://www.jetphotos.com/photographer/602307 https://www.jetphotos.com/photographer/602315 https://www.jetphotos.com/photographer/602323 https://www.jetphotos.com/photographer/602340 https://www.jetphotos.com/photographer/602346 https://www.jetphotos.com/photographer/602741 https://www.jetphotos.com/photographer/602743 https://www.jetphotos.com/photographer/602744 https://www.jetphotos.com/photographer/602745 https://www.jetphotos.com/photographer/602746 https://www.jetphotos.com/photographer/602748 https://www.jetphotos.com/photographer/602749 https://www.jetphotos.com/photographer/602750 https://www.jetphotos.com/photographer/602757 https://www.jetphotos.com/photographer/602758 https://www.jetphotos.com/photographer/602762 https://www.jetphotos.com/photographer/602763 https://www.jetphotos.com/photographer/602764 https://www.jetphotos.com/photographer/602769 https://www.jetphotos.com/photographer/602770 https://www.jetphotos.com/photographer/602772 https://www.jetphotos.com/photographer/602773 https://www.jetphotos.com/photographer/602774 https://www.jetphotos.com/photographer/602775 https://www.jetphotos.com/photographer/601186 https://www.jetphotos.com/photographer/601188 https://www.jetphotos.com/photographer/601189 https://www.jetphotos.com/photographer/601191 https://www.jetphotos.com/photographer/601192 https://www.jetphotos.com/photographer/601194 https://www.jetphotos.com/photographer/601196 https://www.jetphotos.com/photographer/601197 https://www.jetphotos.com/photographer/601248 https://www.jetphotos.com/photographer/601249 https://www.jetphotos.com/photographer/601250 https://www.jetphotos.com/photographer/601251 https://www.jetphotos.com/photographer/601252 https://www.jetphotos.com/photographer/601254 https://www.jetphotos.com/photographer/601255 https://www.jetphotos.com/photographer/601256 https://www.jetphotos.com/photographer/601258 https://www.jetphotos.com/photographer/601260 https://www.jetphotos.com/photographer/601261 https://www.jetphotos.com/photographer/601263 https://www.jetphotos.com/photographer/601264 https://www.jetphotos.com/photographer/601265 https://www.jetphotos.com/photographer/601266 https://www.jetphotos.com/photographer/601267 https://www.jetphotos.com/photographer/601268 https://www.jetphotos.com/photographer/601269 https://www.jetphotos.com/photographer/601270 https://www.jetphotos.com/photographer/601272 https://www.jetphotos.com/photographer/601273 https://www.jetphotos.com/photographer/602779 https://www.jetphotos.com/photographer/602780 https://www.jetphotos.com/photographer/602781 https://www.jetphotos.com/photographer/602782 https://www.jetphotos.com/photographer/600111 https://www.jetphotos.com/photographer/600112 https://www.jetphotos.com/photographer/600148 https://www.jetphotos.com/photographer/600151 https://www.jetphotos.com/photographer/600155 https://www.jetphotos.com/photographer/600157 https://www.jetphotos.com/photographer/600159 https://www.jetphotos.com/photographer/600161 https://www.jetphotos.com/photographer/600163 https://www.jetphotos.com/photographer/600647 https://www.jetphotos.com/photographer/600648 https://www.jetphotos.com/photographer/600649 https://www.jetphotos.com/photographer/600650 https://www.jetphotos.com/photographer/602889 https://www.jetphotos.com/photographer/602890 https://www.jetphotos.com/photographer/602891 https://www.jetphotos.com/photographer/602895 https://www.jetphotos.com/photographer/602897 https://www.jetphotos.com/photographer/602900 https://www.jetphotos.com/photographer/602904 https://www.jetphotos.com/photographer/602907 https://www.jetphotos.com/photographer/602913 https://www.jetphotos.com/photographer/602916 https://www.jetphotos.com/photographer/602918 https://www.jetphotos.com/photographer/602922 https://www.jetphotos.com/photographer/602923 https://www.jetphotos.com/photographer/602925 https://www.jetphotos.com/photographer/602926 https://www.jetphotos.com/photographer/600534 https://www.jetphotos.com/photographer/600535 https://www.jetphotos.com/photographer/600536 https://www.jetphotos.com/photographer/600538 https://www.jetphotos.com/photographer/600539 https://www.jetphotos.com/photographer/600540 https://www.jetphotos.com/photographer/600542 https://www.jetphotos.com/photographer/600543 https://www.jetphotos.com/photographer/600544 https://www.jetphotos.com/photographer/600547 https://www.jetphotos.com/photographer/600548 https://www.jetphotos.com/photographer/600549 https://www.jetphotos.com/photographer/600550 https://www.jetphotos.com/photographer/600552 https://www.jetphotos.com/photographer/600553 https://www.jetphotos.com/photographer/600555 https://www.jetphotos.com/photographer/600558 https://www.jetphotos.com/photographer/600565 https://www.jetphotos.com/photographer/600566 https://www.jetphotos.com/photographer/600567 https://www.jetphotos.com/photographer/600568 https://www.jetphotos.com/photographer/600571 https://www.jetphotos.com/photographer/600573 https://www.jetphotos.com/photographer/600575 https://www.jetphotos.com/photographer/600576 https://www.jetphotos.com/photographer/600577 https://www.jetphotos.com/photographer/600578 https://www.jetphotos.com/photographer/600666 https://www.jetphotos.com/photographer/600668 https://www.jetphotos.com/photographer/600669 https://www.jetphotos.com/photographer/600670 https://www.jetphotos.com/photographer/602963 https://www.jetphotos.com/photographer/601276 https://www.jetphotos.com/photographer/601280 https://www.jetphotos.com/photographer/601281 https://www.jetphotos.com/photographer/601284 https://www.jetphotos.com/photographer/601285 https://www.jetphotos.com/photographer/601286 https://www.jetphotos.com/photographer/601287 https://www.jetphotos.com/photographer/601288 https://www.jetphotos.com/photographer/601291 https://www.jetphotos.com/photographer/601293 https://www.jetphotos.com/photographer/602776 https://www.jetphotos.com/photographer/602777 https://www.jetphotos.com/photographer/602955 https://www.jetphotos.com/photographer/602956 https://www.jetphotos.com/photographer/602957 https://www.jetphotos.com/photographer/602959 https://www.jetphotos.com/photographer/602960 https://www.jetphotos.com/photographer/602961

edit SideBar

WhyIsCottontailSoSlow

Because cottontail is the head node, it is constantly running many requests. Users are always logging on, submitting jobs, and manipulating files. However, even if a job is submitted to the scheduler to be sent to a compute node for calculation, it will often increase the burden on cottontail. This is because for our MD simulations, we are writing output periodically throughout the run. Every time this happens, cottontail has to open a connection to the compute node, write the data, and close the connection. For many users with many jobs, this can overwhelm the node.

A potential solution is to use a scratch directory. This entails navigating into an automatically generated job-specific directory on the node where your job is running. You then (in the script) copy over any input files, run your calculations, and write the output on your job node. Once the run finishes, the data can be copied over to a storage node such as mindstore. The following would be an example script using scratch. One consideration is that the localscratch storage for exx96 nodes is only 800GB. The data should therefore be deleted from localscratch once you have confirmed that it copied correctly.

   #!/bin/bash
   #BSUB -e 5JUP_COD1_C2_mR146_err
   #BSUB -o 5JUP_COD1_C2_mR146_out
   #BSUB -q exx96
   #BSUB -J 5JUP_mR146_12to13
   #BSUB -n 1
   #BSUB -R "rusage[gpu4=1:mem=6288],span[hosts=1]"

   # cuda
   export CUDA_HOME=/usr/local/n37-cuda-9.2
   export PATH=/usr/local/n37-cuda-9.2/bin:$PATH
   export LD_LIBRARY_PATH=/usr/local/n37-cuda-9.2/lib64:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH="/usr/local/n37-cuda-9.2/lib:${LD_LIBRARY_PATH}"

   # openmpi
   export PATH=/share/apps/CENTOS6/openmpi/1.8.4/bin:$PATH
   export LD_LIBRARY_PATH=/share/apps/CENTOS6/openmpi/1.8.4/lib:$LD_LIBRARY_PATH

   # python
   export PATH=/share/apps/CENTOS6/python/2.7.9/bin:$PATH
   export LD_LIBRARY_PATH=/share/apps/CENTOS6/python/2.7.9/lib:$LD_LIBRARY_PATH

   # amber18
   source /share/apps/CENTOS7/amber/amber18/amber.sh

   # navigate to auto-generated scratch directory
   MYLOCALSCRATCH=/localscratch/$LSB_JOBID
   export MYLOCALSCRATCH
   cd $MYLOCALSCRATCH

   ###########
   # MD Runs #
   ###########

   # stage input data in scratch
   cp /mindstore/home33ext/kscopino/5JUP_COD1_C2_mR146/* .

   # call Amber18
   n37.openmpi.wrapper pmemd.cuda -O -i 20ps_heat.in  -p 5JUP_COD1_C2_mR146_wat.prmtop -c 5JUP_COD1_C2_mR146_emin11.rst -r 5JUP_COD1_C2_mR146_heat_12.rst
   -ref 5JUP_COD1_C2_mR146_emin11.rst -o 5JUP_COD1_C2_mR146_heat_12.out -x 5JUP_COD1_C2_mR146_heat_mdcrd_heat_12

   # copy results back to storage node
   cp 5JUP_COD1_C2_mR146* /mindstore/home33ext/kscopino/5JUP_COD1_C2_mR146

For now, if traffic is significant it should be possible to submit jobs from cottontail2, the backup submission node. It is also a good idea to look at resource usage with 'bjobs -u all' when slowdowns occur. If a user is running jobs consistently during a slowdown, it is possible that they are not using a scratch directory. This can be checked by ssh-ing into their compute node and looking for their job directory.

Additional Resources
Scratch Directory wiki

Edit - History - Print - Recent Changes - Search
Page last modified on July 14, 2020, at 02:53 PM