LS-DYNA - ls971 SMP Version of LS-DYNA for AIX 5.1, AIX 5.2, and AIX 5.3 =================================================================== Multi-threaded programs, such as ls971 (SMP version), sometimes fail under AIX 5.1. The symptom of such failure s exhibited by all threads going into a suspended state with no progress being made. Effectively, all threads are kept in sleep state. This problem is corrected by applying the following APAR: IY25899 (Release date February 13, 2002) You can verify whether your system already has this APAR applied or not by running the following command: instfix -ivqk | grep -i IY25899 You can also avoid this problem by setting the following environment variable before starting the LS-DYNA job: If you are using k-shell: export AIXTHREAD_SCOPE=S If you are using c-shell: setenv AIXTHREAD_SCOPE S During the QA testing it was observed that by setting the above environment variable the performance has improved significantly. This is because the system uses what is known as 1:1 mapping of user threads to system threads. So, it is recommended that users set AIXTHREAD_SCOPE as described above to realize some performance gain. Note: If you are using only one thread (ncpu=1 or ncpu=-1) the program should not hang and the AIXTHREAD_SCOPE does not have much impact. ===================== END OF AIX 5.1 PTHREAD ALERT ============== The following are the prerequisites to run ls971 on IBM platforms: AIX OPERATING SYSTEM: FORTRAN RUNTIME LIBRARIES AIX OPERATING SYSTEM You need AIX 5.1 or AIX 5.2 or AIX 5.3 to run ls971. You can verify this by typing the following command on the console: oslevel For example, when you type the above command, the system will respond with a message similar to the following: 5.1.1.0 If you have back level AIX, please contact your local IBM Sales Representative to order an upgrade package. FORTRAN RUNTIME LIBRARIES The following ls971 executables are built using xlf 8.1.1.4 libraries: ls971_d_rev#_ibmpwr3_aix51_p ls971_m63_rev#_ibmpwr3_51_p The following ls971 executables are built using xlf 9.1.0.1 libraries: ls971_d_rev#_ibmpwr5_aix53_p ls971_s_rev#_ibmpwr3_aix51_p ls971_s_rev#_ibmpwr5_aix53_p If you have back level IBM XLF FORTRAN, (at or earlier than xlf 8.1.1.3 level) or XLF runtime is not installed at all on your system you have two options: 1) Please contact your local IBM Sales Representative to order either and an upgrade package or a new license for IBM XLF FORTRAN. 2) Please contact LSTC to obtain the runtime XLF libraries used to build this version of LS-DYNA. These runtime libraries can be used to run LS-DYNA even if you either do not have IBM XLF FORTRAN installed on your system or if you have back level IBM XLF FORTRAN on your system. You can verify this by typing the following command on the console: lslpp -L | grep -i Fortran For example, when you type the above command, the system will respond with a message similar to the following: Fileset Level State Description ---------------------------------------------------------------------------- xlfcmp 8.1.1.4 APPLIED XL Fortran Compiler xlfrte 8.1.1.4 APPLIED XL Fortran Runtime xlfrte.aix51 8.1.1.4 APPLIED XL Fortran Runtime Environment If you decide to obtain XLF runtime libraries from LSTC, please read the follwoing README files for further instructions: For XLF 8.1.1.4 Runtime Libraries - see README_xlf91_lib.ibm For XLF 9.1.0.1 Runtime Libraries - see README_xlf91_lib.ibm AIX 5.1 If you are operating under AIX 5.1 please follow the following procedure until further notice: Before you initiate any ls-dyna jobs, please set the following environment variable: If you are using k-shell export AIXTHREAD_SCOPE=S If you are using c-shell setenv AIXTHREAD_SCOPE S When multiple processors are used (ncpu > 1), IBM Fortran creates multiple threads so that each thread can run on a separate processor. When the environment variable AIXTHREAD_SCOPE to S, the AIX OS allocates a separate kernel thread for each user thread created. This results in more efficient scheduling of threads. If this variable is not set before starting jobs under AIX 5.1, LS-DYNA program alerts the user to set the environment variable and exits so that the user can set the environment variable and restart the job. =================================================================== Instructions on running LS-DYNA in parallel on IBM RS/6000 machines In order to obtain the best parallel performance, setting several environment variables is recommended. For example, here's a C shell script showing how to run using four processors: # setenv YIELDLOOPTIME 10000000 setenv SPINLOOPTIME 1000000 setenv LSTC_MEMORY auto # ls971 i=car.k ncpu=-4 The YIELDLOOPTIME and SPINLOOPTIME settings will improve parallel efficiency of LS-DYNA. The LSTC_MEMORY setting will automatically increase the memory size so that the memory doesn't need to be set on the command line. Note that LSTC_MEMORY option is for metal forming application only. Do not use it for the crash analysis. The "ncpu" setting on the LS-DYNA command line will override the number of processors specified in the input deck. Users are strongly recommended to read and understand the flags which control consistency of results (*CONTROL_PARALLEL in the keyword manual, control card 16 in the structured deck manual). In this example, setting ncpu to "-4" will run LS-DYNA on four processors and produce numerically consistent results. Parallel performance will, of course, be greatly affected by other jobs running on the same machine. This is normally controlled by a queuing system such as "LoadLeveler" or cooperation among users of a system. Other site-specific options can also be put in place. 64-bit EXECUTABLE LS-DYNA is made available under 64-bit addressing mode only on AIX 5.1, AIX 5.2, and AIX 5.3. A 64-bit executable created under AIX 5.1 and later versions of AIX do not run on AIX 4.3. Please contact LSTC if you are still operating under AIX 4.3.3. LARGE PAGE SUPPORT The POWER4. processor in the IBM - pSeries POWER4 system supports two virtual page sizes. It supports the traditional POWER architecture 4KB page size. It also supports a new 16MB page size. AIX 5L. for POWER version 5.1 with the 5100-02 Recommended Maintenance package contains support for 16MB size 'large' pages. The procedure describing how applications request large pages to back their virtual storage requirements is given below: 1) Please make sure that the system admin has setup the large page support 2) In the script which contains the ls971 invocation, insert the following environment variable which enables all the commands subsequent to this variable to run under large page support. export LDR_CNTRL=LARGE_PAGE_DATA=Y for korn shell sentenv LDR_CNTRL=LARGE_PAGE_DATA=Y for c chell LDR_CNTRL=LARGE_PAGE_DATA=Y specifies that the exec()ed program should use large pages for its data and heap segments. LDR_CNTRL=LARGE_PAGE_DATA=N specifies that the exec()ed program should not use large pages for its data and heap segments. If the invocation of the ls971 is just a small step in a production job, make sure that the large page support is disabled. Otherwise, all the commands subsequent to the completion of the ls971 program will be run under large page support. An example follows: Korn Shell example: export LSTC_FILE=LSTC_FILE export LDR_CNTRL=LARGE_PAGE_DATA=Y ls971 i=input.dat ncpu=-4 export LDR_CNTRL=LARGE_PAGE_DATA=N C Shell example: setenv LSTC_FILE LSTC_FILE setenv LDR_CNTRL="LARGE_PAGE_DATA=Y" ls971 i=input.dat ncpu=-4 setenv LDR_CNTRL="LARGE_PAGE_DATA=N" 3) Finally, once the program is running, you can verify whether it is running under large page or not as described here: A) Submit the ls971 job as shown in the Step 2) A) Issue the following command which lists processes owned by the who submitted the job in Step A) ps -l -f -u For example, the following command lists all the processes owned by the user id "hari" F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 240001 A hari 557188 377070 0 64 22 92972400 244 16:37:38 - 0:00 timex ../ls971 i=../neon.refined.k ncpu= 260001 A hari 925800 557188 499 96 22 ba8f7400 1080292 16:37:38 - 18:17 ../ls971 i=../neon.refined.k ncpu=8 para 200001 A hari 1204382 499962 0 60 20 e299c400 560 16:40:56 pts/0 0:00 ps -f -l -u hari Note that the process id (PID) for the ls971 job is 925800. Also, note that corresponding value in column F is 260001. If the job is running under large page support, the fifth digit from the right in column F should be one of the following values: 2, 3, 6, 7, A, B, E, or F in the 5th digit (counting from the right) of the first column output from ps is a large page data process In this case it is 6 and hence is running under large page support. NOTE: Users are advised to be cautious in their use of the environment variable to specify large page usage. Performance tests have shown there can be a significant performance loss in environments where a number of shell scripts or small, short running applications are invoked. One example saw a shell script's execution time increase over 10 times when the large page environment variable was specified. Customers are advised to only set the large page environment variable around specific applications that can benefit from large page usage.