With the concept of Object-Oriented Programming (OOP) and hybrid parallelism, the TOUGH program structure was carefully redesigned, and the codes were rewritten with Fortran 90 and C++. In TOUGH4, EOS modules were designed as Fortran class objects. Through the use of class pointers, EOS modules can be dynamically linked in the TOUGH4 executable during run-time and simulation of different EOS modules using the same executable is realized. A task-based coarse-grained parallelism was implemented through domain decomposition. In addition, the data-based fine-grained parallelism was developed for the heavy computation components of the program. Assembly of Jacobian matrix and EOS calculation are two most important parts for TOUGH simulation. They were completely rewritten for better performance and well organized. All the new development in TOUGH4 is to improve its efficiency in memory usage, computing performance, software readability and extensibility, and reuse of the codes.
The concepts, underlying physics, assumptions, and governing equations are still the same as previous versions of TOUGH. The governing equations for multiphase fluid and heat flow have the same mathematical form, regardless of the nature and number of fluid phases and components present. Based on this recognition, TOUGH is set up with a modular architecture (Figure 10), including modules for handling the input and output, setting up the mass and energy balance equations, EOS calculations, and solving the strongly coupled nonlinear algebraic equations using Newton-Raphson iterations. At each time step, the main flow and transport module interfaces with the EOS module (to obtain secondary parameters, which are thermophysical properties, as a function of the primary variables), and evaluates mass accumulation, flow, and sinks/sources terms (and their derivatives) for a given set of primary variables, which are the variables to be directly solved. In TOUGH4, we improve original TOUGH core module computation procedure by using a more efficient approach for both data-driven parallel computing and memory usage. In the new approach, the computation procedure is organized in a model-element oriented big loop. The loop goes through all the elements one by one for computing each element secondary parameters from its corresponding primary variables and shifted primary variables (needed for derivative calculation), evaluating mass accumulation, flow, and sinks/sources terms for current element, and then assembling Jacobian matrix with the contribution from current element (Figure 11).
The coding of TOUGH3 or earlier version is structured around two large arrays which hold, respectively, the primary thermodynamic variables for all grid blocks (X), and the secondary thermophysical parameters (PAR) which are needed to assemble the governing flow and transport equations. The array PAR is also for the storage of secondary parameters calculated from shifted primary variables (needed for derivative calculation). It can be extremely large. These arrays are one-dimensional and hold data for all grid blocks of the computational mesh in sequential fashion. The solution of a flow problem essentially consists of a complete set of thermodynamic variables as a function of time. However, the storage of the secondary parameters from shifted primary variables is not necessary anymore in the new approach. This leads to significant saving of memory (only 1/(NumPriV+1) of the PAR is needed in new approach, NumPriV is the number of primary variables at each element). The new approach with a large loop is favored by data-driven parallelism and required by OPENMP or GPU directives-based parallel computing. This portion of the computation consists of a significant part of the whole TOUGH simulation. The new approach was implemented by replacing the complex MULTI subroutine in the previous version of TOUGH with several simple subroutines and redesigning the EOS calculation with subroutines for single mesh element.
