Now at the end of its funding, the project's research and development has significantly paid off. The IPARS framework (Integrated Parallel Accurate Reservoir Simulator), along with some of the reservoir models, has been released to affiliates. The project was originally initiated under the auspices of the ACTI Program and funded by the Office of Science. This project also has been highly leveraged by related work at both Argonne and The University of Texas at Austin throughout the its duration.
Summary of significant accomplishments achieved:
Development of a modular structure framework enabling new physics to be studied with only incremental coding. The IPARS framework provides all memory management, well management, message passing, table lookup, input/output, etc., so that the user only need write code for the relevant physics. The framework is designed for portability, and we have run a variety of physical models on PCs (Windows and Linux), workstations, and several parallel platforms with no special adaptation of the code between machines. One of the key features in the design of the IPARS framework is the capability to select application components at compile-time and at run-time. This greatly facilitates the integration of other codes as well as new enabling technologies as they become available. The design paid off handsomely in the last year when the IPARS framework was successfully coupled to several different external software packages. For example, IPARS is now one of the applications available at the NetSolve project. Developed at the University of Tennessee, NetSolve allows the remote launching of applications on participating machines anywhere in the world. The user can conduct all the usual business of reservoir simulation -- edit IPARS input files, submit the job, monitor its progress, view the results -- entirely through a web-browser interface. IPARS was also integrated with the DISCOVER collaborative environment developed at Rutgers University, which enables multiple users to monitor simulation results simultaneously and in real time, as well as to steer the simulation interactively. Successful live demonstrations of the NetSolve/DISCOVER/IPARS system were given at the ACCESS center in Washington DC, the Industrial Affiliates Meeting of the Center for Subsurface Modeling at UT, and at SuperComputing 2000 in Dallas.
Development of a fully implicit equation-of-state (EOS) 3D compositional model, general-purpose parallel adaptive simulator.
Incorporation of the linear solvers from Argonne's PETSc package into the simulator. (Simulator performance and scalability is largely dependent on solver performance.)
The same code can run on a full range of systems from a single PC or workstation, clusters of PCs or workstations, to the highest-end, highest-performance systems available. Simulations are limited only by the size of the problem to be addressed and the system available, and not by fundamental technology constraints.
Excellent simulator performance and scalability on realistic problems demonstrated on clusters of PCs and proprietary parallel computers. The former, especially, puts simulation within the economic reach of the smaller independent producers. For example, in a series of sample problem runs, the time to solution for a 16-processor PC cluster was roughly twice that of a 16-processor IBM SP system (e.g., 1 hour versus 30 minutes), while the cost of a cluster is roughly one-tenth that of an SP system (approximately $40K versus $400K). Comparisons on PC clusters using several types of communication hardware (Ethernet, Myrinet and Giganet) have been made. PC cluster performance typically has been at 70% to 100% of that seen on the proprietary platforms. Test problems on clusters using the equation-of-state (EOS) compositional model included realistic problems with up to 500,000 gridblocks, with up to 13 wells, with up to 6 components, and with both layered and stochastic permeability reservoir descriptions. The largest problem run to date: four million gridblocks and 32 million unknowns in approximately 23 minutes on a 128-processor IBM SP.
Development of multiblock, multimodel domain decomposition approachs for non-matching grids. In the past, reservoir simulators were developed with a focus on a particular physical process, e.g. waterflooding or miscible-gas flooding. As field development strategies become more complex, several recovery processes often occur simultaneously within the same reservoir. Through the development of the multiblock paradigm, we have broken the restraint of traditional "one-process-at-a-time" simulators that cannot adequately couple the different domains in such fields. The IPARS framework permits rigorous, physically representative coupling of different flow models in different parts of the domain. Recently we extended the treatment of the interface between blocks so that different formulations or physical models can exist on each side of the interface in the same run. This multiblock / multiphysics approach is a unique capability of IPARS as it allows for coupling of non-matching grids (in particular, it allows for local mesh refinement). There are two paradigms for multiblock under IPARS: one is called the mortar approach and the other is called "dual". While their functionality is similar, the convergence rates and efficiency may differ. Both the dual and mortar approachs allow for coupling of different physical models. On parallel platforms, we identified some of the critical issues in efficiency including load balancing. Specifically, we considered the cost ratio between the individual models and size of the model subdomains as the main factors determining efficiency. Also, new interface preconditioners appropriate for multiphysics were developed. The efficiency of the multiphysics / multinumerics approach was demonstrated for cases with two physical models, one of which was substantially slower than the other. Tested examples include, a single-phase model with a two-phase model, a two-phase IMPES model with a two-phase implicit model, and a two-phase implicit model with a black oil model. If the faster model occupied a large subdomain compared to the slower, or if the slower model was assigned a higher number of processors, the multiphysics approach ran faster by up to 30% than a single model with no-domain decomposition. The efficiency increased by a factor of 7 times if multiphysics was also combined with local grid refinement.
Implementation of a restart capability for multimodel cases and multiblock geometries. This capability expands the flexibility of the framework, as it allows for a form of adapativity. In particular, one can run multiblock simulations in which the mortar spaces (dimensions and type) can vary between time steps. This can provide a significant improvement in efficiency depending on the location of saturation fronts relative to the block boundaries.
Development of multiple physical models. Considerable effort has been spent on hardening the code, improving its efficiency and robustness. In particular, this includes i) validation of models, ii) progress in multiblock / multiphysics solvers and preconditioners, and iii) multigrid and other preconditioners, and well implementation. Several of IPARS physical models have been validated against existing data, analytical results like Buckley-Leverett, or other codes (Eclipse). Results from the environmental air-water model agree with some laboratory experiments (by J. Touma and M. Vauclin), etc. In another direction, different formulation of the black oil model was tested in which the chosen set of primary variables (Po,No,Ng) is different from the one we were using so far (Pw, No, Ng). The results of this new model allow a unique opportunity to test the siginificance of choice of primary unknowns in models operating in the same environment.
Development of a lightweight visualization option. This has proven very useful for examining details of flow between wells or through extremely heterogeneous regions of the reservoir.
Development of a portable, scalable interactive visualization tool. The visualization of large-scale simulations present several problems related to 1) size of the problem, 2) irregular grids, and 3) parallel decomposition of the grid cells among processors. A typical large-scale problem has on the order of a million or more cells. For a multiphase flow model (e.g., black oil model) one may want to visualize several variables associated with the flow, and at the very least, all primary variables (3 in this case). The size of the visualization dataset may easily reach 1GB or more. Unfortunately, commercial visualization tools, quite suitable for small size problems, are inadequate for such large datasets. Collaboration with the Argonne Futures Laboratory as well as with University of Minnesota has allowed the use of high-end visualization tools developed by these partners to analyze and interpret IPARS results. These software tools post process the output datasets and create "movies" with which the user, using a CAVE or Immersadesk display device, can "step inside" the 3D image to explore in detail critical regions near wells and faults. The images can simultaneously convey a range of relevant information. For example, pressure is shown using cutting planes and saturations are shown using isosurfaces which can be colored by other variable values. In addition vector velocities and streamlines can be added to the overall picture. The tool is scalable and operates on distributed machines. A desktop workstation can also be used for display, although the small size of the display area limits its usefulness.
In another direction, a project on history matching and geostatistical simulations with IPARS using the software package Active Data Repository has been initiated and will be carried through leveraging by other resources. Another software package, Metachaos, is being used to couple IPARS with the surface water code UTBEST for use in environmental applications.
Multigrid or agglomeration techniques for linear solvers and interpolation and flexible inexact nonlinear solver techniques have been studied, implemented and applied. It appears that physics-based multigrid type approach may lead to faster code than the generally known optimal algebraic type multigrid solvers. Also, fully implicit wells and horizontal wells development is underway. (Further development will require additional funding.)
Development of an object-based Fortran 95 toolkit to streamline the use of automatic differentiation based on Argonne's ADIFOR tool. The principal design idea of the toolkit is the use of opaque objects to represent important components. These objects have a fixed interface, behind which the implementation details are hidden from the toolkit user, which allows the implementation to be modified or replaced entirely without requiring any changes to the physics code that uses the toolkit. The objects are designed to plug together, but can be used separately as well. The application developer is free to pick and choose from among the toolkit components. The use of standardized interfaces allows application developers to take advantage of whatever cutting-edge software is available from other sources. Through the use of these interfaces, the toolkit aims to hide from application developers as many of the implementation details of a large-scale parallel simulator as possible. (Further development of this toolkit will require additional funding.)
27 journal articles, 2 book chapters, 24 conference proceedings , 45 presentations, 13 technical reports, 9 related PhD theses.
Back to New Generation Framework homepage
Page maintained by:
jlc@mail.utexas.edu
Last modified: April 11, 2001
URL: http://www.pe.utexas.edu/CPGE/new_generation/accomplishments.html