VPFFT Proxy Application Fine-scale micro-structure used in multi-scale proxy development.
Solver for Polycrystalline Mechanical Properties
VPFFT is an implementation of a mesoscale micromechanical materials model. By solving the viscoplasticity model, VPFFT simulates the evolution of a material under deformation. The solution time to the viscoplasticity model, described by a set of partial differential equations, is significantly reduced by the application of Fast Fourier Transform in the VPFFT algorithm. The proxy application, VPFFT++, is a reference implementation of the algorithms in VPFFT. While capturing most of its computational complexity, VPFFT++ forgoes most of the machine specific optimizations and additional physics feature sets from the original VPFFT code. An algorithmic specification accompanying VPFFT++, detailing the implemented algorithm, is also available as a documentation.
Basic Into to the Capabilities of the App
Typical materials, such as most metals, are inhomogeneous in its response to mechanical stimuli. This implies that depending on local properties, different regions within a macroscopic body would react differently to uniformly applied external forces. This anisotropy comes from the fact that most materials are polycrystalline, i.e., that bulk material in fact are composed of many connected smaller crystals, or grains, of different crystallographic orientations. Because material response for a single grain is directionally dependent, orientation of each individual grain inside a polycrystalline material contribute to the macroscopic behavior in a different way. Conversely, application of macroscopic body force would also result in different reactions from grains of varying orientations within the sample. To further complicate matter, the local and global response of a material is also dependent on the boundary conditions connecting each of the grains together with its neighbors. This implies that grains with the same orientation within the sample may behave differently because of its local neighborhood environment. The goal of predicting the mechanical behavior of a material must take into account all of the aforementioned complications. The viscoplastic FFT code, VPFFT, is designed to address the problem of a polycrystalline sample under plastic deformation, with the assumption that elastic deformation in the crystal lattice plays little role in the overall evolution.
ExMatEx Usage
Encompassing a large number of numerical methods with varying memory and CPU utilization patterns, VPFFT is expected to be representative to aspects of algorithms used in scientific computing and other multi-physics applications. The myriad paths to optimizing VPFFT provide us with experiments to test various hardware configuration and programming models. For example, a parallelization method effective for the Newton-Raphson solve portion of VPFFT for GPUs may very well result in a dramatic slow down in the FFT section. These tradeoffs could be explored within the proxy app, VPFFT++. As a reference proxy app implementation of the original research code, additional abstractions were inserted into VPFFT++ to improve readability to any non-domain specific scientists. While the reference implementation takes advantage of existing open source libraries, such as FFTW and Eigen for FFT and linear algebra needs, replacement with vendor specific libraries is designed to be straightforward. Interfaces are defined such that individual sections, e.g., Newton-Raphson solve, maybe reimplemented independently from the rest of the program. Both MPI and OpenMP parallelized VPFFT++ are currently available, although alternative programming models maybe explored from the VPFFT++ proxy app.
With future hardware architecture in mind, we have decided to use a traditional Makefile system to reduce the external dependencies requirements to build VPFFT++.
Year 1 Accomplishments
- Defined the mathematical specification of VPFFT as an algorithm a proxy app, made available to vendors.
- Produced a serial version of VPFFT++, a C++ reference implementation of the specification. This is subsequently distributed via github and LLNL svn.
Year 2 Accomplishments
- Implemented MPI and OpenMP parallelized VPFFT++.
- Specified the VPFFT++ interface necessary for the multi-scale proxy app.
- Successful interaction with vendor (AMD), resulting in a version of VPFFT++ was re-implemented with AMD’s proprietary libraries (ACML). (Not yet available.)
Novel characterization techniques in experimental mechanics, combined with the advent of exascale computing and the emerging co-design paradigm, pose a challenge to modelers to devise efficient formulations for interpretation and exploitation of the massive amount of collected data. In response to this challenge, an efficient spectral methodology has been proposed and validated [1] to compute the micromechanical response of polycrystalline materials directly from images of their microstructure. Conceived as an alternative to crystal plasticity (CP) FEM, the intensive use of Fast Fourier Transforms (FFT) and the lack of any large matrix inversion, make this spectral formulation an excellent candidate to incorporate fine-scale microstructural information with a level of fidelity unreachable with CP-FEM.
We have identified the relatively simple viscoplastic version of this formulation (VPFFT) as a useful mesoscale proxy application. Besides the relevance of a continuum-level, microstructure-sensitive mesoscale approach, the opportunity to involve the ubiquitous FFT algorithm was also an important consideration in this choice. An available non-optimal Fortran code was the starting point for the proxification.
A rigorous specification has been created to describe the implementation of VPFFT and the polycrystalline viscoplasticity problem at the meso-scale. This specification represents an abstracted numerical recipe detailing equations solved and algorithm employed in VPFFT. A C++ implementation along with a hard-coded representative problem is also provided as a reference. To encourage vendor and non-domain expert adoption, both the specification and reference implementation are written for the general audience. Function definitions and implementations in the reference code correspond closely to that of the specification to better illustrate the algorithmic requirements and data flow of the proxy app. Several interface abstractions are added to the computational kernels to improve readability and adaptability. For example, Fast-Fourier Transform and matrix inverse can be replaced without significant changes of the source code. As it stands, VPFFT may serve as a simple, self-contained test application for simulators such as SST.
We are currently investigating the development of communication and dimensional reduction schemes between VPFFT and LULESH required for the path forward to a dual-scale adaptive sampling proxy app built upon ASPA. This draws upon our recent experience in similar coupling using a homogenization-based approach at the mesoscale [2]. In the context of VPFFT, this implies choosing a subset of all state variables from the simulation volume such that the macroscopic properties are captured. The effects of this dimensional reduction will result in variations in the numerical solutions both at the coarse and the fine scale.
References
- R. A. Lebensohn, R. Brenner, O. Castelnau, A. D. Rollett Orientation image-based micromechanical modelling of subgrain texture evolution in polycrystalline copper Acta Materialia 56 (2008) 3914–3926
- N. R. Barton, J. V. Bernier, J. Knap, A. J. Sunwoo, E. Cerreta, T. J. Turner, A call to arms for task parallelism in multiscale materials modeling, Int. J. Numer. Methods Eng. 86 (6) (2011) 744–764.
VPFFT Publications
Papers
R. A. Lebensohn, J. P. Escobedo, E. K. Cerreta, D. Dennis-Koller, C. A. Bronkhorst, and J. F. Bingert, “Modeling void growth in polycrystalline materials” Acta Materialia 61, 6918-6932 (2013) [external link]
Software
“VPFFT: Crystal Viscoplasticity Proxy Application Version 1.0” (February 14, 2013) [open source at GitHub]