Compiling TOUGHREACT V3.0-OMP using Cygwin and gfortran

On second thought, the code execution may not be skipping line 258.  Rather, gdb may only report the end of the continuation line, rather than showing each component.  It looks like the code likely runs as expected up to this point, where the first value of K is 10, which proceeds to attempt to read from the ROCKS block of flow.inp.  This is where the issue pops up.  As far as I can tell, no other operations occur between the continuation line #258/261 and line 267(" 1700 NM=1"), but this is where the error message occurs.

**UPDATE

After starting from scratch from the original source code, I was able to compile it and execute the example problem "P1_eos9_nacl_v3omp"

I did this by running the command:

"gfortran -O3 -fstack-arrays -fdefault-integer-8 -fdefault-real-8 -fdefault-double-8 -malign-double -fopenmp -Wl,--stack,0x80000000"

This leads me to think that I may have inadvertently altered one of the source files in a way that cause the mysterious error I reported earlier.  I apologize if anyone spent much time trying to get to the bottom of this.

A couple of additional notes:

1. On my system, I was only able to build an executable with a maximum number of elements (MNEL) of 50000 and maximum number of connections (MNCON) of 200000.  This is a little less than half of what was originally written in the file flowpar_v3.inc that was provided from the LBL Software Center download.

2. I've yet to check the validity of the output files for this example problem, nor have I run the other example problems.  Once I do, I'll post another update.

Eric,

Thanks for the advice!  I'll be sure to recompile with -O2.  So far, I've been able to run problems P1 - P11 with -O3.  P12 is running now and has been for a while.  Perhaps it's running into the convergence issues you mentioned. 

As for the stack size, I think I've maxed it out.  I had already tried increasing it explicitly in the --stack option like you suggested.  I also ran the command "ulimit -s unlimited" to maximize the default stack size.  Neither of these tactics appeared to increase the max problem size.  I'll stick with what I have for now and migrate to Linux or Mac if necessary.

- Mikey

I wanted to provide a couple of more updates:

1. P12 appears to have run successfully, but took quite a while longer to complete.  After digging into the input files, I could see why.  Is there any documentation for this problem?  I was able to find info from P1 to P11 in the Example Problem documentation, but nothing about P12.
2. I have dual-booted my machine with Linux-Ubuntu, and I've since been able to increase the max problem size to 1 million elements/3 million connections.  If anyone else tries this, I have a couple of suggestions:
   • Rather than set your stack limit in the compile options, you should simply set it in the terminal window with the command "ulimit -s unlimited" to max out the stack size based on your computer's capability, or by replacing "unlimited" with a set stack size in kbytes.
   • The inclusion of the "-mcmodel=medium" option significantly increased the maximum number of elements and connections.  In fact, by not including it in the Linux environment, I was running into the same limit as I was with Windows/Cygwin, even when that option was included.  It makes me wonder if the "-mcmodel=medium" or "-mcmodel=large" options have much effect in Windows.

So I have a not-so-good update on my last post.  It appears that, while gfortran compiles TOUGHREACT with very large physical domains (1 million elements/3 million connections), I run into a "Segmentation Fault" runtime error up to about 170000 maximum elements & 650000 maximum connections.  While this is still about 3x more than I could get with the Cygwin compilation, it doesn't quite approach the 1000000 elements/3000000 connections I could compile. 

I did a little digging into this, and found this could be caused by implementation of this code in OpemMP. I ran across this site: http://whoochee.blogspot.com/2009/11/segmentation-fault-while-using-openmp.html .

It states the problem could be that "all the variables declared in the parallel region will go into the process stack", whereas the mother process stores them in global memory.

It states that one possible solution is to use the keyword "save" when declaring all variables that should be shared among all of the processes.

Not knowing the makeup of the source code well enough to know if this is already done, or where the right places would be to include this, I thought I'd reach out to you guys to see if this strategy is already implemented.  If not, could it be applicable to the TOUGHREACT source code?

Thanks!

Yeah, the topic heading, which mentions compiling TOUGHREACT using Cygwin, doesn't help clarify some of the efforts I made setting up my machine with Ubuntu.

It's likely that memory shortage is my issue.  I'm running this on my personal laptop that has 8 GB of memory.  I have a desktop at my job that has 128 GB of RAM, which I just set up with Ubuntu.  I'm going to see how much I can scale up my domain sizes with that and post an update later on.

As far as using the keyword "save" during variable declarations goes, do you think this could help?  I noticed a lot of COMMON blocks in the source code.  Would it be helpful to replace them with MODULEs?  At the very least, it could cut back on the number of repetitive variable entries.  Obviously, the source code is massive, so it wouldn't be a trivial effort to convert those, but do you think it could have an effect on its performance?

Another update**

I compiled the TOUGHREACT code using my desktop workstation with 128 GB of RAM, and it appears to compile and run simulations with MNEL=1e6 and MNCON=6e6.   Thus far, I've only run it with example P1, which has way fewer than MNEL elements and MNCON connections, but this is still an improvement from the executables built with my laptop.  Note that I did have to change the "-mcmodel=medium" option to "-mcmodel=large" to get there. 

Stefan,

I'll have to try out your suggestion.  Note that the '-fopenmp' option that TOUGHREACT-v3-OMP implies the '-frecursive' option, which gets overwritten when using '-fno-automatic'.

I think the main issue I'm trying to address, though, is in the COMMON blocks, which generate large arrays with dimension lengths based on the maximum number of elements MNEL and maximum number of connections MNCON.  It's my understanding that when variables that are shared across threads are declared this way, it adds each of them to the thread's stack rather than in global memory, where they would be stored if declared using MODULEs.  I'm not adept enough in FORTRAN coding or the particulars of the TOUGHREACT source code, so I don't know if this would help or not.

Eric Sonnenthal, would you mind sending me the input files for the EOS1 problem you mentioned?  I believe I have everything working now with my Linux compilation at my 128GB workstation, and I wanted to test it on something a little bigger.

Thanks!