F2PY is a solid tool for interfacing a Python front end application with a Fortran compute engine but we found some limitations and quirks that led us to develop a nonstandard method for using it. Using F2PY in the standard, documented fashion has some problems. If you are not using the canonical Fortran compiler, especially on Windows, you can have difficulty getting F2PY to find and use your compiler. And if you want or need to use certain compiler switches the process for doing that is not easy.
The problem that F2PY makes is trying to store and maintain knowledge about all the major Fortran compilers on different platforms, down to the level of the format that they report their versions with. Inevitably, much of this information is out of date and thus support for your platform/compiler combination within the F2PY system requires a bit of hacking. This is worse on Windows, primarily because the F2PY developer does not use Windows (Objexx has contributed a few patches and problem reports for these issues). There are many F2PY mailing list reports about it failing to find and use the compiler that was requested.
After investing some time in trying to get F2PY to work with alternative compilers on Windows Objexx went a different route and we extracted the necessary parts and do our own build steps via script, just using F2PY to generate the interfaces. This is not hard to do and eliminates much grief.
An overview of our approach follows.
Veneer Interface
A recommendation for fast and reliable F2PY use is to only present a lightweight Fortran veneer with the interfaces that the Python needs to access to F2PY and to keep the actual computational code in separately built libraries. This interface is essentially a set of wrappers that call the real computational code that lives in these separate libraries. This has the benefits of faster F2PY processing and lower likelihood of F2PY seeing something that trips it up. It also keeps !f2py comments (that are used to get the desired interface without customizing the .pyf file by hand) out of your computational Fortran code, where they can confuse developers not familiar with F2PY and could be altered or removed, which is hard to notice in a large source file. Another benefit is that it clearly identifies and limits the interface used by Python.
Pluggable User Fortran Libraries
Using an interface veneer as described above enables pluggable dynamic/shared Fortran libraries that can be built without installing or using F2PY, which is very desirable for end user custom libraries, and swapped in and out for each run of your application. The approach is as follows:
- Put a lib sub-directory under the bin directory where executables go.
- Create "stub" source for each customizable routine or bundle of routines.
- The stub routines should have the argument lists and little else except documentation on the arguments and requirements/recommendations on the computations that must occur in each. They may contain sample/standard computations if that is desired.
- If the stub routines are not actual default implementations it is a good idea to have the application automatically detect if any stub routine is called. One way to do this is to have a flag argument in the stub code that real implementations must alter to indicate that the routine can be called.
- It is recommended to have an identifier string in the stub routine that users are encouraged to set to indicate the routine's source, developer, and content. This can be part of the interface and passed back to the application to include a report of the custom routines used in a run.
- Build the stub libraries as part of your build process and put them in this lib subdirectory.
- Point the application at the stub libraries in lib by default.
- Include the source to the stub routines (or user-specific variants) in the application distribution and instructions on the compiler required for each platform (must be ABI compatible with the Fortran compiler used to build the application).
- Users build their custom dynamic/shared libraries and put them in a directory such as /app/custom and can then "point" your application to them either by a command line argument to the application, a saved user preference, or an environment variable, depending on the approach preferred.
Argument-Free F2PY Callbacks
Objexx has developed a method for using callbacks within the Fortran that does not require passing the callback routine throughout the Fortran argument lists, which is a burden and can degrade the clarity of the code. This requires use of a Fortran 2003 feature called procedure pointers, which are supported by recent versions of a number of the major Fortran compilers including both Intel Fortran and GFortran.
This method involves storing the pointer to the callback procedure in a module that is then made available in any routine by adding a USE module statement in that routine. This is a fairly obvious approach in C-based languages but until procedure pointers were added to Fortran there was no way to get the callback reference to a Fortran routine other than by passing it as an argument.
The callback routine and associated procedure pointer is defined in the callback module like this:
! Callback routine interface
INTERFACE
SUBROUTINE callback_prototype( msg, level )
CHARACTER(*), INTENT(IN) :: msg
INTEGER, INTENT(IN) :: level
END SUBROUTINE callback_prototype
END INTERFACE
! Callback procedure pointer
PROCEDURE( callback_prototype ), POINTER :: callback_ptr => NULL()
On every entry into the Fortran the callback needs to be registered with the Fortran since the Fortran DLL state is not carried over. That is done by calling a routine like this:
SUBROUTINE message_caller( message_callback ) USE CallbackModule IMPLICIT NONE ! Arguments EXTERNAL :: message_callback !f2py character*(*) msg !f2py integer level !f2py call message_callback(msg,level) ! Set the message callback procedure pointer callback_ptr => message_callback END SUBROUTINE message_caller
and putting a call like this at the top of every routine that is an entry point from Python:
CALL message_caller( message_callback )
Then within the Fortran messages can be sent via the registered callback by adding the line:
USE CallbackModule
to the routine and making calls like this:
CALL message_call( msg, LogLevel_FATAL )
where message_call is a wrapper routine that lives in the callback module that looks like this:
SUBROUTINE message_call( msg, level ) CHARACTER(*), INTENT(IN) :: msg INTEGER, INTENT(IN) :: level CALL message_callback_ptr( TRIM( msg ), level ) END SUBROUTINE message_call
This setup appears a bit complicated but all the details are in the callback module and the client code just has to add one USE statement and can then do callbacks without adding arguments throughout the Fortran call tree.
F2PY Logging
Leveraging the F2PY callback capabilities, logging systems can be built in Python and Fortran that work together. First, this means that the Python code has a nicely featured logging system that can do things such as routing log messages to one or more "sinks" that can include the console in a CLI run, a log pane/widget in a GUI run, and a log file for any run. Also, the level at which logging happens, e.g., informational messages and higher or also debug messages when debugging, can be controlled by the user. Python's built-in logging package provides this capability.
The next step is to add a Fortran logging module, that exploits the argument free callback mechanism, that can accept messages from anywhere in the Fortran with levels (info, warning, …) and can both act on those messages within the Fortran and pass them back to the Python where they can be wired into the Python logging mechanism. In addition to simple text messages, interfaces can be added for other message types, such as the time step completed and the current estimate or max number of time steps that the Python GUI can use to display a progress bar. This integration is also important when the Fortran is doing a STOP to terminate a run due to some problem: the Fortran can notify the Python that it is about to terminate, clean up resources, and then do the STOP. This is especially important when using Python multiprocessing so that processes are not left hanging, which happens if the Fortran dies with signaling the Python.
A fairly small amount of boilerplate code is required to make this all work and the results are well worth the effort.