This is just a quick note to inform people that I am currently working with Enthought to bring SciPy to the .NET platform. In particular, I will work on the Fortran parts of SciPy.

The primary strategy will be to improve fwrap enough to make it usable for SciPy, and then move SciPy over to fwrap instead of f2py. The point here is that Carl Witty is already working on a .NET backend for Cython, and since fwrap generates Cython code we get the .NET port that way.

All work is done in Enthought’s “refactor” branches for now [1]. The intention is certainly to merge back to main eventually, but questions of how or when or whether will have to wait; getting things up and running on .NET has priority.

Some details:

a) The most important missing feature in fwrap is callbacks. I’m sure there are other things I’ll have to implement as well.

b) The main strategy is to first move (our own branch of) SciPy over to fwrap and have that work on CPython, and then move to compiling things on .NET

c) fwrap does not support g77, only Fortran 90 compilers like gfortran/ifort etc. For the purposes of the .NET port this is likely to be good enough. Before a merge with the main CPython branch one must perhaps look into implementing g77 support in fwrap. I know that David C. at least earlier stated that g77 support is not going away anytime soon. Feedback on this welcome.

Dag Sverre

[1]
http://github.com/teoliphant/numpy-refactor
http://github.com/jasonmccampbell/scipy-refactor

See the rest of the thread here:

http://thread.gmane.org/gmane.comp.python.scientific.devel/14620

This is great news — fwrap will have significant features implemented, and will become part of a major distribution.  Thanks to everyone involved.  I will pitch in where necessary, if I can carve out some time (thesis, job interviews, papers, etc.).

Without question the most error-prone and dark corner-filled part of fwrap is the build system.  It’s based off of numpy’s distutils, but it is very fragile.  Fwrap demands a lot from a build system: (1) generate C, Fortran and Cython headers from an external python script, (2) compile Fortran 90 sources with module dependencies, (3) generate .c files from a .pyx, and (4) compile all the fortran and C source into a Python extension module.  Now that it is time to incorporate full-on Fortran 90 support in fwrap, distutils just plain doesn’t cut it.  I can either spend my time monkey patching things together or move to another build system.

Guess what I’ve decided to do.

One fundamental requirement is that the new build system not impose yet another external dependency.  So it either needs to already exist as part of every platform out there, or it needs to be distributed with fwrap.  This limits the choices (besides the already rejected distutils) to make or a new build tool named “waf”.

Waf is designed to be small and shipped as part of a larger project.  That way the host project has complete control over its build system and doesn’t have to worry about versioning issues.  Besides that, it is a proper build system (contrary to distutils), and it offers automatic dependency detection & resolution, especially desirable for Fortran 90/95 projects with modules.  I anticipate it taking fwrap’s most error prone component and making it much more robust.

In order to begin working on the new F90 stuff, waf needs to be incorporated into fwrap.  Once that is working, then I can set up the test suite to be module-aware, and move on to incorporating the new features.

Waf is suitable for any project, whether it be C, C++, Java, Go, Python (extensions), Cython, D, Fortran, whatever.  It easily handles mixed-language builds.  Adding support for new languages & compilers is pretty easy, too.  I’m somewhat new to it, but I plan on trying it out for some of my bigger codebases in the near future.

For more info:

http://code.google.com/p/waf/

After speaking with some folks at SciPy-Austin this year (in particular, Fernando Perez), I wanted to try out git & github as an alternative to mercurial & bitbucket.  As with anything with so much overlap, there are pros and there are cons, and it comes down to small differences.  The deciding factor, ultimately, was the hg-git mercurial plugin, which allows someone to use a mercurial client with a git repository, with improved branch functionality.  So all of the nice, smooth, user-friendly mercurial commands you’re used to are available, and you can push/pull bookmarks a-la git branches.  (I’ve not used this extensively, so caveat emptor.)

Some of my reasons for the move:

• As mentioned above, the hg-git plugin allows both mercurial and git users to collaborate using the same repository, so there is no need for current mercurial users to have to change to git.
• The numpy/scipy ecosystem is converging on git & github for SCM.  Arguably Cython (which uses mercurial) would be more influential here, but pretty much every fwrap user would be coming from a numerical & Fortran background, which would be more numpy/scipy-centric, hence git & github.
• The way Git handles branches fits my style much better.  And I really like the index.  Mercurial seems to encourage enabling the ‘queue’ extension for advanced stuff, but it just plain doesn’t fit my brain, and I always screw it up.  Granted, it’s plenty easy to blow off your hand using git, although I’ve become quite proficient at messing up whatever SCM I’m using at the time.
• It seems that mercurial goes to great lengths to keep their design pure, which I can respect.  Unfortunately by not allowing bookmarks to be pushed/pulled as part of the wire protocol, they force you to use mercurial branches, which are much heavier than I need them to be.  If mercurial would just allow bookmarks (which are the closest analogue to git branches) to be pushed-pulled, I’d have a much harder time justifying the switch to git.

Github repository

Fwrap’s new home:

http://github.com/kwmsmith/fwrap

https://sourceforge.net/apps/trac/fwrap/wiki/WikiStart

Specifically, I just posted a rough roadmap on the fwrap wiki, outlining what fortran features are currently supported and what will be supported in upcoming versions:

https://sourceforge.net/apps/trac/fwrap/wiki/FortranSupport

I commend it to your attention.

Broadly, fwrap can be seen as an update of f2py for all of Fortran 90/95, addressing the features that f2py does not support at the present time.  My understanding is that the 3rd version of f2py intends to address many of these features, but that version of f2py is far from complete.  This is not intedended as a critique of f2py, merely a brief comparison of the two projects.

Fwrap focuses on generating Cython wrappers for Fortran code, and the Python-level wrappers come for free, in a sense.  Fwrap uses the Fortran parser–named ‘fparser’–that Pearu Peterson (the author of f2py) created for the latest incarnation of f2py.  In contrast to Fwrap using Cython, f2py generates the C extension module directly–quite an undertaking.  Fwrap focuses on all 3 levels of wrapping: C, Cython and Python; and the C wrappers can be used without any Cython or Python dependency.

Fwrap generates one more wrapping layer than f2py; this allows fwrap to eventually support every desirable feature in Fortran 90/95 (and, in principle, 2003).

F2PY has a very nice interface file functionality, and supports comment directives in the fortran source to specify how it is to handle different arguments.  These features are planned for fwrap, but currently it is easier to make ‘pythonic’ wrappers with f2py, while fwrap just wraps every argument directly, taking into account the argument’s intent.

Fwrap has planned some features that are not in f2py: for instance, a fortan inline module that is similar to scipy’s weave, user derived types, automatic type discovery, and (way in the future) wrapping a Fortran module in a Cython class.

Pearu and I are on good terms and we both collaborate to make fparser better along with the other fparser & f2py developers.  F2PY is an impressive project and provides the skeleton for many Python & Fortran projects, most prominently among them being SciPy.

Fwrap would not exist if f2py had not blazed the trail, and paved over half of it

I am pleased to announce the first release of Fwrap v0.1.0, a utility for
wrapping Fortran code in C, Cython and Python. Fwrap focuses on supporting the
features of Fortran 90/95, but will work with most well-behaved Fortran 77
code, too.

Fwrap is BSD licensed.

Fwrap is beta-level software; all public APIs and commandline options are
subject to change.

Features in the v0.1.0 release:

• Fortran source parsing and automatic wrapper generation;
• Top-level (non-module) Fortran subroutines and functions;
• Supports all intrinsic types (integer, real, complex, logical &
  character);
• Default and non-default intrinsic types properly wrapped;
• Scalar and array arguments supported;
• Supports assumed-shape, assumed-size, and explicit-shape array arguments;
• Intent ‘in’, ‘inout’, ‘out’ and no intent arguments supported;
• Automatic docstring generation for extension module and functions;
• Wrappers are portable w.r.t. both Fortran and C compilers.

Upcoming features:

• “Use” statement support;
• Python & Cython callback support;
• Kind-parameters in type specification;
• Module-level data and parameters;
• User-derived types.

See the README and documentation for requirements, compiler support, etc.

Download

You can get fwrap from pypi or from its sourceforge download page:

https://sourceforge.net/projects/fwrap/files/

More Info

Project homepage, including links to wiki & bug tracker:

http://fwrap.sourceforge.net/

Mailing list:

http://groups.google.com/group/fwrap-users

Development blog:

http://fortrancython.wordpress.com/

My first impression is that this is an ideal sweet spot, allowing the usage of nice high-level fortran array expressions on the host when convenient, and dropping into CUDA code where speed is a must.  Granted the host-device bandwith is painfully slow, so there is a significant overhead, but ideally it would allow incremental speedups, all within fortran.

I’m glad to see Fortran adapting to the latest technology, as it has done every generation or so; brings to mind the quip:

“What language will they be using 30 years from now?”

“I don’t know, but they’ll call it ‘Fortran’.”

And yet the standard is still backwards compatible with code from almost 35 years ago.  AFAICT, it’s the combination of adaptibility and backwards compatibility that keeps Fortran around.

Now if they would only get better I/O syntax, named goto labels (not ‘assign 10 to n’, more like C’s goto), etc…

The most significant features for users are module- and function-level docstrings, but there are many more features you’ll like.

For this trivial fortran function:

function one_arg(a)
    implicit none
    ! Automatic type discovery: the C type correspondances
    ! will be automatically determined at compile-time.
    integer :: one_arg
    ! Assumed-shape arrays handled seamlessly
    real(kind=8), dimension(:, : ), intent(inout) :: a

    a = 1.61803399_8
    ! bonus if you know 1.618... without googling
    one_arg = 42

end function

Fwrapping it:

$ fwrapc source.f90 --build \
--name=one_arg --fcompiler=gnu95 \
--f90exec=/usr/bin/gfortran-4.3 \
-L/usr/lib/gcc/x86_64-linux-gnu/4.3.2 -lgfortran

(Admittedly the commandline can get kinda long, but the -L, -l and --f90exec options can be set to environment options $LDFLAGS and $F90, resp. The fwrapc command needs to know where to find the fortran compiler and how to link in the fortran runtime libraries.)

This will generate a million files in ./one_arg, including an extension module, ./one_arg/one_arg.so. All those other files are useful if and when you want to use the wrappers from C or Cython. For now we’ll look at using the code from Python. Importing that extension module, we see some interesting stuff:

$ cd one_arg
$ ipython
Python 2.5.2 (r252:60911, Jan 24 2010, 17:44:40)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import one_arg

In [2]: one_arg?
Type:		module
Base Class:	<type 'module'>
String Form:	<module 'one_arg' from 'one_arg.so'>
Namespace:	Interactive
File:		/home/ksmith/Devel/fwrap/fwrap-dev/testdir/one_arg/one_arg.so
Docstring:
    The one_arg module was generated with Fwrap v0.1.0dev_a0bf087f24a0+.

    Below is a listing of functions and data types.
    For usage information see the function docstrings.

    Functions
    ---------
    one_arg(...)

    Data Types
    ----------
    fw_character
    fwi_integer
    fwi_npy_intp
    fwr_real_8

In [3]:

You’ll notice a list of functions and data types in the one_arg module. The datatypes fwr_real_8 and fwi_integer correspond to the function argument and return value types, resp. (automatic type-discovery, like I mentioned).

We can take a look at the one_arg.one_arg() docstring:

In [3]: one_arg.one_arg?
Type:		builtin_function_or_method
Base Class:	<type 'builtin_function_or_method'>
String Form:	<built-in function one_arg>
Namespace:	Interactive
Docstring:
    one_arg(a) -> (fw_ret_arg, a,)

    Parameters
    ----------
    a : fwr_real_8, 2D array, dimension(:, : ), intent inout

    Returns
    -------
    fw_ret_arg : fwi_integer, intent out
    a : fwr_real_8, 2D array, dimension(:, : ), intent inout

In [4]:

It tells us the function signature, the argument types and the return tuples.

Let’s kick the tires:

In [4]: one_arg.one_arg([[1,2,3], [4,5,6]])
Out[4]:
(42,
 array([[ 1.61803399,  1.61803399,  1.61803399],
       [ 1.61803399,  1.61803399,  1.61803399]]))

And we find that the fortran function return value is the first element of the return tuple, and the array is the second. In this case, we passed in a python list, and fwrap made an internal copy seeing that it wasn’t compatible with the type and ordering of the fortran dummy argument. If we pass in an array of the proper ordering and type, no copy will be made, and it will be modified in-place (there will be options to have finer control over this behavior, including warnings or exceptions raised if you don’t want any copies made, ever):

In [5]: import numpy as np

In [6]: a = np.empty((5,5), dtype=one_arg.fwr_real_8, order='F')

In [7]: one_arg.one_arg(a)
Out[7]:
(42,
 array([[ 1.61803399,  1.61803399,  1.61803399,  1.61803399,  1.61803399],
       [ 1.61803399,  1.61803399,  1.61803399,  1.61803399,  1.61803399],
       [ 1.61803399,  1.61803399,  1.61803399,  1.61803399,  1.61803399],
       [ 1.61803399,  1.61803399,  1.61803399,  1.61803399,  1.61803399],
       [ 1.61803399,  1.61803399,  1.61803399,  1.61803399,  1.61803399]]))

In [8]: (ret, a_ret) = one_arg.one_arg(a)

In [9]: a_ret is a
Out[9]: True

So a_ret and a are the same python object, no copying took place!

I hope this was instructive. A release is coming soon!

The remaining 3% are related to a trivial feature, namely dimension(0:N) style array declarations that will be supported very soon.

Much work has gone into the front end recently.  The commandline options are taking their final shape and I’ll be getting some improved documentation in the README tomorrow.  Feel free to take it out for a spin.

To try out fwrap from the mercurial repo, see http://fwrap.sourceforge.net/ for instructions and dependencies (numpy, cython & a “modern-enough” Fortran 90 compiler).  Let me know any problems you may have.

Enjoy!

Thanks to a sprint at SciPy 2010, fwrap is quickly gaining parity with f2py.  Fwrap doesn’t yet have interface files working (and it won’t for the first release); neither does it support ‘common’ blocks.  I’ve been focusing on F77-style function & subroutine functionality.

The main Fortran features currently supported:

• All intrinsic scalar datatypes, including with literal integer kind parameters
• Arrays of intrinsic types & literal kind params.
• All dimension declarations (assumed shape, assumed size, explicit shape)
• Runtime array dimension checking
• Automatic type discovery

Fwrap has a pretty good testsuite in place, both unittests & integration tests.  They’re run frequently to ensure no regressions are introduced.

Kyle Mandli helped improve fwrap’s commandline interface during the fwrap sprint at SciPy 2010.  Those changesets have yet to be merged with the main repo, but when they are it will be a *big* improvement, and move fwrap much closer to the 0.1 release.  He also added logging functionality to fwrap and fparser — another big improvement.

I’ve been throwing some fairly big F77 (clawpack, lapack) and F90 (a UW-Madison inertial confinement simulation; thanks, Matt!) codebases at fparser to thresh out glaring bugs, and thus far I’ve been pleased.  No major problems, although I’d very much like to figure out how to test the AST generated by fparser for correctness.  Not sure how to do that yet.

Anyway, just a quick update.  Stay tuned!