So what’s the latest with fwrap?

May 31, 2010

Fwrap is quickly progressing & converging on release.  (Mea culpa for the *lack* of a release in April.)  I’m finding that, like solving a PDE, 80% of the human effort takes place with handling the boundary conditions, even though they are, well, just the boundary.  For fwrap’s purposes, that means a *lot* of work has been taken up in the compilation bits, and by that I mean getting the different fortran compilers, cython and numpy to play nicely (although the fortran compilers are the primadonnas of the bunch).  Whoever comes up with a killer solution for ‘make’ in Python will gain my eternal gratitude.  But I digress.

For those interested in checking things out (alpha sofware, interface is stabilizing, usual caveats, etc.) fwrap is hosted at sourceforge here:

http://fwrap.sourceforge.net/

(I’ll put up more handy links there shortly)

The project’s mercurial repo can be checked out here (although it will likely be converted to git soon):

http://fwrap.hg.sourceforge.net:8000/hgroot/fwrap/fwrap

You’ll need numpy (>=1.3.0 — haven’t checked earlier versions), cython (>= 0.11 — ibid) and fparser (which will be distributed with fwrap when 0.1 comes along).  Fwrap uses nose for unit testing.  You can get an svn checkout of fparser here:

$ svn co http://f2py.googlecode.com/svn/trunk/fparser/ fparser

Once these are in place (recommended that a symlink to fparser be placed in the fwrap_src directory) you can run the “integration tests” (runs fwrap on fortran code, compiles it and runs a doctest suite on the resulting module) from within the toplevel fwrap directory:

$ python runtests.py –fcompiler=gnu95 –no-cleanup

There are other options, the one you’ll probably care the most about is the –fcompiler flag.  By default it is ‘gnu95’.  ‘–no-cleanup’ is also handy, it makes sure to leave everything in place in BUILD after running the tests.

If you care to run the tests and let me know how it goes, I’d be obliged.  Current tests work on Mac OS X 10.5 and Ubuntu 9.04.  Windows testing is sorely lacking and much needed.

Announcing fwrap-users mailing list

March 30, 2010

For all fwrap-related questions, etc.:

http://groups.google.com/group/fwrap-users

It may seem a bit premature, since fwrap isn’t in the wild yet, however, fwrap will have its first release this April.

(And I hope to put out releases more frequently than Ubuntu, in case anyone is wondering about the timing of it all.)

Fwrap update — 21.02.2010

February 21, 2010

Development has been progressing well on fwrap, although the first release is still a month in the future, at the very least.

Since last summer, it became clear that the core of fwrap — the code that generates the fortran wrappers, the C and Cython headers and the cython wrappers — was inextricably tied into the syntax tree generated by fparser, in a bad way.  Fwrap had to muck around too much in the guts of fparser’s classes to get stuff done.  By the end of GSoC, fwrap was in a workable state (still not at first release, though), but it was very brittle and increasingly difficult to add functionality incrementally.

My efforts of late have been to extract fwrap from fparser, and fwrap’s core is in a better state.  I’m finding it easier to add in the necessary functionality, and the overall architecture is easier to grasp and adapt.  I want to make it tolerably easy for others to understand and contribute to the code-base, and keeping fwrap’s core separate from fparser shrinks the mental load considerably.

These are the main areas of functionality that I’m planning on addressing for the first release:

  1. All intrinsic datatypes (integer, real, double precision, complex, logical and character) wrapped end-to-end.
  2. Top-level functions & subroutines (those not inside a module) wrapped, end-to-end.
  3. Assumed shape arrays (declared with ‘integer, dimension(:,:) :: int_array’) wrapped, end-to-end.
  4. Explicit shape arrays with trivial expression extents (‘integer, dimension(d1, d2, d3) :: int_array’) wrapped, end-to-end.  (Assumed-size arrays (‘dimension(d1, d2, *)’) are included here.)
  5. Automatic discovery and compilation of kind-type-parameter values.  For instance, if a module has a parameter ‘SP’ defined and a subroutine uses that module’s parameter as a kind-type-parameter, fwrap will handle things automatically for you.  This is mostly a parsing issue and one where fparser needs work.
  6. Older constructs, like common blocks & implicit typing, shouldn’t be too bad, either.  Since common blocks map to C structs, these might wait until we get to derived-type support.  We’ll see.

With the decoupling between fwrap and fparser, it will be possible to write by hand a ‘pyf’ interface file that gives fwrap all the necessary bits to make the right wrappers.  This makes testing easier and further helps to decouple fwrap from fparser.

There are many more fortran features that fwrap will support, if I can get the time or attract other contributors.  High on the list are function/subroutine arguments (done well), Python callbacks, and full module wrapping.  Module wrapping has the possibility to be rather interesting, since it is possible to have a nice correspondance between Fortran 9x modules and Python extension types, so long as a few guidelines are adhered to on the Fortran side.  The C binding features supported in all major fortran compilers makes this Fortran module -> Python extension type mapping possible.  But that is far in the future.  I should mention that Fortran derived types will be supported soon after the first release, as well.

So there’s an update for you — long on promises & ideas, but things are coming together nicely.  Once the first release is out I’ll put more effort into publicizing things and facilitating contributions, including a more detailed roadmap.

Fwrap update and compilation infrastructure

January 14, 2010

It’s been a while since an update.

Events out of my control have consumed my discretionary time, events that have conspired to delay fwrap’s delivery.  The usual story.  I lean on your patience, and I’m glad to know there is still interest out there.

A critical component of fwrap is its compilation infrastructure.  I’ve been thinking about this a lot lately; any input you care to share with me is very welcome.  At first I was using a simple Makefile for the automated integration tests on my box here.  That’s fine for testing, but won’t cut it for all the different systems out there.  I’ve since moved to a monkeypatched Cython-numpy distutils script that gets the job done, but isn’t the shiniest bit of code I’ve written.

The compilation system has to do a number of disparate things:

  1. Somehow generate the ISO C BINDING mappings from Fortran datatype to C datatype.  For some setups, this may require compiling and running a Fortran source file.  I’m informed by some (David Cournapeau, who has some expertise in this area) that doing this on some platforms is less than ideal.  There will be other setups, though.  More on this in a future post.
  2. Cythonize the Cython source files.
  3. Compile the Fortran source with the appropriate flags.
  4. Compile the C source.
  5. Create the extension module.

So the compilation script will, in an ideal world, be able to compile and run a Fortran source file, cythonize Cython code, compile C & Fortran code, and compile an extension module.  All while being intelligently adaptable to different command-line flags to customize different Fortran compilers & compilation flags, etc.  A tall order.

Here are the candidates, as I see it:

  1. Cython-numpy distutils — get this in shape and make it robust.  This encapsulates lots of work done in numpy.distutils.fcompiler  to automate all the different fortran compilers out there on all different platforms.  I’d be a fool to not make use of it.  On the other hand, distutils stuff can be ugly and error-prone, in my limited experience.  A benefit is that it would require just Cython and Numpy; Cython will be a dependency of fwrap, and Numpy very well might be.  So no extra dependencies, which is great.  All in all, though, I’d like something more robust out of the box, something like…
  2. Scons — I’ve been favorably disposed to what I’ve seen of scons, and I’d love to use it.  I’ve not experimented with its fortran support yet, although browsing its source shows some promise.  I see a number of different fortran compilers there that are supported, although not all (pathscale, xlf, others…)  It would be an extra dependency — is it worth it?  And how hard would it be to support other fortran compilers?  For that I’m looking into…
  3. Numscons — (http://projects.scipy.org/numpy/wiki/NumScons)  This likely is the bees’ knees of what I’m looking for, a combination of scons with numerical compilation.  We know it works for numpy so there’s one success already.  Is it mature?  Will it be supported in the future?
  4. Makefiles (on systems that would support it) — I’m Makefile literate, but no guru.  And it may be a minefield with all the different systems out there, i.e. it might require a guru.

If anyone has the competence to comment on the above I’m all ears.

GSoC est fin, but fwrap lives on!

August 23, 2009

I’m glad to say that my Google Summer of code project was successful, despite the features that didn’t make it in fwrap by August 17th.

Oh, yeah — if you’re wondering what ‘fwrap’ is, that’s the new name of ‘f2cy’.  The rebranding was appropriate since ‘f2cy’ didn’t quite capture all that the utility does.  fwrap wraps fortran for a number of languages (C, Cython & Python), so ‘f2cy’ was deemed a misnomer since it seems to wrap fortran just for cython, besides the confusion with ‘f2py’.

fwrap was presented at the SciPy 2009 conference, and I was glad to find a good amount of interest; two (Kyle & Chris) stepped up to work on fwrap once I can get it to a state that’s comprehensible 🙂  They work on a code called ‘clawpack’ at the U-Washington, and want to use fwrap for their ‘pyclaw’ version of clawpack.  But I’m  getting ahead of myself.

Here’s where you can find the presentation abstract (note that it was written early June — its heavy on promised features that didn’t all make it in fwrap by August 17th), and here are the talk slides.  The presentation itself is here.

What’s the state of fwrap?  It can handle scalar arguments with aplomb.  Assumed-shape arrays are working.  Explict-shaped & assumed-size arrays are soon to come.  I’m sketching out callbacks as we speak (a feature that Kyle & Chris are particularly interested in).

The parser used by fwrap needs stabilization — some obvious things need work (public/private module attributes), and I’m finding out just how unruly the full Fortran language specification is.  The language allows you to be expulsively verbose & clunky (‘integer(kind=FOOBAR), intent(in), dimension(2) :: an_int_array’) — yet difficult to fully parse, since each attribute (‘intent,’ ‘dimension,’ ‘save,’ etc.) can instead be a statement on a line a ways away from the actual ‘integer an_int_array’ line; or you can be cryptic & bug-prone by using the implicit declaration anti-feature well-hated by all those who have ever had a misspelled variable in their code.

The above rant is a long way of saying that there are many little things that fparser chokes on.  This is not to be interpreted as a complaint with fparser — I’m grateful for all of Pearu’s work thus far on it, and for his ambition to tackle parsing such a barnacled language.  He has graciously opened up fparser to me to work on it, and has allowed us to package fparser with fwrap.  Thanks, Pearu!

Anyway, work on fwrap will be a bit slow for the next week or two, as I turn my full attention to my research.  But I’m energized and motivated to get the first release out before 2010.

Update

June 18, 2009

I’m glad to say that the f2cy part of the GSoC miterm evaluation is functional. By that I mean it can wrap functions & subroutine subprograms (not in modules, yet) with any integer or real argument. The testing framework is all in place, and all tests pass. There are still many things that I’d like to add before the midterm, but the basics are all there, and the additions should be fairly modular and easy to do. The code could use some cleanup, but it shouldn’t be too bad.

Here are the some additions I’d like to make before the midterm if time allows — nothing to be inferred by the ordering except where noted:

1) Support locally defined parameter kind-types (easy), e.g.:

subroutine foo(a,b)
    integer :: DP
    parameter  (DP = 8 )
    integer, parameter :: SP = 4
    real(kind=DP), intent(inout) :: a
    real(kind=SP), intent(in) :: b
    ! ...
end subroutine

2) Next, support use’d module kind-type parameters (easy):

module ktps
  integer :: KTP
  parameter (KTP = 10)
end module ktps
integer function bar(a)
  use ktps
  real(kind=KTP), intent(inout) :: a
  ! ...
end function

3) Next, support module-scoped functions (some corner cases to consider, a bit harder) — just to give you an idea…

module container
  use ktps, local_kpts => KTPS
  integer, parameter :: C_KTPS = local_ktps
  contains
  subroutine mod_subr(a)
    use othermod, rename_p => orig_p
    use yetanother, only: yetanother_p => xxx
    implicit none
    integer(kind=C_KTPS) :: a
    integer(kind=other_mod_param), intent(out) :: b
    integer(kind=rename_p) :: c
    ! ...
  end subroutine
end module

4) Logical <-> Integer conversions (probably before the above)

The ‘logical’ type is basically a specially typed integer — whenever a subprogram’s argument is a logical, its C wrapper should use the equivalently-sized integer and do a conversion before & after the call.

5) Character arguments:

It seems that these are functionally equivalent to one-dimensional arrays, with a few bugaboos to watch out for. I’m lumping them in with the second-half of the project, since after working with integer/real arrays I’ll have a better feel as to the best way to handle character arrays.

6) Complex arguments:

Since we can’t assume C99 _Complex support, these need to be passed in as structs, converted to the native complex type in fortran, and converted back on return. Once (4) is done, I’ll have a better feel for these. I’m putting them off until the second half, since they’re equivalent to passing derived types.

Cython side:

I’m turning my full attention to tickets 299, 300 & 178 now, provided that the recent intense discussion about a native Cython array type leave these tickets unchanged (see http://article.gmane.org/gmane.comp.python.cython.devel/6654).

Long-overdue Update — ‘f2cy’

June 11, 2009

Much progress over the past few weeks, I’m glad to say.  Here are the main threads that are out there.

First, the 10000 meter view:  For my GSoC, I’m developing, under the guidance of Dag Sverre Seljebotn, (1) a python package that wraps fortran so it can be easily called from a cython program; (2) improving cython’s buffer support to allow the passing of buffers/arrays to external code (C, fortran, or whatever can be linked in) or cdef functions.

The fortran-side:

Dag Sverre and I discussed a bit about the naming of the fortran wrapper project — right now it’s under the working title “f2cy,” in the long tradition of “<some-language/protocol/format>2<other-language/protocol/format>” converter names.  There are at least 85 similarly-named executables on my ubuntu system (apropos 2 | awk ‘{ print $1 }’ | grep ‘[a-zA-Z]2[a-zA-Z]’ | wc -l).  Since the wrapper and the code it generates has very little direct dependence on python/cython, and since it may be useful for those who want to automate wrapping fortran inside a C program, the name will likely change.  I’m thinking of something along the lines of ‘fwrap’ — Dag suggested ‘fbridge’ to emphasize the connectivity aspect.

f2cy, which is using the fortran parser ‘fparser’ from the latest version of f2py (http://code.google.com/p/f2py/) is able to parse pretty much everything required for the midterm review.  The rest of f2cy is still rapidly developing.  The package lives inside the Tools directory of the cython distribution package. To take a look (it’s still in pre-alpha stage) checkout the mercurial repo here: http://hg.cython.org/gsoc-kurt/.

After fparser generates the parse tree, f2cy makes a few passes on the AST (using the Visitor pattern); one pass adds type ‘annotations’ to the dummy arguments of the subprograms to be wrapped, the next pass generates the autoconfig fortran source file, the next one generates the fortran wrapper.  Only scalar-valued arguments are supported for now, but by August f2cy will support arrays (assumed-shape, assumed-size and explicitly shaped) and passing derived types (structs) end-to-end from cython/C.  How is this done?  Thanks to the iso_c_binding intrinsic module within the Fortran 2003 standard, a Fortran program can be portably linked to C code without too much pain — certainly less pain than was required before.  This module allows f2cy to easily wrap subprograms that take scalars, arrays and derived types — in the distant future, f2cy may support passing C callbacks.  There still is an issue of resolving the kind-type-parameters for the dummy arguments, but that is what the autoconfig file is for.

Our goal in the project is to generate fortran wrappers that are completely portable, in the same way that Cython itself is portable.  A number of projects will generate an extension module with Cython and ship it without a Cython dependence, so the source has to work with any C compiler.  In the same way, f2py will generate wrappers that do not have any compiler-specific knowledge (including the particular values for the kind-type-parameters for the various types).  This isn’t purely academic, since some compilers define integer type KTPs consecutively (1,2,3,…) while for others (gfortran, xlf) the KTPs correspond to the byte-size (1,4,8,10,…).  This makes for some complications and makes the autoconfig file necessary.  More on the autoconfig in a future post.

I’ll address the cython-side in a short while.

References for Cython/Python buffers

April 21, 2009

PEP 3118, the buffer protocol.

http://www.python.org/dev/peps/pep-3118/

CEP 514, native buffer support in cython

http://wiki.cython.org/enhancements/buffer

Cython/Includes/numpy.pxd

http://hg.cython.org/cython-devel/file/58ff50bda693/Cython/Includes/numpy.pxd

Ecce Blogo

April 21, 2009

It was suggested that I create a blog for my Google Summer of Code project, “Improved Fortran support in Cython.”

Ecce blogo! (or whatever the Latin is for ‘blog.’)

Just to test some things out…

Here’s some python code:

def foo():
    print "hello python"

Here’s some C code:

int main(int argc, char **argv)
{
    printf("hello C\n");
    return 0;
}

Unfortunately wordpress doesn’t have specific fortran support just yet.  But I can do unformatted:

subroutine frobnicate(in_arr, out_arr)
  implicit none
  integer, dimension(:,:), intent(in) :: in_arr
  complex, intent(inout), dimension(:,:) :: out_arr
end subroutine frobnicate