GSoC est fin, but fwrap lives on!

I’m glad to say that my Google Summer of code project was successful, despite the features that didn’t make it in fwrap by August 17th.

Oh, yeah — if you’re wondering what ‘fwrap’ is, that’s the new name of ‘f2cy’.  The rebranding was appropriate since ‘f2cy’ didn’t quite capture all that the utility does.  fwrap wraps fortran for a number of languages (C, Cython & Python), so ‘f2cy’ was deemed a misnomer since it seems to wrap fortran just for cython, besides the confusion with ‘f2py’.

fwrap was presented at the SciPy 2009 conference, and I was glad to find a good amount of interest; two (Kyle & Chris) stepped up to work on fwrap once I can get it to a state that’s comprehensible   They work on a code called ‘clawpack’ at the U-Washington, and want to use fwrap for their ‘pyclaw’ version of clawpack.  But I’m  getting ahead of myself.

Here’s where you can find the presentation abstract (note that it was written early June — its heavy on promised features that didn’t all make it in fwrap by August 17th), and here are the talk slides.  The presentation itself is here.

What’s the state of fwrap?  It can handle scalar arguments with aplomb.  Assumed-shape arrays are working.  Explict-shaped & assumed-size arrays are soon to come.  I’m sketching out callbacks as we speak (a feature that Kyle & Chris are particularly interested in).

The parser used by fwrap needs stabilization — some obvious things need work (public/private module attributes), and I’m finding out just how unruly the full Fortran language specification is.  The language allows you to be expulsively verbose & clunky (‘integer(kind=FOOBAR), intent(in), dimension(2) :: an_int_array’) — yet difficult to fully parse, since each attribute (‘intent,’ ‘dimension,’ ’save,’ etc.) can instead be a statement on a line a ways away from the actual ‘integer an_int_array’ line; or you can be cryptic & bug-prone by using the implicit declaration anti-feature well-hated by all those who have ever had a misspelled variable in their code.

The above rant is a long way of saying that there are many little things that fparser chokes on.  This is not to be interpreted as a complaint with fparser — I’m grateful for all of Pearu’s work thus far on it, and for his ambition to tackle parsing such a barnacled language.  He has graciously opened up fparser to me to work on it, and has allowed us to package fparser with fwrap.  Thanks, Pearu!

Anyway, work on fwrap will be a bit slow for the next week or two, as I turn my full attention to my research.  But I’m energized and motivated to get the first release out before 2010.

Update

I’m glad to say that the f2cy part of the GSoC miterm evaluation is functional. By that I mean it can wrap functions & subroutine subprograms (not in modules, yet) with any integer or real argument. The testing framework is all in place, and all tests pass. There are still many things that I’d like to add before the midterm, but the basics are all there, and the additions should be fairly modular and easy to do. The code could use some cleanup, but it shouldn’t be too bad.

Here are the some additions I’d like to make before the midterm if time allows — nothing to be inferred by the ordering except where noted:

1) Support locally defined parameter kind-types (easy), e.g.:

subroutine foo(a,b)
    integer :: DP
    parameter  (DP = 8 )
    integer, parameter :: SP = 4
    real(kind=DP), intent(inout) :: a
    real(kind=SP), intent(in) :: b
    ! ...
end subroutine

2) Next, support use’d module kind-type parameters (easy):

module ktps
  integer :: KTP
  parameter (KTP = 10)
end module ktps
integer function bar(a)
  use ktps
  real(kind=KTP), intent(inout) :: a
  ! ...
end function

3) Next, support module-scoped functions (some corner cases to consider, a bit harder) — just to give you an idea…

module container
  use ktps, local_kpts => KTPS
  integer, parameter :: C_KTPS = local_ktps
  contains
  subroutine mod_subr(a)
    use othermod, rename_p => orig_p
    use yetanother, only: yetanother_p => xxx
    implicit none
    integer(kind=C_KTPS) :: a
    integer(kind=other_mod_param), intent(out) :: b
    integer(kind=rename_p) :: c
    ! ...
  end subroutine
end module

4) Logical <-> Integer conversions (probably before the above)

The ‘logical’ type is basically a specially typed integer — whenever a subprogram’s argument is a logical, its C wrapper should use the equivalently-sized integer and do a conversion before & after the call.

5) Character arguments:

It seems that these are functionally equivalent to one-dimensional arrays, with a few bugaboos to watch out for. I’m lumping them in with the second-half of the project, since after working with integer/real arrays I’ll have a better feel as to the best way to handle character arrays.

6) Complex arguments:

Since we can’t assume C99 _Complex support, these need to be passed in as structs, converted to the native complex type in fortran, and converted back on return. Once (4) is done, I’ll have a better feel for these. I’m putting them off until the second half, since they’re equivalent to passing derived types.

Cython side:

I’m turning my full attention to tickets 299, 300 & 178 now, provided that the recent intense discussion about a native Cython array type leave these tickets unchanged (see http://article.gmane.org/gmane.comp.python.cython.devel/6654).

Long-overdue Update — ‘f2cy’

Much progress over the past few weeks, I’m glad to say.  Here are the main threads that are out there.

First, the 10000 meter view:  For my GSoC, I’m developing, under the guidance of Dag Sverre Seljebotn, (1) a python package that wraps fortran so it can be easily called from a cython program; (2) improving cython’s buffer support to allow the passing of buffers/arrays to external code (C, fortran, or whatever can be linked in) or cdef functions.

The fortran-side:

Dag Sverre and I discussed a bit about the naming of the fortran wrapper project — right now it’s under the working title “f2cy,” in the long tradition of “<some-language/protocol/format>2<other-language/protocol/format>” converter names.  There are at least 85 similarly-named executables on my ubuntu system (apropos 2 | awk ‘{ print $1 }’ | grep ‘[a-zA-Z]2[a-zA-Z]‘ | wc -l).  Since the wrapper and the code it generates has very little direct dependence on python/cython, and since it may be useful for those who want to automate wrapping fortran inside a C program, the name will likely change.  I’m thinking of something along the lines of ‘fwrap’ — Dag suggested ‘fbridge’ to emphasize the connectivity aspect.

f2cy, which is using the fortran parser ‘fparser’ from the latest version of f2py (http://code.google.com/p/f2py/) is able to parse pretty much everything required for the midterm review.  The rest of f2cy is still rapidly developing.  The package lives inside the Tools directory of the cython distribution package. To take a look (it’s still in pre-alpha stage) checkout the mercurial repo here: http://hg.cython.org/gsoc-kurt/.

After fparser generates the parse tree, f2cy makes a few passes on the AST (using the Visitor pattern); one pass adds type ‘annotations’ to the dummy arguments of the subprograms to be wrapped, the next pass generates the autoconfig fortran source file, the next one generates the fortran wrapper.  Only scalar-valued arguments are supported for now, but by August f2cy will support arrays (assumed-shape, assumed-size and explicitly shaped) and passing derived types (structs) end-to-end from cython/C.  How is this done?  Thanks to the iso_c_binding intrinsic module within the Fortran 2003 standard, a Fortran program can be portably linked to C code without too much pain — certainly less pain than was required before.  This module allows f2cy to easily wrap subprograms that take scalars, arrays and derived types — in the distant future, f2cy may support passing C callbacks.  There still is an issue of resolving the kind-type-parameters for the dummy arguments, but that is what the autoconfig file is for.

Our goal in the project is to generate fortran wrappers that are completely portable, in the same way that Cython itself is portable.  A number of projects will generate an extension module with Cython and ship it without a Cython dependence, so the source has to work with any C compiler.  In the same way, f2py will generate wrappers that do not have any compiler-specific knowledge (including the particular values for the kind-type-parameters for the various types).  This isn’t purely academic, since some compilers define integer type KTPs consecutively (1,2,3,…) while for others (gfortran, xlf) the KTPs correspond to the byte-size (1,4,8,10,…).  This makes for some complications and makes the autoconfig file necessary.  More on the autoconfig in a future post.

I’ll address the cython-side in a short while.

References for Cython/Python buffers

PEP 3118, the buffer protocol.

http://www.python.org/dev/peps/pep-3118/

CEP 514, native buffer support in cython

http://wiki.cython.org/enhancements/buffer

Cython/Includes/numpy.pxd

http://hg.cython.org/cython-devel/file/58ff50bda693/Cython/Includes/numpy.pxd

Ecce Blogo

It was suggested that I create a blog for my Google Summer of Code project, “Improved Fortran support in Cython.”

Ecce blogo! (or whatever the Latin is for ‘blog.’)

Just to test some things out…

Here’s some python code:

def foo():
    print "hello python"

Here’s some C code:

int main(int argc, char **argv)
{
    printf("hello C\n");
    return 0;
}

Unfortunately wordpress doesn’t have specific fortran support just yet.  But I can do unformatted:

subroutine frobnicate(in_arr, out_arr)
  implicit none
  integer, dimension(:,:), intent(in) :: in_arr
  complex, intent(inout), dimension(:,:) :: out_arr
end subroutine frobnicate