Diffstruc: A Fortran library for automatic differentiation

Announcements
1

Hi all,

I have just released diffstruc, an open source Fortran library for performing automatic differentiation (AD) using forward- and reverse-mode methods, enabling any order differentiation (but, as is common with AD, this is usually limited to 3rd order for any reasonable task due to memory limitations).

The library utilises derived types and pointers to build the necessary operation graphs to perform differentiation. I’ve spent some time trying to fix all memory errors and leaks, but I’ve probably missed some, so keep an eye out and let me know. The library works well, but isn’t nearly as fast as things like Pytorch, so if anyone wants to contribute to make this faster, feel free to get in touch or make pull requests. I am constantly working on this, so I expect to be tinkering away and trying to reduce memory usage further in the coming weeks/months.

The library has core maths, linear algebra, trigonometric, exponent, broadcasting, and reduction operations coded in, but the library is built with the principle of extendability (or whatever you want to call it), so adding new operations that can handle AD does not require modifying the source code, but instead overloading the operators of the derived type. I have been building this library to enable physics informed neural networks in athena (my Fortran neural network library), so functionality and design has come from needing to operate with that.

For more details, I have written up a ReadtheDocs to document some use-cases and a tutorial on how to add custom operators.

I hope someone else finds this useful in some way.

Feel free to ask any questions or share any suggestions!

25 Likes
2

Really interesting! I was interested in making a Fortran implementation of backward differentiation but (fortunately for me) you beat me to it. Looking at the documentation tutorials I see that the functions are defined with a pointer, for example:

f => 2.*x**2

How could I work with a more complex function? One that require loops or some kind of conditional. For example

f = 0

do i=1,N
    do j=1,N
       if (n(i) /= 1.0) then
            f = f  + n(i) * b(i) * b(j)
        end if
    end do
end do

Just as and example, I work with functions with a lot of different terms and parameters that are also functions of the variables. Also, is it possible to make arithmetic between functions?
Like in:

f => x**2 + 2
g => dot_product(x, y)

h => f + g
3

Thanks @fedebenelli. :slightly_smiling_face: Hopefully the library does what you want so that you don’t have to implement autodiff yourself (or, at least, provide you with a useful starting point).

Yes, your example should be implementable. The form will be something like this (but probably not exactly, it’s the weekend and my mind isn’t working on doing maths now):

f => sum( matmul( mask( n, n .ne. 1.0 )  matmul( b, b ) ) )

If it can’t be implemented that easily (or if you want to optimise for memory), I’d recommend referring to the custom operation making documentation to implement it as a specific function (Supported Operations — diffstruc documentation).

For the arithmetic function, you can totally do exactly what you’ve written (except I haven’t set up dot_product, I should probably do that one soon, so you’d need to use sum(x*y)).

The list of operations will definitely need to be expanded further over time, but a good number are already implemented.

2 Likes