SWIG, c++ to Python: array of pointers (double pointer) not working

[Matteo]

Re-posting in more simple and precise terms from a previous thread
http://groups.google.it/group/comp.lang.python/browse_thread/thread/6...

Problem:
SWIG doesn't properly wrap c++ arrays of pointers, therefore when you
try to call a c++ function which requires them, a TypeError exception
is raised.

Similar story here:
http://osdir.com/ml/programming.swig/2003-02/msg00064.html

Already tried:
- some ctypes functions
- tuple or string instead of list

Possibile solutions:
something like
http://embedded.eecs.berkeley.edu/Alumni/pinhong/scriptEDA/pyTypemapF...
that didn't work either, but I think I was not able to adapt the code
to my case, since the example is poorly explained.

Code to reproduce error:
I made a dptest.cpp function that calculates the sum of an array of
pointers to ints.

#include "dptest.h"
//the header file is just
//int sum(int**, int);

int sum(int** dp, int len){
int sum = 0;
for (int i = 0; i < len; i++){
sum += *(dp[i]);
}
return sum;

}

swig -c++ -python, then setup.py build_ext --inplace gets it nicely
compiled and wrapped for python use. It also is imported without
problems, but then...

***@matteo:~/lab/sandbox$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import dptest as dp
l = [1, 2, 3, 4]
size = len(l)
dp.sum(l,size)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: in method 'sum', argument 1 of type 'int **'

NOTE: A pure c++ program works as expected:

#include <iostream>

int sum(int**, int);

int main(){
int **array_of_ptr = new int*[4];
for (int i = 0; i < 4; i++){
array_of_ptr[i] = new int;
*array_of_ptr[i] = i+1; //fill it with 1,2,3,4: 1+2+3+4 = 10
}
std::cout << sum(array_of_ptr, 4) << std::endl;

}

int sum(int** dp, int len){
int sum = 0;
for (int i = 0; i < len; i++){
sum += *(dp[i]);
}
return sum;

}

compiling and running prints the correct result:
***@matteo:~/lab/sandbox$ ./purecpp
10

[William S Fulton]

Post by Matteo
Re-posting in more simple and precise terms from a previous thread
http://groups.google.it/group/comp.lang.python/browse_thread/thread/6...
SWIG doesn't properly wrap c++ arrays of pointers, therefore when you
try to call a c++ function which requires them, a TypeError exception
is raised.
http://osdir.com/ml/programming.swig/2003-02/msg00064.html
- some ctypes functions
- tuple or string instead of list
something like
http://embedded.eecs.berkeley.edu/Alumni/pinhong/scriptEDA/pyTypemapF...
that didn't work either, but I think I was not able to adapt the code
to my case, since the example is poorly explained.
I made a dptest.cpp function that calculates the sum of an array of
pointers to ints.
#include "dptest.h"
//the header file is just
//int sum(int**, int);
int sum(int** dp, int len){
int sum = 0;
for (int i = 0; i < len; i++){
sum += *(dp[i]);
}
return sum;
}
swig -c++ -python, then setup.py build_ext --inplace gets it nicely
compiled and wrapped for python use. It also is imported without
problems, but then...
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import dptest as dp
l = [1, 2, 3, 4]
size = len(l)
dp.sum(l,size)

File "<stdin>", line 1, in <module>
TypeError: in method 'sum', argument 1 of type 'int **'
#include <iostream>
int sum(int**, int);
int main(){
int **array_of_ptr = new int*[4];
for (int i = 0; i < 4; i++){
array_of_ptr[i] = new int;
*array_of_ptr[i] = i+1; //fill it with 1,2,3,4: 1+2+3+4 = 10
}
std::cout << sum(array_of_ptr, 4) << std::endl;
}
int sum(int** dp, int len){
int sum = 0;
for (int i = 0; i < len; i++){
sum += *(dp[i]);
}
return sum;
}
10

The default typemaps for pointer to pointers do not assume that it is an
array of integer pointers. There is nothing in the type int ** that
suggests that this is the only case -- there are other possibilities
that int ** could be used for. As such the appropriate typemaps need
writing. I can't think of any library typemaps you can use, but there
must be some around somewhere if you search for them, otherwise write
them yourself.

William

[Josh Cherry]

Post by Matteo
I made a dptest.cpp function that calculates the sum of an array of
pointers to ints.
#include "dptest.h"
//the header file is just
//int sum(int**, int);
int sum(int** dp, int len){
int sum = 0;
for (int i = 0; i < len; i++){
sum += *(dp[i]);
}
return sum;
}
swig -c++ -python, then setup.py build_ext --inplace gets it nicely
compiled and wrapped for python use. It also is imported without
problems, but then...
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import dptest as dp
l = [1, 2, 3, 4]
size = len(l)
dp.sum(l,size)

File "<stdin>", line 1, in <module>
TypeError: in method 'sum', argument 1 of type 'int **'

For a function like that that doesn't modify its argument, you could add a
function that takes a const std::vector<int>& and calls the existing
function to do the work.

Josh

[Matteo]

It seems I'm quite close to the solution, but it still doesn't work. I
wrote an appropriate typedef, so that SWIG can understand what I mean
with an array of pointers. I found the code on the link in my first post
and slightly adapted it.
This is my dptest.i file, but I still get the "argument of type int**":


%module dptest

%define array_in(T)
%typemap(in) T **{
if(PyList_Check($input)) {
int size = PyList_Size($input);
int i = 0;
$1 = (T **) malloc(size*sizeof(T*));
for(i = 0; i < size; i++) {
PyObject *o = PyList_GetItem($input,i);
T *ptr;
if ((SWIG_ConvertPtr(o,(void **) &ptr,
SWIGTYPE_p_T,1)) == -1) return NULL;
$1[i] = ptr;
}
} else {
PyErr_SetString(PyExc_TypeError,"not a list type");
return NULL;
}
}

%typemap(freearg) T **{
free($input);
}

%enddef

%{
#define SWIG_FILE_WITH_INIT
#include "dptest.h"
%}

%include "dptest.h"
array_in(int);

[Josh Cherry]

Post by Matteo
%include "dptest.h"
array_in(int);

A typemap must come before a declaration it's supposed to apply to.

Josh

[Matteo]

I had tried it indeed, but it won't compile:

dptest_wrap.cxx: In function ‘PyObject* _wrap_sum(PyObject*, PyObject*)’:
dptest_wrap.cxx:2773: error: ‘SWIGTYPE_p_T’ was not declared in this scope
dptest_wrap.cxx:2790: error: ‘$input’ was not declared in this scope
dptest_wrap.cxx:2795: error: ‘$input’ was not declared in this scope
error: command 'gcc' failed with exit status 1

Am I missing something?

Post by Josh Cherry

Post by Matteo
%include "dptest.h"
array_in(int);

A typemap must come before a declaration it's supposed to apply to.
Josh

[William S Fulton]

You need to learn the C preprocessor. SWIGTYPE_p_T is going to remain as
is. Consider using SWIGTYPE_p_##T which will then expand into
SWIGTYPE_p_int.

William

Post by Matteo
dptest_wrap.cxx:2773: error: ‘SWIGTYPE_p_T’ was not declared in this scope
dptest_wrap.cxx:2790: error: ‘$input’ was not declared in this scope
dptest_wrap.cxx:2795: error: ‘$input’ was not declared in this scope
error: command 'gcc' failed with exit status 1
Am I missing something?

Post by Josh Cherry

Post by Matteo
%include "dptest.h"
array_in(int);

A typemap must come before a declaration it's supposed to apply to.
Josh

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user

[Matteo]

Thank you for your help, it's the very first time I'm using SWIG.

I put ##T, as you said, but I still get a compile error, it seems that
there is a problem with SWIGTYPE_p_int as well:

dptest_wrap.cxx: In function ‘PyObject* _wrap_sum(PyObject*, PyObject*)’:
dptest_wrap.cxx:2773: error: ‘SWIGTYPE_p_int’ was not declared in this scope

Post by William S Fulton
You need to learn the C preprocessor. SWIGTYPE_p_T is going to remain
as is. Consider using SWIGTYPE_p_##T which will then expand into
SWIGTYPE_p_int.
William

Post by Matteo
dptest_wrap.cxx:2773: error: ‘SWIGTYPE_p_T’ was not declared in this scope
dptest_wrap.cxx:2790: error: ‘$input’ was not declared in this scope
dptest_wrap.cxx:2795: error: ‘$input’ was not declared in this scope
error: command 'gcc' failed with exit status 1
Am I missing something?

Post by Josh Cherry

Post by Matteo
%include "dptest.h"
array_in(int);

A typemap must come before a declaration it's supposed to apply to.
Josh

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user

[Josh Cherry]

Or use something like $descriptor(T *).

However, it seems to me that calling SWIG_ConvertPtr is the wrong thing to
do. The object that you get from the list (o) is not a SWIG
representation of an int*. Rather, it is a Python integer. You would use
something like PyInt_AsLong to get the integer, and PyInt_Check to check
it. You would need to store those integers somewhere and create an array
of pointers to them. You may need to call Py_DECREF on o to avoid a
memory leak.

It would be easier to add a function to the interface that takes an int*
or, as I suggested earlier, a const vector<int>&.

Josh

You need to learn the C preprocessor. SWIGTYPE_p_T is going to remain as is.
Consider using SWIGTYPE_p_##T which will then expand into SWIGTYPE_p_int.
William

dptest_wrap.cxx:2773: error: ‘SWIGTYPE_p_T’ was not declared in this scope
dptest_wrap.cxx:2790: error: ‘$input’ was not declared in this scope
dptest_wrap.cxx:2795: error: ‘$input’ was not declared in this scope
error: command 'gcc' failed with exit status 1
Am I missing something?

Post by Josh Cherry

Post by Matteo
%include "dptest.h"
array_in(int);

A typemap must come before a declaration it's supposed to apply to.
Josh

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user

[Nitro]

Post by Josh Cherry
Or use something like $descriptor(T *).
However, it seems to me that calling SWIG_ConvertPtr is the wrong thing to
do. The object that you get from the list (o) is not a SWIG
representation of an int*. Rather, it is a Python integer. You would use
something like PyInt_AsLong to get the integer, and PyInt_Check to check
it. You would need to store those integers somewhere and create an array
of pointers to them. You may need to call Py_DECREF on o to avoid a
memory leak.

Even simpler, you can use the traits fragment stuff (which is part of the
UTL I think) and just use the swig::from or swig::from_ptr. Because of
partial template specialization it will automatically resolve to the
fitting code ( be it PyInt_AsLong if T is int or if T is an
std::vector<T2> it will automatically place the code to covert from a
python sequence to an std::vector<T2> ).
Actually this is a mechanism I'd love to see manifested and used a bit
more in SWIG, because it's very useful and you don't have to whip up
custom and redundant typemap code. The swig::from, swig::from_ptr,
swig::as and swig::as_ptr templates are single entry points to do
conversion between C++ and target language (e.g. Python-to-C++ or
C++-to-Python). One place where I use this is to write generic OUTPUT&
typemaps (e.g. wrapping a function with a prototype like "void
generateVector( std::vector<int>& vec )" ).

Post by Josh Cherry
It would be easier to add a function to the interface that takes an int*
or, as I suggested earlier, a const vector<int>&.

Yes, that's usually the easiest way. If you sequences are very large, I
recommend to take a look at numpy and the numpy.i interface file which
deals with lots of this stuff in a performant, somewhat standard manner.

-Matthias

[Matteo]

OK, now I'm feeling stupid, so I'm going to ask you to be so generous to
write a proper wrapper function for me. I thought that it would have
been enough to do:

#include...blahblahdefinitions...
int py_sum_wrapper(int* list, int len){
return sum(&list, len);
}

but that returns the same dumb runtime error:
TypeError: in method 'py_sum_wrapper', argument 1 of type 'int *'

so that I only achieved removing a * from it :/
Using [] instead of * won't help. I probably won't need performance
tweaking for long lists, since I have a few thousands of them to
process, but they have 3 to 4 elements each.

Post by Nitro

Post by Josh Cherry
It would be easier to add a function to the interface that takes an int*
or, as I suggested earlier, a const vector<int>&.

Yes, that's usually the easiest way. If you sequences are very large,
I recommend to take a look at numpy and the numpy.i interface file
which deals with lots of this stuff in a performant, somewhat standard
manner.
-Matthias

[Nitro]

Post by Matteo
OK, now I'm feeling stupid, so I'm going to ask you to be so generous to
write a proper wrapper function for me. I thought that it would have
#include...blahblahdefinitions...
int py_sum_wrapper(int* list, int len){
return sum(&list, len);
}
TypeError: in method 'py_sum_wrapper', argument 1 of type 'int *'
so that I only achieved removing a * from it :/
Using [] instead of * won't help. I probably won't need performance
tweaking for long lists, since I have a few thousands of them to
process, but they have 3 to 4 elements each.

%include
%{
int py_sum_wrapper( const std::vector<int>& myList )
{
if ( myList.size() > 0 )
return sum( &myList[0], myList.size() );
else
return sum( 0, 0 );
}
%}

Call in python like

myModule.py_sum_wrapper( [1, 2, 3] )
or
myModule.py_sum_wrapper( (1, 2, 3) )


-Matthias

[Matteo]

Thank you Nitro, but this won't compile, because:
dptest_wrap.cxx:2755: error: cannot convert ‘const std::vector<int,
std::allocator<int> >*’ to ‘int**’ for argument ‘1’ to ‘int sum(int**, int)’
error: command 'gcc' failed with exit status 1

:(

Post by Matteo
%include
%{
int py_sum_wrapper( const std::vector<int>& myList )
{
if ( myList.size() > 0 )
return sum( &myList[0], myList.size() );
else
return sum( 0, 0 );
}
%}
Call in python like
myModule.py_sum_wrapper( [1, 2, 3] )
or
myModule.py_sum_wrapper( (1, 2, 3) )
-Matthias

[Matteo]

Do you have any documentation on how to use swig::as_ptr and others? I
can't find any :/

Post by Nitro
Even simpler, you can use the traits fragment stuff (which is part of
the UTL I think) and just use the swig::from or swig::from_ptr.
Because of partial template specialization it will automatically
resolve to the fitting code ( be it PyInt_AsLong if T is int or if T
is an std::vector<T2> it will automatically place the code to covert
from a python sequence to an std::vector<T2> ).
Actually this is a mechanism I'd love to see manifested and used a bit
more in SWIG, because it's very useful and you don't have to whip up
custom and redundant typemap code. The swig::from, swig::from_ptr,
swig::as and swig::as_ptr templates are single entry points to do
conversion between C++ and target language (e.g. Python-to-C++ or
C++-to-Python). One place where I use this is to write generic OUTPUT&
typemaps (e.g. wrapping a function with a prototype like "void
generateVector( std::vector<int>& vec )" ).

[Matteo]

AT LAST! After reading more of the docs, I got it working. Now I need to
push that to the next level. I don't need an interface for int** arrays,
in fact I need one for custom object from a C++ WireHit class, because I
have a function that takes a WireHit** array of pointers. The following
typemap uses the PyInt_AsLong() function, but that would not be
available for a generic class. How could I replace it? Any suggestion to
further improve the code? I'm afraid that the new int is a memory leak,
since it's not deleted anywhere...


%module dptest

%{
#define SWIG_FILE_WITH_INIT
#include "dptest.h"
#include <iostream>
%}

%typemap(in) int **{
if(PyList_Check($input)) {
int size = PyList_Size($input);
int i = 0;
$1 = (int **) malloc((size)*sizeof(int*));
for(i = 0; i < size; i++) {
PyObject *o = PyList_GetItem($input, i);
int *ptr = new int;
*ptr = PyInt_AsLong(o);
$1[i] = ptr;
}
} else {
PyErr_SetString(PyExc_TypeError,"not a list type");
return NULL;
}
}

%typemap(freearg) int **{
free($1);
}

%include "dptest.h"

[Haoyu Bai]

Post by Matteo
AT LAST! After reading more of the docs, I got it working. Now I need to
push that to the next level. I don't need an interface for int** arrays,
in fact I need one for custom object from a C++ WireHit class, because I
have a function that takes a WireHit** array of pointers. The following
typemap uses the PyInt_AsLong() function, but that would not be
available for a generic class. How could I replace it? Any suggestion to
further improve the code? I'm afraid that the new int is a memory leak,
since it's not deleted anywhere...
%module dptest
%{
#define SWIG_FILE_WITH_INIT
#include "dptest.h"
#include <iostream>
%}
%typemap(in) int **{
 if(PyList_Check($input)) {
   int size = PyList_Size($input);
   int i = 0;
   $1 = (int **) malloc((size)*sizeof(int*));
   for(i = 0; i < size; i++) {
     PyObject *o = PyList_GetItem($input, i);
     int *ptr = new int;
     *ptr = PyInt_AsLong(o);
     $1[i] = ptr;
   }
 } else {
   PyErr_SetString(PyExc_TypeError,"not a list type");
   return NULL;
 }
}
%typemap(freearg) int **{
free($1);
}
%include "dptest.h"

Hi,

The following code demostrates what I did once I want to wrap an array
of a C struct called 'feature':


%include <cpointer.i>

/* Wrap the struct feature** as an array */
%pointer_functions(struct feature*, FeatureArr);
%inline %{
struct feature* FeatureArr_get(struct feature** featarr, int i)
{
return *featarr + i;
}

struct feature* FeaturePtrArr_get(struct feature** featptrarr, int i)
{
return *(featptrarr + i);
}
%}

So at Python side you can call FeatureArr_get(feat, i) or
FeaturePtrArr_get(feat, i) to access the array. There's two function
just because there's two means of struct feature**: pointer of an
array, or an array of pointer.

Hope it help!

-- Haoyu Bai