[Swig-user] Memory leak when freeing an array of pointers

Discussion:

Sampsa Riikonen

2017-06-09 11:05:22 UTC

Dear List,

I am having hard time in interfacing python with C++ code featuring an
array of pointers.

My class constructor reserves an array of pointers with "new". Upon
destruction, that array of pointers is released with "delete[]". That
should be all legit.

There's no problem when running the code *without* python. However,
when I "swig'it" and use it as a python module, something weird
happens. The destructor is called correctly upon garbage collection,
but while doing that, a memory leak occurs. Please feel free to answer
alternatively/also here:

https://stackoverflow.com/questions/44455184/memory-leak-with-swig

(1) Preliminaries to get it compiled:

setup.py

from setuptools import setup, Extension, find_packages
import os
import copy
import sys

def make_pps():
d=[]
c=[]
l=[]
lib=[]
s=[]
s+=["-c++"]
return
Extension("_pps",sources=["pps.i","pps.cpp"],include_dirs=d,extra_compile_args=c,extra_link_args=l,libraries=lib,swig_opts=s)

ext_modules=[]
ext_modules.append(make_pps())

setup(
name = "pps",
version = "0.1",
packages = find_packages(),
install_requires = ['docutils>=0.3'],
ext_modules=ext_modules
)

pps.i

%module pps
%{
#define SWIG_FILE_WITH_INIT
%}

%inline %{

class MemoryManager {

public:
MemoryManager(int n);
~MemoryManager();
};

%}

(2) The C++ code itself:

pps.cpp

#include <iostream>
#include <stdio.h>
#include <typeinfo>
#include <sys/time.h>
#include <stdint.h>
#include <cmath>

#include <cstring>
#include <string.h>
#include <stdlib.h>

#include<sys/ipc.h> // shared memory
#include<sys/shm.h>

/*
Stand-alone cpp program:
- UNComment #define USE_MAIN (see below)
- compile with

g++ pps.cpp

- run with

./a.out

=> OK

- Check memory leaks with

valgrind ./a.out

=> ALL OK/CLEAN

Python module:
- Comment (i.e. use) the USE_MAIN switch (see below)
- compile with

python3 setup.py build_ext; cp
build/lib.linux-x86_64-3.5/_pps.cpython-35m-x86_64-linux-gnu.so ./_pps.so

- run with

python3 test.py

=> CRASHHHHH

- Check memory leaks with

valgrind python3 test.py

=> Whoa..

- Try to enable/disable lines marked with "BUG?" ..

*/

// #define USE_MAIN 1 // UNcomment this line to get stand-alone
c-program

using std::cout;
using std::endl;
using std::string;

class MemoryManager {

public:
MemoryManager(int n);
~MemoryManager();

private:
int nmax;
int nc; // next index to be used
uint nsum;
int* ids;
void** buffers;
};

MemoryManager::MemoryManager(int n) : nmax(n),nc(0) {
cout << "MemoryManager: nmax="<<this->nmax<<"\n";

this->buffers =new void*[this->nmax]; // BUG?
this->ids =new int [this->nmax];
this->nsum =0;
}

MemoryManager::~MemoryManager() {
printf("MemoryManager: destructor\n");
delete[] this->buffers; // BUG?
delete[] this->ids;
printf("MemoryManager: destructor: bye\n");
}

#ifdef USE_MAIN
int main(int argc, char *argv[]) {
MemoryManager* m;

m=new MemoryManager(1000);
delete m;

m=new MemoryManager(1000);
delete m;
}
#endif

(3) A test python program:

test.py

from pps import MemoryManager
import time

print("creating MemoryManager")
mem=MemoryManager(1000)
time.sleep(1)
print("clearing MemoryManager")
mem=None
print("creating MemoryManager (again)")
time.sleep(1)
mem=MemoryManager(1000)
time.sleep(1)
print("exit")

Luis C

2017-06-09 13:10:04 UTC

Permalink

Hi Sampsa,

I think the problem is on the C++ code. Nowadays, with the new standard
c++11/14 you shouldn't use the the new and delete operators. Is highly
recommend to change that to shared pointers.

Regards,
Luis

Post by Sampsa Riikonen
Dear List,
I am having hard time in interfacing python with C++ code featuring an
array of pointers.
My class constructor reserves an array of pointers with "new". Upon
destruction, that array of pointers is released with "delete[]". That
should be all legit.
There's no problem when running the code *without* python. However, when
I "swig'it" and use it as a python module, something weird happens. The
destructor is called correctly upon garbage collection, but while doing
that, a memory leak occurs. Please feel free to answer alternatively/also
https://stackoverflow.com/questions/44455184/memory-leak-with-swig
setup.py
from setuptools import setup, Extension, find_packages
import os
import copy
import sys
d=[]
c=[]
l=[]
lib=[]
s=[]
s+=["-c++"]
return Extension("_pps",sources=["pps.i","pps.cpp"],include_dirs=d,
extra_compile_args=c,extra_link_args=l,libraries=lib,swig_opts=s)
ext_modules=[]
ext_modules.append(make_pps())
setup(
name = "pps",
version = "0.1",
packages = find_packages(),
install_requires = ['docutils>=0.3'],
ext_modules=ext_modules
)
pps.i
%module pps
%{
#define SWIG_FILE_WITH_INIT
%}
%inline %{
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
};
%}
pps.cpp
#include <iostream>
#include <stdio.h>
#include <typeinfo>
#include <sys/time.h>
#include <stdint.h>
#include <cmath>
#include <cstring>
#include <string.h>
#include <stdlib.h>
#include<sys/ipc.h> // shared memory
#include<sys/shm.h>
/*
- UNComment #define USE_MAIN (see below)
- compile with
g++ pps.cpp
- run with
./a.out
=> OK
- Check memory leaks with
valgrind ./a.out
=> ALL OK/CLEAN
- Comment (i.e. use) the USE_MAIN switch (see below)
- compile with
python3 setup.py build_ext; cp build/lib.linux-x86_64-3.5/_pp
s.cpython-35m-x86_64-linux-gnu.so ./_pps.so
- run with
python3 test.py
=> CRASHHHHH
- Check memory leaks with
valgrind python3 test.py
=> Whoa..
- Try to enable/disable lines marked with "BUG?" ..
*/
// #define USE_MAIN 1 // UNcomment this line to get stand-alone
c-program
using std::cout;
using std::endl;
using std::string;
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
int nmax;
int nc; // next index to be used
uint nsum;
int* ids;
void** buffers;
};
MemoryManager::MemoryManager(int n) : nmax(n),nc(0) {
cout << "MemoryManager: nmax="<<this->nmax<<"\n";
this->buffers =new void*[this->nmax]; // BUG?
this->ids =new int [this->nmax];
this->nsum =0;
}
MemoryManager::~MemoryManager() {
printf("MemoryManager: destructor\n");
delete[] this->buffers; // BUG?
delete[] this->ids;
printf("MemoryManager: destructor: bye\n");
}
#ifdef USE_MAIN
int main(int argc, char *argv[]) {
MemoryManager* m;
m=new MemoryManager(1000);
delete m;
m=new MemoryManager(1000);
delete m;
}
#endif
test.py
from pps import MemoryManager
import time
print("creating MemoryManager")
mem=MemoryManager(1000)
time.sleep(1)
print("clearing MemoryManager")
mem=None
print("creating MemoryManager (again)")
time.sleep(1)
mem=MemoryManager(1000)
time.sleep(1)
print("exit")
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user

Sampsa Riikonen

2017-06-09 13:35:26 UTC

Permalink

Hi Luis,

Thanks for the tip.. however, that doesn't solve the problem either
(please check the "EDIT" of that stack overflow link)

Regards,

Sampsa

Post by Luis C
Hi Sampsa,
I think the problem is on the C++ code. Nowadays, with the new
standard c++11/14 you shouldn't use the the new and delete operators.
Is highly recommend to change that to shared pointers.
Regards,
Luis
Dear List,
I am having hard time in interfacing python with C++ code
featuring an array of pointers.
My class constructor reserves an array of pointers with "new".
Upon destruction, that array of pointers is released with
"delete[]". That should be all legit.
There's no problem when running the code *without* python.
However, when I "swig'it" and use it as a python module, something
weird happens. The destructor is called correctly upon garbage
collection, but while doing that, a memory leak occurs. Please
https://stackoverflow.com/questions/44455184/memory-leak-with-swig
<https://stackoverflow.com/questions/44455184/memory-leak-with-swig>
setup.py
from setuptools import setup, Extension, find_packages
import os
import copy
import sys
d=[]
c=[]
l=[]
lib=[]
s=[]
s+=["-c++"]
return
Extension("_pps",sources=["pps.i","pps.cpp"],include_dirs=d,extra_compile_args=c,extra_link_args=l,libraries=lib,swig_opts=s)
ext_modules=[]
ext_modules.append(make_pps())
setup(
name = "pps",
version = "0.1",
packages = find_packages(),
install_requires = ['docutils>=0.3'],
ext_modules=ext_modules
)
pps.i
%module pps
%{
#define SWIG_FILE_WITH_INIT
%}
%inline %{
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
};
%}
pps.cpp
#include <iostream>
#include <stdio.h>
#include <typeinfo>
#include <sys/time.h>
#include <stdint.h>
#include <cmath>
#include <cstring>
#include <string.h>
#include <stdlib.h>
#include<sys/ipc.h> // shared memory
#include<sys/shm.h>
/*
- UNComment #define USE_MAIN (see below)
- compile with
g++ pps.cpp
- run with
./a.out
=> OK
- Check memory leaks with
valgrind ./a.out
=> ALL OK/CLEAN
- Comment (i.e. use) the USE_MAIN switch (see below)
- compile with
python3 setup.py build_ext; cp
build/lib.linux-x86_64-3.5/_pps.cpython-35m-x86_64-linux-gnu.so
<http://pps.cpython-35m-x86_64-linux-gnu.so> ./_pps.so
- run with
python3 test.py
=> CRASHHHHH
- Check memory leaks with
valgrind python3 test.py
=> Whoa..
- Try to enable/disable lines marked with "BUG?" ..
*/
// #define USE_MAIN 1 // UNcomment this line to get
stand-alone c-program
using std::cout;
using std::endl;
using std::string;
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
int nmax;
int nc; // next index to be used
uint nsum;
int* ids;
void** buffers;
};
MemoryManager::MemoryManager(int n) : nmax(n),nc(0) {
cout << "MemoryManager: nmax="<<this->nmax<<"\n";
this->buffers =new void*[this->nmax]; // BUG?
this->ids =new int [this->nmax];
this->nsum =0;
}
MemoryManager::~MemoryManager() {
printf("MemoryManager: destructor\n");
delete[] this->buffers; // BUG?
delete[] this->ids;
printf("MemoryManager: destructor: bye\n");
}
#ifdef USE_MAIN
int main(int argc, char *argv[]) {
MemoryManager* m;
m=new MemoryManager(1000);
delete m;
m=new MemoryManager(1000);
delete m;
}
#endif
test.py
from pps import MemoryManager
import time
print("creating MemoryManager")
mem=MemoryManager(1000)
time.sleep(1)
print("clearing MemoryManager")
mem=None
print("creating MemoryManager (again)")
time.sleep(1)
mem=MemoryManager(1000)
time.sleep(1)
print("exit")
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user
<https://lists.sourceforge.net/lists/listinfo/swig-user>

Luis C

2017-06-09 15:15:42 UTC

Permalink

HI Sampsa,

If I use this definition and write a .h file works for me, why your
original crash.

%module pps
%{
#include "pps.h"
%}

%include "pps.h"

%inline %{

%}

By the way you don need to use fancy pointers :)

Regards,
Luis

Post by Sampsa Riikonen
Hi Luis,
Thanks for the tip.. however, that doesn't solve the problem either
(please check the "EDIT" of that stack overflow link)
Regards,
Sampsa
Hi Sampsa,
I think the problem is on the C++ code. Nowadays, with the new standard
c++11/14 you shouldn't use the the new and delete operators. Is highly
recommend to change that to shared pointers.
Regards,
Luis

Burlen Loring

2017-06-09 18:43:33 UTC

Permalink

The cause of the problem is that you have 2 definition of the class,
giving the same class two memory layouts depending which definition is
used by the compiler.

from your .i file

%inline %{
class MemoryManager {
public:
MemoryManager(int n);
~MemoryManager();
};
%}

from your .cpp file

class MemoryManager {
public:
MemoryManager(int n);
~MemoryManager();
private:
int nmax;
int nc; // next index to be used
uint nsum;
int* ids;
void** buffers;
};

each definition appears in one of two translation units during
compilation and these are linked together into your module. Now, you
have only one definition of constructor and destructor, but in the swig
generated code the compiler see's the class definition in your .i, which
does not declare class member variables, and as a result it allocates no
space for the class member variables! Then at run time your
constructor/destructors trash some memory.

A hint is that the first error reported from valgrind is

==21419== Invalid write of size 4
==21419== at 0xD3B9857: MemoryManager::MemoryManager(int)
(pps.cpp:100)

You can confirm conclusively by adding a wrapped function in each
translation unit that returns sizeof(MemoryManager), and print the
values in your test script.

In your .i file add:

%inline %{
int mmsize(){ return sizeof(MemoryManager); }
int mmosize();
%}

In your cpp file add:

int mmosize(){ return sizeof(MemoryManager);

in your test script add:

print mmsize()
print mmosize()

On my system the printed values are 1 and 32.

Basically the take away is: use a single class definition, put it in a
header, pass that to swig by include directive.

Other points: You should always have or delete the copy constructor and
assignment operator for a class like this, and it would also be wise to
implement or delete a default constructor to initialize pointers to null.

Burlen

Post by Sampsa Riikonen
Dear List,
I am having hard time in interfacing python with C++ code featuring an
array of pointers.
My class constructor reserves an array of pointers with "new". Upon
destruction, that array of pointers is released with "delete[]". That
should be all legit.
There's no problem when running the code *without* python. However,
when I "swig'it" and use it as a python module, something weird
happens. The destructor is called correctly upon garbage collection,
but while doing that, a memory leak occurs. Please feel free to
https://stackoverflow.com/questions/44455184/memory-leak-with-swig
setup.py
from setuptools import setup, Extension, find_packages
import os
import copy
import sys
d=[]
c=[]
l=[]
lib=[]
s=[]
s+=["-c++"]
return
Extension("_pps",sources=["pps.i","pps.cpp"],include_dirs=d,extra_compile_args=c,extra_link_args=l,libraries=lib,swig_opts=s)
ext_modules=[]
ext_modules.append(make_pps())
setup(
name = "pps",
version = "0.1",
packages = find_packages(),
install_requires = ['docutils>=0.3'],
ext_modules=ext_modules
)
pps.i
%module pps
%{
#define SWIG_FILE_WITH_INIT
%}
%inline %{
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
};
%}
pps.cpp
#include <iostream>
#include <stdio.h>
#include <typeinfo>
#include <sys/time.h>
#include <stdint.h>
#include <cmath>
#include <cstring>
#include <string.h>
#include <stdlib.h>
#include<sys/ipc.h> // shared memory
#include<sys/shm.h>
/*
- UNComment #define USE_MAIN (see below)
- compile with
g++ pps.cpp
- run with
./a.out
=> OK
- Check memory leaks with
valgrind ./a.out
=> ALL OK/CLEAN
- Comment (i.e. use) the USE_MAIN switch (see below)
- compile with
python3 setup.py build_ext; cp
build/lib.linux-x86_64-3.5/_pps.cpython-35m-x86_64-linux-gnu.so ./_pps.so
- run with
python3 test.py
=> CRASHHHHH
- Check memory leaks with
valgrind python3 test.py
=> Whoa..
- Try to enable/disable lines marked with "BUG?" ..
*/
// #define USE_MAIN 1 // UNcomment this line to get stand-alone
c-program
using std::cout;
using std::endl;
using std::string;
class MemoryManager {
MemoryManager(int n);
~MemoryManager();
int nmax;
int nc; // next index to be used
uint nsum;
int* ids;
void** buffers;
};
MemoryManager::MemoryManager(int n) : nmax(n),nc(0) {
cout << "MemoryManager: nmax="<<this->nmax<<"\n";
this->buffers =new void*[this->nmax]; // BUG?
this->ids =new int [this->nmax];
this->nsum =0;
}
MemoryManager::~MemoryManager() {
printf("MemoryManager: destructor\n");
delete[] this->buffers; // BUG?
delete[] this->ids;
printf("MemoryManager: destructor: bye\n");
}
#ifdef USE_MAIN
int main(int argc, char *argv[]) {
MemoryManager* m;
m=new MemoryManager(1000);
delete m;
m=new MemoryManager(1000);
delete m;
}
#endif
test.py
from pps import MemoryManager
import time
print("creating MemoryManager")
mem=MemoryManager(1000)
time.sleep(1)
print("clearing MemoryManager")
mem=None
print("creating MemoryManager (again)")
time.sleep(1)
mem=MemoryManager(1000)
time.sleep(1)
print("exit")
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
https://lists.sourceforge.net/lists/listinfo/swig-user