Installing/compiling pylzma (lzma python binding)

I've already posted this question on the authors website, but I thought I might ask here as well.

I've been trying to install pylzma with this setup:

• Windows 7 x64
• Python 2.6.6 x64
• the amd64 compiler coming from windows server 2003 sdk
• cloned the git repo git://github.com/fancycode/pylzma.git

With a simple easy_install pylzma I got this:

cl : Command line warning D9025 : overriding '/MD' with '/MT' pylzma.c src/pylzma/pylzma.c(85) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data

src/pylzma/pylzma.c(102) : error C2275: 'CSha256' : illegal use of this type as an expression c:\users\xavier.lapointe\appdata\local\temp\easy_install-2mfkqu\pylzma-0.4.3\src\pylzma../7zip/C/Sha256.h(18) : see declaration of 'CSha256'

src/pylzma/pylzma.c(102) : error C2146: syntax error : missing ';' before identifier 'sha'

src/pylzma/pylzma.c(102) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(103) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(103) : warning C4133: 'function' : incompatible types - from 'int *' to 'CSha256 *'

src/pylzma/pylzma.c(104) : error C2143: syntax error : missing ';' before 'type'

src/pylzma/pylzma.c(105) : error C2143: syntax error : missing ';' before 'type'

src/pylzma/pylzma.c(106) : error C2143: syntax error : missing ';' before 'type'

src/pylzma/pylzma.c(107) : error C2143: syntax error : missing ';' before 'type'

src/pylzma/pylzma.c(108) : error C2065: 'round' : undeclared identifier

src/pylzma/pylzma.c(108) : error C2065: 'round' : undeclared identifier

src/pylzma/pylzma.c(108) : error C2065: 'rounds' : undeclared identifier

src/pylzma/pylzma.c(108) : error C2065: 'round' : undeclared identifier

src/pylzma/pylzma.c(109) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(109) : warning C4133: 'function' : incompatible types - from 'int *' to 'CSha256 *'

src/pylzma/pylzma.c(110) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(110) : warning C4133: 'function' : incompatible types - from 'int *' to 'CSha256 *'

src/pylzma/pylzma.c(111) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(111) : warning C4133: 'function' : incompatible types - from 'int *' to 'CSha256 *'

src/pylzma/pylzma.c(111) : error C2065: 'temp' : undeclared identifier

src/pylzma/pylzma.c(112) : error C2065: 'i' : undeclared identifier

src/pylzma/pylzma.c(112) : error C2065: 'i' : undeclared identifier

src/pylzma/pylzma.c(112) : error C2065: 'i' : undeclared identifier

src/pylzma/pylzma.c(113) : error C2065: 'temp' : undeclared identifier

src/pylzma/pylzma.c(113) : error C2065: 'i' : undeclared identifier

src/pylzma/pylzma.c(113) : error C2109: subscript requires array or pointer type

src/pylzma/pylzma.c(116) : error C2065: 'sha' : undeclared identifier

src/pylzma/pylzma.c(116) : warning C4133: 'function' : incompatible types - from 'int *' to 'CSha256 *'

error: Setup script exited with error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe"' failed with exit status 2

Then after googling I figured it was about the compiler (c90 vs c99), some people suggested to change the .c extension to .cpp. So I did this and changed the setup.py accordingly (mostly files name).

Then success, it compiles after a few tweaks in CpuArch.c(pp) and another file which had syntax errors.

The thing is now the linker choke:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\dev\pylzma\build\temp.win-amd64-2.6\Release\src\pylzma /LIBPATH:C:\Python26\libs /LIBPATH:C:\Python26\PCbuild\amd64 user32.lib oleaut32.lib /EXPORT:initpylzma build\temp.win-amd64-2.6\Release\src/pylzma/pylzma.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_decompressobj.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_compressfile.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_decompress.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_compress.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_streams.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_aes.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_decompress_compat.obj build\temp.win-amd64-2.6\Release\src/pylzma/pylzma_decompressobj_compat.obj build\temp.win-amd64-2.6\Release\src\sdk\LzFind.obj build\temp.win-amd64-2.6\Release\src\sdk\LzmaDec.obj build\temp.win-amd64-2.6\Release\src\sdk\LzmaEnc.obj build\temp.win-amd64-2.6\Release\src\7zip\C\CpuArch.obj build\temp.win-amd64-2.6\Release\src\7zip\C\Aes.obj build\temp.win-amd64-2.6\Release\src\7zip\C\AesOpt.obj build\temp.win-amd64-2.6\Release\src\7zip\C\Sha256.obj build\temp.win-amd64-2.6\Release\src\compat\LzmaCompatDecode.obj build\temp.win-amd64-2.6\Release\src/sdk/LzFindMt.obj build\temp.win-amd64-2.6\Release\src/sdk/Threads.obj /OUT:build\lib.win-amd64-2.6\pylzma.pyd /IMPLIB:build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.lib /MANIFESTFILE:build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.pyd.manifest

pylzma.obj : warning LNK4197: export 'initpylzma' specified multiple times; using first specification Creating library build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.lib and object build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.exp

pylzma.obj : error LNK2001: unresolved external symbol "char const * const doc_decompress" (?doc_decompress@@3QBDB)

pylzma.obj : error LNK2001: unresolved external symbol "struct _object * __cdecl pylzma_decompress(struct _object *,struct _object *)" (?pylzma_decompress@@YAPEAU_object@@PEAU1@0@Z)

pylzma.obj : error LNK2001: unresolved external symbol "char const * const doc_compress" (?doc_compress@@3QBDB)

Aes.obj : error LNK2019: unresolved external symbol "void __cdecl AesCtr_Code_Intel(unsigned int *,unsigned char *,unsigned __int64)" (?AesCtr_Code_Intel@@YAXPEAIPEAE_K@Z) referenced in function AesGenTables

Aes.obj : error LNK2019: unresolved external symbol "void __cdecl AesCbc_Decode_Intel(unsigned int *,unsigned char *,unsigned __int64)" (?AesCbc_Decode_Intel@@YAXPEAIPEAE_K@Z) referenced in function AesGenTables

Aes.obj : error LNK2019: unresolved external symbol "void __cdecl AesCbc_Encode_Intel(unsigned int *,unsigned char *,unsigned __int64)" (?AesCbc_Encode_Intel@@YAXPEAIPEAE_K@Z) referenced in function AesGenTables

AesOpt.obj : error LNK2019: unresolved external symbol _mm_aesenclast_si128 referenced in function "void __cdecl AesCbc_Encode_Intel(union __m128i *,union __m128i *,unsigned __int64)" (?AesCbc_Encode_Intel@@YAXPEAT__m128i@@0_K@Z)

AesOpt.obj : error LNK2019: unresolved external symbol _mm_aesenc_si128 referenced in function "void __cdecl AesCbc_Encode_Intel(union __m128i *,union __m128i *,unsigned __int64)" (?AesCbc_Encode_Intel@@YAXPEAT__m128i@@0_K@Z)

AesOpt.obj : error LNK2019: unresolved external symbol _mm_aesdeclast_si128 referenced in function "void __cdecl AesCbc_Decode_Intel(union __m128i *,union __m128i *,unsigned __int64)" (?AesCbc_Decode_Intel@@YAXPEAT__m128i@@0_K@Z)

AesOpt.obj : error LNK2019: unresolved external symbol _mm_aesdec_si128 referenced in function "void __cdecl AesCbc_Decode_Intel(union __m128i *,union __m128i *,unsigned __int64)" (?AesCbc_Decode_Intel@@YAXPEAT__m128i@@0_K@Z) build\lib.win-amd64-2.6\pylzma.pyd : fatal error LNK1120: 10 unresolved externals

error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe"' failed with exit status 1120

Now I've a hard time trying to figure out what these linker issues are, even with the msdn doc (about linkers error). Probably due to my lack of knowledge with C/C++.

So basically my question(s) are:

• Is there a way to fix these linker issues?
• Am I wrong to rename those files from .c to .cpp. If yes, is it possible to compile it as .c with the msvc amd64 compiler?

EDIT Haven't got much answer, so I think I'll add some questions:

• What's your approach when it comes to linker error?

• Is there any logic I can follow to help myself find answers over the web (combined with this MSDN linker errors)?

• Since I've seen many people compile it successfully for 32 bit system, is it possible that it compiles fine in 64bit, but fails when it comes to linking because of the architecture type?

I'm looking more for hints that might help than a fully working answer.

Thanks a lot for your help (:

I have installed pylzma to try this out myself. For python 2.6 it builds fine with mingw, but msvc compiler chokes for python2.4-2.7. Let's see errors first:

src/pylzma/pylzma.c(102) : error C2275: 'CSha256' : illegal use of this type as an expression c:\users\xavier.lapointe\appdata\local\temp\easy_install-2mfkqu\pylzma-0.4.3\src\pylzma../7zip/C/Sha256.h(18) : see declaration of 'CSha256'

Code in question is:

} else {
    Py_BEGIN_ALLOW_THREADS
    Sha256_Init(&sha);
    CSha256 sha;
    long round;
    int i;
    long rounds = (long) 1 << cycles;
    unsigned char temp[8] = { 0,0,0,0,0,0,0,0 };
    for (round = 0; round < rounds; round++) {

which is clearly invalid C since it's not allowed to declare variables after code. When you renamed .c to .cpp you avoided this error since C++ allows for this (and mingw supports this for C apparently). But switching to C++ led to name mangling and linking errors. To fix those you could place extern "C" { ... } around all the code.

It's better to fix src/pylzma/pylzma.c htough and the fix is trivial - move Py_BEGIN_ALLOW_THREADS and Sha256_Init(&sha); after variable declarations:

} else {
    CSha256 sha;
    long round;
    int i;
    long rounds = (long) 1 << cycles;
    unsigned char temp[8] = { 0,0,0,0,0,0,0,0 };
    Py_BEGIN_ALLOW_THREADS
    Sha256_Init(&sha);
    for (round = 0; round < rounds; round++) {

Now pylzma compiles fine but fails to run manifest tool after linking:

Creating library build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.lib and o bject build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.exp C:\Program Files\Microsoft SDKs\Windows\v7.0\bin\x64\mt.exe -nologo -manifest bu ild\temp.win-amd64-2.6\Release\src/pylzma\pylzma.pyd.manifest -outputresource:bu ild\lib.win-amd64-2.6\pylzma.pyd;2

build\temp.win-amd64-2.6\Release\src/pylzma\pylzma.pyd.manifest : general error c1010070: Failed to load and parse the manifest. The system cannot find the file specified. error: command 'mt.exe' failed with exit status 31

Looking in build directory reveals that there is no pylzma.pyd.manifest there though it's seen from output that link.exe has /MANIFEST:... switch. Quick googling for "link didn't create manifest" finds http://bugs.python.org/issue4431 where it's explained that when using /MT switch manifest doesn't get created and the solution is to add /MANIFEST to linker flags. Alright, let's edit pylzma's setup.py to add that linker flag for MSVC compiler:

    if isinstance(self.compiler, MSVCCompiler):
        # set flags only available when using MSVC
        ext.extra_link_args.append('/MANIFEST') # force linker to create manifest
        if COMPILE_DEBUG:
            ext.extra_compile_args.append('/Zi')
            ext.extra_compile_args.append('/MTd')
            ext.extra_link_args.append('/DEBUG')
        else:
            ext.extra_compile_args.append('/MT')

Voila, pylzma builds with MSVC fine now. I have tested building with python 2.4-2.7 32-bit and 2.6-2.7 64-bit:

08.02.2011  10:08            71 844 pylzma-0.4.3dev-py2.4-win32.egg
08.02.2011  10:09            71 480 pylzma-0.4.3dev-py2.5-win32.egg
08.02.2011  10:07            79 358 pylzma-0.4.3dev-py2.6-win-amd64.egg
08.02.2011  10:09            75 637 pylzma-0.4.3dev-py2.6-win32.egg
08.02.2011  10:08            79 259 pylzma-0.4.3dev-py2.7-win-amd64.egg
08.02.2011  10:09            75 540 pylzma-0.4.3dev-py2.7-win32.egg
               6 File(s)        453 118 bytes

As of general approach to any errors - it probably requires some knowledge and experience to understand what's behind them.