Python/Installing pydasm and pydbg with Python 2.5, WinXP, VC++2008 Express Edition

Yesterday I tried installing pydbg and pydasm on my notepc.
pydasm is popular, famous library to disassemble machine codes (opcodes).
pydbg is also popular, famous library to build lightweight, extensible debugger for Windows platform.
Actually, pydbg is included in PaiMei, windows platform debugger framework.

The journey was hard, full of struggle and traps with python distutils.
I left these notes, memos, and traps for future person (including myself) who want installing these excellent reverse engineering tools written in python and c.

My notepc environments are:

CPU : Intel Pentium M (Centrino) 1.2GHz
RAM : 1GB
OS : Windows XP Professional SP3 (Japanese)
Python: Python 2.5 (install from MSI installer)
        Install Dir : C:\Python25
Visual Studio : Visual C++ 2008 Express Edision (SP1)
Subversion: TortoiseSVN 1.6.x

We require Subversion to obtain PaiMei later.

Okay, Let's begin.

• Installing pydasm step-by-step
  • getting libdasm
  • try compile -> errors!!
  • avoiding distutil's registry key checks
  • try compiling again -> build success!!
  • install pydasm and test ... oh-no, "MSVCR90.dll" not foud !?
  • edit msvccompiler.py again, change compile option -> pydasm works correct!!
• Installing pydbg step-by-step
  • checkout from SVN repositry and install -> "Python26.dll not found" !?
  • remove bundled pydasm.pyd in pydbg -> success!!
  • note for other modules in PaiMei

Installing pydasm step-by-step

pydbg requires pydasm. We have to install pydasm before installing pydbg.
But pydasm is written by c, and no official binary release for win32 platform.

Out environments uses:

Python 2.5
Visual C++ 2008 Express Edition SP1

getting libdasm

pydasm is included "libdasm", simple x86 disassembly library. Author of pydasm is another person to author of libdasm.
libdasm was written by jt (nologin.org), pydasm was written by ero (dkbza.org). pydasm was contributed to libdasm.

• dkbza - pydasm (just showing libdasm download link, short overview, example codes)
  • http://dkbza.org/pydasm.html

Now, let's download libdasm, including pydasm from following url:

• libdasm : nologin -- code
  • http://www.nologin.org/main.pl?action=codeView&codeId=49&

When download success, expand tar.gz into your favorite directory.
Yesterday, I get libdasm-1.5.tar.gz. After expanding, I got following directory trees:

libdasm-1.5\
    libdasm.c
    README.txt
    ...
    bin\
    pydasm\
    ...

try compile -> errors!!

Next, open command prompt, setup your VC++ environment (ex: vcvars32.bat). Confirm "C:\Python25" is added PATH environment or ".py" extension is associated with "C:\Python25\python.exe".
In your command prompt, cd to "libdasm-1.5\pydasm", and type:

> setup.py build_ext
running build_ext
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

Ouch!! Why? My command prompt enviroments are surely prepared for VC++ tools!!

> cl
Microsoft(R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.
...
> link
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

The answer is, Python's distutils tool (used from setup.py) checks Visual Studio's registry keys and confirm proper version is installed or not.
Unfortunately, registry keys installed by VC++2008 Express Edision on my environment are not what distutils expected.

avoiding distutil's registry key checks

Then how to avoid these annoying registry key checks ?
The answer is, edit distutil's library source code. Python is open source, and distutils are written in python.

(Skips my surveys and researches for distutils and its source code.)

As a result, edit following source code:

C:\Python25\Lib\distutils\msvccompiler.py

In msvccompiler.py:

...
class MSVCCompiler (CCompiler) :
    ...
    def __init__ (self, verbose=0, dry_run=0, force=0):
        ...
        # comment out here!!
        self.__macros = MacroExpander(self.__version)

Registry keys checks are implemented in MacroExpander() and its constructor. And, "self.__macros" is optional, not needed. So removing this line doesn't effect to compiling.
After removing unneccesary line, delete old ".pyc" file, msvccompiler.pyc, may be exists in same directory to ".py".

try compiling again -> build success!!

In msvccompiler.py, there's no registry checks when "DISTUTILS_USE_SDK" and "MSSdk" environments are set and "cl.exe" is found in PATH environment.
So we set "DISTUTILS_USE_SDK" and "MSSdk" in our command prompt window:

> set DISTUTILS_USE_SDK=1
> set MSSdk=1

Every thing is okay for each environment values. msvccompiler.py only checks each key is exists, ignoring values.

Okay, let's try compiling:

> setup.py build_ext
running build_ext
building 'pydasm' extension
creating build
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe \
    /c /nologo /Ox /MD /W3 /GX /DNDEBUG \
    -IC:\Python25\include -IC:\Python25\include -IC:\Python25\PC \
    /Tc../libdasm.c /Fobuild\temp.win32-2.5\Release\../libdasm.obj
C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe \
    /c /nologo /Ox /MD /W3 /GX /DNDEBUG \
    -IC:\Python25\include -IC:\Python25\include -IC:\Python25\PC \
    /Tcpydasm.c /Fobuild\temp.win32-2.5\Release\pydasm.obj
creating build\lib.win32-2.5
C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe \
    /DLL /nologo /INCREMENTAL:NO \
    /LIBPATH:C:\Python25\libs /LIBPATH:C:\Python25\PCBuild /EXPORT:initpydasm \
    build\temp.win32-2.5\Release\../libdasm.obj build\temp.win32-2.5\Release\pydasm.obj \
    /OUT:build\lib.win32-2.5\pydasm.pyd \
    /IMPLIB:build\temp.win32-2.5\Release\..\pydasm.lib

(omitting warnings from build output)

When all work correct, we'll get

libdasm-1.5\pydasm\build\lib.win32-2.5\pydasm.pyd

install pydasm and test ... oh-no, "MSVCR90.dll" not foud !?

Now invoking install command:

> setup.py install
running install
running build
running build_ext
running install_lib
copying build\lib.win32-2.5\pydasm.pyd -> C:\Python25\Lib\site-packages
copying build\lib.win32-2.5\pydasm.pyd.manifest -> C:\Python25\Lib\site-packages
running install_egg_info
Removing C:\Python25\Lib\site-packages\pydasm-1.5-py2.5.egg-info
Writing C:\Python25\Lib\site-packages\pydasm-1.5-py2.5.egg-info

Then invoking python and try to import pydasm:

> C:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydasm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: 指定されたモジュールが見つかりません。
>>> quit()

Oh-no, last "ImportError" says "specified module not found". And on my environment, "MSVCR90.dll not found" messagebox was shown.

Checking module dependencies:

> dumpbin /dependents C:\Python25\python.exe
...
   python25.dll
   MSVCR71.dll   <-- Python25 depends on MSVCR71.dll
   KERNEL32.dll
...
> dumpbin /dependents build\lib.win32-2.5\pydasm.pyd
...
   MSVCR90.dll    <-- pydasm depends on MSVCR90.dll !!
   python25.dll
   KERNEL32.dll

On my environment, MSVCR90.dll was installed to Visual Studio installed directory, but not to Windows system directory.If you've already installed VC-Runtime including MSVCR90.dll to your Windows system directory, those ImportError and message box may not be shown, all works correctly.

How do that ? There're 2 ways to resolve this error.

1. install VC-Runtime including MSVCR90.dll to Windows.
   1. This is straightforward, but Python uses MSVCR71.dll (compiled by older version VisualStudio). I'm afraid of conflict between runtime dlls.
2. link VC-Runtime static.
   1. This makes portable module, but need to change compile options, module file size become big.

I'm afraid of conflits between runtime dlls strongly (may be another trap), so I delve into second way.

edit msvccompiler.py again, change compile option -> pydasm works correct!!

Then how to change compile option ? What option should we choose ?
msvccompiler.py setups compiler options in MSVCCompiler class's initialize method.

msvccompiler.py:

...
class MSVCCompiler (CCompiler) :
...
    def initialize(self):
        ...
        self.preprocess_options = None
        if self.__arch == "Intel":
            self.compile_options = [ '/nologo', '/Ox', '/MD', '/W3', '/GX' ,
                                     '/DNDEBUG']
            self.compile_options_debug = ['/nologo', '/Od', '/MDd', '/W3', '/GX',
                                          '/Z7', '/D_DEBUG']

"/MD" and "/MDd" indicates dynamic linking multithreaded library.
For more details about runtime library and compiler options, see following MSDN.

• C Run-Time Libraries
  • http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx
    • "MSDN Library" > "Development Tools and Languages" > "Visual Studio 2010" > "Visual Studio" > "Visual Studio Languages" > "Visual C++" > "Visual C++ Reference" > "Visual C++ Libraries Reference" > "Run-Time Library" > "C Run-Time Libraries"

This time we want to link static multithreaded library, So we choose "/MT" and "/MTd".
Edit msvccompiler.py:

...
class MSVCCompiler (CCompiler) :
...
    def initialize(self):
        ...
        self.preprocess_options = None
        if self.__arch == "Intel":
            # change '/MD' => '/MT'
            self.compile_options = [ '/nologo', '/Ox', '/MT', '/W3', '/GX' ,
                                     '/DNDEBUG']
            # change '/MDd' => '/MTd'
            self.compile_options_debug = ['/nologo', '/Od', '/MTd', '/W3', '/GX',
                                          '/Z7', '/D_DEBUG']

Delete old msvccompiler.pyc and pydasm\build directory, and try build again.

> setup.py build_ext
> dumpbin /dependents build\lib.win32-2.5\pydasm.pyd
 Image has the following dependencies:

   python25.dll
   KERNEL32.dll
...
> setup.py install
...
> C:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydasm
>>> buffer = '\x90\x31\xc9\x31\xca\x31\xcb'
>>> offset = 0
>>> while offset < len(buffer):
...     i = pydasm.get_instruction(buffer[offset:], pydasm.MODE_32)
...     print pydasm.get_instruction_string(i, pydasm.FORMAT_INTEL, 0)
...     if not i:
...         break
...     offset += i.length
...
nop
xor ecx,ecx
xor edx,ecx
xor ebx,ecx
>>> quit()

All-right, pydasm seems to work correctly.

Next, try to install PaiMei, pydbg.

Installing pydbg step-by-step

• PaiMei Official Docs
  • http://pedram.redhive.com/PaiMei/docs/
• PaiMei Google Codes Site
  • http://code.google.com/p/paimei/
• Other Resources:
  • http://thatsbroken.com/?p=26
  • http://maliciousattacker.blogspot.com/2006/12/pydbg-in-vmware.html
  • http://www.openrce.org/downloads/details/208/PaiMei
  • http://maliciousattacker.blogspot.com/2007/01/setting-up-pydbg.html

checkout from SVN repositry and install -> "Python26.dll not found" !?

1st, checkout (or export) PaiMei latest source tree from SVN repositry. Use your favorite SVN tools.
Yesterday, I checkouted r248 trunk source tree.

2nd, open command prompt window, cd to PaiMei directory, and run setup.py.

> setup.py build
(...)
> setup.py install
(...)

3rd, try to import pydbg:

> C:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydbg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydbg\__init__.py", line 47, in <module>
    from pydbg                   import *
  File "C:\in_vitro\SVNWORK\paimei\pydbg\pydbg.py", line 32, in <module>
    import pydasm
ImportError: DLL load failed: 指定されたモジュールが見つかりません。

And message box was shown, saying "Python26.dll not found".

See carefully above python stacktrace. It is very strange that error has occurred at "import pydasm".
At previous section, we confirm "import pydasm" work correctly. What's happening?

remove bundled pydasm.pyd in pydbg -> success!!

The Answer is : "pydasm.pyd" is bundled in pydbg module directory, and it was compiled with Python 2.6.

> dumpbin /dependents C:\Python25\Lib\site-packages\pydbg\pydasm.pyd
                                                    ^^^^^^^^^^^^^^^^ pydasm.pyd was bundled!!
...
  Image has the following dependencies:

    MSVCR90.dll
    python26.dll
    KERNEL32.dll
...

Then, we simply remove bundled "pydbg\pydasm.pyd". Close command prompt, shutdown all python program, and re-open command prompt (clear loaded dlls).

Try-again:

> C:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydbg
>>>

All-right, no errors!!

Finally, invoke calc.exe, get its PID, and test following example:

from pydbg import *
from pydbg.defines import *
def handler_breakpoint (pydbg):
  if pydbg.first_breakpoint:
    print "[*] Hit 1st breakpoint!"
    return DBG_CONTINUE
  print "[*] Hit breakpoint!"
  return DBG_CONTINUE
dbg = pydbg()
dbg.set_callback(EXCEPTION_BREAKPOINT, handler_breakpoint)
dbg.attach(XXXXX) # pid of calc.exe
recv = dbg.func_resolve("user32", "ShowWindow")
dbg.bp_set(recv)
dbg.debug_event_loop()

Example:

> C:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydbg import *
>>> from pydbg.defines import *
>>> def handler_breakpoint (pydbg):
...   if pydbg.first_breakpoint:
...     print "[*] Hit 1st breakpoint!"
...     return DBG_CONTINUE
...   print "[*] Hit breakpoint!"
...   return DBG_CONTINUE
...
>>> dbg = pydbg()
>>> dbg.set_callback(EXCEPTION_BREAKPOINT, handler_breakpoint)
>>> dbg.attach(5084)
<pydbg.pydbg.pydbg instance at 0x00BF8198>
>>> recv = dbg.func_resolve("user32", "ShowWindow")
>>> dbg.bp_set(recv)
<pydbg.pydbg.pydbg instance at 0x00BF8198>
>>> dbg.debug_event_loop()
[*] Hit 1st breakpoint!
[*] Hit breakpoint!
...
(minimize, or restore calc window)
...
[*] Hit breakpoint!
>>> quit()

note for other modules in PaiMei

In this article, we JUST ONLY want to install pydbg in PaiMei.
This article ignored to setup other modules in PaiMei, like PIDA, GUI components.
If you want to setup these modules, PaiMei framework fullset, see documents in official site or "docs/index.html" in SVN source tree.