distorm3 doesn't install on Windows 10 #155
Comments
|
Hi
What VS version you use?
Thanks
…On Tue, Jan 19, 2021 at 22:15 smclinden ***@***.***> wrote:
It doesn't install from PIP and attempting to build from source leads to
the following:
building '_distorm3' extension
Calling 'vcvarsall.bat x86' (version=9.0)
C:\Users\testaccount\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DSUPPORT_64BIT_OFFSET -DDISTORM_DYNAMIC -Isrc -Iinclude -IC:\Python27\include -IC:\Python27\PC /Tcsrc\decoder.c /Fobuild\temp.win32-2.7\Release\src\decoder.obj
decoder.c
C:\Users\testaccount\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DSUPPORT_64BIT_OFFSET -DDISTORM_DYNAMIC -Isrc -Iinclude -IC:\Python27\include -IC:\Python27\PC /Tcsrc\distorm.c /Fobuild\temp.win32-2.7\Release\src\distorm.obj
distorm.c
src\distorm.c(320) : error C2143: syntax error : missing ';' before 'type'
src\distorm.c(321) : error C2275: '_OffsetType' : illegal use of this type as an expression
c:\users\testaccount\distorm\src../include/distorm.h(110) : see declaration of '_OffsetType'
src\distorm.c(321) : error C2146: syntax error : missing ';' before identifier 'offset'
src\distorm.c(321) : error C2065: 'offset' : undeclared identifier
src\distorm.c(321) : warning C4244: '=' : conversion from 'const _OffsetType' to 'int', possible loss of data
src\distorm.c(345) : error C2065: 'offset' : undeclared identifier
src\distorm.c(346) : error C2065: 'size' : undeclared identifier
error: command '"C:\Users\testaccount\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe"' failed with exit status 2
In distorm.c the lines around 320 look like:
unsigned int opcode = di->opcode;
unsigned int prefix = FLAG_GET_PREFIX(di->flags);
mnemonic = (const _WMnemonic*)&_MNEMONICS[opcode]; <--
unsigned int size = di->size;
_OffsetType offset = di->addr & addrMask;
Visual C doesn't permit assignments before declarations in some instances.
The solution is to move the flagged line to be after all declarations are
completed, e.g.,
unsigned int opcode = di->opcode;
unsigned int prefix = FLAG_GET_PREFIX(di->flags);
unsigned int size = di->size;
mnemonic = (const _WMnemonic*)&_MNEMONICS[opcode]; <--
_OffsetType offset = di->addr & addrMask;
A similar issue exists in prefix.c, line 141:
for (unsigned int index = 0;
(ci->codeLen > 0) && (index < INST_MAXIMUM_SIZE);
ci->code++, ci->codeLen--, index++) {
Needs to be replaced with:
unsigned int index;
for (index = 0;
(ci->codeLen > 0) && (index < INST_MAXIMUM_SIZE);
ci->code++, ci->codeLen--, index++) {
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#155>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWXOBNLZ7ZEOO7I6GILS2XR7ZANCNFSM4WJLNRJA>
.
|
|
If you do a PIP install, it requires Visual C++ for Python 2.7 which is VS 9. I can, manually, build with later versions of VC (2019) and then install distorm3 by hand, which works but is a bit of a pain. |
|
Hi @smclinden! |
|
Seems to work for me with special VC for Python 2.7 Waiting for your confirmation anyway. |
|
I will check it out. I can also post the manual steps to build it with newer versions of Visual C, if there is an interest. |
|
@smclinden If you think it might be useful I will add it to the wiki, then yes, sure. Thanks |
|
Interesting... I have often had issues install 2.7 without doing a manual compile, but I could never work out why -- the information PIP gives you (along with a suggestion to look at a log file that you can't find) was absolutely zero help. But I also never had an issue just compiling by hand. I only have VS2017 on this box. Does python 2.7 seriously install VS9 ... that's Visual Studio 2008 -- guess we best keep those I don't know anything about PIP, but I do know it's possible to upload prebuilt binaries. shrug. I'm sure I could whip together a BASH script for the job :) |
|
Yes. I compiled distorm3 using VS2019 after figuring out where to put the output and then built the package manually. I think that if the setup script could be rewritten to check to see if a valid VS was already installed and attempt to build with that, it would probably make the package a little more portable. |
|
This setup.py distutils is F@cked. So much pain from such a simple (supposed to be) tool. |
|
I'm not sure if this helps (I don't use distutils, much): https://stackoverflow.com/questions/51976015/select-visual-studio-version-in-distutils-setup |
|
I'm not going back that path. You can have a look at the history of that
stupid file (setup.py) and see it used to find vcvars on its own and what's
not...
Eventually some guy suggested to keep it plain stupid and now it works.
For me personally it has always worked. But I guess that there are way too
many different environments.
Anyway, does it work for you two now? I can't follow :)
…On Sun, Feb 28, 2021 at 4:06 PM smclinden ***@***.***> wrote:
I'm not sure if this helps (I don't use distutils, much):
https://stackoverflow.com/questions/51976015/select-visual-studio-version-in-distutils-setup
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#155 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWVEQRBLFTK335U3CETTBJEWJANCNFSM4WJLNRJA>
.
|
|
From what is currently on my system, I get this (running from a python2.7 setup.py install --verbose) Let's call it a baseline. They really messed with vcvarsall.bat in 2017 and 2019 and ruined all my own lovely build scripts, so this is not that surprising. Let me investigate further now. Same from pip2 running in the same devenv. I thought I had fix this already with https://github.com/gdabah/distorm/pull/137/commits (pulls and updates setup.py from there) .... ok, no errors, but no compile under 2 or 3, and that was from within a vsdev environment too. From a non-vsdev environment, it's the same story. Lame. :) I don't have the disk-space to install a py27 built-dist (or whatever it's called) on either of my dev machines, I'm down to the megabytes right now, so any fixes I perform aren't going to apply to anyone who isn't already running vs2017 or 2019 (2015 and earlier used the standard vcvarsall.bat arrangement). The only other thing I have to offer is my 2015:
I used to use Enterprise 2017, so I know that not only can the path be 2017 or 2019, but the folder can be Evidently I didn't call vsvarsall.bat, but that has more to do with being in cygwin and needing to be in a Getting the default VS9 compatibility is starting to sound very attractive right about now. |
|
Okay, taking a second look. Despite all the notes I see in distutils/msvc9compiler.py regarding things needing to be compiled with VS2008, my Python (installed via the setup binary downloaded from where-ever python lives) says: That's Visual Studio 2017, right there. So this is all nonsense: [ Now to fix the root problem for all packages. Prepend the following to the function Want a solution that doesn't involve modifying your past end of lifetime, non-supported python 2.7 installation? Call Python, I'm sure they'll be glad to help. Are you a package maker suffering from having to support binary builds on supported and uncooperative Microsoft platforms? Upload pre-compiled binaries. Problem solved. Expediency is my middle name. def find_vcvarsall(version):
# --[SNIP]-- (start of inserted section)
import glob
paths = glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\20*\*\vc\Auxiliary\Build\vcvarsall.bat")
if paths:
print("{} possible locations for vcvarsall.bat found:".format(len(paths)))
for i, path in enumerate(paths):
print("\t[{}] {}".format(i, path))
if len(paths) > 1:
print("picking one at random:")
paths.shuffle()
path = paths[0]
print("selecting: {}".format(path))
return path
# otherwise, do what it would have done normally
# --[SNIP]-- (end of inserted section)
"""Find the vcvarsall.bat file
At first it tries to find the productdir of VS 2008 in the registry. If
that fails it falls back to the VS90COMNTOOLS env var.
"""
vsbase = VS_BASE % version
# ... and so forth, and so on.And here it is purring along nicely. And via the manual route: All happy. |
|
I am telling you. I had it and removed it. As it was quirky.
Do you launch the vs command prompt bat file and then try to build btw?
…On Mon, Mar 1, 2021 at 06:08 sfinktah ***@***.***> wrote:
Okay, taking a second look. Despite all the notes I see in
distutils/msvc9compiler.py regarding things needing to be compiled with
VS2008, my Python (installed via the setup binary downloaded from
where-ever python lives) says:
>>> import sys
>>> print sys.version
2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)]
That's *Visual Studio 2017*, right there. So this is all nonsense: [
python27-x64/lib/distutils/msvc9compiler.py]
raise DistutilsPlatformError(
"""Python was built with Visual Studio 2008;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2008 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.""")
Now to fix the root problem for all packages. Insert the following to the
start of find_vcvarsall
def find_vcvarsall(version):
[msvc9compiler.zip](https://github.com/gdabah/distorm/files/6059091/msvc9compiler.zip)
# --[SNIP]-- (start of changes)
import glob
paths = glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\20*\*\vc\Auxiliary\Build\vcvarsall.bat")
if paths:
print("{} possible locations for vcvarsall.bat found:".format(len(paths)))
for i, path in enumerate(paths):
print("\t[{}] {}".format(i, path))
if len(paths) > 1:
print("picking one at random:")
paths.shuffle()
path = paths[0]
print("selecting: {}".format(path))
return path
# otherwise, do what it would have done normally
# --[SNIP]-- (end of changes)
"""Find the vcvarsall.bat file At first it tries to find the productdir of VS 2008 in the registry. If that fails it falls back to the VS90COMNTOOLS env var. """
vsbase = VS_BASE % version
# ... and so forth, and so on.
And here it is purring along nicely.
E:\git\distorm>py -2 -m pip install distorm3
Collecting distorm3
Using cached https://files.pythonhosted.org/packages/93/c4/71dcf0fa683c887253319d92c5d2319fb94cc3f89f7d908fd8275df48cc5/distorm3-3.5.1.tar.gz
Building wheels for collected packages: distorm3
Running setup.py bdist_wheel for distorm3 ... done
Stored in directory: C:\Users\sfink\AppData\Local\pip\Cache\wheels\36\be\76\10d3d3fd7158c1f26189a546de3ca3db8df61d05187770a249
Successfully built distorm3
Installing collected packages: distorm3
Successfully installed distorm3-3.5.1
And via the manual route:
E:\git\distorm>py -2 setup.py install
('__import__', 'setuptools.msvc')
('__import__', 'distutils.msvc9compiler')
('__import__', 'distutils.msvc9compiler')
('__import__', 'distutils._msvccompiler')
('__import__', 'distutils._msvccompiler')
running install
running bdist_egg
running egg_info
writing python\distorm3.egg-info\PKG-INFO
writing top-level names to python\distorm3.egg-info\top_level.txt
writing dependency_links to python\distorm3.egg-info\dependency_links.txt
reading manifest file 'python\distorm3.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'python\distorm3.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
running build_ext
building '_distorm3' extension
1 possible locations for vcvarsall.bat found:
[0] C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\vc\Auxiliary\Build\vcvarsall.bat
selecting: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\vc\Auxiliary\Build\vcvarsall.bat
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
creating build\temp.win-amd64-2.7\Release\src
creating build\temp.win-amd64-2.7\Release\python
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DSUPPORT_64BIT_OFFSET -DDISTORM_DYNAMIC -Isrc -Iinclude -IC:\Python27-x64\include -IC:\Python27-x64\PC /Tcsrc\decoder.c /Fobuild\temp.win-amd64-2.7\Release\src\decoder.obj
decoder.c
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DSUPPORT_64BIT_OFFSET -DDISTORM_DYNAMIC -Isrc -Iinclude -IC:\Python27-x64\include -IC:\Python27-x64\PC /Tcsrc\distorm.c /Fobuild\temp.win-amd64-2.7\Release\src\distorm.obj
distorm.c
...
All happy.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#155 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWXRQ7B5XVZI5BHEBY3TBMHM3ANCNFSM4WJLNRJA>
.
|
|
@gdabah "it was quirky"... yeah, no arguments here, it simply failed silently when I tested it. The Just to repeat -- this is because of the modification to distutils of the python2 distribution itself. It may be possible to incorporate that (and nothing else) into your own build config, though I am dubious. You could also just provide compiled copies of the Also, having vs2017 or vs2019 is a requirement for the patch to work (well, it won't break without them, it just won't do anything), though vs2015 should conform to the "old school" detection methods. If someone just doesn't have visual studio installed, they're still going to be screwed. |
|
@gdabah I can go ahead and make the changes suggested by @smclinden into a PR, I should have a VM somewhere I can test them on with a vanilla Windows 10 install. Reviewing them, (and it took a while to mentally shift gears back from C++), they're the usual super-strict classic C89 rules. I'll check if there's an option to relax the parsing first, though that option may be /TP (which will enable it to parse as C++). So we'll see how that goes. |
|
I better first look at the history of the setup file to see if there’s
going to be any difference. And to see why we chose to remove it (finding
vcvars bat file).
…On Thu, Mar 4, 2021 at 05:06 sfinktah ***@***.***> wrote:
@gdabah <https://github.com/gdabah> I can go ahead and make the changes
suggested by @smclinden <https://github.com/smclinden> into a PR, I
should have a VM somewhere I can test them on with a vanilla Windows 10
install.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#155 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWQKABHWBFPUJPF2VODTB32L5ANCNFSM4WJLNRJA>
.
|
|
I checked the current build against GCC in -std=c89, and gave it a quick eye-ball looking for I found a couple of tiny things while making a tinycc build, and that was it. Even tinycc doesn't stoop to C89. It would appear the reason all my py2 builds failed was that I monkey-patched my distutils back when I was using Enterprise VS, and hardcoded the path. FWIW the changes I made to your setup.py involved the VsWhere batch (command?) tool that comes with the newer versions of VS, where-as the changes I suggested for distutils don't attempt to get smart (I think that's where they went wrong really). They just use a simple I also did some reading on setuptools vs distutils, and thought that might be the answer (after all, it has it's own .py scripts which have different logic for finding MSVC) but I eventually found them frustrating and probably incapable of providing for compilation of the distorm.dll without employing distutils anyway. |
|
Appendix: I reverted to a stock distutils, and installed the VC9Python as instructed by pip, and compiled the git/distorm folder with |
|
It is probably worth mentioning that this is a 64 bit build, where-as I believe the OP was performing a 32 bit build. I made some changes in my last PR that fixed some issues with 32 bit builds and |
|
@smclinden |
|
@gdabah Well, just looking at my own previous comments regarding having no issues compiling my branch, I attempted an install via pip (since you included my tcc PR), and it seems to build fine. I guess I just have the magic touch. I can only check the 64-bit version though.
|
|
distorm3.pyd That’s the one
…On Sun, May 9, 2021 at 05:53 sfinktah ***@***.***> wrote:
@gdabah <https://github.com/gdabah> Well, just looking at my own previous
comments regarding having no issues compiling *my* branch, I attempted an
install via pip (since you included my tcc PR), and it seems to build fine.
I just can't find distorm3.dll afterwards. I'm really quite confused.
C:\Users\sfink\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27-x64\libs /LIBPATH:C:\Python27-x64\PCbuild\amd64 /LIBPATH:C:\Python27-x64\PC\VS9.0\amd64 /EXPORT:init_distorm3 build\temp.win-amd64-2.7\Release\src\decoder.obj build\temp.win-amd64-2.7\Release\src\distorm.obj build\temp.win-amd64-2.7\Release\src\instructions.obj build\temp.win-amd64-2.7\Release\src\insts.obj build\temp.win-amd64-2.7\Release\src\mnemonics.obj build\temp.win-amd64-2.7\Release\src\operands.obj build\temp.win-amd64-2.7\Release\src\prefix.obj build\temp.win-amd64-2.7\Release\src\textdefs.obj build\temp.win-amd64-2.7\Release\python/python_module_init.obj /OUT:build\lib.win-amd64-2.7\_distorm3.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\src\_distorm3.lib /MANIFESTFILE:build\temp.win-amd64-2.7\Release\src\_distorm3.pyd.manifest
python_module_init.obj : warning LNK4197: export 'init_distorm3' specified multiple times; using first specification
Creating library build\temp.win-amd64-2.7\Release\src\_distorm3.lib and object build\temp.win-amd64-2.7\Release\src\_distorm3.exp
running install_lib
creating C:\Python27-x64\Lib\site-packages\distorm3
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#155 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWT3MC6BSZBVLCVBKA3TMX2K3ANCNFSM4WJLNRJA>
.
|
|
I'm slow, but I get there in the end... or rather, |
|
It can be great if you do it one day :) THX!
…On Sun, Jun 13, 2021 at 11:34 AM sfinktah ***@***.***> wrote:
I'm slow, but I get there in the end... or rather, find . -mtime 0 in
cygwin does. So AFAIK you have no problem anymore, but it would be nice to
test a 32 bit Python install to be 100% sure. I might have a virtual
machine somewhere I don't mind polluting with a 32 bit python...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#155 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADMEWS2SYCQQL6HSGK4EN3TSRUSNANCNFSM4WJLNRJA>
.
|
|
I totally just remembered to finish this. I built a fresh Windows 10 VM, then installed the final python 2.7.18-64bit, distorm3 installed no issues. I removed the 64bit python and installed the 32bit, and got the error: I visited that URL, to find (in true Microsoft style) that it was no longer valid, and they didn't care to fix it. End of test. Can you not just bundle a 32-bit DLL in the same manner as you assumedly bundled a 64-bit DLL? |
|
@sfinktah you mean for pypi? |
|
@gdabah yes -- isn't that the issue here? i mean, yes -- the OP suffered issues compiling, but he was only compiling because it didn't automatically work when obtained from |
The Visual C for Python 2.7 only supports C89 not C99. C89 does not support mixing of assignments and declarations nor does it support variable declarations in for loops.
As a result, distorm3 doesn't install from PIP and attempting to build from source leads to the following:
In distorm.c the lines around 320 look like:
Visual C doesn't permit assignments before declarations in some instances. The solution is to move the flagged line to be after all declarations are completed, e.g.,
A similar issue exists in prefix.c, line 141:
Needs to be replaced with:
The text was updated successfully, but these errors were encountered: