{"id":97,"date":"2013-06-18T23:12:24","date_gmt":"2013-06-19T06:12:24","guid":{"rendered":"https:\/\/www.falatic.com\/?p=97"},"modified":"2013-06-20T22:55:44","modified_gmt":"2013-06-21T05:55:44","slug":"howto-getting-started-with-python-development-in-windows-with-msvsgcc-cython-etc","status":"publish","type":"post","link":"https:\/\/www.falatic.com\/index.php\/97\/howto-getting-started-with-python-development-in-windows-with-msvsgcc-cython-etc","title":{"rendered":"Howto: Getting started with Python development in Windows with MSVS\/GCC, Cython, etc."},"content":{"rendered":"

Note: this is directed at Python 2.7 developers, but the same principles should apply to Python 3.x as well.<\/em><\/p>\n

First, a shout-out for the\u00a0Unofficial Windows Binaries for Python<\/a> site: as noted below you can get the binaries for many commonly used packages simply by downloading and installing them, with no need to compile.<\/p>\n

However, when it comes to things like Cython that NEED a working compiler at some point you’ll have to set one up. I wrote this post to help you with that, hopefully saving you many hours of frustration!<\/p>\n

Conclusions first: properly configuring Windows to compile for Python\/Cython takes a little effort but the hardest part is getting the details in one place (which this post is intended to do). The total setup time if you go the full Windows SDK\/Microsoft Visual Studio Express\/MinGW route isn’t that long – less than 30 minutes with a decent connection on a fast machine, with the majority of it spent in the download\/install phases.<\/p>\n

I DO suggest you install Microsoft Visual Studio Express 2008 and the\u00a0Windows SDK. As for MinGW, I strongly suspect there are limits to what you can ultimately do with it in this context but it’s presented here as an option for the adventurous (and it’s rather nice to have GCC handy in general!)<\/p>\n

<\/p>\n

\n

Windows isn’t the easiest system to work with, especially when you want to go the FOSS development route. In the case of Python 2.7 development involving things like Cython or many pip installs, if you don’t have the right Microsoft compilers installed (why?<\/a>) you’ll most likely see the dreaded “error: Unable to find vcvarsall.bat” at some point… usually the moment you REALLY don’t have time for it. What to do?<\/p>\n

One solution is to install Microsoft Visual Studio 2008 itself (at least the Express version, as well as the Windows SDK – it’s evident now that the SDK alone will NOT suffice). I’ve outlined that path below.<\/p>\n

Another possible solution is to use GCC via MinGW, but you’ll need to jump through a few hoops to do it (and it won’t work with Cython for x64 builds of Python<\/a>!) I can’t say for certain how far this approach will get you, but it worked for me when I tried it for 32-bit build tests.\u00a0YMMV… it’s probably not the most stable way to go for linking with a code-base originally built via MSVS.<\/p>\n

\n

Installing the Windows SDK<\/strong><\/p>\n

I’ve read that you can do everything with the Windows SDK alone, at least for Cython<\/a> (also here<\/a> and here<\/a>, among other places). So, get the\u00a0Windows SDK 7.1<\/a>\u00a0(IMHO) and install it. Note that you should only need the Developer tools, which clock in at 1.1 GB (Documentation and Samples take up an extra 3.5 GB!)<\/p>\n

You may get the following error: SDKSetup – Another running instance of this application was detected. Only one instance of this application may be running at any time.<\/em><\/p>\n

Close that error window and you get the Program Compatibility Assistant. I’m quite used to closing that thing out of habit and searching Google in frustration, which I did, and found zero answers to this problem. I searched running processes – nothing. I ran Process Monitor – nothing stood out. I don’t have another instance installed. WTH?<\/p>\n

After a reboot and a few more failed attempts, I accidentally chose the “Reinstall using recommended settings” option. Suddenly, it worked! I have NO idea what special setting Windows “chose” for me (there’s no way to readily see that) but it worked.<\/p>\n

At this point enjoy your favorite soothing beverage and\/or blood pressure medication while the installation proceeds at the speed of a turtle crossing a busy highway. Poor turtle!<\/p>\n

\n

All done? Good! I created a little build.cmd<\/em> script in the folder with my code that has the following (“::” is the same thing as “REM” in batch\/cmd files). You may need to adjust the paths slightly from what’s shown, and comment\/uncomment the appropriate bits (the SDK path may vary, for example):<\/p>\n

\r\n::@echo off\r\n\r\n:: Selects the compiler AND Python27 target this works with! (x86 or x64)\r\n::set MYTGT=x64\r\nset MYTGT=x86\r\n\r\n:: The SDK you actually installed\r\nset MYSDK=v7.1\r\n\r\n:: I've made basic installs of Python with distribute, pip and cython in\r\n:: folders C:\\Python27_x86 and C:\\Python_x64 (so as not to collide with\r\n:: my usual Python at C:\\Python27)\r\n:: We set the path accordingly\r\nPATH=C:\\Python27_%MYTGT%;C:\\Python27_%MYTGT%\\Scripts;%PATH%\r\n\r\n:: Replicate cmd option \/E:ON (command extensions - haven't needed yet)\r\n::setlocal EnableExtensions\r\n\r\n:: Replicate cmd option \/V:ON (delayed expansion - required)\r\nsetlocal EnableDelayedExpansion\r\n\r\n:: Run SetEnv as the SDK cmd shell does (needs to be the same tgt as Python's:\r\ncall "C:\\Program Files\\Microsoft SDKs\\Windows\\%MYSDK%\\Bin\\SetEnv.cmd" \/%MYTGT% \/release\r\n\r\n:: I've seen this setting mentioned for other projects but haven't had to use it myself\r\n::set MSSdk=1\r\n\r\n:: Select the proper SDK for Python 2.7 (6.0A versus a later one - VERY important!)\r\nset DISTUTILS_USE_SDK=1\r\n\r\necho.\r\necho -- Compiler and Python version check\r\necho.\r\necho Compilation target: %TARGET_PLATFORM%-%TARGET_CPU% for Python27_%MYTGT%\r\necho.\r\npython -c "import sys;print('Python '+sys.version.replace('\\n','')+' on '+sys.platform)"\r\n\r\n:: Your build commands go below, e.g.:\r\n\r\necho.\r\necho -- Building it\r\necho.\r\n\r\n::del *.pyd\r\n::python setup.py build_ext --inplace\r\n\r\necho.\r\necho -- Testing it\r\necho.\r\n\r\n::python testhello.py\r\n\r\necho.\r\necho -- Done!\r\necho.\r\npause\r\n\r\n<\/pre>\n
If running that worked (even with no code to compile it’ll at least turn green and mention that it’s using SDK v6.0A) then congratulations: you should be ready to go on! We hope!<\/p>\n
Note: If you try to run these commands directly in a command prompt it’ll likely *fail*. Either ensure you started cmd.exe with the\u00a0“\/V:ON” option, or start one from where you are at by typing\u00a0“cmd.exe \/V:ON”. Why? Because “setlocal EnableDelayedExpansion” ONLY works in a batch file, that’s why.<\/p>\n
You’ll need to perform these steps before installing any package that requires compilation (such as Cython) or \u00a0building your code. You could make some shortcuts so you can just click and have an environment ready to use.<\/p>\n
\n
Installing Microsoft Visual Studio Express 2008 (might not be necessary)<\/em><\/del> (oh yeah, it’s necessary!)<\/strong><\/p>\n
Though some of what I’ve read suggested MSVS is not required, to avoid the missing vcvarsall.bat<\/em> problem you still need to install MSVS “Express” 2008 as found in (of all things) a closed thread on StackOverflow<\/a>. You can find the official installer at\u00a0http:\/\/go.microsoft.com\/?linkid=7729279<\/a><\/p>\n
Skip the SQL part of the install to save time and space.<\/p>\n
Have another beverage… it’s gonna be a while<\/del>. Actually, it was blazingly fast on this machine. Still… refreshing!<\/p>\n
\n
Dealing with despair<\/strong><\/p>\n
At this point you may be considering wiping your system and installing some version of Linux (or possibly buying an entirely new piece of hardware simply to run a hardware-locked Linux-like commercial OS). But, you’re a trouper and you hung on this long – you may yet be rewarded! (And despite all this, knowing how to build in Windows is a handy if esoteric skill).<\/em><\/p>\n
\n
Installing MinGW\/GCC<\/strong><\/p>\n
Going the GCC route, or thinking about it?\u00a0Install the\u00a032-bit version of\u00a0TDM-GCC<\/a>\u00a0(the 32\/64-bit version will NOT work).<\/p>\n
When using newer versions of MinGW, you need to edit C:\\Python27\\Lib\\distutils\\cygwinccompiler.py<\/em> and remove all refs to -mno-cygwin<\/strong> (this is an obsolete option in GCC 4.7)<\/p>\n
To actually use MinGW32’s gcc instead of MSVS with Python, you’ll need to create C:\\Python27\\Lib\\distutils\\distutils.cfg<\/em> with the contents:<\/p>\n
\r\n\r\n[build]\r\n\r\ncompiler = mingw32\r\n\r\n<\/pre>\n
(If later you decide to use MSVS, be sure to rename or remove this file if it’s present. You probably should reinstall Python and your packages using MSVS at that point to ensure it’s consistent.)<\/p>\n
\n
Installing Python 2.7<\/strong><\/p>\n
Install Python 2.7.x from the main site<\/a>. I had three versions installed for the purpose of all this testing: the x86 one in\u00a0C:\\Python27_x86<\/em>, the x64 one in C:\\Python_x64<\/em>, and my “default” version in C:\\Python27<\/em>. that was really just a copy of one of the other two. You will likely just have one version installed (32- or 64-bit) so you’ll adjust the script above accordingly.<\/p>\n
People seem to use the 32-bit version quite a lot…. the one with “X86-64” by it is the 64-bit version (note that the 64-bit version will identify itself as AMD64 when you run it. Why so many ways to say the same thing, you ask?<\/a>)You can always use the 32-bit version, while the 64-bit one only works with a 64-bit OS. Also, 64-bit modules sometimes have less support so keep that in mind as well. Not sure? Do what I did and build for both! \ud83d\ude42<\/p>\n
Another random note: IA64 is NOT another name for x64\/X86-64\/AMD64<\/a> and it’s totally outside the scope of what I’m presenting here.<\/em> If you have to ask you probably aren’t using it (is anyone anymore?) I mention it only because I got confused by it back when I was using MSVS 2008 more regularly and it led to a lot of confusion. If you are confused now just forget this little paragraph.<\/p>\n
There. All better!<\/p>\n
\n
Installing Pip<\/strong><\/p>\n
The easy way: Install the unofficial WIndows binary<\/a> that matches your version of Python!<\/p>\n
The harder but eminently fulfilling way (based on this comment<\/a>):<\/p>\n
Save http:\/\/python-distribute.org\/distribute_setup.py<\/a> as\u00a0C:\\Python27\\Scripts\\distribute_setup.py<\/em>\u00a0(you may need to create that Scripts directory).\u00a0Python ought to be in your path, as should this Scripts folder. Set your system or user path correctly or ensure it’s set during your command prompt sessions thus:<\/p>\n
set PATH=%PATH%;C:\\Python27;C:\\Python27\\Scripts<\/pre>\n
Run the script:<\/p>\n
distribute_setup.py<\/pre>\n
Now install pip:<\/p>\n
easy_install pip<\/pre>\n
\n
Installing Cython<\/strong><\/p>\n
The easy way:\u00a0Install the\u00a0unofficial WIndows binary<\/a>\u00a0that matches your version of Python!<\/p>\n
The harder but more rewarding way:<\/p>\n
THIS is where that Windows SDK you installed becomes important.<\/p>\n
First, run the setenv and other commands as you did above to test the SDK!<\/strong><\/span><\/p>\n
Then:<\/p>\n
pip install cython<\/pre>\n
It should “just work”, compiling and installing without failures.<\/p>\n
Remember: if you’re using 32-bit python you need to be using the 32-bit x86 settings! 64-bit? Use the 64-bit x64 settings! Got a raft of “unresolved external symbol” errors? You probably chose the wrong one, or forgot to choose one.<\/p>\n
I tried it on both 32- and 64-bit versions of Python 2.7.5 (selecting the proper target for each). It worked. I was pretty amazed. A beverage was forthcoming!<\/p>\n
\n
Finally, create and compile something with Cython<\/strong>, such as\u00a0the “helloworld” example from the Cython tutorial<\/a>.<\/p>\n
(Remember the “handy” build.cmd file I mentioned above? This would be a great time to use it!)<\/em><\/p>\n
Note that the compiled version ends in .pyd NOT .dll (typo in the docs)<\/p>\n
\n
More resources:<\/strong><\/p>\n
https:\/\/twitter.com\/raymondh<\/a> (Subscribe!)<\/p>\n
http:\/\/wiki.cython.org\/InstallingOnWindows<\/a><\/p>\n
http:\/\/stackoverflow.com\/questions\/6034390\/compiling-with-cython-and-mingw-produces-gcc-error-unrecognized-command-line-o<\/a><\/p>\n
http:\/\/stackoverflow.com\/questions\/4750806\/how-to-install-pip-on-windows<\/a><\/p>\n
http:\/\/stackoverflow.com\/questions\/17185389\/how-would-i-get-the-detailed-version-of-my-python-interpreter-via-a-one-liner<\/a>\u00a0(thanks, poolie!)<\/p>\n\n","protected":false},"excerpt":{"rendered":"
Note: this is directed at Python 2.7 developers, but the same principles should apply to Python 3.x as well. First, a shout-out for the\u00a0Unofficial Windows Binaries for Python site: as […]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[86],"tags":[102,32,101,100,117],"class_list":["entry","author-marty","has-more-link","post-97","post","type-post","status-publish","format-standard","category-software-and-hardware-development","tag-cython","tag-gcc","tag-msvs","tag-python","tag-windows"],"_links":{"self":[{"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/posts\/97","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/comments?post=97"}],"version-history":[{"count":0,"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/posts\/97\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/media?parent=97"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/categories?post=97"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.falatic.com\/index.php\/wp-json\/wp\/v2\/tags?post=97"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}