Independent game developers - making Avoyd

Python Google App Engine debugging with PyCharm CE

Juliette Foucaut - 31 Dec 2014 - edited 30 Apr 2016


Alongside PyCharm Professional Edition, JetBrains generously offers a free version of their IDE: PyCharm Community Edition. We've found that PyCharm CE can, with some effort, be made to work with Google App Engine (GAE). Although lacking in some features, it is still useful if all you're after is a python debugger and IDE for GAE.

This post details how to configure the free PyCharm Community Edition to enable debugging of python code destined to run on Google App Engine. Further, we explain how to enable smart code navigation and completion for GAE libraries. The final section explains how to use a PyCharm path variable to make the code more portable, which should help for distributed projects. Throughout the tutorial we'll attempt to use the PyCharm UI as often as possible, but we'll also explain how to do the whole configuration by editing the PyCharm files directly. The specific paths examples are from Windows, but they should also be applicable to other Operating Systems.

Note: Doug figured out the configuration settings described in this tutorial through trial and error, our experience with python GAE debugging with Visual Studio, and by looking at the open-source PyCharm CE code. If you want to enjoy the full power of PyCharm and save yourself some time, skip this tutorial and use PyCharm Professional Edition instead.

Before starting, note that you'll find additional information at the bottom of the post:

Configure python GAE debugging

This section describes the minimum Run/Debug configuration necessary to debug a python for GAE application. It must be set on a per project basis. You’ll need a pre-existing GAE application for this to work.
If you don't have a GAE app to work with, see the prerequisites section for instructions on how to set up an example GAE application.

  1. Open PyCharm

  2. Open your GAE project directory:

    1. Select menu item File > Open…

    2. Select your base project directory (the one which contains an app.yaml file).
      For our examples we’ll be using the folder C:\engineapp, which we’ll refer to as <engineapp_path>.

  3. Add a new python configuration:

    1. Run > Edit Configurations…

    2. Click the green plus sign "+"

    3. In the Add New Configuration list, select Python.

    4. Specify the configuration parameters:

      1. Name:
        GAE_config
        Choose a name, which we’ll refer to as GAE_config

      2. Script:
        <GAE_install_path>\dev_appserver.py
        Here you need to enter the location of the dev_appserver.py file installed in your Google App Engine SDK. On Windows the default installation is C:\Program Files (x86)\Google\google_appengine\dev_appserver.py. We’ll refer to this location as <GAE_install_path>

      3. Script parameters:
        --automatic_restart=no --max_module_instances="default:1" .
        These are the minimum arguments needed for the debugging to work.
        Ensure you include the final argument, ".": it means the current path, i.e. the working directory in this case.

      4. Working directory:
        <engineapp_path>

      5. Tick the Share box.
        This creates configuration file <engineapp_path>\.idea unConfigurations\GAE_config.xml, which can be shared with other users and put in version control.

      6. Press OK

  4. PyCharm python run/debug configuration for Google App Engine

  5. Verify the python debugger settings:

    1. File > Settings…

    2. Expand the tree to Build, Execution, Deployment > Python Debugger

    3. Ensure the option Attach to subprocess automatically while debugging is ticked.

  6. To debug:

    1. Add a breakpoint in the python file (e.g. main.py)

    2. Run in debug mode: Run > Debug 'GAE_config'

PyCharm Google App Engine python breakpoint and step by step debugging

The basic PyCharm configuration for python GAE debugging is done. The following steps are optional.

Enable code navigation for GAE libraries

To enable PyCharm's code navigation and completion, we need to add the GAE SDK to PyCharm's list of External Libraries.

  1. Close PyCharm

  2. Add External Libraries directory and file:

    1. In <engineapp_path>\.idea, create a directory named
      libraries.

    2. In <engineapp_path>\.idea\libraries, create an xml file named
      GAE_SDK.xml

    3. In GAE_SDK.xml, copy and paste the following xml code:

    4. <component name="libraryTable">
        <library name="GAE_SDK" type="python">
          <CLASSES>
            <root url="file://C:/Program Files (x86)/Google/google_appengine" />
            <root url="file://C:/Program Files (x86)/Google/google_appengine/lib/django-1.5" />
            <root url="file://C:/Program Files (x86)/Google/google_appengine/lib/jinja2-2.6" />
            <root url="file://C:/Program Files (x86)/Google/google_appengine/lib/webapp2-2.5.2" />
          </CLASSES>
          <SOURCES />
        </library>
      </component>
    5. Update the paths C:/Program Files (x86)/Google/google_appengine to your <GAE_install_path>.
      If you'd like to use a per user path variable instead, see the next section.

    6. If you need more GAE libraries, add them to the list (in a similar way to webapp2, django and jinja in our example).

    7. Save.

  3. Update the project .iml file with the GAE SDK reference:

    1. Open <engineapp_path>\.idea\<engineapp>.iml

    2. Add the line
      <orderEntry type="library" name="GAE_SDK" level="project" />
      to element <component name="NewModuleRootManager">:

    3. <?xml version="1.0" encoding="UTF-8"?>
      <module type="PYTHON_MODULE" version="4">
        <component name="NewModuleRootManager">
          <content url="file://$MODULE_DIR$" />
          <orderEntry type="inheritedJdk" />
          <orderEntry type="sourceFolder" forTests="false" />
          <orderEntry type="library" name="GAE_SDK" level="project" />
        </component>
        <component name="TestRunnerService">
          <option name="PROJECT_TEST_RUNNER" value="Unittests" />
        </component>
      </module>
    4. Save.

  4. Open PyCharm.
    GAE_SDK is listed under External Libraries in the Project viewer (View > Tool Windows > Project). In the python code, you can open class definitions (right-click on class) and code completion works.

PyCharm python Google App Engine external libraries integration for code navigation and completion

The PyCharm configuration for python GAE code navigation and completion is done. The following steps are optional: replacing <GAE_install_path> with a per user path variable is mainly useful for large, shared projects.

Set a path variable for GAE

The configurations described above have to be repeated for each project. To be able to store the settings in files that can be shared across projects and users, it is useful to define a path variable for the library location. For the same result, there are two ways to replace <GAE_install_path> with the PyCharm path variable GAE_PATH: through the PyCharm UI, or by editing the PyCharm files directly.

Method A: use the PyCharm UI

  1. Edit python configuration GAE_config:

    1. Open PyCharm.

    2. Open project <engineapp>

    3. Run > Edit Configurations...

    4. Select GAE_config

    5. Replace <GAE_install_path> with the path variable $GAE_PATH$:

      1. Script:
        $GAE_PATH$\dev_appserver.py

    6. Save.

    7. Close PyCharm.

  2. Edit file GAE_SDK.xml:

    1. Open <engineapp_path>\.idea\libraries\GAE_SDK.xml

    2. replace all instances of <GAE_install_path> with $GAE_PATH$.

    3. <component name="libraryTable">
        <library name="GAE_SDK" type="python">
          <CLASSES>
            <root url="file://$GAE_PATH$" />
            <root url="file://$GAE_PATH$/lib/django-1.5" />
            <root url="file://$GAE_PATH$/lib/jinja2-2.6" />
            <root url="file://$GAE_PATH$/lib/webapp2-2.5.2" />
          </CLASSES>
          <SOURCES />
        </library>
      </component>
    4. Save.

  3. Define the value of path variable $GAE_PATH$:

    1. Open PyCharm.

    2. Open project <engineapp>
      Note: at this point GAE_SDK is not listed under External Libraries.

    3. Event Log, a Load error: undefined path variables message about undefined variable GAE_PATH is displayed. Click the Fix it link.

    4. In the Configure Path Variables window, set the value of GAE_PATH to <GAE_install_path> and save.
      Note: This step creates a user file, path.macros.xml, that contains the Path Variable GAE_PATH's value. We haven't found out how to open the Configure Path Variables window other than by triggering the Load error. In addition, opening the window by clicking Fix it only works on the first attempt. Workaround: restart PyCharm.

    5. Close PyCharm.

  4. PyCharm's 'Load error: undefined path variables' error message and path variable configuration window

  5. Open PyCharm project <engineapp>.
    GAE_SDK is listed under External Libraries in the Project viewer.

Method B: edit the PyCharm files

  1. Edit file GAE_config.xml:

    1. Close PyCharm.

    2. Open <engineapp_path>\.idea unConfigurations\GAE_config.xml

    3. Replace <GAE_install_path> with $GAE_PATH$.
      Make sure to include the $ signs on wither side of the variable name as this denotes that it is a PyCharm path variable.

    4.     <option name="SCRIPT_NAME" value="$GAE_PATH$/dev_appserver.py" />
    5. Save.

  2. Edit file GAE_SDK.xml:

    1. Open <engineapp_path>\.idea\libraries\GAE_SDK.xml

    2. Replace all instances of <GAE_install_path> with $GAE_PATH$.

    3. <component name="libraryTable">
        <library name="GAE_SDK" type="python">
          <CLASSES>
            <root url="file://$GAE_PATH$" />
            <root url="file://$GAE_PATH$/lib/django-1.5" />
            <root url="file://$GAE_PATH$/lib/jinja2-2.6" />
            <root url="file://$GAE_PATH$/lib/webapp2-2.5.2" />
          </CLASSES>
          <SOURCES />
        </library>
      </component>
    4. Save.

  3. Add user file path.macros.xml defining the value of path variable GAE_PATH:

    1. In directory C:\Users\<UserName>\.PyCharm<VersionNumber>\config\options\, create an xml file named
      path.macros.xml

    2. In path.macros.xml, copy and paste the following xml code:

    3. <application>
        <component name="PathMacrosImpl">
          <macro name="GAE_PATH" value="<GAE_install_path>" />
        </component>
      </application>
    4. Replace <GAE_install_path> with the actual GAE install path.
      e.g. C:\Program Files (x86)\Google\google_appengine

    5. Save.

  4. Open PyCharm project <engineapp>.
    GAE_SDK is listed under External Libraries in the Project viewer.



Additional information

Overview of Python for GAE configuration files

To summarise, we need four files to record the debug settings, external libraries mapping and path value.

Configuration files on GitHub

Variables Definitions

  • <GAE_install_path>
    Default GAE installation path where dev_appserver.py and lib, the Google libraries directory, are located.
    In this tutorial: using Windows,
    <GAE_install_path> = C:\Program Files (x86)\Google\google_appengine

  • <engineapp>
    Name of the GAE python application and the PyCharm Project.
    In this tutorial:
    <engineapp> = engineapp

  • <engineapp_path>
    Location of <engineapp>.
    In this tutorial:
    <engineapp_path> = C:\engineapp

  • GAE_config
    Name of the PyCharm project's Run/Debug Configuration used to specify dev_appserver.py as the configuration to run when executing python code.
    In this tutorial, GAE_config is used as is.

  • GAE_SDK
    Name of the GAE libraries in PyCharm's list of external libraries. If you choose a different name, ensure you update the file name and the xml code accordingly.
    In this tutorial, GAE_SDK is used as is.

  • GAE_PATH
    PyCharm Path Variable used to replace <GAE_install_path>
    In this tutorial, GAE_PATH is used as is.
    You'll encounter the notation $GAE_PATH$. The $ signs tell PyCharm it is a variable.

Prerequisites

  • Download and install

  • If you don't have an existing GAE application, create a new app:

    • Run the Google App Engine Launcher GoogleAppEngineLauncher.exe

    • Select File > Create New Application…

    • In the Add New Application window, set:

      • Application Name: engineapp

      • Parent Directory: C:/

    • Select Create

    Default application 'engineapp' in the Google App Engine Launcher

Style convention

  • Emphasized text
    Is for text you'll find in tools UI's, menus, labels, paths etc. e.g. File > Open....

  • Courier New text
    is for values to be entered as specified.

  • <Text in angle brackets>
    is for values that are provided as an example and may need to be replaced by your own (e.g. absolute paths).

  • Small text in italics
    is for comments, extra information or tips.



[Edit 30 Apr 2016 - updated for PyCharm CE 2016.1, rewritten for clarification, added new screenshots, added configuration files on GitHub. Thanks to Олег and David Manns for your feedback.]


comments powered by Disqus
 › 2017
 › Speeding up Runtime Compiled C++ compile times in MSVC with d2cgsummary
 › Multiplayers toxic last hit kill and how to heal it
 › Avoyd Editor Prototype
 › 2016
 › Black triangles and Peter Highspot
 › Colour palettes and lighting
 › Concept art by Rebecca Michalak
 › 2015
 › Internals of a lightweight task scheduler
 › Implementing a lightweight task scheduler
 › Feral Vector
 › Normal generation in the pixel shader
 › 2014
 ›› Python Google App Engine debugging with PyCharm CE 
 › Lighting voxel octrees and procedural texturing
 › Patterns and spheres
 › Python Google App Engine debugging with PyTools
 › Interview
 › Domain masking using Google App Engine
 › Octree streaming - part 4
 › Black triangles and nervous_testpilot
 › Presskit for Google App Engine
 › Octree streaming - part 3
 › Octree streaming - part 2
 › Octree streaming
 › 2013
 › LAN discovery with multiple adapters
 › Playing with material worlds
 › Developer Diary archive
 › Website redesign
 › First Person Editor
 › First Avoyd tech update video
 › Implementing a static website in Google App Engine
 › Multiplayer editing
 › First screenshots
 › Thoughts on gameplay modes
 › Back in 1999
 › 2002
 › ECTS 2002
 › Avoyd Version 1.6.1 out
 › Avoyd Version 1.6 out
 › 2001
 › Biting the bullet
 › Avoyd version 1.5 out
 › Monday Mayhem
 › Avoyd version 1.5 alpha 1 out
 › Avoyd version 1.4 out
 › ECTS 2001
 › Fun with Greek letters
 › Closer just a little closer
 › Back already
 › Artificial Humanity
 › Products and promises
 › Ecommerce
 › Explosions galore
 › Spring fixes
 › Open source and ports to other operating systems
 › Avoyd LAN Demo Version 1.1 is out
 › Thanks for the support
 › Avoyd LAN Demo Ready
 › Game Tech
 › Speeding up Runtime Compiled C++ compile times in MSVC with d2cgsummary
 › Internals of a lightweight task scheduler
 › Implementing a lightweight task scheduler
 › Normal generation in the pixel shader
 › Lighting voxel octrees and procedural texturing
 › Octree streaming - part 4
 › Octree streaming - part 3
 › Octree streaming - part 2
 › Octree streaming
 › LAN discovery with multiple adapters
 › enkiTS
 › Internals of a lightweight task scheduler
 › Implementing a lightweight task scheduler
 › RCC++
 › Speeding up Runtime Compiled C++ compile times in MSVC with d2cgsummary
 › Web Tech
 ›› Python Google App Engine debugging with PyCharm CE 
 › Python Google App Engine debugging with PyTools
 › Domain masking using Google App Engine
 › Presskit for Google App Engine
 › Implementing a static website in Google App Engine
 › Avoyd
 › Multiplayers toxic last hit kill and how to heal it
 › Avoyd Editor Prototype
 › Black triangles and Peter Highspot
 › Colour palettes and lighting
 › Concept art by Rebecca Michalak
 › Feral Vector
 › Patterns and spheres
 › Interview
 › Black triangles and nervous_testpilot
 › Playing with material worlds
 › Website redesign
 › First Person Editor
 › First Avoyd tech update video
 › Multiplayer editing
 › First screenshots
 › Thoughts on gameplay modes
 › Back in 1999
 › Avoyd 1999
 › Developer Diary archive
 › Back in 1999
 › ECTS 2002
 › Avoyd Version 1.6.1 out
 › Avoyd Version 1.6 out
 › Biting the bullet
 › Avoyd version 1.5 out
 › Monday Mayhem
 › Avoyd version 1.5 alpha 1 out
 › Avoyd version 1.4 out
 › ECTS 2001
 › Fun with Greek letters
 › Closer just a little closer
 › Back already
 › Artificial Humanity
 › Products and promises
 › Ecommerce
 › Explosions galore
 › Spring fixes
 › Open source and ports to other operating systems
 › Avoyd LAN Demo Version 1.1 is out
 › Thanks for the support
 › Avoyd LAN Demo Ready