Last Updated:

Modules and packages in Python

Modules and packages in Python

 

One of the key features of Python is that the actual base language is quite small. This is an intentional design feature to maintain simplicity. Most of the powerful functionality is delivered through plug-ins and packages.

The main work of the installation so far has been to supplement the Python kernel with useful modules for scientific analysis.

Modules

A module is simply a file that contains Python definitions, functions, and statements. Putting code into modules is useful because of the ability to import module functionality into your script or IPython session, for example:

importastropy
importastropy.table
data = astropy.table.Table.read(‘my_table.fits’)

You'll see imports in almost every Python script, and soon it will become second nature to you.

Question:

Importing modules and putting the module name in front of them is such a problem, why would I do that?

Answer:

It keeps everything modular and separate. For example, many modules have a read() function, as this is a common thing. Without the use of <module> syntax <function>(...) there would be no way to know which one to invoke.

Packages

A package is simply a way to put related modules together within a single tree hierarchy. Very complex packages, such as NumPy or SciPy, have hundreds of separate modules, so placing them in a directory-like structure allows you to organize things and avoid name collisions. For example, here is a partial list of subpackages available in SciPy.

  • scipy.fftpack- Discrete Fourier transform algorithms
  • scipy.stats — statistical functions
  • scipy.lib Python - for external libraries
  • scipy.lib.blas — for blas library
  • scipy.lib.lapack — for LAPACK library
  • scipy.integrate — integration procedures
  • scipy.linalg — for a linear algebra procedure
  • scipy.sparse.linalg — for linear algebra
  • scipy.sparse.linalg.eigen – to resolve eigenvalues
  • scipy.sparse.linalg.eigen.arpack – to resolve eigen values using iterative methods.

Exercise: Import a package module and learn more about it

Import a linear algebra module from the SciPy package and see what functions it provides.

Find and install other packages

If you've come that far, you have a working Python scientific environment that has most of what you'll ever need. However, it's almost certain that you'll end up finding a need that won't be met within your current setup. Here we will learn where to find other useful packages and how to install them.

Package Resources

When you find a package on the Internet, pay attention to a few things:

  • Good up-to-date documentation with examples
  • Easy to install without a lot of dependencies (or has detailed installation instructions)
  • Actively developing

Resource Lists

There are a number of sites specifically dedicated to Python for astronomy, with organized lists of useful resources and packages.

  • Astropython.org resources
  • Comfort at 1 AU
  • AstronomicalPython

The Python package index is the main repository for third-party Python packages (about 14,000 packages and growing). More and more astronomy-related packages are available on PyPI, but there are many available options on this list.

The advantage of using PyPI is the ease of installation with pipinstall<package_name>.

Exercise: Finding packets for coordinate manipulation

Locate one or more Python packages that convert coordinates from Galactic to Ecliptic FK5.

Tip: Tags are useful when astropython.org and don't forget the "next" button at the bottom.

Installing the Package

There are two standard ways to install a package.

Installing pip

The pip installation script is available in our scientific Python installation and is very easy to use (when it works). During the installation process, you have already seen many examples of installing pip in action. Features include:

  • If it has a package name, it will query the PyPI site to find out about the package. Assuming the package exists, pipinstall will automatically download and install the package.
  • Will accept a local tar file (as long as it contains a Python package to be installed) or a URL that points to the tar file.
  • You can install in the custom package area via pipinstall<packageor URL> -user (but see discussion below)

pythonsetup.py install

Some packages may not be installed using pipinstall. More often than not, there will be some obvious (or not) compilation error message or a missing dependency. In this case, the likely next step is to download the installation tar file and extract it. Navigate to the package directory and locate files such as:

INSTALL
README
setup.py
setup.cfg

If there is an INSTALL file or README, then hopefully you will find useful installation instructions. Most well-established python packages install through a standard setup.py script. This is used as follows:

pythonsetup.py —help # get options
python setup.py install # install in python scope (root / adminreq'd)
pythonsetup.py install —user # install in user package scope

Where are the packages installed?

An important parameter during the installation process is where to place the package files. We saw the —user option in pipinstall and python setup.py install. - And what is it? In the section below, we'll document how this works. Cm. A discussion in A few pythons on your computer for the reason why you might want to do this, but first, please read this warning:

Warning

We strongly recommend that you do not install packages with -user unless you are an expert and really do not understand what you are doing. This is because the local custom version will always take precedence and thus could potentially break other Python settings and cause difficult-to-understand issues. Large analytics packages such as CIAO, STSci_Python or CASA are thoroughly tested as long as there is an integrated environment that they provide. If you start spoiling everything, then all bets will be removed.

C —user

Packages are installed in a local directory owned by the user when you do one of the following:

pipinstall —useraplpy
python setup.py installation —user

This places the packets in:

Mac ~/Library/Python/2.x/lib/python/site-packages
Linux ~/.local/lib/python-2.x/site-packages
Windows %APPDATA%/Python/Python2x/site-packages

Note

On a Mac, if you haven't used Anaconda or the PythonFramework EPD, you may see custom packages in ~/.local/lib, just like for linux. It depends on whether Python is installed as a macOS framework or not.

WITHOUT —user

If you are using Anaconda or a non-root Python installation, then there is no problem with permissions on any platform, as the entire installation is local in your own directory.

However, installing on a system-wide Python installation will require root or admin privileges. Installing in this way has the advantage of making the package available to all users of the Python installation, but has the disadvantage that it is more difficult to opt out of the changes if necessary. In general, we recommend using only a system package manager (such as yum) to install packages in system Python. This will ensure the integrity of your Python system, which is important even if you are the only user.

How do I find the package after installation?

Finding the file associated with a package or module is very simple, just use the help command in IPython:

importscipy
helpscipy

This gives something like:

scipy

FILE
/usr/local/lib/python2.6/site-packages/scipy/__init__.py

DESCRIPTION
SciPy: A scientific computing package for Python

================================================

Documentation is available in the docstrings and
online at http://docs.scipy.org.
...
Internet at http://docs.scipy.org.
...

Deleting Packages

There's no easy and completely consistent way to do this unless you're using solutions like Anaconda or Canopy. The Python community is working on this. However, in most simple cases, you can simply delete the module file or directory that is discovered using the method shown above.