Python style guidelines [go/cros-pystyle]

Introduction

You must follow this guide for all Python source code that is created as part of the ChromiumOS project i.e. src/platform/*. In doing so, you make it easier for others to work with your code.

This guide may be ignored if you are working on code that is part of an open source project that already conforms to a different style i.e. generally Python code in src/third_party. Autotest is a great example of this - see the other notable Python style guides below.

See also the overall Chromium coding style.

Official Style Guide

The Google Python Style guide is the official Python style guide for ChromiumOS original code.

New projects should use that unmodified public Python Google style guide (4 space indent with methods_with_underscores). Historically, we adopted a style that was congruent with Google internal Python style guide (2 spaces with MethodsAsCamelCase). Individual older projects are in the process of migrating to the public style as time allows; there is no mandate to migrate.

That is, for new projects, use:

Indentation: 4 spaces
Naming: Functions and Method names use lower_with_under()

If that guide is silent on an issue, then you should fall back to PEP-8.

PEP-257 is referenced by PEP-8. If it isn't covered in the Comments section, you can also fall back to PEP-257.

Other notable Python style guides

There are a number of other Python style guidelines out there.

The CODING STYLE file for autotest - When working on autotest code (or any autotest tests), you should ignore the ChromiumOS style guidelines (AKA this page) and use the autotest ones.

Auto-formatting Python

An easy way to get people to agree on formatting, and to reduce the effort required around manual formatting, is to use an auto-formatter. The preferred auto-formatter for Python in ChromiumOS is Black.

To format your code with Black in your project, run:

cros format .

Or for files that cros format doesn't detect as python:

black --config ~/chromiumos/chromite/pyproject.toml file_without_py_suffix

Then, to enforce Black in pre-upload, add to PRESUBMIT.cfg:

[Hook Overrides]
black_check: true

For compatibility with pylint, you'll want to disable the bad-continuation, bad-indentation, invalid-string-quote, invalid-docstring-quote, and invalid-triple-quote options. Don't worry: you'll still be checking these for formatting consistency with Black.

Describing arguments in docstrings

PEP-257 says that "The docstring for a function or method should summarize its behavior and document its arguments, return value(s), side effects, exceptions raised, and restrictions. However, it doesn't specify any details of what this should look like.

Fortunately, the Google Python style guide provides a reasonable syntax for arguments and return values. We will use that as our standard.

The content of each subsection (e.g. Args, Returns, etc...) should use the same indentation level as the file. If the file is using 2 spaces, then the subsection content should use 2 space indents (as seen in the example below). If the file is using 4 space indents, then the subsection content should use 4 space indents as well. In both situations, use 4 space hanging indentation for long argument descriptions (as seen with keys: below).

All docstring content should use full sentences (proper capitalization & periods), including argument descriptions (as seen below). This applies even when the docstring is a one-liner.

def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):
    """Fetches rows from a Bigtable.

    Retrieves rows pertaining to the given keys from the Table instance
    represented by big_table.  Silly things may happen if
    other_silly_variable is not None.

    Args:
        big_table: An open Bigtable Table instance.
        keys: A sequence of strings representing the key of each table row
            to fetch.
        other_silly_variable: Another optional variable, that has a much
            longer name than the other args, and which does nothing.

    Returns:
        A dict mapping keys to the corresponding table row data
        fetched. Each row is represented as a tuple of strings. For
        example:

        {
            "Serak": ("Rigel VII", "Preparer"),
            "Zim": ("Irk", "Invader"),
            "Lrrr": ("Omicron Persei 8", "Emperor"),
        }

        If a key from the keys argument is missing from the dictionary,
        then that row was not found in the table.

    Raises:
        IOError: An error occurred accessing the bigtable.Table object.
    """
    pass

Specifically:

Required of all public methods to follow this convention.
Encouraged for all private methods to follow this convention. May use a one-line sentence describing the function and return value if it's simple.
All arguments should be described in the Args: section, using the format above.
- Ordering of descriptions should match order of parameters in function.
- For two-line descriptions, indent the 2nd line 4 spaces.
- If the function doesn't take any arguments, then the Args: section may be omitted entirely.
For functions, if present, Examples:, Args:, Returns: (or Yields: with generators), and Raises: should be in that order and come last in the docstring, each separated by a blank line.
For classes, only Examples: and Attributes: are permitted. When the the __init__ method arguments line up with the class's attributes (e.g., the function does a lot of self.foo = foo), there is no need to duplicate argument documentation in the __init__ docstring. Putting information in the class's attributes is preferred.

Imports

The following rules apply to imports (in addition to rules already talked about in PEP-8 and the Google Python Style guide):

Relative imports are forbidden (PEP-8 only "highly discourages" them). Where absolutely needed, the from __future__ import absolute_import syntax should be used (see PEP-328).
Do not use import *. Always be explicit about what you are importing. Using import * is useful for interactive Python but shouldn't be used in checked-in code.
While import statements are only for packages and modules, we also except the Path object from pathlib in addition to the other exceptions already mentioned in the Google Python Style guide.

Examples:

# OK, this imports a module.
from chromite.lib import cros_build_lib

# OK, there's an exception for pathlib.Path.
from pathlib import Path

# Bad, there's no exception for subprocess.run.
from subprocess import run

The shebang line

All files that are meant to be executed should start with the line:

#!/usr/bin/env python3

If your system doesn't support that, it is sufficiently unusual for us to not worry about compatibility with it. Most likely it will break in other fun ways.

Files that are not meant to be executed should not contain a shebang line.

String formatting

It is acceptable to mix f-strings & non-f-string formatting in a codebase, module, function, or any other grouping. Do not worry about having to pick only one style.

f-strings

When writing Python 3.6+ code, prefer f-strings. They aren't always the best fit which is why we say you should prefer rather than always use them. See the next section for non-f-string formatting.

Writing readable f-strings is important, so try to avoid packing too much logic in the middle of the {...} sections. If you need a list comprehension or complicated computation, then it's usually better to pull it out into a temporary variable instead. Dynamic code in the middle of static strings can be easy to miss for readers.

% formatting

The Google style guide says that f-strings, % formatting, and "{}".format() styles are all permitted. In CrOS, we prefer f-strings foremost, and then % formatting over "{}".format(), so you should too to match. There is no real material difference between the latter two styles, so sticking with % for consistency is better.

x = "name: %s; score: %d" % (name, n)
x = "name: %(name)s; score: %(score)d" % {"name": name, "score": n}

Logging formatting

Keep in mind that for logging type functions, we don't format in-place. Instead, we pass them as args to the function.

logging.info("name: %s; score: %d", name, n)

Variables in Comprehensions & Generator Expressions

When using comprehensions & generators, it's best if you stick to "throw away" variable names. "x" tends to be the most common iterator.

some_var = [x for x in some_list if x]

While helping with readability (since these expressions are supposed to be kept "simple"), it also helps avoid the problem of variable scopes not being as tight as people expect.

# This throws an exception because the variable is undefined.
print(x)

some_var = [x for x in (1, 2)]
# This displays "1" because |x| is now defined.
print(x)

string = "some string"
some_var = [string for string in ("a", "b", "c")]
# This displays "c" because the earlier |string| value has been clobbered.
print(string)

TODOs

It is OK to leave TODOs in code. Why? Encouraging people to at least document parts of code that need to be thought out more (or that are confusing) is better than leaving this code undocumented.

pylint

Python code written for ChromiumOS should be "pylint clean".

Pylint is a utility that analyzes Python code to look for bugs and for style violations. We run it in ChromiumOS because:

It can catch bugs at "compile time" that otherwise wouldn't show up until runtime. This can include typos and also places where you forgot to import the required module.
It helps to ensure that everyone is following the style guidelines.

Pylint is notoriously picky and sometimes warns about things that are really OK. Because of this, we:

Disable certain warnings in the ChromiumOS pylintrc.
Don't consider it a problem to locally disable warnings parts of code. NOTE: There should be a high bar for disabling warnings. Specifically, you should first think about whether there's a way to rewrite the code to avoid the warning before disabling it.

You can use cros lint so that we can be sure everyone is on the same version. This should work outside and inside the chroot.

Python 2 Compatibility

There is no need to keep code compatible with Python 2. That means all new files should omit the boiler plate lines:

# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import print_function

Other Considerations

Unit tests

All Python modules should have a corresponding unit test module called <original name>_unittest.py in the same directory. We use a few related modules (they're available in the chroot as well):

mock (import mock) for testing mocks
- It has become the official Python mock library starting in 3.3
- pymox (import mox) is the old mock framework
You'll still see code using this as not all have been migrated to mock yet
unittest (import unittest) for unit test suites and methods for testing

Main Modules

All main modules require a main method. Use this boilerplate at the end:
```
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
```
- The definition of main itself should look like:
```
def main(argv: Optional[List[str]] = None) -> Optional[int]:
    ...
    # A standard argparse.ArgumentParser parser.
    opts = parser.parse_args(argv)
```
  This allows re-use of Python's console_scripts wrappers.
- Having main accept the arguments instead of going through the implicit global sys.argv state allows for better reuse & unittesting: other modules are able to import your module and then invoke it with main(["foo", "bar"]).
- The sys.exit allows main to return values which makes it easy to unittest, and easier to reason about behavior in general: you can always use return and not worry if you're in a func that expects a sys.exit call. If your func doesn't explicitly return, then you still get the default behavior of sys.exit(0).
- If you want a stable string for prefixing output, you shouldn't rely on sys.argv[0] at all as the caller can set that to whatever it wants (it is never guaranteed to be the basename of your script). You should set a constant in your module, or you should base it off the filename.
```
# A constant string that is guaranteed to never change.
_NAME = "foo"
# The filename which should be stable (respects symlink names).
_NAME = os.path.basename(__file__)
# The filename which derefs symlinks.
_NAME = os.path.basename(os.path.realpath(__file__))
```
- If your main func wants argv[0], you should use sys.argv[0] directly. If you're trying to print an error message, then usually you want to use argparse (see below) and the parser.error() helper instead.
Main modules should have a symlink without an extension that point to the .py file. All actual code should be in .py files.

Other

There should be one blank line between conditional blocks.
Do not use GFlags, getopt, or optparse anymore -- stick to argparse everywhere.