Developing

A new support module must be placed in the appropriate package:

New functions must be predicates i.e. they must return true if the vulnerability is open or false if it is closed, and show the appropriate message from the main package:

from fluidasserts import show_close, show_open

def has_insecure_deserialization(java_dest: str) -> bool:
    if ...:
        show_open('Code uses insecure deserialization', details=...)
        return True
    else:
        show_close('Code uses secure deserialization', details=...)
        return False

This will sometimes lead to apparently awkward function names, like is_not_https_required but it makes sense considering we are not testing if https is _required_, but rather whether the vulnerability “not requiring httpsis open.

Function names must be written in english and conform to PEP 8 (in particular they must be written in snake_case) so that they can be used like this from a project exploit:

from fluidasserts.lang import javascript
javascript.uses_eval('my-script.js')

Other good function name ideas:

is_code_not_obfuscated
is_disk_not_encrypted
is_linter_bypassed
has_blind_xee
contains_commented_code

Notice that if the answer to any of these questions is true, then we have an open vulnerability.

Bad function names:

is_secure         # Too general
has_antimalware   # Logic inverted
ssh_version       # Not a verb

Comments, variables, parameter names, etc should also be written in English and conform to PEP 8 conventions.

All modules are checked with the linters Pylint, Pyflakes, Flake8, Pycodestyle, and the tools dodgy, mccabe, and pep8 (via Prospector, in highest strictness). It is recommended to install and use overcommit locally before committing.

Required tools

Besides the runtime dependencies (see …), you will need some dependencies which you can install with this command on a Debian-based OS:

$ sudo apt install python3-dev python3-pip python3.6-venv git cloc scons rubygems

And then some python packages via pip:

$ pip3 install invoke configobj tox sphinx mypy

Finally install the pre-commit hooks

$ gem install overcommit
$ overcommit --sign pre-commit

This last step should be done after cloning the repository since it needs the overcommit.yml configuration file in it.

You also need to install Docker-CE. Follow the steps in this guide.

Version control workflow

After receiving developer access and cloning the repository, setup your credentials if you haven’t done so already. In Gitlab (from the website) and your local git installation (with git config), your username must be loginatfluid v.g. dknuthatfluid and your email must be your corporate email, v.g. dknuth@fluidattacks.com. The name should be your real name, v.g. Donald Knuth.

The branching workflow is with topic branches but with one caveat: the name of the branch you work on must be your login. Following the example above, Don should name his branch dknuth.

The merge strategy is by fast-forwards only. When ready to make a merge request, ensure that your branch is ahead of master. This means that you must integrate the latest changes in the master branch before your own commits, i.e. you should rebase the master branch onto your own branch. Don can keep up to date easily using these commands after finishing his commits without ever leaving his branch:

$ git fetch
$ git rebase origin/master

If Don followed these steps, checking their effect with git log, he would see this:

[dknuth@tex asserts]$ git commit -m "My last commit"
[dknuth bc53277] My last commit
1 file changed, 44 insertions(+)
[dknuth@tex asserts]$ git log --pretty=oneline --abbrev-commit
a201834 (HEAD -> dknuth) My last commit
f3dec2a (origin/master) Feature: Add cool new feature
...
[dknuth@tex asserts]$ git fetch
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From gitlab.com:fluidsignal/asserts
20b4133..347d774  master     -> origin/master
+ f56e548...e11188e ltorvalds    -> origin/ltorvalds  (forced update)
+ f56e548...347d774 rstallman    -> origin/rstallman  (forced update)
[dknuth@tex asserts]$ git rebase origin/master
First, rewinding head to replay your work on top of it...
Applying: My last commit
[dknuth@tex asserts]$ git log --pretty=oneline --abbrev-commit
a201834 (HEAD -> dknuth) My last commit
347d774 (origin/rstallman, origin/master, origin/HEAD) Add emacs support
e11188e (origin/ltorvalds) Update to kernel 4.14
f3dec2a Add cool new feature
...
[dknuth@tex asserts]

Now Don is ready to make his merge request, that is, if his pipeline passes…

Continuous Integration

Asserts uses Gitlab CI to make sure that a change in a commit does not break anything in the master branch. Among other things, the CI pipeline:

  1. Builds environments for development and runtime

  2. Lints the entire codebase

  3. Runs the whole test suite

  4. Deploys the project for release

  5. Updates this documentation site

You can run this pipeline locally before pushing using the Nix shell and the local-integration.nix script in the repo.

As a developer, you should be specially concerned about:

  • Not pushing simple mistakes like trailing whitespace or typos.

    $ overcommit --run
    

    Can help avoid these.

  • Not pushing functional but ugly code by linter standards. Run

    $ scons lint
    

    You can use each linter individually as well.

  • Your code passing every test. Run

    $ scons test
    

Finally, keep your commits small and logically atomic, that is, there should be a one-to-one mapping between functional changes to the codebase and commits. If you’re adding a function in the HTTP module, don’t commit every line you add independently, but also don’t include your changes to another module in that same commit.

Documentation

Docstrings should conform to PEP 257 and will be checked by pydocstyle in CI time. Parameters and return specifications should be written in plain Sphinx ReStructured Text style, while parameter and return types must be specified using Type Hints according to PEP 484:

def is_long_line(line: str) -> bool
    """
    Determine if a ``line`` is *too* long.

    :param line: A line of code to test.
    :return: ``True`` if too long, ``False`` if **OK**.
    """

This is further enhanced by MyPy and the typing module which provides means to specify, in particular, optional and multiple return types. Type consistency will be checked by MyPy in CI time, but not strictly.

Whenever possible, docstrings should link to the appropriate entry in FLUIDRules and FLUIDDefends.

References

The following pages contain some ideas to keep in mind when developing Fluid Asserts: