Parse and Conquer

Why Asserts uses Parser combinators

Blog Parse and Conquer

| 6 min read

Table of contents

Contact us

As you might have noticed at Fluid Attacks we like parser combinators, functional programming, and of course, Python. In the parser article, we showed you the essentials of Pyparsing and we also showed how to leverage its power to find SQL injections in a PHP application. Here we will extend those essentials to show you how we used parser combinators in Asserts, our vulnerability closure checker engine (considering that we not longer support this product). Feel free to refer to that article for more details on how Pyparsing works, though I’ll try my best to explain every keyword used here.

Parser combinators are particularly useful analyzing complex expressions. You don’t really need parser combinators to break up email addresses into usernames and domains. For that, a regular expression will suffice. However, when what you need to do is more involved, such as looking for SQL injections or analyzing source code for poor programming practices per our Criteria and recommendations, then parsers are our tool of choice. This is one of the tasks at which Asserts excels. Determine whether a vulnerability that has been found in the source code by one of our analysts is still open by doing a deep search within it with the aid of parser combinators. Let’s see how that works.

Suppose an analyst was auditing some Java source code and found out that it uses the insecure DES cipher to mask the information of a bank transaction in the file DES is insecure due to its small 56-bit key size which could theoretically be brute-forced in 6 minutes. In order to report the vulnerability, they could write a script that automatically checks if the vulnerability is still there.

Asserts script to check DES usage.

from fluidasserts.lang import java

FILE = ''

Simple, right? Just running that script tells you whether or not the insecure DES algorithm is used in that particular file. Or you can even point it at an entire directory and Asserts will test every Java source file for DES usage. But what’s behind the curtain? Since Asserts is now open-source, anyone can actually check out what this function does. See Gitlab for rest of code.

from pyparsing import (CaselessKeyword, Word, Literal, Optional, alphas, Or, alphanums, Suppress, nestedExpr, javaStyleComment, SkipTo, QuotedString, oneOf) from fluidasserts.helper import lang ... def uses_des_algorithm(java_dest: str, exclude: list = None) -> bool:
    Check if code uses DES as encryption algorithm.

    See ``_.

    :param java_dest: Path to a Java source file or package.
    method = 'Cipher.getInstance("DES")'
    tk_mess_dig = CaselessKeyword('cipher')
    tk_get_inst = CaselessKeyword('getinstance')
    tk_alg = Literal('"') ` CaselessKeyword('des') ` Literal('"')
    tk_params = Literal('(') ` tk_alg ` Literal(')')
    instance_des = tk_mess_dig ` Literal('.') ` tk_get_inst + tk_params

    result = False
        matches = lang.check_grammar(instance_des, java_dest, LANGUAGE_SPECS,
        if not matches:
            show_unknown('Not files matched',
            return False
    except FileNotFoundError:
        show_unknown('File does not exist', details=dict(code_dest=java_dest))
        return False
    for code_file, vulns in matches.items():
        if vulns:
            show_open('Code uses {} method'.format(method),
                                   lines=", ".join([str(x) for x in vulns]),
            result = True
            show_close('Code does not use {} method'.format(method),
    return result

Notice how, in the first few lines (17-22)above, the parser instance_des is built from smaller parsers such as tk_mess_dig, which matches the single keyword "Cipher", but also any variations in case, should they happen. CaselessKeyword. Literal does not make any such assumption: if its double quotes, they must be there. Pyparsing also takes care of handling white space by overloading the + operator to mean "followed possibly with some whitespace in between". Building a regex to match the same thing would be perhaps more compact, but never as readable or maintainable. This is one of the many advantages of parser combinators over regular expressions.

Get started with Fluid Attacks' Secure Code Review solution right now

Next, this parser is passed along with the other required parameters to the function check_grammar in the lang module (more on that later). This function should return the matches in said file for the built parser. Thus the actual matching code can be reused. If there are matches, that means the vulnerability is open, hence Asserts will show_open a message like this:

"Asserts open vulnerability message"

Asserts open vulnerability message.

Otherwise it shows a similar message, only with fewer alarming colors.

So, what does the lang module do with the combined parsers and the file? More parsing. It tests whether the path is a single file or a directory whether or not it has the correct extension, but really all the most important stuff is in the get_match_lines method. It parses the text in the given source code file to find out if those lines are comments, i.e., not functional code, so as to skip them. In the future, Asserts will be able to use this comment-code discrimination to find lines of code which were commented out and later abandoned. This would be important because these commented-out lines of code might exhibit unpredictable behavior in the application if they were carelessly uncommented, depending on who has access to the code.

From fluidasserts.helper.lang module. See full code in Gitlab.

def _get_match_lines(grammar: ParserElement, code_file: str, # noqa lang_spec: dict) -> List:  # noqa
    Check grammar in file.

    :param grammar: Pyparsing grammar against which file will be checked.
    :param code_file: Source code file to check.
    :param lang_spec: Contains language-specific syntax elements, such as
                       acceptable file extensions and comment delimiters.
    :return: List of lines that contain grammar matches.
    with open(code_file, encoding='latin-1') as file_fd:
        affected_lines = []
        counter = 0
        in_block_comment = False
        for line in file_fd.readlines():
            counter += 1
                if lang_spec.get('line_comment'):
                    parser = ~Or(lang_spec.get('line_comment'))
            except ParseException:
            if lang_spec.get('block_comment_start'):
                    block_start = Literal(lang_spec.get('block_comment_start'))
                    parser = SkipTo(block_start) + block_start
                    in_block_comment = True
                except (ParseException, IndexError):

                if in_block_comment and lang_spec.get('block_comment_end'):
                        block_end = Literal(lang_spec.get('block_comment_end'))
                        parser = SkipTo(block_end) + block_end
                        in_block_comment = False
                    except ParseException:
                    except IndexError:
                results = grammar.searchString(line, maxMatches=1)
                if not _is_empty_result(results):
            except ParseException:
    return affected_lines

After testing if the code we’re looking at is functional or not, it is simply a matter of invoking the searchString method from PyParsing, which as its name implies, searches the given string for matches of the given parser. The module has a few more tricks up its sleeve, such as turning the parsing search results into pretty strings and parsing chunks of lines of code. All that again with the help of parser combinators.

The most important takeaway from looking at this single function’s source code, and what lies behind it, is that using parser combinators in Asserts allows us not only to have readable, maintainable code for our own use and the use of others but also for this code to be easily extensible and reusable. Due to its object-oriented interface, clear naming conventions, and that coding parsers in it are just pythonic, PyParsing allows our team to write and rewrite static code analysis tools that will change along with its users' needs.

That wouldn’t be possible with regular expressions. Regexes must be tailor-made, carefully designed with one specific objective in mind. One application. So that regex that might search for conditionals without default actions in Javascript, will be useless for the same purpose in a different language. Such is not the case with parser combinators as most code is easily modified or reusable. Also, nesting searches as we did above (parsing before parsing to know if we’re inside a block comment) will definitely require uber-complex regular expressions, if it is possible at all.

Just like uses_des_algorithm above, Asserts packs convenient functions to test for many of our requirements or recommendations for secure coding, for several different languages, and growing daily. Pyparsing enhances a significant part of our static code analysis tools in a way that, as mentioned earlier, with regexes would only be ad hoc or impossible to maintain.


  1. Asserts documentation.

  2. McGuire, Paul (2008). 'Getting Started with Pyparsing'. O’Reilly Short Cuts.

Table of contents


Subscribe to our blog

Sign up for Fluid Attacks' weekly newsletter.

Recommended blog posts

You might be interested in the following related posts.

Photo by Jasmin Egger on Unsplash

If the essential security layer is flawed, you're toast

Photo by Christian Wiediger on Unsplash

The need to enhance security within the fintech sector

Photo by Claudio Schwarz on Unsplash

Is your financial service as secure as you think?

Photo by mitchell kavan on Unsplash

Bringing the zero trust model to life

Photo by Brian Kelly on Unsplash

We need you, but we can't give you any money

Photo by Sean Pollock on Unsplash

Data breaches that left their mark on time

Photo by Roy Muz on Unsplash

Lessons learned from black swans

Start your 21-day free trial

Discover the benefits of our Continuous Hacking solution, which hundreds of organizations are already enjoying.

Start your 21-day free trial
Fluid Logo Footer

Hacking software for over 20 years

Fluid Attacks tests applications and other systems, covering all software development stages. Our team assists clients in quickly identifying and managing vulnerabilities to reduce the risk of incidents and deploy secure technology.

Copyright © 0 Fluid Attacks. We hack your software. All rights reserved.