CodeQL Python Extractor 源码分析

最近定位python codeql 的一个cve 复现，发现涉及到extractor 的逻辑，正好之前也有跟踪过java 的经验，应该大差不差，记录下。

1. 架构

upload successful

https://github.com/github/codeql/blob/main/python/extractor/README.md#2-The-actual-Python-extractor

从流程图上看

2. Trap 生成

2.1 codeql database create

新建一个Java 项目任务，配置Debug

upload successful

中断ProcessBuilder 发现有两处

  • autobuild.sh
  • pre-finalize.sh

2.2 autobuild.sh

upload successful

Debug 断点中发现，部分重要env 如下：

1
2
3
4
5
6
7
8
9
10
"CODEQL_EXTRACTOR_PYTHON_SOURCE_ARCHIVE_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/src"
"CODEQL_JAVA_HOME" -> "/Users/m0d9/study/codeql-home/v2.17.2/codeql/tools/osx64/java-aarch64"
"CODEQL_EXTRACTOR_PYTHON_SCRATCH_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working"
"CODEQL_EXTRACTOR_PYTHON_TRAP_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/trap/python"
"CODEQL_EXTRACTOR_PYTHON_DIAGNOSTIC_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/diagnostic/extractors/python"
"CODEQL_EXTRACTOR_PYTHON_WIP_DATABASE" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb"
"CODEQL_EXTRACTOR_PYTHON_LOG_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/log"
"CODEQL_SCRATCH_DIR" -> "/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working"
"CODEQL_DIST" -> "/Users/m0d9/study/codeql-home/v2.17.2/codeql"
"CODEQL_EXTRACTOR_PYTHON_ROOT" -> "/Users/m0d9/study/codeql-home/v2.17.2/codeql/python"

其中autobuild 具体代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/sh

set -eu

# Legacy environment variables for the autobuild infrastructure.
LGTM_SRC="$(pwd)"
LGTM_WORKSPACE="$CODEQL_EXTRACTOR_PYTHON_SCRATCH_DIR"
export LGTM_SRC
export LGTM_WORKSPACE

if which python3 >/dev/null; then
    exec python3 "$CODEQL_EXTRACTOR_PYTHON_ROOT/tools/index.py"
elif which python >/dev/null; then
    exec python "$CODEQL_EXTRACTOR_PYTHON_ROOT/tools/index.py"
else
    echo "ERROR: Could not find a valid Python distribution. It should be available when running 'which python' or 'which python3' in your shell. Python 2 is no longer supported."
    exit 1
fi

实际作用就是调用index.py

2.3 index.py

为了调试index.py，需要新建Python 项目，把codeql python extractor 目录拷贝至此，配置Debug

upload successful

1
2
3
4
5
6
7
8
9
10
11
12
13
CODEQL_EXTRACTOR_PYTHON_SOURCE_ARCHIVE_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/src
CODEQL_JAVA_HOME=/Users/m0d9/study/codeql-home/v2.17.2/codeql/tools/osx64/java-aarch64
CODEQL_EXTRACTOR_PYTHON_SCRATCH_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working
CODEQL_EXTRACTOR_PYTHON_TRAP_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/trap/python
CODEQL_EXTRACTOR_PYTHON_DIAGNOSTIC_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/diagnostic/extractors/python
CODEQL_EXTRACTOR_PYTHON_WIP_DATABASE=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb
CODEQL_EXTRACTOR_PYTHON_LOG_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/log
CODEQL_SCRATCH_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working
CODEQL_DIST=/Users/m0d9/study/codeql-home/v2.17.2/codeql
CODEQL_EXTRACTOR_PYTHON_ROOT=/Users/m0d9/study/codeql-home/v2.17.2/codeql/python
CODEQL_EXTRACTOR_PYTHON_SCRATCH_DIR=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working
LGTM_SRC=/Users/m0d9/study/codeql-home/v2.17.2/codeql/python/tools
LGTM_WORKSPACE=/Users/m0d9/study/java/codeqlDebug/langflow2.cdb/working

index.py 代码如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import os
import sys

if sys.version_info < (3, 7):
    sys.exit("ERROR: Python 3.7 or later is required (currently running {}.{})".format(sys.version_info[0], sys.version_info[1]))

from python_tracer import getzipfilename

if 'SEMMLE_DIST' in os.environ:
    if 'CODEQL_EXTRACTOR_PYTHON_ROOT' not in os.environ:
        os.environ['CODEQL_EXTRACTOR_PYTHON_ROOT'] = os.environ['SEMMLE_DIST']
else:
    os.environ["SEMMLE_DIST"] = os.environ["CODEQL_EXTRACTOR_PYTHON_ROOT"]

tools = os.path.join(os.environ['SEMMLE_DIST'], "tools")
zippath = os.path.join(tools, getzipfilename())
sys.path = [ zippath ] + sys.path

import buildtools.index
buildtools.index.main()

index.py 中的逻辑逻辑比较简单，就是配置一些env 变量，然后调用buildtools

2.4 buildtools

buildtools.index.main 实现如下，逻辑如下

1
2
3
4
5
6
7
8
9
10
def main():
    version = discover.get_version()
    tracer = os.path.join(os.environ["SEMMLE_DIST"], "tools", "python_tracer.py")
    args = extractor_executable() + site_flag(3) + [tracer] + extractor_options(version)
    print("Calling " + " ".join(args))
    sys.stdout.flush()
    sys.stderr.flush()
    env = os.environ.copy()
    env["CODEQL_EXTRACTOR_PYTHON_ANALYSIS_VERSION"] = get_analysis_version(version)
    subprocess.check_call(args, env=env)

2.5 python_tracer.py

upload successful

Debug 发现

upload successful

2.6 extractor

python_tracer.py

1
2
3
4
5
6
7
8
9
10
def main(sys_path = sys.path[:]):
	...
    run(options, args, the_traverser, logger)
    ...

def run(options, args, the_traverser, logger: logging.Logger):
	...
    try:
        pool = worker.ExtractorPool.from_options(options, trap_dir, archive, logger)
	...

worker.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class ExtractorPool(object):
	...
    def __init__(self, outdir, archive, proc_count, options, logger: Logger):
    	...
        self.procs = [
            ctx.Process(target=_extract_loop, args=(n+1,) + args + (n == 0,)) for n in range(proc_count)
        ]
        
    def from_options(options, trap_dir, archive, logger: Logger):
    	...
        return ExtractorPool(trap_dir, archive, procs, options, logger)

def _extract_loop(proc_id, queue, trap_dir, archive, options, reply_queue, logger: Logger, write_global_data):
	...
            extractor = SuperExtractor(options, trap_dir, archive, renamer, logger, diagnostics_writer)
                    imports = extractor.process(unit)

super_extractor.py

1
2
3
4
5
6
7
8
class SuperExtractor(object):
    def process(self, unit):
        for extractor in self.extractors:
            self.logger.debug("Trying %s on %s",extractor.name, unit)
            res = extractor.process(unit)
            if res is not NotImplemented:
                self.logger.debug("%s extracted by the %s.", unit, extractor.name)
                break

upload successful

2.7 extractors

Each extractor process runs a loop which extracts files or modules from the queue, one at a time. Each file or module description is passed, in turn, to one of the extractor objects which will either extract it or reject it for the next extractor object to try. Currently the default extractors are:

  • Builtin module extractor: Extracts built-in modules like sys.
  • Thrift extractor: Extracts Thrift IDL files.
  • Python extractor: Extracts Python source code files.
  • Package extractor: Extracts minimal information for package folders.
  • General file extractor: Any files rejected by the above passes are added to the database as a text blob.

Python extraction

The Python extractor is the most interesting of the processes mentioned above. The Python extractor takes a path to a Python file. It emits TRAP to the specified folder and a UTF-8 encoded version of the source to the source archive. It consists of the following passes:

  1. Ingestion and decoding: Read the contents of the file as bytes, determine its encoding, and decode it to text.
  2. Tokenizing: Tokenize the source text, including whitespace and comment tokens.
  3. Parsing: Create a concrete parse tree from the list of tokens.
  4. Rewriting: Rewrite the concrete parse tree to an AST, annotated with scope, variable information, and locations.
  5. Write out lexical and AST information as TRAP.
  6. Generate and emit TRAP for control-flow graphs. This is done one scope at a time to minimize memory consumption.
  7. Emit ancillary information, like TRAP for comments.

Template file extraction

Most Python template languages work by either translating the template into Python or by fairly closely mimicking the behavior of Python. This means that we can extract template files by converting them to the same AST used internally by the Python extractor and then passing that AST to the backend of the Python extractor to determine imports, and generate TRAP files including control-flow information.

主要是 py_extractor

2.8 tokenizer

upload successful

module.py

1
2
3
4
5
6
7
class PythonSourceModule(object):
	def tokens(self):
        if self._tokens is None:
            with timers["tokenize"]:
                tokenizer = semmle.python.parser.tokenizer.Tokenizer(self._source)
                self._tokens = list(tokenizer.tokens())
        return self._tokens

2.9 CST

虽然实现改功能的几个文件名为ast，但是实际上是CST

  • ast (标准库)
    生成抽象语法树 (Abstract Syntax Tree, AST)，关注代码的逻辑结构，忽略非逻辑元素（如空格、注释、括号位置）。
    用途：代码分析、优化、转换（如 Linter、静态检查）、生成字节码。
  • blib2to3 (Black 的分支)
    生成具体语法树 (Concrete Syntax Tree, CST) 或解析树，保留所有原始细节（空格、注释、格式）。
    用途：代码格式化（如 Black）、保留格式的源码转换（如 2to3 迁移工具）。

__init__.py

1
2
3
4
5
6
7
8
def parse(tokens, logger):
    """Given a string with source, return the lib2to3 Node."""
    for name, grammar in GRAMMARS:
        try:
            with timers["parse"]:
                cpt = parse_tokens(grammar, tokens)
            with timers["rewrite"]:
                return ast.convert(logger, cpt)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def parse_tokens(gr, tokens):
    """Parse a series of tokens and return the syntax tree."""
    p = Parser(gr, convert)
    p.setup()
    for tkn in tokens:
        type, value, start, end = tkn
        if type in (tokenize.COMMENT, tokenize.NL):
            continue
        if type == token.OP:
            type = grammar.opmap[value]
        if type == token.INDENT:
            value = ""
        if p.addtoken(type, value, (start, end)):
            break
    else:
        # We never broke out -- EOF is too soon (how can this happen???)
        raise parse.ParseError("incomplete input",
                               type, value, ("", start))
    return p.rootnode

整体上都用的blib2to3 这个库，为什么没用默认的AST 呢？

dump_ast

为了更好理解此处的ast 结构，codeql 也提供了dump_ast.py 脚本

1
python -m semmle.python.parser.dump_ast /Users/m0d9/study/python/codeql-python-extractor/data/python/stubs/six/moves/urllib_robotparser.py --old
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
Module
  body: [
    Import
      names: [
        alias
          value:
            ImportMember
              module:
                ImportExpr
                  level: 0
                  name: 'six'
                  top: False
              name: 'PY2'
          asname:
            Name
              variable: Variable('PY2', None)
              ctx: Store
        alias
          value:
            ImportMember
              module:
                ImportExpr
                  level: 0
                  name: 'six'
                  top: False
              name: 'PY3'
          asname:
            Name
              variable: Variable('PY3', None)
              ctx: Store
      ]
    If
      test:
        Name
          variable: Variable('PY2', None)
          ctx: Load
      body: [
        Import
          names: [
            alias
              value:
                ImportExpr
                  level: 0
                  name: 'robotparser'
                  top: False
              asname:
                Name
                  variable: Variable('_1', None)
                  ctx: Store
          ]
        Assign
          targets: [
            Name
              variable: Variable('RobotFileParser', None)
              ctx: Store
          ]
          value:
            Attribute
              value:
                Name
                  variable: Variable('_1', None)
                  ctx: Load
              attr: 'RobotFileParser'
              ctx: Load
        Delete
          targets: [
            Name
              variable: Variable('_1', None)
              ctx: Del
          ]
      ]
      orelse: None
    If
      test:
        Name
          variable: Variable('PY3', None)
          ctx: Load
      body: [
        Import
          names: [
            alias
              value:
                ImportExpr
                  level: 0
                  name: 'urllib.robotparser'
                  top: False
              asname:
                Name
                  variable: Variable('_1', None)
                  ctx: Store
          ]
        Assign
          targets: [
            Name
              variable: Variable('RobotFileParser', None)
              ctx: Store
          ]
          value:
            Attribute
              value:
                Name
                  variable: Variable('_1', None)
                  ctx: Load
              attr: 'RobotFileParser'
              ctx: Load
        Delete
          targets: [
            Name
              variable: Variable('_1', None)
              ctx: Del
          ]
      ]
      orelse: None
  ]

2.10 Converter

ast.py

1
2
3
4
5
def convert(logger, cpt):
    '''Covert concrete parse tree as specified by blib2to3/Grammar.txt
    to the AST specified by semmle/python/master.py
    '''
    return Convertor(logger).visit(cpt)

逻辑是将blib2to3 的AST 转成CodeQL 定义的AST 结构

1
2
3
4
5
6
7
8
9
10
11
class ParseTreeVisitor(object):
    '''Standard tree-walking visitor,
    using `node.name` rather than `type(node).__name__`
    '''

    def visit(self, node, extra_arg=None):
        method = 'visit_' + node.name
        if extra_arg is None:
            return getattr(self, method)(node)
        else:
            return getattr(self, method)(node, extra_arg)

具体实现是在visit_xxx 中

2.11 生成Trap

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def _extract_trap_file(self, ast, comments, path):
    writer = TrapWriter()
    file_tag = get_source_file_tag(self.src_archive.get_virtual_path(path))
    writer.write_tuple(u'py_Modules', 'g', ast.trap_name)
    writer.write_tuple(u'py_module_path', 'gg', ast.trap_name, file_tag)
    try:
        for ex in self.passes:
            with timers[ex.name]:
                if isinstance(ex, FlowPass):
                    ex.set_filename(path)
                ex.extract(ast, writer)
        with timers['lexical']:
            self.lexical.extract(ast, comments, writer)
        with timers['object']:
            self.object_pass.extract(ast, path, writer)
    except Exception as ex:
        self.logger.error("Exception extracting module %s: %s", path, ex)
        self.logger.traceback(WARN)
        return None
    return writer.get_compressed()

upload successful

  • 先写文件信息
  • 之后交由Passes.extract 处理

2.12 Passes

Passes 可以简单理解为遍历器

  • ASTPass: 遍历所有的AST 结构，保存为Trap
  • ExportsPass: __al__ 中的exprorts
  • FlowPass: 遍历CFG，保存为Trap，对应API::DataFlow 等之类的

2.12.1 ASTPass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class ASTPass(Pass):

	def extract(self, root, writer):
        try:
            self.writer = writer
            if root is None:
                return
            self._emit_variable(ast.Variable("__name__", root))
            self._emit_variable(ast.Variable("__package__", root))
            # Introduce special variable "$" for use by the points-to library.
            self._emit_variable(ast.Variable("$", root))
            writer.write_tuple(u'py_extracted_version', 'gs', root.trap_name, get_analysis_major_version())
            self._walk(root, None, 0, root, None)
        finally:
            self.writer = None

    def _walk(self, node, parent, index, scope, description):
        self._get_walker(node)(node, parent, index, scope, description)
        
    def _get_walker(self, node):
        if isinstance(node, list):
            return self._walk_list
        elif isinstance(node, ast.AstBase):
            return self._walk_node
        else:
            return self._emit_primitive
  • list
  • 基础节点，例如class、Function、Module、alias、arguments等
  • 原始类型，例如Variable

upload successful

2.12.2 ExportsPass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def exports_from_ast(node):
    'Get a list of symbols exported by the module from its ast.'
    #Look for assignments to __all__
    #If not available at top-level, then check if-statements,
    #but ignore try-except and loops
    assert type(node) is ast.Module
    exports = __all___from_stmt_list(node.body)
    if exports is not None:
        return exports
    # No explicit __all__ assignment so gather global assignments
    exports = set()
    globals_from_tree(node.body, exports)
    return [ ex for ex in exports if not is_private_symbol(ex) ]

class ExportsPass(Pass):
	...
    def extract(self, ast, writer):
        exported = exports_from_ast(ast)
        write_exports(ast, exported, writer)

upload successful

2.12.3 FlowPass

  • FlowNode: CFG 中的节点
  • FlowGraph: CFG 图
  • FlowScope: 整个CFG 图
FlowNode
1
2
3
4
5
6
7
8
9
10
11
12
13
14
class FlowNode(object):
    __slots__ = [ 'node' ]

    def __init__(self, node):
        self.node = node

    def __repr__(self):
        if hasattr(self.node, "lineno"):
            return 'FlowNode(%s at %d)' % (type(self.node), self.node.lineno)
        else:
            return 'FlowNode(%r)' % self.node

    def copy(self):
        return FlowNode(self.node)
FlowGraph

FlowGraph 有几个重点的field

  • pred: dict类型，当存在边x-y,那么则pred[y].add(x)
  • succ: 与pred 方向相反

为了弄清其含义，看看图最重要的两个接口，如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
   def add_node(self, n):
       'Add a node to the graph'
       if n not in self.succ:
           self.pred[n] = SmallSet()
           self.succ[n] = SmallSet()
           self.all_nodes.append(n)

def add_edge(self, x, y):
       '''Add an edge (x -> y) to the graph. Return true if x, y was
       previously in graph'''
       if x in self.succ:
           if y in self.succ[x]:
               return True
       else:
           self.add_node(x)
       self.add_node(y)
       self.pred[y].add(x)
       self.succ[x].add(y)
       return False

use 猜测是SSA 的产物

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def add_use(self, node, var):
    assert node in self.succ, node
    self.uses[node] = var

def _walk_name(self, node, predecessors, ctx_type = None):
    # Too many exception edges make analysis slower and adds almost no accuracy
    # Assume that Name may only raise an exception if global in scope and
    # not a store
    res = self.add_successor(predecessors, node)
    if ctx_type is None:
        ctx_type = type(node.ctx)
        assert ctx_type not in (ast.AugAssign, ast.AugLoad)
    #Only generate SSA variables for variables local to scope
    if node.variable.scope == self.scope.ast_scope:
        if ctx_type in (ast.Store, ast.Param, ast.AugStore):
            for flow_node, kind in res:
                self.scope.graph.add_definition(flow_node, node.variable)
        elif ctx_type is ast.Del:
            for flow_node, kind in res:
                self.scope.graph.add_deletion(flow_node, node.variable)
        elif ctx_type in (ast.Load, ast.AugLoad):
            for flow_node, kind in res:
                self.scope.graph.add_use(flow_node, node.variable)

可以看到，FlowGraph 的接口add_use，参数顾名思义是node 是ASTNode，Var 是Variable

upload successful

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Variable(object):
    'A variable'

    def __init__(self, var_id, scope = None):
        assert isinstance(var_id, str), type(var_id)
        self.id = var_id
        self.scope = scope

    def __repr__(self):
        return 'Variable(%r, %r)' % (self.id, self.scope)

    def __eq__(self, other):
        if type(other) is not Variable:
            return False
        if self.scope is None or other.scope is None:
            raise TypeError("Scope not set")
        return self.scope == other.scope and self.id == other.id

upload successful

FlowScope
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class FlowScope(object):

    def __init__(self, depth, ast_scope):
        self.entry = FlowNode(ast_scope)
        self.graph = graph.FlowGraph(self.entry)
        self.exceptional_exit = FlowNode(ast_scope)
        self.graph.add_node(self.exceptional_exit)
        self.graph.annotate_node(self.exceptional_exit, EXCEPTION_EXIT)
        self.depth = depth
        self.exception_stack = BlockStack()
        self.exception_stack.push_block()
        self.breaking_stack = BlockStack()
        self.continuing_stack = BlockStack()
        self.return_stack = BlockStack()
        self.return_stack.push_block()
        self.ast_scope = ast_scope
1
2
3
4
5
6
7
8
9
10
def _walk_scope(self, scope_node):
    '''Returns: whether this scope raises an exception (or not)'''
    prev_flow_scope = self.scope
    if prev_flow_scope is None:
        self.scope = FlowScope(0, scope_node)
    else:
        self.scope = prev_flow_scope.inner(scope_node)
    predecessors = SingletonNodeSet(self.scope.entry, NORMAL)
    for _, _, child_node in iter_fields(scope_node):
        predecessors = self._walk(child_node, predecessors)

predecessors 这中间有个递归调用

  • iter_fields 可以理解为遍历ast Node
  • _walk 可以理解为根据当前节点类型，遍历

upload successful

upload successful

1
2
3
4
5
6
7
8
9
self._walkers = {
            list : self._walk_list,
            bool : self.skip,
            int : self.skip,
            float : self.skip,
            bytes : self.skip,
            str : self.skip,
            complex : self.skip,
            type(None) : self.skip,

walkers 针对不同的ast 类型，有不同的处理方式，想上面列的，str 之类的，无需放到CFG 中，CFG 主要关注的是方法调用、行参实参的传播，我们重点看这一块的处理。

LexicalPass
1
2
3
4
5
6
7
8
9
class LexicalPass(Pass):

    def extract(self, ast, comments, writer):
        'The entry point'
        LexicalModule(ast, comments, writer).extract()


class LexicalModule(object):
    'Object for extracting lexical information for the given module.'

用于从给定模块中提取词汇信息的对象

ObjectPass
1
2
3
4
5
6
7
8
9
10
11
12
class ObjectPass(Pass):
    '''Generates relations for objects. This includes information about
    builtin objects, including their types and members.
    It also generates objects for all literal values present in the Python source.'''

    def extract(self, ast, path, writer):
        self.writer = writer
        try:
            self._extract_py(ast)
            self._extract_possible_module_names(path)
        finally:
            self.writer = None

生成类信息

2.13 疑问

  • 跨文件分析是如何实现的？CST 都是单文件的分析

3. Trap2DB

3.1 pre-finalize

upload successful

upload successful

  1. trap 文件读取

FileSubtask

1
2
3
public InputStream makeNewStream() throws IOException {
    return CompressedFileInputStream.fromFile(this.trapFile);
}

upload successful

  1. ImportTasksProcessor
1
2
3
4
5
6
TrapInputStream trapInputStream = new TrapInputStream(subtask.getOriginalInputStream(), subtask);

try {
    TRAPReader trapReader = new TRAPReader(trapInputStream, subtaskName, this.getThreadWriter(), this.ids, this.trapReaderConfig, this.cancelToken);
    trapReader.importTuples((LinkTarget[])subtask.getAdditionalInformation());
    bytesRead += trapReader.getBytesRead();

upload successful

  1. TRAPReader
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    protected void scanTuplesAndLabels(TrapScanner scanner, ScanMode scanMode) throws IOException {
        int tryResolveTuplesThreshold = 1000;
        ArrayList<Object> fields = new ArrayList();
        String previousLabel = null;

        while(true) {
            this.cancelToken.checkCancelled();
            TokenKind token = scanner.nextToken();
            String labelToModify = previousLabel;
            previousLabel = null;
            switch (token) {
                case LABEL:
                    if (!scanMode.computeLabels) {
                        skipLabel(scanner);
                    } else {
                        String label = scanner.getLabelValue();
                        expectToken(scanner.nextToken(), TRAPReader.TokenKind.EQ, "=");
                        this.scanLabelValue(scanner, label);
                        previousLabel = label;
                    }

upload successful

每个Trap 内容都是单独的，需要整合所有的trap 文件

3.2 Trap 格式

3.2.1 基础结构

Tuple 结构
1
2
3
public class Tuple {
    private String tableName;
    private Object[] fields;

顾名思义，python tuple的意思，不过不经相同，以trap 文件内的containerparent(#10007, #10006) 为例

upload successful

tuple 表名会最终对应codeql db 中的一个.rel 表

PreThreadWriter

upload successful

RelationEntry
1
2
3
4
5
6
7
8
private class RelationEntry {
    private final String name;
    private final RelationType type;
    private final List<Path> fragmentPaths = new ArrayList();
    private ConcreteRelationWriter relation;
    private ClosableSink sink;
    private int writtenTuples = 0;

addTuple 接口

1
2
3
4
private void addTuple(long[] tuple) {
    this.sink.addTuple(tuple);
    this.increaseTupleCount();
}
BTreeRelationWriter

BTree Relation 指的是数据库中的关系型数据存储，它使用了B 树结构作为索引，从而实现对数据的快速查询和访问。

3.2.2 流程

Trap 文件解析
  • LABEL
  • IDENTIFIER

TRAPReader

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
protected void scanTuplesAndLabels(TrapScanner scanner, ScanMode scanMode) throws IOException {
    while(true) {
        this.cancelToken.checkCancelled();
        TokenKind token = scanner.nextToken();
        String labelToModify = previousLabel;
        previousLabel = null;
        switch (token) {
            case LABEL:
            case IDENTIFIER:
            case PUSH:
            case POP:
            case IF_FRESH:
            case END_IF_FRESH:
            case IMPLEMENTATION:
            case EOF:
LABEL
1
#10000 = @"('/Users/m0d9/Downloads/langflow-main/scripts/ci/update_starter_projects.py', 'utf-8;sourcefile')"

LABEL = #10000
Value = @”(‘/Users/m0d9/Downloads/langflow-main/scripts/ci/update_starter_projects.py’, ‘utf-8;sourcefile’)”
LABELValue 会根据Value 生成ID

例如：
upload successful

IDENTIFIER
1
files(#10000, "/Users/m0d9/Downloads/langflow-main/scripts/ci/update_starter_projects.py")

Identifier 需要指定Label
tableName = file

最终结果会被保存在TaggedTuple 结构中

upload successful

Merge

先写cache/working/xxx.relcheck

再通过Merge().merge 合并

upload successful

那么cache/working/xxx.relcheck 是何时生成的呢？

答案是在RelationEntry 每一次addTuple 中就会触发，实际上会直接往这个relcheck 文件里面写

upload successful

Saver

Saver addTag/addTuple/flush 都会进行文件写入

addTag

1
2
3
4
5
6
7
8
9
10
11
12
13
private int addTag(byte tag) {
    ByteBuffer newBuffer = ByteBuffer.allocate(1);
    newBuffer.put(tag);
    newBuffer.flip();
    this.checksum.update(tag);

    try {
        return this.channel.write(newBuffer);
    } catch (IOException var4) {
        IOException e = var4;
        throw this.makeResourceError(e);
    }
}
1
2
3
4
5
6
7
8
9
public void addTuple(long... longTuple) {
    this.updateIntTuple(longTuple);
    if (this.ibuf.remaining() < this.intTuple.length) {
        this.flush();
    }

    this.ibuf.put(this.intTuple);
    this.noTuples = false;
}

upload successful

至于文件格式，为codeql 独有的

3. blib2to3 VS AST

5. 其他

可能有安全风险

upload successful

6. 参考

  1. https://github.com/github/codeql/blob/main/python/extractor/README.md

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true