Home » Programming » Python » Modules » Tokenize Module
📍 Unpin 🔗 Copy Link 📋 Copy Markdown ⬇️ Download
2 min read

Python Tokenize Module

The tokenize module provides a lexical scanner for Python source code, implemented in Python. It returns a stream of tokens as seen by the Python parser. This is useful for implementing syntax highlighters or code analysis tools.

Importing the Module

import tokenize

Tokenizing Code

To tokenize a stream of bytes, use tokenize.tokenize(). It takes a callable that returns bytes (like readline of a binary file object).

import tokenize
import io

code = b"def hello():\n    print('Hello')"
tokens = tokenize.tokenize(io.BytesIO(code).readline)

for token in tokens:
    print(token)

Token Attributes

The tokens returned are TokenInfo objects (named tuples) with the following attributes:

  • type: The token type (integer). You can map this to a name using tokenize.tok_name.
  • string: The actual string of the token.
  • start: A (row, column) tuple indicating where the token starts.
  • end: A (row, column) tuple indicating where the token ends.
  • line: The logical line on which the token started.
import tokenize
import io

code = b"x = 42"
tokens = tokenize.tokenize(io.BytesIO(code).readline)

for token in tokens:
    token_type_name = tokenize.tok_name[token.type]
    print(f"{token_type_name}: {token.string!r}")

Untokenizing

The tokenize.untokenize() function can reconstruct a Python source code string from a sequence of tokens.

import tokenize
import io

code = b"x = 1 + 1"
tokens = list(tokenize.tokenize(io.BytesIO(code).readline))

# Reconstruct the code
reconstructed_code = tokenize.untokenize(tokens)
print(reconstructed_code.decode('utf-8'))

Command Line Usage

You can use the tokenize module from the command line to see how a file is tokenized.

python -m tokenize myscript.py

programming/python/python

Created: March 17, 2026 • Last Modified: January 27, 2026