# Python Tokenize Module

The `tokenize` module provides a lexical scanner for Python source code, implemented in Python. It returns a stream of tokens as seen by the Python parser. This is useful for implementing syntax highlighters or code analysis tools.

## Importing the Module

```python
import tokenize
```

## Tokenizing Code

To tokenize a stream of bytes, use `tokenize.tokenize()`. It takes a callable that returns bytes (like `readline` of a binary file object).

```python
import tokenize
import io

code = b"def hello():\n    print('Hello')"
tokens = tokenize.tokenize(io.BytesIO(code).readline)

for token in tokens:
    print(token)
```

## Token Attributes

The tokens returned are `TokenInfo` objects (named tuples) with the following attributes:

*   `type`: The token type (integer). You can map this to a name using `tokenize.tok_name`.
*   `string`: The actual string of the token.
*   `start`: A `(row, column)` tuple indicating where the token starts.
*   `end`: A `(row, column)` tuple indicating where the token ends.
*   `line`: The logical line on which the token started.

```python
import tokenize
import io

code = b"x = 42"
tokens = tokenize.tokenize(io.BytesIO(code).readline)

for token in tokens:
    token_type_name = tokenize.tok_name[token.type]
    print(f"{token_type_name}: {token.string!r}")
```

## Untokenizing

The `tokenize.untokenize()` function can reconstruct a Python source code string from a sequence of tokens.

```python
import tokenize
import io

code = b"x = 1 + 1"
tokens = list(tokenize.tokenize(io.BytesIO(code).readline))

# Reconstruct the code
reconstructed_code = tokenize.untokenize(tokens)
print(reconstructed_code.decode('utf-8'))
```

## Command Line Usage

You can use the `tokenize` module from the command line to see how a file is tokenized.

```bash
python -m tokenize myscript.py
```

[[programming/python/python]]