CS336 学习笔记（Assignment 1）

Assignment 1, Writeup

包含：

1. Byte-pair encoding (BPE) tokenizer
2. Transformer language model (LM)
3. The cross-entropy loss function and the AdamW optimizer
4. The training loop, with support for serializing and loading model and optimizer state

Byte-Pair Encoding (BPE) Tokenizer

Problem (unicode1): Understanding Unicode (1 point)

(a) What Unicode character does chr(0) return?

Null 字符。

(b) How does this character’s string representation (__repr__()) differ from its printed representation?

交互环境下输入 chr(0) 就是调用 __repr__()，看到 ‘\x00’。print(chr(0)) 什么都不显示。

(c) What happens when this character occurs in text? It may be helpful to play around with the following in your Python interpreter and see if it matches your expectations:

>>> chr(0) 
>>> print(chr(0)) 
>>> "this is a test" + chr(0) + "string" 
>>> print("this is a test" + chr(0) + "string")  

>>> chr(0)
'\x00'
>>> print(chr(0))

>>> "this is a test" + chr(0) + "string"
'this is a test\x00string'
>>> print("this is a test" + chr(0) + "string")
this is a teststring

Problem (unicode2): Unicode Encodings

(a) What are some reasons to prefer training our tokenizer on UTF-8 encoded bytes, rather than UTF-16 or UTF-32? It may be helpful to compare the output of these encodings for various input strings.

UTF-32 是定长的，每个字符固定 4 字节，导致处理英文文本时空间浪费。

UTF-8 对 ASCII 字符（英文、数字）完全兼容（只占 1 字节），在互联网上最通用。

UTF-16 存在大端序/小端序问题。

(b) Consider the following (incorrect) function, which is intended to decode a UTF-8 byte string into a Unicode string. Why is this function incorrect? Provide an example of an input byte string that yields incorrect results.

def decode_utf8_bytes_to_str_wrong(bytestring: bytes): 
    return "".join([bytes([b]).decode("utf-8") for b in bytestring])  
>>> decode_utf8_bytes_to_str_wrong("hello".encode("utf-8")) 'hello'

Deliverable: An example input byte string for which decode_utf8_bytes_to_str_wrong produces incorrect output, with a one-sentence explanation of why the function is incorrect.

>>> decode_utf8_bytes_to_str_wrong("牛".encode("utf-8"))
Traceback (most recent call last):
  File "<python-input-7>", line 1, in <module>
    decode_utf8_bytes_to_str_wrong("牛".encode("utf-8"))
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "<python-input-4>", line 2, in decode_utf8_bytes_to_str_wrong
    return "".join([bytes([b]).decode("utf-8") for b in bytestring])
                    ~~~~~~~~~~~~~~~~~^^^^^^^^^

原因解释： UTF-8 是变长的。一个汉字占三个字节。

(c) Give a two byte sequence that does not decode to any Unicode character(s).

Deliverable: An example, with a one-sentence explanation.

>>> b'\xFF'.decode("utf-8")
Traceback (most recent call last):
  File "<python-input-11>", line 1, in <module>
    b'\xFF'.decode("utf-8")
    ~~~~~~~~~~~~~~^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

0xff 这种字节在 UTF-8 规范中不出现。