11.1.2 Data Types

The grammar rules for expressions that apply procedures are:

In Python, nearly every data value (including lists and strings) is an object. This means the way we manipulate data is to invoke methods on objects. To invoke a method we use the same rules, but the PrimaryExpression of the CallExpression specifies an object and method:

The name AttributeReference is used since the same syntax is used for accessing the internal state of objects as well.

The tokenize procedure includes five method applications, four of which are tokens.append(current). The object reference is tokens, the list of tokens in the input. The list append method takes one parameter and adds that value to the end of the list.

The other method invocation is c.isspace() where c is a string consisting of one character in the input. The isspace method for the string datatype returns true if the input string is non-empty and all characters in the string are whitespace (either spaces, tabs, or newlines).

The tokenize procedure also uses the built-in function len which takes as input an object of a collection datatype such as a list or a string, and outputs the number of elements in the collection. It is a procedure, not a method; the input object is passed in as a parameter. In tokenize, we use len(current) to find the number of characters in the current token.