Decorated Anonymous Functions in Python
Python functions are generally first-class objects, and that is generally all that’s needed to construct higher-order functions and to engage in the style of functional programming those enable. There is, however, no language support for multiline anonymous functions, leaving a very large and unfortunate gap between
def named functions and
lambda expressions. I show in this article how decorators can be abused to lend anonymity to
def function definitions.
Functions as Values
First things first. In Python, a function is a block.
def add(a, b): return a + b
When it fits on a single line, you have the option of writing a function with lambda syntax.
add = lambda a, b: a + b
A lambda expression is especially convenient when you need to pass a simple operation to a higher-order function (a function which accepts another function as an argument).
xs = [1,2,3,4,5] squares = map(lambda x: x**2, xs) assert squares == [1,4,9,16,25]
If your function argument is more complex, you’ve got to define it beforehand, and pass it by reference. This can get messy quickly.
string = "Long live Guido!" # I have a strange desire to separate # vowels from consonants # helper functions is_vowel = lambda c: c.lower() in "aeiou" is_cons = lambda c: c.lower() in "bcdfghjklmnpqrstvwxyz" # define reducer def reduce_letters((vs, cs), ch): """ Reducer function that collects a string's vowels and consonants into a (vs,cs) tuple """ vs = vs + ch if is_vowel(ch) else vs cs = cs + ch if is_cons(ch) else cs return (vs, cs) # reduce with reducer, with ("","") initial argument vowels, conss = reduce(reduce_letters, string, ("","")) assert vowels == "oieuio" and conss == "LnglvGd"
In practice, the above code written in an iterative style, with local variables and a
for loop instead of a reducing function, would be more readable and concise (more pythonic). In many cases, though, abstraction via higher-order functions and combinators lends to more modular and clearly organized code than the equivalent imperative constructs. I think this is a damn shame, because Python actually has great support for passing functions around as objects. When blocks intended as one-off operations require being defined beforehand as local functions, the distinction between functions as reusable, modular operations and lambdas as single-serving operations—a distinction that is typically very helpful in reading python code—is eroded. A significant factor keeping this functional style from being easily authored and read is syntactic support for expression-level multiline functions, which would allow the definition of operations exactly where they are needed (not sooner).
In Python, a function definition is a statement, not an expression. This means the
end keyword, or wrapping the body in curly braces). While they often lead to trouble, function expressions open up possibilities for very elegant and robust APIs.
Of course, Python has no need for such toys (because flat is better than nested…right?). But if it wanted, it could almost fake them using tools it does have. I will show how.
Decorators and Local Assignment
If you are not familiar with them, decorators are a way to modify the behavior of functions in Python. An introduction is beyond the scope of the present article, but here’s a good decorator tutorial, and if you’re up for it you can read the full PEP 318. Decorators are commonly used to registers functions as handlers for certain events, to add automatic behavior for certain inputs or outputs (memoization, error handling, etc.), or to allow for easier authoring of more “advanced” functionality (
functools.wraps, etc.) Fundamentally, decorators are functions that accept a single function argument, and return a new function that replaces the defined function in the defined function’s own local scope. Inside, they look like this.
from functools import wraps def log_finished(func): @wraps(func) def wrapped(*args, **kwargs): func(*args, **kwargs) print func.func_name, "finished" return wrapped @log_finished def useless(): pass useless() # prints "useless finished"
@wraps helper is unnecessary, but I include it because decorators should not be authored in practice without it. It helps keep the behavior of a decorated function from changing in hidden ways.
As it turns out, decorator functions aren’t written in a special way so much as they adhere to a certain convention that allows them to be used as decorators. Most simply, decorators are functions (callables) that accept a function as an argument and return a function as result. More concretely, decorators are simply syntactic sugar for the following form.
@decorate def func(): pass # essentially desugars to: def func(): pass func = decorate(func)
In english, a decorated function definition is effectively equivalent to defining a plain function, passing that function to the decorator, and overwriting the local function variable with the return value of the decorator. Note that there is nothing stopping the decorator from returning something other than a function to replace the original local function variable, which makes decorator syntax abusable.
Perhaps surprisingly, the following code runs.
def call(func): return func() @call def return_two(): return 2 assert return_two == 2
That is, the decorator
call gets called as the function
return_two is defined, with
return_two passed as an argument, and returns the value that
return_two returns, the int value
2 is returned by the decorator, the local variable
return_two, which was originally a function object, gets immediately overwritten with
2. In other words, whenever a decorator decorates a function, the function gets replaced by whatever the decorator returns, whether that’s a function or any other Python object. Funky!
A word of warning: The (ab)use of decorators for these effects are not consistent with their intended use when added to the language, nor with the universal expectation that decorated functions can still be used as functions. This author encourages both wild experimentation in isolation and responsible treatment of colleagues and library users.
Having discovered this secret about decorators, we can exploit them to implement a facility for passing an anonymous block into a function that expects one. Because a decorator only gets called with a single argument (the function to decorate), a function that supports this style of anonymous blocks will need its block to be passed in both
- as the last argument, and
- in a separate function call, e.g.
To upgrade the builtin
reduce function to behave this way, we can write a helper.
def block_reduce(xs, initial=None): def with_block(block): if initial is None: return reduce(block, xs) return reduce(block, xs, initial) return with_block
And with this, we can rewrite our previous example as the following.
@block_reduce(string, ("","")) def vowels_conss((vs, cs), c): vs = vs + c if is_vowel(c) else vs cs = cs + c if is_cons(c) else cs return (vs, cs) assert vowels_conss == ("oieuio", "LnglvGd")
block_reduce first accepts the iterable and (optional) initial argument for
reduce before accepting a function that gets applied as the reducer, and then its return value replaces
vowels_conss in the local scope, such that the variable
vowels_conssis no longer a function at all, but the result of the whole
block_reduce operation. Ignore for a moment that the syntax used this way is obviously whack, and notice that
vowels_conss didn’t need to be defined as a named function before it was used (its definition is its use), and that at no point in the execution of the above code does
vowels_conss even exist in the local scope as a function that can be mistakenly referenced elsewhere. Anonymous!
Function currying is a style of defining n-ary functions in terms of unary functions. It’s more common in practice in functional languages, including Haskell. A curried function that expects multiple arguments is built in terms of a series of functions which each only expect one of those arguments (unary functions), the last of which ultimately returns the result. In a language like Python, where a normal function would be invoked as
func(arg1, arg2, arg3), an equivalent curried function would be invoked as
curried_func(arg1)(arg2)(arg3). As you can imagine, writing curried functions by hand is unfortunately verbose in Python.
The important part is that, with a bit of introspection, a function can be written as usual in idiomatic Python and be extended to support (optional) curried invocation with the addition of a simple
@curried decorator. An example implementation, as well as a slightly better introduction to optionally-curried functions can be found on my github (a more comprehensive set of functional utilities can be found in fn.py or funcy.
With optional curried invocation, a function expecting a block as well as non-block arguments can be written in idiomatic python and can be subsequently used with our pseudo-anonymous blocks.
from curry import curried # curried helpers c_map = curried(lambda xs, fn: map(fn, xs)) c_join = curried(lambda s, xs: s.join(xs)) # --- Ex. 1 --- words = "Long live Guido!".split() @c_join(" ") @c_map(words) def exclaim_string( word ): return word.upper() + '!' assert exclaim_string == "LONG! LIVE! GUIDO!!" # --- Ex. 2 --- @c_map(range(1,101)) def fizzbuzz( i ): s = "Fizz" if i%3 is 0 else "" s += "Buzz" if i%5 is 0 else "" return s or i # fizzbuzz == the standard FizzBuzz sequence # --- Ex. 3 --- import json untrusted_json = '[["a", "b"], ["c"]]' untrusted_obj = json.loads(untrusted_json) # ensure untrusted_obj is a list of # lists of single-character strings assert type(untrusted_obj) == list @c_map(untrusted_obj) def valid_obj(untrusted_xs): assert type(xs) == list @c_map(untrusted_xs) def valid_xs(c): assert type(c) == unicode assert len(c) == 1 return c return valid_xs assert valid_obj == [["a", "b"], ["c"]]
Elegant? Hell no. Useful? Perhaps not, but I found this possibility of hacky anonymous functions in Python surprising and interesting, and I wanted to share. Mostly, I just wish multiline anonymous function definitions (as expressions) were compatible with Python’s indentation-based syntax. Tell me what you think [@ryanartecona](https://twitter.com/ryanartecona).