Advanced Python Concepts

In this tutorial, a few advanced concepts (not class related) are introduced. This includes

• packing and unpacking

• context manager

• decorator and factories

• exceptions

import time

Packing and unpacking of values

Using * or ** we can pack/unpack list-like objects and dict-like objects, respectively. They act as a “removal” of the parenthesis when situated on the right and as an “adder” of parenthesis when situated on the left of the assigmenemt operator (=).

Let’s play around…

a, c, *b = [3, 4, 4.5, 5, 6]

b

[4.5, 5, 6]

As can be seen, b catches now all the remaining elements in a list. Interesting to see is also the special case if no element is left.

d1, d2, *d3, d4 = [1, 2, 3]  # nothing left for d3

d3

[]

This is simply an empty list. However, this has the advantage that we know that it is always a list.

a = [3, 4, 5]

Multiple unpackings can be added together (however, the other way around does not work: multiple packings are not possible as it is ill-defined which variable would get how many elements).

d, e, f, g, h, i = *a, *b

Now we should be able to understand the *args and **kwargs for functions. Let’s look at it:

def func(*args, **kwargs):
    print(f'args are {args}')
    print(f"kwargs are {kwargs}")

mykwargs = {'a': 5, 'b': 3}
myargs = [1, 3, 4]
func(*myargs, *mykwargs)

args are (1, 3, 4, 'a', 'b')
kwargs are {}

func(5, a=4)

args are (5,)
kwargs are {'a': 4}

# play around with it!

Context manager

A context manager is an object that responds to a with statement. It may returns something. The basic idea is that some action is performed when entering a context and again when exiting it.

with context as var:
    # do something

translates to

# execute context entering code
var = return_from_context_entering_code
# do something
# execute context leaving code

The great advantage here is that the “leaving code” is automatically executed whenever we step out of the context!

This proved to be incredibly useful when operations have cleanup code that we need to execute yet that is tedious to write manually and can be forgotten.

Using yield

One way to create a context manager is to have a function that has a yield.

What is ``yield``?: It’s like a return, except that the executioin stops at the yield, lets other code execute and, at some point, continues again where the yield was. Examples are: - iterator: a function that yields elements. Everytime it is called, it is supposed to yield an element and then continue from there - asynchronous programing: it stops and waits until something else is finished - in the context manager, as we will see

import contextlib


@contextlib.contextmanager
def printer(x):
    print(f'we just entered the context manager and will yield {x}')
    yield x
    print(f'Finishing the context manager, exiting')

with printer(5) as number:
    print(f"we're inside, with number={number}")
print("left manager")

we just entered the context manager and will yield 5
we're inside, with number=5
Finishing the context manager, exiting
left manager

Where is this useful

Basically with stateful objects. This includes anything that can be set and changed (mutable objects).

with open('tmp.txt', 'w') as textfile:
    textfile.write('asdf')

The implementation roughly looks like this:

import contextlib


@contextlib.contextmanager
def myopen(f, mode):
    opened = open(f, mode)
    yield opened
    opened.close()

Exercise: create a context manager that temporarily sets a 'value' key to 42 of a dict and switches it back to the old value on exit

testdict = {'value': 11, 'name': 'the answer'}

to be invoked like this

with manager(testdict) as obj:
    # here the value is 42
# here the value is 11

# SOLUTION
@contextlib.contextmanager
def func(x):
    yield x


with func(5) as var1:
    print('inside')
print(var1)

inside
5

@contextlib.contextmanager
def set_answer(obj):
    old_value = obj.get('value')
    obj['value'] = 42
    yield obj
    obj['value'] = old_value

Using a class

Instead of using the yield, we can have advanced control over the enter and exit methods by creating a class and implementing the two methods __enter__ and __exit__

class MyContext:

    def __init__(self, x):
        self.x = x

    def __enter__(self):
        x = self.x
        print('entered')
        return x ** 2

    def __exit__(self, type_, value, traceback):  # but let's not go into things in detail here
        self.x = 42
        print('exited')

with MyContext(5) as x:
    print(x)

entered
25
exited

While a class is way more powerful and offers ways to catch exceptions and more in the exit, ususally the functional way is enough and should then be preferred. If it doesn’t give you enough flexibility, remember the class, look it up and figure out all the things needed.

Decorators and factories

Sometimes we can’t write a function fully by hand but want to create it programatically. This pattern is called a “factory”. To achieve this, instead of having a function that returns an integer (an object), a list (an object), a dict (an object) or an array (an object), we return a function (an object). We see that the concept of Python, “everything is an object”, starts being very useful here.

def make_power_func(power):
    def func(x):
        return x ** power
    return func

pow3 = make_power_func(3)

pow3(2)

8

def make_power_func(power):
    def func(x):
        return x ** power
    power = 42
    return func

pow3 = make_power_func(3)

pow3(2)

4398046511104

# Exercise: test it here

Another example is to create a timing wrapper. Exercise: create a timing function that can be used as follows

timed_pow3 = fime_func(pow3)
pow3(...)

HINT, scetch of solution

def time_func(func):
    def new_func(...):
        print('start')
        func(...)
        print('stop')
    return new_func

# SOLUTION
def timed_func(func):
    def wrapped_func(*args, **kwargs):
        print(args)
        print(kwargs)
        start = time.time()
        func(*args, **kwargs)
        end = time.time()
        print(f'time needed: {end - start}')
    return wrapped_func

def add_notime(x, y):
    return x + y

add_timed = timed_func(add_notime)

import time

add_timed(y=4, x=5)

()
{'y': 4, 'x': 5}
time needed: 9.5367431640625e-07

# test it here

Decorator

There is another way, just syntactical sugar, to make this automatic: a decorator. It is invoked as below

@timed_func
def add(x, y):
    return x + y

Again, as for the contextmanager, we can also use a class here to give more flexibility and create a decorator that takes arguments.

Exceptions

Exceptions are used to stop the execution at a certain point and surface to higher stacks in the code, e.g. to go up in the call stack. A typical use-case is when an error is encountered, such as the wrong type of object is given. Exceptions can also be caught in a try ... except ... block in order to handle the exception.

There are a few built-in exceptions, the most common ones are: - TypeError: object has the wrong type, e.g. string instead of float - ValueError: the value of the object is illegal, e.g. negative but should be positive - RuntimeError: if a function is illegally executed or a status is wrong. E.g. if an object first has to be loaded before it gets parsed. It covers any error that does not fall into an other category. - KeyError, IndexError: if a key or index is not available, e.g. in a dict or list

An Exception can manually be raised by

raise TypeError("Has to be int, not str")

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[35], line 1
----> 1 raise TypeError("Has to be int, not str")

TypeError: Has to be int, not str

Note that it is often convenient to create an instance such as in the example above where the first argument is the message (as we see in the raised Exception above), but we can also raise an exception by only using the class itself

raise TypeError

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[36], line 1
----> 1 raise TypeError

TypeError:

Custom Exception

In Python, exceptions are simply a class. And as such, we can inherit from it and create our own exception.

Attention: inherit from Exception or subclasses of it such as TypeError, ValueError, but NEVER from BaseException.

class MyError(Exception):
    pass

raise MyError("Hello world")

---------------------------------------------------------------------------
MyError                                   Traceback (most recent call last)
Cell In[38], line 1
----> 1 raise MyError("Hello world")

MyError: Hello world

An exception can also be created by inheriting from an already existing exception if it is more specific and provides hints on the nature of the exception.

class NegativeValueError(ValueError):
    pass

Catching exceptions

An exception can be caught in a try..except block. This works as follows: - if an exception is raised in the try block, the next except is invoked - it is tested whether the raised exception is of type subclass of the exception type specified to be caught. For example, except TypeError checks if the raised error is of type TypeError or a subclass of it. - if that is not the case, it goes to the next except statement (yes, there can be multiple) - … more below

try:
    raise NegativeValueError("Negative value encountered")
except ValueError as error:
    print(f"Caught {error}")

Caught Negative value encountered

By using the as keyword, the error that is raised is assigned to a variable. We can inspect the error now if we want or, as above, just print it.

If no error is specified, any error is caught (this should NOT be used, except for special cases

try:
    raise TypeError
# Anti-pattern, do NOT use in general!
except:  # any exception if not specified
    pass

try:
    raise TypeError("Type was wrong, unfortunately")
except TypeError as error:  # any exception
    print(f'caught TypeError: {error}')
    raise
except ValueError as error:
    print(f'caugth ValueError: {error}')

caught TypeError: Type was wrong, unfortunately

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[42], line 2
      1 try:
----> 2     raise TypeError("Type was wrong, unfortunately")
      3 except TypeError as error:  # any exception
      4     print(f'caught TypeError: {error}')

TypeError: Type was wrong, unfortunately

To continue from above: after the last except, an else statement is looked for. The else is executed if no exception was raised.

try:
    print('no error raised')
#     raise TypeError("Type was wrong, unfortunately")
except TypeError as error:  # any exception
    print(f'caught Type {error}')
except ValueError as error:
    print(f'caugth Value: {error}')
else:
    print("No error")

print("Executed after block")

no error raised
No error
Executed after block

…and finally, after the else, a finally block is looked for. This is guaranteed to be executed! Whether an exception is raised, whether it is caught or not, whether there is an else or not, the finally is always executed.

Therefore it is suitable for any cleanup code such as closing files, removing temporary files and more.

try:
#     pass
#     raise TypeError("Type was wrong, unfortunately")
    raise RuntimeError("Type was wrong, unfortunately")
except TypeError as error:  # any exception
    print(f'caught Type {error}')
except ValueError as error:
    print(f'caugth Value: {error}')
else:
    print("No error")
finally:  # POWERFUL! Guarantied to be executed
    print('Finally run')
print("Executed when passed")

Finally run

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[44], line 4
      1 try:
      2 #     pass
      3 #     raise TypeError("Type was wrong, unfortunately")
----> 4     raise RuntimeError("Type was wrong, unfortunately")
      5 except TypeError as error:  # any exception
      6     print(f'caught Type {error}')

RuntimeError: Type was wrong, unfortunately

Note that in the above example, the error was not caught! All the other statements could also be omitted and only a try...finally block can be created.

try:
    raise ValueError
finally:
    print('raised')

raised

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[45], line 2
      1 try:
----> 2     raise ValueError
      3 finally:
      4     print('raised')

ValueError:

pitfall “guaranteed execution”

As the finally is guaranteed to be executed, this can have an odd effect: possible return statements can be ignored before the finally IF the finally also has a return statement. The logic says here that the finally return must be executed, as it is guaranteed to be executed.

def func(x):
    try:
        if x == 5:
            raise RuntimeError('called inside func')
    except RuntimeError as error:
        return error
    else:
        print('else before 42')
        return 42
        print('after else 42')
    finally:
        print("cleaned up")
        return 11

func(6)

else before 42
cleaned up

11

Exceptions as control-flow

We are used to control-flow elements such as if...elif...else blocks. However, exceptions can also be used for this. They do not replace other ways of control-flow, however there are sometimes situations in which they provide a golden solution to steer the execution.

As an example, consider an add function that sometimes can add three elements - which is, for the sake of a good example, favorable as more performant (real world cases of this exist in larger scale, but too complicated to explain here) - and sometimes not. Also assume that the add function is called again maybe inside. A solution is to execute add with three elements. If an error is raised, we catch it (the specific one), and run the function again with two arguments and add the third argument by calling add again.

Note that this also solves the problem if add is called deeper nested again: we don’t care where it is called, we just try again with only two numbers. The advantage is that we don’t need to check the output of the function; this will always be a number (and not a None or something like this).

def add(a, b, c=None):
    if c is not None:
        raise MyError
    return a + b

add(1, 2, 3)

---------------------------------------------------------------------------
MyError                                   Traceback (most recent call last)
Cell In[49], line 1
----> 1 add(1, 2, 3)

Cell In[48], line 3, in add(a, b, c)
      1 def add(a, b, c=None):
      2     if c is not None:
----> 3         raise MyError
      4     return a + b

MyError:

try:
    result = add(1, 2, 3)
except MyError as error:
    result = add(add(1, 2), 3)
result

6