Snake Charmer’s Secret: Metaprogramming the Python SDK

Starting from today, our ecosystem offers everything you need to get easily started with Contentful. We have the usual goodies - SDKs for working with our Content Management (CMA) and Content Delivery (CDA) APIs, but also a detailed Getting Started guides and a tutorial on building your first Django app. And for those of you who like to learn by deconstructing things, there is a Django Demo App available too. The Python SDKs are notable for being some of the fastest SDKs to be developed we’ve released to date. The secret behind this productivity streak was one of the most underlooked programming hacks - the use of metaprogramming.

Wait, what is metaprogramming?

In simple terms, metaprogramming is creating code that interacts with the code itself. Like programming to modify your program's code.

There are many different applications of metaprogramming, but it’s usually divided into two categories:

• Program modification

• Introspection

Program modification allows you to modify the behavior of your existing objects and classes during runtime. You can do this by altering existing method behavior or by adding and removing methods and properties. You can even create entire classes during runtime.

With introspection, you get to take a look at how objects are composed internally, what their current state is, what methods it knows and what metadata is associated with it.

With the proper usage of the metaprogramming tools that Python provides us, we can augment the power of our developer skills significantly, and as a result, create simpler programs that make both the maintenance and their developer experience better.

Metaprogramming inside the SDK

Now that we know a little bit better what metaprogramming is let’s dive into how we use these principles to improve our SDKs.

Property Accessors

We use program modification within our SDKs to provide you access to sys and fields properties in your Contentful Resources in a straightforward and direct way so that you don't have to deal with the object internals yourself and can simplify your code.

The following snippet shows how property access looks like when you’re trying to get the ID of the space to which an entry belongs to, first without and then with some metaprogramming magic.

This approach makes the developers think less about the API responses and allows them to focus on the problem at hand.

Let's take a look at an example of how to implement this yourself:

Say you want to have an object that responds to a property that looks like hello_*, where * could be any name separated by _, and we want the return value of hello_my_name to be "Hello My Name!". For this purpose, we use a method called __getattr__ which allows us to redefine property access within an object.

In our case, we want properties that start with hello_ to behave differently. Therefore, we'll override the behavior for properties matching that pattern, and leave the rest to behave normally.

How do we do this? Let's take a look at what the Python code would be like:

This example is very similar to how the SDKs work internally; I just simplified it a bit for the purpose of illustrating the concept of metaprogramming better.

Object Creation

Now let's go through a different use of metaprogramming within the SDKs, involving the mapping of JSON to Resource objects.

If you have already looked at our SDKs usage guide, you should have noticed that we map API JSON responses to Python objects, each with a class that corresponds to its sys.type property, with properties and methods that make sense for their usage.

But how do we do this? Well, there are two ways to tackle this problem.

You can manually map each class. This approach would require a lot of code duplication and may lead to errors. Alternatively, you can use a nifty introspection trick to reduce the overall amount of code and improve its quality and readability. Let's see how that works in practice.

Looking into the internals. What’s in our namespace?

Python has a function called globals which returns a map of all the names loaded in the current namespace. This function includes declared variables, classes, functions, etc. In our previous example, the globals function would contain {'Greeter': Greeter}.

Inside our SDKs, we have a list of whitelisted names that can be present on the sys.type field of a resource. Additionally, we have the classes that match those names loaded on our namespace, so, if the JSON includes a value from within our whitelist, we get the matching class from globals and use it to create the appropriate object.

With this approach, we achieve simplicity within our code and can create all resources without having to duplicate the code responsible for the Resource object creation.

Going Meta

These examples I showed you are just a small portion of the vast amount of things you can do with metaprogramming.

If you want to learn more about metaprogramming and Python, you can take a look at how we deal with request retries on rate limits by using decorator classes where we change the behavior of the class to work as a decorator by re-implementing its __call__ method which allows us to treat it as a function. There are a lot more tricks to be learned, and I’ll be here writing a lot more about them in the future.

As always, please feel free to contribute or submit issues to the SDKs and examples: your feedback is invaluable, and we're looking forward to hearing what you think and how we can further improve.

So stay tuned, and IMPORT ALL THE THINGS!