How to throttle a Django view

In the last post, we built a todo list application with Django and Ember. It contained a register view to allow users to create accounts.

Right now, there is nothing to prevent an attacker from flooding that view and filling up the database with fake users. We want to throttle access to the view and limit the damage an attacker can cause.

I investigated several packages that provide throttling functionality, but the cleanest one I found is django-ratelimit. Here is how to use it:

Installing django-ratelimit and setting up the cache

First, we need to install it using pip:

Before we can use it to throttle views, we need to set up a cache for Django. Memcached is the recommended solution.

On Ubuntu 14.04, we can install Memcached and the python-memcached binding that Django needs with two commands:

Then we must add the CACHES dictionary to settings.py.

Check the Django documentation here for more detailed instructions.

Throttling a function-based view

Throttling the register view is as simple as applying a decorator to it.

We are setting the key argument to "ip", which means that we are throttling requests from the same IP address. It is possible to use different keys, but the IP address will work for our purposes.

We are also setting the rate to 10/h, or ten times per hour.

The method argument specifies which HTTP methods are to be throttled. ratelimit.UNSAFE is a shorthand for the list ['DELETE', 'PATCH', 'POST', 'PUT'].

If the limit is exceeded, the server will return 403 Forbidden.

Throttling a class-based view

If we had to throttle a class-based view, we would use the mixin that django-ratelimit provides.

The mixin works in much the same as the decorator, but instead of decorator arguments we are using class attributes prefixed with ratelimit_.

Specifying rates

We can specify rates in the format N/u or N/Mu, where N and M are integers and u is a unit. Here are some examples:

  • 1000/s – One thousand times per second
  • 5/m – Five times per minute
  • 10/h – Ten times per hour
  • 20/d – Twenty times per day
  • 100/5h – One hundred times every five hours

That’s it for now. django-ratelimit is a pretty good solution for throttling normal views. If you are using an API library such as Django Rest Framework, you should check if it has built-in throttling functionality.

Making Ember and Django play nicely together: a to-do app walkthrough

I’ve wanted to get started building applications with Ember for a while, but I never invested the time to figure out how to integrate it with Django. I’ve spent the last few days doing just that, however, and it’s been a nightmare of outdated libraries and vague documentation. The main obstacle was getting authentication working. At one point I ended up on page 7 of Google, so that will give you an idea of how bad it was. I’m writing this post so you don’t have to go through the same pain.

Here is what we are going to build:

  • An Ember todo list CRUD app
  • Using a JSON API-compliant backend built with Django Rest Framework
  • Secured using token authentication
  • With the ability to login and register new users

Here are some screenshots of the (unstyled) finished application:

new_todo todo_list_index user_login user_registration

This tutorial is for Ember 2.4 and Django 1.9. I recommend that you go through the basic Ember tutorial first to orientate yourself, although I will try to explain everything as I go along.

This tutorial is pretty long. Want a PDF?

Just type in your email address and I'll send a PDF version to your inbox.

Powered by ConvertKit

Setting up the project

We’re going to put the Ember project and the Django project side by side in the same directory. We are not going to embed the Ember project within Django’s static files, as some people do. There is really no reason to do that because the whole point of front-end Javascript frameworks is that they are backend-agnostic – they should work equally well with any backend that implements the necessary API calls. It is not even obvious that we would put the Ember client and the Django API on the same server, so it is best to keep them separate.

Create a directory to hold everything and cd into it:

Use the ember new command to generate an Ember application called todo-ember in that directory:

cd into the directory and install some Ember libraries that we are going to need:

Now we will generate the Django project. We will also start a virtualenv for it. From the root directory – the one where we ran ember new – run the following commands:

And we will generate an app inside our project to hold our todo list implementation:

We’re going to need some Django packages to build an API that plays nice with Ember. Install them using pip:

Now add everything to INSTALLED_APPS in settings.py.

We need the django-cors-headers package to add Cross Origin Resource Sharing headers to our API responses. This will allow our Ember development server running on localhost:4200 to talk to the Django development server on localhost:8000. Add the middleware from that package to MIDDLEWARE_CLASSES :

And set the CORS_ORIGIN_ALLOW_ALL setting to True (this will be fine for development, but don’t run it like that in production):

Now add the Django Rest Framework settings to settings.py. Most of this is overriding Django Rest Framework defaults with classes from the djangorestframework-jsonapi , which make Django Rest Framework conform to the JSON API specification that Ember Data expects. We also set TokenAuthentication as a default authentication class:

Implementing the Django TodoItem model

With the basic project skeleton in place, we can create a model for todo items. Open models.py in the todo app and implement the following:

Creating a serializer for TodoItem

Because we are exposing the model with Django Rest Framework, it is going to need a serializer. Make a file called serializers.py in the todo app directory. It should look like this:

Exposing TodoItem as an API

We can take advantage of standard Django Rest Framework functionality to expose REST endpoints for the model. In views.py in the todo app, add the following:

Before we can actually call the endpoints, we need to register the ViewSet in urls.py.

In todo_django/urls.py, register it as follows:

One important point: when instantiating the DefaultRouter, make sure to pass the trailing_slashes=False argument. Otherwise Django will try to redirect calls to /api/todos to /api/todos/, which confuses Ember Data.

As usual when we create a new Django model, we need to migrate:

Now run the Django development server and go to http://localhost:8000/api/todos. You should see the standard Django Rest Framework browseable API screen.

Generating an Ember scaffold with ember-cli-scaffold

We can use the pretty cool ember-cli-scaffold to automatically generate most of what we need to CRUD todo items on the Ember side. From inside the todo-ember directory, run this command:

You should get a printout of everything that was generated. As you can see, it generated a model, routes and templates for us. You can look through the generated files to see what is going on in them.

Something that sucks about the scaffold is that it generated input text components for the boolean attribute on the model. We can change that easily enough.

Open the handlebars template in todos/-form.hbs and find the form component linked to the model’s done component. It should look like this:

Just change it to this:

It also automatically set up a Mirage mock api server to test against, but we are going to use the Django development server, so we don’t need that in our project. Get rid of it like so:

And it generated an adapter in adapters/todos.js that we won’t need, so delete that too.

Adding an application adapter

Ember Data uses adapters to take requests for information from the store and translate them into calls to the persistence layer. We will create an application adapter that Ember Data will use for all API calls by default:

Open up adapters/application.js and add the following:

You’ll notice that we have imported ENV from todo-ember/config/environment.js. This is the Ember equivalent of Django’s settings.py where we can set up different options for when the app is running in dev, test or production.

Let’s set values for ENV.host in there. We’re pointing to the Django development server in development mode:

With that in place, the Ember development server will direct all its API calls to localhost:8000 where your Django development server is running.

Start both servers now and in your browser go to http://localhost:4200/todos. You should find that you can create, read, update and delete records on the server from your Ember frontend. Keep an eye on the Django dev server output to see the requests going through.

Setting up Django for token authentication

So far we’ve built a lot of functionality with nor much code, but we’ve got a problem: anybody can access the server and change our data. Let’s fix that so that only registered users can access the todos route.

We are going to use the token authentication mechanism that comes with Django Rest Framework.

First we will add endpoint in Django that our Ember application can call to receive a token that it will use to validate future API calls. Edit urls.py:

How does this endpoint actually work?

We are going to set up our Ember application to POST some JSON to the endpoint. The JSON will contain a username and password. If the username and password are correct, the endpoint will return a 200 response containing a token.

If the username and password are not correct, it will return a 400 response with an error message. You can try it out with CURL.

First, let’s see what it does with bad credentials:

Now with good credentials:

We will use an Ember addon called Ember Simple Auth to store this token and add it as a header to future API calls.

Setting up Ember Simple Auth

In the Ember project, add the following settings to environment.js. They control the behaviour of Ember Simple Auth:

The code is quite self explanatory, but if you don’t get it right now, relax. You will see what those settings are used for later when we start adding route mixins.

Now generate an application controller (you’ll need to generate a route first):

And add the following code to it:

We are implementing the invalidateSession action on this controller so we can have a logout link on every page. Add the following snippet to the application template in application.hbs:

Creating the login form and controller

Let’s generate an Ember route called login where our login form will live.

While we’re at it, let’s generate a controller for the route:

Open the template for the route – login.hbs – and add a simple login form:

This won’t work until we implement the authenticate action on the controller, so open up controllers/login.js and add the following:

The code in this file deserves some explanation. At the top of the controller, we are injecting the Ember Simple Auth session service, which manages session state for the application.

Then, in the authenticate action, we are calling authenticate on the service. You have probably noticed that the first argument to authenticate is drf-token-authenticator. This refers to a custom Ember Simple Auth authenticator that we have not implemented yet, so let’s do that.

Implementing a custom authenticator

Inside your ember app directory, make a directory called authenticators. Inside that directory, make a file called drf-token-authenticator.js This is where our custom authenticator will live.

Add the following code to the file:

A few words of explanation are required:

We are extending the Ember base authenticator and implementing the restore and authenticate methods.

restore “restores the session from a session data object. This method is invoked by the session either on application startup if session data is restored from the session store.” If we don’t implement this then if we log in and then refresh the page for instance we will be kicked back to the login screen.

authenticate is where we call the /api-auth-token/ endpoint we created in Django. The method returns an Ember Promise that resolves or rejects based on the response to the API call.

The API call itself is made with jQuery, which is embedded into Ember. Notice that we are using the ENV.host that we placed in environment.js earlier to direct the AJAX request to the proper endpoint.

If you run the application now and go to http://localhost:4200/login you should see a login form. Try to log in with an invalid username and password first. You should see the error JSON from the server underneath the form. Then try it with a valid username and password. You should see the “login” link at the top of the page change to a “logout” link.

Securing the API

So far so good, but we’re not actually preventing anyone from sending unauthenticated requests to the API. To do that, we need to edit the ViewSet in Django:

Notice that we added two new properties – authentication_classes and permission_classes.

Right now if you go to http://localhost:4200/todos you will find that our application is broken. Django expects each incoming request on the /todos/ route to have a valid Authorization header with the value Token my_token_value. In order to automatically add that header to outgoing requests, we need to write a custom authorizer.

Writing a custom authorizer

Authorizers are components of Ember Simple Auth that add authorization information to requests made by Ember.

Make a directory inside your app called authorizers and make a file inside that directory called drf-token-authorizer.js. Add the code below to that file:

We are extending the base authorizer from Ember Simple Auth and implementing authorize on it. This checks that the session is authenticated, grabs the token from the passed in sessionData variable, and adds it as a header by calling the passed in block with the header name and the header value.

To make sure that the header is added to all API calls that Ember makes, we need to modify the application adapter. Open adapters/application.js and change it from what we had earlier:

You can see here that we are importing the DataAdapterMixin from Ember Simple Auth and adding it to the adapter as a mixin. We also need to set up authorizer to point to our custom authorizer.

At this point, if you try your application again you should be able to successfully view, edit, create and delete todo list items.

Adding Ember Simple Auth route mixins

A problem with our application right now is that it does not prevent you from accessing the todos route without logging in (although after the last change the API call to fetch the todo items won’t work). We would like users who go straight to todos without logging in to be sent instead to the login route.

It doesn’t automatically send you to todos after login either.

And it would be great if users who go to login while already logged were sent back to todos.

It is quite simple to achieve these things using route mixins. The first one we need to add is in the application route.

Open routes/application.js and change it so it looks like this:

This will enable Ember Simple Auth to automatically change the route based on the authentication state.

Now we need to protect the todos route so that it cannot be accessed without being logged in. Open each route in routes/todos/ and add the AuthenticatedRouteMixin. For example, routes/todos/edit.js should look like this:

Do the same for the nested index and new routes.

If you go to http://localhost:4200/todos now without being logged in you will be kicked back to the login page. Where you go depends on the value for authenticationRoute that we added to environment.js earlier. routeAfterAuthentication and routeIfAlreadyAuthenticated control where you do when you log in and when you come back to the site after already being logged in.

The last mixin we need to add is the UnauthenticatedRouteMixin in the login route. It prevents authenticated users from seeing the login page. Here is what your routes/login.js should look like after you add it.

Registering users

At the moment all we can do is log in with existing users. Nobody else can register an account. Let’s fix that.

We’re going to need a registration endpoint in Django that Ember can use. Here is a one I pulled out of an old project. It’s not going to win any beauty contests, but it does the trick:

It relies on a user registration form that takes a username, email address, a password and a copy of the password to make sure they are the same. Here it is (it lives in forms.py):

This is all pretty standard Django stuff, but note the use of JsonResponse and the use of the HTTP status code to signal the outcome of the request.

The view has to be added to urls.py, naturally:

Now that the Django side is ready, let’s take care of the Ember side. We have to generate a register route and controller.

Then we can put a registration form in register.hbs.

Modify the autogenerated register route to include the UnauthenticatedRouteMixin:

Now we will implement the register action on the register controller that the registration form uses. It looks like this:

This follows the same pattern as the login form. We grab the values in the form fields then use JSON.stringify to assemble them into a JSON payload that we POST to the server. The server then returns either 201 Created response or a 400 response with information about what went wrong.

If the success callback is fired, we hide the login form and show the “Signup Complete” message. Otherwise, we write the error message below the form.

In the same was as with the login form, we are just writing the error responses in raw JSON below the registration form. Exactly how you alert the user to errors will depend on your CSS framework and the facilities it provides for, e.g. form element highlighting, etc. In any case, it is a trivial matter to display the errors more nicely on the page, so we won’t waste time with that today.

Now you should be able to register new users and log in with them.

Make a link to the registration form by editing the application.hbs template. Change the line with the “login” link to include a “register” link too:

Linking Todo items to users

Right now when users log in they see all the todo items and not just the ones that belong to them. Let’s fix that so that todo items are linked to users and users can only see and edit their own.

Add a foreign key to the TodoItem model that points to the user:

Ass with all model changes, we need to migrate. When we run makemigrations we will be prompted to specify a default value for the user field on existing rows. Just give it the primary key of an existing user:

Then apply the generated migration:

To restrict the todo items to the current user, override the get_queryset method on the ViewSet:

And remove the queryset:

At this point we need to add the base_name to the router to avoid errors:

When new todo items are created, we want them to be linked to the current user. We can achieve that by overriding perform_create on the ViewSet:

We also want object-level permissions that prevent people from directly accessing or modifying other users’ todo items, so we will implement a custom Django Rest Framework permission class.

You can put the following code anywhere, but I like to put it in a file called permissions.py in the todo app:

This class implements has_object_permission(self, request, view, obj) from the base class and performs a simple check to see if the user on the object is the same as the authenticated user.

Now change permission_classes on the ViewSet to apply this permission:

That’s it! At this point if you register a bunch of users they will all have their own todo items.

The end

Phew! This has been a pretty long post, but if you have followed along you have gotten over the biggest initial hurdles of working with Ember and Django.

Check out the resources in the next section to learn more.

Resources

The Ember Quickstart Guide should be the first destination in your Ember journey.

The Ember Simple Auth documentation will help you get to grips with this indispensable Ember library.

The Django Rest Framework JSON API package makes linking Ember Data and Django Rest Framework pretty seamless.

Built With Ember showcases the sophisticated user experience that can be achieved with this cool framework.

Why Is My Django/MySQL Application Showing Unicode as Question Marks?

Back up your database before you try anything here. Sometimes character set conversions can change your data in ways you don’t want. Be sensible and use mysqldump or something to safeguard it before you start messing around. Needless to say, you should try everything in a test environment first.

When you run a Django application (or any other web application, for that matter) on top of a stock MySQL install, you might hit a problem with storing Unicode characters. I saw it in a Django project that had to deal with Arabic text. Instead of the Arabic characters, it just showed a bunch of question marks.

Here’s how to fix it.

Check your MySQL character set

Out of the box, your MySQL character set is probably latin1 . We’re going to change it to utf8 .

First, run this command to check that you are in fact dealing with an incorrect character set:

In the output, you will probably see the following line:

If you do, keep going. We’re going to sort it out.

Edit my.cnf

The main MySQL configuration file is called my.cnf . On Ubuntu it is located at /etc/mysql/my.conf  . You can check where it is on your own system by running locate my.cnf .

The file is divided into sections and the start of each section is indicated with a name in square brackets. We’re interested in the sections [client]  and [mysqld] .

After making a backup of the current state of the file

open it in your text editor of choice and find the [client]  section. Add the following line to it:

Next, find the [mysqld]  section and add the following three lines to it:

Be careful that you add the code to the right sections. If you make a mistake here then MySQL will not start and it won’t write any useful error message to the logs.

Save my.cnf  and restart MySQL. On many systems, you can do this with the service  command:

Alter each table to use the new character set

First, you want to generate the script you are going to use to convert each table one by one to the new character set. Change the database name, username and password to the correct values and run this in the terminal.

It will generate the SQL you need to change each of your tables. For example, if your database contained three tables called users , comments  and posts , the generated code would look this this:

Run that code against the database using your tool of choice. It might take a while, depending on the size of your tables. You’ll know when you try it on your test environment. When it’s done, those question marks should be history.

How to fix database race conditions in Django views

Today I’m going to show you how to fix an extremely common error in Django applications. My guess is about 90% of Django applications deployed in the wild suffer from this error, and like 72% of statistics I just made that one up on the spot. Seriously though, it’s pretty common.

Imagine you’ve got an online bookstore application with a Book  model that has a quantity attribute. When somebody buys a copy of one of your books, you want to decrease the quantity attribute by 1. Here is the naive way to do it:

At the start when you’ve got a small load on your system, this will seem to work fine. Now imagine your bookstore grows, you open some new branches, and there are multiple updates being run on your application every second. That’s when strange things will start to happen. Here is how two concurrent updates might play out with our current code. book1 represents the first concurrent update and book2 represents the second:

At the start of both concurrent updates, an identical copy of the data in the database is loaded into memory. The inventory quantity is decreased on each copy, then the new quantity is written back to the database, with the second update clobbering the first. Result: it is as if one of the updates never happened.

In database terms, what we need is called a SELECT FOR UPDATE. Basically, this locks the row in the database until the new information is written back, preventing a second instance from reading and modifying data that might be in the process of changing.

Since Django 1.4, implementing SELECT FOR UPDATE through the ORM is really simple:

That will lock the row selected with get until the end of the transaction block, which since Django 1.5 corresponds to the end of the request by default.

select_for_update is compatible with the postgresql_psycopg2, oracle, and mysql database backends. It doesn’t work for the sqlite backend.

How to integrate New Relic with Django, Apache and mod_wsgi

I just finished setting up New Relic application monitoring on Recommendify. It was a little bit of a painful process and it wasn’t properly described in their docs, so I’m going to note the steps I went through here. Hopefully it will help somebody else in the same situation.

Install the New Relic agent

First, install the newrelic package using pip.

Generate the New Relic configuration

Then copy the command to generate the configuration file from your New Relic dashboard. It will look something like this:

LICENSE-KEY will be the specific license key for your account.

This command will create a file called newrelic.ini. Copy it somewhere safe on your server and make sure it has permissions to be read by the user Apache runs under.

Edit wsgi.py

The next step took some time to figure out. New Relic provides a wrapper script for Python applications, but it doesn’t work for setups using embedded interpreters, such as Apache with mod_wsgi. As this is how Recommendify is configured, I had to find another way.

I edited the wsgi.py file from my Django project to wrap the WSGI application object.

This script imports the socket module and used the socket.gethostname() function to get the hostname of the current machine. I did this so that New Relic would only log data in production and not during development.

At the bottom of the script, I check if the application is running on the production environment. If it is, I import newrelic.agent, initialize it by passing it the path to the newrelic.ini created during the previous step, then wrap the WSGI application object using the WSGIApplicationWrapper class.

You will need to use your own Django settings module, hostname string, and path to newrelic.ini, of course.

Once you deploy the modified wsgi.py, you should start seeing monitoring information in New Relic after about five minutes. Here’s what it looks like for me.

new_relic

Understand WSGI by building a microframework

When you’re learning web development in Python, it’s tempting to go straight for higher level frameworks like Django and Flask that abstract the interactions between the web server and the application. While this is certainly a productive way of getting started, it’s a good idea to go back to the lower level at some point so you understand what these frameworks are doing for you. In this post, you will learn about the Web Server Gateway Interface (WSGI) – the standard interface between web servers like nginx and Apache and Python applications. You’ll do that by working from a simple “Hello, world!” WSGI example up to a microframework that supports Flask-like URL routing with decorators, templating, and lets you code your application logic inside simple Django-like controller functions.

The simplest WSGI application

Take a look at this “Hello, world!” code:

The make_server function imported from wsgiref.simple_server  is part of the reference WSGI implementation included in the Python standard library. It returns a web server instance that you can start by calling its serve_forever  method. make_server  takes three arguments: the hostname, the port and the WSGI application itself. In this case, the hello_world  function is the WSGI application. WSGI applications have to be Python callables, i.e. either functions or classes that implement a  __call__  method.

Now let’s take a look at hello_world . You’ll notice that it has two arguments: environ  and start_response . environ  is a dictionary that holds information about the execution environment and the request, such as the path, the query string, the body contents and HTTP headers. start_response  is a function that starts the HTTP response by writing out the status code and the headers. Finally, the function returns a list with a single string in it. This is because the return value of a WSGI application must be

Finally, the function returns a list with a single string in it. This is because the return value of a WSGI application must be an iterable. (Strings are iterable too, of course, but iterating over a string and writing it out one character at a time is pretty slow.)

Save this code in a helloworld.py  and run it. Then open a browser and go to http://localhost:8000 . You should see “Hello, world!”. (If you don’t, check that you don’t have anything else running on port 8000.)

What the framework will look like

So far, the application is very limited. In your browser, go to http://localhost:800/helloworld/ . You will see “Hello, world!” again. As it stands, the application returns the same response for every path you try. It would be much nicer to be able to write something like this (MicroFramework is the framework class, which you will see later.):

You’ll learn about the route decorators in a moment. For the moment, just take a look at what is happening in the home and hello_world controller functions.

The controller functions are taking a Request  object as a parameter and returning a Response object. This is much cleaner than trying to take everything out of the WSGI environ  dictionary and then calling start_response , setting headers manually every time.

Here is the Request  class. It’s just a wrapper that extracts information from the environ dictionary and makes it accessible in a more convenient way:

The Request class

Most of the code in this class is just extracting keys from environ , but a few lines deserve a special mention. HTTP headers sent with the request are stored in environ in keys with the “HTTP_” prefix, so you can use a dict comprehension to extract them and store them in self.headers:

GET and POST data

Parsing the query string to extract GET and POST data is also interesting. The urlparse  module in the standard library contains a function called parse_qs  that takes a standard HTTP query string in the format ?key1=value1&key2=value2  and converts it into a dictionary that maps each key to a list of values. To extract GET data and store it in self.GET , you can call parse_qs(environ["QUERY_STRING"]) .

Extracting POST data is a bit more complicated as it is contained in the body of the HTTP request. First, you have to check if the HTTP method is POST, then read the content length, then get the POST query string by reading that number of bytes from the input stream. Finally, you call parse_qs  on the query string.

With the Request class in place, can instantiate a request object like so:

The Response class

What about Response ? It follows a similar principle:

As in the Request  class, most of the code here is just for convenience. You can assume that the default response code will be “200 OK” and allow the code to be set manually when it differs.

It is more natural to manipulate response headers as a dictionary, but they need to be passed to start_response  as a list of tuples, so the wsgi_headers  method, with the @property  decorator, returns them in that structure. You will see how this is used when you take a look at the __call__  method in the framework class.

Now you can build different types of responses by passing different keyword arguments. How about a normal “200 OK” response?

What about a “404 Not Found”?

Or what if an error occurs?

And what if you want to issue a redirect? It is as simple as setting the “302” status code and adding a location header.

The MicroFramework class

This is where it all comes together:

Look at the __call__  function. It gets back a instance of Response  by first building a  Request  object based on  environ  and passing that to the framework’s dispatch  method. Then it calls  start_response  with the status code and the headers from the response. Notice how wsgi_headers  is used.

Dispatching requests to controllers

The next interesting thing in the framework class is the dispatch  method. In the constructor, the  self.routes dictionary is initialized. It contains a mapping from regular expressions that represent request paths to controller functions. The method iterates through the regular expressions until it finds one that matches the request path, then it calls the associated controller function and returns the response from it to __call__ . If no route matches the path, it returns a “404” response generated by the not_found  static method.

If an error occurs while executing the controller function, the framework grabs the stack trace, prints it, and returns a “500” response generated by the internal_error  static method.

The route decorator

How do routes get into the self.routes  dictionary in the first place? That’s where the route  decorator comes in. All it does is add a mapping from the regex provided as an argument to the decorator to the controller function itself. The regexes can also contain capture groups that are stored in the request.params  list and made available to controller functions, as in the following example:

In this route regex, the capture group is a sequence of one or more numeric characters. The question mark (?) after the last slash makes the slash optional.

Integrating Jinja2

So far, the framework has no support for templating, but it is easy to integrate Jinja2 or any other template engine. Here is a simple example, which assumes that you have a template directory called “templates” with a file called “helloworld.html” in it.

This post showed you how to build a super simple web framework in Python, but this is just the beginning. For instance, the framework doesn’t support cookies or sessions, and there is no database access facility. Caching, form handling and other niceties that Django and Flask provide either out of the box or as plugins would also need to be added to turn this into a fully featured framework.

Why don’t you try to implement some of these features yourself?

How to validate web requests in Django using a shared secret

I was talking to one of the guys in my co-working space the other day. He’s learning Python and building a backend for a mobile app that sends JSON data to a web service endpoint he has built using Django. The issue he was running into was how to validate that the requests were actually coming from his app. This post explains how to do it with a shared secret.

Restricting access to views

Let’s say you have a Django view function that looks something like this.

Notice how you are using the require_http_methods  decorator to restrict the allowed HTTP methods to POST. If somebody makes a non-POST request to this view, Django will return a 405 status code.

Verifying requests using HMAC

You can use a HMAC (hash-based message authentication code) to make sure that (1) the sender of the message has the shared secret, and (2) that the message contents has not been tampered with.

There are two inputs to the HMAC function: the shared secret, or the “key”, and the message itself, which for us is the body of the HTTP request. The hash function is SHA256. The Base-64 encoded HMAC can then be sent as a HTTP header and used on the receiving end to verify the message.

In this case, the HMAC is sent in the “X-MY-APP-HMAC” header. The following piece of code extends the Django view function to validate the request, assuming the shared secret is stored in your Django settings file at myapp.settings.MY_APP_SHARED_SECRET .

The 401 HTTP status code is used when some authentication has not been provided, so you can use it here to tell the client that the request has been denied because of the invalid HMAC.

Wrapping it up in a decorator

The code above will work fine, but it’s a bit messy. What if you have multiple views that need to verify their requests? You would end up copying and pasting code. You could move the verification code out to a function and call it first thing, but a much better option is to turn it into a decorator.

Then you can simply decorate your view function like so:

Remember that decorators get executed from the inside out, so the require_http_methods  decorator will get called before validate_request_hmac .

That’s it. How to generate requests with the proper headers on the sender’s side depends on the platform. I’ll leave it as an exercise.

How to generate PDF reports with Jinja2 and PyQt

There are quite a few options for PDF generation in Python, but nothing fully open-source that ticks all the boxes. The Reportlab library is probably the most fully-featured solution available right now. Unfortunately, the open-source version doesn’t support templates, so unless you cough up for a license you’re going to be stuck manipulating the PDF format at a very low level.

Recently, I had a requirement to generate some simple PDF reports. I didn’t want to write lots of boilerplate and hoped I would be able to use some kind of templating. In the end I came up with a solution based on Jinja2 and Qt’s QWebView widget. First it renders the contents as HTML, then it uses the “Print as PDF” functionality of the QWebView  to save a PDF file. It’s a bit of a hack, but it gets the job done.

Here’s the main file, htmltopdf.py .

I also made a subdirectory for Jinja2 templates called, originally enough, templates. Inside it are two files, base.html  and report.html .

These templates are just examples. This is where you can get creative with CSS, etc. All I’ve done is provide a minimal scaffold so you can see what’s going on.

Let’s walk through htmltopdf.py  and figure out how it works.

The first thing we need to do is set up the Jinja2 environment. Although it is possible to instantiate a jinja2.Template  object by passing it a string holding the template text, instantiating the environment gives us access to template inheritance and other cool features. To set up the environment, we have to pass it a loader object. For our purposes we can use the basic package loader, which takes as arguments the name of the package (or the name of the file, in this case) and the template directory inside it. Our template directory is  templates .

Here is the function that we use to render a particular template. You can get the Template  object from the given template file name by calling get_template  on the environment. Once you have it, you  call render  on it, passing in the necessary information as keyword arguments.

Now let’s take a look at the print_pdf  function, which handles laying out the HTML from our render_template  function and printing it to a file.

We won’t be able to do much with the QWebView  unless we instantiate QApplication , so we bookend print_pdf  by constructing a QApplication  instance and finally calling exit  on it.

Next, we create the QWebView . Seeing as we already have the HTML we want to display in it, we can call setHtml  on the webview. If you want to load an external URL, for snapshotting web pages, etc., you have to use the QWebView.load  function to set its URL. In that case, you will need to register a signal handler to listen for the loadFinished()  signal, but seeing as we are just providing the HTML directly we don’t need to bother with that.

After we have injected the HTML into the webview, we get a QPrinter  and configure it to print an A4-size PDF document, by calling its  setPageSize , setOutputFormat  and setOutputFileName  methods. Other page sizes and output formats are also supported.

That covers everything novel in this approach. The main function just ties it all together and generates some sample data for the template. I found the loremipsum  package handy for quickly getting my hands on placeholder text.

Here’s what our generated PDF looks like:

rendered_pdf