A primer on Database Transactions and Asynchronous Requests in Django

Hello, Namaste.  Today we are going to  look at few Django web framework cookies that makes our life more sweeter.  Let us learn few things which helps us implement the functionality when situation demands. The topics are following:

  1. Implementing Database transactions in  Django
  2.  Making asynchronous HTTP requests from Django code

1) Django DB Transactions

I am creating a REST API. I want to insert POST data  into database. But here a list is received in POST. I want to validate each element in list and  make sure to insert data. Here there are two rules to say this insertion operation atomic.

  • Insert data if all elements pass the validation criteria.
  • While inserting, if  there is duplicate data then abort the transaction  and return integrity error.

Demo project will be available at https://github.com/narenaryan/trans-cookie

Let me create a sample Django project to illustrate all the things we are going to discuss. I am doing this on Ubuntu14.04 Machine with Python2.7, Django1.8 and MySQL

$ virtualenv cookie
$ source cookie/bin/activate
$ pip install django==1.8.5 python-mysqldb

Now let us create a sample project called cookie

$ django-admin startproject cookie

Here in cookie I am going to create a view which takes a list of numbers and if all numbers are primes then it will store those numbers in db. If invalid prime or any duplicate entry  it aborts the operation.

$ django-admin startapp primer

Now do the following to create a model called Prime in primer app.

# primer/models.py

from django.db import models
import re

class Prime(models.Model):
    number = models.IntegerField(unique=True)
    def __str__(self):
        return str(self.number)
    def prime_check(self):
        if re.match(r'^1?$|^(11+?)\1+$', '1' * self.number):
            raise Exception('Number is not prime')

Prime_check is the function we defined to validate data before inserting into db. Always validate your data using a model class method. Now go and modify settings.py to change database to MySQL. Add primer app to settings.py and run migrations

 # cookie/settings.py
...

INSTALLED_APPS = (
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'primer'
)

...
DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': 'cookie',
 'USER': 'root',
 'PASSWORD': 'passme',
 'HOST': 'localhost',
 'PORT': '3306',
 }
}
...
$ python manage.py makemigrations
$ python manage.py syncdb

Now MySQL tables User,Prime will be created. Now let us create a url and view that takes list of numbers as POST and inserts into db, if all are primes. Now modify primer/urls.py and primer/models.py as below:

# primer/urls.py
from django.conf.urls import include, url
from django.contrib import admin
from primer import views

urlpatterns = [
   url(r'^admin/', include(admin.site.urls)),
   url(r'^supply_primes/$', views.supply_primes, name="prime")
]
# primer/views.py
from django.shortcuts import render
from django.http import HttpResponse, JsonResponse
from django.views.decorators.csrf import csrf_exempt
from primer.models import Prime
import json 

# Create your views here.
@csrf_exempt
def supply_primes(request):
    if request.method == 'GET':
        return JsonResponse({'response':'prime numbers insert API'})
    if request.method == 'POST':
        primes = json.loads(request.body)['primes']
        #Validating data before inserting
        valid_prime = Prime()
        for number in primes:
            valid_prime.number = number
            try:
                valid_prime.prime_check()
            except Exception:
                message = {'error': {
                      'prime_number': 'The Prime number : %s \
                       is invalid.' % number}}
                return JsonResponse(message)
        return JsonResponse({"response":"data successfully stored"})

We can filter data before inserting anything. Integration error comes only when we insert data into db.

If we insert [11, 13, 15]  and next try to insert [14, 15, 13] then in the second case error will be returned while inserting 13 because it is duplicate.  But already 14, 15 are inserted. This is where transactions comes handy. Now we can modify code to

from django.shortcuts import render
from django.http import HttpResponse, JsonResponse
from django.views.decorators.csrf import csrf_exempt
from primer.models import Prime
from django.db import transaction,IntegrityError
import json 

# Create your views here.

@csrf_exempt
def supply_primes(request):
    if request.method == 'GET':
        return JsonResponse({'response':'prime numbers insert API'})
    if request.method == 'POST':
        primes = json.loads(request.body)['primes']
        #Validating data before inserting
        valid_prime = Prime()
        for number in primes:
            valid_prime.number = number
            try:
                valid_prime.prime_check()
            except Exception:
                message = {'error': {
                      'prime_number': 'The Prime number : %s \
                       is invalid.' % number}}
                return JsonResponse(message)
         #Carefully look for exceptions in real time at inserting
        transaction.set_autocommit(False)
        for number in primes:
            try:
                Prime(number=number).save()
            except IntegrityError:
            # We got error. undo all previous insertions
                transaction.rollback()
                message = {'error': {'prime_number': 'This prime number(%s) is already registered.' % number}}
                return JsonResponse(message)
         # If everything is fine, Commit the changes and flush db
        transaction.commit()
        return JsonResponse({"response":"data successfully stored"})

 

The three statements I used for transactions

  • transaction.set_autocommit(False)
  • transaction.rollback()
  • transaction.commit()

First statement tells remove auto pilot and make it manual . Let me chose whether to save something or not

Second statement tells rollback whatever changes I did until last commit

Third statement pinpoints that commit and flush to db.

These statements gives us full control of storage pattern in databases. Without transactions you have one single statement Prime(number=number).save(),  which directly push changes to database. If we need to put something into DB through our own logic then use transaction library in Django.

Let us see it in action

Run Django web  server as below

  $ python manage.py runserver 0.0.0.0:8200

It runs our Django project on localhost with PORT 8200

Let us use fire postman to make a POST request to http://localhost:8200/supply_primes . You can also use CURL.

Screenshot from 2015-10-26 16:46:51

It is showing that data is successfully stored. Because all are primes. If we see the data.

Screenshot from 2015-10-26 16:50:04

Now let me try to insert [26, 13, 17]. Because 26 is not prime it returns me following response.

Screenshot from 2015-10-26 16:51:01

cool. Then try to insert [29, 13, 67]. If you observe we are trying to insert duplicate.

Screenshot from 2015-10-26 16:52:42

and database looks like

Screenshot from 2015-10-26 16:53:38

Here 29 is not inserted. It is inserted actually but rolled back when 13 generates IntegrityError. This is how transactions work.

2) Asynchronous Requests from the Django Code

Think that your Django code base is too large and slow. Some one is asking you to insert a hook in the code which posts some data to external URL. Then your django behaves more slower. If you are making 100 sequential requests then the last hook is executed after a long time. critical code should not be blocked because of side players.

The solution to overcome this problem is to make asynchronous non-blocking requests from the Django code.

* Synchronous code

import requests
res = requests.get('http://localhost:8200/supply_primes')
# some other django task
counter += 1

Here counter will be incremented after a successful or erratic request made by second statement. It means blocking request is making django to pause until request is processed.

* Asynchronous code

If we are using Python3 we have a wonderful library called asyncio to make parallel HTTP requests. visit this diligent link if you are using Python3. http://geekgirl.io/concurrent-http-requests-with-python3-and-asyncio/  . If you are using your Django projects with Python 2.7.x then carry on.

$ pip install requests-futures

This is the library which makes parallel requests through requests library of Python.

from requests_futures.sessions import FuturesSession
session = FuturesSession()
res = session.get('http://localhost:8200/supply_primes')
# Some other django task
count += 1

Here session.get won’t block the increment of counter. So your django code speeds up. Always use this library wherever you want to spawn a separate process for making HTTP requests.

For more details visit demo project at https://github.com/narenaryan/trans-cookie 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s