Build a real time data push engine using Python and Rethinkdb


Namaste everyone.Today we are going to talk about building real time data push engines.How to design models for the modern realtime web will be the lime light point in this article. We are going to build a cool push enigne that notifies “Super Heroes” real time in the Justice League(DC). We can also develop real time chat applications very easily with same principles.

What actually is a Data Push Engine?

Push engine is nothing but a software piece that pushes notifications from the server to all the clients who subscribed to recieve these events.When your app polls for data, it becomes slow, unscalable, and cumbersome to maintain.In order to overcome this burden two proposals were made.

  1. Web Sockets
  2. Server Sent Events(SSE)

But using any one of the above technologies is not sufficient for modern real time web. Think it in this way. The query-response database access model works well on the web because it maps directly to HTTP’s request-response. However, modern marketplaces, streaming analytics apps, multiplayer games, and collaborative web and mobile apps require sending data directly to the client in realtime. For example, when a user changes the position of a button in a collaborative design app, the server has to notify other users that are simultaneously working on the same project. Web browsers support these use cases via WebSockets and long-lived HTTP connections, but relying on database to notify updates is cool.

Seeing is believing

I am going to run my project first to make you confident with it. Project is nothing but a website which does following. Code for this project is available at

  • I am going to start a Justice League website (like the one superman runs).
  • Website collects nickname and email of a SuperHero.
  • Notify all existing heroes about new joinees in real time.

So I am going to tell you a small story. Just click the first image and navigate to last one by one. Don’t forget to read description below in each  image!. Press Esc to exit from slide show.

I think you got something with above story.If you don’t let me explain. Here we are asking information from clients and navigating them to their dashboard. From then all clients who are on dashboard will be notified about newly joined people instantly. No refresh,No ajax polling. Thanks to our push engine.

Are you kidding ,I can implement that using web sockets?

Yes are right. You can purely implement the above notification system using websockets. But why I used few more things to do that. Here is the answer.

“Using websockets code for designing push logic  is cumbersome. Websocket code must do a push from server and recieve that in client . Traditional databases do not know about the websockets or Server Sent Events. There we need to poll the database changes and then push them to intermediate queue and from there to clients. I say remove that headache from our server. Just exploit database capability of pushing changes in realtime whenever a change occurs to it’s data. That is why I chose RethinkDB plus Websockets“.

How I build that Push engine

I used two main ingredients to create data push engine shown above.

  1. Python Tornado web server ( for handling websocket requests and responses)
  2. RethinkDB ( for storing data and also to push real time changes to the server)

What is RethinkDB?

According to RethinkDB official website

RethinkDB is the first open-source, scalable JSON database built from the ground up for the realtime web. It inverts the traditional database architecture by exposing an exciting new access model – instead of polling for changes, the developer can tell RethinkDB to continuously push updated query results to applications in realtime. RethinkDB’s realtime push architecture dramatically reduces the time and effort necessary to build scalable realtime apps.

When is RethinkDB a good choice?

RethinkDB is a great choice when your applications could benefit from realtime feeds to your data.

The query-response database access model works well on the web because it maps directly to HTTP’s request-response. However, modern applications require sending data directly to the client in realtime. Use cases where companies benefited from RethinkDB’s realtime push architecture include:

  • Collaborative web and mobile apps
  • Streaming analytics apps
  • Multiplayer games
  • Realtime marketplaces
  • Connected devices

We know that modern web demands falls in one of the above catagories.So RethinkDB is extremely useful for the people want to exploit it’s real power for building real time apps.

RethinkDB has a dedicated python driver.In our project we are just inserting our dicument and reading the changes on users table.For getting familiar with RethinkDB python client visit these links.

Setup for our data push engine

Install RethinkDB on Ubuntu14.04 in this way.

$ source /etc/lsb-release && echo "deb $DISTRIB_CODENAME main" | sudo tee /etc/apt/sources.list.d/rethinkdb.list
$ wget -qO- | sudo apt-key add -
$ sudo apt-get update && sudo apt-get install rethinkdb
$ sudo cp /etc/rethinkdb/default.conf.sample /etc/rethinkdb/instances.d/instance1.conf
$ sudo /etc/init.d/rethinkdb restart

Create virtualenv for the project and install required libraries

$ virtualenv rethink
$ source rethink/bin/activate
$ pip install tornado rethinkdb jinja2

Now everything is fine.My main applciation will be and there are templates and staticfiles in my project.The project structure looks like this.

|-- requirements.txt
|-- static
|  `-- js
|      `-- sockhand.js
`-- templates
|   `--detail.html

Now letus write our file.

#For tornado server stuff 

import tornado.ioloop
import tornado.web
import tornado.gen
import tornado.websocket
import tornado.httpserver
from tornado.concurrent import Future

from jinja2 import Environment, FileSystemLoader #For templating stuff

import rethinkdb as r #For db stuff

from rethinkdb.errors import RqlRuntimeError, RqlDriverError

from conf import * #Fetching db and table details here

#Load the template environment

template_env = Environment(loader=FileSystemLoader("templates"))

db_connection = r.connect(RDB_HOST,RDB_PORT) #Connecting to RethinkDB server

#Our superheroes who connects to server
subscribers = set() 

#This is just for cross-checking database and table exists 
def dbSetup():
    print PROJECT_DB,db_connection
        print 'Database setup completed.'
    except RqlRuntimeError:
            print 'Table creation completed'
            print 'Table already exists.Nothing to do'
        print 'App database already exists.Nothing to do'

#There is a loop type in python rethinkDB client.set it to tornado

class MainHandler(tornado.web.RequestHandler): #Class that renders details page and Dashbaord
    def get(self):
        detail_template = template_env.get_template("detail.html") #Loads tenplate
    def post(self):
        home_template = template_env.get_template("home.html")
        email = self.get_argument("email")
        name = self.get_argument("nickname")
        connection = r.connect(RDB_HOST, RDB_PORT, PROJECT_DB)
        #Thread the connection
        threaded_conn = yield connection
        result = r.table(PROJECT_TABLE).insert({ "name": name , "email" : email}, conflict="error").run(threaded_conn)
        print 'log: %s inserted successfully'%result

#Sends the new user joined alerts to all subscribers who subscribed
def send_user_alert():
    while True:
            temp_conn = yield r.connect(RDB_HOST,RDB_PORT,PROJECT_DB)
            feed = yield r.table("users").changes().run(temp_conn)
            while (yield feed.fetch_next()):
                new_user_alert = yield
                for subscriber in subscribers:

class WSocketHandler(tornado.websocket.WebSocketHandler): #Tornado Websocket Handler
    def check_origin(self, origin):
        return True

    def open(self):
        subscribers.add(self) #Join client to our league

    def on_close(self):
        if self in subscribers:
            subscribers.remove(self) #Remove client

if __name__ == "__main__":
    dbSetup() #Check DB and Tables were pre created
    #Define tornado application
    current_dir = os.path.dirname(os.path.abspath(__file__))
    static_folder = os.path.join(current_dir, 'static')
    tornado_app = tornado.web.Application([('/', MainHandler), #For Landing Page (r'/ws', WSocketHandler), #For Sockets
(r'/static/(.*)', tornado.web.StaticFileHandler, { 'path': static_folder }) #Define static folder 

    #Start the server
    server = tornado.httpserver.HTTPServer(tornado_app)
    server.listen(8000) #Bind port 8888 to server

I am going to define database configuration parameters like db_name, table_name etc in seperate file.

import os

RDB_HOST = os.environ.get('RDB_HOST') or 'localhost'
RDB_PORT = os.environ.get('RDB_PORT') or 28015
PROJECT_DB = 'userfeed'

That’s it. We had our and ready. I will explain what I did above in point-wise below.

  • importing tornado tools and rethinkDB client drivers
  • writing a function called db_setup that checks whether required database and table were created or not
  • using MainHandler class to handle http requests. For GET request displaying enter details page and for POST showing the dashboard.
  • WSocketHandler is the tornado websocket handler that adds or removes subscribers.
  • We have one method called send_user_alert . It is the actual pusher of changes to the client.It does only two things. “subscribing to database table change” . “sending those changes to client “

In rethinkdb we have a concept called change feeds. It is similar to Redis PUBSUB.We can subscibe to a particular change-feed and rethindb returns us a cursor which is of infinite length.Whenever db recieves a change in particular table it triggers event to that subscribed cursor with new and old values of data.For example.

#cursor is returned when we subscribe to changes on authors table
cursor = r.table("users").changes().run(connection)

#just loop through it infinitely to grab changes that RethinkDB push to cursor
for document in cursor:

I think you got the thing by now. The other files in our project are templates and static files

  • detail.html
  • home.html
  • sockhand.js

The code for templates is quite obvious. You can find templates here

But we need to look into js file

//function that listens to Socket and do something when notification comes
function listen() {
    var source = new WebSocket('ws://' + + '/ws');
    var parent = document.getElementById("mycol")
    source.onmessage = function(msg) {
              var message = JSON.parse(;
              //Return random color for superhero
              var child = document.createElement("DIV");
              child.className = 'ui red message';
              var text = message['new_val']['name'].toUpperCase() + ' joined the league on '+ Date(); 
              var content = document.createTextNode(text);
              return false;

    console.log('I am ready'); 

Here we are defining a listen function when webpage is loaded. That listen function initializes a variable called source which is of type WebSocket and links it to the /ws url that we defined in the Tornado application. It also sets a callback when a message is recieved and that callback code updates the DOM structure and adds information about new user.

If you are still confused ,then run  application yourselves and see the things. The app we wrote above is a data push engine that routes directly from database to client.  Go to this project link . clone it. Install requirements.txt .Then run visit localhost:8000. If you still have any queries on how it works then feel free to comment below or approach

I thought to introduce rethinkDB for absolute beginners but article becomes very lengthy then.Sure I will come up with an article dedicated for RethinkDB in near future.

In this way we can build a real time data push engine using python and Rethinkdb.

Points to ponder

  • Use rethinkDB for building real time applications.It is scalable too.
  • Use Tornado because it can easily handle concurrent connections without any fuss.
  • Remove queuing from your architecturaal design
  • Use websockets for bidirectional communication
  • Try out new things frequently



Build a Collatz conjecture solver with Python and ZeroMQ

Connecting computers is so difficult that software and services to do this is a multi-billion dollar business. So today we’re still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, Websockets. It remains painful, slow, hard to scale, and essentially centralized.

To fix the world, we needed to do two things. One, to solve the general problem of “how to connect any code to any code, anywhere”. Two, to wrap that up in the simplest possible building blocks that people could understand and use easily. It sounds ridiculously simple. And maybe it is. That’s kind of the whole point. Zero MQ comes to rescue us from the problem.With averge hardware configuration, we can handle 2.5-8 Million messages/second using ZeroMQ.

What is ZMQ?

ZeroMQ is a library used to implement messaging and communication systems between applications and processes – fast and asynchronously.It is faster like a bullet train.You can use it for multiple purposes.Like the things listed below.

* Networking and concurrency library

* Asynchronous messaging

* Brokerless communication

* Multiple transport

* Cross-platform and open-source

Why a message queue is required in the distributed applications and how ZeroMQ can be used as the best communication practise between applciations will be explained in few minutes.


I guess you captured the logic from above pic.Instead of hitting the server directly for each request  we can push it into a message queue and process it by workers , then route it to appropriate location. Ok, now context switch to collatz conjecture.

What is Collatz conjecture?

This is the 3n+1 mathematical problem (or Collatz conjecture). Collatz conjecture states that for any number n, the following function f(n) will always boil down to 1 as  result, if you keep feeding the previous result to the function over and over again.
  f(n) = {
           3n+1, if n is odd,
           n/2, if n is even
           1, if n is 1
Eg: if n = 20, then:
           f(20) = 20/2 = 10
           f(10) = 10/2 = 5
           f(5)  = 3*5+1 = 16
           f(16) = 16/2 = 8
           f(8)  = 8/2 = 4
           f(4)  = 4/2 = 2
           f(2)  = 1
The term cycle count refers to the length of the sequence of numbers generated. In the above case, cycle count for f(20) is 9.
We are going to build a collatz conjecture cycle finding server using Python and ZeroMQ. complete code of this project is available at this location

Beauty of collatz conjecture

All numbers leads to one. It is the philosophy of collatz conjecture. visit this site to visually see the construction of collatz numbers for orbital length of 18.

Let us build a Collatz Conjecture Cycle Server

Now come to the coding part.Our aim is to construct the server that takes a number from client and calculates longest collatz cycle from 1 to that number and returns it back. For ex. If we give input of 1000 our server should calculate collatz cycles for 1,2,3,……..,1000 seperately and return the longest cycle of all.


  • Python
  • ZeroMQ
  • Gevent

we can build the same server with Python and Gevent alone , but that setup is vulnerable after 10K connections. For scaling it to millions ,we should use the power of ZeroMQ.


  • Install  ZeroMQ4.1.2 ( ) .Below process shows step wise installation procedure.
$ sudo apt-get install uuid uuid-dev uuid-runtime
$ sudo apt-get install libzmq-dbg libzmq-dev libzmq1
$ sudo apt-get install build-essential gcc
$ cd tmp/ && wget
$ tar -xvf zeromq-4.1.2.tar.gz && cd ./zeromq-4.1.2.tar.gz
$ ./configure && make
$ sudo make install
$ sudo ldconfig
  • Install pyzmq
$ sudo apt-get install python-dev
$ sudo pip install pyzmq
  • Install gevent
$ sudo pip install gevent

Please be aware that ZeroMQ installation will fail if all deapendecies are not installed. pyzmq installation will fail if python-dev library deapendency is not fulfilled. I hope now you are ready with the required set up on a Ubuntu 14.04 machine.

This ZeroMQ server is used to serve requests through a TCP port to which a zmq socket is attached. Server collects data from that bound socket. let us first design function for returning maximum collatz cycle for a given inupt range. That algorithm would look like below

Collatz Conjecture Algorithm

import gevent
from gevent import monkey


#Algorithm for finding collatz conjecture longest cycle.
#Returns max cycle from cycles calculated from 1 to n.
def do_collatz(n):
    def collatz(n,cycle=''):
        while True:
            if n == 1:
                cycle += str(n)
                if n % 2 == 1:
                    n = ( 3 * n ) + 1
                    cycle += str(n)
                    n = n/2
                    cycle += str(n)
       return len(cycle)
    #This Gevent code is for speeding up the calculation of cycles
    jobs = [ gevent.spawn(collatz, x) for x in range(1,n) ]
    return max([g.value for g in jobs])

Now let us use this function in our ZeroMQ server that we are going to write below. It just recieves a number from client and calls this do_collatz function and returns the longest cycle back to the client.

ZeroMQ Collatz Server

import time
import zmq
import gevent
from gevent import monkey


#Create context
context = zmq.Context()

#Set type of socket
socket = context.socket(zmq.REP)

#Bind socket to port 5555

#Algorithm for finding collatz conjecture
def do_collatz(n):
    def collatz(n,cycle=''):
        while True:
            if n == 1:
                cycle += str(n)
                if n % 2 == 1:
                    n = ( 3 * n ) + 1
                    cycle += str(n)
                    n = n/2
                    cycle += str(n)
       return len(cycle)
    #This Gevent code is for speeding up the calculation of cycles 
    jobs = [ gevent.spawn(collatz, x) for x in range(1,n) ]
    return max([g.value for g in jobs])

#Create a loop and listen for clients to send requests 
while True:
    # Wait for next request from client
    number = int(socket.recv())
    print("Received request for finding max collatz cycle between 1.....%s" % number)
    # Send reply back collatz conjecture maximum cycle to client
    num = str(do_collatz(number))


Now we have a server ready to serve any no of clients.Let us build a ZeroMQ client to send request to above server and recieves the maximum collatz cycle for a given no.

ZeroMQ Collatz Client

import zmq
context = zmq.Context()

# Socket to talk to server
print 'Connecting to hello world server'

socket = context.socket(zmq.REQ)

number = raw_input("please give a no to calculate collatz conjecture max cycle: ")
print 'Sending request %s ' % number
#Send number to server
#Wait and print result
message = socket.recv()
print 'Collatz Conjecture max cycle of %s is <[ %s ]>' % (number, message)

That’s it. Our client is ready too. Now open the terminal with three tabs. One for server and other two for two clients to send the request. The output looks like this on my Ubuntu  14.04 machine.

Next try to give input 1000 from one client and 7000 from another client. Server instantly return maximum collatz cycle back to the client. It looks like this.


So it is clearly visible that our ZeroMQ server is working perfectly for serving the clients and solves collatz conjecture problem. This is called Request-Reply pattern of implementation of ZeroMQ. Here communication is acheived through TCP rather than HTTP. There are three more patterns can be implemented using ZeroMQ. They are:

  • Publish/Subscribe Pattern: Used for distributing data from a single process (e.g. publisher) to multiple recipients (e.g. subscribers).
  • Pipeline Pattern: Used for distributing data to connected nodes.
  • Exclusive Pair Pattern: Used for connecting two peers together, forming a pair.

So ZeroMQ has a lot of scope. It is a good scalability solution for current distributed application architectures. All code for above collatz-cycle is availlable at below github link.



Build an API under 30 lines of code with Python and Flask



Hello everyone. Now a days developers need to perform many jobs. Like web development, database development, API development and so on. Some companies are just having jobs called API developer on their openings sheet.What role APIs are playing now and why one should learn building them is our topic today. Developing an API with Python is a very easy task when compared to other languages. So,sit back and grab this skill for you. Take my words ,this skill is hot right now in the market.

What is a REST API?

REST (REpresentational State Transfer) is an architectural style, and an approach to communications that is often used in the development of Web services. The use of REST is often preferred over the more heavyweight SOAP (Simple Object Access Protocol) style because REST does not leverage as much bandwidth, which makes it a better fit for use over the Internet. The SOAP approach requires writing or using a provided server program (to serve data) and a client program (to request data).

In simple three lines REST API is a:

1 ) A way to expose your internal system to the outside world.

2) Programmatic way of interfacing third party systems.

3) Communication between different domains and technologies.

I think we are sounding technical. let us jump into practical things.By the end of this tutorial ,you will be comfortable in creating any API using Python and Flask.

Ingredients to build our API

We are going to use these things to build a running API.

*  Python

*  Flask web framework

*  Flask-RESTFul extension

*  SQLite3

* SQLAlchemy

Let us build Chicago employees salary API under 30 lines of code

I am going to build a Salary info API of Chicago city employees. Do you know ?,it is damn easy. An API can give you computation result or data from a remote database in a nice format. It is what API is intended for.API is a bridge between private databases and applications. I am collecting employee salary details from Chicago city data website

code of this entire project can be found at this link

Let’s begin the show……..

First , i downloaded the data-set as CSV and dumped it into my sqlite database.

$ sqlite3 salaries.db
sqlite> .mode csv salaries
sqlite> .import employee_chicago.csv salaries

and imported CSV.

Now we are going to build a flask app that serves this data as a REST API.

$ virtualenv rest-api
$ source rest-api/bin/activate
$ mkdir ~/rest-app
$ cd ~/rest-app

Now we are in the main folder of app.Create a file called in that folder.We need few libraries to finish our task.Install them by typing below commands.

$ pip install flask
$ pip install flask-restful
$ pip install sqlalchemy

That’s it. We are ready to build a cool salary API that can even be accessed through mobile. Let us recall the REST API design .It has 4 options. GET,PUT,POST,DELETE



here we are  dealing with an open data which can be accessed by multiple applications. So we implement GET here and remaining REST options becomes quite obvious.


from flask import Flask, request
from flask_restful import Resource, Api
from sqlalchemy import create_engine
from json import dumps

#Create a engine for connecting to SQLite3.
#Assuming salaries.db is in your app root folder

e = create_engine('sqlite:///salaries.db')

app = Flask(__name__)
api = Api(app)

class Departments_Meta(Resource):
    def get(self):
        #Connect to databse
        conn = e.connect()
        #Perform query and return JSON data
        query = conn.execute("select distinct DEPARTMENT from salaries")
        return {'departments': [i[0] for i in query.cursor.fetchall()]}

class Departmental_Salary(Resource):
    def get(self, department_name):
        conn = e.connect()
        query = conn.execute("select * from salaries where Department='%s'"%department_name.upper())
        #Query the result and get cursor.Dumping that data to a JSON is looked by extension
        result = {'data': [dict(zip(tuple (query.keys()) ,i)) for i in query.cursor]}
        return result
        #We can have PUT,DELETE,POST here. But in our API GET implementation is sufficient
api.add_resource(Departmental_Salary, '/dept/<string:department_name>')
api.add_resource(Departments_Meta, '/departments')

if __name__ == '__main__':

save it as and run as

 $ python 

That’s it. Your salary API is up and running now on localhost , port 5000. There are two rules we defined in the API. One is to get details of all departments available and second is to get employee full detail, who is working in a particular department.

So now go to


and you will find this.


See how flask is serving database data into JSON through the REST API we defined. Next modify URL to peek all employees who are working in Police department.



Oh man, seems like police officers are well paid in Chicago but they can’t beat a Django or Python developer who earns $ 1,00,000 per annum. just kidding.

My code walk-through is as follows

*  I downloaded latest Salary dataset from chicago data site

*  Dumped that CSV  into my SQLite db.

*  Used SQLAlchemy to connect to database and do select operations.

*  Created Flask-Restful classes to map functions with API URL

*  Returned the queried data as JSON ,which can be used universally.

See how simple it is to create a data API. We can also add support to PUT,POST and DELETE on data too.We can also have an authentication system for fetching data through API. Python and Flask are very powerful tools to create API rapidly. GitHub link is given below. Give it a try and extend it with the things mentioned above.

See you soon with more stories.

Docker , the future of Virtualization for your Django web development




Always use a virtual environment for your software development

Hello friends. I am a DevOps engineer who deals with development , production , monitoring , configuration management etc of a Software. But for every developer , there will be few  house keeping things which are quite irritating. You are having a single Ubuntu system and for the project’s sake , you will install many database servers and other applications in it. The dependencies for your applications will cause trouble , when  you are installing both your favorite games and working projects in a single system.

Why Docker?

What if something went wrong in your working project and everything in your system is messed up.  So I always  suggest you to separate your work with personal things. In this tutorial, I am going to show you how a developer can run his/her project and its applications in a light-weight virtual environment called container. This container is created by the Docker . We can access docker container from the host system for code editing and view execution. So always use a virtual environment for your software development.

What is Docker?

  • Open platform for developers and sysadmins to build, ship, and run distributed applications.
  • Light weight containers
  • It’s fast , not like a VM
  • Minimal resource usage
  • Run thousands of containers
  • Easy to run your whole production stack locally

Docker can create 100 light-weight environments in a single laptop. Unlike a virtual machine , your docker container will launch in one second. It will give an isolated environment for a developer to work with. In this post , I am going to create a docker container and setup a Python Django project and push it to my cloud repository.

Docker achieves its robust application (and therefore, process and resource) containment via Linux Containers (e.g. namespaces and other kernel features). Its further capabilities come from a project’s own parts and components, which extract all the complexity of working with lower-level linux tools/APIs used for system and application management with regards to securely containing processes.

Main Docker Parts

  1. docker daemon: used to manage docker (LXC) containers on the host it runs
  2. docker CLI: used to command and communicate with the docker daemon
  3. docker image index: a repository (public or private) for docker images

Main Docker Elements

  1. docker containers: directories containing everything-your-application
  2. docker images: snapshots of containers or base OS (e.g. Ubuntu) images
  3. Dockerfiles: scripts automating the building process of images

Installation Instructions for Ubuntu 14.04


$ sudo apt-get upgrade
$ sudo apt-get install
$ sudo service start
$ sudo ln -sf /usr/bin/ /usr/local/bin/docker
$ sudo sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/


Using the Docker

Now we use Ubuntu as our base image for creating containers, then modify a freshly created container and commit it.

$ sudo docker pull ubuntu

this pulls the base ubuntu image to your system. After a successful pull we can launch a new container using following command.

$ sudo docker run -i -t -p ubuntu 

This creates a new container with container port 8000 forwarded to the host port 8000 . The flags mean this:

-i        attaches stdin and stdout

-t     allocates a terminal or console

-p     forwards ports from container to host

Things a developer needs

1) Edit Project code from host in your favorite editor.

2) Run project in that virtual container environment

3) See the output through the port that forwarded.

So in-order to edit code you need to access container data in your host. Instead you just mount a host directory in your docker container. Then your project code is in host ,but runs in the container. Cool right?.

So we will modify the command for running the new container as this.

$ sudo docker run -i -t -p -v /home/hproject:/home/cproject ubuntu 

Now a docker container will be created and started  with following things.

* port is forwarded on localhost:8000

* Data volume “/home/hproject” of host is mounted as “/home/cproject” in the container. It means the files lies in the /home/hproject of our host system is accessible perfectly from the container at /home/cproject.

These two things are required by a Django developer because , he need to modify code and view output through the browser. Now he don’t care where code is running. But here code is running in isolation. That too in a light-weight docker container. Vagrant has same port forwarding and volume mounting strategy but Vagrant is a VM , Docker is a VE.

Play with the container

Now we have a container started . It looks like bash with # symbol. So now install following things in it.

* Python


* Virtualenv

* Git

* Setup Django ,MySql and your Project

That’s it. You can install any thing that required to the project.But remember after doing stuff , use exit to come out of container.

# exit

Now if you do not commit the container, all changes you made will be lost. So commit the container first. Before that find out the containerID by typing this command.

$ sudo docker ps -a 

and find out the latest container exited. You can give a name to a container while committing

$ sudo docker commit b03e4fb46172 mycontainer 

This will commit the latest container we played till now and also gives it a name mycontainer. If once again we wish to launch container for working with our project , then use following commands.

$ sudo docker start mycontainer
$ sudo docker attach mycontainer

That’s it. You will enter into a virtual container where you can run your Django project. If you plan to remove a container just do

$ sudo docker rm mycontainer

If you wish to push your container to the cloud , you should use the Docker hub registry. Don’t forget to commit after coming out of container. You can carry your entire project with environment anywhere. Package your Project (Django ,Flask) + Environ (MySql,PostgreSQL,Redis) to a tar file and export it to any place. That is the magic of Docker. For doing that just export a container to a TAR file.

Exporting the containers

$ sudo docker export mycontainer> /home/mycontainer.tar

Instead you can also save images and carry them over system through FTP and then load it in target node.

Loading and saving the images

$ sudo docker save mycontainer > /tmp/mycontainer.tar

and then load it in target host as

$ sudo docker load mycontainer < /tmp/mycontainer.tar

References for dockerizing your mind




Django with Ajax, a modern client-server communication practise



We all know that Django is a powerful framework. Many of us deals with posting data from a HTML form to a Django view and process it. You should have never cared of Ajax way of posting the data. Some of you tried AJAX but faced  few painful intros to setup a perfectly running Django-Ajax implementation. So,I came up this time to tell you how to do that.After this tutorial you will feel quite comfortable about implementing ajax posts and also will keep this article as a reference for future. Source code for upcoming tutorial available at following link.


What is AJAX?

Simply put AJAX is about updating parts of a web page, without reloading the whole page.

Suppose we need to update any parts of our website page ,then we must use Ajax. For posting data back to server ajax is required. Now why we should use it with Django ?. I believe following things will answer you.

  •  Using Ajax, we can craft data format that sent back to server.
  •  We can handle the situations like successful post or failure or error.
  • Drop all burden on client rather than server wherever it is possible.
  • For implementing progress bars ajax post is the best way.
  • Drop messages framework to display something ,I hate it. Use JSON response returned from Ajax post to wake up cool notification models etc.

If you didn’t understand the above points , cool. I too didn’t once. The simple thing I am gonna show is , how to post something back to server from HTML and collect acknowledgement that returned back.

Let us begin the show

We are going to see the demonstration from scratch. Let us create a Django project for that. Remember we am playing with Django >= 1.7 here.

$ pip install Django==1.7.3

This installs Django1.7 on your system.

$ django-admin startproject ajaxpost
#Now create an app called mainapp inside ajaxpost
$ django-admin startapp mainapp

Then the directory structure will be in this way.



Now let us add the superuser to begin with. Default db will be SQilte3. First do a migration for django to create default user tables in db. So do this.

$ python migrate

Now create a super user for admin.

$ python createsuperuser

That’s it. Now we are going to create a home page which has the email field and a password for collecting user data. It is to show data passage from ( client -> server ,server -> client) . Simple,but extremely useful in 100 use-cases.

After creation of superuser ,just create a view for our home page. ajaxpost/ looks like below


from django.contrib import admin
from mainapp import views

urlpatterns = patterns('',
 url(r'^admin/', include(,
 url(r'^',views.home , name=u'home')

Now modify the views of mainapp to the following.

from django.shortcuts import render
from django.http import JsonResponse

def home(request):
     return JsonResponse({"hello":"pythonist"})

Now just save and do a runserver. Next Go to http://localhost:8000 and you will find this thing.

$ python runserver


See how simple it is to send a Json response from a Django1.7 view. Now we are going to crete a HTML file for our home page and try to take data from it to create user. For that create a folder called templates in your project root and make a file called base.html. tree structure of our project looks this.


Ok. Now I am adding  template directories location to Add this line to ajaxpost/

        BASE_DIR + '/templates/',

Now Django can find template files. So add this HTML content to templates/base.html

          <link href="" rel="stylesheet">
          <link href="" rel="stylesheet">


 <div class="container">

 <form class="form-signin" method="POST">
    {% csrf_token %}
      <h2 class="form-signin-heading">Enter Details</h2>
     <label for="inputEmail" class="sr-only">Email address</label>
     <input type="email" id="inputEmail" class="form-control" placeholder="Email               address" required autofocus>
     <label for="inputPassword" class="sr-only">Password</label>
     <input type="password" id="inputPassword" class="form-control"  placeholder="Password" required>
     <button id="submit" class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>


 <script src=""></script>


When you fire runserver and go to http://localhost:8000 , you will find a nice bootstrap Enter details form like this.

I hope things are clear. Follow from the top , if you are in doubt.

Now comes the Ajax part

Instead of doing a normal submit , let us have an Ajax post block in our HTML file. Django relies on csrf token for the validity of data posted to the server. So in addition to data , we should send csrf-token. In default django submission,it is automatically sent. But in case of Ajax , we need to take care of that.

Take the below JS code as Granted and place it in  a new <script></script> tag in base.html


//For getting CSRF token
function getCookie(name) {
          var cookieValue = null;
          if (document.cookie && document.cookie != '') {
                var cookies = document.cookie.split(';');
          for (var i = 0; i < cookies.length; i++) {
               var cookie = jQuery.trim(cookies[i]);
          // Does this cookie string begin with the name we want?
          if (cookie.substring(0, name.length + 1) == (name + '=')) {
            cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
 return cookieValue;

//For doing AJAX post

//When submit is clicked
 $("#submit").click(function(e) {

//Prevent default submit. Must for Ajax post.Beginner's pit.

//Prepare csrf token
 var csrftoken = getCookie('csrftoken');

//Collect data from fields
 var email = $('#inputEmail').val();
 var password = $('#inputPassword').val();

//This is the Ajax post.Observe carefully. It is nothing but details of where_to_post,what_to_post
//Send data  
       url : window.location.href, // the endpoint,commonly same url
       type : "POST", // http method
       data : { csrfmiddlewaretoken : csrftoken, 
       email : email,
       password : password
 }, // data sent with the post request

 // handle a successful response
 success : function(json) {
 console.log(json); // another sanity check
 //On success show the data posted to server as a message
 alert('Hi   '+json['email'] +'!.' + '  You have entered password:'+ json['password']);

 // handle a non-successful response
 error : function(xhr,errmsg,err) {
 console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console
 Now Don’t forget to place {% csrf_token %} next to the form tag in base.html. Let us modify the mainapp/
from django.shortcuts import render
from django.http import JsonResponse

def home(request):
    if request.method == 'POST':
        #POST goes here . is_ajax is must to capture ajax requests. Beginner's pit.
        if request.is_ajax():
            #Always use get on request.POST. Correct way of querying a QueryDict.
            email = request.POST.get('email')
            password = request.POST.get('password')
            data = {"email":email , "password" : password}
            #Returning same data back to browser.It is not possible with Normal submit
            return JsonResponse(data)
    #Get goes here
    return render(request,'base.html')
Now everything for demonstration is built. We need to wrap up things and do a runserver. Before that complete templates/base.html file looks this way.
 <link href="" rel="stylesheet">
 <link href="" rel="stylesheet">


 <div class="container">

 <form class="form-signin" method="POST">
      {% csrf_token %}
      <h2 class="form-signin-heading">Enter Details</h2>
      <label for="inputEmail" class="sr-only">Email address</label>
      <input type="email" id="inputEmail" class="form-control" placeholder="Email address" required autofocus>
      <label for="inputPassword" class="sr-only">Password</label>
      <input type="password" id="inputPassword" class="form-control" placeholder="Password" required>
      <button id="submit" class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>


 <script src=""></script>

//For getting CSRF token
function getCookie(name) {
       var cookieValue = null;
       if (document.cookie && document.cookie != '') {
         var cookies = document.cookie.split(';');
         for (var i = 0; i < cookies.length; i++) {
         var cookie = jQuery.trim(cookies[i]);
         // Does this cookie string begin with the name we want?
         if (cookie.substring(0, name.length + 1) == (name + '=')) {
             cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
 return cookieValue;

//For doing AJAX post
 $("#submit").click(function(e) {


 var csrftoken = getCookie('csrftoken');

 var email = $('#inputEmail').val();

 var password = $('#inputPassword').val();

//This is the Ajax post.Observe carefully. It is nothing but details of where_to_post,what_to_post
         url : window.location.href, // the endpoint,commonly same url
         type : "POST", // http method
         data : { csrfmiddlewaretoken : csrftoken, 
         email : email,
         password : password
 }, // data sent with the post request

 // handle a successful response
 success : function(json) {
      console.log(json); // another sanity check
      //On success show the data posted to server as a message
      alert('Hi '+json['email'] +'!.' + ' You have entered password:'+      json['password']);

 // handle a non-successful response
 error : function(xhr,errmsg,err) {
 console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console


Ok. Now fire up the server using
$ python runserver
Now I went to http://localhost:8000 and entered these details.
email : pikachu@pokemon
password: charizard
When I click sign in this is what I got.
Hurray. We got it. Data is passed from client to server using Ajax and then server sent same data to client as JSON to display alert message.
This is how we pass information to the server using Ajax . Ajax is mainly used for long polling. Long polling is process in which, Ajax sends request from client to server asking the latest information.Then it updates the contents of HTML with data obtained. Here, we  stressed four things. In case of failure of any one, Ajax post may not work.Those things are listed below.
  • {% csrf_token %} just below form tag.
  • csrfmiddlewaretoken in Ajax POST data.
  • is_ajax() handler in Django view.
  • get(‘parameter’) method on request.POST

What is new in Django 1.7?

 In Django 1.7 ,we have JsonResponse which directly encodes a Python dictionary to JSON object without dumping data .
from django.http import JsonResponse

#Usage in view
return JsonResponse({"foo":"bar"})
 Before you used to do this like
import json
return HttpResponse(json.dumps({"foo":"bar"}), content_type="applciation/json")


I hope you got something from this tutorial. I faced problems when I started doing Ajax posts a year back. All tutorials on web misses one or more important things , so things won’t work. That gives us great frustration. I here showed a simple, working example of Ajax post in Django. Use it. source code is available at this link.

Python and Riak DB, a perfect couple for huge scale distributed computing


In this article I am going to show how to use Riak NoSQL database with Python . It been long since I wrote my last post. But this time I am going to introduce a new NoSQL Database which is powerful yet simple,open-source .  ( Riak Database )

What is Riak DB and for god sake why I should learn it ?

Let me tell you a small story . We all know that Amazon is an  E-commerce giant and you some times may be wondering ,how they are handling millions of transactions per second and accuracy is unquestioned . Amazon uses Dynamo DB for all its operations ,some of you have used Amazon S3 service and familiar with concepts like buckets and keys.So it is actually a custom DB developed and used by Amazon . It uses a Dynamo DB database and provides a REST API  access to the users who signs up ,and that setup Amazon call it “Amazon S3 service”.

What Riak to do with Dynamo DB?

Riak DB is created by Basho technologies as an open-source product . It is inspired from Dynamo DB and allow you to enjoy all the features and also additional features than Dynamo DB.If Amazon can do the miracles why can’t you? You can with Riak DB.The structure of Riak DB is almost similar to Dynamo.

In February last year, Facebook acquired the company WhatsApp for $19 billion. WhatsApp serves over 465 million users and there are over 40 billion messages sent in and 19 billion messages sent out every day. Although the company has scaled drastically, they have maintained an uptime of 99%. How have they managed these feats? As WhatsApp engineer, Rick Reed mentioned at Erlang Factory, Erlang is their “secret sauce,” and that has put the language in the spotlight. Riak is built upon Erlang where MongoDB on C++ , Cassandra on Java. That is why Riak is special.

Now what is Riak DB ?

Riak is an open source, distributed database. Riak is architected for:

  • Low-Latency: Riak is designed to store data and serve requests predictably and quickly, even during peak times.
  • Availability: Riak replicates and retrieves data intelligently, making it available for read and write operations even in failure conditions.
  • Fault-Tolerance: Riak is fault-tolerant so you can lose access to nodes due to network partition or hardware failure and never lose data.
  • Operational Simplicity: Riak allows you to add machines to the cluster easily, without a large operational burden.
  • Scalability: Riak automatically distributes data around the cluster and yields a near-linear performance increase as capacity is added.

Riak DB Architecture

Riak uses ring structure for handling keys of multiple nodes


Ok .Leave all the technical things behind construction of DB. I assume you are  a developer who use SQL databases or NoSQL ones like MongoDB,CouchDB. GIve a try to Riak because many big heads like Best Buy  are using Riak comfortably from past few years.

When Riak DB is to be used ?

No database serves all needs. If you are painful while upgrading architecture,and you lose customers or money at down time of server then you should chose Riak because it’s fault-tolerance is supreme. This picture shows where to go.Click on it ,if not visible.



How to install Riak on Linux precisely Ubuntu?

Here we are going to see the installation steps for Riak.You may be guessing it is painful as Cassandra . Nope . It is dead easy to install Riak. Won’t you believe? For other linux installations and Mac OSX visit this link . Linux and Mac Installation Riak

For Ubuntu 12.04+ 64-bit systems follow me now.

First install dependencies for Riak

$ sudo apt-get install libpam0g-dev libssl0.9.8 
$ sudo apt-get install python-dev libffi-dev libssl-dev
 Next use this link to download latest version of Riak.
$ wget
Ok. If your download is too slow,then download from a mirror
then you will get a file called riak_2.0.5-1_amd64.deb in downloaded folder.
Next go to that folder in Linux shell and type the command
$ sudo dpkg -i riak_2.0.5-1_amd64.deb 
 Now Riak will be installed.Start it using command
$ sudo riak start

That’s it a Riak server is running on your system now. See how easy! . For checking whether server is running or not type command

$ sudo riak ping
You will be replied with “pong” message. hurray you installed Riak

How Python can access Riak ?

There is a very good Python client written by Basho for Riak. Using that we can do all CRUD operations on  the database.In Riak,there is a different terminology similar to S3.


  • Bucket  === Table
  • Key-Value === Records


so we are going to store key-value pairs in the buckets. Buckets are name spaces those separate keys. I am going to show you how this can be contrasted with MongoDB documents.

Installing Riak-Python-client

Use pip or easy install command

 $ pip install riak
That’s it . I told you ,it is very simple to setup Riak and jump-start it with Python.

Using Python-client

Let us create buckets,store data in them and next retrieve.

from riak import RiakClient, RiakNode

#All are equivalent , leaving empty means connect to Riak on localhost
RiakClient(protocol='http', host='', http_port=8098)
nodes are the remote systems you are running instances of Riak Servers . When your data size is increasing grow your data horizontally by allocating more nodes to cluster. But for now we are just tweaking single server .So this is required
from riak import RiakClient
client  = RiakClient()
That’s it . We got a client with a connection for Riak. We can create ,delete,modify buckets using that client.Let me create a bucket (table in SQL) called users.

Creating and selecting a bucket

myBucket = myClient.bucket('users')
Remember if bucket already exists it will be selected,else dynamically created instantly.
now myBucket is the bucket for users.I will Insert data into it.

Inserting JSON(or any data) into the bucket

Let us think we got data for insertion from a web form or a product catalog.Normally we store that kind of data MongoDB . I am creating dummy records to insert into Riak.
user1 = {
'name' : 'Naren',
'location' : 'Hyderabad',
'role' : 'Programmer',
'fulname' : ['Naren','Arya','Yellavula']

user2 = {
'name' : 'Saikiran',
'location' : 'Hyderabad',
'role' : 'Dentist',
'fulname' : ['Sai','Kiran','Yellavula']
All python valid data-types  can be inserted into bucket of Riak.
Now we insert those dictionaries into bucket myBucket
obj ='user1',data=user1)
It stores user1 dictionary with key ‘user1’ and user1 information as data. If we wan’t to generate a random key , then use
import uuid
#Keys should be of string type
obj =,data=user1)
or if you skip giving a key ,Riak generates  an arbitrary key and stores value with that key in bucket. Similarly we can add the second user
obj ='user2',data=user2)

Retrieving  data from a bucket

We written data to Riak successfully,now to get it back. get_keys method of bucket returns all keys from that bucket.Using those you can traverse each and every record i.e key=>value pair. use get method  on a bucket to return a record.get takes key of that record.

#Gives <riak.riak_object.RiakObject at 0x7f91607aeb90>
result = myBucket.get('user1')
RiakObject has a property called data where you can see all details of user1. If you need to get keys then use get_keys on bucket.It returns all keys as a list.
#Gives all keys in a bucket
keys = myBucket.get_keys()
Now print roles of all users
for key in myBucket.get_keys():
    print myBucket.get(key).data['role']

#This prints 
I think you got a basic Idea of inserting data into a Riak bucket and also retrieving them.If you are a mongo developer , you got theme by now. There are tons of options for queries and searches. I will discuss them in my next post. First be comfortable with setup and basic CRUD operations as explained above. My next post will be building a REST API with Flask and Riak setup. Don’t miss it, Riak is more powerful than you think.For more details visit these:

Implementing activity streams in Django web framework




What is an activity stream?

It is a sequential chain of all activities done by the users in a website.Like Twitter showing our own feeds ,listing activities of developers on Gitlab dashboard or showing activities of connections on LinkedIn.Whatever you may call it,It is an activity stream.

For ex:

This is the activity stream for some company dashboard.See the smiling faces of company members and their activity beside on their dashboard.If you are a django developer and trying to create similar activity streams,then this post is for you.This post is for everyone who wants to learn how to create LinkedIn like realtime dashboard using powerful Django 1.7 web  framework.

How we can implement activity streams?

Actually there are few advanced concepts like signals and handlers in Django using which we can notify anybody at a particular time i.e when user posts a comment ,send it to email etc.Using that basic functionality of signals ,we should work hard to create dashboard activity streams.Fortunately a young Django developer called Justin Quick wrote a quick usable Django plug-in called django-activity-stream to lift the burden from our shoulders in creating activity streams  with Django in python.Documentation is pretty good but beginners may confuse about it’s usage.Here is the link to that Django plugin library.

What are the requirements?

*  Python2.7

* Django 1.7> (In lower versions,we need to look into many more things,so install safely 1.7 in virtualenv )

* django-activity-stream

Use the below commands to install Django1.7 in a virtual environment

$ pip install virtualenv
$ virtualenv djangoenv

First command install the virtual environment tool and second command creates a virtual environment called djangoenv,in which we are going to install our Django,and its related libraries.Now only environment is created.In order to enter into that environment we need to activate it.

$ source djangoenv/bin/activate

then you will find change in the shell symbol to

(djangoenv)$ ls -l

Now you are in a virtual environment that has a separate copy of python interpreter and fresh libraries to work upon.

Note: In virtual environment use only pip install command instead of sudo pip.

Now let us install django 1.7 ,django-activity-stream

$ pip install Django==1.7.1
$ pip install django-activity-stream

Now everything is ready.

I am creating a new project called “tutorial” and creating an app called “dashboard”.


(djangoenv)$ startproject tutorial

(djangoenv)$ cd tutorial

(djangoenv)$ startapp dashboard

Now your directory structure should be app inside project directory.I also hope that you have chosen your own back-end.I am continuing with MySQL .First there will not be any users.Create database and then use following commands for creating tables.

(djangoenv)$ ./ makemigrations

(djangoenv)$ ./ migrate

makemigrations command scans all your models in apps.So you have a default User model in django.contrib.auth.models.After typing those commands your User table will be created.Now create super user to quick insert data into django models.

(djangoenv)$ createsuperuser

Now after creating the superuser login and populate the users.For me I created 3 users namely,

* naren

* sriman

* manoj


I am going to make users as supervisors.So I create supervisors model and tasks model in my dashboard app.Each supervisor can be allocated with many tasks in hand.

my file looks like

from django.db import models
from django.contrib.auth.models import User
# Create your models here.
class Task(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField(null=True,blank=True)

class Supervisor(models.Model):
    user = models.ForeignKey(User,null=True,blank=True,related_name="supervisor")
    task = models.ManyToManyField(Task,related_name='tasks')

I use foreign key to access user name and one supervisor can have many tasks.So it is a many to many field.

Setting up django-activity-stream?

Your should consists of following things.Add actstream to list of apps.Add that ACTSTREAM_SETTINGS file to your file.
    'MANAGER': 'dashboard.managers.MyActionManager',
    'USE_PREFETCH': True,
    'USE_JSONFIELD': False,

Now you need to create three files in order to proceed further.In dashboard craete,,

from datetime import datetime
from django.contrib.contenttypes.models import ContentType
from actstream.managers import ActionManager, stream

class MyActionManager(ActionManager):
    def mystream(self, obj, verb='posted', time=None):
        if time is None:
            time =
        return obj.actor_actions.filter(verb = verb, timestamp__lte = time)

This file is the time stamp manager that creates and returns action generated time.You can write your own logic here if you wish to return other parameters than time.
from django.apps import AppConfig
from actstream import registry
from django.contrib.auth.models import User

class MyAppConfig(AppConfig):
    name = 'dashboard'
    def ready(self):

This registers the models to the action manager.In order to use activity stream we need to register our models first.So I am registering my User model with dashboard models Task,Supervisor.

Next we need to add one line to of dashboard app.These all steps are followed to use activity-stream

default_app_config = 'dashboard.apps.MyAppConfig'

By this we link the AppConfiguration to activity stream’s MyAppConfig.

Now everything is finished and we are ready to go.I am going to create tasks in the shell to concentrate more on creation of activity streams.



I created three tasks and these can be assigned to any supervisor.I am going to assign tasks to supervisors and at the same time activity will be recorded by activity stream.Normally in our projects assigning task to a user will lie in a view.So you need to send an action signal at the time of creation of assignment action.action_handler in actstream.actions handles the storing of these actions.


Using  action method we send a signal with actor,verb and target as parameters.These actor,target must be registered before using them.That is what we did above in file .

actor – Source object which is performing action. It is like subject in an English statement.

verb – Verb is the description of the action.

target – Target object is the thing upon action is performed. It is like object in an Enlgish statement.

Now i am going to assign planning task to supervisor naren and instantly sends that signal.


Just see the output 31.It returned me the action object that created.It will be stored in the database.

Let us give tasks for two more Supervisors.




Remeber these assignments can occur anywhere in your views at anytime,but actions are stored in a common queue like object based on the time they created. we can fetch actions from that  and list them on the dashboard.

Action is the model that stores all actions generated.Who created model for that ?. django-activity-stream takes care of that.Now we need to list all the occured actions on a page.Just create the dash board view and display all actions in it.

See how the time stamps and all perfectly managed.Just display them on a view.your dash board will be finished.Create more combinations of actions as required and also use full power of django-activity-streams by tweaking all options available

We have tons of options for django-activity-stream like

* fetching actor_stream (actor actions)

* fetching target_stream (target actions)

* fethcing user_stream ( all actions associated with a user)

* following/unfollow  certain stream

The full code for the project is available with sqlite db in this link

:) hope you enjoyed the article.