381292

Build a Collatz conjecture solver with Python and ZeroMQ

Connecting computers is so difficult that software and services to do this is a multi-billion dollar business. So today we’re still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, Websockets. It remains painful, slow, hard to scale, and essentially centralized.

To fix the world, we needed to do two things. One, to solve the general problem of “how to connect any code to any code, anywhere”. Two, to wrap that up in the simplest possible building blocks that people could understand and use easily. It sounds ridiculously simple. And maybe it is. That’s kind of the whole point. Zero MQ comes to rescue us from the problem.With averge hardware configuration, we can handle 2.5-8 Million messages/second using ZeroMQ.

What is ZMQ?

ZeroMQ is a library used to implement messaging and communication systems between applications and processes – fast and asynchronously.It is faster like a bullet train.You can use it for multiple purposes.Like the things listed below.

* Networking and concurrency library

* Asynchronous messaging

* Brokerless communication

* Multiple transport

* Cross-platform and open-source

Why a message queue is required in the distributed applications and how ZeroMQ can be used as the best communication practise between applciations will be explained in few minutes.

Selection_001

I guess you captured the logic from above pic.Instead of hitting the server directly for each request  we can push it into a message queue and process it by workers , then route it to appropriate location. Ok, now context switch to collatz conjecture.

What is Collatz conjecture?

This is the 3n+1 mathematical problem (or Collatz conjecture). Collatz conjecture states that for any number n, the following function f(n) will always boil down to 1 as  result, if you keep feeding the previous result to the function over and over again.
  f(n) = {
           3n+1, if n is odd,
           n/2, if n is even
           1, if n is 1
        }
Eg: if n = 20, then:
           f(20) = 20/2 = 10
           f(10) = 10/2 = 5
           f(5)  = 3*5+1 = 16
           f(16) = 16/2 = 8
           f(8)  = 8/2 = 4
           f(4)  = 4/2 = 2
           f(2)  = 1
The term cycle count refers to the length of the sequence of numbers generated. In the above case, cycle count for f(20) is 9.
We are going to build a collatz conjecture cycle finding server using Python and ZeroMQ. complete code of this project is available at this location

Beauty of collatz conjecture

All numbers leads to one. It is the philosophy of collatz conjecture. visit this site to visually see the construction of collatz numbers for orbital length of 18.
1011-collatz-graph.v01

Let us build a Collatz Conjecture Cycle Server

Now come to the coding part.Our aim is to construct the server that takes a number from client and calculates longest collatz cycle from 1 to that number and returns it back. For ex. If we give input of 1000 our server should calculate collatz cycles for 1,2,3,……..,1000 seperately and return the longest cycle of all.

Ingredients

  • Python
  • ZeroMQ
  • Gevent

we can build the same server with Python and Gevent alone , but that setup is vulnerable after 10K connections. For scaling it to millions ,we should use the power of ZeroMQ.

Requirements

  • Install  ZeroMQ4.1.2 ( http://zeromq.org/intro:get-the-software ) .Below process shows step wise installation procedure.
$ sudo apt-get install uuid uuid-dev uuid-runtime
$ sudo apt-get install libzmq-dbg libzmq-dev libzmq1
$ sudo apt-get install build-essential gcc
$ cd tmp/ && wget http://download.zeromq.org/zeromq-4.1.2.tar.gz
$ tar -xvf zeromq-4.1.2.tar.gz && cd ./zeromq-4.1.2.tar.gz
$ ./configure && make
$ sudo make install
$ sudo ldconfig
  • Install pyzmq
$ sudo apt-get install python-dev
$ sudo pip install pyzmq
  • Install gevent
$ sudo pip install gevent

Please be aware that ZeroMQ installation will fail if all deapendecies are not installed. pyzmq installation will fail if python-dev library deapendency is not fulfilled. I hope now you are ready with the required set up on a Ubuntu 14.04 machine.

This ZeroMQ server is used to serve requests through a TCP port to which a zmq socket is attached. Server collects data from that bound socket. let us first design function for returning maximum collatz cycle for a given inupt range. That algorithm would look like below

Collatz Conjecture Algorithm

import gevent
from gevent import monkey

monkey.patch_all()

#Algorithm for finding collatz conjecture longest cycle.
#Returns max cycle from cycles calculated from 1 to n.
def do_collatz(n):
    def collatz(n,cycle=''):
        while True:
            if n == 1:
                cycle += str(n)
                break
            else:
                if n % 2 == 1:
                    n = ( 3 * n ) + 1
                    cycle += str(n)
                else:
                    n = n/2
                    cycle += str(n)
       return len(cycle)
    #This Gevent code is for speeding up the calculation of cycles
    jobs = [ gevent.spawn(collatz, x) for x in range(1,n) ]
    gevent.joinall(jobs)
    return max([g.value for g in jobs])

Now let us use this function in our ZeroMQ server that we are going to write below. It just recieves a number from client and calls this do_collatz function and returns the longest cycle back to the client.

ZeroMQ Collatz Server

#zmq_collatz_server.py

import time
import zmq
import gevent
from gevent import monkey

monkey.patch_all()

#Create context
context = zmq.Context()

#Set type of socket
socket = context.socket(zmq.REP)

#Bind socket to port 5555
socket.bind("tcp://*:5555")

#Algorithm for finding collatz conjecture
def do_collatz(n):
    def collatz(n,cycle=''):
        while True:
            if n == 1:
                cycle += str(n)
                break
            else:
                if n % 2 == 1:
                    n = ( 3 * n ) + 1
                    cycle += str(n)
                else:
                    n = n/2
                    cycle += str(n)
       return len(cycle)
    #This Gevent code is for speeding up the calculation of cycles 
    jobs = [ gevent.spawn(collatz, x) for x in range(1,n) ]
    gevent.joinall(jobs)
    return max([g.value for g in jobs])

#Create a loop and listen for clients to send requests 
while True:
    # Wait for next request from client
    number = int(socket.recv())
    print("Received request for finding max collatz cycle between 1.....%s" % number)
    # Send reply back collatz conjecture maximum cycle to client
    num = str(do_collatz(number))
    socket.send(num)

 

Now we have a server ready to serve any no of clients.Let us build a ZeroMQ client to send request to above server and recieves the maximum collatz cycle for a given no.

ZeroMQ Collatz Client

#zmq_collatz_client.py

import zmq
context = zmq.Context()

# Socket to talk to server
print 'Connecting to hello world server'

socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

number = raw_input("please give a no to calculate collatz conjecture max cycle: ")
print 'Sending request %s ' % number
#Send number to server
socket.send(number)
#Wait and print result
message = socket.recv()
print 'Collatz Conjecture max cycle of %s is <[ %s ]>' % (number, message)

That’s it. Our client is ready too. Now open the terminal with three tabs. One for server and other two for two clients to send the request. The output looks like this on my Ubuntu  14.04 machine.

Next try to give input 1000 from one client and 7000 from another client. Server instantly return maximum collatz cycle back to the client. It looks like this.

Selection_003

So it is clearly visible that our ZeroMQ server is working perfectly for serving the clients and solves collatz conjecture problem. This is called Request-Reply pattern of implementation of ZeroMQ. Here communication is acheived through TCP rather than HTTP. There are three more patterns can be implemented using ZeroMQ. They are:

  • Publish/Subscribe Pattern: Used for distributing data from a single process (e.g. publisher) to multiple recipients (e.g. subscribers).
  • Pipeline Pattern: Used for distributing data to connected nodes.
  • Exclusive Pair Pattern: Used for connecting two peers together, forming a pair.

So ZeroMQ has a lot of scope. It is a good scalability solution for current distributed application architectures. All code for above collatz-cycle is availlable at below github link.

https://github.com/narenaryan/collatz-cycle-server

References:

https://www.digitalocean.com/community/tutorials/how-to-work-with-the-zeromq-messaging-library

http://zeromq.org/intro:read-the-manual

http://inerciatech.com/post/5251827502/a-rabbitmq-to-zeromq-gateway

https://speakerd.s3.amazonaws.com/presentations/8035a1002fcd013209132673290742c6/ZeroMQ.pdf

 

Build an API under 30 lines of code with Python and Flask

 

python_flute

Hello everyone. Now a days developers need to perform many jobs. Like web development, database development, API development and so on. Some companies are just having jobs called API developer on their openings sheet.What role APIs are playing now and why one should learn building them is our topic today. Developing an API with Python is a very easy task when compared to other languages. So,sit back and grab this skill for you. Take my words ,this skill is hot right now in the market.

What is a REST API?

REST (REpresentational State Transfer) is an architectural style, and an approach to communications that is often used in the development of Web services. The use of REST is often preferred over the more heavyweight SOAP (Simple Object Access Protocol) style because REST does not leverage as much bandwidth, which makes it a better fit for use over the Internet. The SOAP approach requires writing or using a provided server program (to serve data) and a client program (to request data).

In simple three lines REST API is a:

1 ) A way to expose your internal system to the outside world.

2) Programmatic way of interfacing third party systems.

3) Communication between different domains and technologies.

I think we are sounding technical. let us jump into practical things.By the end of this tutorial ,you will be comfortable in creating any API using Python and Flask.

Ingredients to build our API

We are going to use these things to build a running API.

*  Python

*  Flask web framework

*  Flask-RESTFul extension

*  SQLite3

* SQLAlchemy

Let us build Chicago employees salary API under 30 lines of code

I am going to build a Salary info API of Chicago city employees. Do you know ?,it is damn easy. An API can give you computation result or data from a remote database in a nice format. It is what API is intended for.API is a bridge between private databases and applications. I am collecting employee salary details from Chicago city data website data.cityofchicago.org

code of this entire project can be found at this link https://github.com/narenaryan/Salary-API

Let’s begin the show……..

First , i downloaded the data-set as CSV and dumped it into my sqlite database.

$ sqlite3 salaries.db
sqlite> .mode csv salaries
sqlite> .import employee_chicago.csv salaries

and imported CSV.

Now we are going to build a flask app that serves this data as a REST API.

$ virtualenv rest-api
$ source rest-api/bin/activate
$ mkdir ~/rest-app
$ cd ~/rest-app

Now we are in the main folder of app.Create a file called app.py in that folder.We need few libraries to finish our task.Install them by typing below commands.

$ pip install flask
$ pip install flask-restful
$ pip install sqlalchemy

That’s it. We are ready to build a cool salary API that can even be accessed through mobile. Let us recall the REST API design .It has 4 options. GET,PUT,POST,DELETE

rstapi

 

here we are  dealing with an open data which can be accessed by multiple applications. So we implement GET here and remaining REST options becomes quite obvious.

 

from flask import Flask, request
from flask_restful import Resource, Api
from sqlalchemy import create_engine
from json import dumps

#Create a engine for connecting to SQLite3.
#Assuming salaries.db is in your app root folder

e = create_engine('sqlite:///salaries.db')

app = Flask(__name__)
api = Api(app)

class Departments_Meta(Resource):
    def get(self):
        #Connect to databse
        conn = e.connect()
        #Perform query and return JSON data
        query = conn.execute("select distinct DEPARTMENT from salaries")
        return {'departments': [i[0] for i in query.cursor.fetchall()]}

class Departmental_Salary(Resource):
    def get(self, department_name):
        conn = e.connect()
        query = conn.execute("select * from salaries where Department='%s'"%department_name.upper())
        #Query the result and get cursor.Dumping that data to a JSON is looked by extension
        result = {'data': [dict(zip(tuple (query.keys()) ,i)) for i in query.cursor]}
        return result
        #We can have PUT,DELETE,POST here. But in our API GET implementation is sufficient
 
api.add_resource(Departmental_Salary, '/dept/<string:department_name>')
api.add_resource(Departments_Meta, '/departments')

if __name__ == '__main__':
     app.run()

save it as app.py and run as

 $ python app.py 

That’s it. Your salary API is up and running now on localhost , port 5000. There are two rules we defined in the API. One is to get details of all departments available and second is to get employee full detail, who is working in a particular department.

So now go to

http://localhost:5000/departments

and you will find this.

Selection_074

See how flask is serving database data into JSON through the REST API we defined. Next modify URL to peek all employees who are working in Police department.

http://localhost:5000/dept/police

Selection_073

Oh man, seems like police officers are well paid in Chicago but they can’t beat a Django or Python developer who earns $ 1,00,000 per annum. just kidding.

My code walk-through is as follows

*  I downloaded latest Salary dataset from chicago data site

*  Dumped that CSV  into my SQLite db.

*  Used SQLAlchemy to connect to database and do select operations.

*  Created Flask-Restful classes to map functions with API URL

*  Returned the queried data as JSON ,which can be used universally.

See how simple it is to create a data API. We can also add support to PUT,POST and DELETE on data too.We can also have an authentication system for fetching data through API. Python and Flask are very powerful tools to create API rapidly. GitHub link is given below. Give it a try and extend it with the things mentioned above.

https://github.com/narenaryan/Salary-API

See you soon with more stories.

Docker , the future of Virtualization for your Django web development

what_is_docker

 

 

Always use a virtual environment for your software development

Hello friends. I am a DevOps engineer who deals with development , production , monitoring , configuration management etc of a Software. But for every developer , there will be few  house keeping things which are quite irritating. You are having a single Ubuntu system and for the project’s sake , you will install many database servers and other applications in it. The dependencies for your applications will cause trouble , when  you are installing both your favorite games and working projects in a single system.

Why Docker?

What if something went wrong in your working project and everything in your system is messed up.  So I always  suggest you to separate your work with personal things. In this tutorial, I am going to show you how a developer can run his/her project and its applications in a light-weight virtual environment called container. This container is created by the Docker . We can access docker container from the host system for code editing and view execution. So always use a virtual environment for your software development.

What is Docker?

  • Open platform for developers and sysadmins to build, ship, and run distributed applications.
  • Light weight containers
  • It’s fast , not like a VM
  • Minimal resource usage
  • Run thousands of containers
  • Easy to run your whole production stack locally

Docker can create 100 light-weight environments in a single laptop. Unlike a virtual machine , your docker container will launch in one second. It will give an isolated environment for a developer to work with. In this post , I am going to create a docker container and setup a Python Django project and push it to my cloud repository.

Docker achieves its robust application (and therefore, process and resource) containment via Linux Containers (e.g. namespaces and other kernel features). Its further capabilities come from a project’s own parts and components, which extract all the complexity of working with lower-level linux tools/APIs used for system and application management with regards to securely containing processes.

Main Docker Parts

  1. docker daemon: used to manage docker (LXC) containers on the host it runs
  2. docker CLI: used to command and communicate with the docker daemon
  3. docker image index: a repository (public or private) for docker images

Main Docker Elements

  1. docker containers: directories containing everything-your-application
  2. docker images: snapshots of containers or base OS (e.g. Ubuntu) images
  3. Dockerfiles: scripts automating the building process of images

Installation Instructions for Ubuntu 14.04


 

$ sudo apt-get upgrade
$ sudo apt-get install docker.io
$ sudo service docker.io start
$ sudo ln -sf /usr/bin/docker.io /usr/local/bin/docker
$ sudo sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io

 

Using the Docker

Now we use Ubuntu as our base image for creating containers, then modify a freshly created container and commit it.

$ sudo docker pull ubuntu

this pulls the base ubuntu image to your system. After a successful pull we can launch a new container using following command.

$ sudo docker run -i -t -p 127.0.0.1:8000:8000 ubuntu 

This creates a new container with container port 8000 forwarded to the host port 8000 . The flags mean this:

-i        attaches stdin and stdout

-t     allocates a terminal or console

-p     forwards ports from container to host

Things a developer needs

1) Edit Project code from host in your favorite editor.

2) Run project in that virtual container environment

3) See the output through the port that forwarded.

So in-order to edit code you need to access container data in your host. Instead you just mount a host directory in your docker container. Then your project code is in host ,but runs in the container. Cool right?.

So we will modify the command for running the new container as this.

$ sudo docker run -i -t -p 127.0.0.1:8000:8000 -v /home/hproject:/home/cproject ubuntu 

Now a docker container will be created and started  with following things.

* port is forwarded on localhost:8000

* Data volume “/home/hproject” of host is mounted as “/home/cproject” in the container. It means the files lies in the /home/hproject of our host system is accessible perfectly from the container at /home/cproject.

These two things are required by a Django developer because , he need to modify code and view output through the browser. Now he don’t care where code is running. But here code is running in isolation. That too in a light-weight docker container. Vagrant has same port forwarding and volume mounting strategy but Vagrant is a VM , Docker is a VE.

Play with the container

Now we have a container started . It looks like bash with # symbol. So now install following things in it.

* Python

* PIP

* Virtualenv

* Git

* Setup Django ,MySql and your Project

That’s it. You can install any thing that required to the project.But remember after doing stuff , use exit to come out of container.

# exit

Now if you do not commit the container, all changes you made will be lost. So commit the container first. Before that find out the containerID by typing this command.

$ sudo docker ps -a 

and find out the latest container exited. You can give a name to a container while committing

$ sudo docker commit b03e4fb46172 mycontainer 

This will commit the latest container we played till now and also gives it a name mycontainer. If once again we wish to launch container for working with our project , then use following commands.

$ sudo docker start mycontainer
$ sudo docker attach mycontainer

That’s it. You will enter into a virtual container where you can run your Django project. If you plan to remove a container just do

$ sudo docker rm mycontainer

If you wish to push your container to the cloud , you should use the Docker hub registry. Don’t forget to commit after coming out of container. You can carry your entire project with environment anywhere. Package your Project (Django ,Flask) + Environ (MySql,PostgreSQL,Redis) to a tar file and export it to any place. That is the magic of Docker. For doing that just export a container to a TAR file.

Exporting the containers

$ sudo docker export mycontainer> /home/mycontainer.tar

Instead you can also save images and carry them over system through FTP and then load it in target node.

Loading and saving the images

$ sudo docker save mycontainer > /tmp/mycontainer.tar

and then load it in target host as

$ sudo docker load mycontainer < /tmp/mycontainer.tar

References for dockerizing your mind

http://www.jamescoyle.net/how-to/1499-installing-docker-on-ubuntu-14-04

http://www.jamescoyle.net/how-to/1503-create-your-first-docker-container

http://docs.docker.com/linux/started/

https://blog.docker.com/2015/03/docker-tutorial-1-installing-docker/

https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-python-web-applications

https://www.digitalocean.com/community/tutorials/docker-explained-how-to-create-docker-containers-running-memcached

http://www.slashroot.in/docker-tutorial-how-install-and-use-containers

http://www.slashroot.in/dockerfile-tutorial-building-docker-images-for-containers

https://denibertovic.com/talks/supercharge-development-env-using-docker/#/

http://opensource.com/business/14/7/guide-docker

https://www.codementor.io/docker/tutorial/try-docker-in-2015-fig-cloud-services-containers

https://www.linode.com/docs/applications/containers/linode-lamp-container-docker

https://realpython.com/blog/python/docker-in-action-fitter-happier-more-productive/

http://jeff-davis.blogspot.in/2015/02/multi-tier-architecture-tutorial-using.html

 

 

 

Django with Ajax, a modern client-server communication practise

1666323

Intro

We all know that Django is a powerful framework. Many of us deals with posting data from a HTML form to a Django view and process it. You should have never cared of Ajax way of posting the data. Some of you tried AJAX but faced  few painful intros to setup a perfectly running Django-Ajax implementation. So,I came up this time to tell you how to do that.After this tutorial you will feel quite comfortable about implementing ajax posts and also will keep this article as a reference for future. Source code for upcoming tutorial available at following link.

 

What is AJAX?

Simply put AJAX is about updating parts of a web page, without reloading the whole page.

Suppose we need to update any parts of our website page ,then we must use Ajax. For posting data back to server ajax is required. Now why we should use it with Django ?. I believe following things will answer you.

  •  Using Ajax, we can craft data format that sent back to server.
  •  We can handle the situations like successful post or failure or error.
  • Drop all burden on client rather than server wherever it is possible.
  • For implementing progress bars ajax post is the best way.
  • Drop messages framework to display something ,I hate it. Use JSON response returned from Ajax post to wake up cool notification models etc.

If you didn’t understand the above points , cool. I too didn’t once. The simple thing I am gonna show is , how to post something back to server from HTML and collect acknowledgement that returned back.

Let us begin the show

We are going to see the demonstration from scratch. Let us create a Django project for that. Remember we am playing with Django >= 1.7 here.

$ pip install Django==1.7.3

This installs Django1.7 on your system.

$ django-admin startproject ajaxpost
#Now create an app called mainapp inside ajaxpost
$ django-admin startapp mainapp

Then the directory structure will be in this way.

 

Selection_049

Now let us add the superuser to begin with. Default db will be SQilte3. First do a migration for django to create default user tables in db. So do this.

$ python manage.py migrate

Now create a super user for admin.

$ python manage.py createsuperuser

That’s it. Now we are going to create a home page which has the email field and a password for collecting user data. It is to show data passage from ( client -> server ,server -> client) . Simple,but extremely useful in 100 use-cases.

After creation of superuser ,just create a view for our home page. ajaxpost/urls.py looks like below

#ajaxpost/urls.py

from django.contrib import admin
from mainapp import views

urlpatterns = patterns('',
 url(r'^admin/', include(admin.site.urls)),
 url(r'^',views.home , name=u'home')
)

Now modify the views of mainapp to the following.

from django.shortcuts import render
from django.http import JsonResponse


def home(request):
     return JsonResponse({"hello":"pythonist"})

Now just save and do a runserver. Next Go to http://localhost:8000 and you will find this thing.

$ python manage.py runserver

Selection_042

See how simple it is to send a Json response from a Django1.7 view. Now we are going to crete a HTML file for our home page and try to take data from it to create user. For that create a folder called templates in your project root and make a file called base.html. tree structure of our project looks this.

Selection_043

Ok. Now I am adding  template directories location to settings.py. Add this line to ajaxpost/settings.py

     TEMPLATE_DIRS = (
        BASE_DIR + '/templates/',
                 )

Now Django can find template files. So add this HTML content to templates/base.html

 <html>
     <head>
          <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet">
          <link href="http://getbootstrap.com/examples/signin/signin.css" rel="stylesheet">
     </head>

 <body>

 <div class="container">

 <form class="form-signin" method="POST">
    {% csrf_token %}
      <h2 class="form-signin-heading">Enter Details</h2>
     <label for="inputEmail" class="sr-only">Email address</label>
     <input type="email" id="inputEmail" class="form-control" placeholder="Email               address" required autofocus>
     <label for="inputPassword" class="sr-only">Password</label>
     <input type="password" id="inputPassword" class="form-control"  placeholder="Password" required>
     <button id="submit" class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>
 </form>

 </div>

 <script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>

</body>
</html>

When you fire runserver and go to http://localhost:8000 , you will find a nice bootstrap Enter details form like this.

I hope things are clear. Follow from the top , if you are in doubt.

Now comes the Ajax part

Instead of doing a normal submit , let us have an Ajax post block in our HTML file. Django relies on csrf token for the validity of data posted to the server. So in addition to data , we should send csrf-token. In default django submission,it is automatically sent. But in case of Ajax , we need to take care of that.

Take the below JS code as Granted and place it in  a new <script></script> tag in base.html

 
 <script>

//For getting CSRF token
function getCookie(name) {
          var cookieValue = null;
          if (document.cookie && document.cookie != '') {
                var cookies = document.cookie.split(';');
          for (var i = 0; i < cookies.length; i++) {
               var cookie = jQuery.trim(cookies[i]);
          // Does this cookie string begin with the name we want?
          if (cookie.substring(0, name.length + 1) == (name + '=')) {
            cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
              break;
             }
          }
      }
 return cookieValue;
}


//For doing AJAX post

//When submit is clicked
 $("#submit").click(function(e) {

//Prevent default submit. Must for Ajax post.Beginner's pit.
 e.preventDefault();

//Prepare csrf token
 var csrftoken = getCookie('csrftoken');


//Collect data from fields
 var email = $('#inputEmail').val();
 var password = $('#inputPassword').val();

//This is the Ajax post.Observe carefully. It is nothing but details of where_to_post,what_to_post
//Send data  
 $.ajax({
       url : window.location.href, // the endpoint,commonly same url
       type : "POST", // http method
       data : { csrfmiddlewaretoken : csrftoken, 
       email : email,
       password : password
 }, // data sent with the post request

 // handle a successful response
 success : function(json) {
 console.log(json); // another sanity check
 //On success show the data posted to server as a message
 alert('Hi   '+json['email'] +'!.' + '  You have entered password:'+ json['password']);
 },

 // handle a non-successful response
 error : function(xhr,errmsg,err) {
 console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console
 }
 });
});
</script>
 Now Don’t forget to place {% csrf_token %} next to the form tag in base.html. Let us modify the mainapp/views.py.
from django.shortcuts import render
from django.http import JsonResponse

def home(request):
    if request.method == 'POST':
        #POST goes here . is_ajax is must to capture ajax requests. Beginner's pit.
        if request.is_ajax():
            #Always use get on request.POST. Correct way of querying a QueryDict.
            email = request.POST.get('email')
            password = request.POST.get('password')
            data = {"email":email , "password" : password}
            #Returning same data back to browser.It is not possible with Normal submit
            return JsonResponse(data)
    #Get goes here
    return render(request,'base.html')
Now everything for demonstration is built. We need to wrap up things and do a runserver. Before that complete templates/base.html file looks this way.
   
 <html>
 <head>
 <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet">
 <link href="http://getbootstrap.com/examples/signin/signin.css" rel="stylesheet">
 </head>

 <body>

 <div class="container">

 <form class="form-signin" method="POST">
      {% csrf_token %}
      <h2 class="form-signin-heading">Enter Details</h2>
      <label for="inputEmail" class="sr-only">Email address</label>
      <input type="email" id="inputEmail" class="form-control" placeholder="Email address" required autofocus>
      <label for="inputPassword" class="sr-only">Password</label>
      <input type="password" id="inputPassword" class="form-control" placeholder="Password" required>
      <button id="submit" class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>
 </form>

 </div>

 <script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
 <script>

//For getting CSRF token
function getCookie(name) {
       var cookieValue = null;
       if (document.cookie && document.cookie != '') {
         var cookies = document.cookie.split(';');
         for (var i = 0; i < cookies.length; i++) {
         var cookie = jQuery.trim(cookies[i]);
         // Does this cookie string begin with the name we want?
         if (cookie.substring(0, name.length + 1) == (name + '=')) {
             cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
             break;
          }
     }
 }
 return cookieValue;
}


//For doing AJAX post
 $("#submit").click(function(e) {

 e.preventDefault();

 var csrftoken = getCookie('csrftoken');

 var email = $('#inputEmail').val();

 var password = $('#inputPassword').val();

//This is the Ajax post.Observe carefully. It is nothing but details of where_to_post,what_to_post
 
 $.ajax({
         url : window.location.href, // the endpoint,commonly same url
         type : "POST", // http method
         data : { csrfmiddlewaretoken : csrftoken, 
         email : email,
         password : password
 }, // data sent with the post request

 // handle a successful response
 success : function(json) {
      console.log(json); // another sanity check
      //On success show the data posted to server as a message
      alert('Hi '+json['email'] +'!.' + ' You have entered password:'+      json['password']);
 },

 // handle a non-successful response
 error : function(xhr,errmsg,err) {
 console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console
 }
 });
});

</script>

</body>
</html>
Ok. Now fire up the server using
$ python manage.py runserver
Now I went to http://localhost:8000 and entered these details.
email : pikachu@pokemon
password: charizard
When I click sign in this is what I got.
Selection_048
Hurray. We got it. Data is passed from client to server using Ajax and then server sent same data to client as JSON to display alert message.
This is how we pass information to the server using Ajax . Ajax is mainly used for long polling. Long polling is process in which, Ajax sends request from client to server asking the latest information.Then it updates the contents of HTML with data obtained. Here, we  stressed four things. In case of failure of any one, Ajax post may not work.Those things are listed below.
  • {% csrf_token %} just below form tag.
  • csrfmiddlewaretoken in Ajax POST data.
  • is_ajax() handler in Django view.
  • get(‘parameter’) method on request.POST

What is new in Django 1.7?

 In Django 1.7 ,we have JsonResponse which directly encodes a Python dictionary to JSON object without dumping data .
from django.http import JsonResponse

#Usage in view
return JsonResponse({"foo":"bar"})
 Before you used to do this like
import json
return HttpResponse(json.dumps({"foo":"bar"}), content_type="applciation/json")

Conclusion

I hope you got something from this tutorial. I faced problems when I started doing Ajax posts a year back. All tutorials on web misses one or more important things , so things won’t work. That gives us great frustration. I here showed a simple, working example of Ajax post in Django. Use it. source code is available at this link.

Python and Riak DB, a perfect couple for huge scale distributed computing

Distributed-Storage-Simulation-Cluster_f_improf_1024x407

In this article I am going to show how to use Riak NoSQL database with Python . It been long since I wrote my last post. But this time I am going to introduce a new NoSQL Database which is powerful yet simple,open-source .  ( Riak Database )

What is Riak DB and for god sake why I should learn it ?

Let me tell you a small story . We all know that Amazon is an  E-commerce giant and you some times may be wondering ,how they are handling millions of transactions per second and accuracy is unquestioned . Amazon uses Dynamo DB for all its operations ,some of you have used Amazon S3 service and familiar with concepts like buckets and keys.So it is actually a custom DB developed and used by Amazon . It uses a Dynamo DB database and provides a REST API  access to the users who signs up ,and that setup Amazon call it “Amazon S3 service”.

What Riak to do with Dynamo DB?

Riak DB is created by Basho technologies as an open-source product . It is inspired from Dynamo DB and allow you to enjoy all the features and also additional features than Dynamo DB.If Amazon can do the miracles why can’t you? You can with Riak DB.The structure of Riak DB is almost similar to Dynamo.

In February last year, Facebook acquired the company WhatsApp for $19 billion. WhatsApp serves over 465 million users and there are over 40 billion messages sent in and 19 billion messages sent out every day. Although the company has scaled drastically, they have maintained an uptime of 99%. How have they managed these feats? As WhatsApp engineer, Rick Reed mentioned at Erlang Factory, Erlang is their “secret sauce,” and that has put the language in the spotlight. Riak is built upon Erlang where MongoDB on C++ , Cassandra on Java. That is why Riak is special.

Now what is Riak DB ?

Riak is an open source, distributed database. Riak is architected for:

  • Low-Latency: Riak is designed to store data and serve requests predictably and quickly, even during peak times.
  • Availability: Riak replicates and retrieves data intelligently, making it available for read and write operations even in failure conditions.
  • Fault-Tolerance: Riak is fault-tolerant so you can lose access to nodes due to network partition or hardware failure and never lose data.
  • Operational Simplicity: Riak allows you to add machines to the cluster easily, without a large operational burden.
  • Scalability: Riak automatically distributes data around the cluster and yields a near-linear performance increase as capacity is added.

Riak DB Architecture

Riak uses ring structure for handling keys of multiple nodes

riak-ring

Ok .Leave all the technical things behind construction of DB. I assume you are  a developer who use SQL databases or NoSQL ones like MongoDB,CouchDB. GIve a try to Riak because many big heads like Best Buy  are using Riak comfortably from past few years.

When Riak DB is to be used ?

No database serves all needs. If you are painful while upgrading architecture,and you lose customers or money at down time of server then you should chose Riak because it’s fault-tolerance is supreme. This picture shows where to go.Click on it ,if not visible.

tumblr_lry99bsh3x1r2wj1vo1_1280

 

How to install Riak on Linux precisely Ubuntu?

Here we are going to see the installation steps for Riak.You may be guessing it is painful as Cassandra . Nope . It is dead easy to install Riak. Won’t you believe? For other linux installations and Mac OSX visit this link . Linux and Mac Installation Riak

For Ubuntu 12.04+ 64-bit systems follow me now.

First install dependencies for Riak

$ sudo apt-get install libpam0g-dev libssl0.9.8 
$ sudo apt-get install python-dev libffi-dev libssl-dev
 Next use this link to download latest version of Riak.
$ wget http://s3.amazonaws.com/downloads.basho.com/riak/2.0/2.0.5/ubuntu/precise/riak_2.0.5-1_amd64.deb
Ok. If your download is too slow,then download from a mirror
then you will get a file called riak_2.0.5-1_amd64.deb in downloaded folder.
Next go to that folder in Linux shell and type the command
$ sudo dpkg -i riak_2.0.5-1_amd64.deb 
 Now Riak will be installed.Start it using command
$ sudo riak start

That’s it a Riak server is running on your system now. See how easy! . For checking whether server is running or not type command

$ sudo riak ping
You will be replied with “pong” message. hurray you installed Riak

How Python can access Riak ?

There is a very good Python client written by Basho for Riak. Using that we can do all CRUD operations on  the database.In Riak,there is a different terminology similar to S3.

 

  • Bucket  === Table
  • Key-Value === Records

 

so we are going to store key-value pairs in the buckets. Buckets are name spaces those separate keys. I am going to show you how this can be contrasted with MongoDB documents.

Installing Riak-Python-client

Use pip or easy install command

 $ pip install riak
That’s it . I told you ,it is very simple to setup Riak and jump-start it with Python.

Using Python-client

Let us create buckets,store data in them and next retrieve.

from riak import RiakClient, RiakNode

#All are equivalent , leaving empty means connect to Riak on localhost
RiakClient()
RiakClient(protocol='http', host='127.0.0.1', http_port=8098)
RiakClient(nodes=[{'host':'127.0.0.1','http_port':8098}])
nodes are the remote systems you are running instances of Riak Servers . When your data size is increasing grow your data horizontally by allocating more nodes to cluster. But for now we are just tweaking single server .So this is required
from riak import RiakClient
client  = RiakClient()
That’s it . We got a client with a connection for Riak. We can create ,delete,modify buckets using that client.Let me create a bucket (table in SQL) called users.

Creating and selecting a bucket

myBucket = myClient.bucket('users')
Remember if bucket already exists it will be selected,else dynamically created instantly.
now myBucket is the bucket for users.I will Insert data into it.

Inserting JSON(or any data) into the bucket

Let us think we got data for insertion from a web form or a product catalog.Normally we store that kind of data MongoDB . I am creating dummy records to insert into Riak.
user1 = {
'name' : 'Naren',
'location' : 'Hyderabad',
'role' : 'Programmer',
'fulname' : ['Naren','Arya','Yellavula']
}

user2 = {
'name' : 'Saikiran',
'location' : 'Hyderabad',
'role' : 'Dentist',
'fulname' : ['Sai','Kiran','Yellavula']
}
All python valid data-types  can be inserted into bucket of Riak.
Now we insert those dictionaries into bucket myBucket
obj = myBucket.new('user1',data=user1)
obj.store()
It stores user1 dictionary with key ‘user1′ and user1 information as data. If we wan’t to generate a random key , then use
import uuid
#Keys should be of string type
obj = myBucket.new(str(uuid.uuid1()),data=user1)
or if you skip giving a key ,Riak generates  an arbitrary key and stores value with that key in bucket. Similarly we can add the second user
obj = myBucket.new('user2',data=user2)
obj.store()

Retrieving  data from a bucket

We written data to Riak successfully,now to get it back. get_keys method of bucket returns all keys from that bucket.Using those you can traverse each and every record i.e key=>value pair. use get method  on a bucket to return a record.get takes key of that record.

#Gives <riak.riak_object.RiakObject at 0x7f91607aeb90>
result = myBucket.get('user1')
print result.data
RiakObject has a property called data where you can see all details of user1. If you need to get keys then use get_keys on bucket.It returns all keys as a list.
#Gives all keys in a bucket
keys = myBucket.get_keys()
Now print roles of all users
for key in myBucket.get_keys():
    print myBucket.get(key).data['role']

#This prints 
Programmer
Dentist
I think you got a basic Idea of inserting data into a Riak bucket and also retrieving them.If you are a mongo developer , you got theme by now. There are tons of options for queries and searches. I will discuss them in my next post. First be comfortable with setup and basic CRUD operations as explained above. My next post will be building a REST API with Flask and Riak setup. Don’t miss it, Riak is more powerful than you think.For more details visit these:

Implementing activity streams in Django web framework

 

hsrry

 

What is an activity stream?

It is a sequential chain of all activities done by the users in a website.Like Twitter showing our own feeds ,listing activities of developers on Gitlab dashboard or showing activities of connections on LinkedIn.Whatever you may call it,It is an activity stream.

For ex:

This is the activity stream for some company dashboard.See the smiling faces of company members and their activity beside on their dashboard.If you are a django developer and trying to create similar activity streams,then this post is for you.This post is for everyone who wants to learn how to create LinkedIn like realtime dashboard using powerful Django 1.7 web  framework.

How we can implement activity streams?

Actually there are few advanced concepts like signals and handlers in Django using which we can notify anybody at a particular time i.e when user posts a comment ,send it to email etc.Using that basic functionality of signals ,we should work hard to create dashboard activity streams.Fortunately a young Django developer called Justin Quick wrote a quick usable Django plug-in called django-activity-stream to lift the burden from our shoulders in creating activity streams  with Django in python.Documentation is pretty good but beginners may confuse about it’s usage.Here is the link to that Django plugin library.

https://github.com/justquick

What are the requirements?

*  Python2.7

* Django 1.7> (In lower versions,we need to look into many more things,so install safely 1.7 in virtualenv )

* django-activity-stream

Use the below commands to install Django1.7 in a virtual environment

$ pip install virtualenv
$ virtualenv djangoenv

First command install the virtual environment tool and second command creates a virtual environment called djangoenv,in which we are going to install our Django,and its related libraries.Now only environment is created.In order to enter into that environment we need to activate it.

$ source djangoenv/bin/activate

then you will find change in the shell symbol to

(djangoenv)$ ls -l

Now you are in a virtual environment that has a separate copy of python interpreter and fresh libraries to work upon.

Note: In virtual environment use only pip install command instead of sudo pip.

Now let us install django 1.7 ,django-activity-stream

$ pip install Django==1.7.1
$ pip install django-activity-stream

Now everything is ready.

I am creating a new project called “tutorial” and creating an app called “dashboard”.

 

(djangoenv)$ django-admin.py startproject tutorial

(djangoenv)$ cd tutorial

(djangoenv)$ django-admin.py startapp dashboard

Now your directory structure should be app inside project directory.I also hope that you have chosen your own back-end.I am continuing with MySQL .First there will not be any users.Create database and then use following commands for creating tables.

(djangoenv)$ ./manage.py makemigrations

(djangoenv)$ ./manage.py migrate

makemigrations command scans all your models in apps.So you have a default User model in django.contrib.auth.models.After typing those commands your User table will be created.Now create super user to quick insert data into django models.

(djangoenv)$ django-admin.py createsuperuser

Now after creating the superuser login and populate the users.For me I created 3 users namely,

* naren

* sriman

* manoj

Selection_017

I am going to make users as supervisors.So I create supervisors model and tasks model in my dashboard app.Each supervisor can be allocated with many tasks in hand.

my models.py file looks like

#dashboard.models
from django.db import models
from django.contrib.auth.models import User
# Create your models here.
class Task(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField(null=True,blank=True)

class Supervisor(models.Model):
    user = models.ForeignKey(User,null=True,blank=True,related_name="supervisor")
    task = models.ManyToManyField(Task,related_name='tasks')

I use foreign key to access user name and one supervisor can have many tasks.So it is a many to many field.

Setting up django-activity-stream?

Your settings.py should consists of following things.Add actstream to list of apps.Add that ACTSTREAM_SETTINGS file to your settings.py file.

#settings.py
....
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'dashboard',
    'actstream',
)
ACTSTREAM_SETTINGS = {
    'MANAGER': 'dashboard.managers.MyActionManager',
    'FETCH_RELATIONS': True,
    'USE_PREFETCH': True,
    'USE_JSONFIELD': False,
    'GFK_FETCH_DEPTH': 1,
}
....

Now you need to create three files in order to proceed further.In dashboard craete managers.py, signals.py, apps.py.

#dashboard.managers.py

from datetime import datetime
from django.contrib.contenttypes.models import ContentType
from actstream.managers import ActionManager, stream

class MyActionManager(ActionManager):
    @stream
    def mystream(self, obj, verb='posted', time=None):
        if time is None:
            time = datetime.now()
        return obj.actor_actions.filter(verb = verb, timestamp__lte = time)

This file is the time stamp manager that creates and returns action generated time.You can write your own logic here if you wish to return other parameters than time.

#dashboard.apps.py
from django.apps import AppConfig
from actstream import registry
from django.contrib.auth.models import User

class MyAppConfig(AppConfig):
    name = 'dashboard'
    def ready(self):
        registry.register(User,self.get_model('Task'),self.get_model('Supervisor'))

This apps.py registers the models to the action manager.In order to use activity stream we need to register our models first.So I am registering my User model with dashboard models Task,Supervisor.

Next we need to add one line to __init__.py of dashboard app.These all steps are followed to use activity-stream

default_app_config = 'dashboard.apps.MyAppConfig'

By this we link the AppConfiguration to activity stream’s MyAppConfig.

Now everything is finished and we are ready to go.I am going to create tasks in the shell to concentrate more on creation of activity streams.

Selection_001

 

I created three tasks and these can be assigned to any supervisor.I am going to assign tasks to supervisors and at the same time activity will be recorded by activity stream.Normally in our projects assigning task to a user will lie in a view.So you need to send an action signal at the time of creation of assignment action.action_handler in actstream.actions handles the storing of these actions.

Selection_003

Using  action method we send a signal with actor,verb and target as parameters.These actor,target must be registered before using them.That is what we did above in file dashboard.apps.py .

actor – Source object which is performing action. It is like subject in an English statement.

verb – Verb is the description of the action.

target – Target object is the thing upon action is performed. It is like object in an Enlgish statement.

Now i am going to assign planning task to supervisor naren and instantly sends that signal.

Selection_005

Just see the output 31.It returned me the action object that created.It will be stored in the database.

Let us give tasks for two more Supervisors.

Selection_006

 

Selection_007

Remeber these assignments can occur anywhere in your views at anytime,but actions are stored in a common queue like object based on the time they created. we can fetch actions from that  and list them on the dashboard.

Action is the model that stores all actions generated.Who created model for that ?. django-activity-stream takes care of that.Now we need to list all the occured actions on a page.Just create the dash board view and display all actions in it.

See how the time stamps and all perfectly managed.Just display them on a view.your dash board will be finished.Create more combinations of actions as required and also use full power of django-activity-streams by tweaking all options available

We have tons of options for django-activity-stream like

* fetching actor_stream (actor actions)

* fetching target_stream (target actions)

* fethcing user_stream ( all actions associated with a user)

* following/unfollow  certain stream

The full code for the project is available with sqlite db in this link

https://github.com/narenaryan/django-activity-stream-tutorial

:) hope you enjoyed the article.

 

Ultimate guide for scraping JavaScript rendered web pages

We all scraped web pages.HTML content returned as response has our data and we scrape it for fetching certain results.If web page has JavaScript implementation, original data is obtained after rendering process. When we use normal requests package in that situation then responses those are returned  contains no data in them.Browsers know how to render and display the final result,but how a program can know?. So I came with a power pack solution to scrape any JavaScript rendered website very easily.

Many of us use below libraries to perform scraping.

1)Lxml

2)BeautifulSoup

I don’t mention scrapy or dragline frameworks here since underlying basic scraper is lxml .My favorite one is lxml.why? ,It has the element traversal methods rather than relying on regular expressions methodology like BeautifulSoup.Here I am going to take a very interesting example.I am so amazed after finding that ,my article is appeared in recent PyCoders weekly issue 147.So I am taking PyCoders weekly as an example to scrape all useful links from PyCoders archives.link to PyCoders weekly archives is here.

http://pycoders.com/archive/

It is totally a JavaScript rendered website.I want all links for those archives and next all links from each archive post.How to do that?. First I will show that it returned me nothing when used HTTP approach.

import requests
from lxml import html

#storing response
response = requests.get('http://pycoders.com/archive/')

#creating lxml tree from response body
tree = html.fromstring(response.text)

#Finding all anchor tags in response
print tree.xpath('//div[@class="campaign"]/a/@href')

When I run this I got following output

Selection_004

So I returned with only 3 links.How is that possible,because there are nearly 133 archives of PyCoders weekly.So I got nothing in response.Now I will think about tackling the problem.

How can we get the content?

There is one approach of getting data from JS rendered web pages.It is using Web kit library.Web kit library can do everything that a browser can perform.For some browsers Web kit will be the underground element for rendering web pages.Web kit is part of the QT library.So if you installed QT library and PyQT4 then you are ready to go.

You can install it by using command

sudo apt-get install python-qt4

Now everything is finished.We retry the fetching process,but with a different approach.

Here comes the solution

We first give the request through the web kit.We wait until everything is loaded perfectly and then return the completed HTML to a variable.Then we scrape that HTML content using lxml and obtain results.This process is little bit slow but you will be surprised by seeing that content fetched perfectly.

Let us take this code for granted

import sys  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  
  
  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit() 

Render class renders the web page. QWebPage is the input URL of web page to scrape.It does something,don’t bother about details.Remember that when we create Render object, it loads everything and creates a frame containing all information about the web page.

url = 'http://pycoders.com/archive/'  
#This does the magic.Loads everything
r = Render(url)  
#result is a QString.
result = r.frame.toHtml()

We are storing the result HTML into variable result.It is not a string to be processed with lxml.So we need to process before using content by lxml.

#QString should be converted to string before processed by lxml
formatted_result = str(result.toAscii())

#Next build lxml tree from formatted_result
tree = html.fromstring(formatted_result)

#Now using correct Xpath we are fetching URL of archives
archive_links = tree.xpath('//div[@class="campaign"]/a/@href')
print archive_links

It gives us all the links for archives and output is a very populated one.

Selection_005

So next create Render objects with these links as URL and extract the required content.The power of Web kit provides us to render a web page pragmatically then fetches data.So use this technique and get data from any JavaScript rendered web page.

Total code looks like this.

import sys  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

#Take this class for granted.Just use result of rendering.
class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  
  
  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()  

url = 'http://pycoders.com/archive/'  
r = Render(url)  
result = r.frame.toHtml()
#This step is important.Converting QString to Ascii for lxml to process
archive_links = html.fromstring(str(result.toAscii()))
print archive_links

 

I showed you the fully functional way to scrape a JavaScript rendered web page .Apply this technique to automate any no of steps or integrate this technique and override default behavior of a scraping framework.It is slow but 100% result prone.I hope you enjoyed the post.Try now this on any website you think is tricky to scrape.

All the best.