Django with Azure Webjob.. it works!!

Azure Webjob is another great feature from Azure WebApp which provides cost effectiveness for running background job. With all the great features, and lots of buzz about open source supports in other languages like Java or python. I was let down by the available documentations which can properly guide a newbie like me to use the Webjob properly.

Architecture wise, Webjob and WebApp are running on the same logical machine. The Django Codes that I have deployed in production are available in "wwwroot", the root of IIS. On the other hand, Webjob is located in a different folder at the home directory, "data/jobs". This is clearly to show that the Webjob is not running on the IIS. Hence it will not cost any performance on the IIS.

Back to executing Webjob with python, I have been doing a lot of research and trying to ask a lot of resources from MS. Most of the answer that I received are similar, "Webjob and WebApp are sharing the same server". None of the answers were actually showing me the example codes in python. However, I solved the puzzle from different parts of the blogs.

Here are solutions that I have managed to work on and put in run.py for the Webjob.

1. Instead of creating/install a new set of packages via PIP (webjob does not support PIP), we will refer to the existing packages that were installed via PIP in virtual environment:

import sys

sitepackage = "D:\home\site\wwwroot\env\Lib\site-packages"
sys.path.append(sitepackage)

2. With the similar approach at step one, we may need to call functions/models that are defined in different django app as part of the scheduled task in Webjob. Hence, the following are the reference of the django project path:

myproject = "D:\home\site\wwwroot"
sys.path.append(myproject)

3. We still need to refer to django settings.py to make sure that we do not need to replicate the settings that are already defined. With the research of stackoverflow, I managed to find a code that help me to solve the problem:

from optparse import OptionParser

usage = "usage: %prog -s SETTINGS | --settings=SETTINGS"
parser = OptionParser(usage)
parser.add_option('-s', '--settings', dest='settings', metavar='SETTINGS', help="The Django settings module to use", default='DjangoProject.settings')
(options, args) = parser.parse_args()

if not options.settings:
parser.error("You must specgfy a settings module")
os.environ['DJANGO_SETTINGS_MODULE']=options.settings

4. Executing the tasks that are designed for the relevant jobs as usual python code.

Voala! Your webjob is ready to be executed. Remember, you need to zip up the run.py before you upload it. Now, you can run your webjob by reusing the codes in your original django project without replicating them,

Please leave comments for this blog.

Solving CORS error with Azure CDN

There're a lot of blogs and information about CORS error on Azure CDN. Lot's of solutions had been provided but none of them works for me. After a lot of trials and errors, I have finally found the solution. It's the sequence of defining CORS rules to Azure Blog storage and also loading the static files to the blog storage. I found this link, an easy to use Azure CORS rules manager to manage my blog account without writing any code.

The make life easier for other developers who may move from other platform, here are the simple steps to setup Azure CDN without getting CORS error.

1. Create Azure Blog Storage account and a container. Remember to update this container's property to allow Public Access.
2. *Important: Before upload any other static files to the Azure Blog storage, download the Visual Studio solution package for Azure CORS Rules Manager from the link above. Set the rules according to the picture.
3. Once, the rules are defined, you can start to upload the static files to the container that you have created.
4. Create CDN on Azure portal and map to the storage that you have created on the steps above.
5. Now, you can define the customheaders and other rules for the CDN in web.config. You can find the example from the blog. 

After a few minutes/an hour depends on your rules defined in the CORS, your static files will be available and display accordingly.

Please refer to this article: Why and how we migrated babylonjs.com to Microsoft Azure for more live example on how to make use Azure's CDN and storage.

Deploying psycopg2 to Azure Webapp

Another challenge with Azure-Django-Postgres deployment is to install psycopg2. Azure is running on windows and IIS, hence any existing psycopg2 installation with PIP will not work. As the solution of this, I removed the psycopg2 from the requirements.txt and used easy_install to install the driver from http://www.stickpeople.com/projects/python/win-psycopg/

I added the following to the deployment script: deploy.cmd

:: Addition psycopg2 for windows
echo Pip install git+https://github.com/nwcell/psycopg2-windows.git@win32-py27#egg=psycopg2
env\scripts\easy_install http://www.stickpeople.com/projects/python/win-psycopg/2.6.0/psycopg2-2.6.0.win32-py2.7-pg9.4.1-release.exe

IF !ERRORLEVEL! NEQ 0 goto error

Hope, this will help you to kick start the Azure for your existing this Django migration.

Django deployment headache with Azure Webapp

Cost is always the key factor in my decision making. Through the research, I found out that Microsoft Azure is offering some compiling package for startup compare to other big company. It comes with 3 years of $150 worth of Azure credit each month and also with the benefits of accessing other MS products. As for the startup, who just want to start to launch a business, I believe this can be quite cost effective compare of 12 months free-tier with Amazon AWS and also soon to be ended free server access with Heroku.

However, I will advise to start building on some easy to use server like Heroku, it's way too easy to deploy the code into that server within a day.

After a month of effort in activating the MSDN and Azure account due to my profile into the system. Hooray, but wait... how am I going to deploy the code? With Heroku, there's a very clear instruction in deploying the code for python. However, the article in Azure does not give much detailed on the steps to deploy a workable Django/python WebApp. I have been spending about a week to find out a way to deploy my existing django project to Azure environment. 

In this article, I will highlight the steps for Django deployment. The following is a simple architecture design that I am currently working on my Django project. 

1. Azure Webapp - for the website (front end)

2. Azure VM - for the postgresql and elasticsearch server (back end)

First challenge that I am facing.. how can VM and Webapp communicate and VM is not accessible by public. Without any gurus or experts with me as I am bootstrapping, I need to do the work myself. After some research, with different keywords search, I found a solution.

Azure Virtual Network with point-to-site-connectivity!!

Important: The virtual network should be created before start creating the VM. You can follow this link for the instructions to create virtual network.

Now, it's time to create the VM, I am using ubuntu server VM for postgresql and elasticsearch servers. I will not elaborate much in this blog, as the instructions is quite straightforward.

2nd challenge, no simple but detailed instructions in properly deploy a django project into Azure environment. Through some trial and error as well as getting clue from the instructions, I have finally managed to deploy the django project into Azure server successfully but it's taking too much time to do that. Hence, I decide to share the knowledge with others who may encounter the same issue.

1. Read this link to give you an overview on creating, and deploying Django to Azure Webapp.
2. Azure offers different management sites, like http://manage.windowsazure.com and http://portal.azure.com
2. I am using manage.windowsazure.com. Create a new webapp using custom gallery.
3. Follow the instructions and continue with the steps show on the screen, for example:

4. Once you have completed Azure will deploy the source code automatically.
5. However, this is not ready. You will be getting server error as due to missing files such as web.config, which is used by IIS of Azure.
6. Install Azure CLI to any environment you have. You can follow the instructions in Azure for the installation and understand the usage. Make sure you are connected to azure site with your command line.
7. Generate a custom deployment script with Azure CLI. With the following line on the root of your repository.
     azure site deploymentscript
8. A deployment script file will be generated with name, deploy.cmd. Edit the file to include the syncdb, and other relevant deployment steps that you normally performed with python manage.py command. *important: Please remember that you need to put --noinput at the end of syncdb as Azure deployment script will hang when an interaction is needed.
9. Once you have completed all the necessary changes for the settings.py and deploy.cmd. Check in you changes to the git repository, Azure will automatically pickup the changes and run the deployment.
10. Remember, you need to have web.config in the git too.

Voila. The deployment now should have include the steps to run syncdb, loaddata, createsuperuser or execute easy_install if you need to install packages specific for windows environment as Azure webapp is running on Windows.

Send me some comments if you have any questions or suggestions.

My chaotic travel to Heathrow Airport

Travel to heathrow airport from east London with public transport is quite an unpleasant journey as there's not fast and direct to there. It needs several interchange between the tube line to reach destination. Sometimes, the interchange requires walk up or down the staircase. Imagine if you are travelling with more than one luggage, you need to carry them walking up the staircase as the interchange is not wheel friendly.

Nonetheless, my travel to heathrow airport on Thursday evening was chaotic. The train was cancelled and delayed due to the signal failure, which happening so frequent in UK. Sometime, I wonder why this signal failure still happening when we are getting so much advance with the connectivity of Internet, mobile, GPS, satellite , etc. What type of signal system does the railway company use to control and it always failed by different reasons. Don't they have a backup?

The usual 25 minutes economy travel to heathrow, Heathrow Connect at 7.03pm was initially has on time status which make me decide to buy the ticket. By the time i got the ticket, the train status changed to delay. This is really freaked me up. My flight was just 2 hours away and as usual we never know how much is it going to delay. So, I decided to take the expenses train, Heathrow Express which seems more promising as the usual travel time is 15 minutes. Though the sales person advised that was a delayed and it should not take long to depart from the station.

With a ten minutes wait, I manage to get into the train. The train departed 5 minutes later at around quarter passed 7. Unfortunately, the train was hardly moving. Spending about another 10-20 minutes and still within London. I was started to get panic by the time it was 10 to 8pm and I have yet to arrive the airport. I decided to make a call to the airline and asking about the latest closing time at the check in counter and also preparing for the worst case scenario, book for the next flight in case I can't check in.

Luckily, the announcement from the train driver convinced the customer service And she will help me to inform the check-in counter, but the counter will closed at 20 pass 8. By the time she confirmed with me, the train was just outside the Heathrow airport a arrived at 5 pass 8. I was started to rush to the check in counter at terminal 1. Gosh, it was a long run dragging the heavy luggage and manage to check in just on time. Thank god, I do not have to rebook the flight. It was a closed one.

Having jet lag (wake up at 4am) to finish this blog in Hong Kong.




Quiet paddington station without much trains


- Posted using BlogPress from my iPad

Location:London

Norwegian Wedding (16th of October)

16th of October, the big day for my buddy, Runar. This was his wedding day. I managed to travel to Oslo from London early in the morning on that day. In fact, I attended an uninvited wedding to on the same church where the ceremony was held. :) Luckily, I spent most of my time to change to a formal wear and only went into the church during the ceremony. Initially, I thought Runar was changing the church ceremony's time without informing me. After quietly entered the church, I just realised that I did not know the couple up there at the stage. Immediately, I make the decision to get out from the church as I knew that I was attended an uninvited wedding. 

I decided to spend time around the church which is surrounded by graveyard. Though it sounds not great, but i think it's a lovely park. I spent about one and a half hours sitting and wondering around the cemeteries. Even though, it was freezing cold out there, but the sun helps to warm up my body. I managed to have some quick lunch that I had packed from the train station. When it was closed to the time of Runar's wedding, 2:30pm.. I started to see some familiar faces. Friends who I met during Runar's bachelor party. (Yes, I was in Oslo during his bachelor party on September). I was happy that I did not miss anything, in fact I experienced something new, attended uninvited wedding.

The ceremony was started on time, and it was started with some talks from the priest in Norwegian, then following by some songs from Runar's sister, Møyfrid and Runar's father. I would say they are very talented. Even though, I did not understand Norwegian, but I was admiring their songs. The ceremony just took about half an hour, and it ended with bubbles and cars with tin tied on it. With the limited of my optical zoom from Canon S90, I managed to get the following photos.

After the church ceremony, with the courtesy of Rune, Runar's friend. I got a leave to the hotel for the dinner. Lobsy God is the Golf resort near to Oslo. The venue of the wedding dinner was absolutely perfect and nice. When arrived at the car park, I could see the pond near there was frozen. I could imagine that the weather was very cold in Oslo on the past few days before the wedding. The wedding was blessed with the sunny day and it was a great time exploring the Lobsy God.

The wedding dinner started with drink sessions, photographs and finally entering to the dinner hall for proper meals. The head waiter was explaining different type of wine will be served during dinner. Even though he was speaking in Norwegian, I managed to get translation from David, Runar's buddy. During the dinner time, I also learned a few Norwegian's customs for the bride and groom. Stepping the ground by everyone in the room means the couple need to kiss under the table. Hitting the glass with folk or knife will request the couple to stand on the chair and kiss. While the bride left the room, all the ladies will kiss the groom and vice versa that all the gentlemen will kiss the bride when groom left the room. 

It was the funniest wedding that I had entered. Lost of speeches were shared by the Runar and Bjørg's families and friends. Unfortunately, they were spoken in Norwegian until one speech was given in bilingual, Norwegian and English. I felt proud as I was being invited to the wedding, even though I was the only non Norwegian there. The dinner was continued with dance, games and cakes. I was impressed with the cakes being served. The cakes were delicious, unfortunately too many for my stomach. After 2am, we were kicked out from the dance hall, but we kept the party going at Kjell's room. I only called a day off after 2 hours later when I was filled with vodka and polish sausage.

Short stay in Cairo (April 2010)

I have been quite curious about Egypt and it's capital Cairo. It is one of the most ancient city in the world which was established in about 3000 B.C. Finally, I have a chance to visit the country due to the work assignments. After 4 days of staying in Cairo, finally I have some time to spend outside. Today I was joining a day tour to Memphis and Sakkara in the morning and Giza Pyramids and the Sphinx in the afternoon.

The following are some photos taken during the tour.