Introducing Mr.Dummy who has volunteered to hear out all your questions. So next time you have a question, here are four steps to do :

1. Describe your problem to Mr.Dummy in simple English.
2. Draw a visual representation of your problem for Mr.Dummy to understand.
3. Google/Bing your problem [Possibilities are high that someone already had the same question]
4. By now, you should have your answer, if not, you will atleast have a better Question !

Imagine we have a set of numbers representing the fibonacci series in a list and then we traverse the list to get the numbers.

The data in the list is pre-generated using a formula, or manually entered by the user. This list can be really big, and then we have memory considerations. In cases where we really do not know how many of this elements we need to use, but still we have to pre-generate all of it just in case we need it. In such a scenario Pythons Generator comes to the rescue.

Here is a test function which is a generator, as it uses the yield keyword. We will run through the code step by step to understand how it works.

The output of the code would be as follows :

Hello stranger
1
Whats up?
2
Python rocks
3

When we call the test function it doesn’t get executed  instead, the Python interpreter gives a `generator object’.  Calling a.next results the function beginning its execution – it prints ‘Hello Stranger’ and comes to a dead stop at the `yield’ statement, returning the value 1 to the caller. The function has gone back to its caller, but its `local state’ has been fully preserved. Another invocation of a.next results in the function restarting from where it had stopped earlier – it prints `Whats up’ and stops after returning the value 2 to the caller. Yet another invocation of a.next prints `Python rocks’ and returns 3 to the caller. If it is called again a exception will be raised, as the execution of the function has reached the end.

Going back to our first example of traversing a list for fibonacci numbers, we can achieved the same thing using a Generator function where the instead of the list we have a fibonacci function as the Generator. The Generator uses the yield keyword instead of a “return” , but “remembers” the exact point of execution where it returned. Next time the generator is called, it picks up where it left before, both in terms of function flow and in terms of variable values. Hence when needed and the function is called again the next number is calculated, thus no pre-storing of values are needed.  A code snippet to show the yield in action :

So, now we can actually treat a function, as a list containing numbers, which does not need to be pre-generated rather can be calculated on the fly when the function is called. Each time the function is called, the next value is calculated. Very Interesting, isn’t it ?

In a real DHT implementation, each node would be running on a different machine and all calls to them would be needed to be communicated over some socket protocol. However for now, we will only simulate the DHT to help us understand how it works. A complete real DHT implementation will be discussed in the next post.

Each node is itself a standard hash table. All we need to do, to store or retrieve a value from the hash table is find the appropriate node in the network, then store it in that nodes hash table.

Network Overlay

Each node points to another node called its successor. The last node points to the first node and forming the network overlay. For now we will not discuss how to maintain this overlay, rather concentrate on store and lookup assuming we have the overlay. Nevertheless, it is very simple to simulate the overlay, you can use a simple Circular Linked List, where each node points to a successor node forming the circular list.

Choosing Node ID

By hashing the Nodes IP, using a consistent hash function like SHA-1, a m bit identifier is generated which is set as the Node Id . Similarly the value is hashed to generate a m bit identifier to be used as the key. Any key has to be of the same range of the Node Id so that they can be compared.

Store

The start is the node which will perform the store request. It will simply check whether the key is in between its Node ID and its Successors Node ID, if so then it just has to decide whether it key is closer to its Node ID or Successor Node ID. If it is closer it will store it in its Hash Table. If the successor’s Node ID is closer, it will tell the successor node to store the value. But if the key doesn’t fall in the range, then it requests its successor to perform the search. This goes on until the actual node to store the value in is found. This is done using the closest_node and find_node functions.

The closest_node function, given the node and its successor node and the key, returns the node which is closer to the key.

The find_node function searches to see if the key is between the start node and its successor node, if not moves on to the next node, and does the check again. When found, it calls the closest_node function to find out which node is closer to the key and returns that node.

The store function is used to store a value at a appropriate node. It uses the find_node function to find the appropriate node to store the value and then stores it there.

Lookup

It uses the find_node function to find which node holds the value and then gets the value from it.

Analysis

• Storage or Lookup complexity - Storage or lookup takes O(n) messages. We can modify the current DHT implementation into a Chord Implementation. Instead of each Node having only one successor, it can keep a list of successors (finger table). Then the storage or lookup can be done using O(log n) messages.
• Range Queries - In this simple DHT we cannot perform range queries efficiently. We can implement Prefix Hash Tree (PHT) or Range Search Tree (RST) that supports range queries over DHT.
• Value Storage - Each node maintains a standard Hash Table to store in values. In the current implementation if two values come with the same key to be stored, then the previous value will be over written and is lost. But this is not a common situation as usually key itself is produced by hashing, and thus the different values will not have the same key. But nevertheless this needs to be considered if the key is not made sure to be unique for different values. To be able to store different values which might have the same key, we need to use a chained hash table instead of the standard hash table which can only store one value. In a chained has table each key points to a list of values.

Conclusion

I tried to touch the core basics of a DHT and show how to implement one. DHT’s are very useful means to distributed load among servers. It has a lot of other application domains. This implementation given in this post is however not much useful as it is just a basic simulation of how things work under the hood. In the coming post I will write a real DHT implementation in Python which will run in different nodes, and will try to hook it  up with a sample application.  Watch out for that.

The full Python implementation of the simulated DHT presented in this post can be downloaded from here

Everyday we read so many forward mails, wouldn’t it be cool if we at least read one verse from the Holy Quran instead. In pursuit to understand Islam better we enable online access to the Holy Quran, in the most simplistic form, we are introducing the new website www.Quranic-verses.com. The site is targeted to show random verses from the Holy Quran. You can sequentially read the verses which are chained together. In addition is has the following features :

• It enables  you to post a verse directly into Facebook
• See Salat/Namaz/Prayer timings
• Receive daily Quranic verses via e-mail
• Use a iGoogle gadget to read the verses
• Follow us on Twitter to receive verses as tweets daily
• Find out Qibla Direction

Here is a screen shot of the web page.

Quranic Verses

Hope this site will help all of us to read and understand the Holy Quran, in the midst of our busy lives.

]]> http://rezahok.wordpress.com/2009/07/11/reading-quran-in-whole-new-way/feed/ 0 rezahok Quranic Verses http://rezahok.wordpress.com/2009/04/12/accessing-websites-which-are-allowed-only-in-some-countries/ http://rezahok.wordpress.com/2009/04/12/accessing-websites-which-are-allowed-only-in-some-countries/#comments Sun, 12 Apr 2009 19:48:43 +0000 Reza http://rezahok.wordpress.com/?p=161 ]]> Have you come across websites which are restricted to be used by only the users of a certain country. Wasn’t the Internet supposed to be for everyone ? What happened ?

Instead of throwing some words at them, here is a work around to access those sites from anywhere in the world.  In this post lets take Hulu.com as an example.  Hulu.com allows users from USA only to access their web pages and stream video from it. To be able to crack it we need to first understand how it works and how it restricts other users.

When you request the page, the Server receives your request, checks IP from where the request is coming (which is your PC).

And from the IP it is easy to figure out which country it came from.  So simply if its not coming from inside USA they deny access to the contents.

If we just could fake our IP and make it an IP which belongs to USA then the server would be fooled and would server us with the content.  Seems like do able. What we need to do is we simply use a proxy Server.  How does this solve our problem ? We will have a proxy server in USA, and then we will set our machine to use this proxy server so that all traffic goes through this server.

Now when we request for Hulu.com the traffic is going through the Proxy Server which is in USA, so the IP relayed from the Proxy Server will fool the Hulu Server and thus Hulu.com will think the request originated from inside USA and will thus serve its pages and contents to the Proxy Server. And the ProxyServer will in trun relay it to us. Wualaaaaaaaa… there we have it all working.

How to set your machine to use Proxy Server ?

If you are using Mozilla I would recommend you to install the FoxyProxy. Then search for free proxy Servers in google or you can find a list here as well.

Open FoxyProxy and set the ProxyServer’s ip and port.

Choose the option so that all requests are routed through this proxy.

Now you are all good to go. Just simply type Hulu.com in the browser and enjoy the site and its contents.

1. Important Tags
   Important text of tags which are given importance are : <TITLE>, <META> and the domain name. Make sure the keywords used in this tags and domain are relevant to the main content of your web page. Repeating keywords in small text <font size=1> or <h6> maybe tempting, but today’s crawlers are clever to understand if you are spamming or overusing a keyword. And if they find you spamming by using too many <META> tags or using too many keywords in title, they will penalize you in the ranking.
2. Invisible or Semi-Visible Text
   Have important keywords with text color similar to the background color.
   
3. Doorway Pages
   A page that is optimized to score well on one or more search engines for specific keywords or phrases and is linked to one of the main pages within your website. So when a user comes to that page he does not see that optimized page as it is redirected by META refresh command or by javascript code to the user friendly page.
4. Using comments in HTML
   Having keywords in comments was a common trick, but most of today’s intelligent crawlers ignore comments.
5. Alphabetical Title Order
   This is interesting as if a search engine finds two title to be of the same rank it will choose the one which is in alphabetic order to be of more importance. So if possible have a title which ascends alphabetically . Like “A Climate Change” is preferred over “Climate change”. The reason is the ascii value of A is lower that C which is the first word of the second title.
6. Phantom Pixel
   Use a transparent 1px image and use its alt tag for keywords. As the alt tag keywords are considered by the crawler.
7. Redirect Pages
   Do not have broken links, instead redirect them. Reason is if a page is indexed by a search engine crawler already and that link is broken, the user will end up on your page which is not present, so redirect them to the new link. Use a simple javascript code to redirect document.location=’http://www.newdomain.com’
8. Cloaking
   A crawler wants a text oriented page, where as a user is more happy with graphically rich content page. Since the crawler sees everything as text, so images and video contents are not considered while ranking. One solution to make both parties happy is to have a graphically rich content page and an extra text oriented page highly optimized for crawler.  When a crawler request a page, serve the text oriented page and other wise server the normal rich content page. To know if the request is from a crawler you need to have all the listing of the crawler IPs, which is found but often updated, so you need to keep your list of various crawler IPs updated as well.
9. CSS to Hide Text and Links
   Write your links and keywords and put an image or a flash movie on top.
10. Have good ranked pages link to your page
    If you can have a link from a high rank page to yours, your page rank will increase as well. So if possible pay or by other means have a link of your page, in a high ranked page.

But the bottom line is make sure you have relevant text on your page as much as possible, as that is the only thing the crawlers can read and via which they will rank your page. So remember “Content is KIng”.

Initially you get 1GB of online space for free. As users we always want more space, so either you can buy more space online or interestingly you can trade some of your local disc space for online space. It has a very user friendly and sleek interface. Security is a key feature of this online storage system. All of the files which are uploaded are automatically encrypted before the actual upload which makes your content very secure. A sample screen shot of my wuala client:

Now with such a online storage system all your important files and pictures can be stored reliably, no more crying for lost data when your system crashes.

Technically it is a fascinating product as it is distributed file storage system which is based on peer to peer systems. If anyone is interested to know more about the technology behind the product I recommend watching this video http://www.youtube.com/watch?v=3xKZ4KGkQY8

Not only file storage, it has a social side to it as well where you can have friends added and share folders with them.

So now without any further do go and explore for yourself. Happy online storing and sharing !!

It’s pretty interesting how it is done. Over here we will not go into the details of how it works, but here is a picture that might help you visualize what is going on.

The coding complexity is not huge but what is happening under the hood is a bit tricky to understand. So lets get this straight using RMI we can actually call a function of an object which is not residing in the same machine my code is running.

Now what if instead of our software being able to call a function on the remote object we could have our object himself travel over the network. Sounds interesting, then fasten your seat belt and read on how it is done.

This is a pretty interesting concept called Object or Agent mobility. We can send in an object over the network which will wake up on another machine and then it can do some specified task, and then come back. Pretty interesting isn’t it. Here is a picture to visualize it.

So now what we have here is instead of calling a function of a remote object we are sending our own object to go to another remote machine and do some processing over there and then come back with results. If we can make our objects intelligent and they have mobility then we can do wonders.

An example and usage of this agent mobility is distributed search, where we can have multiple agents with different AI in them traveling to different servers and extracting information and then coming back from where they started. All the results of the agents can be deduced to a meaningful search result. For a moment think wild and I am sure you will come up with dozen of other useful things you can do, since now you know something like this can be done. And top of it is it not cool that your software is actually roaming around the network.

The implementation portion is not rocket science at all so here is a bit of idea how it is done. The implementation is two folded.

1- How to transfer objects from one machine to another.
2 – How to receive and send back the object

The first question is simple to answer. We need to learn how to transfer objects via networks. In Java for both TCP and UDP transfer we have functions to pass a serialized object. So this can be used to throw our object to the remote machine.

For the second question, we should make our server aware of the fact that they will and can receive such objects, so on receiving such an object it will know which function of the object to call to start the object and the object should in turn know which function of the server to call to be able to go back home. As the Sever should have some mechanism to throw back an object, the server will get the home address (home ip) of the object from its data and then throw him back functions. Therefore in short there should be a specified interface an agent should follow so that based on that interface the remote machine can call up some functions of the agent. And also the server should have a defined interface so that the agent can call the remote machines function. For a bit of more clarification here are interfaces that may get you started off.

Interface Mobile Agents

Interface of Agent Receiving Server

Apart from the implementations of this interfaces you will need another class or function which will create, initialize the object and throw it to a server and wait to receive the object back, when it is thrown back by the server.

Please note, the server should be listening on a specified port all the time and will receive the objects. On getting an object it will call the objects agentArrived(…) function. When the object decides it wants to go back to home it will call the servers agentSendBackToHome (…) function. The agent might or can travel to another server instead of going home and that can be achieved by calling the servers agentMigrate(..) function.

I hope this post sheds a bit of light on agent mobility. And now its time to play around with this and make interesting stuff. Have fun !

Pause for a moment and think wild, what if we could some how make the object become more intelligent so that single piece of code will be able to think and handle all the scenarios, or maybe to be more realistic, we can say it will be able handle unexpected scenarios. Its like it will have a behaviour and can work its way around in autonomous environment. Not only can they think on their own but can communicate with other agents in their vicinity and collaborate to achieve a goal. It would be great wouldn’t it ? Sure it would. But to give it some intelligence and make agents communicate with each other, makes it even harder on the programmer’s perspective. I let the reader speculate on the door of possibilities it will open up, if it can be done to that extend.

The good news is, people have already started working on this and they call those intelligent objects “Agents” and further the name “Agent Oriented Programming”. I would suggest every one to have a peek into this concepts.

For Java people good news is there are a lot of agent development frameworks have already been developed to aid the development process of Agent Oriented programming in order to build intelligent systems. To name such frameworks JADE is one of them.

This is “end of dumb object era“ and beginning of a new era for “intelligent objects named Agents.