Computing Startup Idea: On Structuring Online Human Knowledge

I'm fascinated by the google knowledge graph. Although i'm actually not sure whether or not this thing is real or just a publicity stunt, but it sure sounds like the future of search.

Now you've probably noticed how much knowledge exists on the web in an unstructured, ad-hoc way, mostly on forums, blogs, and wikis. If you're also addicted to stackoverflow and ubuntuforums you know what i'm talking about.

Here's what I'm suggesting: to introduce a way of structuring knowledge (technical knowledge for now), represented in questions and answers on forums, let's start with computing forums. What I want is a flexible framework that allows the users to ask questions on properties of objects. for example, take this stackoverflow question here:

I have 2 unrelated questions.
1. how many posts does facebook allow you to get with an api?
2. using facepy, facebook, or any other api, how do I get the number of friends a user has (the user is not my friend). The user id is provided.
This is how I get the number of my friends with facebook api:
If i try this:

I get this error:

My proposed framework would allow the question poster to use a structured system that relies on an extensive tag library to say something like this:

Q1:
    facebook.api.posts.size() = ?

Q2:

    using x where x.type = API

    var FRIEND y where y is not in facebook.api.friends_list(me)

    y.friends.size() = ?

Then we can could some similar way of posting the code snippet and indicating that it gives a certain error.

Of course my example script is still a little basic, but I hope it gives the sort of idea I have. You can think of so many extensions to this, including a translation from the official APIs of each source code, and using this to automatically answer such basic questions. Maybe also using the automated doxygen documentation body to answer questions, or at least automatically tag them correctly.

The purpose of such technology would be to provide one more step that further closes the gap between human language and computer language, and enable a more seemless interaction between the two.

Computing Startup Idea: CalendarJini - The Calendar Assistant

Idea:
In addition to normal calendar capabilities, CalendarJini (CJ) should be able to do the following:

• Define a "flexible event." This event is added to the calendar without having to choose a fixed date/time for it. CJ is responsible for defining a suitable slot for it in my busy day.
• Define "time colors," which are fixed periods in my day with a certain "theme" to them, such as "Business", "Sports", "Family Time"... etc. CJ uses these as a guide to place a new flexible event.
• A flexible event can have a duration, loose boundaries (such as a certain date, week, or a specific few days), and type (business, sports, dinner out... etc). CJ also uses this to choose where to put it.
• CJ is smart enough to group similar flexible events together. For example, I can add several "send email to X" events, they're automatically grouped together.
• CJ can use geo-tagging to reorder events according to my current position and the positions of the events. For example, it would recommend that I go to event1 before event 2 since I'm already in event1's vicinity.
• Flexible events are a powerful addition to a normal calendar, specifically when a group of people share an event. CJ can use all their information to decide on the best time (maybe also place) for a meeting or a dinner. Adding new people can cause the event to change place/time, notifying all collaborators.

CalendarJini: "just name your event and go back to work, i'll find a place for it."

The idea came to me in a Hackathon held by facebook during my one-year stay at EPFL. Thinking about the potential of the technologies coming out everyday, making everybody's life easier, one of the ways to make my life easier is having a personal assistant. My idea is related to calendars, and proposes a better way of organizing them.

The reasoning behind the idea mainly is that I noticed I could waste quite some time reorganizing one of my busy days, sometimes shifting unimportant events to following days, or regrouping events because they can be done consecutively and so on. If you're like me you don't want to do this manually, it's such a tedious job. You can see now many of these tasks can be performed programmatically. This is a calendar I would certainly use!

A short on UV decomposition and Hadoop

SVD is probably the single most successful technique used to do recommendations in collaborative filtering. It's been used in the Netflix contest, adding recommender systems to the very long list of problems it can solve. It's not an easy solution though. Not to understand in the beginning, and not the implement either.

A simpler form of the decomposition is called UV, and it's very widely known among students of data mining and information retrieval for being a simple algorithm that does a lot of heavy lifting in a variety of problems. This post is my attempt at building a scalable version of the simple UV algorithm described in Jeff Ullman's book Mining of Massive Datasets, chapter 9. I used Hadoop to decompose a 2.5 gig netflix dataset on a cluster of 88 machines in a class project. The source code is available here, as well as a serial version.

One thing to note about working on a serious Hadoop project. Hadoop can easily drive you crazy. You go through so much trouble you actually manage to pull a few hairs out just setting your distributed cluster on 88 nodes and running your WordCount program, then development is horrifically slow because of all the unexplained bugs, the inability to debug properly, and the sluggish performance of a cluster that's been cursed by Hadoop. But then right before you get the epiphany that programming is just not your thing, and you'd rather chase your dream of being a professional WoW player, everything works after you've fixed a bug, renamed a file, deleted a certain log, or eliminated some other really dumb excuse for a fault-tolerant framework not to work, and it just works beautifully!

On Hadoop, and Mac: Quickstart

Last month I had a project for my advanced databases course, it was to implement a simple iterative UV matrix decomposition algorithm using Hadoop. This post is just about the first step: setting up Hadoop to work, not the algorithm itself. I'm gonna talk about that in a future post.

For those of you noobs like me who never tried Hadoop on more complicated stuff beyond Apache's boring copy & paste WordCount, it can be a real pain in the butt to run anything more complicated, simply because it's just one of the worst-documented, badly written pieces of software that implement a really nice idea!

One good idea is to compile a Hadoop version from scratch, rather than use the pre-compiled libraries. This way you should be able to debug and run your Hadoop applications easily. For that, take a look at this video tutorial.

In this post however, I list very simple steps to quickly get Hadoop running on a Mac. So here we go:

1. Download Hadoop

You can get the latest stable version from here: http://www.apache.org/dyn/closer.cgi/hadoop/common/ or you can go to the archive to find older versions. Sometimes you need a specific version for compatibility, and there are some tutorials that recommend a Hadoop version that's earlier than 0.21.0 so that it works with the infamous Hadoop Eclipse plugin. I tried the plugin, it's definitely not worth it. You're better off either compiling your Hadoop, or just following these steps if you don't need Eclipse's step debugging.

Let's assume you downloaded the version and decompressed it into your ~/Downloads directory.

2. Setting Environment Variables

You need two things set, either export them temporarily, or put your exports in your bash_rc or bash_profile:

• JAVA_HOME: needs to point to the directory for your Java binaries. To find this you can first get which Java binary is set in your path by running which java, then running readlink on the output path from that. Note that sometimes the readlink would just return another symlink, which you need to follow again. Do that till you reach the JRE/bin/java directory.

• HADOOP_HOME: needs to point to the bin directory of your downloaded Hadoop version. In our case, set it to /home/user_name/Downloads/hadoop-*/bin, where hadoop-* is your hadoop version.
  

3. Configuration and Local SSH

Follow apache's pseudo-distributed operation steps for configurations and passphraseless SSH, they're standard. Here.

4. Run it!

First, you need to tell Hadoop to set up its file system by running a format command:

> bin/hadoop namenode -format

Then start the daemons:

> bin/start-all.sh

You should see a few lines appear indicating the starting of the namenode, the datanode, the secondarynamenode, the jobtracker, and the tasktracker.

5. Monitor Your Jobs

You can use Hadoop's graphical administration interface for the jobtracker, by default it's at http://localhost:50030/jobtracker.jsp where you can see all your running, failed, and retired jobs, should look like this:

Don’t Hate Her Because She’s Successful

Thelastpsychiatrist strikes again. On how the system sends misleading signals to women (at least that’s what i gathered from his mind-boggling-metaphores-infested writing!). He argues that women are picking the wrong fights with the system, striving for the trappings of power such as promotions, management, and work prestige that is without a real reason, but just for the sake of itself. Instead, they should fight for real chances to change the unbalanced workforce, and that is, in his opinion, all about how much money you make, you long you work, and other material benefits.

One quote i especially liked:

“It’s so great that Americans will still vote for a white guy even if he’s a little black”