In this second blog post in the series …

• The Internet of Things and Glanceable Data
• The Economies of Digital Assistants and Personalized Experiences
• Artificial Stupidity, Meaning, and Structured Data
• Application Discoverability and Actions with Facebook
• What’s Success and How To Get There
• $0 to $1B… Yes, It’s That Easy
• Working with Data
• Random Bits
• Art

The Internet of Things and Glanceable Data

Tim Berners-Lee talked about the Giant Global Graph (GGG). He used the term to describe a digital world of interconnected, machine-processable data to complement the World-Wide Web (WWW), the human-oriented world of interconnected documents. GGG and WWW… get it? :-) The GGG was all about RDF, OWL and other Semantic Web technologies. I am a huge believer of the vision. I don’t particularly care about the specific technologies even though I do have my biases towards representation models that don’t require me to write many triples :-) Facebook’s OpenGraph showed us how a great user experience providers the incentive for information publishers to incorporate structured data into their pages.

readDIYmate is an interesting Kickstarter project. The company wants to make it very easy for someone to build Web-connected objects that react to information on the Web. Checkout their videos showing a physical object moving and making sounds when you receive “likes” on Facebook, when a friend tweets something, when someone checks-in at your favorite place. I love the idea. It reminds me of Yahoo Pipes but with physical objects reacting to information events on the Web.

I think there is a tremendous opportunity in building automation that gathers, processes, and personalizes information on the Web. Call it agents, digital assistants, personalized aggregator services… call it Bob if you want :-) The point is that we are going to need help in processing the vast information that is being produced by all the Giant Global Graph-connected things. Computer/TV/phone/tablet screens might not be the only way that such aggregated, processed, and personalized information surfaces to us. Projects like readDIYmate demonstrate the possible bridges between the digital and the physical worlds for information awareness. In an internal version of this newsletter series I had pointed to the “glanceable data” concept, which falls in the same category.

And of course… we have to include devices such as the Nest, Nike Fuelband, Fitbit, and Aria (which my partner ordered and now tells me that I have to use :-( in the category of devices connected to the Internet of all things. Everyone seems to be building similar functionality, which leads me to believe that a platform is necessary. I think the people behind the company Presence are on the right track… a platform to connect everything on the Internet. Facebook is successful in connecting people. However, we need a platform that connects everything… people, physical objects, data. I really liked this quote from the founder:

“This is about making your interactions with spaces and objects more similar to your interaction with people and friends” (source TechCrunch interview)

But is the connection of devices to the Internet enough? Companies such as Fitbit and Nike create isolated islands of information, they lock the users data behind their respective walls. Shouldn’t all data and all devices be interconnected in a Giant Global Graph? Who is going to enable that capability? The value to users comes from a world full of bridges between islands.

The Economies of Digital Assistants and Personalized Experiences

I don’t think that there is any doubt that natural language interaction is going to be THE way we consume and produce information in the very near future. The concept of a digital assistant with whom we can interact has been a goal of our industry for decades. Such digital assistants take all forms and shapes, from Siri on the iPhone, to Roomba, to this cute little robot in the car.

As I wrote above, we need help with processing all the information on Web. We need information processing agents that operate in a manner tailored to our needs and interests. My observation is that everything that the big companies do these days focuses on learning as much as possible about their users so that they can offer highly-personalized services to them. Google hasn’t hidden the fact that their recent privacy policy changes are aimed towards that goal (note the “tailored for you” part). Facebook is, of course, already mining their users’ data.

It goes without saying that companies are focusing on how to make profit. As they get to know more about their users, they can sell more targeted ads, they can offer more relevant-to-the-user services. Here are examples of news in this space that I noticed over the last couple of weeks:

• Amazon offers recommendation services for daily deals. They allow the user to express their likes and dislikes. Effectively, Amazon explicitly learns about the user’s interests and, as a result, can offer better deals. But of course, with Amazon collecting so much consumer behavior data, it’s only a matter of time before they can offer targeted deals automatically.
• Glimpse is a service that attempts to leverage Facebook’s likes in order to learn more about users’ products preferences. They effectively join their product database with a user’s likes.
• Narrative Science developed a program that can write articles. It can create stories tailored to the users’ interests and even tone preferences… dry vs sensational. The Wired article “Can an Algorithm Write a Better News Story Than a Human Reporter” is a great read.
• Google introduced Schemer, which is an activity recommendation service. Again, no surprise. You add “schemes”, or in other words things that you like doing, and Schemer will offer recommendations. As you do stuff, Schemer gets better at recommendations.

I would categorize all the above in the “digital assistants” space, as I also discussed in the previous section. They are offered to us users as little helpers that process information on our behalf and notify us about stuff that are of interest to us. Whether they can be considered “intelligent”, that’s a different topic :-) I personally avoid the use of the term because it has so many connotations. I have used the following spectrum on a number of presentations, inside and outside Microsoft…

I used to have “intelligence” instead of “understanding” but since the former term is so misunderstood and overloaded, I stopped using it. But I diverge.

As before, here we have another case of information islands. My (inferred) interests, consumer habits, activity timelines are isolated from service to service. There is no interconnection. Most importantly, it’s mostly the companies that benefit from that data. Yes, I do consume a personalized experience or receive relevant offers but, at the end of the day, it’s other companies that party on my data. Perhaps a new economic model is necessary.

Scott Merrill reviews “The Intention Economy” by Doc Searls who argues that we should really change the above game. Rather than allowing companies to find things about us, we should really express what we want to do. We should trust our data with a “fourth party” and allow companies to come to us based on what we want to accomplish, buy, consume. I sympathize with Searls premise. Searls writes: “We need ways of gathering, organizing, and controlling the data that we generate and that others suck in from our digital crumb trails. We also need new understandings about how personal data might be used.” (source TechCrunch)

Whether an intention-based economy with fourth parties acting as gatekeepers of personal data is going to be possible, I don’t know. I think that the work being done around digital assistants by so many companies will surface this issue big time.

And since the discussion was about digital assistants, let’s take a trip back to 1979. As always, Xerox Parc had the vision.

Artificial Stupidity, Meaning, and Structured Data

As I was writing earlier about my attempt to avoid the use of the term “intelligence”, I remembered of Stephen Wolfram‘s latest post. Regular readers and those in my organization will know that WolframAlpha is one of my favorite services out there. I really admire the work that Stephen and his team are doing.

Stephen talks about how they are “Overcoming Artificial Stupidity” :-) Effectively he explains how they have been improving the natural language understanding capability of WolframAlpha over the years. And since I’ve been hanging out a lot with language understanding folks lately, I get how usage data improves the accuracy of a language understanding system.

In my presentations and discussions around knowledge I always reference WolframAlpha as an example of a knowledge system that can do really great things but fails at some simple ones. I have been asking the following question as an example… “Who are the members of Coldplay?“. The answer I used to get was the definition of “member” from the dictionary. WolframAlpha didn’t know about the music domain so it tried its best to give me something else. Well, WolframAlpha now gives the correct answer. I wonder whether they found my query in their logs from the many times I used it :-) Just joking. They just hadn’t ingested the data.

WolframAlpha can now even answer questions such as “When were Radiohead formed?“. However, it can’t yet answer my next set of test questions: “How many members were there in Deep Purple?” (it knows the members and the years but it doesn’t count them) or “When did Berlin become the capital of Germany?” (it understand Berlin as the capital but it doesn’t answer the specific question).

WolframAlpha is evolving at a very fast pace, it’s improving, and its knowledge base is expanding with more and more domains. As Stephen says, he aspires to make computers do more than humans. I truly wish them the best. Their work is truly inspiring.

Related to language understanding… “Iris” is  Readability‘s new content normalization service. As per the blog post announcing the feature, Iris will attempt to draw meaning from the Web and it’s inspired by IBM’s Watson. In my mind, Iris falls under the category of “content understanding”. It’s absolutely the future… trying to understand documents, language, gestures and connect them with structured data.

Well, talking about structured data, I think that the move by Flickr to push structured data to Pinterest is very interesting. I find it a great example of the proliferation of structured data on the Wed. Of course Flickr wants photographs to be attributed but no matter what the reason, it’s definitely the right strategy.

Application Discoverability and Actions

When we talk about structured data, we cannot ignore Facebook. Every week they seem to announce a new feature around OpenGraph. They execute really fast in this space and they should be congratulated:

• The social stream can now be used to discover applications on Android, with support for deep linking. This a brilliant strategy by Facebook. They continue to make their platform useful to application creators while they collect even more structured data about what the users are doing on their platform.
• Third party developers can now use the Open Graph to register actions in the user’s social streams. As an example, Foursquare checkins can now include a “save this place” action. Not only the actions represent a convenient way for users to perform a task, they can also drive traffic to a service or application because of the network effect on Facebook’s platform.

Just to further emphasize the last point… Traffic to Pinterest increased by 60% when they integrated with the Open Graph.

Advice for Startups

Even though the following articles talk about startups, I believe that the advices they give are equally applicable to teams within large companies. I think any leader, any project would benefit.

• In “How Technology Can Solve The Financial Industry’s Deficit of Trust“, Mike Sha discusses the findings of a study that shows 75% of those questioned trusting technology companies vs 50% for the financial ones. The reasons that he gives can stand as advice to any team starting a project:
  • Put Users First & Trust: Do what’s right for the user.
  • Truth & Trust: The data never lies.
  • Awesomeness & Trust: Focus on design and user experience.
• In “Data-Driven Decisions for Startups“, Uzi Shmilovici talks about the power of having data in the decision making process. He references some interesting works. I liked the quote from Jim Barksdale: “If you don’t have any facts, we’ll just use my opinion.” :-)
• Twitter talks about the value of innovating through experimentation. I couldn’t agree more!!!
• In “Disillusionment of an Entrepreneur“, Prema Gupta talks about his experience building and growing a company to 10M users. Even though he reached his original goal, his appetite for more only grew, giving him the illusion that he needed to do more in order to feel successful. But he didn’t feel happy. This is a reminder to all of us (and mostly me I guess)… there is more to life than the next career goal/success.
• In “The Billion Dollar Mind Trick“, Nir Eyal and Jason Hreha talk about the three steps for capturing and keeping a user, giving Instagramas an example:
  • Educate and Acquire With External Triggers (e.g. links in Twitterstreams and Facebook feeds).
  • Create Desire
  • Affix the Internal Trigger… create habit.

$0 to $1B… Yes, It Was That Easy To Scale

And talking about the billion dollar mind trick, here’s how “easy” it was to scale to 30M users and ultimately to an $1B acquisition… “Scaling Instagram“. They started with 2 engineers and by the time they scaled to millions of users they only had 5 engineers. Very impressive!

It is a tech-oriented talk that every service engineer should read!!! :-) My key takeaways:

• Simplicity!
• Instrument everything! (here it is again :-)
• Loose coupling.
• Extensive monitoring.
• If you are tempted to reinvent the wheel… don’t! (I can’t emphasize this enough)
• Focus on making what you have better!
• Stay nimble = remind yourself of what’s important.
  • Your users around the world don’t care that you wrote your own DB (or that you wrote another distributed computing platform or another messaging layer or another transactions platform)… it’s the functionality offered that matters.

Working With Data

• Google released an “experimental” version of their Fusion Tables service. I see some really nice features, including a graph data visualizer. They make it really easy to work with data on their platform. I like where they are heading.
• The above reminded me of Google’s BigQuery service. Great stuff.
• A startup shared their experience after a year of using MongoDB.

Random Bits

• “Most Americans don’t plan to use mobile payment systems, says survey”. I guess I am not in the majority then! I really can’t wait to have an electronic wallet and get rid of all the physical credit cards, cinema cards, grocery shopping cards, etc. Electronic wallets can’t come fast enough.
• Quantum Computing: 300 qbits. Need I write more?
• Interesting project by Prof. David Karger’s students over at MIT: Own your own data (thanks to Steve Macbeth for the pointer).
• The Future of Science.
• Microsoft created a new subsidiary company that will work with Open Source, Microsoft Open Technologies, Inc. Who would have thought this few years ago? :-) They just released a version of Redis.
• Android in C#: Amazing work by the Xamarin folks. They took the Android Java code and recompiled it in .NET (mono). The speed ups they present are amazing.
  

  

Art

The Instagr/am/bient project encouraged folks to associate ambient music with photographs from Instagram. What ambient track would you associate with a particular photograph?

It took them 238 takes in order to get it right. Very impressive and very cool concept. Check out their “behind the scenes” video to see how they did it. (via Janet’s Facebook wall :-)

Even though it’s a product, I am including evr1 in the art section of this edition. Perhaps, I should have created a “tech-hippie” section :-) I don’t know, you decide. Anyway… How would you feel carrying the human knowledge in your key ring? Would you pay $140 for it (oh… and you won’t be able to read anything… you just carry it). From The Bible and The Koran to Darwin’s “The Origin of Species by Means of Natural Selection “, from the US Army Survival Guide to Plato’s writings.

If you made it all the way down here, thank you!!! :-)

The Windows blog announced the upgraded services yesterday so I can now talk about it :-)… “Making personal cloud storage for Windows available anywhere, with the new SkyDrive“

I am paying $50/year for 100GB of space. That’s in addition to the 25GB I got for free so it’s not a bad deal at all. All my computers, including my iPad, Mac, and Windows Phone have access to the same files. I stopped using Dropbox as a result of SkyDrive’s new features and cheap storage.

Google announced Google Drive today. They have a very good integration story with their apps (similar to SkyDrive’s integration with the Web version of the Office apps). I am going to check it out. You can’t have enough replicas of your work :-)

Cloud storage rocks. Plus… Jim rocks for hosting my backups at the other side of the pond :-) (and I am hosting his)

Update

You have to looooove competition. Prices will only go down. More features will be added :-)

I enjoy keeping up with the world of technology. I take notes, keep links, and write about the things that interest me. I try to identify the connections with what I do at work, with what excites me.

I like to share. I learn a lot from discussions and alternate points of view. I find it very rewarding to engage in dialogs.

Introduction to the The Hermes Series

Hermes was the messenger for the Greek gods.

Whenever I can, I will spend some time aggregating those news I find interesting from around the Web and write down my thoughts. It’s my way of thinking more about what’s reported in the technology news and correlate it, if possible, with my areas of interest and work. This is the first issue!

It’s all about natural language understanding

The speech apps and platform space is getting crowded and extremely interesting. If you haven’t noticed, Apple is pushing Siri as a differentiator feature for the iPhone. TrueKnowledge (one of my favorite companies) released Evi and, of course, Nuance has been around for a while. It’s no surprise that even more companies are looking to enter this space

• WolframAlpha seems to be appearing in many places. I am a huge fan of their work on curating and managing knowledge. Siri already uses it to drive some of the answers and now a new app, which also uses WolframAlpha, has appeared on Apple’s App Store…  Voice Answer. According to its creators “Voice Answer is an assistant“.
• I haven’t tried it but I loved the description of the “Talk to Eve” app: “Eve is not meant to give meaningful advice or provide answers that are factually correct, although she does know a lot of details plus anything you teach her. She is mainly meant for idle conversation.” :-)
• On the platform side, AT&T seems to be making a big investment in this space. They just announced their “Watson” project: “Opening Watson Speech Recognition to Developers With New APIs in June” and “AT&T’s AVP Of Technical Research Explains The New Watson Speech API“. I am a huge believer of this approach. As anything in computing, interesting and unique features ultimately become part of platforms, they become the building blocks for a new generation of applications. I also came across TheVerge’s article on the same subject.

Given my interests, it should be of no surprise that I see this space through the eyes of knowledge representation and reasoning. In order to enable natural language understanding, we need to build knowledge bases and efficient ways of reasoning over those knowledge bases.

Wearable computing – sensors

We are soooo moving into a technology world where every aspect of our lives is going to be recorded. I can’t wait to receive my Basis device (come on… it’s been a while since I pre-ordered it :-) so that I can automatically record and reason over my excitement levels using their galvanic skin response sensor. I recently got to try a more expensive version of the sensor through Microsoft Research.

This is going to be a huge space. I want to ask the question “What was the title of the song that I really liked at my last concert?“. Given that my phone can record my location, I can get a structured data representation of the concerts (e.g eventful), and perhaps get the order of the songs, a platform should be able to determine with a degree of confidence that I was at a Radiohead concert (they were in Seattle two weeks ago and they were awesome!). Now, combine that information with the stream of excitement levels and we could perhaps determine with a certain probability the title of the song that I felt more than the others.

A TechCrunch article on “Report: Wearable Computing Will Soon Intensify The Platform Wars” points to a Forester report about the upcoming explosion of wearable devices. They point out that a platform play is inevitable. I couldn’t agree more.

Going from users’ interests to actions

This space is very much related with the previous topic on wearable computing. There is a lot of information in the data streams that we can collect.

It’s so easy for users to produce content today… updates on Twitter, activity data through Facebook’s OpenGraph, Pinterest, and so on. It’s funny how years ago we were talking about Web 2.0 as the era in which users were equally consumers and producers of information. I don’t think any of us back then realized how user-generated content would be king in such a short period of time :-)

All the major social platforms want to analyze the data they collect and do something about it. Google recently made the news because of the change in their policy on how they use user-related data internally. No surprise really.

Going from data to actions is a natural step. It’s one of the possible value propositions for the companies.

• I love the potential of Pinterest (story about their rise to fame). They are the 3rd most popular social network. WOW! And of course, we are starting to see how new services are starting to leverage the power of Pinterest… Pinerly is a “Pinterest analytics site“.
• Facebook just introduced the “listen” button for band pages. It effectively allows a user to select their favorite application (matching structured data with an action that may be performed against the user’s preferred application).
• The work that Springpad is doing sounds very interesting as well (GigaOM). Lots of good stuff there. I also came across a TechCrunch article on the same topic.
• When talking about the value of user-generated content, one cannot ignore that $1B acquisition of Instagram.

Some random bits

• Just in case there was any doubt that we are moving to a digital world, even for books: “The rise of e-reading“.
• Microsoft’s Metro as a philosophy. Couldn’t agree more. I love simplicity and clean design.
• Erm??? Most of Facebook users are older than me? Seriously? That’s indeed surprising if true. “46% of Facebookers are over 45 & other surprising facts“.
• Wealthier countries are more interested in the future (Nature report as reported by TheVerge).

The Tools

I was doing something similar to “The Hermes Series” for my team inside Microsoft (in my spare time). Folks asked me about the ways I keep up with technology news and what tools I use. The flow goes something like this…

While I exercise or during my night-reading time, I use my iPad to go through my favorite tech blogs (using the Reeder and Flipboard apps). If I find something that is interesting, I send it to Evernote. If an email contains a link to something that looks interesting, it will also find its way to Evernote. Over the weekend, I take time to read those posts more closely and do some more research on a topic if necessary. Then I just write down my notes/thoughts.

When it comes to blogging, Windows Live Writer absolutely rocks. I still find it as the best blogging tool around. And it’s completely free! I bought the Parallels Virtual Machine for my Mac OS just to run this app :-) (ok… I use Windows on my Mac for many other reasons as well).

Feedback/suggestions are more than welcome!

History

It was towards the end of 2003 when Jim and I started working on a personal blogging platform which we called pblog. It was a collection of ASP.NET components and a database-driven backend that allowed one to build a blog-oriented website. Jims blog still runs on pblog 1.0. Few years ago, I created pblog 2.0. It was a much improved and more flexible version. My blog has been running on v2.0 until now.

The move to WordPress

I decided that another update was necessary. I wanted to move to a modern platform with better tooling support. I didnt want to have to worry about the backend or the protocols for publishing content. I also didnt want to have to worry about the implementation details of the presentation or social media integration.

WordPress is the popular choice with a lot of support and a great ecosystem of plugins and themes. After some false starts, I settled on a commercial theme.

I also changed hosting providers. I was unhappy with the management interfaces of BizHostnet. Ive now moved to iPage. They seem to be very responsive, they offer unlimited bandwidth and space, unlimited number of MySQL databases, and many web sites. My blog is now hosted on Linux/Apache. All for $3 a month, which is not bad at all.

I may change the presentation of the blog as I experiment with more themes and tooling but the links and content should remain the same.

What to expect

The move to a WordPress blog will allow me to use many different blogging tools from any platform (Windows, MacOS, iOS). My hope is that its going to be even easier to produce content now.

I am going to be writing more about Semantics, Knowledge Representation & Reasoning, graphs, related news, and so on. You can also expect to read more about my little startup adventure in which Ive been engaged during the weekends for the last 1 1/2 years. Plexipixel, the company with which Ive partnered for their design and user experience skills, has been doing some wonderful work. Im now in the process of coding their recommendations and integrating with the platform that Im hosting on Amazon Web Services.

The coming months are going to be very exciting :-)

In my previous post I shared my excitement about my new role in Microsoft. I work together with some amazing people on hard problems, in a domain that really excites me: data, information, knowledge at scale. We now want to grow the team and so we want YOU! :-)

We are looking for good Developers and Program Managers. If you are passionate about shipping quality services, you need to talk to me. If you are interested in an environment that encourages small teams, agility, and frequent release cycles, you need to talk to me. If you want to have fun while doing very cool stuff and working with the coolest of future technologies and gadgets, you need to talk to me! If you are a scientist in the fields of Semantic Computing, Knowledge Representation & Reasoning, Artificial Intelligence, I am also interested in hearing from you. If you love graphs, do drop me a line.

You don’t have to have a background in knowledge representation. We have many problems to solve in all aspects of software engineering and computer science. Just be passionate about engineering excellence.

While I can’t talk about the specifics of what we are set out to do, I can given you few examples of some of our currently released projects… If you have an Xbox and use search (voice or not), you are already using one of my team’s features. If you are searching for apps on the Windows Phone or talking to it, you are consuming our services. If you are checking in to Facebook/Twitter directly through Windows Phone (not an app), you are our user :-) If you are using Bing Maps through your browser or through any embedded control, you have been exposed to our technology. If you are giving voice commands to your Windows Phone, you are experiencing our work.

The above are just very few examples of what my organization is doing. We are planning to do sooooo much more! I am excited! :-)

If you are interested in finding out more, drop me a line.

Over the last three months I’ve been having an incredible time at work. I opted not to talk about it because I wanted to make sure that it sticks this time :-) Before my current role, I participated in great startup efforts within Microsoft, projects with great vision and immense possibilities. We’ve learnt a lot from the incubation efforts and we’ve even shipped stuff. Sometimes it was frustrating too. It happens :-)

As I mentioned in my post on “Knowledge Representation and Reasoning in 2012“, I am now following my dream to work with knowledge at scale. I am part of a team of brilliantly talented and passionate people whom I am honored to have as colleagues. The expertise and the quality of the people, from the leadership to all the engineers, designers, business folks makes my day-to-day work a lot of fun. The organization is highly committed and the huge investment shows it. Expect to see many great things.

I am excited! :-)

Why all the above now?… That’s the topic of the next post that is coming up.

BTW… My huge thanks go to Steve Macbeth for his leadership style and the trust he’s shown in me in the first few months in the new org!

2011 was a slow year for this blog. Early last year I joined Erik Meijers team in order to work on an incubation project that involved actors, distributed graph-based processing, a highly-scalable and reliable document store, and more. Given the non-public nature of the project, I couldn’t really talk about it. It was a lot of fun and we learnt a lot. The nice folks in the team are continuing the work but I decided to move on, following my dream to work on knowledge representation and reasoning at scale. Unfortunately more months of work-related silence are going to follow.

I’ve already talked (on this blog, in public presentations, and in articles) about the opportunities and challenges associated with the data-information-knowledge spectrum. Semantic Computing is at the age of reason and I believe that 2012 will represent its big breakthrough into commercial applications and experiences.

As I’ve been talking with others about the knowledge representation space, I observed how the terms information and knowledge are often misused. As I expect to be writing more about information and knowledge in the coming months, I felt like sharing few words on the subject.

Information vs Knowledge: I find that the two terms are often used interchangeable. While an authoritative definition doesn’t exist, Bellinger, Castro, and Mills offer an informative description in their article on Data, Information, Knowledge, and Wisdom.

Data is symbols (bits, numbers, characters). Information adds meaning to data through the introduction of relationships. It answers questions such as who, what, where, and when. Knowledge is a description of how the world works. It’s the application of data and information in order to answer how questions.

As an example… while “Savas likes Coldplay” and “Coldplay is a band” represent information facts, the statement “for each person X there exists a female person that gave birth to X” makes an assertion about a truth in the world, about how we perceive the world. Also, “Savas is an adult” is an inferred statement from our general understanding (i.e. knowledge) that “for each person X with Age(X) > 18, X is considered an adult” when it is combined with the information fact that “Savas is 38 years old”. Of course, one might argue that the inference about Savas’ adultness is erroneous because we haven’t accurately described the world in which Savas acts like a teenager on many occasions :-)

The above is just part of the basics of course. Once we incorporate temporal and probabilistic reasoning things get a lot more interesting :-)

I predict that 2012 is going to be the year where parts of Vannevar Bush’s vision are going to start becoming a reality. Experiences such as Siri are leading the way in incorporating natural language understanding in general-purpose computing. However, I believe that Siri is only scratching the surface of what we can achieve today. 2012 is going to be very exciting :-)

As always, feedback/recommendations are always welcomed!

If you are interested in this space, here are the books I recommend:

• R. Brachman and H. Levesque, Knowledge Representation and Reasoning, vol. 1, no. 1. Morgan Kaufmann, 2004, p. 381
• (I am still going through this one) S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Prentice Hall; 3rd edition, 2009, p. 1152
• J. F. Sowa, Knowledge Representation: Logical, Philosophical, and Computational Foundations. Brooks Cole Publishing Co., 2000

This is Tony Hey‘s group so I am always wiling to help if I can. Also, Lee is such a great guy so it’s difficult for me to say “no” :-) I checked my work calendar to make sure that I didn’t have any meetings before committing to the trip. Then the following two occurred:

• I changed roles within Microsoft (yes, again! :-)*. My new boss was more than happy for me to help out MSR, so all was good.
• I checked my personal calendar and realized that I had bought tickets for the Winter Warmth concert with Florence and the Machine on the night I was supposed to fly to Boston :-( I was really looking forward to seeing them live again but I couldn’t let Lee down. So what is a concert-going geek like me to do?

I knew I was going to be already pumped by the “Deck the Hall Ball 2011” the night before so I had to figure something out :-)

I now find myself at the Logan International Airport in Boston, Saturday 6am, waiting for a flight to San Diego. I got tickets for the “91X Wrex the Halls” festival over there. Oh, it’s going to be awesome.

My Wed-Sun schedule:

• Wed: Deck the Hall Ball. It was amazing: Young the Giant, Foster the People, Cage the Elephant, Death Cub for Cutie, Mumford & Sons.
• Thu: Flight to Boston. In the evening I crashed the CHI 2011 Program Committee meeting and went out for dinner with them since, by complete coincidence, Mary happed to also be in Boston :-)
• Fri: DPLA meeting at Harvard Law School. Met my friend Renata in the evening after a long time.
• Sat morning: Flight to San Diego via Dallas.
• Sat evening: 91X Wrex the Halls. Cage the Elephant, Death Cub for Cutie, Noel Gallagher’s High Flying Birds, and of course Florence and the Machine.
• Sun afternoon: Flight to Seattle.

Next time I will try to cover all four corners of the US! Now I am looking for the right place to make a donation in order to offset the resulting carbon footprint for this trip.

Crazy eh? I love it! :-)

* I am super super excited about the new role but I am not going to say anything else until the new effort sticks and delivers to its huge potential. Given Microsoft’s investment and the caliber of people involved, I am extremely optimistic :-) I am having a blast so far!

Trip report – XLSD Conference

Venue: SLAC (Stanford Linear Accelerator Center)
Conference: XLDB 2011

Overview

I attended the XLDB 2011 conference earlier in the week. XLDB seems to be an emerging conference. There were 280-300 people in attendance and more had to be turned away. In fact, there were people who turned up anyway, even though they hadnt secured a place, hoping that there would be cancellations.

My understanding is that XLDB emerged from the need to discuss technology, issues, and requirements for Big Data in the scientific community, especially when using supercomputing facilities. However, it is clear to everyone that Big Data is not specific to those who are doing supercomputing anymore. Even though we are moving into the exascale supercomputing era (the fastest supercomputer today can do >8 PFlops), having to deal with GBs, PBs, or even EBs of data is not an exclusive problem to those who have access to supercomputing facilities. Hence, it was not a big surprise to find out that the conference attracted the interest of the industry.

A number of leading industry leaders had a presence at the conference: Google, Microsoft, eBay, Netflix, LinkedIn, Facebook, Amazon (even though they decided at the last minute not to present), and IBM (to name few). There were talks by scientists as well.

Most of the talks were great and I enjoyed them a lot. I chose to highlight the Netflix, Metamarkets, and Novartis ones as the driving examples for my observations. The conference organizers have promised to publish the slides and the videos of the presentations.

The value of data

In my mind, the Big Data space is not a niche any more. Its not a space that any company offering enabling technologies, solutions, and services to its customers can afford to ignore. Many customers already have real problems, they already take advantage of Big Data processing infrastructures, and their competiveness is based on their ability to extract value and insights from the data they collect.

Take Netflix for example. Their VP of Data Science and Engineering (highlighting Data Science in the title!!!) gave an excellent talk on how Netflix won the DVD-shipping game, how they became competitive. It was all because of the data they collected and then analyzed. They heavily instrumented their DVD-handling equipment. Every single aspect of a DVDs route was recorded and sent to Netflixs data warehouse. Decisions within msecs had to be made about how to best route each disc. Data was collected and then processed in order to optimize all aspects of their business. They had become so good at it that their only bottleneck became the post office. They had reached such a level of data-based business intelligence that they even went to the post office and started helping them optimize their operations.

The VP of Data Science and Engineering at Netflix was a happy person until Netflix decided to get into the streaming business. Their data collection and analysis requirements skyrocketed!

Here comes the cloud

Netflix needed to expand its ability to process data and make business decisions. They really wanted to move away from the business of managing infrastructure. They didnt want to have to deal with operations, data centers, machines, and so on. They went through a migration period of progressively moving their entire data collection and processing infrastructure into Amazons cloud.

Granted, Netflix had to build their own pipeline based on open source technologies. They used the right tool for the job. They used a NoSQL solution for reliably gathering/recording their data at scale. They used an RDBMS where it made sense.

Netflix is a big company. They can build their own data processing infrastructure from the various pieces. However, what about all those smaller companies that want to collect and process data that is critical to their growth, competitiveness, survival? Wouldnt they benefit from cloud solutions that are scalable, reliable, and NOT managed by them!

Take Metamarkets as an example. They are doing predictive analytics that help advertisers around the world. Apparently the advertising game is following that of the financial markets. Advertisers need to be able to make decisions within few seconds. They need to analyze large amounts of data (billions of microtransactions per day) very fast.

Their needs for a very fast engine for doing almost real time analytics was not addressed by any existing solution. Metamarkets was born in the cloud and continues to operate in the cloud. They didnt have to transition to it like Netflix did. Nevertheless, they still had to build their own distributed, in-memory database (Druid) because none of the solutions they tried could meet their requirements. Given their domain of focus, thats effort that could have been avoided. Rather than focusing on infrastructure, they could have diverted their investments in offering better services to their customers. As it turned out, they managed to build a very good infrastructure that serves them well today.

The data analytics ecosystem

Companies like Vertica provide solutions for companies like Metamarkets. The value proposition is obvious. If you want to build a service or a product that is based/depends upon the processing of data at scale, then you dont have to build the infrastructure yourself.

This is not about deploying a database management system. This is not about just deploying Hadoop or a NoSQL store. This is about getting a complete solution for your big data analytics needs, tailored to your specific requirements (e.g. close-to-real-time processing, batch processing, scale, cloud, etc.).

Novartis happens to concentrate on providing solutions for the genomics/life sciences community. They utilize SciDB, an array-oriented parallel database. There are many companies like Novartis out there addressing different domains. Weve all heard about them and already monitoring them. The point is that such companies are offering solutions for real customer needs today. They reuse open source technologies in order to build an ecosystem of tools and services for their customers.

In my mind, a great opportunity resides in democratizing the data analytics ecosystem by offering scalable solutions at scale; that is, solutions that meet the compute- and data-processing scalability requirements of customers while doing so for 100s of millions of customers at the same time. An ecosystem that addresses all aspects of the Big Data space… data collection, management, processing, visualization, analysis, data mining, machine-based reasoning, and many more!

Isnt it a great time to be in the cloud + big data space? :-)

XLDB was a great conference.