I am really enjoying the work that my new team does. We are a Cloud-focused team. I am really looking forward to when we are going to be in a position to talk in public about what we are building. I am learning a lot and the people in the team are fantastic!

 

The following observation is by no means unique or original. Many many out there have been doing similar and even more complicated calculations, illustrating the value of Cloud Computing. Many use similar insights already for their day-to-day operations. Cloud Computing infrastructures have enabled so many businesses to take off and scale, at a fraction of the typical infrastructure costs. So, nothing new here :-)

I am recording it because it led me to the realization that us developers/architects/philosophers*, not just CIOs and CTOs, have to also start embracing the new platforms out there and incorporate economics-related thinking in the way we develop, not just deploy and operate, software and services. Again, the following observations are very simple and obvious, so don’t expect to find anything profound. You’ve been warned :-)

 

The Cloud is not just a platform for deploying applications and services. It’s a great tool for our day-to-day lives as developers as well.

Recently, we had to transfer 2TB of data as a test for one of the components we are developing. As part of the transfer process, we wanted to calculate the MD5s of thousands of files. We used Azure storage as the destination for our files. We effectively used it as a data repository, as a disk, directly from our locally-running software component. Our intake process was running on a local computer, getting the data from the Internet and placing it on Azure. Then, another computer was reading the data from the “disk”, calculating the MD5s, and storing the results back into the Cloud. That was our mistake. Some simple calculations illustrate why.

 

An Azure (or Amazon Web Services customer for that matter), pays for the data it transfers into the Cloud storage and for the data it transfers out. Any transfers inside the Cloud are free.

For Azure (and assuming that the transfers happen in the US), the cost is $0.10/GB in and $0.15/GB out. In other words, we had to pay (I am not adding the per-10k-transactions cost here):

• $0.10 x 2 x 1,024 GB ~= $205 to bring the data into Azure
• $0.15 x 2 x 1,024 GB ~= $307 to get the data out of Azure in order to perform the MD5 calculations

Now, had we used an Azure compute instance to do the calculation of MD5s, we would have dramatically reduced the cost of our task. Assuming we can get 100Mbits from Azure storage to an Azure worker role, we can process ~12.5MB/s. Let’s assume for the moment that we can calculate the MD5 almost instantly and that we are bounded by the bandwidth. We will need ~46.5 hours to calculate all the MD5s. We cannot get 100Mbits out of Azure to a local machine, so it would have been much much more time consuming to do the calculation outside of Azure.

The cost of using an Azure worker role for 46.5h:

• 46.5h x $0.12/h = $5.58

Wow. Compare $307 against $5.58! That’s a huge saving and I still haven’t included the cost of owning, managing, maintaining the infrastructure in order to perform the calculation locally (hardware, software, networking, power, human resources). All we have to do is deploy our app on an Azure compute node and finish our task for $6. More importantly, given that our task at hand is highly parallelizable, we could just use 10 or 20 or 40 Azure instances and finish in a fraction of the time for the same cost. Installing a cluster to scale out an one-time task would have dramatically increased the total cost for that operation.

Our job as developers is to come up with great designs and build good quality software that meets customer requirements. We should also be thinking about the cost of delivering the software. Cloud computing infrastructures are here to help.

 

* :-) “why a philosopher”

Last Sunday, Michele, Dave, and I participated in the “No Pants Light Rail Ride” event. Similar events take place in major US cities every year. This one was the first one in Seattle.

It was an absolute blast. We had such a good time! Just looking at the expressions of unexpected passengers was great :-) Michele and Dave... you were both such a good company. Many thanks!

Videos and photographs of the event are all over the place. Here are some representative ones.

• The Komo News report is fun. The reporter took his pants off at the end :-)
• The Highline Times post has some nice photos (just click on the first one).
• The Seattle PI has some video shots. David is the one talking at the start. I can be seen in two of the shots in the first few seconds, albeit very very briefly. At the end of the video, the shot of the pants on the conveyor belt at the airport is great :-)
• King 5 news: Michele and Dave can be seen. The “happy faces” guy is blocking me :-(

A random YouTube video.

 

And here are some of Dave’s excellent photographs (I can’t believe I am posting photos of myself without pants :-)

  
  

It’s been long time in the making but, finally, Tony is blogging :-) In fact, it seems that the entire External Research team might be posting through the same blog. Definitely worth following! And no, I am not just saying this because it’s my previous team and I really like everyone over there :-)

External Research blog

The inspiration behind this blog is a strong desire to foster connections that lead to meaningful breakthroughs; to engage in ongoing dialog in an open forum; to discuss and debate the information and ideas critical to harnessing the power of science and technology to address the most urgent global challenges.

Every day, I have the privilege of witnessing the wonder of discovery, regardless of where it takes place, or whether it’s undertaken by academic researchers and scientists around the world or within Microsoft External Research. For those of us at Microsoft External Research, the opportunity to collaborate with the finest and researchers and scientists working across the globe today is the core of everything we do.

This blog is being launched at a critical juncture in the research field. Over the next decade, it’s predicted that more scientific data will be accumulated than has been collected thus far in all of human history. For every member of the global research community, that fact represents a serious responsibility. Given the richness of available resources, it’s important to make the most of that data by sharing not only ideas but inspiration as well, and to challenge one another to contribute the best work possible to the global, virtual collective of dedicated research and technology professionals. The work being undertaken by the global research community is significant: the impact of the research being done throughout the community extends far beyond today. And, by collaborating in order to share the best work, the pursuits will benefit generations to come.

Please, let us know what’s on your mind. Let us know what’s important to you and how the research community can make a meaningful contribution to your work. On behalf of all of us at Microsoft External Research, we look forward to the conversation.

Tony Hey, Corporate Vice President

Microsoft External Research

(source: Welcome to the Microsoft External Research blog)

Last February, I linked to a video, which I thought was very powerful, from Konstantinos Pilavios. Here’s another one called “Small Pleasures” (make sure you watch it all the way to the end).

(thanks to Ioannis Kavouras for the link)

Our upcoming book (tentative title: “REST in Practice”) has a Facebook page, so go become a fan. We are going to start posting news/updates about the book soon. You’ll also be able to ask questions, monitor our progress, and give us feedback. However, if you are not a Facebook fan, you can always contact us directly:

Savas Parastatidis: Web, Facebook, Twitter, email

Jim Webber: Web, Facebook, Twitter, email

Ian Robinson: Web, Twitter

We are also considering launching the “restbucks.com” Web site.* “Restbucks” is a coffee shop. It is used throughout the book as an example of how Web technologies could be used to build Web-based services solutions (inside and outside of an organization), apply hypermedia in the implementation of business processes and documents, address scalability and fault tolerance issues through caching, utilize Atom and Atom Pub, etc.

We are very excited about the book and we are making our best to bring it to your hands as soon as possible. We are also thrilled to see our work already influencing the thinking behind efforts like Restfulie.

 

* We need to check with a lawyer first for obvious reasons :-)

Tini, Nuno, and I went to Stevens Pass today. It was a beautiful beautiful day. Blue sky and sunshine. The snow was not great. The runs were a bit icy but the beautiful day made up for it. No powder unfortunately :-(

Stevens Pass is not like Whistler for sure, not even close. However, it’s nearby (it took us 1h45m in the morning to get there). We also got a nice surprise when we found out that since we are Edge Card holders (for Whistler) we were entitled to a free Stevens Pass “Advantage Pass” (the equivalent of the Edge Card) at no cost. This saved us around $90 and we got to ski the entire day for free (the next time, the lift pass will be $10 cheaper). What a great surprise :-)

On the way back, we drove around 40mi towards Seattle only to find out that a bridge got damaged by a track. We had to drive all the way back, pass Leavenworth, on to Highway 97, and then I90. A 3 hour detour :-( I made sure that there was some excitement on the road by refusing to refuel at the various Shell stations on the way*, which meant that we almost run out of fuel in the middle of nowhere. We were practically on fumes when we reached a gas station :-)

 

The usual panorama :-)

And here’s a video I made with various clips (using my iPhone) and photos from the day. Unfortunately, Facebook didn't let me upload the version of the video with "Somebody Told Me" by The Killers as background music (due to copyright). It's soooooooo much better that way!

 

* For a number of years I’ve been boycotting Shell stations after their exploits in Nigeria.

I loved, of course, Michael Palin in all the Monty Python sketches and movies but I used to be soooo envious (and still am) of his travels for his documentary series. He’s been to 94 countries and not just touching down... he’s really visited and explored those countries.

BBC News has a 5-min interview with Michael Palin. I was smiling as he mentioned in this interview all the wonderful places in Peru that were also part of my month-long adventure back in 2001! What a wonderful experience that was :-)

If you haven’t watched his travel documentaries, I highly recommend them. If you haven’t watched any Monty Python, you are really missing out on some excellent comedy :-) (granted, it’s not to everyone’s taste :-)

This reminded me of John Cleese whom I had the fortune to see live here in Seattle few weeks ago. Time to watch “Life of Brian” or “The Holly Grail” again? :-)

The photographs that made it to the final round of the Bing homepage contest are amazing. The winning one is a photo of the Seattle skyline. It’s gorgeous.

Justin Kraemer/Justin Kraemer Photography
This photograph of the Seattle skyline won the second Bing homepage-photo contest. It will be featured as Bing's background image on Jan. 6.

(source “Seattle skyline wins Bing 'hometown pride' photo contest”, from “The Microsoft Blog”, SeattlePI blogs)

The skiing season has started really strong. After the great snow at Crystal Mountain few weeks ago, I went to Whistler for the weekend with Tini and Nuno. We also met my very good friends (and usual skiing buddies when I go to Whistler... they practically live there in the winter :-) Theoni and Simran. Simran (yes, I still owe him $5 for taking his top off in the snow at the end of last season) is an amazing skier! I think I just had a breakthrough in my overall slalom style when coming down black diamonds as a result of a tip he gave me.

The first day was absolutely amazing. Great snow and great weather. The second day was wet :-( It rained so we didn’t ski all that much but we still had fun chatting.

Tini, Nuno, and I also explored a couple of bars. It wasn’t as busy as usual but I guess it’s only the start of the season.

 

Simran and Theoni at the top of Blackcomb.

With Nuno at the top of Blackcomb.

  
Following Nuno with my camera as he was skiing down. Then up the mountain again.

It was a gorgeous day (the back of the Blackcomb mountain)

 

Before the last run of the day (facing towards the village).

  
 
 
Out and about at Whistler village. We HAD to go to the Longhorn saloon since Tini was part of the marketing team of the Windows version with that codename :-) I am not going to say what I am doing with the melted chocolate and the spoon :-)

And here’s a video of me skiing down with my iPhone in my hands (and yes, falling after hitting Nuno’s skis :-) There is another good clip but I do not have permission from Tini to post it :-))) This was taken on Sunday, so visibility and conditions were not great :-(

It looks cool too :-)

Thanks again for the credit to our book (tentative title “REST in Practice”).

I was forwarded a translated to Greek version of the “Bar Stool Economics” anecdote. It effectively tries to exemplify how the American tax system works. It uses clever language and math to hide the fact that the system is indeed biased towards larger savings for the richer. It also makes the assumption that the richer want to contribute to the common good and that the majority of the society does not understand the difference between absolute vs percentage-based savings.

It turns out that the anecdote is attributed to Dr. David R. Kamerschen, Distinguished Professor of Economics at the University of Georgia. However, after just a simple check, it turns out that Dr. Kamerschen was not the original author. I guess someone wanted to add “authority” to the anecdote by simply adding his name :-(

I always believed that the more one earns, the more they should be taxed (not in absolute amounts but as a percentage of their income at an increasing scale... without upper limits that further amplify the imbalance of wealth accumulation in favor of the few). It’s as simple as that.

The New York Times has a nice article (“A Deluge of Data Shapes a New Era in Computing”) about the “Fourth Paradigm: Data-Intensive Scientific Discovery” book, which also happens to include my “A platform for all that we know: creating a knowledge-driven research infrastructure” essay :-) Check out my October blog entry about the book.

It’s great that the article focused on Jim Gray’s observations around data-intensive computing, the availability of scientific data and literature online, and interoperability. Jim’s work and thinking continues to inspire me. I am soooo honored to have met and worked with him. He’s inspired so many!!!

Just read Guilherme’s interview over at InfoQ about Restfulie. Nice read.

It’s great to see that our (upcoming) book is already helping people create Web applications and frameworks out there.

“Jim Webber, Savas Parastatidis and Ian Robinson will release a book on using the web as one's infrastructure in early 2010 and after reading it I finally got to understand how hypermedia fit into the rest scene.”

Guilherme was very kind to give us credit but it’s his engineering effort and ideas that created a buzz and brought attention to Restfulie. Great great work!

I think that Restfulie beautifully captures the hypermedia concepts and helps developers build hypermedia applications.

I can’t wait for our book to become available (I’ve been working on a chapter most of the day today) so that more people can read about our take towards REST and hypermedia when building distributed applications. As everything else that we’ve done, I am expecting some controversy, which is good (TM) :-)

I realized today that I didn’t write* about my long weekend in London over Thanksgiving (Nov. 26 - Dec 1), which was absolutely amazing! It was a very last minute thing but it turned out to be great, as it is usually the case. I had such a beautiful time.

I spent some time with my friend Renata, drunk few beers with Meropi and her friends, and briefly saw Paul Watson together with Jim for lunch. I stayed one night with the two Dr. Webbers at their place, which was absolutely great. We had a great Indian dinner and few drinks. I miss hanging out with my pal Jim :-( Ian Robinson also dropped by and we chatted about the book (on which I am working at the moment) and just hang out. It was great seeing him!

I enjoyed the main part of the weekend with Natasa, who traveled from Greece. We walked a lot, went to a great Mexican place (keep forgetting its name) with the two Webbers, Ronnie Scott’s for an excellent Jazz night (Lucinda Belle was great and Soulive were amazing), saw Wicked (the musical), watched Disney’s Xmas Carole in 3D at the IMAX in London, went to see “The Habit of Art” at the National Theatre (it was absolutely amazing!!!), visited the Globe theater (where Shakespeare’s plays were performed), danced at the Fabric (enjoyed the dancing but the club was not great), tasted different ethnic food, drunk too much wine, and chatted a lot :-) I loved it!

It was a memorable weekend. One of those that makes me want move there. I keep having such weekends in London. Hmmm!!! Jim, will you twist my arm a bit more? :-)

Here are some photos...

       

 

* Sorry Lee… yet another travel trip report but not the one you wanted :-)

I just watched “The Rage in Placid Lake” (an Australian indie film), which was cute. I liked these lines delivered by the main character towards the end of the movie.

“There are moments when you know what you need to do.

If you stopped and analyzed it, it’d be crazy.

You’d lose pros and cons and so think “no”.

But you don’t think, you are listening to yourself. These moments can save us.”