Marc J. Greenberg

Codemarc's Blog

Category Archives: Cloud

Stevedore in the Fog

I have always been connected to the water. Staring out of my office window, gazing over downtown New York, I sometimes think about the what the city looked like during mid 19th century “Gangs of New York” times. I imaging myself as a dockworker loading and unloading ships only to realize I am a stevedore even now.

For the past year, many of the projects that I have engaged in have been about developing cloud delivered containers for large scale applications. The work allows me to maintain a sharp focus on a “Big Things” agends while continously delivering useful and reusable components.  To me the cloud is a reborn technology that encourages better architecture. The cloud of the now is simply understood as “Compute, Storage and Networking”; pure constructs in the cyberspace of Gibsons, Neuromancer.

Docker… Docker… Docker…   is the enabling technology that brings the devops estibador, super-scripting-power. The question is: how do we unravel the bigger messes and produce cloud native components that are robust but simple. Hopefully the answer will materialize soon enough

Tags: , ,

Big Data Floating in the Clouds

For the last few months I have been building a prototype on top of an Apache Hadoop 1.0.4 cluster that I  built from scratch out of six virtual machines running  Ubuntu Server 12.04.2 LTS. It has been an interesting experience. Simply put, this is the actual learning process that every hacker goes through on every new project whether its a programming language, platform or technology. So now that I got a handle on the basics and I can take an earnest look at other peoples packaging.

Today I am checking out the current offering from Cloudera. I found the download named Clouder Manager 4.5 Free Edition, and proceeded with the installation. Of course I need to install it on a few nodes so I am back to setting up some more servers.

Cluster up

This time I decide to use my mac pro server configured with virtual box. I planned on running a three server cluster (cloud1,cloud2,cloud3) so I set it up and run into a few networking problems. I get my ops dept to fix my port to allow for multiple mac addresses. Here are some of the issues and solutions I encountered when setting up the environment:

For each cloned virtual server I needed to change (persistently) its host name and mac address. The tools ( virtual box in this case ) should have properly handled this. It did NOT. So I did the following  hand job on each machine.

  1. sudo vi /etc/hosts
  2. sudo vi /etc/hostname
    (remove cloud definition from each)
  3. sudo vi /etc/dhcp/dhclient.conf
  4. sudo rm /etc/udev/rules.d/70-persistent-net.rules
    sudo mkdir /etc/udev/rules.d/70-persisitent-net.rule
    (thank you Peter Mount)

Install Cloudera Manager (Free Edition)

So my first installation was from my remote desktop linux to my cluster and it failed. I then decided to allocate another local instance (cloud0) and try again. The installer runs ok and i point my web browser at http://cloud0:7180, login as admin/admin and away we go:

This installer will deploy the following services on your cluster:

  • Apache Hadoop (MapReduce, HDFS, Common)
  • Apache HBase
  • Apache ZooKeeper
  • Apache Oozie
  • Apache Hive
  • Hue (Apache licensed)
  • Apache Flume NG
  • Cloudera Impala (Apache licensed)

You are using Cloudera Manager (Free Edition) to install and configure your system.

I specify cloud[1-3] and get the following results:

Expanded Query Hostname (FQDN) IP Address Currently Managed Result
cloud1 No Host ready: 9 ms response time.
cloud2 No Host ready: 7 ms response time.
cloud3 No Host ready: 16 ms response time.

While it took a few tries I finally got the following:


So now It asks me decide which CDH4 services I should install. I pick core hadoop for my first attempt withan embedded PostgreSQL database setup:

Database Host Name: Database Type: Database Name : Username: Password:
cloud0:7432 PostgreSQL hive hive aflhU8ZThz

and all defaults for the rest. 13 steps later  and viola:


Now What

cm Ok so its installed, and we can see. I guess I have to spend some time installing my parts and working with this version to see what happens and how it behaves. But thats for another day.

iPad – DellLaptop + iPod – Blackberry + iPhone + MacBookPro = iPerson

From my trip to San Francisco, Dreamforce 2010

Monday December 6th, 2010

So for this trip I took both my iPad and my laptop. I finally signed up and powered on the AT&T wireless cellular plan (the cheapest one) as I am sure 30,000 people on wifi may not really work all the time. I have every intention of trying to not really use the laptop (if I can) we shall see.

I am not loving the WordPress app on my iPad. Sitting here, 7 miles high tapping away, sharing my thoughts on a westbound AA flight to the San Francisco high-tech homeland to attend a “global gathering” feels so familiar, and yet so different. A simple finger brush off of the writing surface and my unsaved local draft has vanished (for the second time). Oh well, more time to kill while I retype it in. I guess it must be a feature of “airplane mode”.

On another note, i got this free book on my iPad and have been reading about quantum physics and reality, awesome stuff. No wonder I often feel lost in space.

Tuesday December 7th, 2010

So the conference is off In full swing and today I decide to carry both devices. Of corse, my bag is now extra heavy. Good thing I did a full workout this morning. All morning I use my iPad, writing, tweeting, chatting answering emails, using apps like clipboard (my new best friend). Connectivity is somewhat spotty. Sometimes I can get a signal, sometimes it fades and I switch to cellular (which also works 50/50). The only missing element is a camera. Ok I still carry a blackberry and it has one so a quick email and were in business. I wonder if I can pair the two using Bluetooth. Have to try that later.

Now it’s afternoon and I am listening to a session on what’s coming in the next version of chatter. The dude speaking is the primary developer so at this point I whip out my laptop and load up my dev environment so that I can make noted and code changes right here. Sorta glad I had the laptop cause typing code snippets on the iPad is a little painful. It is evident that if I am going to use this as mobile device I probably need to invest in some sorta keyboard.

Wednesday December 8th, 2010

Today I decide to leave the laptop in the hotel. This causes me issues on many fronts. First, I am paranoid and am always concerned about theft. So I feebly hide my laptop in some dirty clothes in my suitcase cause it make me feel better and I am off to the races. I get a good seat at the keynote and establish connectivity early. The keynote starts about 15 minutes late and guess what happens. People are tweeting about things that have not yet happened. Talk about great marketing, poorly executed. Not too many people really notice or if they do they don’t think its real important.

Later in the day, as Stevie Wonder and Bill Clinton speak having the iPad to watch the feeds in real time is awesome.

By the way as, even though I am a republican, I will always be a friend of Bill C.

Wednesday December 9th, 2010

Mornings uneventful, conference ends and I am off to the airport. Once again even though there is free wi-fi, it is not as available and clean as it is in NY. So here I sit in the plane home, tapping out this entry. Of corse since I eat with my hands the screen is getting a bit salty. All in all I am convinced, if your not writing code you can probably leave the laptop at home.

If any one has thought on this I would love to hear about it. In the meantime I think I’ll go back to reading “butterflies are free to fly” by Stephen Davis.

Peace 🙂