Marc J. Greenberg

Codemarc's Blog

Monthly Archives: March 2013

A Fresh Start?

“Spring is the time of plans and projects.” – Leo Tolstoy (Anna Karenina)

myoldell
As a software developer with a background in hardware it is always interesting and often painful to setup a new work environment. As it happens to be a sunny spring day (good friday almost easter) it seem that the theme of renewal is here. Coincidentally my old Dell precision workstation (circa 2004, perhaps the first 64-bit machine i ever used) has finally died.

Over the past few years I have bee running Ubuntu as my primary desktop working environment. This is probably the reason I to so much milage out of the old system. Each time someone else’s Dell precision workstation died, I would cannibalize the good parts and add them to my system. Net result is that for a grand total of $65 (for a galaxy force 6500 video card, an old but awesome card appropriate for the old system)  I had a great 2 dual core cpu’s (4 total), 24G memory and about 1 terabyte of disk space dual monitor developer workstation. So what happened? One of the three old fans died probably about a month ago but ubuntu didn’t care, it kept on ticking. eventually I needed to use the dvd drive for something and noticed it was dead, so eventually I powered down the system to check it out. And that was all she wrote. Funny thing is that although it was obvious that some of the insides had issues, the hard drives (2 beautiful SATA drives) were just fine. Oh well.

optiplex7010So I removed the good parts and scrapped the old system and ordered a new one. You know what, sometimes new is really better. I ordered up a Dell Optiplex 7010. with 3rd Gen Intel Core I7 (quad core, bug cache, real fast) so basically a 8 cpu machine with 16G of memory, 2TB disk, no monitors no fluff  for about $800 bucks.  It came in today so I ware ready to go to work on it.

On boot I got a brand new super clean version of window 7 running. I quickly configured the dual monitors and that I thought about how to configure the machine for real. After think about it for a while I decided there was no real reason torero what was done and as long as I am vigilant in the way I use the host (in this case windows 7) it should service as a great platform for development. So I painfully located all my windows application license codes for the basic stuff. For me this means Google Chrome, Dropbox.  Office 2010 (64-bit), Adobe Acrobat and virtualization software (I typically use virtual box). A few hours I finish installing the latest UBUNTU as a guest and start using it. AND it sucks. What—–

It just doesn’t feel right to me. Several discussions later and I realize it. Linux or OS X on the bottom is always the way to go. So now what. I guess I can use something to suck up my windows drive and virtualize it. Or may be I should go right ahead and boot Linux from a different physical drive. Then I can use the actual physical drive (with windows as the guest) and maybe that won’t suck at all. By the way I’ve also had a great deal of success using the latest wine stuff, so maybe I can look at that as a another option.

Funny thing about the story is that while I sit here waiting for all the file transfer/shuffling to finish (I needed to rearrange some of the stuff in disk) I am typing on my mac using its notes app. I really just use my mac and don’t ever really think about it.  So thats it for now I wait:

“Expect to have hope rekindled. Expect your prayers to be answered in wondrous ways. The dry seasons in life do not last. The spring rains will come again.” – Sarah Ban Breathnach

What do you do?

I am often asked, what do you do? I typically respond, I am in computers. I have no idea why this is an appropriate response but in the beginning it was quite literally true

dc_la120

I learned beginners programming (BASIC) in 1977, where the Dec-Writer was king and the ribbon was always broken. You literally got your hands dirty if you wanted to get anything done. I then took an after school job at Oliver Office Equipment Company, an original typewriter repair company that sold and delivered office supplies.
The ugly part of fixing typewriters was crawling around the basement of the store to find a replacement part from the hundreds of broken buybacks. I was just beginning to learn how to repair the IBM Selectric (with its million or so unique parts) when the first shipment of Kaypro II computers arrived. As an employee, I got the 50% discount and my first real box for about $850.00. I still have this computer on my window sill and it still works (although half of the 360K, 5 1/4″, single sided, single density are de-maged). Taking that bad boy apart and putting it back together was alway fun.

Then there was my work study job at Syracuse University with the campus computer services department. Day 1, I was given a cool briefcase full of tools and handed a bunch of trouble tickets and sent out to fix terminals. About half of the tickets were to replace ribbons on old Dec-Writers around campus and the other half was reseating video cards in volker-craig terminals. Then there was the day where a friend of mine tried to replace a card on terminal where he had inappropriately grounded the static discharge. And calling the ambulance after he flew across the room convinced me it was probably time to get out of being in computers.

So what do I do? I read, I write, I hack, I invent, I think and I teach. Creativity is a part of who I am. As an mathematical artist, I find beauty in music. I love its patterns and symmetry. I am interested in the layers and complexity that I hear and feel effortlessly. The math of technology is harder for me. I love simple logic, truths, repeatable certainty. I am fearful of complexity in that it can be a huge distraction. I am a fan of good field position, keeping your eye on the goal, and getting it done and while I dispise the notion of “good enough”, as an adult I understand it. The essence of Zen is on not identifying with one thought or its opposite, it is about getting to the awareness that is behind the thought.

I leave you with this thought: A distraught man approached his master. “Please, Master, I feel lost, desperate. I don’t know who I am. Please, show me my true self!” But the teacher just looked away without responding. The man began to plead and beg, but still the master gave no reply. Finally giving up in frustration, the man turned to leave. At that moment the master called out to him by name. “Yes!” the man said as he spun back around. “There it is!” exclaimed the master.

Big Data Floating in the Clouds

For the last few months I have been building a prototype on top of an Apache Hadoop 1.0.4 cluster that I  built from scratch out of six virtual machines running  Ubuntu Server 12.04.2 LTS. It has been an interesting experience. Simply put, this is the actual learning process that every hacker goes through on every new project whether its a programming language, platform or technology. So now that I got a handle on the basics and I can take an earnest look at other peoples packaging.

Today I am checking out the current offering from Cloudera. I found the download named Clouder Manager 4.5 Free Edition, and proceeded with the installation. Of course I need to install it on a few nodes so I am back to setting up some more servers.

Cluster up

This time I decide to use my mac pro server configured with virtual box. I planned on running a three server cluster (cloud1,cloud2,cloud3) so I set it up and run into a few networking problems. I get my ops dept to fix my port to allow for multiple mac addresses. Here are some of the issues and solutions I encountered when setting up the environment:

For each cloned virtual server I needed to change (persistently) its host name and mac address. The tools ( virtual box in this case ) should have properly handled this. It did NOT. So I did the following  hand job on each machine.

  1. sudo vi /etc/hosts
  2. sudo vi /etc/hostname
    (remove cloud 127.0.0.1 definition from each)
  3. sudo vi /etc/dhcp/dhclient.conf
  4. sudo rm /etc/udev/rules.d/70-persistent-net.rules
    sudo mkdir /etc/udev/rules.d/70-persisitent-net.rule
    (thank you Peter Mount)

Install Cloudera Manager (Free Edition)

So my first installation was from my remote desktop linux to my cluster and it failed. I then decided to allocate another local instance (cloud0) and try again. The installer runs ok and i point my web browser at http://cloud0:7180, login as admin/admin and away we go:

This installer will deploy the following services on your cluster:

  • Apache Hadoop (MapReduce, HDFS, Common)
  • Apache HBase
  • Apache ZooKeeper
  • Apache Oozie
  • Apache Hive
  • Hue (Apache licensed)
  • Apache Flume NG
  • Cloudera Impala (Apache licensed)

You are using Cloudera Manager (Free Edition) to install and configure your system.

I specify cloud[1-3] and get the following results:

Expanded Query Hostname (FQDN) IP Address Currently Managed Result
cloud1 cloud1.ibi.com 172.30.240.110 No Host ready: 9 ms response time.
cloud2 cloud2.ibi.com 172.30.240.111 No Host ready: 7 ms response time.
cloud3 cloud3.ibi.com 172.30.240.112 No Host ready: 16 ms response time.

While it took a few tries I finally got the following:

cm3

So now It asks me decide which CDH4 services I should install. I pick core hadoop for my first attempt withan embedded PostgreSQL database setup:

Database Host Name: Database Type: Database Name : Username: Password:
cloud0:7432 PostgreSQL hive hive aflhU8ZThz

and all defaults for the rest. 13 steps later  and viola:

cm8

Now What

cm Ok so its installed, and we can see. I guess I have to spend some time installing my parts and working with this version to see what happens and how it behaves. But thats for another day.