April 17, 2015Posted by on
I have always been connected to the water. Staring out of my office window, gazing over downtown New York, I sometimes think about the what the city looked like during mid 19th century “Gangs of New York” times. I imaging myself as a dockworker loading and unloading ships only to realize I am a stevedore even now.
For the past year, many of the projects that I have engaged in have been about developing cloud delivered containers for large scale applications. The work allows me to maintain a sharp focus on a “Big Things” agends while continously delivering useful and reusable components. To me the cloud is a reborn technology that encourages better architecture. The cloud of the now is simply understood as “Compute, Storage and Networking”; pure constructs in the cyberspace of Gibsons, Neuromancer.
Docker… Docker… Docker… is the enabling technology that brings the devops estibador, super-scripting-power. The question is: how do we unravel the bigger messes and produce cloud native components that are robust but simple. Hopefully the answer will materialize soon enough
July 18, 2014Posted by on
I have always been a scripter. To me scripting is the glue that binds the universe together. I have claimed that I am like a classic composer, creating beautiful symphonies out of concepts and code. Each movement, designed to evoke strong emotional reaction.
As a conductor it is script that drives the story home. It brings together a prelude, body and epilog in a way that allows us to look at something of interest in a holistic manner.
Scripting is often explained as the nirvana of high-level programming that is both abstract and concise, devoid of difficut low-level details (memory, performance, etc…)
Scripting appeals to the “hacker” element in our collective conscience by providing us with the
ability to drive a project’s design and development. There is a famous column by a well known
language designed Paul Graham entitled hackers and painters, where he says:
“What hackers and painters have in common is that they’re both makers. Along with
composers, architects, and writers, what hackers and painters are trying to do is
make good things. They’re not doing research per se, though if in the course of trying
to make good things they discover some new technique, so much the better.”
So this in a nutshell describes a core premise of my own working life.
“Sometimes what the hackers do is called “software engineering,” but this term is just as misleading.
Good software designers are no more engineers than architects are. The border between
architecture and engineering is not sharply defined, but it’s there. It falls between what and how:
architects decide what to do, and engineers figure out how to do it.”
So a final thought, script; as it relates to life the universe and everything. For many years I have been a fan of one of the more interesting authors of our times, the late great Douglas Adams who somehow had a finger on the british humorist version of computing where the earth is itself is hadoop
relational humor (:
scott adams (dilbert), douglas adams (hhgh), douglas cutting (hadoop)
And thinking along those lines are do we live our lives by acting out our life-script.
Is our life-script instantiated at birth as:
var script = new life();
for(var yearofyourlife=0; yearofyourlife < 100; yearofyourlife++)
March 29, 2013Posted by on
“Spring is the time of plans and projects.” – Leo Tolstoy (Anna Karenina)
As a software developer with a background in hardware it is always interesting and often painful to setup a new work environment. As it happens to be a sunny spring day (good friday almost easter) it seem that the theme of renewal is here. Coincidentally my old Dell precision workstation (circa 2004, perhaps the first 64-bit machine i ever used) has finally died.
Over the past few years I have bee running Ubuntu as my primary desktop working environment. This is probably the reason I to so much milage out of the old system. Each time someone else’s Dell precision workstation died, I would cannibalize the good parts and add them to my system. Net result is that for a grand total of $65 (for a galaxy force 6500 video card, an old but awesome card appropriate for the old system) I had a great 2 dual core cpu’s (4 total), 24G memory and about 1 terabyte of disk space dual monitor developer workstation. So what happened? One of the three old fans died probably about a month ago but ubuntu didn’t care, it kept on ticking. eventually I needed to use the dvd drive for something and noticed it was dead, so eventually I powered down the system to check it out. And that was all she wrote. Funny thing is that although it was obvious that some of the insides had issues, the hard drives (2 beautiful SATA drives) were just fine. Oh well.
So I removed the good parts and scrapped the old system and ordered a new one. You know what, sometimes new is really better. I ordered up a Dell Optiplex 7010. with 3rd Gen Intel Core I7 (quad core, bug cache, real fast) so basically a 8 cpu machine with 16G of memory, 2TB disk, no monitors no fluff for about $800 bucks. It came in today so I ware ready to go to work on it.
On boot I got a brand new super clean version of window 7 running. I quickly configured the dual monitors and that I thought about how to configure the machine for real. After think about it for a while I decided there was no real reason torero what was done and as long as I am vigilant in the way I use the host (in this case windows 7) it should service as a great platform for development. So I painfully located all my windows application license codes for the basic stuff. For me this means Google Chrome, Dropbox. Office 2010 (64-bit), Adobe Acrobat and virtualization software (I typically use virtual box). A few hours I finish installing the latest UBUNTU as a guest and start using it. AND it sucks. What—–
It just doesn’t feel right to me. Several discussions later and I realize it. Linux or OS X on the bottom is always the way to go. So now what. I guess I can use something to suck up my windows drive and virtualize it. Or may be I should go right ahead and boot Linux from a different physical drive. Then I can use the actual physical drive (with windows as the guest) and maybe that won’t suck at all. By the way I’ve also had a great deal of success using the latest wine stuff, so maybe I can look at that as a another option.
Funny thing about the story is that while I sit here waiting for all the file transfer/shuffling to finish (I needed to rearrange some of the stuff in disk) I am typing on my mac using its notes app. I really just use my mac and don’t ever really think about it. So thats it for now I wait:
“Expect to have hope rekindled. Expect your prayers to be answered in wondrous ways. The dry seasons in life do not last. The spring rains will come again.” – Sarah Ban Breathnach
March 19, 2013Posted by on
I am often asked, what do you do? I typically respond, I am in computers. I have no idea why this is an appropriate response but in the beginning it was quite literally true
I learned beginners programming (BASIC) in 1977, where the Dec-Writer was king and the ribbon was always broken. You literally got your hands dirty if you wanted to get anything done. I then took an after school job at Oliver Office Equipment Company, an original typewriter repair company that sold and delivered office supplies.
The ugly part of fixing typewriters was crawling around the basement of the store to find a replacement part from the hundreds of broken buybacks. I was just beginning to learn how to repair the IBM Selectric (with its million or so unique parts) when the first shipment of Kaypro II computers arrived. As an employee, I got the 50% discount and my first real box for about $850.00. I still have this computer on my window sill and it still works (although half of the 360K, 5 1/4″, single sided, single density are de-maged). Taking that bad boy apart and putting it back together was alway fun.
Then there was my work study job at Syracuse University with the campus computer services department. Day 1, I was given a cool briefcase full of tools and handed a bunch of trouble tickets and sent out to fix terminals. About half of the tickets were to replace ribbons on old Dec-Writers around campus and the other half was reseating video cards in volker-craig terminals. Then there was the day where a friend of mine tried to replace a card on terminal where he had inappropriately grounded the static discharge. And calling the ambulance after he flew across the room convinced me it was probably time to get out of being in computers.
So what do I do? I read, I write, I hack, I invent, I think and I teach. Creativity is a part of who I am. As an mathematical artist, I find beauty in music. I love its patterns and symmetry. I am interested in the layers and complexity that I hear and feel effortlessly. The math of technology is harder for me. I love simple logic, truths, repeatable certainty. I am fearful of complexity in that it can be a huge distraction. I am a fan of good field position, keeping your eye on the goal, and getting it done and while I dispise the notion of “good enough”, as an adult I understand it. The essence of Zen is on not identifying with one thought or its opposite, it is about getting to the awareness that is behind the thought.
I leave you with this thought: A distraught man approached his master. “Please, Master, I feel lost, desperate. I don’t know who I am. Please, show me my true self!” But the teacher just looked away without responding. The man began to plead and beg, but still the master gave no reply. Finally giving up in frustration, the man turned to leave. At that moment the master called out to him by name. “Yes!” the man said as he spun back around. “There it is!” exclaimed the master.
March 15, 2013Posted by on
For the last few months I have been building a prototype on top of an Apache Hadoop 1.0.4 cluster that I built from scratch out of six virtual machines running Ubuntu Server 12.04.2 LTS. It has been an interesting experience. Simply put, this is the actual learning process that every hacker goes through on every new project whether its a programming language, platform or technology. So now that I got a handle on the basics and I can take an earnest look at other peoples packaging.
Today I am checking out the current offering from Cloudera. I found the download named Clouder Manager 4.5 Free Edition, and proceeded with the installation. Of course I need to install it on a few nodes so I am back to setting up some more servers.
This time I decide to use my mac pro server configured with virtual box. I planned on running a three server cluster (cloud1,cloud2,cloud3) so I set it up and run into a few networking problems. I get my ops dept to fix my port to allow for multiple mac addresses. Here are some of the issues and solutions I encountered when setting up the environment:
For each cloned virtual server I needed to change (persistently) its host name and mac address. The tools ( virtual box in this case ) should have properly handled this. It did NOT. So I did the following hand job on each machine.
- sudo vi /etc/hosts
- sudo vi /etc/hostname
(remove cloud 127.0.0.1 definition from each)
- sudo vi /etc/dhcp/dhclient.conf
- sudo rm /etc/udev/rules.d/70-persistent-net.rules
sudo mkdir /etc/udev/rules.d/70-persisitent-net.rule
(thank you Peter Mount)
Install Cloudera Manager (Free Edition)
So my first installation was from my remote desktop linux to my cluster and it failed. I then decided to allocate another local instance (cloud0) and try again. The installer runs ok and i point my web browser at http://cloud0:7180, login as admin/admin and away we go:
This installer will deploy the following services on your cluster:
- Apache Hadoop (MapReduce, HDFS, Common)
- Apache HBase
- Apache ZooKeeper
- Apache Oozie
- Apache Hive
- Hue (Apache licensed)
- Apache Flume NG
- Cloudera Impala (Apache licensed)
You are using Cloudera Manager (Free Edition) to install and configure your system.
I specify cloud[1-3] and get the following results:
|Expanded Query||Hostname (FQDN)||IP Address||Currently Managed||Result|
|cloud1||cloud1.ibi.com||172.30.240.110||No||Host ready: 9 ms response time.|
|cloud2||cloud2.ibi.com||172.30.240.111||No||Host ready: 7 ms response time.|
|cloud3||cloud3.ibi.com||172.30.240.112||No||Host ready: 16 ms response time.|
While it took a few tries I finally got the following:
So now It asks me decide which CDH4 services I should install. I pick core hadoop for my first attempt withan embedded PostgreSQL database setup:
|Database Host Name:||Database Type:||Database Name :||Username:||Password:|
and all defaults for the rest. 13 steps later and viola:
April 24, 2012Posted by on
So today, I woke up as usual, ran thru the morning rituals, ate breakfast, grabbed the keys and phone and out the door I went, A very typical day. I jumped on the train, found a seat and then started my work day (I get 1 hour each way, 5 days a week, 4 weeks a month. So I basically work an additional 40 hour week on the train).
First I check my phone and notice it is off. I try to boot it a few times and realize, no power, way lo battery as it flickers off. Who knows why, maybe the charger wasn’t really plugged in or maybe some-one switch batteries on me (again!). Like I said a typical day. So I pull out the laptop, as this is Ubuntu 12.04 LTS release time, now is a great time to play with it. As I open the lid, I notice that it is powered off. Like the phone, I try to boot and cant get passed the bios check. Like the phone, another device is dead again. Next up I the iPad and it’s batter meter is red with 5% left. I try using it a but but shortly device #3 is doa. So I pull out my workout iPod nano, at least I can tune out the trip. Guess what, another fail.
Holy moly, did we have an EMP burst? So here I sit on the electric train with no electronics to use. Perhaps its time to read. I check my bag and no books, no mags, no papers except an old paycheck envelope. So I grab a pen and begin to write, Old school. Why does the simple act of writing with a pen seem like such a luxury in 2012. Here I sit on a quiet train, traveling with my thoughts and the time to compose and record them by hand (which is beginning to hurt since its been a long time since I’ve written more then a signature). As I look around I notice I am the only one with a pen, in hand. Everyone is glaring at the tiny devices; tablets, phones, laptops, etc. I am surrounded by energy and yet I have none of my own. I guess this is the real reason why, I don’t like Mondays. Ive always know that I had no real power. It’s a good thing we don’t need to remember to recharge ourselves.
Live Long and Prosper
June 17, 2011Posted by on
Last week I spent three days attending a Learning Tree course entitled “Agile Project Management With Scum” Course 918.
I took this course because I was looking for some validation on what my scrum team had been doing and I figured that this was an easy way to achieve it. And anyway the company had some education vouchers so it was pretty easy to arrange.
Getting up real early and riding the subway downtown had some real surprises. First of all, I had forgotten that if you leave the house real early, then parking at the station is easy as there are many many spots. Also the subway downtown pre-rush hour is pleasant, although the extra 15-20 mins each way kinda sucks. But the biggest thing that I had forgotten is how simply wonderful lower Manhattan is in the morning.
The facility is located at One New York Plaza-31st Floor. This spot is across the street from South Ferry, and has a great view of the waterfront including Governors Island, the Statue of Liberty, and the rest of the harbour. I have not really spent much time downtown since the summer before 9-11, so this was a pleasure. I roamed around quite a bit and kept thinking about the “Gangs of New York“.
The course was lead be a guy named Maurice Hagar who was reasonably well trained as a “professional teacher”. These classes always feel like they are a little heavy in the marketing aspects of “professional education” but I guess everyone needs to make a buck. There were at least 25 attendees and while the information disseminated can be learned from books there is nothing like being forced into participation with a group of strangers. I had initially thought that I was going to hate this, but it turned out to be really ok. And the whole experience was worthwile.
Sometime agile is referred to as fragile. In the end the all we need to do to tighten up our current agile process is to get a little stricter with out adherence to the principles and maybe do a little less scrum but.
December 11, 2010Posted by on
From my trip to San Francisco, Dreamforce 2010
Monday December 6th, 2010
So for this trip I took both my iPad and my laptop. I finally signed up and powered on the AT&T wireless cellular plan (the cheapest one) as I am sure 30,000 people on wifi may not really work all the time. I have every intention of trying to not really use the laptop (if I can) we shall see.
I am not loving the WordPress app on my iPad. Sitting here, 7 miles high tapping away, sharing my thoughts on a westbound AA flight to the San Francisco high-tech homeland to attend a “global gathering” feels so familiar, and yet so different. A simple finger brush off of the writing surface and my unsaved local draft has vanished (for the second time). Oh well, more time to kill while I retype it in. I guess it must be a feature of “airplane mode”.
On another note, i got this free book on my iPad and have been reading about quantum physics and reality, awesome stuff. No wonder I often feel lost in space.
Tuesday December 7th, 2010
So the conference is off In full swing and today I decide to carry both devices. Of corse, my bag is now extra heavy. Good thing I did a full workout this morning. All morning I use my iPad, writing, tweeting, chatting answering emails, using apps like clipboard (my new best friend). Connectivity is somewhat spotty. Sometimes I can get a signal, sometimes it fades and I switch to cellular (which also works 50/50). The only missing element is a camera. Ok I still carry a blackberry and it has one so a quick email and were in business. I wonder if I can pair the two using Bluetooth. Have to try that later.
Now it’s afternoon and I am listening to a session on what’s coming in the next version of chatter. The dude speaking is the primary developer so at this point I whip out my laptop and load up my dev environment so that I can make noted and code changes right here. Sorta glad I had the laptop cause typing code snippets on the iPad is a little painful. It is evident that if I am going to use this as mobile device I probably need to invest in some sorta keyboard.
Wednesday December 8th, 2010
Today I decide to leave the laptop in the hotel. This causes me issues on many fronts. First, I am paranoid and am always concerned about theft. So I feebly hide my laptop in some dirty clothes in my suitcase cause it make me feel better and I am off to the races. I get a good seat at the keynote and establish connectivity early. The keynote starts about 15 minutes late and guess what happens. People are tweeting about things that have not yet happened. Talk about great marketing, poorly executed. Not too many people really notice or if they do they don’t think its real important.
Later in the day, as Stevie Wonder and Bill Clinton speak having the iPad to watch the feeds in real time is awesome.
By the way as, even though I am a republican, I will always be a friend of Bill C.
Wednesday December 9th, 2010
Mornings uneventful, conference ends and I am off to the airport. Once again even though there is free wi-fi, it is not as available and clean as it is in NY. So here I sit in the plane home, tapping out this entry. Of corse since I eat with my hands the screen is getting a bit salty. All in all I am convinced, if your not writing code you can probably leave the laptop at home.
If any one has thought on this I would love to hear about it. In the meantime I think I’ll go back to reading “butterflies are free to fly” by Stephen Davis.
November 24, 2010Posted by on
So it’s thanksgiving day 2010 and we are all still here, plugged into the matrix, living out our lives, growing more dependent on technology then ever before. As I sit on the long island railroad headed east, I peck out this post on my IPad, listening to Coldplay (the escapist) on my iPod, surrounded by other iPeople and their devices. Communicating constantly, tapping or talking, looking for instant amusement. I carry no less then 5 separate devices with global connectivity capability and yet I feel more alone and isolated then ever. And I am sure it will only get worse. I remember reading the Arthur C. Clarke novel 2010, as a boy and thinking how far away that was. This year my father turned 80 and he told me when he looked in the mirror he was happy to see his father smiling back. Anyway, I remember thinking about the line, all these worlds are yours and how I never considered the concept as applied to inner space. Now I simply think well maybe something is going to happen, something wonderful. Its good to be a dad.
So I am thankful, for my family, for health and happiness and hockey and grapes. Gobble-Gobble!
Ps I stumbled across the greatest secret in the world, so I thought I’d share…
October 29, 2010Posted by on
So this summer I finally broke down and purchased some new backyard furniture. My wife and I went to the Macy*s Outdoor Furniture shop and bought a very nice, “fairly priced” set. Hayley opened a new account for this purchase that included special financing. Of course Macy*s promised speedy delivery. Of course they failed on that. So to placate us Macy*s offered and then sent a $200 Gift Card as an I’m sorry.
I never shop, even though I work in Manhattan (across from Macy*s). Hayley, my wife insisted that I take the gift card and buy myself some clothes. So, I plopped the card into my wallet and forgot about it till this morning when my she made a little remark on how the bottoms of my pants were ripped.
So today at lunchtime I made my way into Macy*s. I picked out 4 pairs of pants. I tried them all on. I had plans to buy more but it had already been 45 mins and I had enough of the “shopping” experience. I go to a register to check out. And like magic 2 pairs of the pants were on sale, how random, how lucky. I go to pay with the gift card and guess what, the clerk says: Sir, this card has $2.46 cents on on it. After a fair amount of discussion (loud angry discussion), I decide to just pay. The clerk then informs me to get the sale price I must use my Macy*s credit card which I do not have in my pocket. He tries to use the automatic lookup system but it fails. So they call someone and tell me the card is in my wife’s name and I am not authorized. At this point I get angry, I have had enough I am done. But after a small rant I apologize to the clerk and ask for a manager. The clerk politely explains I need customer service to help me straighten this out and I should see the girl upstairs at the desk. I find the girl by the front door and explain it all to her. She tell me I should go up to the 9th Floor executive offices and find someone up there to straighten it out.
I decided that I am too hungry to continue with those retail nonsense. So now Hayley is on the phone with Macy*s credit. I cant wait to see how this turns out. Well see who wears the pants😉
Turns out the gift card was used before it was sent to us in topanga, California. A place I’ve never been. More fraud, more fun. Still even though they will send me another card, I have no pants. Oh well!