Posted on

Cert Addict!

Wow.  I cannot believe it has almost been two months since the last post.  That is a fair proportion of my two year target.

So to recap, it is interesting to look back on the last post.  For example, I did pick up AWS SysOps again and did this:

My very top secret project has progressed up to a point but stalled when it came to building a UI.  And also – guess what!  – I had another epiphany or lights-on moment, whatever you want to call it.  Given the nature of the application it struck me that that the whole thing would be best suited to a serverless-Lamdba-API Gateway-S3 architecture.  I am now a Certified Architect so I can do these things.  It is not that big a step.  Many of my Python routines can be pulled into Lambda as-is.

This does mean of course that I will be moving from EC2 to Serverless before serving any real customers.

It also means I cannot reveal the brilliant idea and name for the application just yet.

Certification Addict

I have decided to conclude my year of AWS Certification hoarding by doing the Associate Developer exam.  There is a lot to be leveraged from knowledge learnt on previous exams so not leaving it too late is an advantage.  I also think that the topics will support my various projects nicely.  In fact it was while going through API Gateway modules that I decided to rework my very top secret project.  Yes, I know I am a certification junkie.

After that, I think I will give AWS studying a rest for a while.  The next stage up hurts my head thinking just thinking about it.

DevOpsDays London 2018


I attended my first DevOpsDays in September.  Will definitely do it again.  Very chilled,  great talks – much more varied than I had anticipated.  I really enjoyed the Open Spaces too.  I will add links to my favourite talks in My Picks below.

I was particularly inspired by a lightning talk on how some deliberately over-engineered their profile web-site in order to learn as many technologies as possible.  This is a sentiment I share.  Most of the things I work on privately are done this way.

My Picks

I thought I had read and seen everything about the making of 2001: A Space Odyssey so was surprised to enjoy this book as much as I did.  

“At this point it dawned on him that the leopard had not seen him in the monkey suit before” was one go the funniest lines I have read in a while.  Well recommended.

I also saw Tina: The Musical which gives me an excuse to share gifs like this:

Finally, links to DevOpsDays talks I enjoyed:

Overengineering Your Personal Website – How I Learn Things Best – Jamie Tanna – here.

Neurodiversity and the Essence of DevOps – Jeff Sussna – here.

Why are Distributed Systems so hard? A network partition survival guide – Denise Yu – here.

Tickets and Silos Ruin Everything – Damon Edwards – here.

Posted on

Start With The API

A Pet Project

I have just come to the end of a holiday during which I used a lot of downtime (in very pleasant environments) to start a small dev project. I will provide some more details in a future post of exactly what I am trying to do.

As you will have seen in previous posts, I have been meaning to move off the studying and reading track and onto an actual building and doing track. I have chosen my project well. It is small enough to move quickly but complex enough to introduce a number of different of technologies.

Learning From Doing

Here are some of the things that I have learned to do over the past few weeks.

  • I have used Python and Flask to expose some simple API’s
  • I have served those API’s from a Docker container both locally and on AWS. This means that the API’s can be hit from anywhere on the Web
  • I wanted to add persistence so used AWS DynamoDB both locally and with the managed service
  • I have used Boto to call AWS API’s from my Python code.

It crossed my mind whilst doing all this that maybe I was just hacking. Where exactly does this fit into DevOps practices and my Mission?  My conclusion – I have been following good CI/CD principles (although my tests need some TLC):

  • I have checked everything in to a remote GitHub repo – Dockerfiles, test files, Python code
  • I have pushed Docker images onto Dockerhub
  • I have a pipeline. Update code, build new image, test locally, push image to DockerHub, pull image to an EC2 instance and run from there.   Very manual but simple
  • I have regularly destroyed everything and rebuilt everything using Dockerfiles and Python scripts to do things like recreate DynamoDB tables. Not quite full automation, but very easy to set up elsewhere.

Additionally I have thought hard about how to approach iterations of my code so as to avoid breaking things and to be very quickly able to build and test an end product rather than diving in and changing 10 things at once and then spending hours making it work again.

I have been making sure my code is in a releasable state at all times. An example would be using a variable to point to an in-code list and then pointing it to a database equivalent so that I could easily switch back if it didn’t work. Then when the database was working, delete the in-code list.

Two Key Learning Points

Predictably, it is amazing how much more you learn when actually doing things rather than reading about them. Examples include the difference between using IAM roles and AWS access keys or the flexibility of a managed service NoSQL database.

Secondly, and this is where the title came from, I have started with the APIs. Rather than thinking of an app, I designed a set of API’s before starting and worked out how I wanted each to behave. I have made sure that they are independent and stateless so they can be assembled and called in different ways. At some point I will build a lightweight page that will draw them together in some meaningful way.

This is very different to my old iOS hacking days. I also intend to split the API’s out into their own independent containers which will give me a real world example to play with orchestration tools.

What About AWS SysOps?

I have partially parked this as I was unable to find time to do the exam before my holiday. However, I have created a GitHub repo where I am building cheat sheets.

I have just found out that there is a late September deadline, after which the course changes which means I will have to pick this up again soon.

My Picks

I have been so ensconced in the above, going back to work and family matters that I don’t have too much to add here. I still love GLOW. I still love The Replacements. And here’s a clue, my pet project is very Madonna-centric! More to follow in a future blog.

My other favourite thing recently has been watching Stewart Lee’s Content Provider show. I saw it live but watching it on TV made me laugh just as much second time around.

I have also seen David Byrne live for the first time ever and was blown away. Going again.

Finally, for those studying AWS, I thoroughly recommend A Cloud Guru – they can be found here.

Posted on

Reflections

So clearly an inspirational post title is not happening today.  Still, “Reflections” sounds very bloggy does it not?  For some reason I thought my time was almost up, but I checked back to my first post and realised I have a few months to go.  I also reflected, again, without success on how I would know if I had succeeded.

Two months since the last post!  What have I been up to?  Well reading as always.  I did read Accelerate – some great lessons about how to activate culture changes (do it by changing the things you do, not by imposing culture change itself).

I also appeared to be one of a handful of people left who hadn’t read The Goal so I read that too.  As with many others I found it insightful.  In particular the section where a machine was operating on parts that were already defective reminded me there is financial and processing cost if quality issues are not detected as early as possible.  Baking in quality early is not just about shorter feedback cycles and reducing fix costs.

I have been studying too.  I have decided to take on AWS SysOps certification.  Yes I know.  #certificationJunkie.  However I may have run out of time to finish that before a rapidly approaching holiday.

I have decided to have some fun too.  My reading list had list expanded so I read this Dr Who novel and, 30+ years since I first read it, Do Androids Dream of Electronic Sheep.

What I haven’t done is build anything for a while.

TV and DevOps

I have also been indulging in box sets.  Westworld 2 surprised me by being way better than I thought it would be.  It made heavy weather of getting there, but the finale was delightful.  Their attitude towards operability and high availability left a lot to be desired.  <<SPOILER ALERT>> The humans had all of their critical data in a single data centre.  They deserved all they got.

My Picks

I have also just finished GLOW which hooked me completely. One episode in particular had me in bits.  Might chain watch it again.  Alison Brie – I salute you!  Perfect excuse for a Zoya The Destroya gif.

Musically, it seems one listen of any Replacements song is enough to have it pinging around my head for days.  This includes some of the outtake bonus songs too – Birthday Gal and Valentine being two recent examples.

Finally happy 60th birthday Madonna!  I salute you too.

Posted on

Increasing Post Cadence and Python

Upping my blog post cadence then.  This even briefer than usual.  I am thinking that for the record I should say what I am doing.

Firstly, I am back on the Python trail, spending train journeys going through books and exercises.  I decided to restart the book after stopping a few months ago.  This went better than expected.  I had forgotten less than I thought and expect to be moving into new territory this week.

Secondly, the Accelerate book turned up so I look forward to spending a bit of reading time on that too.

Finally, I attended the AWS London Summit this week.  Not the best AWS day I have spent, but did get to see Dr Werner Vogels speak.

My Picks

Really enjoying some music documentaries.  Shane Meadows’ Made of Stone always reminds me of how brittle and brilliant The Stone Roses and why I love them so much still.

Posted on

Happy Birthday Mission!

Well, it’s oh so quiet here.  So quiet that it escaped me until this week that my mission is one year old.  Eek!  That means there is only year to go.

I have reflected recently on what being a “DevOps practitioner” actually means.  However, at this point I will avoid the navel gazing and crack on.

Since last post, I have become AWS again.  This time as a Certified Solutions Architect.  I am tempted to tackle the sysops exam soon – there appears to be overlaps with what I have been studying but I am feeling a bit revised out.  I am feeling the urge to read – my reading list expands constantly – and to build something.

Over the past couple of weeks I finally finished reading Continuous Delivery.  I am pleased that I am operating on a level where I can understand, value and relate to the material.

Next up, I am spending a couple of days out of office attending this year’s Pipeline conference for the first time.  I have also signed up for free (free!) Kubernetes training provided by Google.    Reading-wise, I will be looking at some Value Stream materials and building-wise I plan to pick up Python again and see what I can do.

My Picks

I enjoyed the latest Food Fight podcast on incident management here.

Away from the mission, I really enjoyed the Pig in a Day course at River Cottage.  Steve Lamb might just be my new favourite person.

Ok, I finally get The Flight of the Conchords.

Posted on

Next Certificates, then Pipelines

Another month flies by without a new post.  In fairness, I have been busy starting work with a new client.

So what has been happening in Mission land?  Well, as seen in my last post, I became AWS Certified as a Cloud Practitioner in January.  It struck me that I had a lot of momentum as well as considerable overlaps in content with other AWS learning paths so I have decided to keep going and study for the a Solutions Architect certificate.  I am planning to take the exam very soon.

I then need to get reading and building again.  New client work means that finishing CD/CI/pipeline/value stream materials are more relevant than ever.

My Picks

I attended an “AWS Builders Day” in January.  Lots of interersting container talks that can be found here and here.  Abby Fuller talks were particularly entertaining.

The LA LA Land soundtrack has been on heavy rotation too.

 

Posted on

Oh Man, So Much to Update on ….

Terraform

Obviously after listing next areas of study in previous post, I would pick up something else entirely – Terraform. Back in James Turnbull land with this marvellous book. I had been looking at Ansible but was put off by all the Vagrant stuff which seemed too anti-container to me.

I have had some crunchy issues to work through. Frustrating at times but I have learnt a hell of a lot working through them – as ever learning a lot of newbie stuff on top.

One such lesson was what happens if you check your AWS access keys into a public Github repo. Turns out you get the attention of AWS and to a lesser extent Github pretty damn quick. Very impressive response particularly as I had no idea what I had done it initially.

As well as another lesson in just how easily an idiot can introduce a vulnerability, I had to figure out how to do the following to get back onto an even keel:

  • rotate IAM access keys
  • remove commits from a public repo (admittedly not really necessary once step above had been completed)

Big hugs to AWS for their response to this issue.

As the Terraform tutorial makes extensive use of Git, this has also been a great way to reinforce my Git skills.

I have realised I am in a space where I have learnt enough to be bold but not enough to avoid doing dangerous things. As with my Docker API faux pas, I am grateful I am doing this on my personal AWS account. Like Luke Skywalker in Empire Strikes Back taking on Vader before finishing his training.  Handy (haha) but ultimately doomed.

Weekly Webinar

Not been great at sticking to this goal, but a colleague referred me to this – particularly relevant to recent work challenges.

Back on SRE

Reading-wise, same colleague recommended the SRE book as a capacity modelling resource. He didn’t know I had a copy. Perfect opportunity to jump forward a few chapters to read about Intent Based Planning.

Black Friday

When trying to find Ansible learning materials that didn’t use Vagrant, I came across Udemy who had some Black Friday deals. I have purchased Kubernetes, Ansible and Python courses.

My Picks

Bladerunner 2049. Just because. I probably love it for all the reasons others don’t.

Curb Your Enthusiasm. Makes Mondays worthwhile. Will miss it when it is done.

Noel Gallagher – Who Built the Moon?. I adore this. And I am not a slavish follower of all things Oasis either. I took my copy of Standing on the Shoulders of Giants back to the shop on the grounds it was pants.

Posted on

What next?

Update from last post.  I have just finished this …

… and this …

The former was wonderful, should have dug into this much more years ago.  I am going to try out the approach of checking in absolutely everything from this point.

The latter was not so good, but did demystify Mongo for me.  Made me realise that demystification of any subject is a great place to start quickly.  Mental note to purchase and read this when available.

Reading wise, I have finished the Bowie book (yeah, I know that doesnt really count but has cleared my reading backlog) and am 25% through Continuous Delivery.

So what next?  In my previous post, I said Kubernetes next.  I may still do that but have enticing Puppet, Terraform and Logstash books to look at.  And there is still the small matter of the Art of Monitoring to return to.  The key thing is to do something that has a hands-on element to keep the learning momentum going.  That in accompaniment to something like Continuous Delivery is a good mix.

Posted on

chmod 600 and helping Bitcoin miners

I have just completed my second pass through The Docker Book.  Running through it again was a good decision.  I flew through it this time and surprised myself at how much I have learnt.

I even learnt what happens when you run the command ‘chmod 600’ on it’s own (a cut and paste error).  Even that turned out to be a positive learning experience as it pushed me down the route (or root haha) of using AWS snapshots and volumes for real to resolve the issue.

This time around I noticed the following text which escaped my attention first time:

The significance of this was not lost on me.  First time around I completed the Docker API chapter and being short of time left the “authenticate your API” chapter until a few days later.  When I came back to it, imagine my surprise at seeing Bitcoin mining containers happily whirring away on my host.

In both of these examples I realised that learning by your mistakes is a powerful learning tool, particularly as I am using my own AWS hosts.  Clearly either of these mistakes would have been disastrous in a real life production context.  This time around I was very thorough in only opening up as much access as was needed rather than having everything open to the world.

So what next?  I had been planning to Dockerise Riemann for the Art of Monitoring book.  I will get to that, but feel the need to branch out a bit even if Docker still plays a part.  Current thinking is to study and play with:

  • MongoDB
  • Git (I am embarrassed by how little I use it)
  • Kubernetes (really excited about this)

I am also going to try and organise my reading a little so am planning to tackle the following:

  • The DevOps Handbook
  • Sam Newman’s Microservices
  • Google SRE
  • The Lean Enterprise

Some choices there are influenced by a possible change of role – which will be a good step towards something more DevOpsy (sorry) – for which I am going to have to know my stuff.  I hope re-reading the first two will be as rewarding as a second pass through The Docker Book.

To further assist my mission, I have set myself the objective of watching a webinar each week. I am amazed at how much great material is out there.

My Picks

Finally, I liked the Food Fight’s Picks” so much, I am going to borrow it.  My first one in homage to Food Fight is one of their very own podcasts on the Netflix OSS.  This is a few years old (2013) but I enjoyed it immensely.  Find it here..

Posted on

A forest in a bottle in a spaceship in a maze

The title of this post is one of my favourite quotes from one of my favourite episodes of my favourite shows.  See the foot of this post for more info.

I was reminded of this quote whilst tackling chapter five of The Docker Book.  I cannot recall finishing this section last time around as I was running Docker on my Mac, not an EC2 instance running Ubuntu as I am now.   I ran into limitations on the Mac Docker implementation.

When I had completed the section I had (deep breath), Jenkins running in a Docker container on an EC2 instance, creating Docker containers to run Ruby apps.  I connected to the EC2 instance from iTerm running on my Mac.

I am particularly proud of completing this one as I ran into a number of issues through which I logically debugged to fix.  I learnt  a lot as I did so but even better I was using tools, techniques, knowledge and logic that would have been beyond me first time around.

What I am not particularly proud of is that I managed to lose a Dockerfile that had an issue so could not compare it to the Dockerfile that ultimately worked – opportunity lost there to learn something else.  However, I know the area where the issue occurred so all is not lost.

If you want to know more about forests, bottles, rockets and mazes, visit here.