DevOps status report: HackOregon 2019 season

One of my colleagues on the HackOregon project this year sent around “Nice post on infrastructure as code and what getting solid infra deploys in place can unlock” https://www.honeycomb.io/blog/treading-in-haunted-graveyards/

I felt immediately compelled to respond, saying:

Provocative thinking, and we are well on our way I’d say.

I’ve been the DevOps lead for HackOregon for three years now, and more often than not delivering 80% of the infrastructure each year – the CI/CD pipeline, the automation scripts for standardizing and migrating configuration and data into the AWS layers, and the troubleshooting and white-glove onboarding of each project’s teams where they touch the AWS infrastructure.

There’s great people to work with too – on the occasions when they’ve got the bandwidth to help debug some nasty problem, or see what I’ve been too bleary-eyed to notice is getting in our way, it’s been gratifying to pair up and work these challenges through to a workable (if not always elegant) solution.

My two most important guiding principles on this project have been:

  • Get project developers productive as soon as possible – ensure they have a Continuous Deployment pipeline that gets their project into the cloud, and allows them to see that it works so they can quickly see when a future commit breaks it
  • “working > good > fast” – get something working first, make it “good” (remove the hard-coding, the quick workarounds) second, then make it automated, reusable and documented

We’re married pretty solidly to the AWS platform, and to a CloudFormation-based orchestration model.  It’s evolved (slowly) over the years, as we’ve introspected the AWS Labs EC2 reference architecture, and as I’ve pulled apart the pieces of that stack one by one and repurposed that architecture to our needs.

Getting our CloudFormation templates to a place where we can launch an entirely separate test instance of the whole stack was a huge step forward from “welp, we always gotta debug in prod”. That goal was met about a month ago, and the stack went from “mysterious and murky” to “tractably refactorable and extensible”.

Stage two was digging deep enough into the graveyard to understand how the ECS parts fit together, so that we could swap EC2 for Fargate on a container-by-container basis. That was a painful transition but ultimately paid off – we’re well on our way, and can now add containerised tasks without also having to juggle a whole lot of maintenance of the EC2 boxes that are a velocity-sapping drag on our progress.

Stage 3 has been refactoring our ECS service templates into a standardised single template used by whole families of containerised tasks, from a spray of copypasta hard-coded replicas that (a) had to be curated by hand (much like our previous years’ containerised APIs has to be maintained one at a time), and (b) buried the lede on what unique configuration was being used in each service. Any of the goofy bits you need to know ahead of deploying the next container are now obvious and all in one place, the single master.yaml.

I can’t speak for everyone, but I’ve been pretty slavish about pushing all CF changes to the repo in branches and merging when the next round of stable/working infra has been reached. There’s always room for improvement, however:

  • smaller changes are always better
  • we could afford more folks who are trained and comfortable with the complex orchestration embedded in our infrastructure-as-code
  • which would mean being able to conduct good reviews before merge-to-master
  • I’d be interested in how we can automate the validation of commit-timed-upgrades (though that would require more than a single mixed-use environment).

Next up for us are tasks like:

  • refactoring all the containers into a separate stack (out of master.yaml)
  • parameterising the domains used for ALB routing
  • separating production assets from the development/staging environment
  • separating a core infra layer from the staging vs production side-by-side assets
  • refactoring the IAM provisions in our deployment (policies and attached roles)
  • pulling in more of the coupled resources such as DNS, certs and RDS into the orchestration source-controlled code
  • monitoring and alerting for real-time application health (not just infra-delivery health)
  • deploying *versioned* assets (not just :latest which becomes hard to trace backwards) automatically and version-locking the known-good production configuration each time it stabilises
  • upgrading all the 2017 and 2018 APIs to current deployment compatibility (looking for help here!)
  • assessing orchestration tech to address gaps or limitations in our current tools (e.g. YAML vs. JSON or TOML, pre-deploy validation, CF-vs.-terraform-vs-Kubernetes)
  • better use of tagging?
  • more use of delegated IAM permissions to certain pieces of the infra?

This snapshot of where we’re at doesn’t capture the full journey of all the late nights, painful rabbit holes and miraculous epiphanies

Occupied Neurons, October edition

Melinda Gates Asked For Ideas to Help Women in Tech: Here They Are

https://backchannel.com/an-open-letter-to-melinda-gates-7c40d8696b63#
I am psyched that a powerhouse like Gates is taking up the cause, and I sincerely hope she reads this (and many other) articles to get a sense of the breadth of the problem (and how few working solutions there are).  The overlap with race, the attempts to bring more women into classrooms, the tech industry bias towards the elite schools and companies (and not the wealth of other experiences). It’s a target-rich environment to solve.

Building a Psychologically Safe Workplace: Amy Edmondson at TEDxHGSE

https://m.youtube.com/watch?feature=youtu.be&v=LhoLuui9gX8

I am super-pleased to see that the concept of Psychological Safety is gaining traction in the circles and organizations I’m hanging with these days.  I spend an inordinate amount of time in my work making sure that my teammates and colleagues feel like it’s OK to make a mistake, to own up to dead ends and unknowns, and will sure make the work easier when I’m not the only one fighting the tide of mistrust/worry/fear that creates an environment where learning/risks/mistakes are being discouraged.

Three Books That Influenced CorgiBytes Culture

http://corgibytes.com/blog/2016/09/15/three-influential-books/

Andrea and Scott are two people who have profoundly changed my outlook on what’s possible to bring to the workplace, and how to make a workplace that truly fits what you want (and sometimes need) it to be. Talking about empathy as a first-class citizen, bringing actual balance to the day and the communications, and treating your co-workers better than we treat ourselves – and doing it in a fun line of business with real, deep impact for individual customers.

This is the kind of organization that I could see myself in. And which would draw in the kinds of people I enjoy working with each day.

So after meeting them earlier this year in Portland, I’ve followed their adventures via their blog and twitter accounts. This article is another nuanced look at what has shaped their workplace, and I sincerely hope I can do likewise someday.

Reducing Visual Noise for a Better User Experience

https://medium.com/@alitorbati/reducing-visual-noise-for-a-better-user-experience-ae3407ff9c99

View at Medium.com

These days I find myself apprehensively clicking on Design articles on Medium.  While there’s great design thinking being discussed out there, I seem to be a magnet for finding the ones that complain why users/managers/businesses don’t “get it”.

As I’d hoped, this was an honest and detailed discussion of the inevitable design overload that creeps into most “living products”, and the factors that drove them to improve the impact for non-expert users.

(I am personally most interested in improving the non-expert users’ experience – experts and enthusiasts will always figure out a way to make shit work, even if they don’t like having to beat down a new door; the folks I care to feed are those who don’t have the energy/time/inclination/personality for figuring out something that should be obvious but isn’t.  Give me affordances, not a learning experience e.g. when you’ve got clickable/tappable controls on your page, give me lines/shadows/shading to signify “this isn’t just text”, not just subtle whitespace that cues the well-trained UI designer that there’s a button around that otherwise-identically-styled text.

Meetups where you’ll find Mike’s hat, Spring 2016 edition

Occasionally I’ll tell people I meet about all the meetups I have so much fun at.

Or rather, I’ll try to enumerate them all, and fail each and every time.

Primarily because there’s so many meetups I like to check in on.

So occasionally I’ll enumerate them like this, so that my friends have a valiant hope of crossing paths with me before the amazing event has passed.

Meetups I’m slavishly devoted to

Meetups I’ll attend anytime they’re alive

Meetups I sample like caviar – occasionally and cautiously

Recent additions that may soon pass the test of my time

 

Non-Diversity Apologists still abound in Comics

I did a dumb thing today.

I read the comments below this post at the Marvel: The Untold Story page:

https://www.facebook.com/MarvelComicsUntoldStory/posts/634186903384002

And immediately lost another month off my life due to the immediate blood pressure spike.

I recovered some of those lost future days by reading the source article upon which the comments were theoretically made:
http://www.vanityfair.com/culture/2015/07/kelly-sue-deconnick-profile

Why do the moron patrol spout their ill-informed opinions when the confounding data is so easy to obtain?  If there was no systemic discrimination biasing the work-for-hire systems at the Big Two (DC and Marvel) towards white males, I’d expect to see a relative population among female creators somewhere better than the female engineering population we see in the tech industry (which according to the best of the paucity of data out there, is around 12%).

Here’s the latest figures for DC’s male-to-female stats:
http://www.bleedingcool.com/2015/07/08/gendercrunching-special-edition-dcyou/

And for reference, here’s a similar study of both DC and Marvel from 2012:
http://www.bleedingcool.com/2012/10/30/gendercrunching-august-2012/

Analysis/conjecture: the % of creators hired by the Big Two who are female has consistently peaked in the low teens, and even % of back-office employees is generally lower than the magic 30% threshold (where “gender diversity” is celebrated, and beyond which females start to experience a marked uptick in pushback, and “diversity initiatives” start to see institutionalized resistance).  And sadly, this is years after fans demand DC take action to change it (and they agreed).  Maybe DC thinks that <10% is their end goal?

Hell, when one of the paragons of progressive thinking and hiring, Apple, can’t even make it past 30%, where do smaller creative companies and the masses of ‘invisible’ tech corporations stand a chance?  Google’s 30% is even held up as an achievement, which is DEPRESSING.

Contrary to what I’ve heard in the comics press and hallways for years, comics fans are about half female these days.  Hasn’t always been that way, but it’s also fascinating that readership has achieved near-parity when many (men of course) predicted it would never, and that maybe it shouldn’t (to preserve the enclave of “what we white males have always enjoyed in the safe past”).  It’s not like there isn’t interest from females, and I’d predict this to bleed into the creative population who’s trying to get a gig there – it would blow my mind that only 10% of the people trying to create comics are women.  (This article mentions women outnumber men in an undergraduate cartooning setting, for example, and then quotes the instructor, “It’s also not uncommon that they’re the best students in class”.)

So assume for a moment the fantastic proposition that there’s a greater proportion of women clamouring for the gig than are getting the gig, what does that say about the selection filter: are those making the hire decisions going to explain this by saying that the greater proportion of women aren’t as talented as the men?  That they aren’t applying through the correct/operative/off-the-books channels?  Or that they are actually just more comfortable hiring people like them (white, male, cis)?

[Relevant anecdote: Bobbie Chase, a female editorial director for DC, was quoted in this article saying, “We’re pursuing people all the time who could be new voices for comic books, but it’s still going to be a predominantly male industry. I don’t think that has to change, but we can certainly make a much better balance.”]

[Another relevant anecdote: Ann Nocenti in this article said, “Its undeniable when you look at industry-wide statistics, women are hired low, their salaries are lower; I think statistically you can say women aren’t treated fairly in any industry.”]

Any answer raises troubling questions that I really hope are being examined and answered honestly, and which will drive systemic change in both the comics and the tech industry.

robliefeldcaptainamerica
Even freakish Cap thinks there’s a problem here

 

Discovered artist: Justin Ponseur

https://m.facebook.com/JPoColors
At the DevSigner conference this morning, Justin McDowell (@revolt_puppy) turned me on to the realistic, natural colour work of Justin Ponseur. 

After a few minutes flipping through his site, I am definitely a fan. His colour work adds another level of gorgeous to even the best artists (and probably covers a lot of sins from the lesser artists).

Epiphany of Volunteering

Been struggling with the desire to volunteer – to take my skills out to organizations and people who don’t normally have access to the kind of big corporate expertise – and to give myself opportunities to give back to my community.

Only problem is: the kinds of groups in which I want to volunteer (eg. Hack Oregon) are filled with amazing coders who might not feel friendly and welcoming to a “business/product/design” guy who wants to help out but isn’t a coder or database geek.

I’ve been out to a couple of events, and watched the participants gather together in their natural tendencies. I start out feeling self-conscious and a deficit for any group I force myself into, and end up just chatting with whoever it feels like might also be feeling disconnected.

I’ve lost my nerve with such organizations and ended up not finding an outlet for my desire to help, contribute my energy and experience, and effect change.

Epiphany
Today for no explicable reason, it occurred to me that rather than approaching volunteering as a place to contribute, and instead set my goal to “learning”.

I thought of this when Catherine Nikolovsky talked about the number of Big Data and data visualization nerds her organization, and I lit up thinking, “I want to learn about Big Data and Dataviz!”

What if I showed up and attempted to simply ask questions, see how Big Data apps are built, and what kinds of decisions are made in developing an effective data visualization?

Do I have the nerve to show up and insert myself without any ego – without an intention to help, but rather just to listen?

And now, a random picture from today’s Facebook distractions:

IMG_2277