Update my Contacts with Python: thinking through the options

Alright folks, that’s the bell.  When LinkedIn stops thinking of itself as a professional contact manager, you know there’s no profit in it, and it’s time to manage this stuff yourself.

Problem To Solve

I’ve been hemming and hawing for a couple of years, ever since Evernote shut down their Hello app, about how to remember who I’ve met and where I met them.  I’m a Meetup junkie (with no rehab in sight) and I’ve developed a decent network of friends and acquaintances that make it easy for me to attend new events and conferences in town – I’ll always “know” someone there (though not always remember why/how I know them or even what their name is).

When I first discovered Evernote Hello, it seemed like the perfect tool for me – provided me a timeline view of all the people I’d met, with rich notes on all the events I’d seen them at and where those places were.  It never entirely gelled, it sporadically did and did NOT support business card import (pay for play mostly), and it was only good for those people who gave me enough info for me to link them.  Even with all those imperfections, I remember regularly scanning that list (from a quiet corner at a meetup/party/conference) before approaching someone I *knew* I’d seen before, but couldn’t remember why.  [Google Glasses briefly promised to solve this problem for me too, but that tech is off somewhere, licking its wounds and promising to come back in ten years when we’re ready for it.]

What other options do I have, before settling in to “do it myself”?

  • Pay the big players e.g. SalesForce, LinkedIn
    • Salesforce: smallest SKUs I could find @ $25/month [nope]
    • LinkedIn “Sales” SKU: $65/month [NOPE]
  • Get a cheap/trustworthy/likely-to-survive-more-than-a-year app
    • Plenty of apps I’ve evaluated that sound sketchy, or likely to steal your data, or are so under-funded that they’re likely to die off in a few months

Requirements

Do it myself then.  Now I’ve got a smaller problem set to solve:

  1. Enforce synchronization between my iPhone Contacts.app, the iCloud replica (which isn’t a perfect replica) and my Google Contacts (which are a VERY spotty replica).
    • Actually, let’s be MVP about this: all I *need* right now is a way of automating edits to Contacts on my iPhone.  I assume that the most reliable way of doing this is to make edits to the iCloud.com copy of the contact and let it replicate down to my phone.
    • the Google Contacts sync is a future-proofing move, and one that theoretically sounded free (just needed to flip a toggle on my iPhone profile), but which in practice seems to be built so badly that only about 20% of my contacts have ever sync’d with Google
  2. Add/update information to my contacts such as photos, “first met” context (who introduced, what event met at) and other random details they’ve confessed to me (other attempts to hook my memory) – *WITHOUT* linking my iPhone contacts with either LinkedIn or Facebook (who will of course forever scrape all that data up to their cloud, which I do *not* want to do – to them or me).

Test the Sync

How can I test my requirements in the cheapest way possible?

  • Make hand edits to the iCloud.com contacts and check that it syncs to the iPhone Contacts.app
    • Result: sync to iPhone within seconds
  •  Make hand edits to contacts in Contacts.app and check that it syncs to iCloud.com contact
    • Result: sync to iCloud within seconds

OK, so once I have data that I want to add to an iCloud contact, and code (Python for me please!) that can write to iCloud contacts, it should be trivial to edit/append.

Here’s all the LinkedIn Data I Want

Data that’s crucial to remembering who someone is:

  • Date we first connected on LinkedIn
  • Tags
  • Notes
  • Picture

Additional data that can help me fill in context if I want to dig further:

  • current company
  • current title
  • Twitter ID
  • Web site addresses
  • Previous companies

And metadata that can help uniquely identify people when reading or writing from other directories:

  • Email address
  • Phone number

How to Get my LinkedIn connection data?

OK, so (as of 2016-12-15 at 12:30pm PST) there’s three ways I can think of pulling down the data I’ve peppered into my LinkedIn connections:

  1. User Data Archive: request an export of your user data from LinkedIn
  2. LinkedIn API: request data for specified Connections using LinkedIn’s supported developer APIs
  3. Web Scraping: iterate over every Connection and pull fields via CSS using e.g. Beautiful Soup

User Data Archive

This *sounds* like the most efficient and straightforward way to get this data.  The “Relationship Section” announcement even implies that I’ll get everything I want:

If you want to download your existing Notes and Tags, you’ll have the option to do so through March 31, 2017…. Your notes and tags will be in the file named Contacts.

The initial data dump included everything except a Contacts.csv file.  The later Complete_LinkedInDataExport_12-16-2016 [ISO 8601 anyone?] included the data promised and nearly nothing else:

  • Connections.csv: First Name, Last Name, Email Address, Current Company, Current Position, Tags
  • Contacts.csv: First Name, Last Name, Email (mostly blank), Notes, Tags

I didn’t expect to get Picture, but I was hoping for Date First Connected, and while the rest of the data isn’t strictly necessary, it’s certainly annoying that LinkedIn is so friggin frugal.

Regardless, I have almost no other source for pictures for my professional contacts, and that is pretty essential for recalling someone I’ve met only a handful of times, so while helpful, this wasn’t sufficient.

LinkedIn API

The next most reliable way to attack this data is to programmatically request it.  However, as I would’ve expected from this “roach motel” of user-generated data, they don’t even support an API to request all Connections from your user account (merely sign-in and submit data).

Where they do make reference to user data, it’s in a highly-regulated set of Member Profile fields:

  • With the r_basicprofile permission, you can get first-name, last-name, positions, picture-url plus some other data I don’t need
  • With the r_emailaddress permission, you can get the user’s primary email address
  • For developers accepted into “Apply with LinkedIn”, and with the r_fullprofile permission, you can further get date-of-birth and member-url-resources
  • For those “Apply with LinkedIn” developers who have the r_contactinfo permssion, you can further get phone-numbers and twitter-accounts

After registering a new application, I am immediately given the ability to grant the following permissions to my app: r_basicprofile, r_emailaddress.  That’ll get me picture-url, if I can figure out a way to enumerate all the Connections for my account.

(A half-hour sorting through Chrome Dev Tools’ Network outputs later…)

Looks like there’s a handy endpoint that lets the browser enumerate pretty much all the data I want:

https://www.linkedin.com/connected/api/v2/contacts?start=40&count=10&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

That bears further investigation.

Web Scraping

While this approach doesn’t have the built-in restrictions with the LinkedIn APIs, there’s at least three challenges I can forsee so far:

  1. LinkedIn requires authentication, and OAuth 2.0 at that (at least for API access).  Integrating OAuth into a Beautiful Soup script isn’t something I’ve heard of before, but I’m seeing some interesting code fragments and tutorials that could be helpful, and it appears that the requests package can do OAuth 1 & 2.
  2. LinkedIn has helpfully implemented the “infinite scroll” AJAX behaviour on the Connections page.
    • There are ways to work with this behaviour, but it sure feels cumbersome – to the point I almost feel like doing this work by hand would just be faster.
  3. Navigating automatically to each linked page (each Connection) from the Connections page isn’t something I am entirely confident about
    • Though I imagine it should be as easy as “for each Connection in Connections, load the page, then find the data with this CSS attribute, and store it in an array of whatever form you like”.  The mechanize package promises to make the link navigation easy.

Am I Ready for This Much Effort?

It sure feels like there’s a lot of barriers in the way to just collecting the info I’ve accumulated in LinkedIn about my connections.  Would it take me less time to just browse each connection page and hand copy/paste the data from LinkedIn to iCloud?  Almost certainly.  To together a Beautiful Soup + requests + various github modules solution would probably take me 20-30 hours I’m guessing, from all the reading and piecing together code fragments from various sources, to debugging and troubleshooting, to making something that spits out the data and then automatically uploads it without mucking up existing data.

Kinda takes the fun out of it that way, doesn’t it?  I mean, the “glory” of writing code that’ll do something I haven’t found anyone else do, that’s a little boost of ego and all.  Still, it’s hard to believe this kind of thing hasn’t been solved elsewhere – am I the only person with this bad of a memory, and this much of a drive to keep myself from looking like Leonard Shelby at every meetup?

What’s worse though, for embarking on this thing, is that I’d bet in six months’ time, LinkedIn and/or iCloud will have ‘broken’ enough of their site(s) that I wouldn’t be able to just re-use what I wrote the first time.  Maintenance of this kind of specialized/unique code feels pretty brutal, especially if no one else is expected to use it (or at least, I don’t have any kind of following to make it likely folks will find my stuff on github).

Still, I don’t think I can leave this itch entirely unscratched.  My gut tells me I should dig into that Contacts API first before embarking on the spelunking adventure that is Beautiful Soup.

Highlights from latest Lean Coffee

A lively crowd around the table at last Sunday’s Lean Coffee session, and fresh faces to the discussion (thank you to Scott for inviting your colleagues, and to all for coming out).

There’s no way I can do justice to the breadth and depth of the discussion, so I’m just going to mention those things I wrote down on sticky notes to myself – the things that I thought, “Boy, I should get this tattooed on myself somewhere”:

  • Don’t Automate Waste – a killer principle from the Lean camp that Dan Walsh graced us with, it speaks to the tension of not optimizing early, and to my instinct not to assume you have the solution without experimentation
  • “Agile/Scrum is a Problem Discovery Framework, not a Project Management Methodology” – courtesy of Scott Henderson, every word here lends subtle meaning to the mental shift it encourages
  • Lean Coffee has been used successfully in at least two settings I haven’t tried – as the basis for both the Retrospective and Brainstorming sessions (which helps get ideas on the table that might be ‘swallowed’ by the time attention comes around to the less-confident individual)
  • Code 46 and Sully were the two movies that came up in conversation, so off to Netflix I go

2016-12-04 11.59.58.jpg

I posed a question to the group which came back with some great thoughts: “how to workaround a situation [which I’ve observed at many software companies] where the testing infrastructure/coverage isn’t reliable, and there’s no quick route to addressing that?”

  1. Ensure that you at least have Unit Tests included in the Definition of Done
  2. Try an experiment where for a single sprint, the team only works on writing unit tests – when this was tried at one organization, it surprised everyone how much progress and coverage could truly be made
  3. Try a regular “Game Day” exercise – run tabletop simulation of a production bug that takes out one or more of your customer-facing services.  This identifies not only who must be involved, but also how long it can take to execute corrective action once identified, and ultimately can result in significant time savings by making upstream changes in product/devops.
  4. Run an occasional discussion at Retrospective to ask “what’s the worst thing we could do to the product?”  This can uncover issues and concerns that otherwise go unspoken by folks who are worried about retribution or downplaying.
  5. And the most obvious, start out future sprints by planning tests up front (either via TDD or manually between QA and Dev)

Lean Coffee September insights report

That’s our Sunday morning Lean Coffee practice. Here’s where we landed after a good 1.5-ish hours of structured-and-friendly conversation.

 On the subject of landing a job as a Scrum Master

  • You must be very familiar with the SCRUM Guide, and especially the “Why” behind each practice – so that you can address real questions about when you’ll recommend a practice and when you’ll recommend evolving past it
  • Should be very comfortable with trying new things AS EXPERIMENTS
  • Must avoid “always pitying the SCRUM team” at the expense of the overall business goals, or else business will hamstring your influence and bypass your role
  • Relies heavily on Situational Leadership abilities
  • Starts with CI, graduate to Continuous Learning

On the subject of what’s changed and what is changing

  • According to our discussion of Crossing the Chasm, once those beyond the chasm start adopting, then rather than chasing the downward slope, you should chase a new curve starting from the other side of the chasm
  • We’re seeing signs that other non-software disciplines are adopting Agile practices eg. Marketing functions, DevOps
  • Perhaps we’re merely waiting for the rest of the org to catch up to those of us who are post-Agile and delivering continuously
  • The VUCA model (Volatility, Uncertainty, Complexity, Ambiguity) made it into a Harvard Business Review article
  • Neurodiversity is getting broader consciousness

 On the subject of creating success as a Scrum Master

  • The basic SM is a “boundary manager”
  • They’re there not only to help the team “learn to be a team” and more to help the team “learn how to be Agile as a team”
  • They’re there to work with the team and enable them to determine what process solutions to try, rather than dictate or even “guide” the team to specific outcomes
  • Tip: should be very familiar with the Agile Fluency model
  • When interviewing for a SM role, an insightful question is to ask what are the inputs and outputs of the engineering team?
  • Geoff Watts published an article asking What kind of support a Scrum Master would need?

On the subject of no estimates

  • Analogy of a cook: asking for precise estimates is like asking them to cook a dinner with a menu they’ve never cooked before
  • Analogy of car mechanic: they can only give predictable, tight estimated of when the repair will be completed for operations they’ve already done before (enough time to have codified the standard timeframe) and with mechanics who are highly experienced

Miscellaneous Insights

  • Meetup (as in the collection of fluid communities) is like a grand ongoing Un-Conference – people announcing a topic they’d like to talk about, those who wish to attend come, people obeying the law of two feet as the meetup’s theme no longer keep their attention
  • Check out Rachel Davies’ Agile Coaching book
  • There’s growing insight that SAFe can find better flow mechanics across the portfolio if it uses Kanban rather than SCRUM – but that a prerequisite is that the teams must already have in place high-quality technical practices (eg. low big output, continuous integration, short distance from idea to value) and functioning teams before Kanban at scale will create consistent results
  • Book: check out the free mini-book on Scrumban by Henrik Kniberg

See you there next time (if you’re lucky).

Do You Demo? Do you act on the feedback? No? Then you ain’t agile

I am convinced that there are few practices in Agile (aka SCRUM to most people) that can’t be revised, bent, paused or outright abandoned – in the pursuit of a healthy, adaptive and productive engineering squad.

However, the end-of-iteration demo is one I am vehement about.  If you aren’t doing demos well, it’s my believe that you shouldn’t be calling yourself agile (let alone Agile(tm)).

Aside: AgilePDX is a local meetup community of people zealous about improving our employers’ ability to deliver better product – faster, more transparently and most importantly, with more value to our customers.

Our last pub lunch roundtable discussion was “The End of Agile?”, and Billy McGee posed a great question that still rings in my head:

Which of the rituals/rules/ceremonies can we abandon and still call ourselves Agile?

My initial (silent) response to this was “almost all of them”.

As I actually contributed to the discussion, I still claim that one of The Most Important ceremonies to stick with is the end-of-iteration DEMO. Get your feedback early and often (if more frequently than end-of-iteration, even better), and for dogs’ sake get at least *one* person to say something who’s from *outside* the team.  Then take an action on that feedback.

code demo

Until you get outside-the-team inspection, you can’t hope to truly adapt to the trouble you’ve just gotten into by being away from outsiders for that long.  Groupthink, too-close-to-the-problem, love for the solution you derived – it all clouds the objective reality that you’ve almost certainly done it wrong.

If you do this one thing [demo/feedback/react] consistently/more-than-haphazardly, and *act* on the feedback (discuss, change a story, throw away code, replan your roadmap), you will have done more to imbue confidence in your team from the rest of the organization, and they’ll give you a lot more room to be experimental and not “plan-to-perfection” drowned.

The most frustrating thing for management/leadership that aren’t there every day is feeling like we have no CONTROL over getting the right things delivered more effectively.  Not seeing the work that goes into delivering the product, the engineering process becomes just a black box – even for those who’ve been in the trenches, distance + time just leads the mind to question.

When things feel out of control, leaders use what few tools they have to do what they can to keep things steered correctly – demand reports and metrics to give them *something* to wrap hands around, start dictating process to get more ‘structure’ around this messy process, and asking for more detailed plans.

These are all proxies for “how can I help make sure we’re doing the right things?”  And like they say, a picture’s worth a thousand words.  Heck, a customer’s reaction is worth at least that much too.

I’ve seen myself react *far* better to the growing uncertainties when I get to see what’s been delivered so far – live UI, screenshots, even a tour of the source code.  Takes away so much anxiety just to *see* something is there, let alone whether it truly meets my personal objectives.  Give me something – anything – to comment on, and the fact that I engage in comments on the thing means (a) I’m asking for help to buy in, (b) I’m getting committed to the thing in front of me and (c) you’re getting early clues what I’ll want to see before the last responsible moment.

Even this is better than no feedback, no matter how painful it might be:

code quality

Coda: one of my colleagues asked me, “So, abandon retrospectives?”  It’s a good question.  They’re one of the few rituals that’s meant to help the team evolve to better performance and outcomes.  And here’s where I’m going to take a radical/lazy/wait-and-see position:

Personally, I’m inclined to skip (the process part of) the retro until and unless the team gets behind the “inspect and adapt” angle on the product in response to what happens during demo. My experience, engineers are more likely to start in that habit if it’s focused on their code, and if they’re not even willing to engage for the technology, I’m less inclined to skate uphill on the process side of things – as a PO/PM participant/observer, I’ve seen retro degrade quite often into a “here’s what happened”, “good/bad/ugly” historical review, and lose sight of the “what would you like to try changing next time?”

Occupied Neurons, early May 2016

The continuing story of the intriguing ideas and happenings that I can’t shake off…

Pigsinspace222

(Have you ever seen an episode of Pigs In Space?  If not, go sample one now, and you’ll get my droll reference)

Infinite Scrolling, Pagination or “Load More” Buttons? Usability Findings in eCommerce

https://www.smashingmagazine.com/2016/03/pagination-infinite-scrolling-load-more-buttons/

Summary (and something I plan to bias towards in future designs, under similar conditions): The “Load More” design pattern is the most well-received by users and creates a minimum of friction while still enabling access to the page footer.

How Spotify’s Poor API Hygiene Broke a Bunch of Hardware and Software

http://www.programmableweb.com/news/how-spotifys-poor-api-hygiene-broke-bunch-hardware-and-software/analysis/2016/02/23

This is a pretty epic rant on the fallout for independent Spotify developers from a haphazard approach to managing the APIs offered over the years by this consumer entertainment service. Having worked on the other side of these kinds of decisions, I can well imagine how this came to be: thin staffing levels keeping from putting adequate attention on developer communications and engineering maintenance, plus distracted attention by PMs (or possibly even frequent PM turnover) such that late in the game, no one even remembers lets alone still believes in the original value prop behind the original APIs.

It doesn’t excuse the broken promises behind the APIs, and especially not the lack of communication in obvious channels when changes were made (eliminated), but I’ve been in such positions as a Product guy and found myself making decisions that feel just as compromised – trading off one disappointment for a better-mitigated disappointment elsewhere. It happens, especially when the product being extended through those APIs has a pretty low profit margin, and when the staff devoted to managing those concerns are terribly compromised (higher priorities and all).

Theory of Constraints

https://en.m.wikipedia.org/wiki/Theory_of_constraints

At the Intel-sponsored Accelerate Results gathering, a few themes/durable concepts kept coming up (and have come up in this community repeatedly over the years). One is the Theory of Constraints, which seems popular among all systems thinkers, even in big software design (at least in concept if not in execution).

I firmly believe we have a duty to consider outside perspectives on our industry, even when they appear to have no direct applicability – myopia, tools bias and fad-driven design/execution are the restraints I make deliberate effort to resist in my own practices.

Standing on the Shoulders of Giants

http://www.business-improvement.eu/toc/Goldratt_Standing_On_The_Shoulders_Of_Giants.php

Eliyahu Goldratt is a huge influence on the thought leaders at the Accelerate Results conference, and many made reference to his seminal essay that seems to have kicked off this whole revolution. Worth a skim, even if it’s only to be able to nod thoughtfully when others keep talking about this.

Everyday Internet Users Can Stand Up for Encryption — Here’s How

https://blog.mozilla.org/blog/2016/03/30/everyday-internet-users-can-stand-up-for-encryption-heres-how/
I worked with Mark Surman a long time ago back in Toronto for a non-profit Internet Service Provider. It’s more than a little amazing to me to see how our paths have diverged and yet how he’s speaking about issues today that are very near and dear to my heart.

Meetups where you’ll find Mike’s hat, Spring 2016 edition

Occasionally I’ll tell people I meet about all the meetups I have so much fun at.

Or rather, I’ll try to enumerate them all, and fail each and every time.

Primarily because there’s so many meetups I like to check in on.

So occasionally I’ll enumerate them like this, so that my friends have a valiant hope of crossing paths with me before the amazing event has passed.

Meetups I’m slavishly devoted to

Meetups I’ll attend anytime they’re alive

Meetups I sample like caviar – occasionally and cautiously

Recent additions that may soon pass the test of my time

 

PDX-local Meetups in my coherent rotation

(Hah – I meant to type “current rotation” but sometimes autocorrect makes me sound much more nuanced than I meant)

Here’s where you’ll find me lurking and entertaining the not-so-innocent bystanders: meetup-in-a-bar

Occasional fly-bys: