Bug Reports: hoopla + comics

An occasional series of the bugs I attempt to report to vendors of software I enjoy using.

Bug #1: re-borrow, can’t read

I borrow a comics title on Hoopla, it eventually expires. I re-borrow it, then when I try to read it reports “There was an error loading Ex Machina Book Two.” error.

I tried a half-dozen times to Read it. I killed the app and restarted it, then tried to Read, still the same error.  I am unable to find a delete feature in the app, so I cannot delete and re-download the content.

This same error has happened to me twice with two different comics titles.  I only read comics via hoopla, so I cannot yet report if this happens for non-comics content.

Repro steps

  • Open Hoopla app on my device, browse to the title Ex Machina Book Two
  • Tap the Borrow button, complete the Downloading phase
  • Tap the Read button – result: content loads fine
  • Wait 21+ days for DRM license to expire
  • Browse to the same title, tap Borrow
    (Note: it take no time at all to switch to the Read button, which implies it just downloads a fresh DRM license file)
  • Tap the Read button

Expected Result

Book opens, content is readable.

Actual Result

App reports Error “There was an error loading…”, content does not load:

hoopla error re-borrowing comic.png

User Environment

iPad 3, iOS 9.3.5, hoopla app version 4.10.2

Bug #2: cannot re-sort comics

I browse the “Just added to hoopla” section of Comics, and no matter which sorting option I choose, the list of comics appears in the exact same order. Either this is a coincidence, or the sorting feature doesn’t work (at least in this particular scenario).

Repro steps

  • Open the hoopla app on my device, tap the Books tab
  • Tap the Comics selector across the top of the app window, then tap the Genres link at the top-right corner
  • Select the option Just added to hoopla
  • Scroll the resulting comics titles in the default popular view, noting that [at time of writing] three Jughead titles appear before Superman, Betty & Veronica and The Black Hood
  • Tap the new arrivals and/or A-Z view selectors along the top

Expected Result

The sort order of the displayed comics would change under one or both views (especially under the A-Z view, where Jughead titles would be listed after Betty & Veronica). The included titles may or may not change (perhaps some added, some removed in the new arrivals view, if this is meant to show just the most recently-added titles).

Actual Result

The sort order of the displayed comics appears identical to the naked eye.  Note that in the A-Z view, the Jughead comics continue to appear at the top, ahead of the Betty & Veronica comic:

hoopla sort order in A-Z view.png

User Environment

iPad 3, iOS 9.3.5, hoopla app version 4.10.2

Advertisements

Update my Contacts with Python: thinking about how far to extend PyiCloud to enable PUT request?

I’m on a mission to use PyiCloud to update my iCloud Contacts with data I’m scraping out of LinkedIn, as you see in my last post.

From what I can tell, PyiCloud doesn’t currently implement support for editing existing Contacts.  I’m a little out of my depth here (constructing lower-level requests against an undocumented API) and while I’ve opened an issue with PyiCloud (on the off-chance someone else has dug into this), I’ll likely have to roll up my sleeves and brute force this on my own.

[What the hell does “roll up my sleeves” refer to anyway?  I mean, I get the translation, but where exactly did this start?  Was this something that blacksmiths did, so they didn’t burn the cuffs of their shirts?  Who wears a cuffed shirt when blacksmithing?  Why wouldn’t you go shirtless when you’re going to be dripping with sweat?  Why does one question always lead to a half-dozen more…?]

Summary: What Do I Know?

  • LinkedIn’s Contacts API can dump most of the useful data about each of your own Connections – connectionDate, profileImageUrl, company, title, phoneNumbers plus Tags (until this data gets EOL’d)
  • LinkedIn’s User Data Archive can supplement with email address (for the foreseeable) and Notes and Tags (until this data gets EOL’d)
  • I’ve figured out enough code to extract all the Contacts API data, and I’m confident it’ll be trivial to match the User Data Archive info (slightly less trivial when those fields are already populated in the iCloud Contact)
  • PyiCloud makes it darned easy to successfully authenticate and read in data from the iCloud contacts – which means I have access to the contactID for existing iCloud Contacts
  • iCloud appears to use an idempotent PUT request to write changes to existing Contacts, so that as long as all required data/metadata is submitted in the request, it should be technically feasible to push additional data into my existing Contacts
  • It appears there are few if any required fields in any iCloud Contact object – the fields I have seen submitted for an existing Contact include firstName, middleName, lastName, prefix, suffix, isCompany, contactId and etag – and I’m not convinced that any but contactID are truly necessary (but instead merely sent by the iCloud.com web client out of “habit”)
  • The PUT operation includes a number of parameters on the request’s querystring:
    • clientBuildNumber
    • clientId
    • clientMasteringNumber
    • clientVersion
    • dsid
    • method
    • prefToken
    • syncToken
  • There are a large number of cookies sent in the request:
    • X_APPLE_WEB_KB–QNQ-TAKYCIDWSAXU3JXP7DXMBG
    • X-APPLE-WEBAUTH-HSA-TRUST
    • X-APPLE-WEBAUTH-LOGIN
    • X-APPLE-WEBAUTH-USER
    • X-APPLE-WEBAUTH-PCS-Cloudkit
    • X-APPLE-WEBAUTH-PCS-Documents
    • X-APPLE-WEBAUTH-PCS-Mail
    • X-APPLE-WEBAUTH-PCS-News
    • X-APPLE-WEBAUTH-PCS-Notes
    • X-APPLE-WEBAUTH-PCS-Photos
    • X-APPLE-WEBAUTH-PCS-Sharing
    • X-APPLE-WEBAUTH-VALIDATE
    • X-APPLE-WEB-ID
    • X-APPLE-WEBAUTH-TOKEN

Questions I have that (I believe) need an answer

  1. Are any of the PUT request’s querystring parameters established per-session, or are they all long-lived “static” values that only change either per-user or per-version of the API?
  2. How many of the cookies are established per-user vs per-session?
  3. How many of the cookies are being marshalled already by PyiCloud?
  4. How many of the cookies are necessary to successfully PUT a Contact?
  5. How do I properly add the request payload to a web request using the PyiCloud functions?  How’s about if I have to drop down to the requests package?

So let’s run these down one by one (to the best of my analytic ability to spot the details).

(1) PUT request querystring parameter lifetime

When I examine the request parameters submitted on two different days (but using the same Chrome process) or across two different browsers (but on the same day), I see the following:

  1. clientBuildNumber is the same (16HProject79)
  2. clientMasteringNumber is the same (16H71)
  3. clientVersion is the same (2.1)
  4. dsid is the same (197715384)
  5. method is obviously the same (PUT)
  6. prefToken is the same (914266d4-387b-4e13-a814-7e1b29e001c3)
  7. clientId uses a different UUID (C1D3EB4C-2300-4F3C-8219-F7951580D3FD vs. 792EFA4A-5A0D-47E9-A1A5-2FF8FFAF603A)
  8. syncToken is somewhat different (DAVST-V1-p28-FT%3D-%40RU%3Dafe27ad8-80ce-4ba8-985e-ec4e365bc6d3%40S%3D1432 vs. DAVST-V1-p28-FT%3D-%40RU%3Dafe27ad8-80ce-4ba8-985e-ec4e365bc6d3%40S%3D1427)
    • which if iCloud is using standard URL encoding translates to DAVST-V1-p28-FT=-@RU=afe27ad8-80ce-4ba8-985e-ec4e365bc6d3@S=1427
    • which means the S variable varies and nothing else

Looking at the PyiCloud source, I can find places where PyiCloud generates nearly all the params:

  • base.py: clientBuildNumber (14E45), dsid (from server’s authentication response), clientId (a fresh UUID on each session)
  • contacts.py: clientVersion (2.1), prefToken (from the refresh_service() function), syncToken (from the refresh_service() function)

Since the others (clientMasteringNumber, method) are static values, there are no mysteries to infer in generating the querystring params, just code to construct.

Further, I notice that the contents of syncToken is nearly identical to the etag in the request payload:

syncToken: DAVST-V1-p28-FT=-@RU=afe27ad8-80ce-4ba8-985e-ec4e365bc6d3@S=1436
etag: C=1435@U=afe27ad8-80ce-4ba8-985e-ec4e365bc6d3

This means not only that (a) the client and/or server are incrementing some value on some unknown cadence or stepping function, but also that (b) the headers and the payload have to both contain this value.  I don’t know if any code in PyiCloud has performed this (b) kind of coordination elsewhere, but I haven’t noticed evidence of it in my reviews of the code so far.

It should be easy enough to extract the RU and S param values from syncToken and plop them into the C and U params of etag.

ISSUE

The only remaining question is, does etag’s C param get strongly validated at the server (i.e. not only that it exists, and is a four-digit number, but that its value is strongly related to syncToken’s S param)?  And if so, what exactly is the algorithm that relates C to S?  In my anecdotal observations, I’ve noticed they’re always slightly different, from off-by-one to as much as a difference of 7.

(2) How many cookies are established per-session?

Of all the cookies being tracked, only these are identical from session to session:

  • X-APPLE-WEBAUTH-USER
  • X-APPLE-WEB-ID

The rest seem to start with the same string but diverge somewhere in the middle, so it’s safe to say each cookie changes from session to session.

 

(3) How many cookies are marshalled by PyiCloud?

I can’t find any of these cookies being generated explicitly, but I did notice the base.py module mentions X-APPLE-WEBAUTH-HSA-TRUST in a comment (“Re-authenticate, which will both update the 2FA data, and ensure that we save the X-APPLE-WEBAUTH-HSA-TRUST cookie.”) and fingers X-APPLE-WEBAUTH-TOKEN in an exception thrower (“reason == ‘Missing X-APPLE-WEBAUTH-TOKEN cookie'”), so presumably most or all of these are being similarly handled.

I tried for a bit to get PyiCloud to cough up the cookies sent down from iCloud during initial session setup, but didn’t get anywhere.  I also tried to figure out where they’re being cached on my filesystem, but I haven’t yet figured out where the user’s tmp directory lives on MacOS.

(4) How many cookies are necessary to successfully PUT a Contact?

This’ll have to wait to be answered until we actually start throwing code at the endpoint.

For now, it’s probably a reasonable assumption for now that PyiCloud is able to automatically capture and replay all cookies needed by the Contacts endpoint, until we run into otherwise-unexplained errors.

(5) How to add the request payload to endpoint requests?

I can’t seem to find any pattern in the PyiCloud code that already POSTs or PUTs a dictionary of data payload back to the iCloud services, so that may be out.

I can see that it should be trivial to attach the payload data to a requests.put() call, if we ignore the cookies and preceding authentication for a second.  If I’m reading the requests quickstart correctly, the PUT request could be formed like this:

import requests
url = 'https://p28-contactsws.icloud.com/co/contacts/card/'
data_payload = {"key1" : "value1", "key2" : "value2",  ...}
url_params = {"contacts":[{contact_attributes_dictionary}]}
r = requests.put(url, data = data_payload, params = url_params)

Where key(#s) includes clientBuildNumber, clientId, clientMasteringNumber, clientVersion, dsid, method, prefToken, syncToken, and contact_attributes_dictionary includes whichever fields exist or are being added to my Contacts (e.g. firstName, lastName, phones, emailAddresses, contactId) plus the possibly-troublesome etag.

What feels tricky to me is to try to leverage PyiCloud as far as I can and then drop to the reuqests package only for generating the PUT requests back to the server.  I have a bad feeling I might have to re-implement much of the contacts.py and/or base.py modules to actually complete authentication + cookies + PUT request successfully.

I do see the same pattern used for the authentication POST, for example (in base.py’s PyiCloudService class’ authenticate() function):

req = self.session.post(
 self._base_login_url,
 params=self.params,
 data=json.dumps(data)
 )

Extension ideas

This all leads me to the conclusion that, if PyiCloud is already properly handling authentication & cookies correctly, that it shouldn’t be too hard to add a new function to the contacts.py module and generate the URL params and the data payload.

update_contact()

e.g. define an update_contact() function:

def update_contact(self, contact_dict)

# read value of syncToken
# pull out the value of the RU and S params 
# generate the etag as ("C=" + str(int(s_param) - (increment_or_decrement)) + "@U=" + ru_param
# append etag to contact_dict
# read in session params from session object as session_params ???
# contacts_url = 'https://p28-contactsws.icloud.com/co/contacts/card/'
# req = self.session.post(contacts_url, params=session_params, data=json.dumps(contact_dict))

The most interesting/scary part of all this is that if the user [i.e. anyone but me, and probably even me as well] wasn’t careful, they could easily overwrite the contents of an existing iCloud Contact with a PUT that wiped out existing attributes of the Contact, or overwrote attributes with the wrong data.  For example, what if in generating the contact_dict, they forgot to add the lastName attribute, or they mistakenly swapped the lastName attribute for the firstName attribute?

It makes me want to wrap this function in all sorts of warnings and caveats, which are mostly ignored and aren’t much help to those who fat-finger their code.  And even to generate an offline, client-side backup of all the existing Contacts before making any changes to iCloud, so that if things went horribly wrong, the user could simply restore the backup of their Contacts and at least be no worse than when they started.

edit_contact()

It might also be advisable to write an edit_contact(self, contact_dict, attribute_changes_dict) helper function that at least:

  • takes in the existing Contact (presumably as retrieved from iCloud)
  • enumerated the existing attributes of the contact
  • simplified the formatting of some of the inner array data like emailAddresses and phones so that these especially didn’t get accidentally wiped out
  • (came up with some other validation rules – e.g. limit the attributes written to contact_dict to those non-custom attributes already available in iCloud, e.g. try to help user not to overwrite existing data unless they explicitly set a flag)

And all of this hand-wringing and risk management would be reduced if the added code implemented some kind of visual UI so that the user could see exactly what they were about to irreversibly commit to their contacts.  It wouldn’t eliminate the risk, and it would be terribly irritating to page through dozens of screens of data for a bulk update (in the hopes of noticing one problem among dozens of false positives), but it would be great to see a side-by-side comparison between “data already in iCloud” and “changes you’re about to make”.

At which point, it might just be easier for the user to manually update their Contacts using iCloud.com.

Conclusion

I’m not about to re-implement much of the logic already available in iCloud.com.

I don’t even necessarily want to see my code PR’d into PyiCloud – at least and especially not without a serious discussion of the foreseeable consequences *and* how to address them without completely blowing up downstream users’ iCloud data.

But at the same time, I can’t see a way to insulate my update_contact() function from the existing PyiCloud package, so it looks like I’m going to have to fork it and make changes to the contacts module.

Update my Contacts with Python: exploring LinkedIn’s and iCloud’s Contact APIs

TL;DR Wow is it an adventure to decipher how to interact with undocumented web services like I found on LinkedIn and iCloud.  Migrating data from LinkedIn to iCloud looks possible, but I got stuck at implementing the PUT operation to iCloud using Python.

Background: Because I have a shoddy memory for details about all the people I meet, and because LinkedIn appears to be de-prioritizing their role as a professional contact manager, I want to make my iPhone Contacts my system of record for all data about people I meet professionally.  Which means scraping as much useful data as possible from LinkedIn and uploading it to iCloud Contacts (since my people-centric data is currently centered more around my iPhone than a Google Contacts approach).

In our last adventure, I stumbled across the a surprisingly well-formed and useful API for pulling data from LinkedIn about my Connections:

https://www.linkedin.com/connected/api/v2/contacts?start=40&count=10&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

Available Data

Which upon inspection of the results, gives me a lot of the data I was hoping to import into my iCloud Contacts:

  • crucial: Date we first connected on LinkedIn (“connectionDate” as time-since-epoch), Tags (“tags” as list of dictionaries), Picture (“profileImageUrl” as URI), first name (“firstName” as string), last name (“lastName” as string)
  • want: current company (“company” as dictionary), current title (“title” as string)
  • metadata: phone number (“phoneNumbers” as dictionary)

What doesn’t it give?  Notes, Twitter ID, web site addresses, previous companies, email address.  [What else does it give that could be useful?  LinkedIn profile URL (“profileUrl” as the permanent URL, not the “friendly URL” that many of us have generated such as https://www.linkedin.com/in/mikelonergan.  I can see how it would be helpful at a meetup to browse through my iPhone contacts to their LinkedIn profile to refresh myself on their work history.  Creepy, desperate, but something I’ve done a few times when I’m completely blanking.]

What can I get from the User Data Archive?  Notes are found in the Contacts.csv, and email address is found in Connections.csv.  Matching those two files’ data together with what I can pull from the Contacts API shouldn’t be a challenge (concat firstName + lastName, and among the data set of my 684 contacts, I doubt I’ll find any collisions).  Then matching those records to my iCloud Contacts *should* be just a little harder (I expect to match 50% of my existing contacts by emailAddress, then another fraction by phone number; the rest will likely be new records for my Contacts, with maybe one or two that I’ll have to merge by hand at the end).

Planning the “tracer bullet”

So what’s the smallest piece of code I can pull together to prove this scenario actually works?  It’ll need at least these features (assumes Python):

  1. can authenticate to LinkedIn via at least one supported protocol (e.g. OAuth 2.0)
  2. can pull down the first 10 JSON records from Contacts API and hold them in a list
  3. can enumerate the First + Last Name and pull out “title” for that record
  4. can authenticate to iCloud
    • Note: I may need to disable 2-factor authentication that is currently enabled on my account
  5. can find a matching First + Last Name in my iCloud Contacts
  6. can write the title field to the iCloud contact
    • Note: I’m worried least about existing data for the title field
  7. can upload the revised record to iCloud so that it replicates successfully to my iPhone

That should cover all the essential operations for the least-complicated data, without having to worry about edge cases like “what if the contact doesn’t exist in iCloud” or “what if there’s already data in the field I want to fill”.

Step 1: authenticate to LinkedIn

There are plenty of packages and modules on Github for accessing LinkedIn, but the ones I’ve evaluated all use the REST APIs, with their dual-secrets authentication mechanism, to get at the data.  (e.g. this one, this one, that one, another one).

Or am I making this more complicated than it is?  This python module simply used username + password in their call to an HTTP ‘endpoint’.  Let’s assume that judicious use of the requests package is sufficient for my needs.

I thought I’d build an anaconda kernel and a jupyter notebook to experiment with the modules I’m looking at.   And when I attempted to install the requests package in my new Anaconda environment, I get back this error:

LinkError:
Link error: Error: post-link failed for: openssl-1.0.2j-0

Quick search turns up a couple of open conda issues that don’t give me any immediate relief. OK, forget this for a bit – the “root” kernel will do fine for the moment.

Next let’s try this code and see what we get back:

import requests
r = requests.get('https://www.linkedin.com/connected/api/v2/contacts?start=40&count=10&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007', auth=('mikethecanuck@gmail.com', 'linkthis'))
r.status_code

Output is simply “401”.  Dang, authentication wasn’t *quite* that easy.

So I tried that URL in an incognito tab, and it displays this to me without an existing auth cookie:

{"status":"Member is not Logged in."}

And as soon as I open another tab in that incognito window and authenticate to the linkedin.com site, the first tab with that contacts query returns the detailed JSON I was expecting.

Digging deeper, it appears that when I authenticate to https://www.linkedin.com through the incognito tab, I receive back one cookie labelled “lidc”, and that an “lidc” cookie is also sent to the server on the successful request to the contacts API.

But setting the cookie manually with the value returned from a previous request still leads to 401 response:

url = 'https://www.linkedin.com/connected/api/v2/contacts?start=40&count=10&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007'
cookies = dict(lidc="b=OGST00:g=43:u=1:i=1482261556:t=1482347956:s=AQGoGetJeZPEDz3sJhm_2rQayX5ZsILo")
r2 = requests.get(url, cookies=cookies)

I tried two other approaches that people have used in the past – some even successfully with certain pages on LinkedIn – but eventually I decided that I’m getting ratholed on trying to reverse-engineer an undocumented (and more than likely unusually-constructed) API, when I can quite easily dump the data out of the API by hand and then do the rest of my work successfully.  (Yes I know that disqualifies me as a ‘real coder’, but I think we both know I was never going to win that medal – but I will win the medal for “results-oriented” not “pedantically chasing my tail”.)

Thus, knowing that I’ve got 684 connections on LinkedIn (saw that in the footer of a response), I submitted the following queries and copy-pasted the results into 4 separate .JSON files for offline processing:

https://www.linkedin.com/connected/api/v2/contacts?start=0&count=200&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

https://www.linkedin.com/connected/api/v2/contacts?start=200&count=200&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

https://www.linkedin.com/connected/api/v2/contacts?start=400&count=200&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

https://www.linkedin.com/connected/api/v2/contacts?start=600&count=200&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

Oddly, the four sets of results contain 196, 198, 200 and 84 items – they assert that I have 684 connections, but can only return 678 of them?  I guess that’s one of the consequences of dealing with a “free” data repository (even if it started out as mine).

Step 2: read the JSON file and parse a list of connections

I’m sure I could be more efficient than this, but as far as getting a working result, here’s the arrangement of code I used to start accessing structured list data from the Contacts API output I shunted to a file:

import json
import os
contacts_file = open("Connections-API-results.json")
contacts_data = contacts_file.read()
contacts_json = json.loads(contacts_data)
contacts_list = contacts_json['values']

Step 3: pulling data out of the list of connections

It turns out this is pretty easy, e.g.:

for contact in contacts_list:
 print(contact['name'], contact['title'])

Messing around a little further, trying to make sense of the connectionDate value from each record, I found that this returns an ISO 8601-style date string that I can use later:

import time
print(strftime("%Y-%m-%d", time.localtime(contacts_list[15]['connectionDate'] / 1000)))

e.g. for the record at index “15”, that returned 2007-03-15.

Data issue: it turns out that not all records have a profileImageUrl key (e.g. for those oddball security geeks among my contacts who refuse to publish a photo on their LinkedIn profile), so I got to handle my first expected exception 🙂

Assembling all the useful data for all my Connections I wanted into a single dictionary, I was able to make the following work (as you can find in my repo):

stripped_down_connections_list = []

for contact in contacts_list:
 name = contact['name']
 first_name = contact['firstName']
 last_name = contact['lastName']
 title = contact['title']
 company = contact['company']['name']
 date_first_connected = time.strftime("%Y-%m-%d", time.localtime(contact['connectionDate'] / 1000))

picture_url = ""
 try:
 picture_url = contact['profileImageUrl']
 except KeyError:
 pass

tags = []
for i in range(len(contact['tags'])):
tags.append(contact['tags'][i]['name'])

phone_number = ""
try:
 phone_number = {"type" : contact['phoneNumbers'][0]['type'], 
 "number" : contact['phoneNumbers'][0]['number']}
except IndexError:
 pass

stripped_down_connections_list.append({"firstName" : contact['firstName'], 
 "lastName" : contact['lastName'], 
 "title" : contact['title'], 
 "company" : contact['company']['name'],
 "connectionDate" : date_first_connected, 
 "profileImageUrl" : picture_url,
 "tags" : tags,
 "phoneNumber" : phone_number,})

Step 4: Authenticate to iCloud

For this step, I’m working with the pyicloud package, hoping that they’ve worked out both (a) Apple’s two-factor authentication and (b) read/write operations on iCloud Contacts.

I setup yet another jupyter notebook and tried out a couple of methods to import PyiCloud (based on these suggestions here), at least one of which does a fine job.  With picklepete’s suggested 2FA code added to the mix, I appear to be able to complete the authentication sequence to iCloud.

APPLE_ID = 'REPLACE@ME.COM'
APPLE_PASSWORD = 'REPLACEME'

from importlib.machinery import SourceFileLoader

foo = SourceFileLoader("pyicloud", "/Users/mike/code/pyicloud/pyicloud/__init__.py").load_module()
api = foo.PyiCloudService(APPLE_ID, APPLE_PASSWORD)

if api.requires_2fa:
    import click
    print("Two-factor authentication required. Your trusted devices are:")

    devices = api.trusted_devices
    for i, device in enumerate(devices):
        print(" %s: %s" % (i, device.get('deviceName',
        "SMS to %s" % device.get('phoneNumber'))))

    device = click.prompt('Which device would you like to use?', default=0)
    device = devices[device]
    if not api.send_verification_code(device):
        print("Failed to send verification code")
        sys.exit(1)

    code = click.prompt('Please enter validation code')
    if not api.validate_verification_code(device, code):
        print("Failed to verify verification code")
        sys.exit(1)

Step 5: matching on First + Last with iCloud

Caveat: there are a number of my contacts who have appended titles, certifications etc to their lastName field in LinkedIn, such that I won’t be able to match them exactly against my cloud-based contacts.

I’m not even worried about this step, because I quickly got worried about…

Step 6: write to the iCloud contacts (?)

Here’s where I’m stumped: I don’t think the PyiCloud package has any support for non-GET operations against the iCloud Contacts service.  There appears to be support for POST in the Reminders module, but not in any of the other services modules (including Contacts).

So I sniffed the wire traffic in Chrome Dev Tools, to see what’s being done when I make an update to any iCloud.com contact.  There’s two possible operations: a POST method call for a new contact, or a a PUT method call for an update to an existing contact.

Here’s the Request Payload for a new contact:

{“contacts”:[{“contactId”:”2EC49301-671B-431B-BC8C-9DE6AE15D21D”,”firstName”:”Tony”,”lastName”:”Stank”,”companyName”:”Stark Enterprises”,”isCompany”:false}]}

Here’s the Request Payload for an update to that existing contact (I added homepage URL):

{“contacts”:[{“firstName”:”Tony”,”lastName”:”Stank”,”contactId”:”2EC49301-671B-431B-BC8C-9DE6AE15D21D”,”prefix”:””,”companyName”:”Stark Enterprises”,”etag”:”C=1432@U=afe27ad8-80ce-4ba8-985e-ec4e365bc6d3″,”middleName”:””,”isCompany”:false,”suffix”:””,”urls”:[{“label”:”HOMEPAGE”,”field”:”http://stark.com”}]}]}

There are four requests being made for either type of change to iCloud contacts (at least via the iCloud.com web interface that I am using as a model for what the code should be doing):

  1. https://p28-contactsws.icloud.com/co/contacts/card/
  2. https://webcourier.push.apple.com/aps
  3. https://p28-contactsws.icloud.com/co/changeset
  4. https://feedbackws.icloud.com/reportStats

Here’s the details for these calls when I create a new Contact:

  1. Request URL: https://p28-contactsws.icloud.com/co/contacts/card/?clientBuildNumber=16HProject79&clientId=63D7078B-F94B-4AB6-A64D-EDFCEAEA6EEA&clientMasteringNumber=16H71&clientVersion=2.1&dsid=197715384&prefToken=914266d4-387b-4e13-a814-7e1b29e001c3&syncToken=DAVST-V1-p28-FT%3D-%40RU%3Dafe27ad8-80ce-4ba8-985e-ec4e365bc6d3%40S%3D1426
    Request Payload: {“contacts”:[{“contactId”:”E2DDB4F8-0594-476B-AED7-C2E537AFED4C”,”urls”:[{“label”:”HOMEPAGE”,”field”:”http://apple.com”}],”phones”:[{“label”:”MOBILE”,”field”:”(212) 555-1212″}],”emailAddresses”:[{“label”:”WORK”,”field”:”johnny.appleseed@apple.com”}],”firstName”:”Johnny”,”lastName”:”Appleseed”,”companyName”:”Apple”,”notes”:”Dummy contact for iCloud automation experiments”,”isCompany”:false}]}
  2. Request URL: https://p28-contactsws.icloud.com/co/changeset?clientBuildNumber=16HProject79&clientId=63D7078B-F94B-4AB6-A64D-EDFCEAEA6EEA&clientMasteringNumber=16H71&clientVersion=2.1&dsid=197715384&prefToken=914266d4-387b-4e13-a814-7e1b29e001c3&syncToken=DAVST-V1-p28-FT%3D-%40RU%3Dafe27ad8-80ce-4ba8-985e-ec4e365bc6d3%40S%3D1427
  3. Request URL: https://webcourier.push.apple.com/aps?tok=bc3dd94e754fd732ade052eead87a09098d3309e5bba05ed24272ede5601ae8e&ttl=43200
  4. Request URL: https://feedbackws.icloud.com/reportStats
    Request Payload: {“stats”:[{“httpMethod”:”POST”,”statusCode”:200,”hostname”:”www.icloud.com”,”urlPath”:”/co/contacts/card/”,”clientTiming”:395,”uncompressedResponseSize”:14469,”region”:”OR”,”country”:”US”,”time”:”Wed Dec 28 2016 12:13:48 GMT-0800 (PST) (1482956028436)”,”timezone”:”PST”,”browserLocale”:”en-us”,”statName”:”contactsRequestInfo”,”sessionID”:”63D7078B-F94B-4AB6-A64D-EDFCEAEA6EEA”,”platform”:”desktop”,”appName”:”contacts”,”isLiteAccount”:false},{“httpMethod”:”POST”,”statusCode”:200,”hostname”:”www.icloud.com”,”urlPath”:”/co/changeset”,”clientTiming”:237,”uncompressedResponseSize”:2,”region”:”OR”,”country”:”US”,”time”:”Wed Dec 28 2016 12:13:48 GMT-0800 (PST) (1482956028675)”,”timezone”:”PST”,”browserLocale”:”en-us”,”statName”:”contactsRequestInfo”,”sessionID”:”63D7078B-F94B-4AB6-A64D-EDFCEAEA6EEA”,”platform”:”desktop”,”appName”:”contacts”,”isLiteAccount”:false}]}

I am 99% sure that the only request that actually changes the Contact data is the first one (https://p28-contactsws.icloud.com/co/contacts/card/), so I’ll ignore the other three calls from here on out.

Here’s the details of the first request when I edit an existing Contact:

Request URL: https://p28-contactsws.icloud.com/co/contacts/card/?clientBuildNumber=16HProject79&clientId=792EFA4A-5A0D-47E9-A1A5-2FF8FFAF603A&clientMasteringNumber=16H71&clientVersion=2.1&dsid=197715384&method=PUT&prefToken=914266d4-387b-4e13-a814-7e1b29e001c3&syncToken=DAVST-V1-p28-FT%3D-%40RU%3Dafe27ad8-80ce-4ba8-985e-ec4e365bc6d3%40S%3D1427
Request Payload: {“contacts”:[{“lastName”:”Appleseed”,”notes”:”Dummy contact for iCloud automation experiments”,”contactId”:”E2DDB4F8-0594-476B-AED7-C2E537AFED4C”,”prefix”:””,”companyName”:”Apple”,”phones”:[{“field”:”(212) 555-1212″,”label”:”MOBILE”}],”isCompany”:false,”suffix”:””,”firstName”:”Johnny”,”urls”:[{“field”:”http://apple.com”,”label”:”HOMEPAGE”},{“label”:”HOME”,”field”:”http://johnny.name”}],”emailAddresses”:[{“field”:”johnny.appleseed@apple.com”,”label”:”WORK”}],”etag”:”C=1427@U=afe27ad8-80ce-4ba8-985e-ec4e365bc6d3″,”middleName”:””}]}

So here’s what’s puzzling me so far: both the POST (create) and PUT (edit) operations include a contactId parameter.  Its value is the same from POST to PUT (i.e. I believe that means it’s referencing the same record).  When I create a second new Contact, the contactId is different than the contactId submitted in the Request Payload for the first new Contact (so it’s presumably not a dummy value).  And yet when I look at the request/response for the initial page load when I click “+” and “New Contact”, I don’t see a request sent from the browser to the server (so the server isn’t sending down a contactID – not at that moment at least – perhaps it’s cached earlier?).

Explained another way, this is how I believe the sequence works (based on repeated analysis of the network traffic from Chrome to the iCloud endpoint and back):

  1. User loads icloud.com, Contacts page (#contacts), clicks “+” and selects “New Contact”
    • Browser sends no request, but rather builds the New Contact form from cached code
  2. User adds data and clicks the Done button for the new Contact
    • Browser sends POST request to https://p28-contactsws.icloud.com/co/contacts/card/ with a bunch of form data on the URL, a whole raft of cookies and the JSON request payload [including contactId=x]
    • Server sends response
  3. User clicks Edit on that new contact, updates some data and clicks Done
    • Browser sends PUT request to https://p28-contactsws.icloud.com/co/contacts/card/ with form data, cookies and JSON request payload [including the same contactId=x]
    • Server sends response

So the question is: if I’m creating a net-new Contact, how does the web client get a valid contactId that iCloud will accept?  Near as I can figure, digging through the javascript-packed.js this page uses, this is the function that generates a UUID at the client:

Contacts.Contact = Contacts.Record.extend({
 primaryKey: "contactId",
 contactId: CW.Record.attr(String, {
 defaultValue: function() {
 return CW.upperCaseUUID()
 }
 })

Using this function (IIUC):

UUID: function() {
 var e = new Array(36),
 t = 0,
 n = ["8", "9", "a", "b"];
 if (window.crypto && window.crypto.getRandomValues) {
 var r = new Uint8Array(18);
 crypto.getRandomValues(r);
 for (t = 0; t < 18; t++) e[t * 2 + 1] = (r[t] >> 4).toString(16), e[t * 2] = (r[t] & 15).toString(16);
 e[19] = n[r[9] >> 6]
 } else {
 while (t < 36) e[t] = (Math.random() * 16 | 0).toString(16), t++;
 e[19] = n[Math.random() * 4 | 0]
 }
 return e[8] = e[13] = e[18] = e[23] = "-", e[14] = "4", e.join("")
 }

[Aside: I sincerely hope this is a standard library for UUID, not something Apple wrote themselves.  If I ever think that I’m going to need to generate iCloud-compatible UUIDs.]

Whoa – Pause

I need to take a step back and re-examine my goals and what I can specifically address.  I have learned a lot about both LinkedIn and iCloud, but I didn’t set out to recreate them, just find a way to make consistent use of the data I already have.

Update my Contacts with Python: thinking through the options

Alright folks, that’s the bell.  When LinkedIn stops thinking of itself as a professional contact manager, you know there’s no profit in it, and it’s time to manage this stuff yourself.

Problem To Solve

I’ve been hemming and hawing for a couple of years, ever since Evernote shut down their Hello app, about how to remember who I’ve met and where I met them.  I’m a Meetup junkie (with no rehab in sight) and I’ve developed a decent network of friends and acquaintances that make it easy for me to attend new events and conferences in town – I’ll always “know” someone there (though not always remember why/how I know them or even what their name is).

When I first discovered Evernote Hello, it seemed like the perfect tool for me – provided me a timeline view of all the people I’d met, with rich notes on all the events I’d seen them at and where those places were.  It never entirely gelled, it sporadically did and did NOT support business card import (pay for play mostly), and it was only good for those people who gave me enough info for me to link them.  Even with all those imperfections, I remember regularly scanning that list (from a quiet corner at a meetup/party/conference) before approaching someone I *knew* I’d seen before, but couldn’t remember why.  [Google Glasses briefly promised to solve this problem for me too, but that tech is off somewhere, licking its wounds and promising to come back in ten years when we’re ready for it.]

What other options do I have, before settling in to “do it myself”?

  • Pay the big players e.g. SalesForce, LinkedIn
    • Salesforce: smallest SKUs I could find @ $25/month [nope]
    • LinkedIn “Sales” SKU: $65/month [NOPE]
  • Get a cheap/trustworthy/likely-to-survive-more-than-a-year app
    • Plenty of apps I’ve evaluated that sound sketchy, or likely to steal your data, or are so under-funded that they’re likely to die off in a few months

Requirements

Do it myself then.  Now I’ve got a smaller problem set to solve:

  1. Enforce synchronization between my iPhone Contacts.app, the iCloud replica (which isn’t a perfect replica) and my Google Contacts (which are a VERY spotty replica).
    • Actually, let’s be MVP about this: all I *need* right now is a way of automating edits to Contacts on my iPhone.  I assume that the most reliable way of doing this is to make edits to the iCloud.com copy of the contact and let it replicate down to my phone.
    • the Google Contacts sync is a future-proofing move, and one that theoretically sounded free (just needed to flip a toggle on my iPhone profile), but which in practice seems to be built so badly that only about 20% of my contacts have ever sync’d with Google
  2. Add/update information to my contacts such as photos, “first met” context (who introduced, what event met at) and other random details they’ve confessed to me (other attempts to hook my memory) – *WITHOUT* linking my iPhone contacts with either LinkedIn or Facebook (who will of course forever scrape all that data up to their cloud, which I do *not* want to do – to them or me).

Test the Sync

How can I test my requirements in the cheapest way possible?

  • Make hand edits to the iCloud.com contacts and check that it syncs to the iPhone Contacts.app
    • Result: sync to iPhone within seconds
  •  Make hand edits to contacts in Contacts.app and check that it syncs to iCloud.com contact
    • Result: sync to iCloud within seconds

OK, so once I have data that I want to add to an iCloud contact, and code (Python for me please!) that can write to iCloud contacts, it should be trivial to edit/append.

Here’s all the LinkedIn Data I Want

Data that’s crucial to remembering who someone is:

  • Date we first connected on LinkedIn
  • Tags
  • Notes
  • Picture

Additional data that can help me fill in context if I want to dig further:

  • current company
  • current title
  • Twitter ID
  • Web site addresses
  • Previous companies

And metadata that can help uniquely identify people when reading or writing from other directories:

  • Email address
  • Phone number

How to Get my LinkedIn connection data?

OK, so (as of 2016-12-15 at 12:30pm PST) there’s three ways I can think of pulling down the data I’ve peppered into my LinkedIn connections:

  1. User Data Archive: request an export of your user data from LinkedIn
  2. LinkedIn API: request data for specified Connections using LinkedIn’s supported developer APIs
  3. Web Scraping: iterate over every Connection and pull fields via CSS using e.g. Beautiful Soup

User Data Archive

This *sounds* like the most efficient and straightforward way to get this data.  The “Relationship Section” announcement even implies that I’ll get everything I want:

If you want to download your existing Notes and Tags, you’ll have the option to do so through March 31, 2017…. Your notes and tags will be in the file named Contacts.

The initial data dump included everything except a Contacts.csv file.  The later Complete_LinkedInDataExport_12-16-2016 [ISO 8601 anyone?] included the data promised and nearly nothing else:

  • Connections.csv: First Name, Last Name, Email Address, Current Company, Current Position, Tags
  • Contacts.csv: First Name, Last Name, Email (mostly blank), Notes, Tags

I didn’t expect to get Picture, but I was hoping for Date First Connected, and while the rest of the data isn’t strictly necessary, it’s certainly annoying that LinkedIn is so friggin frugal.

Regardless, I have almost no other source for pictures for my professional contacts, and that is pretty essential for recalling someone I’ve met only a handful of times, so while helpful, this wasn’t sufficient.

LinkedIn API

The next most reliable way to attack this data is to programmatically request it.  However, as I would’ve expected from this “roach motel” of user-generated data, they don’t even support an API to request all Connections from your user account (merely sign-in and submit data).

Where they do make reference to user data, it’s in a highly-regulated set of Member Profile fields:

  • With the r_basicprofile permission, you can get first-name, last-name, positions, picture-url plus some other data I don’t need
  • With the r_emailaddress permission, you can get the user’s primary email address
  • For developers accepted into “Apply with LinkedIn”, and with the r_fullprofile permission, you can further get date-of-birth and member-url-resources
  • For those “Apply with LinkedIn” developers who have the r_contactinfo permssion, you can further get phone-numbers and twitter-accounts

After registering a new application, I am immediately given the ability to grant the following permissions to my app: r_basicprofile, r_emailaddress.  That’ll get me picture-url, if I can figure out a way to enumerate all the Connections for my account.

(A half-hour sorting through Chrome Dev Tools’ Network outputs later…)

Looks like there’s a handy endpoint that lets the browser enumerate pretty much all the data I want:

https://www.linkedin.com/connected/api/v2/contacts?start=40&count=10&fields=id%2Cname%2CfirstName%2ClastName%2Ccompany%2Ctitle%2Clocation%2Ctags%2Cemails%2Csources%2CdisplaySources%2CconnectionDate%2CsecureProfileImageUrl&sort=CREATED_DESC&_=1481999304007

That bears further investigation.

Web Scraping

While this approach doesn’t have the built-in restrictions with the LinkedIn APIs, there’s at least three challenges I can forsee so far:

  1. LinkedIn requires authentication, and OAuth 2.0 at that (at least for API access).  Integrating OAuth into a Beautiful Soup script isn’t something I’ve heard of before, but I’m seeing some interesting code fragments and tutorials that could be helpful, and it appears that the requests package can do OAuth 1 & 2.
  2. LinkedIn has helpfully implemented the “infinite scroll” AJAX behaviour on the Connections page.
    • There are ways to work with this behaviour, but it sure feels cumbersome – to the point I almost feel like doing this work by hand would just be faster.
  3. Navigating automatically to each linked page (each Connection) from the Connections page isn’t something I am entirely confident about
    • Though I imagine it should be as easy as “for each Connection in Connections, load the page, then find the data with this CSS attribute, and store it in an array of whatever form you like”.  The mechanize package promises to make the link navigation easy.

Am I Ready for This Much Effort?

It sure feels like there’s a lot of barriers in the way to just collecting the info I’ve accumulated in LinkedIn about my connections.  Would it take me less time to just browse each connection page and hand copy/paste the data from LinkedIn to iCloud?  Almost certainly.  To together a Beautiful Soup + requests + various github modules solution would probably take me 20-30 hours I’m guessing, from all the reading and piecing together code fragments from various sources, to debugging and troubleshooting, to making something that spits out the data and then automatically uploads it without mucking up existing data.

Kinda takes the fun out of it that way, doesn’t it?  I mean, the “glory” of writing code that’ll do something I haven’t found anyone else do, that’s a little boost of ego and all.  Still, it’s hard to believe this kind of thing hasn’t been solved elsewhere – am I the only person with this bad of a memory, and this much of a drive to keep myself from looking like Leonard Shelby at every meetup?

What’s worse though, for embarking on this thing, is that I’d bet in six months’ time, LinkedIn and/or iCloud will have ‘broken’ enough of their site(s) that I wouldn’t be able to just re-use what I wrote the first time.  Maintenance of this kind of specialized/unique code feels pretty brutal, especially if no one else is expected to use it (or at least, I don’t have any kind of following to make it likely folks will find my stuff on github).

Still, I don’t think I can leave this itch entirely unscratched.  My gut tells me I should dig into that Contacts API first before embarking on the spelunking adventure that is Beautiful Soup.

Problems-to-solve: finding meetup-friendly spaces in Portland

Preamble

Sometimes I encounter a problem in my day to day life that I find so frustrating – and to me, so obvious (hasn’t been thought of by some PM already; or should’ve been caught by PO/PM acceptance validation, or during usability testing, or in the User Story’s acceptance criteria) that I can’t help thinking of how I’d have pitched this to the engineering team myself.

Think of this as a Product Guy’s version of “fantasy football” – “fantasy product ownership/management”.

Summary

User Story: as the organizer of a Meetup in Portland, I want to be able to quickly find all the meetup-friendly spaces in Portland so that I can book my meetup in a suitable space.

BDD Scenario: Given that I have an existing meetup group AND that the meetup does not have an booked meetup space, when I search for available meetup-friendly spaces in Portland, then I see a listing of such spaces in Portland including address, contact info and maximum number of attendees.

Background

I’ve been an active participant in the meetup scene in Portland for a few years now. I’ve briefly co-led a meetup as well, and been solicited to help organize a number of other meetups.

One of the phenomena I’ve observed is how challenging it can be for some meetups to find a space for their meetings. Many meetups find one space, lock it in for a year and never roam. Some meetups have to change spaces from month to month, and regularly put out a call to attendees to help them find suitable locations. And once in a while, a meetup has to change venues for space or other logistical reasons (e.g. a very popular speaker is coming to town).

Whenever I talk to meetup organizers about this part of the job, it strikes me as odd that they’re all operating like this is a high-school gossip circle: no one has all the information, there is no central place to find out where to go/who to talk to, and most people are left to ask friends if they happen to know of any spaces.

In a tech-savvy city such as Portland, where we have dozens of meetups every day, and many tech conferences a month, it’s surprising to find that getting a meetup successfully housed relies so much on word of mouth (or just using your employer’s space, if you’re lucky to be in such a position).

I’ve been at meetups in some great spaces, nearly all of them in a public-friendly space of tech employers across Portland. Where is the central directory of these spaces? Is there an intentional *lack* of public listing, so that these spaces don’t get overrun? Is this a word-of-mouth resource so that only those event organizers with a personal referral are deemed ‘vetted’ for use?

From the point of view of the owners of these spaces, I can imagine there’s little incentive to make this a seven-nights-a-week resource. Most of these employers don’t employ staff to stick around at night to police these spaces; many of them seem to leave the responsibility up to an employee [often an existing member of the meetup group] to chaperone the meetup attendees and shoo them out when they’re too tired or have to go home/back to work.

My Fantasy Scenario

Any meetup organizer in Portland will be able to find suitable meetup spaces and begin negotiating for available dates/times. A “suitable” space would be qualified on such criteria as:

  • Location
  • Number of people the space can legally accommodate
  • Number of seats available
  • Days and hours the space is potentially available (e.g. M-F 5-8, weekends by arrangement)
  • A/V availability (projector, microphone)
  • Guest wifi availability
  • Amenities (beer, food, snacks, bike parking)
  • Special notes (e.g. door access arrangements, must arrange to have employee chaperone the space)
  • Contact info to inquire about space availability [email, phone, booking system]

Future features

I can also see a need for a service that similarly lists conference-friendly spaces around town – especially for low-budget conferences that can’t afford the corporate convention spaces. I’ve been at many community-oriented conferences here in Portland, and I’m betting the number of spaces I’ve visited [e.g. Eliot Center, Armory, Ambridge, Portland Art Museum, Center for the Arts], still aren’t anywhere near the secret treasures that await.

  • Number of separate/separable rooms and their seating
  • Additional limitations/requirements e.g. if food/drinks, must always use the contracted catering

Workarounds Tried

Workaround: the http://workfrom.co service includes a filter for “Free Community Spaces”, labelled Community spaces are free and open to all, no purchase required. Common community spaces include libraries, student unions and banks. Unfortunately, as of now there are only five listings (three of them public library spaces).

Workaround: I was told by a friend that Cvent has a listing of event spaces in Portland. My search of their site led to this searchable interface. Unfortunately, this service appears to be more oriented to helping someone plan a conference or business meeting and keeping attendees entertained/occupied – where “venue type” = “corporate