Docker container commands: which goes where?

So far in my DevOps class I’ve encountered three separate places where we use parameterized commands to perform some of the Docker setup and runtime execution.

I’ve been having a conceptual crisis of confidence because I don’t definitively understand which kinds of commands go where.  So this is my exercise in deciphering the lay of the land.

Dockerfile, docker-compose.yml, docker-compose run

Here’s what I think I have inferred from what we’ve done in class tutorials:

  • Dockerfile
    • can contain one or more commands prefaced with the RUN directive
    • these will run commands outside of the to-be-built Docker container, to setup the appropriate files and environment to make the build successful
    • presumably you should do as little as necessary out here
    • example command: RUN pip install -r requirements.txt
  • docker-compose.yml
    • configures the set of containers that will be built and run together (one or more)
    • command to be run once the container is up and running
    • ?? Only one command can be configured per container ??
    • example command: command: gunicorn fooapi.wsgi:application -b :8000
  • docker-compose run [container_name]
    • commands can be run one time, arbitrarily, outside the build process itself
    • example command: docker-compose run web django-admin.py startproject fooapi .

My Confusions

  1. If my web application needs to have all the python packages installed as dependencies, why isn’t pip install -r requirements.txt being run once the appropriate container is up and running?
  2. If the use of a RUN command in Dockerfile creates a scratch space outside all containers, why would I need to install python dependencies to be able to create a PostgreSQL container cf. Assignment 3? [Leaving aside the advice I’ve heard that putting databases in containers isn’t generally necessary or advisable]
  3. What is the net effect of running commands inside the “docker-compose run [container_name]” wrapper?  Why couldn’t/shouldn’t I run that command as a RUN command from the Dockerfile, and then copy the resulting files into the /code folder that we’re creating in Assignment 3?
  4. Does docker-compose run run commands inside an already-built container?

As I learn answers to these questions, with any luck I’ll return here to annotate what I’ve learned.

Notes to self: merging my fork with upstream

It’s supposed to be as natural as breathing, right?  See a neat repository on Github, decide you want to use the code and make some minor changes to it right?  So you fork the sucker, commit some change, maybe push a PR back to the original repo?

Then, you want to keep your repo around – I dunno, maybe it’s for vanity, or maybe you’re continuing to make changes or use the project (and maybe, just maybe, you’ll find yourself wanting to push another PR in the future?).  Or maybe messages like this just bother your OCD:

github-branch-is-xx-commits-behind

Eventually, most developers will run into a situation in which they wish to re-sync their forked version of a project with the updates that have been made in “upstream”.

Should be dead easy, yes?  People are doing this all the time, yes?  Well, crap.  If that’s the case, then I’m an idiot because I’d tried this a half-dozen times and never before arrived at the beautiful message “This branch is even with…”.  So I figured I’d write it out (talk to the duck), and in so doing stumble on the solution.

GitHub help is supposed to help, e.g. Syncing a fork.  Which depends on Configuring a remote for a fork, and which is followed by Pushing to a remote.

Which for a foreign repo named e.g. “hackers/hackit” means the following stream of commands (after I’ve Forked the repo in GitHub.com and git clone‘d the repo on my local machine):

git remote add upstream git@github.com:hackers/hackit.git
git fetch upstream
git checkout master
git merge upstream/master

That last command will often result in a bunch of conflicts, if you’ve made any changes, e.g.:

git merge upstream/master
Auto-merging package.json
CONFLICT (content): Merge conflict in package.json
Auto-merging README.md
Auto-merging .travis.yml
CONFLICT (content): Merge conflict in .travis.yml
Auto-merging .babelrc
Automatic merge failed; fix conflicts and then commit the result.

At this point I temporarily abandon the command line and dive into my favourite editor (Visual Studio Code with a handful of extensions) to resolve the conflicting files.

Once I’d merged changes from both sources (mine and upstream), then it was a simple matter of the usual commands:

git add .
git commit -m "merged changes from upstream"
git push

And the result is…

github-branch-is-xx-commits-ahead

(No it wasn’t quite the “even” paradise, but I’ll take it.)

Aside

I somehow got myself into a state where I couldn’t get the normal commands to work.  For example, when I ran git push origin master, I get nowhere:

git push origin master
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Or git push:

git push
ERROR: Permission to hackers/hackit.git denied to MikeTheCanuck.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Then when I added upstream…:

git remote add upstream git@github.com:hackers/hackit.git

…and ran git remote -v…:

git remote -v
upstream git@github.com:hackers/hackit.git (fetch)
upstream git@github.com:hackers/hackit.git (push)

…it appears I no longer had a reference to origin. (No idea how that happened, but hopefully these notes will help me not go astray again.)  Adding back the reference to origin seemed the most likely solution, but I didn’t get the kind of results I wanted:

git remote add origin git@github.com:mikethecanuck/hackit.git
git remote -v
origin git@github.com:mikethecanuck/hackit.git (fetch)
origin git@github.com:mikethecanuck/hackit.git (push)
upstream git@github.com:hackers/hackit.git (fetch)
upstream git@github.com:hackers/hackit.git (push)
git push origin master
To github.com:mikethecanuck/hackit.git
 ! [rejected]        master -> master (fetch first)
error: failed to push some refs to 'git@github.com:mikethecanuck/hackit.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

 

And when I pushed with no params, I went right back to the starting place:

git push
ERROR: Permission to hackers/hackit.git denied to MikeTheCanuck.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

(I finally rm -rf‘d my forked repo, cloned it again, and started over – that’s how I got to the first part of the article.)

AWS wrangling, round 4: automating the setup of a static website

This time around, let’s automate the steps I manually performed last time (as much as the S3 APIs allow CLI interaction, which is sadly not much).

Install the AWS CLI (command line interface)

Follow the instructions for your operating system from here:

https://aws.amazon.com/cli/

Run the following commands

Create a new bucket (CLI)

Note: “mikecanuckbucket” is the bucket I’m creating.

aws s3 mb s3://mikecanuckbucket

Which returns the following output:

make_bucket: mikecanuckbucket

Configure the static website (CLI)

aws s3 website s3://mikecanuckbucket --index-document index.html

Which returns nothing if it succeeds.

Set bucket-level permissions (Console)

You have to use the console for this one (or at least, I couldn’t find a CLI command to interact with bucket-level permissions).  This time around I tried the bucket policy approach, and used this policy example as my template:

  • Load the AWS S3 console
  • select your bucket
  • click the Properties button
  • expand the Permissions section
  • click the Add bucket policy button
  • Paste the bucket policy you’ve constructed – in my example, I simply substituted “mikecanuckbucket” for “examplebucket” from the example
  • Click Save

Note: the bucket policy is immediately applied.

Upload a web page (CLI)

Note: “helloworld.html” is the example file I’m uploading to my bucket.

aws s3 cp helloworld.html s3://mikecanuckbucket

Set file-level permissions (Console)

Hah!  If you used the bucket policy like me, this won’t actually be necessary.  One convenient advantage of this inconvenience.

If you didn’t, then you’ll have to configure file-level permissions like I did in the last post.

Retrieve the page link (Console)

  • Select the new file in your new bucket
  • Click the Properties button
  • Click (or copy) the URL listed as Link

Comments

I’m disappointed that the S3 team hasn’t made it possible to eliminate the manual (Console) steps in an operation like this.  I must be missing something, or perhaps the AWS shell is where all incomplete features will be added in the future?

I happened to notice a git issue asking whether aws-shell or SAWS was the future, and according to the most active developer in both, it appears to be aws-shell.  Presumably this project is being actively developed then (though after a year and a half, it’s still labelled “The aws-shell is currently in developer preview” – and the vast majority of commits are from a year ago).  However, it’s sad to see some pretty significant issues have remained open over a year – that’s a project that sounds like it’s on life support, not fully staffed.  Or maybe the AWS APIs are still just that incomplete?

It’s also discouraging to see it mention that “The aws-shell accepts the same commands as the AWS CLI, except you don’t need to provide the aws prefix”.  This implies that it’s simply relying on the AWS CLI to implement additional commands, rather that implement themselves.  And certainly my cursory inspection of the autocomplete commands bears this out (no new options to the base s3 command palette).  [That’s understandable – it’s painful as a development organization to duplicate functionality that’s ostensibly being supported already by one branch of the org.]

Still, it is disappointing to see that at this point in the maturity lifecycle of AWS, there are still this many discontinuities to create such a disjoint experience.  My experience with the official tutorials is similar – some are great, some are frankly terrible botched efforts, and I would’ve expected better from The Leader in Cloud.  [Hell, for that matter, some capabilities themselves such as CodeDeploy are still a botched mess, at least as far as how difficult it was for me to succeed EVEN WITH a set of interdependent – admittedly poorly-written – tutorials.]

 

AWS wrangling, round 3: simplest possible, manually-configured static website

Our DevOps instructor Dan asked us to host a static HelloWorld page in S3.  After last week’s over-scoped tutorial, I started digging around in the properties of an S3 bucket, and discovered it was staring me in the face (if only I’d stared back).

Somehow I’d foolishly gotten into the groove of AWS tutorials and assumed if I found official content, that must be the most well-constructed approach based on the minimal needs of most of their audience, so I didn’t question the complexity and poorly-constructed lessons until long after I was committed to see it through.  [Thankfully I was able to figure out at least one successful path through those vagaries, or else I’d probably still be stubborn-through-seething and trying to debug the black box that are IAM policy attachments.]

Starting Small: S3 bucket and nothing else

Create Bucket

Modify the bucket-level Permissions

  • Select the new bucket and click Properties
  • Expand the Permissions section, click Add more permissions
  • In the Grantee selector, choose Everyone
  • check the List checkbox
  • click Save

Enable static website hosting

  • In the Bucket Properties, expand the Static Website Hosting section
  • Select enable website hosting
  • In the Index Document textbox, enter your favoured homepage name (e.g. index.html)
  •  click Save

Upload your web content

  • In the Actions button (on the top-left of the page), select Upload
  • Click Add files, select a simple HTML file, and click Start Upload
    • If you don’t have a suitable HTML file, copy the following to a text editor on your computer and save it (e.g. as helloworld.html)
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
      <html>
      <head>
       <title>Hello, World!</title>
       <style>
       body {
       color: #ffffff;
       background-color: #0188cc;
       font-family: Arial, sans-serif; 
       font-size:14px;
       }
       </style>
      </head>
      <body>
       

      Hello, World!

      You have successfully uploaded a static web page to AWS S3

      </body> </html>

Modify the content-level Permissions

  • Select the newly-uploaded file, then click Properties
  • expand the Permissions section and click Add more permissions
  • In the Grantee selector, choose Everyone
  • check the Open/Download checkbox and click Save

Now to confirm that your web page is available to web users, find the Link in the Properties for the file and click it – here’s my test file:

Screenshot 2017-01-15 10.26.39.png

If you’ve done everything correctly, you should see something like this:

Screenshot 2017-01-15 10.31.01.png

If one or more of the Permissions aren’t correct, you’ll see this (or at least, that’s what I’m getting in Chrome):

Screenshot 2017-01-15 10.29.47.png

 

AWS Tutorial wrangling, effort 2: HelloWorld via CodeDeploy

I’m taking a crash course in DevOps this winter, and our instructor assigned us a trivial task: get Hello World running in S3.

I found this tutorial (tantalizingly named “Deploy a Hello World Application with AWS CodeDeploy (Windows Server)”), figured it looked close enough (and if not, it’d keep me limber and help me narrow in on what I *do* need) so I foolishly dove right in.

TL;DR I found the tutorial damnably short on explicit clarity – lots of references to other tutorials, and plenty of incomplete or vague instructions, it seems this was designed by someone who’s already overly-familiar with AWS and didn’t realize the kinds of ambiguities they’d left behind.

I got myself all the way to Step 3 and was faced with this error – little did I know this was just the first of many IAM mysteries to solve:

Mac4Mike:aws mike$ aws iam attach-role-policy --role-name CodeDeployServiceRole --policy-arn arn:aws:iam::aws:policy:/service-role/AWSCodeDeployRole
An error occurred (AccessDenied) when calling the AttachRolePolicy operation: User: arn:aws:iam::720781686731:user/Mike is not authorized to perform: iam:AttachRolePolicy on resource: role CodeDeployServiceRole

Back the truck up, Mike

Hold on, what steps preceded this stumble?

CodeDeploy – Getting Started

Well, first I pursued the CodeDeploy Getting Started path:

  • Provision an IAM user
    • It wasn’t clear from the referring tutorials, so I learned by trial and error to create a user with CLI permissions (Access/Secret key authN), not Console permissions
  • Assigning them the specified policies (to enable CodeDeploy and CloudFormation)
  • Creating the specifiedCodeDeployServiceRole service role
    • The actual problem arose here, where I ran the command as specified in the guide

I tried this with different combinations of user context (Mike, who has all access to EC2, and Mike-CodeDeploy-cli, who has all the policies assigned in Step 2 of Getting Started) AND the –policy-arn parameter (both the Getting Started string and the one dumped out by the aws iam create-role command (arn:aws:iam::720781686731:role/CodeDeployServiceRole)).

And literally, searching on this error and variants of it, there appear to be no other people who’ve ever written about encountering this.  THAT’s a new one on me.  I’m not usually a trailblazer (even of the “how did he fuck this up *that* badly?” kind of trailblazing…)

OK, so then forget it – if the CLI + tutorial can’t be conquered, let’s try the Console-based tutorial steps.   [Note: in both places, they state it’s important that you “Make sure you are signed in to the AWS Management Console with the same account information you used in Getting Started.”  Why?  And what “account information” do they mean – the user with which you’re logged into the web console, or the user credentials you provisioned?]

I was able to edit the just-created CodeDeployServiceRole and confirm all the configurations they specified *except* where they state (in step 4), “On the Select Role Type page, with AWS Service Roles selected, next to AWS CodeDeploy, choose Select.”  Not sure what that means (which should’ve pulled me in the direction of “delete and recreate this role”), but I tried it out as-is anyway.  [The only change I had to make so far was to attach AWSCodeDeployRole.]

Reading up on the AttachRolePolicy action, it appears that –policy-arn refers to the permissions you wish to attach to the targeted role, and –role-name refers to the role getting additional permissions.  That would mean I’m definitely meant to attach “arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole” policy to CodeDeployServiceRole.  (Still doesn’t explain why I lack the AttachRolePolicy permission in either of the IAM Users I’ve defined, nor how to add that permission.)

Instead, with no help from any online docs or discussions, I discovered that it’s possible to assign the individual permission by starting with this interface: https://console.aws.amazon.com/iam/home?#/policies:

  • Click Create Policy
  • Select Policy Generator
  • AWS Service: Select AWS Identity and Access Management
  • Actions: Attach Role Policy
  • ARN: I tried constructing two policies with (arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole) and (arn:aws:iam::720781686731:role/CodeDeployServiceRole)
    • The first returned “AccessDenied” and with the second, the command I’m fighting with returned “InvalidInput”

Then I went to the Users console:

  • select the user of interest (Mike-CodeDeploy-cli in my journey)
  • click Add Permissions
  • select “Attach existing policies directly”
  • select the new Policy I just created (where type = Customer managed, so it’s relatively easy to spot among all the other “AWS managed” policies)

As I mentioned, the second construction returned this error to the command:

An error occurred (InvalidInput) when calling the AttachRolePolicy operation: ARN arn:aws:iam::720781686731:role/CodeDeployServiceRole is not valid.

Nope, wait, dammit – the ARN is the *policy*, not the *object* to which the policy grants permission…

Here’s where I started ranting to myself…

Tried it a couple more times, still getting AccessDenied, so screw it.  [At this point I conclude AWS IAS is an immature dog’s breakfast – even with an explicit map you still end up turned in knots.]

So I just went to the Role CodeDeployServiceRole and attached both policies (I’m pretty sure I only need to attach the policy AWSCodeDeployRole but I’m adding the custom AttachRolePolicy-CodeDeployRole because f it I just need to get through this trivial exercise).

[Would it kill the folks at AWS to draw a friggin picture of how all these capabilities with their overlapping terminology are related?  Cause I don’t know about you, but I am at the end of my rope trying to keep these friggin things straight.  Instead, they have a superfluous set of fragmented documented and tutorials, which it’s clear they’ve never usability tested end-to-end, and for which they assume way too much existing knowledge & context.]

I completed the rest of the InstanceProfile creation steps (though I had to create a second one near the end, because the console complained I was trying to create one that already existed).

CodeDeploy – create a Windows instance the “easy” way

Then of course we’re on to the fun of creating a Windows Instance in AWS.  Brave as I am, I tried it with CloudFormation.

I grabbed the CLI command and substituted the following two Parameter values in the command:

  • –template-url:

    ttp://s3-us-west-2.amazonaws.com/aws-codedeploy-us-west-2/templates/latest/CodeDeploy_SampleCF_Template.json (for the us-west-2 region I am closest to)

  • Parameter-Key=KeyPairName: MBP-2009 (for the .pem file I created a while back for use in SSH-managing all my AWS operations)

The first time I ran the command it complained:

You must specify a region. You can also configure your region by running "aws configure".

So I re-ran aws configure and filled in “us-west-2” when it prompted for “Default region name”.

Second time around, it spat out:

{
    "StackId": "arn:aws:cloudformation:us-west-2:720781686731:stack/CodeDeployDemoStack/d1c817d0-d93e-11e6-8ee1-503f20f2ade6"
}

They tell us not to proceed until this command reports “CREATE_COMPLETED”, but wow does it take a while to stop reporting “None”:

aws cloudformation describe-stacks --stack-name CodeDeployDemoStack --query "Stacks[0].StackStats" --output text

When I went looking at the cloudformation console (blame it on lack of patience), it reported my instance(s)’ status was “ROLLBACK_COMPLETE”.  Now, I’m no AWS expert, but that doesn’t sound like a successful install to me.  I headed to the details, and of course something else went horribly wrong:

  • CREATE_FAILED – AWS::EC2::Instance – API: ec2:RunInstances Not authorized for images: [ami-7f634e4f]

CodeDeploy – create a Windows instance the “hard” way

So let’s forget the “easy” path of CloudFormation.  Try the old-fashioned way of creating a Windows instance, and see if I can make it through this:

  • Deciding among Windows server AMI’s is a real blast – over 600 of them!
  • I narrowed it down to the “Windows_Server-2016-English-Full-Base-2016.12.24”
    • Nano is only available to Windows Assurance customers
    • Enterprise is way more than I’d need to serve a web page
    • Full gives you the Windows GUI to manage the server, whereas Core only includes the PowerShell (and may not even allow RDP access)
    • I wanted to see what the Manage Server GUI looks like these days, otherwise I probably would’ve tried Core
    • Note: there were three AMI all prefixed “Windows_Server-2016-English-Full-Base”, I just chose the one with the latest date suffix (assuming it’s slightly more up-to-date with patches)
  • I used the EC2 console to get the Windows password, then installed the Microsoft Remote Desktop client for Mac to enable me to interactively log in to the instance

Next is configuring the S3 bucket:

  • There is some awfully confusing and incomplete documentation here
  • There are apparently two policies to be configured, with helpful sample policies, but it’s unclear where to go to attach them, or what steps to take to make sure this occurs
  • It’s like the author has already done this a hundred times and knows all the steps by heart, but has forgotten that as part of a tutorial, the intended audience are people like me who have little or no familiarity with the byzantine interfaces of AWS to figure out where to attach these policies [or any of the other hundred steps I’ve been through over the last few weeks]
  • I *think* I found where to attach the first policy (giving permission to the Amazon S3 Bucket) – I attached this policy template (substituting both the AWS account ID [111122223333] and bucket name [codedeploydemobucket] for the ones I’m using):
    { "Statement": [ { "Action": ["s3:PutObject"], "Effect": "Allow", "Resource": "arn:aws:s3:::codedeploydemobucket/*", "Principal": { "AWS": [ "111122223333" ] } } ] }
  • I also decided to attach the second recommended policy to the same bucket as another bucket policy:
    { "Statement": [ { "Action": ["s3:Get*", "s3:List*"], "Effect": "Allow", "Resource": "arn:aws:s3:::codedeploydemobucket/*", "Principal": { "AWS": [ "arn:aws:iam::80398EXAMPLE:role/CodeDeployDemo" ] } } ] }
  • Where did I finally attach them?  I went to the S3 console, clicked on the bucket I’m going to use (called “hacku-devops-testing”), selected the Properties button, expanded the Permissions section, and clicked the Add bucket policy button the first time.  The second time, since it would only allow me to edit the bucket policy, I tried Add more permissions – but that don’t work, so I tried editing the damned bucket policy by hand and appending the second policy as another item in the Statement dictionary – after a couple of tries, I found a combination that the AWS bucket policy editor would accept, so I’m praying this is the intended combination that will all this seductive tutorial to complete:
    {
     "Version": "2008-10-17",
     "Statement": [
     {
     "Effect": "Allow",
     "Principal": {
     "AWS": "arn:aws:iam::720781686731:root"
     },
     "Action": "s3:PutObject",
     "Resource": "arn:aws:s3:::hacku-devops-testing/*"
     },
     {
     "Action": [
     "s3:Get*",
     "s3:List*"
     ],
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::hacku-devops-testing/*",
     "Principal": {
     "AWS": [
     "arn:aws:iam::720781686731:role/CodeDeployDemo-EC2-Instance-Profile"
     ]
     }
     }
     ]
    }

CodeDeploy – actually deploying code

I followed the remaining commands (closely – gotta watch every parameter and fill in the correct details, ugh).  But thankfully this was the trivial part.  [I guess they got the name “CodeDeploy” right – it’s far more attractive than “CodeDeployOnceYouFoundTheLostArkOfTheCovenantToDecipherIAMIntricacies”.]

Result

Success!  Browsing to the public DNS of the EC2 instance showed me the Hello World page I’ve been trying to muster for the past three days!

Conclusion

This tutorial works as a demonstration of how to marshall a number of contributing parts of the AWS stack: CodeDeploy (whose “ease of deployment” benefits I can’t yet appreciate, considering how labourious and incomplete/error-prone this tutorial was), IAM (users, roles, groups, policies), S3, EC2 and AMI.

However, as a gentle introduction to a quick way to get some static HTML on an AWS endpoint, this is a terrible failure.  I attacked this over a span of three days.  Most of my challenges were in deciphering the mysteries of IAM across the various layers of AWS.

In a previous life I was a security infrastructure consultant, employed by Microsoft to help decipher and troubleshoot complex interoperable security infrastructures.  I prided myself on being able to take this down to the lowest levels and figure out *exactly* what’s going wrong.  And while I was able to find *a* working pathway through this maze, my experience here and my previous expertise tells me that AWS has a long way to go to make it easy for AWS customers to marshall all their resources for secure-by-default application deployments.  [Hell, I didn’t even try to enhance the default policies I encountered to limit the scope of what remote endpoints or roles would have access to the HTTP and SSH endpoints on my EC2 instance.  Maybe that’s a lesson for next time?]

 

Update my Contacts with Python: using pyobjc, Contacts.app & vCards, Swift or my own two hands?

I’m still on a mission to update my iCloud Contacts using PyiCloud to consolidate the data I’ve retrieved from LinkedIn.  Last time I convinced myself to add an update_contact() function to a fork of PyiCloud’s contacts module, and so far I haven’t had any nibbles on the issue I’ve filed in the PyiCloud project a couple of days ago.

I was looking further at the one possibly-working pattern in the PyiCloud project that appears to implement a write back to the iCloud APIs: the reminders module with its post() method.  What’s interesting to me is that in that method, the JSON submitted in the data parameter includes the key:value pair “etag”: None.  I gnashed my teeth over how to construct a valid etag in my last post, and this code implies to me (assuming it’s still valid and working against the Reminders API) that the etag value is optional (well, the key must be specified, but the complicated value may not be needed).

Knowing that this sounds too easy, I watched a new Reminder getting created through the icloud.com web client, and sure enough Chrome Dev Tools shows me that in the Request Payload, etag is set to null.  Which really tells me nothing now about the requirement for the Contacts API…

Arrested Development

Knowing that this was going to be a painful brick wall to climb, I decided to pair up with a python expert to look for ways to dig out from this deep, dark hole.  Lucky me, I have a good relationship with the instructor from my python class from late last year.  We talked about where I am stuck and what he’d recommend I do to try to break through this issue.

His thinking?  He immediately abandoned the notion of deciphering an undocumented API and went looking around the web for docs and alternatives.  Turns out there are a couple of options:

  1. Apple has in its SDKs a Contacts framework that supports Swift and Objective-C
  2. There are many implementations of Python & other languages that access the MacOS Contacts application (Contacts.app)

Contacts via Objective-C on MacOS

  • Contacts Framework is available in XCode
  • There appears to be a bidirectional bridge between Python and Objective-C
  • There is further a wrapper for the Contacts framework (which gets installed when you run pip install pyobjc)
  • But sadly, there is nothing even resembling a starter kit example script for instantiating and using the Contacts framework wrapper

Contacts via Contacts.app on MacOS

  • We found a decent-looking project (VObject) that purports to access VCard files, which is the underlying  data layout for import/export from Contacts.app
  • And another long-lived project (vcard) for validating VCards
  • This means I would have to manually import VCard file(s) into Contacts.app, and would still have to figure out how Contacts.app knows how to match/overwrite an imported Contact with an existing Contact (or I’ll always be backing up, deleting and importing)
  • HOWEVER, in exploring the content of the my Contacts.app and comparing to what I have in my iPhone Contacts, there’s definitely something extra going on here
    • I have at least one contact displayed in Contacts.app who is neither listed in my iPhone/iCloud contacts nor Google Contacts – given the well-formed LinkedIn data in the contact record, I’m guessing this is being implicitly included via Internet Accounts (the LinkedIn account configured here):
      screenshot-2017-01-06-09-45-24
    • What would happen if I imported a vCard with the same UID (the iCloud UUID)?
    • What would happen if I imported a vCard that exists in both iCloud and LinkedIn – would the iCloud (U)UID correctly match and merge the vCard to the right contact, or would we get a duplicate?
  • Here at least I see others acknowledge that it’s possible to create non-standard types for ADR, TEL (and presumably email and URL types, if they’re different).
  • Watch out: if you have any non-ASCII characters in your Address Book, exporting will generate the output as UTF-16.
  • Watch out: here’s a VObject gotcha.

Crazy Talk: Swift?

  • I *could* go learn enough Swift to interface with the JSON data I construct in Python
  • There’s certainly a plethora of articles (iOS-focused) and tutorials to help folks use the Contacts framework via Swift – which all seem to assume you want to build a UI app (not just a script) – I guess I understand the bias, but boy do I feel left out just wanting to create a one-time-use script to make sure I don’t fat-finger something and lose precious data (my wetware memory is lossy enough as it is)

Conclusion: Park it for now

What started out as a finite-looking piece of work to pull LinkedIn data into my current contacts of record, turned into a never-ending series of questions, murky code pathfinding  and band-aiding multiple technologies together to do something that should ostensibly be fairly straightforward.

Given that the best options I have at this point are (a) reverse-engineer an undocumented Apple API, (b) try to leverage an Objective-C bridge that *no one* else has tried to use for this new Contacts framework, or (c) decipher how Contacts.app interacts in the presence of vCards and all the interlocking contacts services (iCloud, Google, LinkedIn, Facebook)…I’m going to step away from this for a bit, let my brain tease apart what I’m willing to do for fun and how much more effort I’m willing to put in for fun/”the community”, and whether I’ve crossed The Line for a one-time effort and should just manually enter the data myself.

Notes to self: installing Vagrant via homebrew

It’s as easy as

brew install vagrant

Ha! Nice try there, buddy:

Error: No available formula with the name "vagrant" 
==> Searching for similarly named formulae...
Error: No similarly named formulae found.
==> Searching taps...
These formulae were found in taps:
homebrew/completions/vagrant-completion  Caskroom/cask/vagrant-manager
Caskroom/cask/vagrant-bar                Caskroom/cask/vagrant
To install one of them, run (for example):
  brew install homebrew/completions/vagrant-completion

According to this discussion, it’s not allowed by homebrew but the homebrew-cask project enables users to install vagrant.  (So *that’s* why I couldn’t get Chrome, Dropbox, 1Password, VLC and other apps installed view homebrew – there’s some rule or constraint in homebrew that only enables non-GUI apps.)

So let’s install the Cask tools – this article makes me feel silly even – you just have to know to use the “cask” keyword, as in:

brew cask install vagrant

Thus do I get vagrant v1.9.1 in one (slightly unexpected) command line.  (And don’t forget virtualbox and vagrant-manager!)

Which is super-cool, because many package managers end up with non-current builds of the tools in their catalogs.

And as a bonus, they mention a bunch of quicklook plugins that I never even thought to go looking for – markdown, syntax-highlighted code,  JSON, CSV and more!