AWS Tutorial wrangling, effort 2: HelloWorld via CodeDeploy

I’m taking a crash course in DevOps this winter, and our instructor assigned us a trivial task: get Hello World running in S3.

I found this tutorial (tantalizingly named “Deploy a Hello World Application with AWS CodeDeploy (Windows Server)”), figured it looked close enough (and if not, it’d keep me limber and help me narrow in on what I *do* need) so I foolishly dove right in.

TL;DR I found the tutorial damnably short on explicit clarity – lots of references to other tutorials, and plenty of incomplete or vague instructions, it seems this was designed by someone who’s already overly-familiar with AWS and didn’t realize the kinds of ambiguities they’d left behind.

I got myself all the way to Step 3 and was faced with this error – little did I know this was just the first of many IAM mysteries to solve:

Mac4Mike:aws mike$ aws iam attach-role-policy --role-name CodeDeployServiceRole --policy-arn arn:aws:iam::aws:policy:/service-role/AWSCodeDeployRole
An error occurred (AccessDenied) when calling the AttachRolePolicy operation: User: arn:aws:iam::720781686731:user/Mike is not authorized to perform: iam:AttachRolePolicy on resource: role CodeDeployServiceRole

Back the truck up, Mike

Hold on, what steps preceded this stumble?

CodeDeploy – Getting Started

Well, first I pursued the CodeDeploy Getting Started path:

  • Provision an IAM user
    • It wasn’t clear from the referring tutorials, so I learned by trial and error to create a user with CLI permissions (Access/Secret key authN), not Console permissions
  • Assigning them the specified policies (to enable CodeDeploy and CloudFormation)
  • Creating the specifiedCodeDeployServiceRole service role
    • The actual problem arose here, where I ran the command as specified in the guide

I tried this with different combinations of user context (Mike, who has all access to EC2, and Mike-CodeDeploy-cli, who has all the policies assigned in Step 2 of Getting Started) AND the –policy-arn parameter (both the Getting Started string and the one dumped out by the aws iam create-role command (arn:aws:iam::720781686731:role/CodeDeployServiceRole)).

And literally, searching on this error and variants of it, there appear to be no other people who’ve ever written about encountering this.  THAT’s a new one on me.  I’m not usually a trailblazer (even of the “how did he fuck this up *that* badly?” kind of trailblazing…)

OK, so then forget it – if the CLI + tutorial can’t be conquered, let’s try the Console-based tutorial steps.   [Note: in both places, they state it’s important that you “Make sure you are signed in to the AWS Management Console with the same account information you used in Getting Started.”  Why?  And what “account information” do they mean – the user with which you’re logged into the web console, or the user credentials you provisioned?]

I was able to edit the just-created CodeDeployServiceRole and confirm all the configurations they specified *except* where they state (in step 4), “On the Select Role Type page, with AWS Service Roles selected, next to AWS CodeDeploy, choose Select.”  Not sure what that means (which should’ve pulled me in the direction of “delete and recreate this role”), but I tried it out as-is anyway.  [The only change I had to make so far was to attach AWSCodeDeployRole.]

Reading up on the AttachRolePolicy action, it appears that –policy-arn refers to the permissions you wish to attach to the targeted role, and –role-name refers to the role getting additional permissions.  That would mean I’m definitely meant to attach “arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole” policy to CodeDeployServiceRole.  (Still doesn’t explain why I lack the AttachRolePolicy permission in either of the IAM Users I’ve defined, nor how to add that permission.)

Instead, with no help from any online docs or discussions, I discovered that it’s possible to assign the individual permission by starting with this interface: https://console.aws.amazon.com/iam/home?#/policies:

  • Click Create Policy
  • Select Policy Generator
  • AWS Service: Select AWS Identity and Access Management
  • Actions: Attach Role Policy
  • ARN: I tried constructing two policies with (arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole) and (arn:aws:iam::720781686731:role/CodeDeployServiceRole)
    • The first returned “AccessDenied” and with the second, the command I’m fighting with returned “InvalidInput”

Then I went to the Users console:

  • select the user of interest (Mike-CodeDeploy-cli in my journey)
  • click Add Permissions
  • select “Attach existing policies directly”
  • select the new Policy I just created (where type = Customer managed, so it’s relatively easy to spot among all the other “AWS managed” policies)

As I mentioned, the second construction returned this error to the command:

An error occurred (InvalidInput) when calling the AttachRolePolicy operation: ARN arn:aws:iam::720781686731:role/CodeDeployServiceRole is not valid.

Nope, wait, dammit – the ARN is the *policy*, not the *object* to which the policy grants permission…

Here’s where I started ranting to myself…

Tried it a couple more times, still getting AccessDenied, so screw it.  [At this point I conclude AWS IAS is an immature dog’s breakfast – even with an explicit map you still end up turned in knots.]

So I just went to the Role CodeDeployServiceRole and attached both policies (I’m pretty sure I only need to attach the policy AWSCodeDeployRole but I’m adding the custom AttachRolePolicy-CodeDeployRole because f it I just need to get through this trivial exercise).

[Would it kill the folks at AWS to draw a friggin picture of how all these capabilities with their overlapping terminology are related?  Cause I don’t know about you, but I am at the end of my rope trying to keep these friggin things straight.  Instead, they have a superfluous set of fragmented documented and tutorials, which it’s clear they’ve never usability tested end-to-end, and for which they assume way too much existing knowledge & context.]

I completed the rest of the InstanceProfile creation steps (though I had to create a second one near the end, because the console complained I was trying to create one that already existed).

CodeDeploy – create a Windows instance the “easy” way

Then of course we’re on to the fun of creating a Windows Instance in AWS.  Brave as I am, I tried it with CloudFormation.

I grabbed the CLI command and substituted the following two Parameter values in the command:

  • –template-url:

    ttp://s3-us-west-2.amazonaws.com/aws-codedeploy-us-west-2/templates/latest/CodeDeploy_SampleCF_Template.json (for the us-west-2 region I am closest to)

  • Parameter-Key=KeyPairName: MBP-2009 (for the .pem file I created a while back for use in SSH-managing all my AWS operations)

The first time I ran the command it complained:

You must specify a region. You can also configure your region by running "aws configure".

So I re-ran aws configure and filled in “us-west-2” when it prompted for “Default region name”.

Second time around, it spat out:

{
    "StackId": "arn:aws:cloudformation:us-west-2:720781686731:stack/CodeDeployDemoStack/d1c817d0-d93e-11e6-8ee1-503f20f2ade6"
}

They tell us not to proceed until this command reports “CREATE_COMPLETED”, but wow does it take a while to stop reporting “None”:

aws cloudformation describe-stacks --stack-name CodeDeployDemoStack --query "Stacks[0].StackStats" --output text

When I went looking at the cloudformation console (blame it on lack of patience), it reported my instance(s)’ status was “ROLLBACK_COMPLETE”.  Now, I’m no AWS expert, but that doesn’t sound like a successful install to me.  I headed to the details, and of course something else went horribly wrong:

  • CREATE_FAILED – AWS::EC2::Instance – API: ec2:RunInstances Not authorized for images: [ami-7f634e4f]

CodeDeploy – create a Windows instance the “hard” way

So let’s forget the “easy” path of CloudFormation.  Try the old-fashioned way of creating a Windows instance, and see if I can make it through this:

  • Deciding among Windows server AMI’s is a real blast – over 600 of them!
  • I narrowed it down to the “Windows_Server-2016-English-Full-Base-2016.12.24”
    • Nano is only available to Windows Assurance customers
    • Enterprise is way more than I’d need to serve a web page
    • Full gives you the Windows GUI to manage the server, whereas Core only includes the PowerShell (and may not even allow RDP access)
    • I wanted to see what the Manage Server GUI looks like these days, otherwise I probably would’ve tried Core
    • Note: there were three AMI all prefixed “Windows_Server-2016-English-Full-Base”, I just chose the one with the latest date suffix (assuming it’s slightly more up-to-date with patches)
  • I used the EC2 console to get the Windows password, then installed the Microsoft Remote Desktop client for Mac to enable me to interactively log in to the instance

Next is configuring the S3 bucket:

  • There is some awfully confusing and incomplete documentation here
  • There are apparently two policies to be configured, with helpful sample policies, but it’s unclear where to go to attach them, or what steps to take to make sure this occurs
  • It’s like the author has already done this a hundred times and knows all the steps by heart, but has forgotten that as part of a tutorial, the intended audience are people like me who have little or no familiarity with the byzantine interfaces of AWS to figure out where to attach these policies [or any of the other hundred steps I’ve been through over the last few weeks]
  • I *think* I found where to attach the first policy (giving permission to the Amazon S3 Bucket) – I attached this policy template (substituting both the AWS account ID [111122223333] and bucket name [codedeploydemobucket] for the ones I’m using):
    { "Statement": [ { "Action": ["s3:PutObject"], "Effect": "Allow", "Resource": "arn:aws:s3:::codedeploydemobucket/*", "Principal": { "AWS": [ "111122223333" ] } } ] }
  • I also decided to attach the second recommended policy to the same bucket as another bucket policy:
    { "Statement": [ { "Action": ["s3:Get*", "s3:List*"], "Effect": "Allow", "Resource": "arn:aws:s3:::codedeploydemobucket/*", "Principal": { "AWS": [ "arn:aws:iam::80398EXAMPLE:role/CodeDeployDemo" ] } } ] }
  • Where did I finally attach them?  I went to the S3 console, clicked on the bucket I’m going to use (called “hacku-devops-testing”), selected the Properties button, expanded the Permissions section, and clicked the Add bucket policy button the first time.  The second time, since it would only allow me to edit the bucket policy, I tried Add more permissions – but that don’t work, so I tried editing the damned bucket policy by hand and appending the second policy as another item in the Statement dictionary – after a couple of tries, I found a combination that the AWS bucket policy editor would accept, so I’m praying this is the intended combination that will all this seductive tutorial to complete:
    {
     "Version": "2008-10-17",
     "Statement": [
     {
     "Effect": "Allow",
     "Principal": {
     "AWS": "arn:aws:iam::720781686731:root"
     },
     "Action": "s3:PutObject",
     "Resource": "arn:aws:s3:::hacku-devops-testing/*"
     },
     {
     "Action": [
     "s3:Get*",
     "s3:List*"
     ],
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::hacku-devops-testing/*",
     "Principal": {
     "AWS": [
     "arn:aws:iam::720781686731:role/CodeDeployDemo-EC2-Instance-Profile"
     ]
     }
     }
     ]
    }

CodeDeploy – actually deploying code

I followed the remaining commands (closely – gotta watch every parameter and fill in the correct details, ugh).  But thankfully this was the trivial part.  [I guess they got the name “CodeDeploy” right – it’s far more attractive than “CodeDeployOnceYouFoundTheLostArkOfTheCovenantToDecipherIAMIntricacies”.]

Result

Success!  Browsing to the public DNS of the EC2 instance showed me the Hello World page I’ve been trying to muster for the past three days!

Conclusion

This tutorial works as a demonstration of how to marshall a number of contributing parts of the AWS stack: CodeDeploy (whose “ease of deployment” benefits I can’t yet appreciate, considering how labourious and incomplete/error-prone this tutorial was), IAM (users, roles, groups, policies), S3, EC2 and AMI.

However, as a gentle introduction to a quick way to get some static HTML on an AWS endpoint, this is a terrible failure.  I attacked this over a span of three days.  Most of my challenges were in deciphering the mysteries of IAM across the various layers of AWS.

In a previous life I was a security infrastructure consultant, employed by Microsoft to help decipher and troubleshoot complex interoperable security infrastructures.  I prided myself on being able to take this down to the lowest levels and figure out *exactly* what’s going wrong.  And while I was able to find *a* working pathway through this maze, my experience here and my previous expertise tells me that AWS has a long way to go to make it easy for AWS customers to marshall all their resources for secure-by-default application deployments.  [Hell, I didn’t even try to enhance the default policies I encountered to limit the scope of what remote endpoints or roles would have access to the HTTP and SSH endpoints on my EC2 instance.  Maybe that’s a lesson for next time?]

 

Getting to know AWS with Python (the free way)

I’ve run through all the easy AWS tutorials and I was looking to roll sleeves up and take it one level deeper, so (given I’ve recently completed a Python class, and given I’m trying to stay on a budget) I hunted around and found some ideas for putting a sample Flask app up on AWS Elastic Beanstalk, that cost as little as possible to use.  Here’s how I wove them together (into an almost-functional cloud solution first time around).

Set aside Anaconda, install Python3

First step in the tutorial I found was building the Python app locally (then eventually “freezing” it and uploading to AWS EB).  [Note there’s a similar tutorial from AWS here, which is good for comparison.]  [Also note: after I mostly finished my work, I confirmed that the tutorial I’m using was written for Python 2.7, and yet I’d blundered into it with Python 3.4.  Not for the faint of heart, nor for those who like non-Degraded health status.]

Which start with using virtualenv to build up the app profile.

But for me, virtualenv threw an error after I installed it (with pip install virtualenv):

ERROR: virtualenv is not compatible with this system or executable 

Okay…so Ryan Wilcox gave us a clue that you might be using the wrong python.  Running which python told me I’m directed to the anaconda version, so I commented that PATH modification out of .bash_profile and installed python3 using homebrew (which I’d previously installed on my MacBook).

Setup the Flask environment Debug virtualenv

Surprising no one but myself, by removing anaconda and installing python3, I lost access not only to virtualenv but also to pip, so I guess all that was wrapped in the anaconda environment.  Running brew install pip reports the following:

Updating Homebrew...
Error: No available formula with the name "pip" 
Homebrew provides pip via: `brew install python`. However you will then
have two Pythons installed on your Mac, so alternatively you can install
pip via the instructions at:
  https://pip.readthedocs.io/en/stable/installing/

The only option from that article seems to be running get-pip.py, so I ran python3 get-pip.py (in case the script installed a different pip based on which version of python was running the script).  Running pip –version returned this, so I felt safe enough to proceed:

pip 9.0.1 from /usr/local/lib/python3.5/site-packages (python 3.5)

Ran pip install virtualenv, now we’re back where I started (but without the anaconda crutch that I don’t entirely understand, and didn’t entirely trust to be compatible with these AWS EB tutorials).

Running virtualenv flask-aws from within the local clone of the tutorial repo throws this:

Using base prefix '/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5'
Overwriting /Users/mike/code/flask-aws-tutorial/flask-aws/lib/python3.5/orig-prefix.txt with new content
New python executable in /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3.5
Not overwriting existing python script /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python (you must use /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3.5)
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 11, in 
    sys.exit(main())
  File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 925, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 1370, in install_python
    os.symlink(py_executable_base, full_pth)
FileExistsError: [Errno 17] File exists: 'python3.5' -> '/Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3'

Hmm, is this what virtualenv does?  OK, if that’s true why is something intended to insulate from external dependencies seemingly getting borked by external dependencies?

Upon closer examination, it appears that what the virtualenv script tried to do the first time was generate a symlink to a python3.5 interpreter, and got tripped when it found a symlink already there.  However, the second time I ran the command (hoping that it is self-healing), I got yelled at about “too many symlinks”, and discover that the targets now have a circular loop:

Mac4Mike:bin mike$ ls -la
total 24
drwxr-xr-x  5 mike  staff  170 Dec 11 12:26 .
drwxr-xr-x  6 mike  staff  204 Dec 11 12:26 ..
lrwxr-xr-x  1 mike  staff    9 Dec 11 12:26 python -> python3.5
lrwxr-xr-x  1 mike  staff    6 Dec 11 11:52 python3 -> python
lrwxr-xr-x  1 mike  staff    6 Dec 11 11:52 python3.5 -> python

If I’m reading the above stack trace correctly, they were trying to create a symlink from python3.5 to some other target.  So all it should require is deleting the python3.5 symlink, yes?  Easy.

Aside: then running virtualenv flask-aws from the correct location (but an old, not-refreshed shell) spills this:

Using base prefix '/Users/mike/anaconda3'
Overwriting /Users/mike/code/flask-aws-tutorial/flask-aws/lib/python3.5/orig-prefix.txt with new content
New python executable in /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python
Traceback (most recent call last):
  File "/Users/mike/anaconda3/bin/virtualenv", line 11, in 
    sys.exit(main())
  File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 925, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 1387, in install_python
    raise e
  File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 1379, in install_python
    stdout=subprocess.PIPE)
  File "/Users/mike/anaconda3/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/Users/mike/anaconda3/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 62] Too many levels of symbolic links

Try again from a shell where anaconda’s been removed from $PATH?  Works fine:

Installing setuptools, pip, wheel...done.

Step 2: Setting up the Flask environment

Installing the requirements (using pip install -r requirements.txt) went *mostly* smooth, but it broke down on either the distribute or itsdangerous dependency:

Collecting distribute==0.6.24 (from -r requirements.txt (line 11))
  Downloading distribute-0.6.24.tar.gz (620kB)
    100% |████████████████████████████████| 624kB 1.1MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "", line 1, in 
      File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/__init__.py", line 2, in 
        from setuptools.extension import Extension, Library
      File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/extension.py", line 2, in 
        from setuptools.dist import _get_unpatched
      File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/dist.py", line 103
        except ValueError, e:
                         ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/

Unfortunately the “/pip-build-6gettd5v/” folder was already deleted by the time I got there to look at its contents.  So I re-ran the script and noticed that all dependencies through to distribute reported as “Using cached distribute-0.6.24.tar.gz”.

Figured I’d just pip install itsdangerous by hand, and if I got into too much trouble I could always uninstall it right?  Well, that didn’t seem to help arrest this error – presumably pip caches all the requirements files first and then installs them – so I figured I’d divide the requirements.txt file into two (creating requirements2.txt and stuffing the last three files there instead) and see if that helped bypass the problem in installing the first 11 requirements.

Nope, that didn’t work, so I also moved the line “distribute==0.6.24” out of requirements.txt into requirements2.txt:

Successfully installed Flask-0.10.1 Flask-SQLAlchemy-2.0 Flask-WTF-0.10.3 Jinja2-2.7.3 MarkupSafe-0.23 PyMySQL-0.6.3 SQLAlchemy-0.9.8 WTForms-2.0.1 Werkzeug-0.9.6 argparse-1.2.1

Yup, that did it, so off to pip install requirements2.txt:

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-5y_r3gg0/distribute/

OK, so then I removed the “distribute==0.6.24” line entirely from requirements2.txt:

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-5g9i5fpl/wsgiref/

Crap, wsgiref too?  OK, pip install wsigiref==0.1.2 by hand:

Collecting wsgiref==0.1.2
  Using cached wsgiref-0.1.2.zip
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "", line 1, in 
      File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-suqskeut/wsgiref/setup.py", line 5, in 
        import ez_setup
      File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-suqskeut/wsgiref/ez_setup/__init__.py", line 170
        print "Setuptools version",version,"or greater has been installed."
                                 ^
    SyntaxError: Missing parentheses in call to 'print'

Shit, even worse – the package itself craps out.  Good news is, I know exactly what’s wrong here, because the print method (function?) requires () in Python3.  (See, I *did* learn something from that anti-Zed rant.)

Double crap – wsgiref‘s latest version IS 0.1.2.

Triple-crap – that code’s docs haven’t been touched in forever, and don’t exist in a git repo against which issues can be filed.

Damn, this *really* sucks.  Seems I’ve been trapped in the hell that is “tutorials written for Python2”.  The only way I could get around this error is to edit the source of the cached wsgiref-0.1.2.zip file – do such files get deleted from $TMPDIR when pip exits?  Or are they stuffed in ~/Library/Caches/pip in some obfuscated filename?

Ultimately it didn’t matter – I found that pip install –download ~/ wsgiref==0.1.2 worked to dump the file exactly where I wanted.

And yes, wrapping lines 170 and 171 with () after the print verb (and re-zip’ing the folder contents) allowed me to fully install wsgiref-0.1.2 using pip install –no-index –find-links=~/ wsgiref-0.1.2.zip.

OK, finally ran pip install boto==2.28.0 and that’s all the dependencies installed.

Next!

Step 3: Create the AWS RDS

While the AWS UI has changed in subtle but UX-improving ways, this part wasn’t hard to follow.  I wasn’t happy about seeing the “all traffic, all sources” security group applied here, but not knowing what my ultimate target configuration might look like, I made the classic blunder of saying to self, “I’ll lock it down later.”

Step 4: Add tables

It wasn’t entirely clear, so here’s the construction of the SQLALCHEMY_DATABASE_URI parameter in config.py:

  • = Master Username you chose on the Specify DB Details page
    e.g. “awsflasktst”
  • = Master Password you chose on the Specify DB Details page
    e.g. “awsflasktstpwd”
  • = Endpoint string from DB Instance page
    e.g. “flask_tst.cxcmcajseabc.us-west-2.rds.amazonaws.com:3306”
  • = Database Name you chose on the Configure Advanced Settings page
    e.g. “awstflasktstdb”

So in my case that line read (well, it didn’t because I’m not telling you my *actual* configuration, but this lets you know how I parsed the tutorial guidance):

SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://awsflasktst:awsflasktstpwd@flask_tst.cxcmcajseabc.us-west-2.rds.amazonaws.com:3306/awstflasktstdb'

Step 4.6 Install NewRelic

Because I’m a former Relic and I’m still enthused about APM technology, I decided to install the newrelic package via pip install newrelic.  A few steps into the Quick Start guide and it was reporting all the…tiny amounts of traffic I was giving myself.

God knows if this’ll survive the pip freeze, but it’s sure worth a shot.  [Spoiler: cloud deployment was ultimately unsuccessful, so it’s moot.]

Step 5: Elastic Beanstalk setup

Installing the CLI was easy enough:

Successfully installed awsebcli-3.8.7 blessed-1.9.5 botocore-1.4.85 cement-2.8.2 colorama-0.3.7 docker-py-1.7.2 dockerpty-0.4.1 docopt-0.6.2 docutils-0.13.1 jmespath-0.9.0 pathspec-0.5.0 python-dateutil-2.6.0 pyyaml-3.12 requests-2.9.1 semantic-version-2.5.0 six-1.10.0 tabulate-0.7.5 wcwidth-0.1.7 websocket-client-0.40.0

Setup of the user account was fairly easy, although again it’s clear that AWS has done a lot of refactoring of their UI flows.  Just read ahead a little in the tutorial and you’ll be able to fill in the blanks.  (Although again, it’s a little disturbing to see a tutorial be this cavalier about the access permissions being granted to a simple web app.)

When we got to the eb init command I ran into a snag:

ERROR: ConfigParseError :: Unable to parse config file: /Users/mike/.aws/config

I checked and I *have* a config file there, and it’s already been populated with Packer credentials info (for a separate experiment I was working on previously).

So what did I do?  I renamed the config and credentials files to set them aside for now, of course!

Like me, you might also see an EB application listed, so I chose “2” not “1”.  No big deal.

There was also an unexpected option to use AWS CodeCommit, but since I’m not even sure I’m keeping this app around, I did not take that choice for now.

(And once again, this tutorial was super-cavalier about ignoring the SSH option that *everyone* always uses, so just closed my eyes and prayed I never inherit these bad habits…)

“Time to deploy this bad boy.”  Bad boy indeed – Python 2.7, *no* security measures whatsoever.  This boy is VERY bad.

Step 6: Deployment

So there’s this apparent dependency between the EBCLI and your local git repo.  If your changes aren’t committed, then what gets deployed won’t reflect those changes.

However, the part about running git push on the repo seemed unnecessary.  If the local CLI tool checks my repo, they’re only checking the local one, not the remote one, so once we run git commit, it shouldn’t matter whether those changes were pushed upstream.

However, knowing that I monkeyed with the requirements.txt file, I thought I’d also git add that one to my commit history:

git add requirements.txt
git commit -m "added newrelic requirement"

After initiating eb create, I was presented with a question never mentioned in the tutorial (another new AWS feature!), and not documented specifically in any of the ELB documentation I reviewed:

Select a load balancer type
1) classic
2) application
(default is 1): 

While my experience in this space tells me “application” sounds like the right choice, I chose the default for sake of convenience/sanity.

Proceeded with eb deploy, and it was looking good until we ran into the problem I was expecting to see:

ERROR: Your requirements.txt is invalid. Snapshot your logs for details.
ERROR: [Instance: i-01b45c4d01c070555] Command failed on instance. Return code: 1 Output: (TRUNCATED)...)
  File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1. 
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
WARN: Environment health has transitioned from Pending to Degraded. Command failed on all instances. Initialization completed 7 seconds ago and took 3 minutes.
ERROR: Create environment operation is complete, but with errors. For more information, see troubleshooting documentation.

Lesson

Ultimately unsuccessful, but was it satisfying?  Heck yes, grappling with all the tools and troubleshooting various aspects of the procedures was a wonderful learning experience.  [And as it turned out, was a great primer to lead into a less hand-holding tutorial I found later that I’ll try next.]

Coda

Well, heck.  That was easier than I thought.  Succeeded after another run-through by forcing Python 2.7 on both the local app configuration and in a new Elastic Beanstalk app.

Occupied Neurons, November edition (late)

Docker In Production: a History of Failure

A cautionary tale to counter some of the newbie hype around the new Infrastructure Jesus that is Docker. I’ve fallen prey to the hype as well, assuming that (a)Docker is ready for prime time, (b) Docker is universally beneficial for all workloads and (c) Docker is measurably superior to the infrastructure design patterns that it intends to replace.

That said, the article is long on complaints, and doesn’t attempt to back its claims with data, third-party verification or unemotional hyperbole. I’m sure we’ll see many counter-articles claiming “it works for me”, “I never saw these kinds of problems” and “what’s this guy’s agenda?”  I’ll still pay attention to commentary like this, because it reads to me like the brain dump of a person exhausted from chasing their tail all year trying to find a tech combo that they can just put in production and not devote unwarranted levels of monitoring and maintenance to. I think their expectations aren’t unreasonable. It sure sounds like the Docker team are more ambitious or cavalier than their position and staffing levels warrant.

Wat

This is one of the most hilarious and horrifying expeditions into the dark corners of (un?)intended consequences in coding languages.  Watching this made me feel like I’m more versed in the lessons of the absurd “stupid pet tricks” with many languages, even if I’d never use 99% of these in real life.  It also made me feel like “did someone deliberately allow these in the language design, or did some nearly-insane persons just end up naturally stumbling on these while trying to make the language do things it should never have done?”

Is Agile dying a slow death?  Or is it being reborn?

This guy captures all my attitudes about “Agile according to the rules” versus “getting an organization tuned to collaborate and learn as fast as possible”.  While extra/unnecessary process makes us feel like we have guard rails to keep people from making mistakes, in my experience what it *actually* does it drive DISengagement and risk aversion in most employees, knowing that unless they have explicit permission to break the rules, their great new idea is likely to attract organizational antibodies.

Stanford’s password policy shuns one-size-fits-all security

This is better than a Bigfoot sighting! An actual organization who’ve thought about security risk vs punishing anti-usability and come up with an approach that should satisfy both campaigns! This UX-in-security bigot can finally die a happy man.

A famed hacker is grading thousands of programs – and may revolutionise software in the process

May not get to the really grotty code security issues that are biting us some days, and probably giving a few CIOs a false sense of security.  Controversial?  Yes.

A necessary next step as software grows up as an engineering discipline? Absolutely.

Let’s see many more security geeks meeting the software developer where they live, and stop expecting em to voluntarily become part-time security experts just because someone came up with another terrific Hollywood Security Theater plot.

A Rebuttal for Python 3

Why are some old-school Pythonistas so damned pissy about Python 3 – to the point of (in at least one egregiously dishonest case) writing long articles trying to dissuade others from using it? Are they still butthurt at Guido for making breaking changes that don’t allow them to run their old Python 2 code on the Python 3 runtime? Do they not like change? Are they aware that humans are imperfect and sometimes have to admit mistakes/try something different? I find it fascinating to watch these kinds of holy wars – it gives the best kinds of insights into what frailties and hot buttons really motivate people.

The best quote’s in the comments: “Wow, I haven’t seen this much bullshit in a “technical” article in a while. A Donald Trump transcript is more honest and informative than that. I seriously doubt Zed Shaw himself believes a single paragraph there; if he actually does, he should stop acting like a Python expert and admit he’s an idiot.”

How The Web Became Unreadable

It’s painful to see some designers slavishly devote their efforts more to the third hand fashion they hear about from other designers, than to the end users of the sites and services to which they deliver their designs. I love a lot of the design work that’s come out the last few years – the jumbled mess that was web design ten years ago was painful – but the practical implications of how that design is consumed in the wild must be paramount.  And it is where I am the final decision maker on shipping software.

Occupied Neurons, early May 2016

The continuing story of the intriguing ideas and happenings that I can’t shake off…

Pigsinspace222

(Have you ever seen an episode of Pigs In Space?  If not, go sample one now, and you’ll get my droll reference)

Infinite Scrolling, Pagination or “Load More” Buttons? Usability Findings in eCommerce

https://www.smashingmagazine.com/2016/03/pagination-infinite-scrolling-load-more-buttons/

Summary (and something I plan to bias towards in future designs, under similar conditions): The “Load More” design pattern is the most well-received by users and creates a minimum of friction while still enabling access to the page footer.

How Spotify’s Poor API Hygiene Broke a Bunch of Hardware and Software

http://www.programmableweb.com/news/how-spotifys-poor-api-hygiene-broke-bunch-hardware-and-software/analysis/2016/02/23

This is a pretty epic rant on the fallout for independent Spotify developers from a haphazard approach to managing the APIs offered over the years by this consumer entertainment service. Having worked on the other side of these kinds of decisions, I can well imagine how this came to be: thin staffing levels keeping from putting adequate attention on developer communications and engineering maintenance, plus distracted attention by PMs (or possibly even frequent PM turnover) such that late in the game, no one even remembers lets alone still believes in the original value prop behind the original APIs.

It doesn’t excuse the broken promises behind the APIs, and especially not the lack of communication in obvious channels when changes were made (eliminated), but I’ve been in such positions as a Product guy and found myself making decisions that feel just as compromised – trading off one disappointment for a better-mitigated disappointment elsewhere. It happens, especially when the product being extended through those APIs has a pretty low profit margin, and when the staff devoted to managing those concerns are terribly compromised (higher priorities and all).

Theory of Constraints

https://en.m.wikipedia.org/wiki/Theory_of_constraints

At the Intel-sponsored Accelerate Results gathering, a few themes/durable concepts kept coming up (and have come up in this community repeatedly over the years). One is the Theory of Constraints, which seems popular among all systems thinkers, even in big software design (at least in concept if not in execution).

I firmly believe we have a duty to consider outside perspectives on our industry, even when they appear to have no direct applicability – myopia, tools bias and fad-driven design/execution are the restraints I make deliberate effort to resist in my own practices.

Standing on the Shoulders of Giants

http://www.business-improvement.eu/toc/Goldratt_Standing_On_The_Shoulders_Of_Giants.php

Eliyahu Goldratt is a huge influence on the thought leaders at the Accelerate Results conference, and many made reference to his seminal essay that seems to have kicked off this whole revolution. Worth a skim, even if it’s only to be able to nod thoughtfully when others keep talking about this.

Everyday Internet Users Can Stand Up for Encryption — Here’s How

https://blog.mozilla.org/blog/2016/03/30/everyday-internet-users-can-stand-up-for-encryption-heres-how/
I worked with Mark Surman a long time ago back in Toronto for a non-profit Internet Service Provider. It’s more than a little amazing to me to see how our paths have diverged and yet how he’s speaking about issues today that are very near and dear to my heart.

This week in after-work design & tech events in PDX UX Happy Hour, BSides PDX, Flux

Did any of you get out to Rose City Comicon this weekend? Tell me what you thought – I’d love to hear how it went down. I was in the outskirts of hippie country, catching up with friends who are decidedly veering off the grid.

This week: hackers. You ever wanted to hear how security researchers work their magic, or what they’re interested in breaking (or fixing)? BSides PDX will definitely give you your fill – I’ve been to the past two cons and they’re a fascinating look inside the mind the of the hacker. Sometimes scary, always educational.

This Week in my kind of PDX fun
· Tues Sept 24th: Portland UX Happy Hour “Combo meal with PDX Web & Design” (Calagator)

· Tues Sept 24th: Internet of Things Meetup PDX “The $1K Hardware Contest and Oregon’s Plans for a Hardware Incubator” (Meetup)

· Wed Sept 25th: Lean Coffee (7:30am) (Calagator)

· Sept 27-28th: BSides PDX Annual Security Hackers Gathering (EventBrite)

· Fri Sept 27th: Flux Feminist Hackerspace Grand Opening (Calagator)

On the Radar
· Wed Oct 2nd: CHIFOO “Finding and Measuring the Awesome in Education” (Eventbrite)

· Sat Oct 5th: PDX Code Retreat – Fall 2013 (EventBrite)

· Tues Oct 8th: UX Book Club “The UX Team Of One” by Leah Buley (Twitter)

· Oct 7-12th: Design Week Portland – just ignore the horrible gimmicky page design, I’m sure there’ll be some talented designers featured there

· Jan 11th 2014: Portland Code Camp 2014 (EventBrite)

· March 2014: CHIFOO talk “Show, Don’t Tell: Storytelling Experience Design in Modern Comic Books

Got the "2012 security suite" infection? This should fix it

I heard from colleagues who’ve been infected with the “2012 security suite” – yet another in a long line of malware that pretends to be an antivirus program, able to shut down legitimate antimalware like McAfee, and gets itself embedded deeply in your system.

Another colleague posted the following instructions on how they got rid of the “2012 security suite” infection.  I’m posting as-is with no firsthand knowledge of how well this would work for others, but in the spirit of sharing whatever smarter people have tried in hopes it helps a few others.

“I had to fix this one for a friend recently.  I had to use 3 different tools and it took several passes to get in entirely cleaned out.  Here’s what worked in the end (I make no guarantees of course):

  1. Download, install, and update Malwarebytes and SpyBot Search & Destroy.  They are free. Do this on two computers, the one that’s infected and another one.
  2. Remove the drive from the system and connect it to the other system as a secondary drive. Boot that system into Safe Mode with Networking.  Scan with Malwarebytes and allow it to remove anything it finds.  Reboot into Safe Mode with Networking and run SpyBot and allow it to remove anything it finds.
  3. DO NOT REBOOT YET (if you reboot at this point you will be re-infected).  First delete the following:
    • %AllUsersProfile%\Application Data\ LocalAppData\kdn.exe
    • %LocalAppData%\Temp\%UserProfile%\Templates
    • All files in c:\windows\temp (Yes you will lose your browsing history and cookies, sorry)
    • Your Temporary Internet Files (on the infected drive, not the system you’re running on)
  4. Now remove the drive from the second computer and put it back in the original computer.
  5. Boot into Safe Mode with Networking
  6. Turn off System Restore.
  7. 7. Delete the following registry keys if they exist:
    • HKEY_USERS\.DEFAULT\Software\Microsoft\Internet Explorer\BrowserEmulation “TLDUpdates” = ‘1’
    • HKEY_CURRENT_USER\Software\Classes\.exe\shell\open\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “%1” %*’
    • HKEY_CURRENT_USER\Software\Classes\exefile\shell\open\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “%1” %*’
    • HKEY_CLASSES_ROOT\.exe\shell\open\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “%1” %*’
    • HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\FIREFOX.EXE\shell\open\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “C:\Program Files\Mozilla Firefox\firefox.exe”‘
    • HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\FIREFOX.EXE\shell\safemode\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “C:\Program Files\Mozilla Firefox\firefox.exe” -safe-mode’
    • HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\IEXPLORE.EXE\shell\open\command “(Default)” = ‘”%LocalAppData%\kdn.exe” -a “C:\Program Files\Internet Explorer\iexplore.exe”‘
    • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Security Center “AntiVirusOverride” = ‘1’
    • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Security Center “FirewallOverride” = ‘1’ 
  8. Run Malwarebytes and allow it to clean anything it finds.
  9. Reboot into Safe Mode with Networking
  10. Run Spybot and allow it to clean anything it finds.
  11. Assuming you are able to run your antivirus software, run a full scan of your system.
  12. Reboot normally.
  13. Scan with both tools and your anti-virus software.
  14. Re-enable system restore.

NOTE: If at any time during this process you find you can no longer start your applications, follow the instructions here: http://www.techerator.com/2010/03/virus-blocks-exe-files-from-opening-2/ You’ll find a fix that you can download and run.”

Found: Minimum permissions to edit Site Columns in SharePoint 2010

Our organization has a requirement to delegate the maintenance of a number of Site Columns (re-used throughout our business application) to colleagues who do not have Site-wide responsibilities or authority.  We didn’t want to simply grant one of the all-powerful permissions/groups (such as Site Collection Administrators, Site Collection Owner, [Site] Owners, or just granting Full Control) as that would leave them open to accusatory questions every time the site or application started behaving badly.  We’d like them to be able to act fairly autonomously, without having to worry that they could cause unintentional damage to the site or application due to excessive permissions.

Knowing that most every Microsoft product has a rich, granular set of DACLs applied at every object level, and having exercised some of the rich permissions available in SharePoint 2010, I was pretty sure there should be a way to combine some permissions together to enable Site Column edits, without giving away the farm.  However I was unable to find any documentation or research that definitively asserted they knew what those permissions were.

Finally I asked my colleague Dale Cox to pair up with me and, together with four hours and an experimental sub-site, we were able to work out a tight sub-set of permissions that (a) definitely allow Site Columns edits to propagate to inheriting Lists and (b) were ratcheted down stepwise until we were comfortable there was no place lower to go.

Permissions: Summary

There are two sets of permissions we tested for: site-level permissions needed to edit an existing Site Column (that is being used as a column in one or more Lists in the SharePoint site), and List-level permissions needed to propagate Site Column changes down to inheriting columns in a List.

The former permissions *only* exist at the site level (even though some permissions in the permissions set are labelled “site permissions” and others “list permissions”) – we’ve found there are *no* permissions that can be managed directly related to Site Columns, but only indirectly via the site-level permissions.

That we also require the latter permissions was surprising at first (once we configured the inheritance, why should we need ongoing permissions on every inheriting List?), but that’s just a design decision that Microsoft’s SharePoint team made.  It means we have to manually *track* all the Lists that inherit changes from Site Columns, and make sure our site column maintenance people have enough permissions to the Lists.

Site-level Permissions

  • Manage Lists (labelled “List Permissions”)
  • View Items (labelled “List Permissions”)
  • Add and Customize Pages (labelled “Site Permissions”)
  • Browse Directories (labelled “Site Permissions”)
  • View Pages (labelled “Site Permissions”)
  • Open (labelled “Site Permissions”)

List-level Permissions

  • Manage Lists (individual permission – which includes View Items, View Pages and Open)
  • Contribute (permission set)

There is one case we did *not* explicitly test for, but found was part of the resultant permissions we arrived at: creating a new Site Column.  While we believe this probably requires only a subset of the site-level (site and/or List) permissions, we didn’t spend the extra cycles to isolate the minimum permissions needed there.  Perhaps it’s implied in the results, and if we need to know we’ll pursue it, but it’s not something we needed to know right now.

Steps We Followed To Distill These Permissions

We had to work as a team to test each permissions combination, as I’d found early on that as soon as I removed myself from the all-powerful groups, I was unable to restore myself or change permissions later on.  I manipulated the permissions, and Dale ran the test cases.  I pre-created the following groups in our experimental site and assigned them specific site-level permissions, to make it easy to switch from one permission set to another:

Group name

Permission Levels assigned to group (custom permissions included)

Approve Approve
Design Design
Manage lists Manage Lists (Manage Lists, View Items, View Pages, Open, Manage Personal Views)
Manage web site Manage Web Site (View Items, Manage Web Site, Add and Customize Pages, Browse Directories, View Pages, Enumerate Permissions, Browse User Information, Open)
Site viewers Contribute, Read, View Only
Edit Site Columns (Add and Customize Pages, Browse Directories, View Pages, Open)

 

  1. Dale’s starting position:
    • Permissions: member of Site Viewers group
    • Result: can view site Libraries and Lists but cannot even load the Site Columns page (/_layouts/mngfield.aspx)
  2. Added Dale to “Manage Lists” group
    • Result: can load Site Settings page but still cannot access Site Columns page
  3. Removed Dale from “Manage Lists” group, added to “Manage web site” group
    • Result: Can load Site Columns page but cannot edit site columns
  4. Granted Dale the Contribute permission on one of the Lists that inherits a Column definition from the Site Column being tested.
    • Result: can still load Site Columns page but still cannot edit the site column being tested
    • Dale even tried just updating the site column definition without propagating the change to the inheriting Lists, but that still didn’t work
    • NOTE: from here onwards we started making a point of trying to separate out the ability to edit the site column definition from the ability for that site column definition to propagate to inheriting lists
  5. Removed Dale from “Manage web site” group, added to “Approve” group
    • Result: Dale cannot load Site Columns page
  6. Added “Manage Web Site” permission to a new permission group called “Approve + MWS”
    • That is, copied the “Approve” permission group, then checked the “Manage Web Site” permission (and left the dependent permissions that came with it)
    • Result: Dale can load the Site Columns page, but cannot commit changes to any Site Columns
  7. Removed Dale from “Approve + MWS”, added Dale to “Manage Hierarchy” group
    • Note: while the difference in included permissions between “Manage Web Site” and “Manage Hierarchy” is large, still the “Manage Hierarchy” permission set is a superset of “Manage Web Site”
    • Result: Dale can access the Site Columns page, but cannot commit changes to Site Columns
    • However, we noticed that Dale can create a new Site Column, and can edit that new Site Column
    • Dale attempted just a non-propagating edit with one of the existing Site Columns, and was successful.
    • Theory: current permissions on the Lists that inherit from the existing Site Columns are what’s blocking the previous Site Column edits
  8. Granted to Dale “Contribute” permission on all of the three Lists which inherit from the existing Site Columns
    • Result: Dale cannot commit a propagating edit on an existing, inherited Site Column
    • Theory: the individual permission “Manage Lists” is what’s needed on each inheriting List for the Site Column propagation to succeed
  9. Granted to Dale “Design” permission (which includes the individual “Manage Lists” permission) to one of the three inheriting Lists
    • Result: Dale could not commit a propagating edit on an existing, inherited Site Column.  Further, the change did not even inherit to the inheriting List (where the permissions needed for propagation could have allowed the change).
    • Theory: propagating edits to a Site Column are a transactional request – if the request doesn’t succeed on every inheriting List, then the edit “transaction” is rolled back, rather than partially succeeding.
  10. Changed the following permissions to all three inheriting Lists: changed from “Design” + “Contribute” to “Manage Lists” + “Contribute”
    • Result: Dale’s propagating edit finally succeeded!
  11. Decided to go back over the site-level permissions to determine whether there was an unknown interaction between the site-level and list-level permissions that we didn’t account for.
  12. Kept the List-level permissions as-is, but changed Dale’s site-level permissions – removed him from “Manage Hierarchy” and added him to “Approve + MWS” group.
    • Result: propagating edit failed, and non-propagating edit failed.
  13. Removed Dale from “Approve + MWS” group, added Dale to Design group (which has Design permissions set).
    • Result: propagating edit succeeded.
  14. Compared the permissions between the “Design” and the “Manage Hierarchy” permissions groups, and found the following eight permissions in common:
    • Add and Customize Pages
    • Browse Directories
    • View Pages
    • Browse User Information
    • Use Remote Interfaces
    • Use Client Integration Features
    • Open
    • Edit Personal User Information
  15. Analysis of these permissions, looking for the “secret sauce” that they share:
    • The “Manage Web Site” site-level individual permission isn’t the one that Dale needs (as it isn’t part of the Design permission set, and yet Dale was still able to propagate the edit when he had the Design permission)
    • Based on the descriptions of each individual permission listed above, the most likely candidate is “Add and Customize Pages”
  16. Created a new Group called “Edit Site Columns”.  Created a new permissions set called “Edit Site Columns”.  To that permissions set, we added only “Add and Customize Pages”.  Removed Dale from Design group, added him to Edit Site Columns group.
    • Note: when “Add and Customize Pages” is checked, the following permissions were automatically checked: “Browse Directories”, “View Pages”, “Open”, “View Items”.
    • Result: propagating and non-propagating edits failed.  Adding a new Site Column also failed.  Dale could access the Site Columns page.
  17. Added back all eight site-level permissions that were in common between the Design and Manage Hierarchy permissions sets.
    • Note: the “Edit Site Columns” permission set at this point doesn’t include any of the List-level permissions that I’ve so far ignored.  Continued failure means that to edit Site Columns, a user needs some of these “list permissions” assigned at the site level.  [Confused yet?]
    • Result: propagating and non-propagating edits failed.  Dale is still able to access the Site Columns page.
  18. Re-examined the “list permissions” that were included in both of the site-level permissions sets (Design and Manage Hierarchy) that were successful for Dale:
    • Manage Lists  –  Create and delete lists, add or remove columns in a list, and add or remove public views of a list. 
    • Override Check Out  –  Discard or check in a document which is checked out to another user. 
    • Add Items  –  Add items to lists and add documents to document libraries. 
    • Edit Items  –  Edit items in lists, edit documents in document libraries, and customize Web Part Pages in document libraries. 
    • Delete Items  –  Delete items from a list and documents from a document library. 
    • View Items  –  View items in lists and documents in document libraries. 
    • Open Items  –  View the source of documents with server-side file handlers. 
    • View Versions  –  View past versions of a list item or document. 
    • Delete Versions  –  Delete past versions of a list item or document. 
    • Create Alerts  –  Create alerts. 
    • View Application Pages  –  View forms, views, and application pages. Enumerate lists. 
  19. Added all of the above permissions to the “Edit Site Columns” permission set except View Items.
    • Result: propagating edits were successful.
  20. Stripped down the site permissions from the “Edit Site Columns” permission set to leave only the following: ACP, Browse Directories, View Pages, Open.
    • Result: propagating edits were successful.
  21. Stripped out a number of List-level permissions from the “Edit Site Columns” permission set to leave only the following: ML, VI.  Left the site permissions as in the last step.
    • Result: propagating edits were successful.
  22. Wanted to ensure this result is reproducible, so we created a new permissions set “Edit Site Columns – Confirmation”, added the same permissions to this new set (i.e. ML, VI, ACP, BD, VP, Open),  created new group “Edit Site Columns – Confirmation”, granted the new permission set to the new group, added Dale to the new group and removed him from the older “Edit Site Columns” group.
    • Result: all operations were successful – view Site Columns page, commit propagating and non-propagating edit of inherited Site Columns, add new Site Column.