I’ve run through all the easy AWS tutorials and I was looking to roll sleeves up and take it one level deeper, so (given I’ve recently completed a Python class, and given I’m trying to stay on a budget) I hunted around and found some ideas for putting a sample Flask app up on AWS Elastic Beanstalk, that cost as little as possible to use. Here’s how I wove them together (into an almost-functional cloud solution first time around).
Set aside Anaconda, install Python3
First step in the tutorial I found was building the Python app locally (then eventually “freezing” it and uploading to AWS EB). [Note there’s a similar tutorial from AWS here, which is good for comparison.] [Also note: after I mostly finished my work, I confirmed that the tutorial I’m using was written for Python 2.7, and yet I’d blundered into it with Python 3.4. Not for the faint of heart, nor for those who like non-Degraded health status.]
Which start with using virtualenv to build up the app profile.
But for me, virtualenv threw an error after I installed it (with pip install virtualenv):
ERROR: virtualenv is not compatible with this system or executable
Okay…so Ryan Wilcox gave us a clue that you might be using the wrong python. Running which python told me I’m directed to the anaconda version, so I commented that PATH modification out of .bash_profile and installed python3 using homebrew (which I’d previously installed on my MacBook).
Setup the Flask environment Debug virtualenv
Surprising no one but myself, by removing anaconda and installing python3, I lost access not only to virtualenv but also to pip, so I guess all that was wrapped in the anaconda environment. Running brew install pip reports the following:
Updating Homebrew... Error: No available formula with the name "pip" Homebrew provides pip via: `brew install python`. However you will then have two Pythons installed on your Mac, so alternatively you can install pip via the instructions at: https://pip.readthedocs.io/en/stable/installing/
The only option from that article seems to be running get-pip.py, so I ran python3 get-pip.py (in case the script installed a different pip based on which version of python was running the script). Running pip –version returned this, so I felt safe enough to proceed:
pip 9.0.1 from /usr/local/lib/python3.5/site-packages (python 3.5)
Ran pip install virtualenv, now we’re back where I started (but without the anaconda crutch that I don’t entirely understand, and didn’t entirely trust to be compatible with these AWS EB tutorials).
Running virtualenv flask-aws from within the local clone of the tutorial repo throws this:
Using base prefix '/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5' Overwriting /Users/mike/code/flask-aws-tutorial/flask-aws/lib/python3.5/orig-prefix.txt with new content New python executable in /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3.5 Not overwriting existing python script /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python (you must use /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3.5) Traceback (most recent call last): File "/usr/local/bin/virtualenv", line 11, in sys.exit(main()) File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 713, in main symlink=options.symlink) File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 925, in create_environment site_packages=site_packages, clear=clear, symlink=symlink)) File "/usr/local/lib/python3.5/site-packages/virtualenv.py", line 1370, in install_python os.symlink(py_executable_base, full_pth) FileExistsError: [Errno 17] File exists: 'python3.5' -> '/Users/mike/code/flask-aws-tutorial/flask-aws/bin/python3'
Hmm, is this what virtualenv does? OK, if that’s true why is something intended to insulate from external dependencies seemingly getting borked by external dependencies?
Upon closer examination, it appears that what the virtualenv script tried to do the first time was generate a symlink to a python3.5 interpreter, and got tripped when it found a symlink already there. However, the second time I ran the command (hoping that it is self-healing), I got yelled at about “too many symlinks”, and discover that the targets now have a circular loop:
Mac4Mike:bin mike$ ls -la total 24 drwxr-xr-x 5 mike staff 170 Dec 11 12:26 . drwxr-xr-x 6 mike staff 204 Dec 11 12:26 .. lrwxr-xr-x 1 mike staff 9 Dec 11 12:26 python -> python3.5 lrwxr-xr-x 1 mike staff 6 Dec 11 11:52 python3 -> python lrwxr-xr-x 1 mike staff 6 Dec 11 11:52 python3.5 -> python
If I’m reading the above stack trace correctly, they were trying to create a symlink from python3.5 to some other target. So all it should require is deleting the python3.5 symlink, yes? Easy.
Aside: then running virtualenv flask-aws from the correct location (but an old, not-refreshed shell) spills this:
Using base prefix '/Users/mike/anaconda3' Overwriting /Users/mike/code/flask-aws-tutorial/flask-aws/lib/python3.5/orig-prefix.txt with new content New python executable in /Users/mike/code/flask-aws-tutorial/flask-aws/bin/python Traceback (most recent call last): File "/Users/mike/anaconda3/bin/virtualenv", line 11, in sys.exit(main()) File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 713, in main symlink=options.symlink) File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 925, in create_environment site_packages=site_packages, clear=clear, symlink=symlink)) File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 1387, in install_python raise e File "/Users/mike/anaconda3/lib/python3.5/site-packages/virtualenv.py", line 1379, in install_python stdout=subprocess.PIPE) File "/Users/mike/anaconda3/lib/python3.5/subprocess.py", line 947, in __init__ restore_signals, start_new_session) File "/Users/mike/anaconda3/lib/python3.5/subprocess.py", line 1551, in _execute_child raise child_exception_type(errno_num, err_msg) OSError: [Errno 62] Too many levels of symbolic links
Try again from a shell where anaconda’s been removed from $PATH? Works fine:
Installing setuptools, pip, wheel...done.
Step 2: Setting up the Flask environment
Installing the requirements (using pip install -r requirements.txt) went *mostly* smooth, but it broke down on either the distribute or itsdangerous dependency:
Collecting distribute==0.6.24 (from -r requirements.txt (line 11)) Downloading distribute-0.6.24.tar.gz (620kB) 100% |████████████████████████████████| 624kB 1.1MB/s Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/extension.py", line 2, in from setuptools.dist import _get_unpatched File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-6gettd5v/distribute/
Unfortunately the “/pip-build-6gettd5v/” folder was already deleted by the time I got there to look at its contents. So I re-ran the script and noticed that all dependencies through to distribute reported as “Using cached distribute-0.6.24.tar.gz”.
Figured I’d just pip install itsdangerous by hand, and if I got into too much trouble I could always uninstall it right? Well, that didn’t seem to help arrest this error – presumably pip caches all the requirements files first and then installs them – so I figured I’d divide the requirements.txt file into two (creating requirements2.txt and stuffing the last three files there instead) and see if that helped bypass the problem in installing the first 11 requirements.
Nope, that didn’t work, so I also moved the line “distribute==0.6.24” out of requirements.txt into requirements2.txt:
Successfully installed Flask-0.10.1 Flask-SQLAlchemy-2.0 Flask-WTF-0.10.3 Jinja2-2.7.3 MarkupSafe-0.23 PyMySQL-0.6.3 SQLAlchemy-0.9.8 WTForms-2.0.1 Werkzeug-0.9.6 argparse-1.2.1
Yup, that did it, so off to pip install requirements2.txt:
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-5y_r3gg0/distribute/
OK, so then I removed the “distribute==0.6.24” line entirely from requirements2.txt:
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-5g9i5fpl/wsgiref/
Crap, wsgiref too? OK, pip install wsigiref==0.1.2 by hand:
Collecting wsgiref==0.1.2 Using cached wsgiref-0.1.2.zip Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-suqskeut/wsgiref/setup.py", line 5, in import ez_setup File "/private/var/folders/dw/yycg4bz1347cx5v_crcjl5580000gn/T/pip-build-suqskeut/wsgiref/ez_setup/__init__.py", line 170 print "Setuptools version",version,"or greater has been installed." ^ SyntaxError: Missing parentheses in call to 'print'
Shit, even worse – the package itself craps out. Good news is, I know exactly what’s wrong here, because the print method (function?) requires () in Python3. (See, I *did* learn something from that anti-Zed rant.)
Double crap – wsgiref‘s latest version IS 0.1.2.
Triple-crap – that code’s docs haven’t been touched in forever, and don’t exist in a git repo against which issues can be filed.
Damn, this *really* sucks. Seems I’ve been trapped in the hell that is “tutorials written for Python2”. The only way I could get around this error is to edit the source of the cached wsgiref-0.1.2.zip file – do such files get deleted from $TMPDIR when pip exits? Or are they stuffed in ~/Library/Caches/pip in some obfuscated filename?
Ultimately it didn’t matter – I found that pip install –download ~/ wsgiref==0.1.2 worked to dump the file exactly where I wanted.
And yes, wrapping lines 170 and 171 with () after the print verb (and re-zip’ing the folder contents) allowed me to fully install wsgiref-0.1.2 using pip install –no-index –find-links=~/ wsgiref-0.1.2.zip.
OK, finally ran pip install boto==2.28.0 and that’s all the dependencies installed.
Step 3: Create the AWS RDS
While the AWS UI has changed in subtle but UX-improving ways, this part wasn’t hard to follow. I wasn’t happy about seeing the “all traffic, all sources” security group applied here, but not knowing what my ultimate target configuration might look like, I made the classic blunder of saying to self, “I’ll lock it down later.”
Step 4: Add tables
It wasn’t entirely clear, so here’s the construction of the SQLALCHEMY_DATABASE_URI parameter in config.py:
- = Master Username you chose on the Specify DB Details page
- = Master Password you chose on the Specify DB Details page
- = Endpoint string from DB Instance page
- = Database Name you chose on the Configure Advanced Settings page
So in my case that line read (well, it didn’t because I’m not telling you my *actual* configuration, but this lets you know how I parsed the tutorial guidance):
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://awsflasktst:awsflasktstpwd@flask_tst.cxcmcajseabc.us-west-2.rds.amazonaws.com:3306/awstflasktstdb'
Step 4.6 Install NewRelic
Because I’m a former Relic and I’m still enthused about APM technology, I decided to install the newrelic package via pip install newrelic. A few steps into the Quick Start guide and it was reporting all the…tiny amounts of traffic I was giving myself.
God knows if this’ll survive the pip freeze, but it’s sure worth a shot. [Spoiler: cloud deployment was ultimately unsuccessful, so it’s moot.]
Step 5: Elastic Beanstalk setup
Installing the CLI was easy enough:
Successfully installed awsebcli-3.8.7 blessed-1.9.5 botocore-1.4.85 cement-2.8.2 colorama-0.3.7 docker-py-1.7.2 dockerpty-0.4.1 docopt-0.6.2 docutils-0.13.1 jmespath-0.9.0 pathspec-0.5.0 python-dateutil-2.6.0 pyyaml-3.12 requests-2.9.1 semantic-version-2.5.0 six-1.10.0 tabulate-0.7.5 wcwidth-0.1.7 websocket-client-0.40.0
Setup of the user account was fairly easy, although again it’s clear that AWS has done a lot of refactoring of their UI flows. Just read ahead a little in the tutorial and you’ll be able to fill in the blanks. (Although again, it’s a little disturbing to see a tutorial be this cavalier about the access permissions being granted to a simple web app.)
When we got to the eb init command I ran into a snag:
ERROR: ConfigParseError :: Unable to parse config file: /Users/mike/.aws/config
I checked and I *have* a config file there, and it’s already been populated with Packer credentials info (for a separate experiment I was working on previously).
So what did I do? I renamed the config and credentials files to set them aside for now, of course!
Like me, you might also see an EB application listed, so I chose “2” not “1”. No big deal.
There was also an unexpected option to use AWS CodeCommit, but since I’m not even sure I’m keeping this app around, I did not take that choice for now.
(And once again, this tutorial was super-cavalier about ignoring the SSH option that *everyone* always uses, so just closed my eyes and prayed I never inherit these bad habits…)
“Time to deploy this bad boy.” Bad boy indeed – Python 2.7, *no* security measures whatsoever. This boy is VERY bad.
Step 6: Deployment
So there’s this apparent dependency between the EBCLI and your local git repo. If your changes aren’t committed, then what gets deployed won’t reflect those changes.
However, the part about running git push on the repo seemed unnecessary. If the local CLI tool checks my repo, they’re only checking the local one, not the remote one, so once we run git commit, it shouldn’t matter whether those changes were pushed upstream.
However, knowing that I monkeyed with the requirements.txt file, I thought I’d also git add that one to my commit history:
git add requirements.txt git commit -m "added newrelic requirement"
After initiating eb create, I was presented with a question never mentioned in the tutorial (another new AWS feature!), and not documented specifically in any of the ELB documentation I reviewed:
Select a load balancer type 1) classic 2) application (default is 1):
While my experience in this space tells me “application” sounds like the right choice, I chose the default for sake of convenience/sanity.
Proceeded with eb deploy, and it was looking good until we ran into the problem I was expecting to see:
ERROR: Your requirements.txt is invalid. Snapshot your logs for details. ERROR: [Instance: i-01b45c4d01c070555] Command failed on instance. Return code: 1 Output: (TRUNCATED)...) File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI. INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1]. WARN: Environment health has transitioned from Pending to Degraded. Command failed on all instances. Initialization completed 7 seconds ago and took 3 minutes. ERROR: Create environment operation is complete, but with errors. For more information, see troubleshooting documentation.
Ultimately unsuccessful, but was it satisfying? Heck yes, grappling with all the tools and troubleshooting various aspects of the procedures was a wonderful learning experience. [And as it turned out, was a great primer to lead into a less hand-holding tutorial I found later that I’ll try next.]
Well, heck. That was easier than I thought. Succeeded after another run-through by forcing Python 2.7 on both the local app configuration and in a new Elastic Beanstalk app.