Are you impatient? Do you want to get your code deployed immediately and skipped Code Review and Quality Assurance testing? If the answer is yes and you use ElasticBeanStalk on AWS then this little tip will save you loads of time. Let us forcefully abort a deployment to ElasticBeanStalk.
We are currently in the development phase of a new prototype portal where I am designing and building a RESTful API that is housed on EBS, and since I am the sole DevOps, Developer, Cloud Engineer / everything else, I don’t have a lot of time for waiting for eb abort to work its wonders.
If you deploy using eb deploy and notice after the fact that your service has moved from an Ok to Severe state but don’t want to wait the extra couple minutes to be able to deploy a fixed version do the following.
Have access to the instance
If you don’t have access then you’ll just have to wait for the abort command to finish through; otherwise ssh into the instance.
ssh beanstalk.instance -l ec2-user -i .ssh/SomeKey.pem
Find the deployment process
ps auwxf | grep appdeploy -B3
root 28610 0.0 0.0 158352 3100 pts/0 S 14:37 0:00 \_ su -
root 28611 0.0 0.0 115324 3464 pts/0 S 14:37 0:00 \_ -bash
root 7807 0.0 0.0 117292 2488 pts/0 R+ 19:03 0:00 \_ ps auwxf
root 7808 0.0 0.0 110460 2160 pts/0 S+ 19:03 0:00 \_ grep --color=auto appdeploy -B3
--
root 2566 0.2 0.6 599320 26372 ? Ssl 2015 122:15 python /opt/aws/bin/cfn-hup
root 7640 0.0 0.0 115184 3076 ? S 19:02 0:00 \_ /bin/bash -e /opt/elasticbeanstalk/bin/command-processor
root 7641 2.4 0.6 230300 26600 ? Sl 19:02 0:00 \_ /opt/elasticbeanstalk/lib/ruby/bin/ruby /opt/elasticbeanstalk/lib/ruby/bin/command-processor
root 7790 1.5 0.3 203748 13824 ? S 19:03 0:00 \_ python /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py
Kill it with fire!
kill -9 7790 # This is the 01flip.py script
Fair warning: There were no negative consequences outside of logging that the process had failed (See Next); but I don’t know what this process is doing. I assume since its called 01flip.py that it is taking /opt/python/current and flipping it w/ the object in S3. But alas I don’t have time to look at the source.
Events on AWS
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances.
WARN: User initiated abort was received, however the current step of the operation in progress is not cancellable. The current operation will be aborted as soon as the non-cancellable step(s) complete.
WARN: Environment health has transitioned from Info to Severe. 100.0 % of the requests are failing with HTTP 5xx. Insufficient request rate (24.0 requests/min) to determine application health. 25.0 % of the requests to the ELB are failing with HTTP 5xx. Insufficient request rate (2.0 requests/min) to determine application health (6 minutes ago). Command is executing on all instances. ELB health is failing or not available for all instances.
ERROR: [Instance: i-0f4b4ab9] Command failed on instance. Return code: null Output: httpd: stopped
httpd: started
httpd RUNNING pid 7444, uptime 0:00:03.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
WARN: User initiated abort was received. Canceling the current operation.
ERROR: Failed to deploy application.
ERROR: During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
-- Events -- (safe to Ctrl+C)
Redeploy
Now lets push your code back up.
$ eb deploy
Creating application version archive "app-48b2-160201_131124".
Uploading MyAPI/app-48b2-160201_131124.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances.
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.
Yeah; you just broke your immutable secure infrastructure, plus some compliance rules. But... got your application deployed. W00t.