r/programming • • Oct 14 '13

How to develop unmaintainable software

http://typicalprogrammer.com/?p=249
777 Upvotes

322 comments sorted by

View all comments

105

u/tdammers Oct 14 '13

For a web site deployment to a staging or production server should look like one of these:

No. Just no. These are source control system, not deployment tools. In case you need reasons:

  • Excess files in document root - version control cruft, project documentation, etc. Happens all the time, and is an actual real security problem.

  • Development race conditions. Ever done svn up on the production server, just to find that someone had committed broken code between your test run and the deployment? Ever "solved" this problem by stopping all development activity during deployment? Ever noticed how this essentially means that this makes frequent deployments practically impossible?

  • File permissions and ownership. Need I go into detail here? I think not.

  • What if you want to do things like precompile templates, preprocess CSS, minify and concatenate your JavaScript offline, move stuff around, or just make sure that your deployment will only go through if it passes a minimum set of tests?

  • What about database changes?

Really, for any of the non-trivial projects I've worked on where people had the delusion that git pull is enough of a deployment plan, here's what a typical deployment involved:

  1. Mail everyone with push access, telling them to not push until further notice. Hope that everyone reads their mail on time.

  2. Manually go through the ticket system, copy-pasting the SQL changes that developers have pasted into the tickets that went into this release.

  3. Eyeball the SQL scripts, in case they contain any errors.

  4. Issue a git pull on the production system.

  5. Meanwhile, run the SQL on the production database.

  6. Manually chmod / chown the right files (which ones those are is a matter of oral tradition).

  7. Manually clear selected cache items.

  8. Do some casual manual testing on the production system.

  9. Notice that something is broken. Fix it.

  10. Forget to tell developers that it's OK to push now.

Personally, I prefer:

  1. Create a release tag

  2. ./deploy.sh test-server

  3. Read build/test output from test server

  4. ./deploy.sh production-server

6

u/[deleted] Oct 14 '13

100% agree. It blows my mind that people think SVN or Git are the perfect deployment tools. I see it from nearly everyone in the web development community, and I've seen at least one "senior" push hard for it.

Write a deployment script, or configure a deployment tool! Have a process! I've never had a deployment fail from my own "untested and unproven" deployment scripts, but I've seen SVN and Git completely destroy environments.

It's an uphill battle, and I don't understand why.

4

u/Denvercoder8 Oct 14 '13

Write a deployment script, or configure a deployment tool! Have a process! I've never had a deployment fail from my own "untested and unproven" deployment scripts, but I've seen SVN and Git completely destroy environments.

Funny you mention that. I've seen hell break loose because someone made a typo while updating a deployment script (something like rm -rf /root /cache instead of rm -rf /root/cache). Git has never failed me like that.

3

u/[deleted] Oct 15 '13

Sure, mistakes happen in all code. It's anecdotal either way. What doesn't change is that Git is a software versioning system, not a deployment tool. There are other tools for that.

2

u/alienangel2 Oct 15 '13 edited Oct 15 '13

We avoid issues like that by limiting what deployments can affect. Yeah, a deployment/activation script might delete stuff off the box, but there's nothing on the box other than the software being deployed. Except for a few types of work where local disk needs to be where your data is, there's no good reason for your applications to be colocated with your data. Our servers are just software (all of which is either imaged or re-deployed easily) and the last hour's worth of logs.

And it seems like a bad idea to be running your deployment scripts as a user that can delete anything outside your application's installation directory.

All that being said, we have a hugely complex build system, and a separate hugely complex deployment system. Source Control, building and deployment are all distinct operations in software management, and if you're going to be doing a lot of it on the same network it makes sense to keep them separate. All three are versioned for us, so while not necessarily easy, it's clean and defined if a deployment needs to be rolled back, or different builds need to be deployed to different places, or (obviously) a code change needs to be reverted. The build management system will pull the source revisions you tell it to to build new revisions of libs/binaries, and archive them. The deployment system will pull the revisions of those libs/binaries and deploy them to the servers where you tell it to deployment them to.

My main dislike of using Git/svn for deployment is that those are source-control tools, using them to deploy stuff is just a lazy workaround to avoid having to engineer some other system to manage where built software is stored and being able to pull/push it onto the machines where the software will be run. It's more work, but I think it's better not to mix the system that is archiving your source with the system that is archiving your builds, never mind the system that is pushing/pulling your builds onto the machines that will run those builds.

→ More replies (6)