A Proper Environment for Web Development? 66
umdenken wonders: "I'd like to know how others on Slashdot do their server-side web programming. We have dozens of Perl CGI scripts, and are currently doing development by editing these production scripts in place on the web server (!). Our sysadmins have finally installed an SVN client on the server (Solaris), and have offered to create a new virtual host that we can use as the development server. What are some of the practices you use for organizing this kind of set up?"
Re: (Score:3, Informative)
Re: (Score:1, Troll)
Tell your boss he should have hired someone experienced, rather than some "wet behind the ears" kid? :-)
What is training? (Score:1, Offtopic)
How does "some 'wet behind the ears' kid" become "someone experienced"?
Re: (Score:2)
Re: (Score:2)
Option 2 - work by himself for a few years, making a few thousand (highly educational) mistakes along the way.
Re: (Score:1)
Re:Er, uh (Score:4, Insightful)
His COMPANY was doing dev directly on the server, he has implemented SVN and a Test environment, and is wondering what the Best Practices might be. He knows there's a better way, and rather brilliantly knows he doesn't have all the answers.
For the record just a dev and prod environment isn't enough, ideally you would have multiple dev environments (individual playgrounds plus common test areas, two QA environments (New releases and current release for bugfix testing), and possibly even a User Acceptance Testing area. There should be no code updates a release is migrated through the environments, all environmental variables get read from the environment...
Of course, every dev environment will be different, with different needs, release cycles, etc...
Re: (Score:1)
Re: (Score:1)
Exactly!
Re: (Score:2)
For the record just a dev and prod environment isn't enough
Actually, it could be. It sounds like this is a pretty small shop, so it's not unreasonable that developers develop something in dev, then have their account managers (or other responsible-as-in-accountable persons) review the changes, then roll them into production. Lots of web projects are easily broken down into discrete non-interdependent pieces that can be developed and rolled out separately. This is certainly true for the Perl and PHP shops I've worked in, much less so for Java (since you often ha
Easy: (Score:2)
- Emacs of vi (depending on which side of the flame war you fight for)
- a W3 validation tool
Virtualisation (Score:5, Informative)
Subversion is definitely a stride forward - well worth using and getting used to, it's good that you have a client there. You should be able to fix your config scripts such they they recognise the environment (prod, dev, uat) and can be deployed directly from a tag in svn. A tag of course, not the trunk. Given the constraints it sounds like you're up against, I would definitely be looking to virtualise at least three environments - one dev, one system test, one UAT. You may have multiple dev virtual machines depending on your needs.
Cheers,
Ian
Re:Virtualisation (Score:4, Informative)
You are going to have to restart your web server in development more often than you would like to in production. You don't want to bounce production every time you change a module, right?
Re: (Score:2)
Agreed, and I should have clarified - by virtualisation I mean full machines, such as VMWare instances. Not Apache virtual hosts. Ideally I'd get separate hardware for UAT and dev/system test but it sounds like the constraints the submitter is under won't allow for that. That's why I'm suggesting falling back on virtual machines as a good second base.
Cheers,
Ian
Re: (Score:1)
Re: (Score:3, Informative)
Cheers,
Ian
Re: (Score:2)
Separate your environments (Score:5, Interesting)
Developers do their development in one environment (if possible, each developer should be isolated), and when their code is written it goes to a testing server where it can be hammered by your QA/Testing team.
When it passes QA, it goes into Pre-Production, which is what your Production environment should look like when you push your changes live. Any kind of external integration can test against this environment, since it is as close to your production environment as possible.
Then, you have your production environment where everything is live.
It is VITAL that each environment is set up the exact same way, or as close as possible, to every other environment. Each one should have its own separate hardware running identical software versions - unless you are testing software upgrades, in which case you FREEZE THE CODE, update Development, then QA, the Pre-Production, then Production (testing everything, everywhere), and THEN resume writing your code again. It is incredibly frustrating for a developer when code works on servers A, B, and D, but breaks on C and E due to non-matching hardware and software.
It is also important that your developers and qa team have access to fresh, live data if at all possible. It is easier to develop when you have real data to develop with; Plan on updating your QA and development data once a week.
If you have a set up like this, then development, testing, and deployment will be very smooth. It can be a bit of a pain to set up at first, especially if you are not used to the idea, but once you have something like this I promise that you will never go back.
Other people may recommend different set ups, but the basic idea is the same. Keep the developers separate from the testers, keep everyone separate from production. The pre-production just makes deployment easier - push code that passes QA to pre-production, then when the time comes, just rsync the files over.
Re:Separate your environments (Score:5, Insightful)
I like for the pre-production and production environments to be as close as possible.
But...
I like the dev and testing environments to be different from each other... and from the production environment.
I've found that doing this helps me shake out some "dependencies" that I may not have thought about.
Taking care of those "dependencies" helps me write code that is easier to move to another environment if the customer wants to upgrade their systems.
YMMV
Re: (Score:3, Insightful)
I like the dev and testing environments to be different from each other... and from the production environment.
Simple solution that mostly works: run a lot of the dev stuff on your desktop. Depending on your environment, that can isolate developers from each other and free up a test server for 'more stable' code.
Re: (Score:2)
I've found that doing this helps me shake out some "dependencies" that I may not have thought about.
That's okay, but the problem is that those "dependendies" aren't being revealed until the product breaks in production. And by then the damage has already been done.
I can see merit to having a different configuration for the QA/test environment than for the development environment, but you need to b
Re: (Score:2)
Actually, they break in pre-production. That is why it is supposed to be exactly like the production system...
Re: (Score:1)
Re: (Score:1)
First off, our developers are either Java (resin engine) or
Pretty much the only perl that gets written is by me and the other sysadmin. We do still have one ancient PHP app, I'd LOVE to nuke it, but development resources to rewrite it
Java developers run Linux kickstarted and set up to our environment (build system, installs required packages, sysadmins control the yum repository) they run dev on their own boxes, with their own java engines and apache.
Test mirrors pr
Re: (Score:2)
In some environments this is a serious security risk, since production data can contain SSN's, medical or other "private" data, etc. Use at your own risk.
Re: (Score:2)
Plan on updating your QA and development data once a week.
It's better if you simply give your developers the ability to update their development databases at will (perhaps by restoring from a weekly snapshot). I know several times I've set up some complex (and sometimes 'illegal', constraint-wise) data relationships to ferret out a particularly evil bug, and it can take time to recreate those. Plus, in a dynamic development environment, updating the database can require updating your code base as well (tables get added/merged/split/converted to views), which ca
Re: (Score:1)
Re: (Score:1)
My perspective... (Score:4, Informative)
Most web development environments I've been exposed to have a development, UAT (User Acceptance Testing) and production environment. Alternatively your development environment can be local and you can stick a "system testing" environment in between dev and UAT. Your UAT environment should mirror production, and before you apply any changes should have whatever code is currently in production.
You do your development, and if it's being done locally you integrate your changes with everyone else's in system test and do your automated testing and so forth. Once the developers/testers are happy a release is packaged up and deployed to UAT. You should probably run your automated tests again here for a sanity check.
The end users (business or whoever) do their testing in UAT, and if they're happy (and this is important) you take the same package you applied to UAT and apply it in production.
In some environments the developers aren't the same people with access or rights to apply changes in produciton, so you've got different groups performing different roles, but you get the idea.
Disclaimer: this is just my experiences of corporate web development, your mileage may vary, but I believe this sort of setup is pretty common (with differences here and there).
Re: (Score:2)
Unless you meant "instead of mod_perl"... I call troll.
Hell, I call troll anyway...
Let the language wars begin!
Re: (Score:2)
Oh puh-lease. Perl is used for important things all the time. I mean, Slashdot is written in Perl!
Okay, seriously: It's not clear whether you're referring to Perl or CGI as inappropriate for serious work. I agree that CGI deserves that status, but not Perl.
Re: (Score:2)
I was thinking as compared to say an MVC Java application where they're editing their business objects directly on the server.
Buggie production server... (Score:2, Interesting)
Process with Tomcat (Score:2, Insightful)
Instead I build up a
Re: (Score:2, Informative)
Re: (Score:2)
That's a feature/flaw of your servlet container configuration, not of JSP itself. Websphere, for example, can allow JSP templates to be edited and recompiled on-the-fly.
Re: (Score:2)
But it makes no sense if hey are deployed via a
JSP as part of J2EE defines that it is/must be possible to edit JSP files and let them automatically recompiled.
angel'o'sphere
Minimize complexity (Score:5, Insightful)
One setup that works (Score:5, Informative)
A. Development Box
B. Test Server
C. Production Server
A. Development Box
Every developer has Apache (or IIS respectively) and PHP/SQL on his box. People without experience can just install one of the premade packages that exist (like XAMPP or whatever its name was). This setup is isolated from the outside and responds only to 127.0.0.1 and the virtual domains. Each virtual host in Apache is a separate project.
Server Side Developers work in Eclipse PHP IDE with SubClipse, designers/client side work in Dreamweaver/Photoshop with SVN4DW & TortoiseSVN.
B. Test Server
This is used for few purposes: devs can checkout a revision and run it there on a "real" server to test, QA (well we have no dedicated QA.. it's a small team) can test on this server too.
If everything is ok it goes to...:
C. Production Server
That's it, it works really well though, everyone has his own server that can run files right of his PC, and this helps a lot in quick development. Showing to clients is as easy as checking out a revision on the test server.
Re: (Score:2)
Re: (Score:2)
Lets say you have 1 machine in production (not likely but makes it easy). Machine A is serving all the traffic. Machine B is out of the "loop" via load balancer. You deploy your staging code to machine B and run some smoke tests. Once all is go with Machine B, the load balancer flips or slowly migrates traffic from A to B (can do all or stage it by percentage of u
Seperate environments, deploy from source control (Score:2)
Each developer should get their own virtualhost on the dev server that can mimic production (apache virtualhosts if possible, or use vmware), that they can upload and self-test to. Use wildcard dns for hostnames like username.dev.company.com.
Depending on how many paralell things are tested, you might want more testing systems. Testing boxes should be mirrors of production software-wise (or at least as close as you feel comfortable).
Staging/Prepr
PXE and Kickstart (Score:3, Interesting)
1. Use PXE as a way to get an install running as quickly as possible. Do a minimal install to get the machine up and on the network.
2. Use Yum, Apt or Yast according to your distro to install everything required to support your development and application stack.
3. Your code should be done as a package (RPM/Deb). Yes, even for web stuff. If you customize your Apache install, that should be a package, too.
4. Use Subversion. CVS makes me sad, because it isn't Subversion.
5. Have a build environment that pulls your code out of subversion and builds it for QA. A package that is built here is what will be pushed live.
The bottom line is make it as easy as possible to reproduce your environment.
And back everything up over the network to a server in another state. Prefereablle, two different servers, at least one on another coast.
Local PHP (Score:1)
how we do it (Score:4, Informative)
* My code always has a standard layout (bin, conf, src, lib, and so on). No exceptions, because you never know when that little script will become a big app (this happens to me at least once a year).
* Use good coding practices: unit tests, continuous integration, whatever
* The code is checked into CVS/Subversion/Darcs, whatever. Use branches and so forth intelligently (dev on the trunk, release branches which are bugfix only, whatever). Make it "obvious" where the latest stable code always lives, so that someone besides yourself can deploy it.
* I have a script which will deploy the app to any server with rsync (excluding CVS, config, test, and dev files). There is also a flag that will "pull" the files from the server, in case of an emergency fix that was done directly on the server.
* There's a "config" directory with all system-dependent configs. No passwords or other stuff is hard-wired into the app.
* As others have described, you have your various "dev" machines, a "staging" server (identical to production but non-critical), and a "production" server. NEVER WORK ON THE PRODUCTION SERVER EXCEPT IN EMERGENCIES. Also, resist the temptation to install extra stuff on the staging server like some do (MRTG or Nagios or whatever). The staging server must be identical to the production in all respects.
If you do this properly, all you have to do to work is the following:
1) go to your laptop or other dev environment and check out the code.
2) review the REQUIREMENTS file for any packages you might need to install
3) adjust the app config files appropriately
3) code, test, checkin, code, test, checkin,
Then when you're ready to update the code on the server, first sync to the staging server and test as needed (user tests, whatever). Virtual hosts can work this way too (I've done it this way, for instance "foo.com" is the app and "dev.foo.com" is the staging version, on the same machine).
Once everything is working, push to the production servers (my script will also restart anything that's needed
I also agree with the poster above who said it's good to have different OS or environments on your dev machines, to catch any hidden assumptions. I dev on Linux and OSX and push to FreeBSD usually. Use conditional code as appropriate, and sparingly.
Ideally, you can take a blank server, install the OS, install the REQUIREMENTs, push the app, config the app, run a setup script, and go. No undocumented requirements. No weird "procedures". Just "push button install".
You should also get into the habit of making apps "learn" as much about their environment as possible. For instance, in my Ruby, Perl, PHP code, I use the __FILE__ variable or equivalent to determine the install dir, that way I don't have to configure it. A common library sets up all the necessary paths based on that.
Write your code to be flexible and backwards-compatible. For instance, if you need to move some data files around or change database fields, write code that detects the old version and does the update at startup time. A little extra work, but oh-so-automatic.
Once you get this working, you'll never want to work any other way. Being able to check out and deploy your code ANYWHERE in just a few steps is a very powerful feeling. Heck, just being able to check out in a different directory on your dev machine is useful. Having separate release branches are awesome when the client reports a bug but you've already started the big changes for 2.0
Where I work (Score:2)
The process has evolved slowly over time to cope with different numbers of clients / developers / environments. We have somewhere around 15-20 client production environments to support at different versions.
Developers are expected to document how to test their change before they commit.
Every week we build from trunk and deploy to an Integration Test environment.
Developers are then expected to re-test their change to ensure it's all in there and playing nice with any other changes.
The build is then deplo
Ghetto (Score:2)
For Apache and perl CGI, why can't every developer have his own Apache instance on his own development workstation? Use Subversion, and away you go.
You have a simple architecture. I don't see why you need to be more complex than this.
How does this change with Database sites (Score:1)
Solaris 10 zones (Score:2)
Yeah. That exclamation point does not begin to describe what a Bad Idea this is.
Here's what you should do first, this is similar to what we are doing with one of our setups (note, I am thinking cheaply here - this won't be the "best practice" but it will be good enough): Get two more servers. Put Solaris 10 on both of them.
Call the first one your development/integration machine and create a bunch of Solaris 10 zones on it, one for each de
Emacs (Score:2)
How do you configure SVN for automatic checkout? (Score:1)
I work in a small dev team that produces a specialized server that exposes some of its functionality via a web-UI. The dev-team is fantastic, and does everything right (and still uses CVS!)
Outside of my team we have some people in a pre-sales role, and part of their job is putting together quick customer-specific demos online for potential customers. It drives me nuts that if I have to go in and fix an issue, then my changes will be overwritten if the presales team decide to tweak t
Re:How do you configure SVN for automatic checkout (Score:2)
The Proper Environment for Web Design (Score:1)