I have spent the last 3 months in an exhaustive - and exhausting - requirements-gathering phase of a major project and have learned, above all else, that there is no one-size-fits-all solution. There is no process, no secret, that will work in every case. Requirements analysis is a genuine skill, and just when you think you've finally figured it all out, you get exposed to a totally different group of people and have to throw everything you know out the window.
Different stakeholders think at different levels of abstraction.
It is easy to say "talk at a business level, not technical", but it's not necessarily that easy to do. The system you're designing is an elephant and your stakeholders are the blind men examining it. Some people are so deeply immersed in process and routine that they don't even realize that there is a business. Others may work at the level of abstraction you want but be prone to making exaggerated or even false claims, or engage in wishful thinking.
Unfortunately, you simply have to get to know all of the individuals as individuals and understand how they think, learn how to interpret the things they say, and even decide what to ignore.
Divide and Conquer
If you don't want something done, send it to a committee.
Don't meet with committees. Keep those meetings as small as possible. YMMV, but in my experience, the ideal size is 3-4 people (including yourself) for open sessions and 2-3 people for closed sessions (i.e. when you need a specific question answered).
I try to meet with people who have similar functions in the business. There's really very little to gain and very much to lose from tossing the marketing folks in the room with the bean counters. Seek out the people who are experts on one subject and get them to talk about that subject.
A meeting without preparation is a meeting without purpose.
A couple of other answers/comments have made reference to the straw-man technique, which is an excellent one for those troublesome folks that you just can't seem to get any answers out of. But don't rely on straw-men too much, or else people will start to feel like you're railroading them. You have to gently nudge people in the right direction and let them come up with the specifics themselves, so that they feel like they own them (and in a sense, they do own them).
What you do need to have is some kind of mental model of how you think the business works, and how the system should work. You need to become a domain expert, even if you aren't an expert on the specific company in question. Do as much research as you can on your business, their competitors, existing systems on the market, and anything else that might even be remotely related.
Once at that point, I've found it most effective to work with high-level constructs, such as Use Cases, which tend to be agreeable to everybody, but it's still critical to ask specific questions. If you start off with "How do you bill your customers?", you're in for a very long meeting. Ask questions that imply a process instead of belting out the process at the get-go: What are the line items? How are they calculated? How often do they change? How many different kinds of sales or contracts are there? Where do they get printed? You get the idea.
If you miss a step, somebody will usually tell you. If nobody complains, then give yourself a pat on the back, because you've just implicitly confirmed the process.
Defer off-topic discussions.
As a requirements analyst you're also playing the role of facilitator, and unless you really enjoy spending all your time in meetings, you need to find a way to keep things on track. Ironically, this issue becomes most pernicious when you finally do get people talking. If you're not careful, it can derail the train that you spent so much time laying the tracks for.
However - and I learned this the hard way a long time ago - you can't just tell people that an issue is irrelevant. It's obviously relevant to them, otherwise they wouldn't be talking about it. Your job is to get people saying "yes" as much as possible and putting up a barrier like that just knocks you into "no" territory.
This is a delicate balance that many people are able to maintain with "action items" - basically a generic queue of discussions that you've promised to come back to sometime, normally tagged with the names of those stakeholders who thought it was really important. This isn't just for diplomacy's sake - it's also a valuable tool for helping you remember what went on during the meetings, and who to talk to if you need clarification later on.
Different analysts handle this in different ways; some like the very public whiteboard or flip-chart log, others silently tap it into their laptops and gently segue into other topics. Whatever you feel comfortable with.
You need an agenda
This is probably true for almost any kind of meeting but it's doubly true for requirements meetings. As the discussions drag on, people's minds start to wander off and they start wondering when you're going to get to the things they really care about. Having an agenda provides some structure and also helps you to determine, as mentioned above, when you need to defer a discussion that's getting off-topic.
Don't walk in there without a clear idea of exactly what it is that you want to cover and when. Without that, you have no way to evaluate your own progress, and the users will hate you for always running long (assuming they don't already hate you for other reasons).
Mock It
If you use PowerPoint or Visio as a mock-up tool, you're going to suffer from the issue of it looking too polished. It's almost an uncanny valley of user interfaces; people will feel comfortable with napkin drawings (or computer-generated drawings that look like napkin drawings, using a tool like Balsamiq or Sketchflow), because they know it's not the real thing - same reason people are able to watch cartoon characters. But the more it starts to look like a real UI, the more people will want to pick and paw at it, and the more time they'll spend arguing about details that are ultimately insignificant.
So definitely do mock ups to test your understanding of the requirements (after the initial analysis stages) - they're a great way to get very quick and detailed feedback - but keep them lo-fi and don't rush into mocking until you're pretty sure that you're seeing eye-to-eye with your users.
Keep in mind that a mock up is not a deliverable, it is a tool to aid in understanding. Just as you would not expect to be held captive to your mock when doing the UI design, you can't assume that the design is OK simply because they gave your mock-up the thumbs-up. I've seen mocks used as a crutch, or worse, an excuse to bypass the requirements entirely; make sure you're not doing that. Go back and turn that mock into a real set of requirements.
Be patient.
This is hard for a lot of programmers to believe, but for most non-trivial projects, you can't just sit down one time and hammer out a complete functional spec. I'm not just talking about patience during a single meeting; requirements analysis is iterative in the same way that code is. Group A says something and then Group B says something that totally contradicts what you heard from Group A. Then Group A explains the inconsistency and it turns out to be something that Group C forgot to mention. Repeat 500 times and you have something roughly resembling truth.
Unless you're developing some tiny CRUD app (in which case why bother with requirements at all?) then don't expect to get everything you need in one meeting, or two, or five. You're going to be listening a lot, and talking a lot, and repeating yourself a lot. Which isn't a terrible thing, mind you; it's a chance to build some rapport with the people who are inevitably going to be signing off on your deliverable.
Don't be afraid to change your technique or improvise.
Different aspects of a project may actually call for different analysis techniques. In some cases classical UML (Use Case / Activity diagram) works great. In other cases, you might start out with business KSIs, or brainstorm with a mind map, or dive straight into mockups despite my earlier warning.
The bottom line is that you need to understand the domain yourself, and do your homework before you waste anyone else's time. If you know that a particular department or component only has one use case, but it's an insanely complicated one, then skip the use case analysis and start talking about workflows or data flows. If you wouldn't use the same tool for every part of an app implementation, then why would you use the same tool for every part of the requirements?
Keep your ear to the ground.
Of all the hints and tips I've read for requirements analysis, this is probably the one that's most frequently overlooked. I honestly think I've learned more eavesdropping on and occasionally crashing water-cooler conversations than I have from scheduled meetings.
If you're accustomed to working in isolation, try to get a spot around where the action is so that you can hear the chatter. If you can't, then just make frequent rounds, to the kitchen or the bathroom or wherever. You'll find out all kinds of interesting things about how the business really operates from listening to what people brag or complain about during their coffee and smoke breaks.
Finally, read between the lines.
One of my biggest mistakes in the past was being so focused on the end result that I didn't take the time to actually hear what people were saying. Sometimes - a lot of the time - it might sound like people are blathering on about nothing or harping about some procedure that sounds utterly pointless to you, but if you really concentrate on what they're saying, you'll realize that there really is a requirement buried in there - or several.
As corny and insipid as it sounds, the Five Whys is a really useful technique here. Whenever you have that knee-jerk "that's stupid" reaction (not that you would ever say it out loud), stop yourself, and turn it into a question: Why? Why does this information get retyped four times, then printed, photocopied, scanned, printed again, pinned to a particle board, shot with a digital camera and finally e-mailed to the sales manager? There is a reason, and they may not know what it is, but it's your job to find out. Good luck with that. ;)
