What is the value in hiding the details through abstractions? Isn't there value in transparency?

Question

Background

I am not a big fan of abstraction. I will admit that one can benefit from adaptability, portability and re-usability of interfaces etc. There is real benefit there, and I don't wish to question that, so let's ignore it.

There is the other major "benefit" of abstraction, which is to hide implementation logic and details from users of this abstraction. The argument is that you don't need to know the details, and that one should concentrate on their own logic at this point. Makes sense in theory.

However, whenever I've been maintaining large enterprise applications, I always need to know more details. It becomes a huge hassle digging deeper and deeper into the abstraction at every turn just to find out exactly what something does; i.e. having to do "open declaration" about 12 times before finding the stored procedure used.

This 'hide the details' mentality seems to just get in the way. I'm always wishing for more transparent interfaces and less abstraction. I can read high level source code and know what it does, but I'll never know how it does it, when how it does it, is what I really need to know.

What's going on here? Has every system I've ever worked on just been badly designed (from this perspective at least)?

My philosophy

When I develop software, I feel like I try to follow a philosophy I feel is closely related to the ArchLinux philosophy:

Arch Linux retains the inherent complexities of a GNU/Linux system, while keeping them well organized and transparent. Arch Linux developers and users believe that trying to hide the complexities of a system actually results in an even more complex system, and is therefore to be avoided.

And therefore, I never try to hide complexity of my software behind abstraction layers. I try to abuse abstraction, not become a slave to it.

Question at heart

1. Is there real value in hiding the details?
2. Aren't we sacrificing transparency?
3. Isn't this transparency valuable?

Answer 1

The reason for hiding the details isn't to keep the details hidden; it's to make it possible to modify the implementation without breaking dependent code.

Imagine that you've got a list of objects, and each object has a Name property and other data. And a lot of times, you need to find an item in the list whose Name matches a certain string.

The obvious way is to loop over each item one by one, and check to see if Name matches the string. But if you find that that's taking far too much time, (as it would if you have several thousand items in the list,) you might want to replace it with a string-object dictionary lookup.

Now if all of your lookups were done by retrieving the list and looping over it, you've got a huge amount of work to do to fix this. It's even tougher if you're in a library and third-party users are using it; you can't go out and fix their code!

But if you had a FindByName method to encapsulate the process of the name lookup, you can simply change the way it's implemented and all the code that calls it will continue working, and get a lot faster for free. That's the real value of abstraction and encapsulation.

Answer 2

I just finished reading the section in Code Complete about abstraction, so that's where most of this sources.

The point of abstraction is to remove the need to ask "how is this implemented?". When you call user.get_id(), you know that an id is what you're going to get back. If you have to ask "how does this get the user id?" then you probably either don't need an id, or get_id() returns something that is unexpected and is poorly designed.

You use abstraction to allow you to design:

a house with doors and windows

not design

a box with four walls,
    with 3 holes,
        two of which fit panes of glass surrounded by wood frames,
        one that fits a large plank of wood with hinges and a metal knob,
etc.

Answer 3

Is there real value in hiding the details?

Yes. By presenting abstractions we can think and program at a higher level.

Imagine modeling physical systems without calculus or matrix algebra. It's completely impractical. Similarly, if we can only program at a scalar level, we will be unable to solve interesting problems. Even relatively simple web applications can benefit greatly from abstractions like tag libs. It is much easier to insert a tag that means "address entry fields" than to repeatedly create four text fields and a select box. And if you decide to expand overseas, you can just modify the tag definition, rather than fixing every form to handle international addresses. Effective use of abstraction is what makes some programmers ten times as effective as others.

Humans have a limited working memory. Abstraction allows us to reason about large systems.

Aren't we sacrificing transparency?

No. If abstractions are not used, then the purpose of a software component is buried in repeated details. Developers spend their days wading through code like this:

for (i = 0; i < pilgrim.wives.size(); ++i) {
  wife = pilgrim.wives[i];
  for (j = 0; j < wife.sacks.size(); ++j) {
     sack = wife.sacks[i];
     for (k = 0; j < sack.cats.size(); ++j) {
        cat = sack.cats[k];
        for (m = 0; m < cat.kits.size(); ++m) {
           ++count;
        }
     }
  }
}

and thinking "oh yes another four-level loop over the kits", instead of seeing

pilgrim.kits.each { ++count; }

Isn't this transparency valuable?

As you pointed out, there is a cost to indirection. There is no point in creating layers "just in case". Use abstraction to reduce duplication and clarify code.

Answer 4

When people say that abstractions hide implementation details, they don't actually mean "hide" in the sense to make it hard to find. What they mean is separate implementation details from public interface, to keep the interface simple, concise, and manageable. Just like a car "hides" most of its vital parts, and only offers a fairly rudimentary set of controls to operate them, a software module "hides" most of its functionality deep in its bowels and only exposes a limited number of access methods to drive it. Imagine a car where you had to manually operate all the engine's internals (and there's a whole freaking lot of them), you'd have a really hard time keeping an eye on the traffic and finding the way.

But keeping the interface simple is not merely an aesthetic thing; it can make the difference between a successful project and a Death March. Let's play devil's advocate for a minute; imagine a software project without any abstractions at all. If you need to keep a value around, you use a global variable. If you need to use functionality more than once, you copy-paste it. If you need two different versions of a certain code section, you copy-paste, wrap it in an if statement, and modify both branches. Technically speaking, it works, but a few months down the road, you'll be fighting a few really nasty problems:

• When you find and fix a bug, it is likely to also exist in other copy-pasted instances of similar code, so on top of finding and fixing the bug, you also have to go hunting for other occurrences and fix them, too.
• In order to find a bug or implement a change, a maintenance programmer must be able to understand the relevant code. The difficulty in doing this increases with the size of the relevant code section, but even more with its scope. Keeping half a dozen variables in your head while mentally stepping through some code is doable; but if you have a few hundred of them, your productivity is severely impacted (I like to compare the thought process with a program that runs out of physical RAM and has to dip into the swapfile: instead of reading through the code fluently in one go, the programmer has to jump back and forth to look things up).
• The scope of a piece of code also impacts the size of the codebase one has to dig through in order to find the bug. If you have a ten-line function with two parameters, and no globals, and you know the values of the input and the line at which it crashes, finding the bug is usually trivial and often requires nothing more than looking at the code. If it's a few hundred lines, twenty parameters, fifteen globals, and calls a few other functions of similar nature, you're in for some serious pain.
• Without proper abstraction, any change can potentially impact large parts of the codebase, as practically anything may depend on the code to be changed. A typical symptom with such codebases is that you make a small, seemingly innocent change, and a completely unrelated feature suddenly breaks. With abstraction, you can limit the amount of damage a change can do, and you make the impact more predictable. If you change the name of a private field, you only have one source file to check; if you change the name of a global variable, you need to run through the entire codebase.

In a badly-abstracted codebase, the impact typically grows exponentially with the size of the codebase, that is, adding a constant amount of code increases the maintenance effort by a constant factor. To make matters worse, adding more programmers to a project does not increase productivity linearly, but logarithmically at best (because the larger your team, the more overhead is required for communication).

Answer 5

I think you should understand how it works if needed. Once you've determined it does what you thought it would do, then you have peace of mind. I never thought the goal was to hide it for ever and ever.

Once you set an alarm on a clock that you are confident will work, you can get some sleep knowing it will go off at the corret time. Waking up an hour early just so you can watch the seconds tick away is a waste.

Answer 6

To answer your questions specifically:

Is there real value in hiding the details?

Yes. As you acknowledge in the first line of your question.

Aren't we sacrificing transparency?

Not really. A well written abstraction will make it easy to understand the details if needed.

Isn't this transparency valuable?

Yes. Abstractions should be designed and implemented to make understanding the details easy when needed/desired.

Answer 7

I would say that hiding the details is great when the stuff that's hidden works.

For instance, say we develop an interface which defines behaviors (i.e. GetListItems, SendListItem), which are two features that are initiated by the user via some button click or something.. NOW, each user can have their own "ListItemStore".. say one's on facebook, ones on myspace.. (for instance).. and say it's saved as a user property / setting somewhere in the app via user prefrences.. and say it's possible for the App developers to add additional ListItemStore's over the course of time (mybook, facespace, etc..)

now there's a lot of details in connecting to facebook and getting them items.. and there's equally as much details when connecting to to myspace.. and so on...

now, after the initial "store access" code is written, it may not need to be modified (well in facebooks case we probably need a fulltime developer to keep up with the changes, zing..) ,

so when you use the code it's something like:

    new ItemManager(user) //passes in user, allowing class to get all user properties
    ItemManager.GetListItems()

and now you've got the User's data from wherever they've stored it, and since all i'm worried about is getting the list of items and doing something with it, and since it only took 2 lines which will work no matter how many more stores are added i can get back to answering / posting questions on stack... lol..

So, all the plumbing to make that happen is "hidden" and who really cares how it does it as long as i get the correct list of items.. if you have unit tests, then you can even rest easier because the results should've been quantified already..

Answer 8

What you refer to as "Hiding", many see as a separation of concerns (e.g. Implementation vs. Interface).

In my opinion, one major benefit of abstraction is to reduce the clutter of unneeded details from the developer's limited brainspace.

If implementation code were obfuscated, I could see that as hindering transparency, but abstraction as I see it, is just good organization.

Answer 9

First off, anything beyond a single machine code instruction is essentially abstraction--a while...do loop is an consistent symbolic way of representing the comparisons and address calls required to repeat a set of instructions until a condition is met. Likewise the int type is an abstraction for X number of bits (depending on your system). Programming is all about abstraction.

You'd probably agree that those primitive abstractions are mighty useful. Well, so is being able to build your own. OOAD and OOP are all about.

Suppose you've got a requirement where the users want to be able to export the data from a screen in a variety of formats: delimited text, excel, and pdf. Isn't it handy that you can create an interface called "Exporter" with a method export(data), based on which you can build a DelimitedTextExporter, an ExcelExporter, and a PDFExporter, each of which knows how to create it's particular output? All the calling program needs to know is that it can call the export(data) method, and whichever implementation is used will do its thing. Moreover, if the delimited text rules change, you can change the DelimitedTextExporter without having to mess with the ExcelExporter, possibly breaking it.

Pretty much all of the well known design patterns used in OO programming depend on abstraction. I'd recommend reading Freeman and Freeman's Head First Design Patterns to get a better feeling for why abstraction is a good thing

Answer 10

I think I understand your feeling about this, and I think I have a similar opinion.

I have worked with Java developers who turn 50 code line class into 3 classes and 3 interfaces because it's easy to understand. And I couldn't stand it.

The thing was awfully hard to understand, almost impossible to debug and never ever needed to "switch the implementation".

On the other hand, I've also seen code where multiple objects share similar behavior and are used in one place, and could really use common sorting/processing loops if the methods would have been exposed through common interface.

So, IMHO, core objects that are likely to be used in similar scenarios usually benefit from common behavior which should be accessible through interface. But that's pretty much it, abstracting simple things because it's right, or makes it possible to switch implementations is just a way to make code messy.

Then again, I prefer longer smarter classes over explosive amount of small classes with all the lifetime management issues, and hard to see relationships, and spaghetti call graphs. So some people will disagree with me.

Answer 11

The guiding purpose of hiding and abstraction should be decoupling the user from the implementation so they can be changed independently If the consumer is coupled with the implementation details, due to fiddling with their internals both are cast in stone and it becomes harder to introduce new features or better algorithms in the future.

When writing a module, hidden parts of the implementation gives you the piece of mind to be able to change them without risking breaking other code you can't think of.

Another advantage to providing opaque interfaces is that they significantly reduce the surface area between subsystems. By reducing the amount of ways they can interact they can become more predictable, easier to test and have fewer bug. Interactions between modules also increase quadratically with the number of modules so there is a value in trying to control this growth of complexity.

---

That said, it is of course possible to hide too much and nest interfaces too deep. It is the job of the programmer, as an intelligent human, to design the system so that it is maximally useful while also minimizing complexity and maximizing maintainability.

Answer 12

In so many cases you simply don't need to know how things are implemented. I can almost guarantee that you'll write code like this people.Where(p => p.Surname == "Smith") so many times a day yet you will hardly ever think "how does this Where() method actually work?" You just don't care - you know this method is there and that it gets you the results you want. Why would you care how it works?

This is exactly the same for any in-house software; just because it's not written by Oracle, Microsoft, etc doesn't mean that you should have to go hunting into how it's implemented. You can reasonably expect that a method called GetAllPeopleWithSurname(string name) to return you a list of people who have that surname. It might iterate over a list, it might use a dictionary, it might do something completely crazy but you just shouldn't care.

There is an exception to this rule of course (it wouldn't be a rule without one would it!) and that is if there is a bug in the method. So in the above example, if you have a list with 3 people in it and you know that one of them has the surname Smith and they aren't returned in the list then you care about the implementation of that method because it is clearly broken.

Abstraction, when done correctly, is wonderful because it allows you to filter out all of the stuff that isn't useful when you have to read it at a later date. Don't forget that much more time is spent reading code than is spent writing it, therefore the emphasis must be on making that task as easy as possible. You might also be thinking that abstraction means a hierarchy of objects as long as your arm but it can be as simple as refactoring a 100 line method into 10 methods, each 10 lines long. So what was once 10 steps all bundled together is now 10 separate steps allowing you to go straight to the place where this pesky bug is hiding.

Answer 13

Abstractions result in information hiding. This should end up in lower coupling. This should lead to fewer risks in change. This should lead to happy programmers, not getting nervous when touching code.

Those ideas are expressed through three essential laws in software architecture:

Simon's Law: "Hierarchies reduce complexity." (Hierarchies introduce abstraction)

Parna's Law: "Only what is hidden can be changed without risk."

Constantin's Law: Robust programs need low coupling and high cohesion

Answer 14

I am in the enterprise application business too, and this question grabbed my attention because I have the same question myself. I had some insight regarding the abstraction issue over the course of my career thus far, but my insight is by no means the universal answer. I am continuing to learn/listen to new ideas and thoughts, so what I believe now may change.

When I was a maintaining a large and complex healthcare application, just liked you, I hated all the abstractions there. Figuring where the all the code goes was pain in the neck. Jumping around different classes made me dizzy. So I said to myself "abstraction sucks, I will minimize abstraction when I design stuff".

Then, it came to the time I had to design an application (relative small web service component) from ground up. Remembering all the pain, I had a pretty flat design of the component. Problem was, when requirements changed I had to made change to many different places (requirements were quite fluid, and I couldn't do anything about that). It was so bad, I basically throw away my initial design and re-design with abstraction, and things got better - I didn't have to make changes to many places when requirements changed anymore.

I delivered the application, sat around for few weeks then was told to start maintaining the application. I had been a while, I didn't remember everything so I struggled a little bit to understand my own code, and the abstraction was not helping.

I was put on many other different projects afterwards and had the chance to play around with the abstraction levels a little bit more. What I actually find is, and this is just my personal opinion, abstraction helps a lot in development but has negative impact when you didn't write the code and are trying to understand everything to the deepest level of an application; you will spend more time jumping around different classes, and trying to make the connections.

My feeling is that abstraction is so valuable during development time that the trouble we go through as maintainer when trying to understand the code is worth while. Software exist to solve business problems, business problems evolve over time; hence, software has to evolve over time. Without abstraction, evolving the software is very hard. One can design abstraction in a way that the maintainer can navigate around the code base easily once they see the pattern of the code structure, so that only the initial learning curve is frustrating.

Answer 15

As others have said, "hiding details" behind an abstraction enables them to be changed without affecting users. This idea comes from Parnas' On the Criteria to Be Used in Decomposing Systems Into Modules (1972), and is related to the idea of abstract data types (ADT) and object oriented programming.

At around the same time, Codd's A Relational Model of Data for Large Shared Data Banks (1970) was motivated (see the abstract and intro) by wanting to change the internal storage representation of databases, without affecting users of the database. He had seen programmers regularly taking days, modifying pages of code, to cope with minor storage changes.

That said, an abstraction isn't very useful if you have to see what's inside it to be able to use it. It can be very difficult to design it well. An example of a good abstraction is addition - when was the last time you had to think about what happens inside? (but at times you do, e.g. for overflow).

The essential problem (IMHO) is that to design modules well (in Parnas' sense), you need to predict what will change and what won't. Predicting the future is difficult - but if you have a lot of experience with something, and understand it clearly, then you can do a pretty good job of predicting. And therefore, you can design a module (abstraction) that works well.

However, it does seem the fate of all abstractions - even the very best - that eventually there will be unforeseen (and arguably, unforeseeable) changes that require breaking the abstraction. To address this, some abstractions have an escape, where you can get access to a deeper level if you really need it.

This all seems very negative. But I think the truth is we are surrounded by abstractions that work so well that we don't notice them, or realize what they are hiding. We only notice the poor abstractions, so we have a jaundiced view of them.

Answer 16

Abstractions are mostly for the benefit of their consumers (e.g., application programmers). The system (designer) programmers have more work to do to make them beautiful and useful, which is why good design is not usually done by beginners.

Maybe you don't like abstractions because they always add complexity? Maybe the systems you've worked on had abstractionitis (over use of abstractions)? They're not a panacea.

The extra work and complexity of a useful abstraction should pay off, but it's hard to know for sure. If you think of an abstraction as a pivot point, then the software design can flex on either side: implementations of the abstraction can be modified without breaking client code, and/or new clients can easily re-use the abstraction to make new things.

You could almost measure the return-on-investment of abstractions by showing that they've been "flexed" over time in one or both of these directions: relatively painless implementation changes and additional new clients.

For example: Using the abstraction of the Socket class in Java, I'm sure my application code from Java 1.2 still works fine under Java 7 (although their may be some performance changes). Since Java 1.2, there have definitely been a lot of new clients that used this abstraction, too.

As for unhappiness with abstractions, if I talked to the developers who maintained the code behind the Socket class, then maybe their lives are not as peachy and rosy as the clients who used Socket to write fun applications. Working on the implementation of an abstraction surely is more work than using it. But that doesn't make it bad.

As for transparency, in a top-down design strategy, total transparency makes for bad design. Smart programmers tend to make the most out of the information they have at their disposal, and the system then becomes tightly coupled. The smallest change of detail (e.g., changing the order of bytes in a data structure) in a module might break some code somewhere else, because a smart programmer used that information to do something useful. David Parnas pointed it out this problem in articles as old as 1971 where he proposed hiding information in designs.

Your reference to ArchLinux makes sense to me if you consider the "inside" of the operating system as being the complex implementation of the abstraction that are the OS to the applications that run on it. Keep it simple in the guts of the abstraction.

Answer 17

I'll answer your question with a question; when you drove to work this morning (I'll assume you did in fact do so), did you care exactly how the engine opened valves to let in fuel-air mixtures, and then ignited them? No. You don't care how your car's engine works when you're driving down the road. You care that it does work.

Suppose, one day, your car fails to work. Doesn't start, throws a rod, breaks a belt, inexplicably plows into that concrete barrier through no fault of your own while you were busy texting. Now, you need a new car (at least temporarily). Do you care exactly how this new car works? No. What you care about is first that it does work, and second that you can use the same knowledge and skills that you used to drive your old car to drive the new one. Ideally, it should appear to you that there has been no change in the car you are driving. Realistically, the way this new car works should give you as few "surprises" as possible.

These basic tenets are the core principle behind encapsulation and abstraction. Knowledge of how an object does what it does should not be a requisite to using it to do what it does. Even in computer programming, the details of electrical paths within the CPU running your program are abstracted behind at least half a dozen layers of I/O instructions, drivers, OS software and runtime. Many very successful software engineers write perfectly good code without once worrying about the exact hardware architecture, or even OS build, that will run it. Including me.

Encapsulation/information hiding allows the "don't care how it does, just care that it does" mentality. Your object should expose what is useful to the consumer, in a way that the consumer can easily consume. Now, back in the real world, this doesn't mean that a car shouldn't give the user any information about the inner workings, or that the car should only allow the user the most basic of functionality like the ignition, steering wheel, and pedals. All cars have speedometers and fuel gauges, tachometers, idiot lights, and other feedback. Virtually all cars also have switches for various independent subsystems, like headlights, turn signals, the radio, seat adjustment, etc. Some cars allow for some pretty esoteric user input, like the sensitivity of the limited-slip center differential. In all cases, if you know enough, you can open it up and change things to make it work a slightly different way. But, in most cases, maybe, just maybe, the user shouldn't be able to directly and independently control the fuel pumps from inside the cabin? Maybe, just maybe, the user shouldn't be able to activate their brake lights without actually depressing the brake pedal?

Abstraction allows the "this isn't the same as that, but because they are both X I can use them as I would any X" mentality. If your object inherits or implements an abstraction, your consumers should expect your implementation to produce the same or similar result as other known implementations of the abstraction. A Toyota Camry and a Ford Fusion are both "cars". As such, they have a common set of expected functionality, such as a steering wheel. Turn it counterclockwise, car goes left. Turn it clockwise, car goes right. You can get in any car in the United States and expect the car to have a steering wheel and at least two pedals, the one on the right being the "car goes" pedal and the one in the center being the "car stops" pedal.

A corollary of abstraction is the "theory of least astonishment". If you got behind the wheel of a new car for a test drive, rotated the steering wheel clockwise and the car turned left, you'd be pretty astonished to say the least. You'd accuse the dealer of peddling a POS, and would be unlikely to listen to any of his reasons why the new behavior is "better" than what you're used to, or how well this behavior is "documented" or how "transparent" the control system is. Despite this new car and all the other ones you've driven still being "cars", when driving this car you have to change some fundamental concepts of how a car is supposed to be driven in order to drive the new car successfully. That's typically a bad thing, and it only happens when there is an intuitive advantage to the new paradigm. Perhaps the addition of seat belts is a good example; 50 years ago you just got in and went, but now you have to buckle up, the intuitive advantage being that you don't go through the windshield or into the passenger seat if you get in an accident. Even then, drivers resisted; many car owners cut the seat belts out of the car until laws were passed mandating their use.