long (or bizarre) file paths

Question

I know that (on Linux at least, with native file systems like Ext3) file paths can in theory be quite long, and PATH_MAX is often (but not always) as big as 4096 (see this; and this explains it could be longer or runtime defined!).

However, file paths are often constructed on the fly, and it not uncommon in C to code something like

#define MY_PATH_MAX 256
char pathbuf[MY_PATH_MAX];
snprintf (pathbuf, sizeof(pathbuf), "%s/%s", getenv("HOME"), somevar);
FILE *f = fopen(pathbuf, "r");
//// etc, avoiding error checks for simplicity

Now, I believe that having MY_PATH_MAX being not too big (in particular to avoid consuming a lot of space on the call stack frame) is practically important and sensible. So, in the above example doing #define MY_PATH_MAX PATH_MAX would be silly (no need to spend 4Kbytes in my call frame).

I'm coding since 1974 and I never met a case where a file path was larger than a screen width, e.g. 80 characters or so. All the file systems I heard about did not have very deep directory hierarchy (a file with 8 directories above it is very unusual).

So, is my intuition correct? Does real software care about such long or bizarre file paths? Likewise, I never met any file name containing a newline (and I strongly recommend against spaces in file names!).

^{IIUC, you won't be able to build a GNU software like GCC in a directory whose name contain spaces or newlines... very likely autoconf related scripts would suffer.... Also, there is simply no way to give in your /etc/passwd a $HOME containing newline or even a colon : -since passwd(5) does not support it.}

Is my practical limit of 256 bytes in a file path a good limit?

^{Per the comments, I just upgraded it to 384...}

Do you have a practical example where long file path names are relevant? I'm mostly interested in POSIX or Linux systems!

Answer 1

I am very opposed to building unnecessary restrictions into our software, unless we really are in a very resource-constrained environment (e.g. embedded systems). Whenever we think that “nobody is ever going to need that”, somebody will certainly run into these arbitrary restrictions.

A maximum path length is fundamentally at odds with the hierarchical, tree-oriented, recursive model of modern file systems. A path addresses a node in this tree, and there should be no fundamental difference in addressing between a tree and its subtree. Yet imposing a length limit on the overall path means that in extreme cases we can address a file from a subdirectory that can't be addressed from a parent directory. E.g. if our path lengths were limited to ten bytes, to address the file /a/b/c/d.txt (12 bytes + terminator), we could chdir into /a/b and then address the file as c/d.txt. Having to chdir just to open a file really sucks.

In practice, long path names can occur due to a variety of reasons:

• File systems are a database. Some programs use hashes as key in this database, e.g. Git or Firefox. while both are very tame as they only use fairly short paths such as ./.git/objects/07/5f3cb2c7deb3abb3ce6b7e6eb8b83ff34c883c (relative path, 56 bytes + terminator) or ~/.cache/mozilla/firefox/********.default/cache2/entries/1101BEDA093DDE5562CCC4FE6CF1C4538D36022E (relative path, 97 bytes + terminator), there is no reason why someone shouldn't have multiple hashes in a path.
• Many programming toolchains use long paths. For example, Java has a convention where a package name such as com.example.product.awesome.backend.dataaccesslayer.util can lead to a path ./src/com/example/product/awesometool/backend/dataaccesslayer/util/SecondaryProductInformationVisitorManagerFactory.java (facetious example, relative path, 120 bytes + terminator). I ran a query on my file system to find the longest path, and it turned up a file in a NodeJS package called /usr/local/lib/node_modules/phonegap/node_modules/cordova/node_modules/cordova-lib/node_modules/cordova-js/node_modules/browserify/node_modules/insert-module-globals/node_modules/combine-source-map/node_modules/inline-source-map/node_modules/source-map/lib/source-map/indexed-source-map-consumer.js (real example, no symlinks, 298 bytes + terminator). Though I have to admit, that appears to be a horrible project layout.
• Symlinks are fun. Especially when they are at the end of a long path and point to a directory. Of course, symlinks have real uses, e.g. for configuration management.

I'd strongly suggest to get away from a fixed-size mindset. Scale as needed, since one size doesn't fit all. If you don't want to allocate a 4KB buffer on the stack, then don't. Your example in the question is particularly bad because you can easily find out how much space you need, and simply malloc that much – or use variable length arrays, if your compiler allows. If it turns out that someone has passed you megabytes of data, you can explicitly fail with that reason, but trying to access a half-copied filename is even worse. I think Steele and Sussman have put it better than me in their 1979 “Design of LISP-Based Processors or, SCHEME: a Dielectric LISP or, Finite Memories Considered Harmful or, LAMBDA: The Ultimate Opcode” memo:

As far as the LISP programmer is concerned, new data objects are available in endless supply. They can be conveniently called forth to serve some immediate purpose and discarded when they are no longer in use. […]

The immense freedom this gives the programmer may be seen by an example taken from current experience. A sort of error message familiar to most programmers is “too many nested DO loops”, or “more than 200 declared arrays” or “symbol table overflow”. Such messages typically arise within compilers or assemblers which were written in languages requiring data tables to be pre-allocated [at compile time] to some fixed length. The author of a compiler, for example, might well guess, “No one will even use more than, say, ten nested DO loops; I'll double that for good measure, and make the nested-DO-loop-table 20 long.” Inevitably, someone eventually finds some reason to write 21 nested DO loops, and finds the compiler overflows its fixed table and issues an error message (or worse yet, doesn't issue an error message). On the other hand, had the compiler writer made the table 100 long or 1000 long, most of the time most of the memory space devoted to that table would be wasted.

Of course, they are talking about early Fortran and its aversion to dynamic memory management. The irony here is that C absolutely does support dynamic memory allocation, although it is admittedly cumbersome. If you need some help to keep track of all that memory, you can always use C++…

Answer 2

Don't make users hate you: allow whatever the underlying system is likely to allow.

I recall running into file path problems in a reasonably large business in which assets (of several types, for products which contained content subdivided components) were stored in a folder structure that went something like:

• Department name
• Team name
• Product family name
• Product name
• Release name
• Component
• Sub-component
• Asset type

That's eight levels not including an identifier for the network drive and filename. Virtually none of these would be close to eight characters. Release name might include a date; product names would often be be of the order of "Inspirational Product Title Mark 3 Client Manual (Swiss German)".

And that's optimistic: very often another level or two of hierarchy would creep in.

Answer 3

Does real software care about such long or bizarre file paths?

There is no such thing as a long or bizarre file path. There are only valid and invalid file paths, and real software -- correct software -- sees the valid ones as strings of bytes and doesn't care about the contents.

Is my practical limit of 256 bytes in a file path a good limit?

Absolutely not. Nor is your revised limit of 384.

The practical limit on any system that defines PATH_MAX is PATH_MAX whether the actual value behind it is 5, 50, 500 or 5,000. (256 is the minimum required by POSIX.)

PATH_MAX is the system's way of telling you exactly how many characters to expect in a valid path. Using buffers that are any smaller limits your program to being able to handle a subset of the valid paths on the system. That will serve to aggravate users of your program that run afoul of this limitation and makes more work for you when they ask that you adhere to the established standard.

Unless you can describe a specific, actual problem that is solved by using truncated buffers, there's no reason to be using anything other than PATH_MAX. The 1970s were over 35 years ago, and PATH_MAX bytes for a path on most stacks isn't an issue. If you're in a stack-constrained environment, odds are good you're probably not going to be opening a lot of files, and if you are, that environment's PATH_MAX will be something small enough to accommodate it.