134

I want to expose a resource on the web. I want to protect this resource: to make sure it is only accessible to certain individuals.

I could set up some kind of password-based authentication. For example, I could only allow access to the resource through a web server that checks incoming requests for correct credentials (perhaps against some backing database of users) before serving up the file.

Alternately I could just use a private URL. That is, I could simply host the resource at some impossible-to-guess path, for example https://example.com/23idcojj20ijf..., which restricts access to those who know the exact string.

From the perspective of an evildoer who wants to access this resource, are these approaches equivalent? If not, what makes them different? And as far as maintainability, are there pros and cons to either approach that I should be aware of before implementing one or the other?

D.W.
  • 447
  • 8
  • 11
GladstoneKeep
  • 2,629
  • 4
  • 19
  • 15
  • 46
    This approach is generally only used *without authentication* for things like password resets. The unguessable link typically expires within a short period of time, and can only be used by someone already *semi-authenticated* (i.e. the website already knows the email address to which the link was sent). – Robert Harvey Jul 26 '16 at 15:34
  • 6
    Related discussion on security.SE: http://security.stackexchange.com/q/58215/37496 – Mark Jul 26 '16 at 20:39
  • 9
    @MonkeyZeus it is not security through obscurity. The secret, that normally is a password, in this case is a URL. – Davidmh Jul 27 '16 at 08:19
  • 16
    @MonkeyZeus: Security-through-obscurity refers to keeping the *implementation* of the mechanism secret, not using obscure keys. If unguessable urls are security through obscurity, then strong passwords are also. – JacquesB Jul 27 '16 at 08:30
  • Not really an answer to your question but you should look into authentication using client certificates. – JimmyJames Jul 27 '16 at 14:12
  • You should also ask whether a user trusted for the content at one of these URLs should be trusted for all of them. Does receiving one of these make it easier to guess what other randomly generated ones might be out there? – Joshua Taylor Jul 27 '16 at 16:06
  • 1
    @GladstoneKeep keep in mind the URL shorteners. Once someone malicious uses one of them the link will be much easier to guess / write down. – Roke Jul 28 '16 at 13:46
  • 1
    @JacquesB, this is security through obscurity. If I gain access to your code, I can see the URL is 23idcojj20ijf. You've not secured it, you've merely obscured it. This is not the same as gaining access to your salted & hashed password. – Chris G Jul 28 '16 at 19:35
  • 1
    @ChrisG: You can salt and hash the URL the same way, that is not the problem. – JacquesB Jul 28 '16 at 19:45
  • Another difference is that for account passwords, X wrong guesses can disable the account or alert the owner, admin etc. This might not be possible with a URL depending on the site structure. – Nick Westgate Jul 29 '16 at 00:37
  • Of course it isn't. – Dave Newton Jul 29 '16 at 11:04
  • There's no such thing as "unguessable." It's in the same category as massless ropes, frictionless pulleys, and spherically symmetric cows. – Carl Witthoft Jul 29 '16 at 14:05
  • Back in the days of Geocities, my high-school girlfriend put a "secret page" on her site. There was a link that generated a JavaScript dialog box asking for the password; the JavaScript then applied a very weak but non-reversible encryption to the input, compared it to a target value (which was stored in the script) and if they matched, passed the input to a `location` command that navigated to [siteURL]/[input].html – Dan Henderson Jul 31 '16 at 07:47

7 Answers7

208

A private URL is somewhat weaker than authentication with credentials, even if the bit size of the URL is the same as that of the credentials. The reason is the URL may more easily "leak". It is cached in the browser, logged on the server and so on. If you have outbound links, the private URL may show up in the referrer header on other sites. (It can also be seen by people looking over your shoulder.)

If it leaks (by accident or due to carelessness by the user), it may end up being public and even indexed by Google, which would allow an attacker to easily search for all leaked URLs to your site!

For this reason, private URLs are typically used only for one-shot operations like password resets, and typically they are only active for a limited time.

There is a related thread over at Information security: Are random URLs a safe way to protect profile photos? - one answer shares this story: Dropbox disables old shared links after tax returns end up on Google. So it is not just a theoretical risk.

Community
  • 1
JacquesB
  • 57,310
  • 21
  • 127
  • 176
  • 7
    Minor quibble, but "typically used only for one-shot operations" seems an over-statement given that Dropbox (and perhaps other cloudy services) are using them to "protect" access to files. – Steve Jessop Jul 27 '16 at 08:56
  • 4
    I would add that users are taught, with limited success, to protect their passwords, but not to protect their URLs. – svavil Jul 27 '16 at 10:33
  • 1
    You should add, that many of the technical concerns can be mitigated by using a parameter in the private URL so xzy.com/myDoc?auth=8502375 and an automated redirect to a plain url after the authentication checks out. - But this doesn't solve all possible problems – Falco Jul 27 '16 at 11:25
  • Another possible leak is sending this link to a gmail user, as Google parses all gmails and would therefore know about the "secret" link. – Chuck Krutsinger Jul 27 '16 at 17:42
  • 3
    TL;DR - there's no such thing as a secret URL. There's a current attack that exposes URLs to a malicious actor at WiFi hotspots even if you normally send the URL over HTTPS. The attack abuses Web Proxy Autodiscovery (WPAD), forcing your browser to send all of your URLs to the attacker. See (e.g.) http://arstechnica.com/security/2016/07/new-attack-that-cripples-https-crypto-works-on-macs-windows-and-linux/ – Harald Jul 27 '16 at 18:19
  • I think the indexing is the biggest thing here. If it gets indexed, the secret isn't so secret. – RubberDuck Jul 27 '16 at 18:54
  • 5
    @JacquesB Aren't some of the risks you identified mitigated by putting the private part in the fragment portion of the URL (i.e. after the "#", as e.g. Stack Exchange does for its Oauth authentication responses)? For example, the referer header is [not allowed to include the fragment](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.36). – Jason C Jul 28 '16 at 02:38
  • 2
    +1 for One Time Use. This not only means that if it ever leaks it has become useless (since it's used up), but it also means that if on what should be the first access the user is denied (already used), then they can *realize* it was used by someone/something else and take appropriate actions. – Matthieu M. Jul 28 '16 at 12:53
  • @Harald: What kinds of clients actually honor this WPAD ridiculousness, and why would they send a full URL rather than just a domain name? It's impossible to proxy HTTPS except for transparent passthrough of the encrypted stream, assuming correct certificate policy is followed by the client, so there is no possible use case where it would be the right thing to tell an autoconfig proxy the URL. – R.. GitHub STOP HELPING ICE Jul 29 '16 at 15:29
  • A problem with url's to access content is that an attacker can guess at the hashing function and try to access other files. Bitly urls have been show to use too small a hash space so guessing will get access to other content. Since url access is unauthenticated there is no way to stop the attacker from trying lots of guesses. – Michael Shopsin Jul 29 '16 at 15:29
  • @R.. pretty much all of the ones in common use. The problem isn't that HTTPS URLs are sent in clear to the proxy; the problem is PAC. PAC is a bit of JavaScript that's run to decide whether to proxy a request, and if so, to what proxy. PAC receives the whole URL (because "hey, why not") and is capable of leaking information by issuing arbitrary DNS queries (possibly other ways, I didn't see the talk). – hobbs Jul 30 '16 at 06:36
  • @hobbs: I don't think "pretty much all of the ones in common use" is helpful because it's not specific enough to confirm or deny. Knowing specific browser/OS combinations affected would at least allow checking the claim. From what I read so far it seems like it would require cooperation between a bad dhcp client and a bad browser that wants to use the information and pass private information into js from an untrusted & unauthenticated source. Just the dhcp/browser integration requirement should be sufficient to rule it out on OS's not under tight vendor control, though. – R.. GitHub STOP HELPING ICE Jul 30 '16 at 14:21
  • @ChuckKrutsinger That's not relevant. We could say exactly the same thing about emailing the password, or otherwise transmitting any secret in any way that any entity might be able to harvest. But the OP said "private, unguessable URL" - _private_ presumably indicating that they're not going to go shouting it by email or any other medium, just as they wouldn't with the password. The question is whether giving out a URL is inherently less secure than giving out a URL and a password, i.e. excluding irresponsible handling of either. The answer is yes - but not for this reason. – underscore_d Jul 30 '16 at 15:47
  • @R.. If you read the linked article it specifically mentions that the threat is primarily from malicious public wifi providers, who do have control over DHCP. As far as what clients are vulnerable, I'd imagine it includes any clients that have an "autodetect proxy" option. – barbecue Jul 30 '16 at 21:15
  • @barbecue: I understand that the threat is from malicious dhcp servers. My skepticism was about how the junk setting would get from the dhcp server to the web browser. This would require some communication channel between the dhcp client and web browser which should not even exist (and certainly does not on standard Linux setups, though maybe systems like Android or some recent systemd hell adds them). – R.. GitHub STOP HELPING ICE Jul 31 '16 at 03:16
  • @R.. but that's *exactly* what WPAD does, via either DHCP or DNS. "Internet Explorer and Konqueror are currently the only browsers offering support for both the DHCP and DNS methods; the DNS method is supported by most major browsers." – barbecue Jul 31 '16 at 21:52
48

Note:

A lot of people seem to be confusing a "private" URL with authentication. Also, there seems to be some confusion that sending the link via a trusted entity is an attempt at two-factor authentication. To be clear, we're talking about a publicly accessible resource, albeit one that is sufficiently hard to guess.

When using a private URL, you should always assume that it can be compromised -- you should design such a URL so that even if it is compromised, the resource will not leak information to the attacker.

Private/hard to guess URLs are not equivalent to password-based authentication. By nature, private URLs are not private at all -- they are publicly accessible resources. I think the term "private" URL is a misnomer, rather they're "obscure" URLs.

There are certain cases where using a "private" URL is acceptable, but they are inherently less secure than traditional authentication methods such as password authentication or key-based authentication.

Some of the places I've commonly seen "private" URLs used are:

  1. Password Reset emails
  2. Certificate Generation emails
  3. Account / email confirmation emails
  4. Delivery of purchased content (ebooks, etc)
  5. Other misc things like flight check-in, print boarding pass, use private URLs in addition to traditional authentication

The commonality here is that random URLs are typically only good for one-shot operations. Also, traditional authentication and random URLs are not mutually exclusive -- indeed, they can be used in conjunction with each other to provide additional security when delivering a resource.

As Robert Harvey has pointed out, the only way to securely use a random/private URL is to generate the page dynamically and submit the URL to the user in a way such that the user can be considered semi-authenticated. This could be email, SMS, etc.

A randomly generated/private URL typically has a few properties:

  1. It should expire after a reasonable amount of time
  2. It should be a single-use URL: IE it should expire after the first time it's accessed.
  3. It should defer the user's authentication to some other entity that it trusts to securely authenticate the user. (By sending the link via email or SMS, etc)
  4. It should be impossible for a modern computer to brute force the URL in the timeframe preceding expiration -- either by rate limiting the API that exposes the resource or by creating a url endpoint with sufficient entropy such that it cannot be guessed.
  5. It should not leak information about the user. IE: If the page is to reset a password: the page should not display the requestors account information. If Alice requests a password reset link and Bob somehow guesses the URL, Bob should have no way of knowing whose password he is resetting.
  6. If it does leak information about the user, it should be used on top of traditional authentication, for instance a page may consider a user authenticated if they have a cookie set or if their session_id is still valid.

Different resources require different levels of security. If you want to share a secret recipe with some friends, for instance, it would be acceptable to use a random/private URL to share it with them. However, if the resource could be used to steal somebody's identity or compromise their accounts with other service providers, you'd likely care much more about restricting access to that resource.

Community
  • 1
Charles D Pantoga
  • 806
  • 5
  • 9
  • 4
    If I wanted to share the secret recipe for Coke with my product development team, that would require something somewhat different from if I wanted to share the recipe for the potato salad I served the neighbors during a neighborhood barbecue party. Again, context. :-) – user Jul 26 '16 at 19:08
  • 7
    @MichaelKjörling I'm not sure how somebody would infer diff differently from my post. Quite clearly I stated that different resources require different levels of authentication. The recipe for coke would is much more valuable than the recipe for grandma's cookies. – Charles D Pantoga Jul 26 '16 at 19:46
  • 9
    @CharlesAddis Clearly you've never tasted my grandma's cookies! – Brian Jul 26 '16 at 21:29
  • 1
    I think, although I might be wrong, that @Michael's saying your 5-point description of the properties a secret URL should have, is already overkill for sharing a secret recipe with friends. Making each one single-use (and therefore needing a separate URL per friend who accesses the recipe) in particular seems a lot of hassle for negligible benefit, especially if there is *also* an expiry time. I read your answer to mean, "it's acceptable to use a private URL, but private URLs should have these 5 properties", and IMO that's slightly wrong. – Steve Jessop Jul 27 '16 at 08:59
  • I downvoted this answer because **email and SMS are not secure**. Any information you send via email or text should be considered potentially compromised; anyone can read data sent by email. This is why the usual app-sec advice is never to send passwords by email. It is a very poor choice for sending URLs which should only be accessed by the recipient. – Benjamin Hodgson Jul 27 '16 at 09:20
  • 1
    @BenjaminHodgson that is precisely the reason for item #5 => if the link/URL ends up in the wrong hands, it should not leak any information about the user who requested it. – Charles D Pantoga Jul 27 '16 at 14:53
  • "SMS-based two factor authentication is insecure and should be banned, warns US standards body" - http://www.ibtimes.co.uk/sms-based-two-factor-authentication-insecure-should-be-banned-warns-us-standards-body-1572676 and others – Mawg says reinstate Monica Jul 27 '16 at 15:36
  • @mawg 2-factor auth has nothing to do with this post? Given that the link you posted has nothing to do with this post, I'm not sure why you've posted it? – Charles D Pantoga Jul 27 '16 at 15:39
  • @Mawg - This description of a private/secret URL is in contrast with authentication. The functionality is completely different as the URL provides no mechanism of authentication, and defers that mechanism to a trusted entity that delivers the URL. Simply knowing the URL in this case is enough, since the URL doesnt leak any information about the requestor. In authentication, knowing a public resource (url, username, etc) is not enough -- you must also know a secret in order to access the resource. – Charles D Pantoga Jul 27 '16 at 15:49
  • I was just expounding on BenjaminHodgson 's comment – Mawg says reinstate Monica Jul 27 '16 at 16:12
  • @Brian We all know *your* grandma's cookies are worth more than Coke, but Charles Addis didn't specify *who's* grandma. He just said "grandma". – wizzwizz4 Jul 27 '16 at 16:43
8

Pretty much all authentication schemes boil down to proving that you know a secret. You authenticate yourself to some service by proving that you know a secret password, or a secret URL or,...

Some more advanced protocols (e.g., OAUTH, Kerberos, ...) enable you to prove that you know the secret without actually transmitting the secret. This is important because there are more ways to obtain an "unguessable" secret besides guessing it.

I could be sitting in the same Internet cafe as yourself, eavesdropping on your WiFi connection when you type in your "unguessable" URL. At that point, if you weren't using SSL, or if I can exploit the latest new bug in your SSL implementation, then I would know your secret too.

Solomon Slow
  • 1,213
  • 9
  • 14
  • 1
    To be fair, this also holds true for authentication, or _any_ communication. – Andy Jul 26 '16 at 18:13
  • "eavesdropping on your WiFi connection" work on anything: URLs, CSRF protected `
    `s, JavaScript military grade encrypted data (maybe active sniffing will be needed). It's simple to fix it: use HTTPS.
     – Gustavo Rodrigues Jul 26 '16 at 21:42
  • @GustavoRodrigues First of all, if eavesdropping really did "work on _anything_", then it would work on HTTPS. Otherwise, what does "anything" mean? Or, what magic do you think is in HTTPS that puts it above everything else? – Solomon Slow Jul 26 '16 at 22:11
  • 2
    ...but there _is_ magic that wards off eavesdroppers: It's public-key cryptography. Here's a simple example: A service sends me a _challenge_ containing a random number and a timestamp. I sign the challenge with my private key and return it. If they can validate my response with my registered public key, then that proves that I know the secret (the secret is my private key), but I proved it without ever revealing it to a potential eavesdropper. It won't help the eavesdropper to replay my response, because the service will never send the same challenge twice. – Solomon Slow Jul 26 '16 at 22:16
  • HTTPS (That is, HTTP over SSL) uses public-key crypto, but FYI, the most popular implementations of SSL have a history of bugs that have allowed the eavesdroppers to break in even in spite of the cryptography. New bugs (a.k.a, "exploits") seem to be discovered two or three times each year, and all of the developers of all of the products that use SSL have to run around with their hair on fire until the latest exploit is patched. (Don't ask me how I know!) – Solomon Slow Jul 26 '16 at 22:25
  • I'm no expert in the area, but - *even if there are exploits* - there is better than HTTPS for web sites? Protecting authentication data putting it ouside URLS, which is the focus of this question, will not solve it. Many websites try to use JavaScript to replace HTTPS, which simply don't work. Even if HTTPS is being used using JavaScript to improve security (something like MEGA's secure boot or some JavaScript Secure Remote Password), as browsers don't verify authentic for code, just for the connection, the server can disable it: it's meaningless. – Gustavo Rodrigues Jul 27 '16 at 11:49
3

Lots of good answers already in this thread, but to directly address the question:

From the perspective of an evildoer who wants to access this resource, are these approaches equivalent? If not, what makes them different?

Let me establish a definition. "Authentication" is the providing of credentials to prove a claim of identity. Access control is usually based on the identification of the user.

Your secret URL is not bound to a specific user. As others have pointed out, it could end up in a proxy's log file, or a search request that gets indexed by google, or many other ways it could leak out.

On the other hand, a password is tied to a unique user account. You have the ability to reset it, or only allow it to be used form the user's home location, or known IP address, or any number of other controls.

A username/password gives you much more granular control of the access.

Access control allows an identity (subject) access to a resource (object). In your URL example the identity is "anyone who ever gets the URL, via any means."

Go with the username/password if you can. URLs leak out in all sorts of unexpected ways over time.

JesseM
  • 139
  • 3
1

A secret URL is just as secure as a secret password. However, passwords are easier to keep secret than URLs, because everyone and their programs knows that passwords must remain secret.

For instance, your browser will not show a password on screen, only store passwords with your permission, and offer means to protect that password storage (such as encrypted storage unlocked by a master password). In contrast, it will always show URLs on screen, may possibly leak them through the referrer header, and store them automatically in your browsing history without any further protection.

Likewise, HTTP proxies will not usually log passwords, while URLs are commonly logged.

Using URLs for authentication also means that sharing URLs shares authentication, which makes it hard to individually revoke (or record) access.

And of course, secret URLs inherit all weaknesses of secret passwords. In particular, using a password for authentication will reveal that password to the server and anyone able to read your communication.

meriton
  • 4,022
  • 17
  • 18
  • 3
    This answer makes a lot of assumptions which are wrong. If you log in to a site with HTTPS, and type in your username and password, the hops in between and the proxies won't know your password. – Pieter B Jul 27 '16 at 08:52
  • By "intercept your communication" I meant the ability to reconstruct its contents. This may or may not be prevented by HTTPS. In particular, trusting a single bad certificate (for instance by some corporate HTTPS proxy that uses the same private key for all installations) allows an attacker to man-in-the-middle the HTTPS traffic. – meriton Jul 27 '16 at 12:29
  • 2
    but by encoding the secret in the url you basically make HTTPS totally unusable. Every hop between the client and the server has the secret.No compromised certificates needed. – Pieter B Jul 27 '16 at 13:02
  • 4
    Nonsense; in HTTPS, the URL is only transmitted in the encrypted stream. You must be confusing it with the domain or IP, which are visible in DNS lookup and IP header fields, respectively. – meriton Jul 27 '16 at 20:28
1

Another item not noted anywhere is throttling of 'guesses'. For most password authentication systems, you get a limited number of attempts at guessing a password for a user before further authentication attempts are either locked out, or limited.

While you could do something similar with 'guesses' against your URL scheme, it would be somewhat harder to implement. If there is a recognizable pattern to your URL generation, then it may be hard to stop someone setting up to work their way through your 'random' URL space.

Paddy
  • 2,623
  • 16
  • 17
0

There's another aspect which I didn't see mentioned yet - URL shorteners.

In a recent publication (April 2016), it was claimed that URL shortener services completely nullify the increased security provided by random generated "unguessable" URLs. The URL space of the shorterner service is considerably smaller than your randomly generated URL - meaning that any such "secure" URL shared with a shortener service can be guessed in an easier fashion than anticipated.

To illustrate - let's assume your random URL is 128bit long (i.e a guid). Also, let's assume that your random number generator is really strong and that you generate those bits in a uniform way. Guessing a 128bit number is very hard and can take a considerable time - your URL is effectively 128bit key protected.

Then, let's assume someone shared this URL on the google URL shortener. Today that service emits a 10 character long ID, composed of letters and numbers. (26+10)^10 ~= 3.6*10^15 < 2^52 - so we've effectively halved the key strength from 128 bit to 52 bit.

Add to that fact that the generators do not use the entire space due to other consideration and you can mount an effective attack that combines brute force with side channels (most likely pre-allocated random URL buffers).

The original article: http://arxiv.org/pdf/1604.02734v1.pdf
A blog post summarizing the article (by the author): https://freedom-to-tinker.com/blog/vitaly/gone-in-six-characters-short-urls-considered-harmful-for-cloud-services/

Ran Biron
  • 457
  • 2
  • 5
  • 10
  • 2
    Well, yeah, but one would hope anyone using such services for sensitive data would know better than to post the URLs _anywhere_, including to a shortening service. This isn't really any different from saying `Gah! My password/private key is too long and complex. I know! I'll just write it in a text document and put that in a zip file with an easier password.` Both are transparent fails, which one hopes against hope people wouldn't be silly enough to do. But yes, in reality, sadly your warning is probably needed ;) – underscore_d Jul 30 '16 at 15:59
  • @underscore_d yeah exactly - if you know this subject in enough detail to comment then this isn't a blog-worthy point. – Rob Grant Jul 30 '16 at 17:40