37

We’re working on a new service – this service will potentially be called directly from applications on user devices. These applications will be developed and supported by multiple development teams from all over the organisation, all depending on the data we provide.

We’re keen to identify which applications are sending which requests, so that we can identify usage patterns and developers responsible. (For the avoidance of doubt, user authentication is handled separately.)

Our solution is to require API keys, one per application – then we have contact details for the development team.

We don’t want getting the API keys to be a source of friction, but we’re concerned that developers will share them to colleagues in other teams, meaning we can no longer identify traffic for just one application.

How can we incentivise developers not to share API keys internally?

Community
  • 1
Oli
  • 525
  • 1
  • 4
  • 4
  • 5
    How will these teams access the API? Via the internal network? Generally different teams are put in different subnetworks so you could enforce the use of a certain API key by network... Anyway the social solution is to tell them "it is important that you do not share this API key not for security but because we need the metrics of different users to improve it. If somebody asks you just tell them to ask us and we will gladly and efficiently provide an API key to them". – Giacomo Alzetta Sep 11 '19 at 09:47
  • 3
    If you don't want people to share keys to colleagues, make it easy to include a configuration file that is ignored by the versioning system (so that the key is never commited) and also make it easy to create new keys. Nobody will bother sharing a secret key if another developer can easily create a new key by themselves. The problem with sharing personal keys is usually a problem caused by the fact that getting new keys takes time. – Sulthan Sep 11 '19 at 17:52
  • Have you considered requiring registration when the application is first started? It could show a splash screen asking for the user's contact details (or whatever information it is that you need) and then issue the API key on the spot. – John Wu Sep 12 '19 at 03:31
  • "How can we incentivise developers not to share API keys internally?" Simply stated, tie every key to the MAC address of the network card(s) in the computers that run them. Networking protocols prevents the same MAC addresses from being used in the same network, so that keeps people from using the same keys over and over again. I'd have made this into an answer, but I don't have the rep for it currently. – Blerg Sep 12 '19 at 11:25
  • Curiously, I don't see the word "rotation" (as in *key rotation* -- credential expiration/rotation) anywhere on this page at the moment. Once someone obtains a key, does it have a finite lifetime after which it must be rotated out of use and replaced by a new one? If not, why not? – Michael - sqlbot Sep 12 '19 at 12:36
  • @Blerg You cannot tell the MAC address of the machine that sent an IP packet (or created a TCP connection), if the machine is on a different network segment. IP address would work, but does suppose that you have one-machine-one-API-token, which likely isn't true in highly-available setups. – James_pic Sep 12 '19 at 12:36
  • This is a social and management issue. And I would on the contrary encourage API keys sharing between developers inside your company. This is more productive at company level. Read also [Bullshit jobs](https://en.wikipedia.org/wiki/Bullshit_Jobs) and [this](http://www.shoshanazuboff.com/new/recent-publications-and-interviews/big-other-surveillance-capitalism-and-the-prospects-of-an-information-civilization/) ...., it is extremely relevant to your question – Basile Starynkevitch Sep 13 '19 at 06:02
  • @BasileStarynkevitch Without having read those links in detail, perhaps you could explain a little more? My really specific use case is identifying patterns of queries that are 'good' (work quickly, never fail, etc), versus patterns of queries that are 'bad' (fail often, consume system resources, etc) or will soon be 'bad' (depend on functionality or data soon to be retired). If I want to speak to the developers (not end-user) responsible for writing a particular query in an organisation of 200k people, how would you suggest I identify them without something unique on the request? – Oli Sep 13 '19 at 13:16
  • Read more about Open Source. Suggest the same approach for your entire company. Several corporations have been successful with this approach for their proprietary software. – Basile Starynkevitch Sep 13 '19 at 17:09
  • @James_pic you are correct on that. What I should have said was that you can have each system that connects to the service send the local system's MAC address, and possibly a key as well, for authentication. I don't know of any protocols that would prevent finding out the MAC address of the local machines, so that could work. Change and spoofing MAC addresses is possible, but that would just make systems need keys more often. – Blerg Sep 18 '19 at 05:54

8 Answers8

75

In order to share those keys between teams, the teams need to talk to each other, agree to share, then share them. This takes time. So if a team can request API keys from you more quickly and more easily, there's no incentive to share.

And the easiest way for them to request those keys is for you to pre-empt them. Assuming you know all the other teams that will need API keys, create them and share them before making the service available to them.

There's one other incentive that you can offer: debugging support. Those teams will want your help when things don't quite work properly when they integrate their work with your service. Those API keys allow you to track their specific requests and thus to assist in debugging what's going wrong. So sell that as the reason for the keys, rather than "identify usage patterns and developers responsible", which sounds like you are spying on their activities.

David Arno
  • 38,972
  • 9
  • 88
  • 121
  • 2
    Re: sharing, in an ideal world you’re right - but you’re presupposing they have perfect information (though I can help by directing them to docs from the schema, root of the endpoint, and any errors) and cooperative management, and not the reality where all they may have is the endpoint and some code copied from another team which they were forwarded by a senior manager. – Oli Sep 11 '19 at 07:27
  • 3
    Re: pre-empting, you’re absolutely right, we should aggressively create and send out keys to interested parties. What I didn’t mention is that some of these apps use a common framework/interface, and we could perhaps semi-automatically inject or enforce unique keys at that layer, but that doesn’t apply to all apps. – Oli Sep 11 '19 at 07:29
  • Re: debugging support, good idea, thanks! – Oli Sep 11 '19 at 07:30
  • @Oli: You're technically correct about the presupposition of perfect information; but it's impossible to find a universally fitting solution to pre-emptively counter anyone's potentially uninformed and unpredictable behavior. No system can ever be used as a way to guide people who ignore/don't know the existence of the system. – Flater Sep 11 '19 at 08:12
  • you should obviously be as helpful as possible, but this doesn't help you when people share the api key anyway. You need to be able to detect when it happens and chase them – Ewan Sep 11 '19 at 10:15
  • 1
    @Ewan if you simply disable access to given key all the users will run to you - no need to chase them. And then you can give them unique keys :) – Alexei Levenkov Sep 11 '19 at 20:58
  • 15
    Pre-empting should take this form: accessing the API without the key produces an error message with a link to the page where you apply for the key. Don't expect anyone to read documentation. Incentives don't work when no one knows about them. – candied_orange Sep 11 '19 at 22:25
  • 1
    If error messages or debug logs were automatically emailed to an address linked to the key, anyone who wanted the data would have an incentive to use their own key - and anyone who's key was shared would have an incentive to track down the person who shared it and get them to stop. – Robin Bennett Sep 13 '19 at 09:29
20

Good answers already, I just thought of a different approach which may or may not work for you.

Rather than issuing keys to be included you could require the header of requests to include the name of the front end application, to be created and formatted by the developer of the front end application, like web browsers do. That way front ends could still pretend to be a different application but there would be no benefit to doing that so that seems unlikely. Just let the front end identify itself and accept any non-empty string.

Martin Maat
  • 18,218
  • 3
  • 30
  • 57
  • That would make life somewhat harder if ownership of an application changed - for example, we could just update the API key details stored on our end, but if we accept free form text that requires the application to make a code change. – Oli Sep 11 '19 at 09:32
  • 1
    @Oli If ownership of an application changed and the (new) developer would deem it appropriate to update the application's identity (which it truely has because someone else is maintaining it), what would be the problem? You could differentiate between pre ownership change and post ownership change, just regard them as two different applications. I would not expect any new owner to notice the name in the header anytime soon though. – Martin Maat Sep 11 '19 at 09:48
  • 3
    This is what I do. have the client have a construction parameter which is the name of the app and/or use reflection to pull other stuff like the machine its running on, its version etc. The key thing is to enable you to report when people are NOT following your api policy – Ewan Sep 11 '19 at 10:13
  • 1
    If the organization has a common approach to version control, e.g. everybody keeps their code on the org's GitHub server, have each app send an URL to it's repo and the commit hash from which it was built. The commit hash can be included in the code as part of the build process, so no need for developers to update anything. Having the repo URL lets you see who the owner is, and getting specific commits would let you notice behavior differences between versions. – Caleb Sep 11 '19 at 12:36
  • @Caleb If things were centralized like that the OP would probably not have this problem. From what I understand the front end application developers are anonymous lot to the OP, with private ways of software development. – Martin Maat Sep 11 '19 at 15:38
16

In short:

First: facilitation and benefits; If necessary: friction and police.

Some more words

Facilitation: First, make it easy for a team to get a new API key. For instance add a reminder in the corporate procedures for launching new projects, and offer an easy to use service to request a new keys, without asking for justification.

Benefits: Make the usage of an own API key be a benefit for the team or the product owner. For example, propose some feedback about app usage based on that key.

Friction: Depending on the key feature, you can create friction, for example if key is linked to somme app-defined domain (i.e. reusing keys would not necessarily give access to all desired services).

Policing: Finally, you may need to foresee some policing measures. For example, you may monitor usage of api functions by api key and after a given time to establish a baseline, inquiry about use of api parts which is not expected in view of the baseline. Or if this is not realistic, simply include in the corporate project-review checklists the verification that a valid key was used.

Remark: you may need to be very clear on your API key policy: Would a new major version require its own API key ? What with a fork, or if an app is split up ? what if another team is in charge, etc...

Christophe
  • 74,672
  • 10
  • 115
  • 187
6

Generally the easiest way to get developers to "do the right thing", is to make it easy for them to do so.

To that end I would suggest building a API key issuing web page/site. In its simplest form it could be just a login (ideally tied to your corporate AD/LDAP) and the page that just asks for the application name and issues the key.

At the end of the day you can always revoke keys later, so all you really need the site to do is record who (username) requested the key and what (Application Name) they want to do with it - along with any info needed to revoke the key later.

You could do something similar with a ticketing system, but at the end of the day it's very easy for me to copy and paste a key from one app to another, so it has to be really easy to request a new key, to avoid bad behavior.

DavidT
  • 1,074
  • 3
  • 5
1

Be pro-active.

Identify likely developers and GIVE them unique API keys in a secure channel, ahead of time. Provide an easy means of requesting new API keys. Provide an easy means of new people requesting new API keys. When new interns or hires join the team, give them a JIRA ticket or similar "Request an API key" with the steps in the description.

Keep track of which API keys have been used, and which ones haven't. If Bob has submitted tickets in the project but hasn't been using his API keys, then he's probably borrowed someone else's.

Have Management's Support. Don't be a Nosy Nancy going any making up rules that don't matter. Literally convince Management that it's important, and then they are the ones to convince the group that it's important. Don't work on convincing everyone.

And the most annoying and tyranny-prone suggestion : Be aware of misuse, and crack down on the same day. Same hour is best. Don't say "Bad Naughty Developer" say "Here are the proper steps." If they do it repeatedly, disable the misused key. This hassles both the Sharer and the One Who Borrowed, and the sharer will say "No, do it properly" in the future. Avoid disabling keys that are in live projects.

Christopher Hostage
  • 119
  • 2
1

How can we incentivise developers not to share API keys internally?

  • Generate keys as a result of self-service application registration.
  • Require a point of contact before keys become active.
  • And ask them not to share. (Create a terms of service and/or tell them why it's better for them not to share.)

You should also implement rate-limiting. This in itself could discourage sharing of keys. It protects your system to some extent against abusive applications. (And downright malicious ones.) And, it ensures you'll be somewhat informed prior to a massive increase in serviceable traffic. (Giving you time to add capacity, I hope!)

And, with rate limiting, when an application does require a higher limit, it opens dialog with the POC registered for the key. You get an opportunity to ask if keys are being shared, explain why that's harmful and so forth, and you can offer additional keys when it's appropriate instead of the requested rate limit changes. Etc.

svidgen
  • 13,414
  • 2
  • 34
  • 60
0

One way to do things, especially if the teams use a shared build system (or at least a sufficiently common one) is to set up an internal server that creates and issues API keys (given a few basic bits of info about the product using it). Then use a script that grabs a new API key from the server for each build, or for each version update. Let devs run the script to get a different key for their local builds as well. (Where possible, automate this as part of the build so they don't even need to think about it.)

This would let you tell whether it was something in production, QA, or dev, and at what version/build the problems started.

Miral
  • 101
  • 3
0

The first and best thing you can do is to format the keys so that they include the application name in an easily readable form, and don't work if you change it.

If it's obvious when teams are using the wrong key, then they'll endeavor not to.

Then, periodically expire keys. You should do this anyway, and when a key is nearing expiration you can send a new one to the team that owns it. The team that uses a key will then be motivated to ensure that they are the team that owns it, so that they'll get the new one when it expires.

Matt Timmermans
  • 537
  • 2
  • 6
  • 1
    in practice though expiring keys may be too much of a hurdle for the adopting application - i can see managers saying "fuggetaboutit" as it's just going to be trouble later. – davidbak Sep 12 '19 at 23:38