Moving on from REST to a Unified API

APIs and Data Models

One of my hats at my current job is API and Data Model Guy. My official title is something like Principal Engineer, but really I’m the guy who makes the API work. For our current product line that API is done with Spring Data, but at the last job it was Rails, before that it was ColdFusion and PHP, etc, etc.

I’ve been thinking quite a bit lately about how your data model is transferred across your API, and what effects that has on the complexity of your app.

Stick with me for a few minutes while I walk back through how we got here.

Web 1.0: HTML

These were the bad old days when your UI and your data were sent in the same response. Let’s not dwell.

Web 1.5: SOA, AJAX, and SOAP

In case you’d forgotten, the X in AJAX stood for XML. The first foray into truly dividing UI from data was with XML (remember XSLT?), which was only much later beat out by Javascript and JSON.

What we ended up with we didn’t really have a name for at the time—we just called it “the API”. It was what we now know as feature-specific or screen-specific endpoints. You’d have an endpoint for user registration, one for sign-in, one for your account details, etc, each with a data model your data that probably looked exactly like your UI widget layout:

>>> GET /api/account
<<< 200 OK
    {
      "user": {
        "id": 1234567,
        "name": "ricko",
        "email": "ricko@aol.com",
        "realName": "Rick Osborne
      },
      "friends": [
        {
          "id": 2345678,
          "name": ...
        }
      ]
    }

If you were thinking about data model at this time, it was almost assuredly a term that your server-side devs worried about, but your UI devs probably didn’t. Even with structured data representations like SOAP and WDDX the data model usually collapsed as it went across the API.

Web 2.0: REST

For most people this was just the unabashed commitment to JS and JSON. But for some, UI frameworks that pushed the idea of a real data model started to become popular. Backbone.js and Ext JS brought data model and MVC concepts to the UI. Concurrent with this, REST became popular because people got really tired of maintaining APIs with hundreds of screen-specific endpoints:

>>> GET /api/v1/session
<<< 200 OK
    {
      "userId": 1234567
    }
>>> GET /api/v1/user/1234567
<<< 200 OK
    {
      "id": 1234567,
      "name": "ricko",
      "email": "ricko@aol.com",
      "realName": "Rick Osborne
    }

The complexity shift here is interesting. The endpoint naming is more uniform (/api/v1/:noun/:id), dividing up your data model along the types. The main upside to this is that it’s easier for everyone to reason about. The main downside is that you then have to put thought into how to do things like lists and relationships. The latter were generally done as sub-resources:

>>> GET /api/v1/user/1234567/friends
<<< 200 OK
    {
      "friends": [ 2345678, 3456789, 4567890, 5678901, ... ]
    }

But then you were left with the list problem: how do you easily get lists of things without making a zillion little fetches? Maybe your API added something like an expand option that let you (non-RESTfully) embed related resources as if they were sub-resources:

>>> GET /api/v1/user/1234567/friends?expand=1
<<< 200 OK
    {
       "friends": [
         {
           "id": 2345678,
           "name": ...
         }, ...
       ]
    }

This replaces a zillion little fetches with aggregated responses, as long as you were willing to not think too hard about browser caching, multiple representations of the same data, etc. Is the user 2345678 data you got from your friend endpoint different than the one you get from the /user/2345678 endpoint? If you request both and they come back different, how do you know which one is more current? If something changes with one, how do you invalidate the browser’s cache of the other? Etc, etc, etc.

Or maybe your API added bulk-fetch:

>>> GET /api/v1/user/bulk?id=2345678&id=3456789&...
<<< 200 OK
    {
      "2345678": {
        "id": 2345678,
        "name": ...
      }, ...
    }

This is effectively the same from a complexity and trade-off standpoint. Sure, you were dropping the “relation” pretense, but you were trading it for some kind of unofficial “bulk” format. You weren’t solving the caching problems, or the concurrency ambiguities, but you were still at least thinking in terms of the data model, data types, and how you could break things down into atomic components reusable by any configuration of the UI.

Web 2.1: HATEOAS

Sure, it’s probably the worst acronym in the history of the Web. And it’s shockingly ambiguous from a “yeah, but what do I do?” standpoint. But it did move a few extra pieces of business logic out of the UI and into the data model—namely the “where is the canonical representation of this data?” (rel="self") and “what am I allowed to do with it?” (rel=":verb") parts.

It didn’t really solve the list problem, though the canonical links did help with the concurrency problem and at least pointed toward the right direction of solutions for the caching problems.

Today

Let me make a maybe-controversial assertion: REST is okay but not great and needs to evolve.

  • Lists and relationships suck.
  • Bulk fetch sucks.
  • You still have a bunch of hard-coded paths in your UI. (Users are at /user, etc.)
  • For complex data models REST drops you squarely into a covariant/contravariant quagmire.

That last one is on the propellerhead side, but easy to explain. Say that you accept a number of different credentials to sign into your app. Maybe AmazonCredential and EmailPasswordCredential and SamlCredential. In your code they all implement an interface like ICredential. But how does that work across a REST API? You might get a list of credentials:

>>> GET /api/v1/user/1234567/credentials
<<< 200 OK
    {
      "credentials": [
        {
          "id": 663,
          "type": "AmazonCredential",
          "accessKey": "...",
          ...
        },
        {
          "id": 664,
          "type": "SamlCredential",
          "idp": "...",
          ...
        }
      ]
    }

In more concrete terms, we might say we have a List<ICredential> credentials. But then questions are raised:

  • What’s the canonical endpoint for an AmazonCredential? Is it /credential or /amazonCredential or /credential/amazon? Presumably, SAML credentials should mirror this: /credential or /samlCredential or /credential/saml.
  • If you can GET /credential/663 for the Amazon credential and GET /credential/664 for the SAML credential, what does that mean? You’d have two different schemas coming out of the same endpoint. If you restrict the endpoint to only send the common subset of fields for all credentials so they are uniform, then you have yet another non-canonical format.
  • What happens if you PATCH /credential/663 the Amazon credential with an idp field, which should only exist on a SAML credential? In concrete terms, this is the equivalent of credentials.get(0).setIdp(...) which must fail because setIdp(...) does not exist on ICredential.
  • Say you decide /credential is too tricky and instead allow only /amazonCredential and /samlCredential. But when given an ID (665) you have no way of resolving it if you don’t also know its type, and therefore its endpoint.

Proposal: Unified Endpoints

Over the last three years at work we’ve been moving to a system we call Unified Endpoint. It does a few big things:

  1. Unify all CRUD endpoints together into one: /api/data.
  2. Unify object ID namespaces into one, so that IDs are globally unique across the entire system, no matter what the type is. The format isn’t important, but we use Base64-encoded UUIDs.
  3. Unify the field namespaces into one, so that any field name has exactly one domain (type to which it can belong) and one range (type to which it can point).
  4. Unify the JSON representation into a single abstract format. Again, the format isn’t critical, but we use a dialect of JSON-LD.
  5. Unify the query and filtering abstraction. In our case it’s a dialect of JSONPath though you could just as easily use GraphQL or similar.

For example:

>>> GET /api/data/1234567
<<< 200 OK
    {
      "@id": 1234567,
      "@context": {
        "1234567": {
          "@id": 1234567,
          "@type": "User",
          "fullName": "Rick Osborne",
          "credential#1": {
            "@id": 663,
            "@type": "AmazonCredential"
          },
          "credential#2": {
            "@id": 664,
            "@type": "SamlCredential"
          }
        }
      }
    }

Above you can see the @id + @type combination reference that acts sort of like a join/relationship or HATEOAS link. The @type is necessary because the "credential" key just gets you List<ICredential>—thanks to the unified field namespace you know that credential always has a User domain and a Credential range (or ICredential, though all those extra I prefixes aren’t helpful). But the UI might make decisions about whether or not to follow the link based on the concrete type, so we include it in the reference just in case. The @id alone is enough to allow you to fetch it, obviously.

Similarly, deserialization on the UI side uses @type to figure out the concrete class to use for deserialization. Maybe all of your subtypes have concrete types, maybe they don’t. But your UI can decide where to make a new concrete implementation, or it can fall back to the Javascript bag-of-things approach.

I’m omitting optimistic concurrency controls for brevity (we use an opaque nonce in a @hash key), but I might change my name via:

>>> PATCH /api/data/1234567
    {
      "@id": 1234567,
      "@context": {
        "1234567": {
          "fullName": "Rick O"
        }
      }
    }

All that @context nonsense and @id duplication seems silly until you consider the list/bulk case. If I wanted to fetch my User as well as any direct relationships, I might ask for the same thing with an expand parameter:

>>> GET /api/data/1234567?expand=1
<<< 200 OK
    {
      "@id": 1234567,
      "@context": {
        "1234567": {
          ...
        },
        "663": {
          "@id": 663,
          "@type": "AmazonCredential",
          "awsAccessKey": "...",
          ...
        },
        "664": {
          "@id": 664,
          "@type": "SamlCredential",
          "samlIdp": "...",
          ...
        }
      }
    }

Here, my top-level @id is the same because I’m still requesting my User. But I’ve added full versions of my credentials embedded in the @context so you don’t need to fetch them separately. Duplicating the @id as both the @context key and then again inside its object seems redundant, and it is, but it also means that you can easily extract and pass those context objects on to parts of the UI without also having to pass their @id because it’s already embedded. We could just make the @context object an array, but that would make assembling them from disparate parts of the UI trickier because you’d constantly be searching the array for objects with the matching @id.

The expand I listed here is naive, it just gives you all relationships one level out from your requested object, but you could also specify a directed expand (a traversal) via your query/filter abstraction.

So why have that top-level @id at all? Why not just promote everything in the @context to the top level? Two reasons. The easy one is because we do have some other top-level fields for request-level metadata (so we don’t have to use custom HTTP headers). This includes things like pagination metadata.

But the more interesting case is lists of things. Say you wanted to bulk-create a number of objects in the same transaction:

>>> POST /api/data
    {
      "@list": [ "!a", "!b" ],
      "@context": {
        "!a": {
          "@id": "!a",
          "@type": "User",
          "fullName": "Alice Aardvark",
          "credential#1": {
            "@id": "!b"
          },
          "friend#1": {
            "@id": 1234567
          }
        },
        "!b": {
          "@id": "!b",
          "@type": "EmailPasswordCredential",
          "epcEmail": "...",
          "epcPassword": "..."
        }
      }
    }
<<< 200 OK
    {
      "@list": [ 98765432, 667 ],
      "@context": { ... }
    }

Above, I’m creating a new user and her email/password credential at the same time. I’ve provided two transient identifiers, !a and !b, in my @list because I want to know what @id values they end up with later. I might have !c, !d, etc, in my @context below, but I only care about the final @id values of what I put in the @list, which is what I got in the response. You can see that I also added the relationship to the Rick User 1234567 at the same time. And yes, you can POST or PUT or PATCH or DELETE with either a top-level @list or @id and there’s no real semantic difference, just whether you want back many or one identifier.

The epcEmail and epcPassword fields might offend you, due to the unified field namespace. Another option, which we use at work, would be to formalize a field naming scheme based on something like URNs: "credential:emailPassword:email" and "credential:emailPassword:password". It’s more verbose, to be sure, but it’s also unambiguous and prevents arguments about semantic differences between similarly-named fields. (Does "name" imply the full name, the Unix-style name, etc?)

You may have also noticed that we don’t use arrays in @context values. Arrays are troublesome things, and we’ve found that they cause more problems than they are worth. They may or may not imply an order. When a single-valued field gets promoted to accommodate multiple values, how does that work? We’ve decided to instead handle multi-valued fields by appending their field identifier to the field name, URL-style with a #:

{
  "credential#abc123": {
    "@id": 663,
    "@type": "AmazonCredential",
    ...
  },
  "credential#def456": {
    "@id": 664,
    "@type": "SamlCredential",
    ...
  }
}

The abc123 and def457 parts are the field identifiers. Notice that they are not the same as the @id of the object they reference. Those field identifiers aren’t just random garbage—they uniquely identify the combination of the domain, the relationship (field), and the range. In the cases above, the relationship from the Rick User account to each of his Credentials. We don’t just use it for relationships/references, but for all fields:

{
  "@id": 1234567,
  "@type": "User",
  "fullName#ghi789": "Rick Osborne",
  "credential#abc123": { ... },
  "credential#def456": { ... },
  ...
}

This seems silly until you see that it can be used for single-value PUT and PATCH requests, where you don’t have to worry about the domain (User) object. For example, this would be another way of changing my User name:

>>> PATCH /api/data/ghi789
    {
      "@id": "ghi789",
      "@context": {
        "ghi789": {
          "@value": "Rick O"
        }
      }
    }

This means that every single value in our system is individually addressable. When using optimistic concurrency you can modify a single field on an object without having to assert the state of the entire object. Sure, there will be business logic cases where some fields must be entangled with other fields, but that’s for the service to worry about. From the UI’s perspective, it can now tie that field identifier to a UI widget and not worry about which object it came from—it can request changes to it without any other context.

Traversal

I’ve already spent far too long for what was supposed to be a mile-high view, so I won’t spend as much time here. Suffice it to say that the other half of the relationship problem requires something a bit more radical. That is: without resorting to a bunch of key-value GET requests, how do you walk along the entities and relationships in your data model to encounter objects about which you don’t know anything?

As I mentioned before, we’ve chosen to use a very simple JSONPath approach. We call it Find/Like because it’s a find that works with a like query parameter:

>>> GET /api/data?like=$[?(@.id==1234567)].friend.fullName

That $[?(@.id == 1234567)].friend.fullName expression is a very simple example: starting with the object identified by 1234567 (which we know is a User, but is not necessary for the query), follow its friend relationship(s) and extract the fullName of whatever you find there. We don’t use GraphQL, but it wouldn’t be hard to translate:

query FriendsNamesQuery {
  user(id: $id) {
    friend {
      fullName
    }
  }
}

This is obviously a trivial and contrived example. It’s far more common for Find/Like queries to work with types:

$[?(@.type =~ 'User' && @.friend[?(@.fullName == 'Rick Osborne')])]

This translates to “find all Users, and anything that subclasses User, with a friend whose name is exactly ‘Rick Osborne'”. Again, maybe a little contrived, but you get the point.

Find/Like queries are where @list and expand really come together to shine: you can fetch a list of things and some additional context, while the @list unambiguously tells you which objects in your @context are the top-level results.

About Complexity

I’ve really come to love our Unified Endpoint setup, but I am aware that I’m in the minority at work. For our UI team, it’s not a slam dunk:

  • There’s an extra layer of abstraction between the response and the UI. Not only is the JSON response not in a feature-specific or REST-like format, it’s in a JSON-LD dialect that requires some processing before you work with it.
  • On the way back to the API, assembling a body for a POST, PUT, or PATCH request isn’t as simple as nesting objects and relying on JSON.stringify.
  • Caching is both harder and simpler. On the one hand you’re effectively giving up on browser-managed caching via Etag and other headers, due to the expand mechanism. On the other hand, since you must have that processing layer anyway, it’s relatively straightforward to build an app-managed object caching layer because of the uniformity of the responses.
  • As neat as field-atomic updates sound, it’s not as useful in practice as it could be. It’s useful for metadata where there aren’t any concurrency concerns or field entanglements, but for anything transactional it’s all but worthless.

Complexity hasn’t been eliminated, it’s been shifted: your API and your data going across it are now far more uniform and far easier to reason with, but you’ve traded that for translation layers on either side. One could make the arguments that we’ve just reimplemented AMF or Protobuf over a JSON-LD dialect, while reimplementing an RDF triplestore-like data schema. And those would be fair points.

In the end I’d argue that while Unified Endpoint isn’t perfect it has made significant leaps of the deficiencies of naive REST. REST was a great leap forward, but we need to keep up that momentum and figure out what we want next.

Published by

Rick Osborne

I am a web geek who has been doing this sort of thing entirely too long. I rant, I muse, I whine. That is, I am not at all atypical for my breed.