Blog Image

Personal API Experience

Daily musings

Sharing my day to day lessons, ideas and contemplations of architecting, developing and maintaining services (API) in the global marketplace we call the internet.

Chain of Trust

API Posted on Sun, February 24, 2019 14:00:54

In the
traditional two or three-tier world of web development, there is an inherent
trust between the client and the server. Each is contextually aware of the
other and withholds or shares information depending on the level of trust. The
server may generate and persist session information about the client to enforce
control or to simplify interaction. The client may do the same for a better
user experience. In the world of APIs, this contextual awareness is not
guaranteed since APIs can call or orchestrate other components where the context
of the client may be different or unknown.

API architecture
can be, and often is, layered. Since APIs offer abstraction via their
interfaces, the consumer need not know, nor care about, what goes on behind the
scenes of the interface. This also means that it should not matter to the consumer
what the API does, as long as it sticks to its interface contract. This abstraction
allows an API to do any orchestration, composition, translation or enrichment to
the original request; be that calling other APIs, interfaces, databases, systems,
etc. Basically, the consumer of the API may not know where their request is
actually being fulfilled, or how many other components/interfaces are being
consumed in the process.

On the
counter side of this, components being orchestrated or composed are abstracted
from the original request by the API consumer. They may not be contextually
aware of the overall intentions of the original consumer, and ideally, need not
be; they should only be concerned with responding to their specific request.

This potentially
disparate awareness between the original API consumer and the components being manipulated
by the API can be a problem for the traditional mindset. Some might struggle
with this lack of awareness and see it as a potential risk in security, authorization,
authentication or trust. This often becomes the excuse for not adhering to interface-based
architecture principles and the insistence in sticking to a two or three-tiered
architecture, even with APIs.

The trick
here is to implement and understand a chain of trust. Each conversation in an
end-to-end request should be based on an established trust between consumer and
producer (or requester and responder). The client making the original request should
have a trust relationship with the original API. The API, in turn, should have
an established trust relationship with each of the components it orchestrates
or consumes. And each of these components should have an established trust with
their consumed components.

This trust
and context can be established using a combination of patterns and approaches.
These could include:

· Mutual TLS (or similar) to restrict
boundaries to known systems,

· API Gateways to manage the consuming
systems

· JWT to manage user authentication
and authorization (using OAuth2, SAML, OIDC, etc.)

Plus, many
others.

Picture from Pexels.com



API Misconceptions

API Posted on Fri, February 01, 2019 08:12:00


I’d like to
clarify some things about the use of HTTP verbs for RESTful APIs and the
apparent misconceptions these can bring about.

Firstly, there
are a few people who believe that RESTful APIs are predominantly for CRUD (Create,
Read, Update, Delete) operations. I covered how this can be extended to other operations
in a previous article; Nouns and Verbs – in the world of APIs.

Secondly,
there’s also a large contingency of people who believe the use of HTTP verbs,
and their alignment to CRUD operations, implies that RESTful APIs are simply a
representation of an underlying database table – or JDBC over HTTP as I like to
call it.

In a
green-field situation, it’s quite possible to create your database structure
based on your API contract, but eventually, these will move apart. It’s far
more expensive to change a database structure than an API contract and you cannot
really version a database table change and still maintain a previous version
(unless you create a new table and inherit all the complexity in your code that
that creates). In any other situation, where your database is already defined,
exposing your database tables as APIs is exposing your data complexity directly
to your consumers – APIs should abstract complexity.

So, if the
HTTP verbs of a RESTful imply CRUD operations, but these may not directly
correlate with database tables; what does CRUD mean in this context? Quite
simple, it means CRUD operations at the API interface level – you still need to
implement the validation, transformation and mapping to wherever the data is
being processed; be that your underlying database or another API.

Photo from
Pexels.com



API Fragments – simple flexibility

API Posted on Sun, January 27, 2019 14:56:07

Designing
APIs often leads to a compromise between producers and/or consumers. Consumers
of APIs would like the interface to be as close to their requirements as
possible; they want to avoid possible composition or contract navigation. API
producers would like their interface to be as simple as possible; they want to
support maximum reuse and consumption.

I’ve
touched on this in a previous article, Should
APIs be Pre-made or Deli style?
, where I offered different options
for consumer flexibility or producer simplicity.

Pre-made. Defining API interfaces
based on specific consumer needs – simple for each consumer, but not
necessarily reusable.

Deli-style. Defining an API
interface that allows consumers to select the attributes they need via
arguments in the request – maximum flexibility for the consumer, but at a cost
for the producer.

Providing the entire model as one
interface
. Allowing the consumer to pick the elements they want from the
response and discard the rest – simple for the producer, requires contract
navigation for the consumer.

For this
discussion, let’s look at how we can make the third option a little more
flexible for the consumer while keeping it simple for the producer.

Assume this
is the entire model of a car:

It’s a fairly
simple model, but I’ll use it to explain the approach.

A producer
could expose the entire model via GET: /cars/1234abcd. This would force the
consumers to download the entire model and find the values they need. It’s the
simplest for the producer, but will probably get challenged by consumers if the
model is large.

An
alternative is to expose each sub-object as a separate URI – giving consumers
the basic information about the car and links to allow them to discover more about
the model through hyperlinks (HATEOUS).

This reduces
the payload of the response, and allows consumers to retrieve the additional
data they might need independently. From a producer’s perspective; each object
can be managed in isolation, it’s still simple and supports maximum reuse. For
the producer it offers flexibility and reduced payload.

However,
even this simple approach leads to another consumer argument: “What if I need
more than one object? I will have to make repeated calls to compose all the
data I need!
”. By example, assuming a consumer was interested in a car’s engine
and body, this would result in the consumer having to make three API calls:

GET: /cars/1234abcd
GET: /cars/1234abcd/engine
GET: /cars/1234abcd/interior

Is there another
alternative here?

What about
using the HTTP URL Fragment? By definition (RFC 3986): “The
fragment identifier component of a URI allows indirect identification of a
secondary resource …”. This is donated in a URL as the “#” (hash) symbol and
allows users to navigate to a specific portion of an HTML document.

We could
use the HTTP URL fragment to identify secondary objects within a model, and allow
the API consumer to define which objects in the model they are interested in. Since
we already have these objects exposed via end-points, we can offer a
short-circuit URI using the fragment, as follows:

GET:
/cars/1234abcd#engine,interior

The resource
in question is still the car, but the consumer is asking for additional sub-resources
of the car known by “engine” and “interior”.

Obviously,
only requesting one object via a fragment is redundant, but I believe this does
offer a lot more flexibility for the consumer without introducing too much complexity
for the producer.

Let me know
your thoughts.

Photo by Lukas from Pexels



API: Arrays or named elements?

API Posted on Tue, January 15, 2019 16:58:24

The humble
restaurant menu follows a common pattern where menu items are generally
separated into a logical hierarchy. At the top level, we may have breakfast
items, dinner items or drinks items. Each of these could be grouped further
into categories like meat dishes, fish dishes or pasta dishes (dinner items),
and again into burgers and steaks (meat dishes). Each item is then described so
that we (as humans) can make a choice on what to eat.

Now,
consider making that same menu available as an API (for machine consumption).
Think of the challenge that becomes immediately apparent; how do we describe
objects that contain similar properties? How should we model burgers for
example?

The natural
human tendency might be to describe each burger as an object with defined
properties. We may describe the “cheeseBurger”, the “baconBurger”, the
“hawaiianBurger” – this follows the menu approach (the intended business
purpose). Alternatively, we could define the “burgers” as an array and identify
each burger as an element with that array based on its property values.

There are a number of pros and cons to consider for each approach.

The “Named Object” approach

Pros

· The model is easy to understand (from a human perspective – developers are sometimes human)

· Consumers can navigate directly to the object needed; e.g. json.burgers.baconBurger

· Each object (burger) can evolve in isolation

· Each object can be managed via individual API end-points; e.g. http://api.sample.com/burgers/cheeseBurger

Cons

· Renaming or removing objects will lead to contract-breaking changes – forcing a new API version

· Filtering is usually done on attribute values. Doing so on objects is not intuitive.

· The object name provides identity within the interface schema. This name may need to be translated into an appropriate UI label and also makes localization challenging. The obvious solution would be to include a “label” attribute – but wouldn’t that be redundant?

The “Array” approach

Pros

· We can add, rename or remove objects (burgers) without effecting the interface model (no potential version changes).

· We can still retrieve a defined object via an API end-point; e.g. http://api.sample.com/burgers?type=Cheese%20Burger

Cons

· Consumers may need to loop through the array to find the required object by value.

· Objects have to maintain the same structure and cannot evolve in isolation.

The choice of which approach to take depends on the domain you are modeling. However, as the domain expert, the decision is yours – just understand the consequences of your choice.



DDD – what isle is your attribute in?

API Posted on Wed, January 09, 2019 18:41:59

During a Domain Driven Design (DDD) workshop this week, one of my colleagues very cleverly used grocery store aisles as an analogy to explain why attributes should be grouped into entities. He explained that grocery stores group logical produce together to make it easier for customers to find what they are looking for. Thereby suggesting that grouping attributes together in entities would help API consumers find relevant data. I really liked this analogy since it highlights some of the challenges of Domain Driven Design.

Have a look next time you go to your local grocery store. The teas, coffees and other hot drinks will be in one aisle. Condiments may be in another, so too will dry goods. Each isle can be seen as a group of items that logically go together. But, and there is always the exception, there are some items that don’t clearly fit into one category; like sugar for example. Should sugar be with the hot drinks, the dry goods or the baking goods? The stores around me can’t seem to agree, but at least they’ve each made a decision and mostly keep to it. As regular shoppers, we begin to know where to find our favorite items and understand the store’s layout.

DDD can be a very time-consuming and on-going exercise. Not everyone is going to agree with the models you define, but you can’t go on ad infinitum. Eventually, at some point, you have to make a decision. That decision will be based on the best available knowledge your team has at that point in time. Yes, it may not be 100% right, but it is a decision none-the-less.

Many of these decisions will be around which entities attributes logically belong to. Once all the decisions have been made, you are going to publish your model as an API. Consumers may initially be confused by the placement of one or more attributes and will either accept or challenge them.

Being challenged on a model is not a big deal. The beauty of APIs is that they can be versioned. This allows us a little flexibility in making a decision on our models. A change to the model (that breaks the current contract of your API) can be published as a new API version. That’s not to say we are free to change the model whenever the whim takes us, but, we don’t have to wait for the absolutely perfect model before releasing our APIs. We can start with a minimum viable product (MVP) and evolve it as our knowledge and experience in the domain grows.

The alternative is, of course, to not model your domain at all, and simply publish an API with a long list of root attributes. Imagine trying to shop in a store that randomly puts stuff on shelves without any thought…

Photo by Fancycrave.com from Pexels



Fireworks and APIs – “meh”

API Posted on Fri, January 04, 2019 14:34:58

Every New
Year’s Eve we are wowed by extravagant fireworks from around the world – we anticipate
them and we expect them. This New Year’s Eve was no different, except for me – this
year I was in Copenhagen city center; on the balcony of a friend’s apartment, watching.

In Denmark
anyone, over a certain age, can purchase and set off fireworks during New Year’s
Eve. It’s quite spectacular – bangs, fizzes, pops and smoke for a number of
hours; explosions are constant and everywhere.

Some of the
fireworks are spectacular and clearly cost a small fortune, others are unexceptional,
but the result of all of them is the same. Firstly, there’s great expectation
followed by excitement in lighting the fuse. Then there’s anticipation; waiting
for them to go off, tailed by wonder as the explosion and accompanying light
show is released. In the end, there’s the “meh” moment – either because it was
the same as the previous one, not as good as the next one, or a little lack-luster
when compared to others.

Finally, it’s
all done. The show is over, the money spent and everyone goes home.

Since this
is a new year, and we generally try to aspire to greater things, let’s try with
our APIs. Let’s not simply build excitement and anticipation only to release an
API that’s “meh” and soon forgotten. Let’s try to do them right this time.

Understand

Before you
begin, make sure you understand your business. Make sure you understand how your
API will be used in the context of your business. Make sure your API will be
used to grow or enhance your business. And finally, make sure the purpose of your
API is clearly understood.

Collaborate

Work with
your colleagues, your business, your partners and your consumers. Understand
their needs, work through their ideas, grasp their constraints – work with the requirements.
Agree the contractual interface (payload, URLs, operations, etc.) and simulate
them (using SwaggerHub or similar). Let everyone play and plan ahead with the simulations
– there is no need for consumers to wait for the final product. Get feedback
early.

Simple and reusable

Keep your
APIs simple and try make them useable by as many parties as possible (avoid single
consumer builds). Make them intuitive and keep them constrained to a single
functional responsibility. If you have to document your APIs too much, they are
probably too complicated. Mocking your APIs will help you understand if you’re
on the right path.

Loose Coupling

Don’t place
assumptions on consumers; don’t assume they understand the complexity of the system
behind your APIs – it’s not their responsibility. Your API interface should
abstract them from your complexity. After all, managing complexity is part of the service that
an API offers.

Manage

Finally,
manage you APIs through their lifecycle. Give them an explicit version (using Symantec
versioning) and let your consumers know when new versions are available. Of
course, if you are already collaborating with your consumers, they should be
ready for your API changes.

So, fireworks
offer all the excitement, anticipation and wonder; followed by the “meh” moment.
Your API’s can offer the same, but try not to aspire to that – try to maintain
the feel-good excitement and wonder by delivering something lasting and worthwhile
to all.



What event are you waiting for?

API Posted on Fri, December 14, 2018 15:58:22

In one of my
previous articles, the Hollywood Principle, I introduced the concept of an
event-driven architecture. This is a popular distributed asynchronous architecture
pattern used to produce highly scalable applications. It is underpinned by asynchronous
messaging and often implemented using the Publish and Subscribe (PubSub)
pattern. The premise of this pattern is that an event triggers messages for
distribution to multiple subscribers. However, in order to implement an
effective event-driven architecture, it is important to understand what an event
is in context.

Let’s consider
the first type of Event. Application developers, and more specifically UI
developers are all too familiar with concept of an event. A user clicking a
button or moving a mouse is considered an event. Each event has an associated
event handler to perform the specific action anticipated by the user. We’ll
call these UI events and exclude them from this discussion since they are generally
handled within the UI container.

Secondly, during
some form of business process, an artefact may be produced (a file for example)
which needs to be handed over to another system. The system creating the
artefact needs to let the other system know that that artefact is ready for
further processing. This can be considered an event. Likewise, a UI event
handler may need to trigger a process in another system. These can also be
considered events. We’ll call these process events.

Finally, we
have the scenario where a business domain or entity is updated or created, via some
process or another. The update of this entity can be considered an event since this
update may be relevant or of importance to other systems. A good example of
these could include a customer update, a payment or an account update. We’ll
call these business events.

Process
events nearly always involve only two parties; the producer of the event and
the consumer of the event message. The producer of the event needs to be aware
of the consumer of the event message. To support asynchronous behavior these
events should be transported via standard messaging. Using PubSub is overkill
in this context and should not be advocated as it clutters the PubSub topic range.

Business
events are the ideal contenders for using PubSub since these are business
activities that others may care about.

Let’s walk
through an example – an online payment.

Using the
UI channel, the user clicks the submit payment button. This “click” (UI event)
might be handled by a function that validates all the required fields are populated.
All these activities are encapsulated with the UI. Assuming successful
validation, the payment request might be submitted to a payments engine asynchronously
(a process event) since the payment may only be cleared later.

Within the
payments engine, the payment request is processed. This process might involve using
several external systems, including funds check, AML and fraud checks. Each of
these systems might raise an event that other systems might care about (analytics,
alerts, etc.). These systems might publish these events via PubSub using a
topic relevant to the event: “overdrawn”, suspect persons” and “suspicious behavior”.
Finally, the payment is cleared – this will trigger the final event – and publish
the transaction via PubSub for other systems to react to (account balance update,
balance SMS, analytics, etc.).

Reacting to
events can be a very powerful architectural pattern – it promotes real-time responses
and reduces “overnight” processing. But, it is important to know which messaging
pattern to implement.

Photo by
rawpixel.com from Pexels



Nouns and Verbs – in the world of APIs

API Posted on Fri, December 07, 2018 16:13:30

Please note that this is a little more
technical than my previous articles – sometimes I need to let my inner-geek out.

Simply
because RESTful APIs are based on resources and use the HTTP verbs (GET, POST,
PUT, DELETE, PATCH), does not mean they should only support CRUD (Create, Read,
Update, Delete) operations. RESTful APIs can also be used for performing other actions
on resources.

I was
challenge with this recently when explaining the concepts of a layered API
architecture (see my previous
article
for more information) during a Domain Driven Design
workshop. Core APIs, which expose core domains objects, support pure CRUD
operations and make full use of the HTTP verbs. These are well within the perceived
CRUD definitions of RESTful services. Process and experience APIs, which
orchestrate or compose core APIs to create a defined process, tend to move away
from this understanding; and are sometimes a cause for confusion. What HTTP
verb should be used for an operation outside of the CRUD set and how should we graft
the URL?

Let’s start
with the basics. The naming convention for resources in RESTful APIs should be
noun based; we work with “accounts”, “customers”, “products”, etc. The basic
CRUD operations we perform on these resources are defined by the HTTP verbs.
So, for example:

GET http://www.api.com/customers
– gets a list of customers

POST http://www.api.com/customers
– creates a new customer based on the representation in the body of the request

PUT http://www.api.com/customers/123
– replaces the customer identified by “123” with the representation in the body
of the request

PATCH http://www.api.com/customers/123 – updates
the customer identified by “123” with the representation in the body of the
request

DELETE http://www.api.com/customers/123
– deletes a customer identified by “123”.

The PUT, PATCH
and DELETE HTTP verbs imply very specific Update and Delete actions in the
realm of HTTP; they will not work for activity-based APIs. A GET request, only consists
of the URL to fulfill the HTTP request; you cannot provide content in the
request body (we’ll ignore the use of HTTP Headers for now – they serve a
completely different purpose). A POST request allows you to provide content
within the request body, so is best placed for an activity-based API.

Now that we’ve
identified that the HTTP POST verb is the appropriate choice, how then do we graft
the URL to identify what activity (“action”) we want to perform on the resource?

A method
driven approach to URL design is often used: http://www.api.com/authenticateCustomer
or http://www.api.com/validatePaymentRequest.
However, while this seems to make sense,
in my view, it moves away from the resource orientation of RESTful URLS. You may
struggle to consider one resource in isolation, but rather have to be aware of
multiple related or interacting resources and actions. This can lead to a plethora
of inconsistent URL patterns and approaches – making it difficult for developers
to learn how to use your APIs.

Maintaining
a resource-based URL representation is surprisingly simple. I use a verb-based terms
to identify the action I want to perform on the resource in question. The URL
is grafted by appending the verb to the resource-based URL. Using the POST
method, I can provide the details required for the activity in the request
body.

For example,
consider: POST http://www.api.com/customers/123/authenticate – “authenticate” is the action I want to
execute on the customer identified by “123”. The request body contains a
representation of the customer required for the authentication.

The
important principle to be aware of here is that resources MUST be noun-based
(describe the resource) and actions MUST be verb-based (describe the action).

By
definition and for clarity, REST (Representational State Transfer) is a
software architecture style that defines a set of constraints to be used when
creating web services. These constraints include:

1. A client-server architecture which
supports the separation of concerns principle – allowing components to evolve
independently.

2. A stateless client-server
communication – where each request from the client contains all the data
required to service the request. No client context is stored on the server.

3. Responses must, implicitly or
explicitly, define themselves as cacheable or not to prevent clients from
getting stale or inappropriate data in response to further requests.

4. A layered system where the client
cannot ordinarily tell whether it is connected directly to the end server, or
to an intermediary along the way.

5. A uniform interface is fundamental
to the design of any RESTful system. It simplifies and decouples the
architecture, which enables each part to evolve independently.

Being resource
based is inherent in RESTful APIs and the standard set of HTTP verbs tend to imply
CRUD operations only; so, I do understand were the misconception comes from.



Next »