Request Flow

These diagrams summarise the flow of a user request from a client, executed on a CCF node. Each shows execution of a POST /copy/A/B endpoint which does a KV read from the key specified in A, and writes the obtained value back to the KV at key B.

These show the progression from parsing through several layers of dispatch, in code that is part of the framework, down to the app code which is under the control of application developers.

Note

This page only discusses execution, and does not show how execution results are then replicated to reach consensus.

Normal flow

This is the simple, usual flow, where the request is submitted to a primary node capable of writing to the KV. Only the receiving node interacts with the request. The entire execution is synchronous, writing the response back to the client before proceeding with any other work.

sequenceDiagram participant User participant NetStack participant Frontend participant App participant KV User->>NetStack: POST /copy/A/B rect rgba(191, 223, 255, 0.5) note over NetStack,KV: Inside single CCF node NetStack->>NetStack: TLS decrypt request NetStack->>NetStack: HTTP parse request NetStack->>+Frontend: Frontend Dispatch note left of Frontend: Tx is created here Frontend->>Frontend: is_open(tx) Frontend->>App: find_endpoint(tx, ctx) App->>App: h = find_handler_for(tx, ctx) App-->>Frontend: return h Frontend->>Frontend: get_authenticated_identity(tx, ctx) Frontend->>Frontend: forward? Frontend->>App: execute_endpoint(tx, ctx, h) App->>KV: tx.get(A) KV-->>App: return a App->>KV: tx.put(B, a) KV-->>App: return App->>App: ctx.set_claims_digest(...) App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}") App-->>Frontend: return Frontend->>Frontend: tx.commit() Frontend->>-Frontend: response.set_header(TX_HEADER, tx.commit_id()) note left of Frontend: Tx is destroyed here Frontend-->>NetStack: return NetStack->>NetStack: HTTP serialise response NetStack->>NetStack: TLS encrypt response end NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"

Forwarding flow

When write request are submitted to a follower node, they must be forwarded to the primary for execution. This diagram shows how that is done, between a follower node A and a primary B. Decryption and some dispatch still occurs on the follower, as it must lookup the correct endpoint’s metadata to determine whether this request should be forwarded. When A establishes that the request should be forwarded, it queues a node-to-node (N2N) forwarding message to the primary describing the original request. The synchronous execution the follower A now completes without writing any response to the user, but maintaining an open TLS session and some local state that a response is pending.

When the primary B receives the forwarded command, it executes the same dispatch and execution that it would if it had directly received the request, but with a different stack at the top level. Specifically, it will eventually write its response back over the encrypted node-to-node channel to A, rather than the original caller.

When follower A receives the forwarded response, it writes this to the TLS session that was maintained earlier, and marks the pending response as completed.

sequenceDiagram participant User participant NetStackA participant FrontendA participant N2NA participant N2NB participant FrontendB participant App participant KV User->>NetStackA: POST /copy/A/B rect rgba(191, 223, 255, 0.5) note over NetStackA,N2NA: Inside CCF node A NetStackA->>NetStackA: TLS decrypt request NetStackA->>NetStackA: HTTP parse request NetStackA->>+FrontendA: Frontend Dispatch note left of FrontendA: Tx is created here FrontendA->>FrontendA: is_open(tx) FrontendA->>FrontendA: find_endpoint(tx, ctx) FrontendA->>FrontendA: get_authenticated_identity(tx, ctx) FrontendA->>-FrontendA: forward? FrontendA->>N2NA: forward() N2NA->>N2NA: Queue forwarded msg N2NA-->>FrontendA: return FrontendA->>FrontendA: ctx.pending_response = true note left of FrontendA: Tx is destroyed here FrontendA-->>NetStackA: return end N2NA->>N2NB: forwarded_cmd rect rgba(191, 223, 255, 0.5) note over N2NB,KV: Inside CCF node B N2NB->>N2NB: N2N parse N2NB->>+FrontendB: Frontend Dispatch note left of FrontendB: Tx is created here FrontendB->>FrontendB: is_open(tx) FrontendB->>App: find_endpoint(tx, ctx) App->>App: h = find_handler_for(tx, ctx) App-->>FrontendB: return h FrontendB->>FrontendB: get_authenticated_identity(tx, ctx) FrontendB->>FrontendB: forward? FrontendB->>App: execute_endpoint(tx, ctx, h) App->>KV: tx.get(A) KV-->>App: return a App->>KV: tx.put(B, a) KV-->>App: return App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}") App-->>FrontendB: return FrontendB->>FrontendB: tx.commit() FrontendB->>-FrontendB: response.set_header(TX_HEADER, tx.commit_id()) FrontendB-->>N2NB: return note left of FrontendB: Tx is destroyed here N2NB->>N2NB: HTTP serialise response end N2NB-->>N2NA: forwarded_response N2NA->>N2NA: N2N Parse N2NA->>NetStackA: reply_async(session, response) NetStackA->>NetStackA: TLS encrypt response NetStackA-->>User: 200 OK "Copied {a} from {A} to {B}"

Redirection flow

CCF supports HTTP redirections as an alternative to forwarding. When a request arrives that cannot be executed locally, rather than forwarding it to an appropriate node over the node-to-node channels, the node can return a HTTP redirect response advising the caller to resubmit the request directly to that node. This uses standard HTTP semantics, reporting the redirect target in a Location header. Most HTTP clients will have an option to follow this redirect automatically, and all should have an option to enable this behaviour if desired. Alternatively, client applications may choose to intercept this redirect response and manually interpret it, perhaps to alter the resubmitted request or to update the target node for future requests.

Warning

Many HTTP clients will strip out Authorization headers when following Cross-Origin redirects. This means that if your client is automatically following redirects, and you submit a request with a JWT token as authorization, if you are redirected you may see a surprising authorization failure. In this scenario we recommend intercepting the redirect responses manually, so that the request can be resubmitted without stripping headers.

Similar to forwarding, the redirect behaviour is partly controlled by per-endpoint metadata, so the initially receiving node must parse the request and go through endpoint dispatch before making a forwarding decision.

There are currently 2 supported modes for redirections. In the first, the response sends the user directly to the suggested node. This will only work if that node has an accessible name, which can be included in the Location header and accessed by the user.

sequenceDiagram autonumber participant U as User participant B as Backup (nodeA.ccf.com) participant P as Primary (nodeB.ccf.com) U->>B: POST /copy/A/B B->>B: Lookup endpoint B->>B: Decide request should be redirected B->>B: Build redirect response B-->>U: 307 REDIRECT Location: nodeB.ccf.com/copy/A/B U->>P: POST /copy/A/B P->>P: Lookup endpoint P->>P: Decide request can be executed P->>P: Execute request P-->>U: 200 OK "Copied {a} from {A} to {B}"

For deployments where nodes are not directly accessible, redirections can still be supported via multiple load balancers. All that is required is a public name for each redirect purpose, with up-to-date balancing to the correct nodes. More simply, that currently means maintaining a write load balancer which can direct external traffic to a primary.

sequenceDiagram autonumber participant U as User participant LB as General LB (service.ccf.com) participant B as Backup participant WLB as Write LB (write.service.ccf.com) participant P as Primary U->>LB: POST /copy/A/B LB->>B: POST /copy/A/B B->>B: Lookup endpoint B->>B: Decide request should be redirected B->>B: Build redirect response B-->>U: 307 REDIRECT Location: write.service.ccf.com/copy/A/B U->>WLB: POST /copy/A/B WLB->>P: POST /copy/A/B P->>P: Lookup endpoint P->>P: Decide request can be executed P->>P: Execute request P-->>U: 200 OK "Copied {a} from {A} to {B}"

To use redirection behaviour, and choose whether to redirect to a node or a load balancer, set the redirections field in the cchost launch configuration.

External executor flow

This shows the flow for the in-development external executor app, where implementation of the user endpoints is offloaded to an external trusted executor. This is achieved by providing a remote KV API over which the executor can invoke actions of the local KV, using a persistent Tx object shared between multiple requests.

The result is that the user’s interaction is unchanged - they send a HTTPS request to a single CCF node and get the same format of response, but the app logic can be decoupled from the CCF enclave.

Note

Some steps are elided/abbreviated for clarity. This diagram does not show the registration of executors.

sequenceDiagram participant User participant Executor participant NetStack participant Frontend participant App participant KV User->>NetStack: POST /copy/A/B rect rgba(191, 223, 255, 0.5) note over NetStack,App: Inside single CCF node NetStack->>NetStack: TLS decrypt request NetStack->>NetStack: HTTP parse request NetStack->>Frontend: Frontend Dispatch activate Frontend note left of Frontend: tx1 is created here Frontend->>Frontend: is_open(tx1) Frontend->>App: find_endpoint(tx1, ctx) App->>App: e = find_executor_for(ctx) App-->>Frontend: return e Frontend->>Frontend: get_authenticated_identity(tx1, ctx) Frontend->>Frontend: forward? Frontend->>App: execute_endpoint(tx1, ctx, e) note over Frontend,App: tx1 is stolen here deactivate Frontend activate App App->>App: pending_reqs[e].append(tx1, ctx) App->>App: ctx.pending_response = true App-->>Frontend: return Frontend-->>NetStack: return end Executor->>NetStack: POST /StartTx rect rgba(191, 223, 255, 0.5) NetStack->>NetStack: TLS decrypt request NetStack->>NetStack: HTTP parse request NetStack->>Frontend: Frontend Dispatch activate Frontend Frontend->>Frontend: is_open(tx2) Frontend->>App: find_endpoint(tx2, ctx) App->>App: h = find_handler_for(tx2, ctx) App-->>Frontend: return h Frontend->>Frontend: get_authenticated_identity(tx2, ctx) Frontend->>Frontend: forward? Frontend->>App: execute_endpoint(tx2, ctx, h) App->>App: active_reqs[e] = pending_reqs.pop(e) App->>App: ctx.set_response(OK, describe_request(active_reqs[e])) App-->>Frontend: return Frontend->>Frontend: tx.commit() Frontend->>Frontend: response.set_header(TX_HEADER, tx.commit_id()) Frontend-->>NetStack: return deactivate Frontend NetStack->>NetStack: HTTP serialise response NetStack->>NetStack: TLS encrypt response end NetStack-->>Executor: 200 OK {RequestDescription} activate Executor Executor->>Executor: Process RequestDescription Executor->>NetStack: POST /KV.Get {key=A} rect rgba(191, 223, 255, 0.5) note over NetStack,App: ... Frontend->>App: execute_endpoint(tx3, ctx, h) App->>App: tx = active_reqs[e].tx Note right of App: // Gets tx1 App->>KV: tx1.get(A) KV-->>App: return a App->>App: ctx.set_response(OK, {value=a}) App-->>Frontend: return note over NetStack,Frontend: ... end NetStack-->>Executor: 200 OK {value=a} Executor->>NetStack: POST /KV.Put {key=B, value=a} rect rgba(191, 223, 255, 0.5) note over NetStack,App: ... Frontend->>App: execute_endpoint(tx4, ctx, h) App->>App: tx = active_reqs[e].tx Note right of App: // Gets tx1 App->>KV: tx1.put(B, a) KV-->>App: return App->>App: ctx.set_response(OK) App-->>Frontend: return note over NetStack,Frontend: ... end NetStack-->>Executor: 200 OK Executor->>NetStack: POST /EndTx {code=200, body="Copied {a} from {A} to {B}"} rect rgba(191, 223, 255, 0.5) note over NetStack,App: ... Frontend->>App: execute_endpoint(tx5, ctx, h) App->>App: tx = active_reqs[e].tx Note right of App: // Gets tx1 App->>App: result = tx1.commit() App->>App: response = (result, code, body) App->>App: HTTP serialise response App->>NetStack: reply_async(active_reqs[e].ctx.session, response) App->>App: ctx.set_response(OK) App->>App: active_reqs.pop(e) note over App: tx1 is destroyed here deactivate App App-->>Frontend: return note over NetStack,Frontend: ... end NetStack-->>Executor: 200 OK deactivate Executor NetStack->>NetStack: TLS encrypt response NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"