Request Flow
These diagrams summarise the flow of a user request from a client, executed on a CCF node. Each shows execution of a POST /copy/A/B
endpoint which does a KV read from the key specified in A
, and writes the obtained value back to the KV at key B
.
These show the progression from parsing through several layers of dispatch, in code that is part of the framework, down to the app code which is under the control of application developers.
Note
This page only discusses execution, and does not show how execution results are then replicated to reach consensus.
Normal flow
This is the simple, usual flow, where the request is submitted to a primary node capable of writing to the KV. Only the receiving node interacts with the request. The entire execution is synchronous, writing the response back to the client before proceeding with any other work.
sequenceDiagram
participant User
participant NetStack
participant Frontend
participant App
participant KV
User->>NetStack: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStack,KV: Inside single CCF node
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>+Frontend: Frontend Dispatch
note left of Frontend: Tx is created here
Frontend->>Frontend: is_open(tx)
Frontend->>App: find_endpoint(tx, ctx)
App->>App: h = find_handler_for(tx, ctx)
App-->>Frontend: return h
Frontend->>Frontend: get_authenticated_identity(tx, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx, ctx, h)
App->>KV: tx.get(A)
KV-->>App: return a
App->>KV: tx.put(B, a)
KV-->>App: return
App->>App: ctx.set_claims_digest(...)
App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}")
App-->>Frontend: return
Frontend->>Frontend: tx.commit()
Frontend->>-Frontend: response.set_header(TX_HEADER, tx.commit_id())
note left of Frontend: Tx is destroyed here
Frontend-->>NetStack: return
NetStack->>NetStack: HTTP serialise response
NetStack->>NetStack: TLS encrypt response
end
NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"
Forwarding flow
When write request are submitted to a follower node, they must be forwarded to the primary for execution. This diagram shows how that is done, between a follower node A and a primary B. Decryption and some dispatch still occurs on the follower, as it must lookup the correct endpoint’s metadata to determine whether this request should be forwarded. When A establishes that the request should be forwarded, it queues a node-to-node (N2N) forwarding message to the primary describing the original request. The synchronous execution the follower A now completes without writing any response to the user, but maintaining an open TLS session and some local state that a response is pending.
When the primary B receives the forwarded command, it executes the same dispatch and execution that it would if it had directly received the request, but with a different stack at the top level. Specifically, it will eventually write its response back over the encrypted node-to-node channel to A, rather than the original caller.
When follower A receives the forwarded response, it writes this to the TLS session that was maintained earlier, and marks the pending response as completed.
sequenceDiagram
participant User
participant NetStackA
participant FrontendA
participant N2NA
participant N2NB
participant FrontendB
participant App
participant KV
User->>NetStackA: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStackA,N2NA: Inside CCF node A
NetStackA->>NetStackA: TLS decrypt request
NetStackA->>NetStackA: HTTP parse request
NetStackA->>+FrontendA: Frontend Dispatch
note left of FrontendA: Tx is created here
FrontendA->>FrontendA: is_open(tx)
FrontendA->>FrontendA: find_endpoint(tx, ctx)
FrontendA->>FrontendA: get_authenticated_identity(tx, ctx)
FrontendA->>-FrontendA: forward?
FrontendA->>N2NA: forward()
N2NA->>N2NA: Queue forwarded msg
N2NA-->>FrontendA: return
FrontendA->>FrontendA: ctx.pending_response = true
note left of FrontendA: Tx is destroyed here
FrontendA-->>NetStackA: return
end
N2NA->>N2NB: forwarded_cmd
rect rgba(191, 223, 255, 0.5)
note over N2NB,KV: Inside CCF node B
N2NB->>N2NB: N2N parse
N2NB->>+FrontendB: Frontend Dispatch
note left of FrontendB: Tx is created here
FrontendB->>FrontendB: is_open(tx)
FrontendB->>App: find_endpoint(tx, ctx)
App->>App: h = find_handler_for(tx, ctx)
App-->>FrontendB: return h
FrontendB->>FrontendB: get_authenticated_identity(tx, ctx)
FrontendB->>FrontendB: forward?
FrontendB->>App: execute_endpoint(tx, ctx, h)
App->>KV: tx.get(A)
KV-->>App: return a
App->>KV: tx.put(B, a)
KV-->>App: return
App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}")
App-->>FrontendB: return
FrontendB->>FrontendB: tx.commit()
FrontendB->>-FrontendB: response.set_header(TX_HEADER, tx.commit_id())
FrontendB-->>N2NB: return
note left of FrontendB: Tx is destroyed here
N2NB->>N2NB: HTTP serialise response
end
N2NB-->>N2NA: forwarded_response
N2NA->>N2NA: N2N Parse
N2NA->>NetStackA: reply_async(session, response)
NetStackA->>NetStackA: TLS encrypt response
NetStackA-->>User: 200 OK "Copied {a} from {A} to {B}"
Redirection flow
CCF supports HTTP redirections as an alternative to forwarding. When a request arrives that cannot be executed locally, rather than forwarding it to an appropriate node over the node-to-node channels, the node can return a HTTP redirect response advising the caller to resubmit the request directly to that node. This uses standard HTTP semantics, reporting the redirect target in a Location
header. Most HTTP clients will have an option to follow this redirect automatically, and all should have an option to enable this behaviour if desired. Alternatively, client applications may choose to intercept this redirect response and manually interpret it, perhaps to alter the resubmitted request or to update the target node for future requests.
Warning
Many HTTP clients will strip out Authorization
headers when following Cross-Origin redirects. This means that if your client is automatically following redirects, and you submit a request with a JWT token as authorization, if you are redirected you may see a surprising authorization failure. In this scenario we recommend intercepting the redirect responses manually, so that the request can be resubmitted without stripping headers.
Similar to forwarding, the redirect behaviour is partly controlled by per-endpoint metadata, so the initially receiving node must parse the request and go through endpoint dispatch before making a forwarding decision.
There are currently 2 supported modes for redirections. In the first, the response sends the user directly to the suggested node. This will only work if that node has an accessible name, which can be included in the Location
header and accessed by the user.
sequenceDiagram
autonumber
participant U as User
participant B as Backup (nodeA.ccf.com)
participant P as Primary (nodeB.ccf.com)
U->>B: POST /copy/A/B
B->>B: Lookup endpoint
B->>B: Decide request should be redirected
B->>B: Build redirect response
B-->>U: 307 REDIRECT Location: nodeB.ccf.com/copy/A/B
U->>P: POST /copy/A/B
P->>P: Lookup endpoint
P->>P: Decide request can be executed
P->>P: Execute request
P-->>U: 200 OK "Copied {a} from {A} to {B}"
For deployments where nodes are not directly accessible, redirections can still be supported via multiple load balancers. All that is required is a public name for each redirect purpose, with up-to-date balancing to the correct nodes. More simply, that currently means maintaining a write load balancer which can direct external traffic to a primary.
sequenceDiagram
autonumber
participant U as User
participant LB as General LB (service.ccf.com)
participant B as Backup
participant WLB as Write LB (write.service.ccf.com)
participant P as Primary
U->>LB: POST /copy/A/B
LB->>B: POST /copy/A/B
B->>B: Lookup endpoint
B->>B: Decide request should be redirected
B->>B: Build redirect response
B-->>U: 307 REDIRECT Location: write.service.ccf.com/copy/A/B
U->>WLB: POST /copy/A/B
WLB->>P: POST /copy/A/B
P->>P: Lookup endpoint
P->>P: Decide request can be executed
P->>P: Execute request
P-->>U: 200 OK "Copied {a} from {A} to {B}"
To use redirection behaviour, and choose whether to redirect to a node or a load balancer, set the redirections
field in the cchost launch configuration.
External executor flow
This shows the flow for the in-development external executor app, where implementation of the user endpoints is offloaded to an external trusted executor. This is achieved by providing a remote KV API over which the executor can invoke actions of the local KV, using a persistent Tx
object shared between multiple requests.
The result is that the user’s interaction is unchanged - they send a HTTPS request to a single CCF node and get the same format of response, but the app logic can be decoupled from the CCF enclave.
Note
Some steps are elided/abbreviated for clarity. This diagram does not show the registration of executors.
sequenceDiagram
participant User
participant Executor
participant NetStack
participant Frontend
participant App
participant KV
User->>NetStack: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: Inside single CCF node
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>Frontend: Frontend Dispatch
activate Frontend
note left of Frontend: tx1 is created here
Frontend->>Frontend: is_open(tx1)
Frontend->>App: find_endpoint(tx1, ctx)
App->>App: e = find_executor_for(ctx)
App-->>Frontend: return e
Frontend->>Frontend: get_authenticated_identity(tx1, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx1, ctx, e)
note over Frontend,App: tx1 is stolen here
deactivate Frontend
activate App
App->>App: pending_reqs[e].append(tx1, ctx)
App->>App: ctx.pending_response = true
App-->>Frontend: return
Frontend-->>NetStack: return
end
Executor->>NetStack: POST /StartTx
rect rgba(191, 223, 255, 0.5)
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>Frontend: Frontend Dispatch
activate Frontend
Frontend->>Frontend: is_open(tx2)
Frontend->>App: find_endpoint(tx2, ctx)
App->>App: h = find_handler_for(tx2, ctx)
App-->>Frontend: return h
Frontend->>Frontend: get_authenticated_identity(tx2, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx2, ctx, h)
App->>App: active_reqs[e] = pending_reqs.pop(e)
App->>App: ctx.set_response(OK, describe_request(active_reqs[e]))
App-->>Frontend: return
Frontend->>Frontend: tx.commit()
Frontend->>Frontend: response.set_header(TX_HEADER, tx.commit_id())
Frontend-->>NetStack: return
deactivate Frontend
NetStack->>NetStack: HTTP serialise response
NetStack->>NetStack: TLS encrypt response
end
NetStack-->>Executor: 200 OK {RequestDescription}
activate Executor
Executor->>Executor: Process RequestDescription
Executor->>NetStack: POST /KV.Get {key=A}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx3, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>KV: tx1.get(A)
KV-->>App: return a
App->>App: ctx.set_response(OK, {value=a})
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK {value=a}
Executor->>NetStack: POST /KV.Put {key=B, value=a}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx4, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>KV: tx1.put(B, a)
KV-->>App: return
App->>App: ctx.set_response(OK)
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK
Executor->>NetStack: POST /EndTx {code=200, body="Copied {a} from {A} to {B}"}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx5, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>App: result = tx1.commit()
App->>App: response = (result, code, body)
App->>App: HTTP serialise response
App->>NetStack: reply_async(active_reqs[e].ctx.session, response)
App->>App: ctx.set_response(OK)
App->>App: active_reqs.pop(e)
note over App: tx1 is destroyed here
deactivate App
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK
deactivate Executor
NetStack->>NetStack: TLS encrypt response
NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"