Request Flow
These diagrams summarise the flow of a user request from a client, executed on a CCF node. Each shows execution of a POST /copy/A/B
endpoint which does a KV read from the key specified in A
, and writes the obtained value back to the KV at key B
.
These show the progression from parsing through several layers of dispatch, in code that is part of the framework, down to the app code which is under the control of application developers.
Note
This page only discusses execution, and does not show how execution results are then replicated to reach consensus.
Normal flow
This is the simple, usual flow, where the request is submitted to a primary node capable of writing to the KV. Only the receiving node interacts with the request. The entire execution is synchronous, writing the response back to the client before proceeding with any other work.
sequenceDiagram
participant User
participant NetStack
participant Frontend
participant App
participant KV
User->>NetStack: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStack,KV: Inside single CCF node
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>+Frontend: Frontend Dispatch
note left of Frontend: Tx is created here
Frontend->>Frontend: is_open(tx)
Frontend->>App: find_endpoint(tx, ctx)
App->>App: h = find_handler_for(tx, ctx)
App-->>Frontend: return h
Frontend->>Frontend: get_authenticated_identity(tx, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx, ctx, h)
App->>KV: tx.get(A)
KV-->>App: return a
App->>KV: tx.put(B, a)
KV-->>App: return
App->>App: ctx.set_claims_digest(...)
App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}")
App-->>Frontend: return
Frontend->>Frontend: tx.commit()
Frontend->>-Frontend: response.set_header(TX_HEADER, tx.commit_id())
note left of Frontend: Tx is destroyed here
Frontend-->>NetStack: return
NetStack->>NetStack: HTTP serialise response
NetStack->>NetStack: TLS encrypt response
end
NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"
Forwarding flow
When write request are submitted to a follower node, they must be forwarded to the primary for execution. This diagram shows how that is done, between a follower node A and a primary B. Decryption and some dispatch still occurs on the follower, as it must lookup the correct endpoint’s metadata to determine whether this request should be forwarded. When A establishes that the request should be forwarded, it queues a node-to-node (N2N) forwarding message to the primary describing the original request. The synchronous execution the follower A now completes without writing any response to the user, but maintaining an open TLS session and some local state that a response is pending.
When the primary B receives the forwarded command, it executes the same dispatch and execution that it would if it had directly received the request, but with a different stack at the top level. Specifically, it will eventually write its response back over the encrypted node-to-node channel to A, rather than the original caller.
When follower A receives the forwarded response, it writes this to the TLS session that was maintained earlier, and marks the pending response as completed.
sequenceDiagram
participant User
participant NetStackA
participant FrontendA
participant N2NA
participant N2NB
participant FrontendB
participant App
participant KV
User->>NetStackA: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStackA,N2NA: Inside CCF node A
NetStackA->>NetStackA: TLS decrypt request
NetStackA->>NetStackA: HTTP parse request
NetStackA->>+FrontendA: Frontend Dispatch
note left of FrontendA: Tx is created here
FrontendA->>FrontendA: is_open(tx)
FrontendA->>FrontendA: find_endpoint(tx, ctx)
FrontendA->>FrontendA: get_authenticated_identity(tx, ctx)
FrontendA->>-FrontendA: forward?
FrontendA->>N2NA: forward()
N2NA->>N2NA: Queue forwarded msg
N2NA-->>FrontendA: return
FrontendA->>FrontendA: ctx.pending_response = true
note left of FrontendA: Tx is destroyed here
FrontendA-->>NetStackA: return
end
N2NA->>N2NB: forwarded_cmd
rect rgba(191, 223, 255, 0.5)
note over N2NB,KV: Inside CCF node B
N2NB->>N2NB: N2N parse
N2NB->>+FrontendB: Frontend Dispatch
note left of FrontendB: Tx is created here
FrontendB->>FrontendB: is_open(tx)
FrontendB->>App: find_endpoint(tx, ctx)
App->>App: h = find_handler_for(tx, ctx)
App-->>FrontendB: return h
FrontendB->>FrontendB: get_authenticated_identity(tx, ctx)
FrontendB->>FrontendB: forward?
FrontendB->>App: execute_endpoint(tx, ctx, h)
App->>KV: tx.get(A)
KV-->>App: return a
App->>KV: tx.put(B, a)
KV-->>App: return
App->>App: ctx.set_response(OK, "Copied {a} from {A} to {B}")
App-->>FrontendB: return
FrontendB->>FrontendB: tx.commit()
FrontendB->>-FrontendB: response.set_header(TX_HEADER, tx.commit_id())
FrontendB-->>N2NB: return
note left of FrontendB: Tx is destroyed here
N2NB->>N2NB: HTTP serialise response
end
N2NB-->>N2NA: forwarded_response
N2NA->>N2NA: N2N Parse
N2NA->>NetStackA: reply_async(session, response)
NetStackA->>NetStackA: TLS encrypt response
NetStackA-->>User: 200 OK "Copied {a} from {A} to {B}"
External executor flow
This shows the flow for the in-development external executor app, where implementation of the user endpoints is offloaded to an external trusted executor. This is achieved by providing a remote KV API over which the executor can invoke actions of the local KV, using a persistent Tx
object shared between multiple requests.
The result is that the user’s interaction is unchanged - they send a HTTPS request to a single CCF node and get the same format of response, but the app logic can be decoupled from the CCF enclave.
Note
Some steps are elided/abbreviated for clarity. This diagram does not show the registration of executors.
sequenceDiagram
participant User
participant Executor
participant NetStack
participant Frontend
participant App
participant KV
User->>NetStack: POST /copy/A/B
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: Inside single CCF node
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>Frontend: Frontend Dispatch
activate Frontend
note left of Frontend: tx1 is created here
Frontend->>Frontend: is_open(tx1)
Frontend->>App: find_endpoint(tx1, ctx)
App->>App: e = find_executor_for(ctx)
App-->>Frontend: return e
Frontend->>Frontend: get_authenticated_identity(tx1, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx1, ctx, e)
note over Frontend,App: tx1 is stolen here
deactivate Frontend
activate App
App->>App: pending_reqs[e].append(tx1, ctx)
App->>App: ctx.pending_response = true
App-->>Frontend: return
Frontend-->>NetStack: return
end
Executor->>NetStack: POST /StartTx
rect rgba(191, 223, 255, 0.5)
NetStack->>NetStack: TLS decrypt request
NetStack->>NetStack: HTTP parse request
NetStack->>Frontend: Frontend Dispatch
activate Frontend
Frontend->>Frontend: is_open(tx2)
Frontend->>App: find_endpoint(tx2, ctx)
App->>App: h = find_handler_for(tx2, ctx)
App-->>Frontend: return h
Frontend->>Frontend: get_authenticated_identity(tx2, ctx)
Frontend->>Frontend: forward?
Frontend->>App: execute_endpoint(tx2, ctx, h)
App->>App: active_reqs[e] = pending_reqs.pop(e)
App->>App: ctx.set_response(OK, describe_request(active_reqs[e]))
App-->>Frontend: return
Frontend->>Frontend: tx.commit()
Frontend->>Frontend: response.set_header(TX_HEADER, tx.commit_id())
Frontend-->>NetStack: return
deactivate Frontend
NetStack->>NetStack: HTTP serialise response
NetStack->>NetStack: TLS encrypt response
end
NetStack-->>Executor: 200 OK {RequestDescription}
activate Executor
Executor->>Executor: Process RequestDescription
Executor->>NetStack: POST /KV.Get {key=A}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx3, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>KV: tx1.get(A)
KV-->>App: return a
App->>App: ctx.set_response(OK, {value=a})
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK {value=a}
Executor->>NetStack: POST /KV.Put {key=B, value=a}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx4, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>KV: tx1.put(B, a)
KV-->>App: return
App->>App: ctx.set_response(OK)
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK
Executor->>NetStack: POST /EndTx {code=200, body="Copied {a} from {A} to {B}"}
rect rgba(191, 223, 255, 0.5)
note over NetStack,App: ...
Frontend->>App: execute_endpoint(tx5, ctx, h)
App->>App: tx = active_reqs[e].tx
Note right of App: // Gets tx1
App->>App: result = tx1.commit()
App->>App: response = (result, code, body)
App->>App: HTTP serialise response
App->>NetStack: reply_async(active_reqs[e].ctx.session, response)
App->>App: ctx.set_response(OK)
App->>App: active_reqs.pop(e)
note over App: tx1 is destroyed here
deactivate App
App-->>Frontend: return
note over NetStack,Frontend: ...
end
NetStack-->>Executor: 200 OK
deactivate Executor
NetStack->>NetStack: TLS encrypt response
NetStack-->>User: 200 OK "Copied {a} from {A} to {B}"