SCA DesignNotes

This document, "SCA Design Notes," details design changes, reasoning, and implicit assumptions for the SCAMPI (System Communication Architecture Message Passing Interface) system. It addresses various aspects of system operation, including concurrency, connection management, and buffer handling, often in the context of specific routines (e.g., SC.RAP, SC.ERR, SC.DIS, SC.ALC).

Key areas covered include:

1. Concurrency and Protection: SCAMPI employs mechanisms like Critical I/O Off (CIOFF), connection block locks, and NOSKED (no rescheduling) to manage concurrent access and prevent race conditions. Routines are categorized by their locking behavior (Lock, Honor, Lock and Honor), with SC.DIS (disconnect) being specially handled with CIOFF to ensure atomic state changes and defer other operations.
2. Connection Management: The system manages lists of connection blocks and "don't-care listeners." It defines processes for adding, removing, and traversing these lists, ensuring their integrity. Virtual Circuit (V.C.) states are meticulously handled, including sequences for opening and closing V.C.s by both SCAMPI and PHYKLP, along with error scenarios. The document explains why connection blocks are "reaped" in a process context (not at interrupt level), primarily for buffer return and efficiency. Issues like incoming packets on closed V.C.s, which can lead to state inconsistencies, are addressed with new checks and buffer handling procedures.
3. Buffer Management: A significant portion details the allocation, deferral, and return of message and datagram buffers within SCA.
   • Allocation: Routines such as SC.ALC, SC.ABF, and SC.ALD allocate buffers from a central pool, maintaining minimum thresholds (MINMSG, MINDG) that are dynamically adjusted based on the number of online systems. If buffer counts drop below these thresholds, SC.ALC is triggered to create more buffers. SC.CMG and SC.CDG are responsible for creating new buffers from memory pages using SC.BBF.
   • Deferral: SC.DEF handles "stuck" connect blocks that are awaiting buffers, working to acquire them and send pending connection management requests.
   • Return & Error Handling: SC.RBF and SC.RLD return buffers to free queues. PHYKLP (the physical layer protocol handler) is crucial in returning locally-generated buffers to port queues upon error, particularly when the "response" bit is not set, to maintain buffer integrity and system stability.
4. State Transitions: Extensive tables illustrate the various states of connections and connection blocks, message types, and the resulting state transitions when SCAMPI routines are called or messages are received (e.g., CONNECT_REQ, DISCONNECT_RSP).

In essence, the document serves as an internal guide to the complex interplay of routines, locks, buffers, and state changes necessary for reliable communication within the SCAMPI architecture, with a particular emphasis on handling concurrent events and errors gracefully.