Digital PDFs

MISC-68363AB5

December 1987

24 pages

Original

1.0MB

Document:	Firefox Workstation Dual-CVAX Processor Module Functional Specification
Order Number:	MISC-68363AB5
Revision:	0
Pages:	24
Original Filename:

OCR Text

Firefox Workstation Dual-CV AX Processor
Module Functional Specification
Revision 3.0

David Smith (DECWSE::DSMITH)
Tom Furlong (DECWSE::FURWNG)
Worlcstation Systems Engineering
Digital Equipment Corporation
100 Hamilton A venue
Palo Alto, CA 94301
415-853-6730

December 21, 1987

RESTRICTED DISTRIBUTION

Copyright 1986, 1987 by Digital Equipment Corporation
The information in this document is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital Equipment Corporation assumes no responsibility for
any errors that may occur in this document. Tiris specification does not describe any program or product
currently available from Digital Equipment Corporation. Nor does Digital Equipment Corporation commit
to implement this specification in any product or program. Digital Equipment Corporation makes no commitment that this document accurately describes any product it might ever make.

Blank Page

Table of Contents

6. Dual-CV AX Processor Module

6.1. Functionality ................................................................................................................... .
6.1.1. CVAX CPU/FPA ................................................................................................
6.1.2. Two-1-evel Caches .... .. ... ..... .. .. .. ...... .. .. ... ....... ..... ....... ..... ................... .. ... ..... .. ......

2
3

6.1.2.1. 1-evel-1 Cache ..................................................................................................
6.1.2.2. 1-evel-2 Snoopy Cache .....................................................................................

3
3

6.1.2.2.1. Cache Reads ..................................................................................................
6.1.2.2.2. Cache Writes ...................................... .............................................. ....... ......
6.1.2.2.3. Victim Processing and Cache Fills ...............................................................
6.1.2.2.4. Shared Reads .... .. .. ... ....... ....... ................. .. ... ....... ....... ... .... ... ..... .. .. ... ..... .... ... .
6.1.2.2.5. Shared Writes ................................................................................................

4
5
5
5
5

6.1.2.3. Order for Enabling/Disabling Caches ..............................................................

6.1.3. M-Bus Interface ...........................................................................................
6.1.4. S?\1P Requirements ... .. ... .. ..... ............ .. ........ .. ..... .. ..... ..... .. ............... .. .. ... .. ... .. .. ... .
6.1.5. Diagnostic/Boot ROM ........................................................................................
6.1.6. Test-Mode Inputs and Status Indicators ................. .. .. ... ....... ................. ....... ......
6.1. 7. External Connections .. ... .. ... .. .. ... .. ... .. .. ... .. ... .... ... .. .. . .. .. . .. .. ... .. .. ... .. ............ ... .... ... .

6
6
6
6
6

6.2. Implementation ................................................................................................................

6.2.1. CVAX CPU ........................................................................................................
6.2.2. Caches . ... .. ... .... ... .. ..... .. ... .. ..... ..... .. ... .. ..... ..... .. .. ... .. ....... ... .. ..... .. ........ .. .. ... .. ... .. .. ... .

8
8

6.2.2.1. CVAX uvel-1 Cache .....................................................................................
6.2.2.2. 1-evel-2 Cache ............... ......................................... .............. ........ ....... .. ..... .. ... .

8
9

6.2.2.2. l. 1-evel-2 Data Store ... .. ... .. ..... ....... .. ... ......... ..... ....... ... .. ....... .......... .. ... ..... .. .. ... .
6.2.2.2.2. 1-evel-2 Tag Store .........................................................................................

10
10

6.2.2.3. FBIC Support of the uvel-2 Cache ................................................................

6.2.3. M-Bus Interface ............. ................................. ..... .......... .. ... ......... ..... .. ... ..... .... ....
6.2.4. SMP Requirements . .. .. .. . .... . ... . ... .. ... .. .. ... .. ... .. .. .. . .. ... .. .. ... ..... ..... .. .. ... .. .. ... .. ... .. .. ... .

11
11

6.2.4.1. Halting Processors ...........................................................................................
6.2.4.2. Interprocessor Interrupts ..................................................................................
6.2.4.3. SMP Processor Registers .................................................................................
6.2.4.4. S?\1P Console Support .................................................................. .............. ......
6.2.4.5. S?vfP CPU Census ............................................................................................

11
12
12
12
12

6.2.5. Diagnostic/Boot ROM ........................................................................................
6.2.6. Test-Mode Inputs and Status Indicators ..... ........................................ ................

12
13

iii

6.3. Programming ...................................................................................................................

6.3.l~Address Map.·.......................................................................................................

6.3.1.1. Firefox I/0 Space .............................................................................................
6.3.1.2. Slot-Specific I/0 ..............................................................................................
6.3.1.3. External Processor Registers (EPRs) ...............................................................

14
15
17

6.3.2. Locating Modules ...............................................................................................
6.3.3. Initialization ........................................................................................................
6.3.4. Base Workstation ROM ......................................................................................

17
18
18

Revision History

- -r oate

112 Nov 87

Version

Content/Changes

3.0

Updated KA600 to KA60; corrected initialization code;
Operations, BOM, and Placement moved to Engineering
Specification

/ 14 Aug 87

2.1

Updated BOM and unqualified parts list;
added new power requirements and placement

I
I 01May87
I

2.0

Updated functionality, implementation
Added programming information
Revised as per Revision 2 M-Bus
Revised as per Revision 2 FBIC

I 30Jan 87

1.1

Integrate with System Specification

I 101an 81

1.0

First external release

I 21 Dec 86

0.0

Preliminary draft

I
!

I
II

Blank Page

6. Dual-CV AX Processor Module

This functional specification for the Firefox dual-CVAX processor module, KA60, describes the implementation of the modular CPU element of a Firefox multiprocessor. It does not describe the detailed function of the VLSI devices on the module, pertinent functional specifications for which are included in the
Appendices to the Firefox System Specification Designer's Guide.
The functionality provided by the KA60 forms the basis of the expandable CPU architecture of the system.
The circuiuy required to implement two CVAX CPU elements of the Firefox multiprocessor is provided on
this single, quad-height module, which contains all per-processor functions necessary to construct a symmetric multiprocessor (SMP). Supported functions include the following:
•

Two CVAX CPU/FPA chipsets

•

Two 64-Kbyte level-2 snoopy caches

•

Two sets of SMP-required IPRs

•

Firefox M-bus interface

•

Two sets of diagnostic/boot ROMs

•

Test mode inputs and status indicators

At least one KA60 module is required in all Firefox configurations. Certain configurations specifv two processor modules, providing four CVAX processors. The architecture of the KA60 does not preclude
configurations consisting of more than two processor modules. The factors that limit the number of CPUs
in a Firefox multiprocessor a.re the backplane slots, M-bus bandwidth. and I/O throughput Every effort
has been made to ensure that as many KA60 modules as other system components are able to support can
be con.figured in a Firefox.
Two versions of the KA60 are planned; they will differ only in the speed sort of the CVAX chipset and the
cache RAMs that support it. Although the KA60 is being designed to support a CMOS II CVAX chipset
with a 60-ns cycle time, the initial version of the KA60 will use a binned CMOS I chipset running at an
80-ns cycle time. As necessary throughout this docwnent, the different versions will be referred to as
KA60-60 and KA60-80 respectively.

6.1. Functionality
The KA60 provides two CVAX CPUs in a Firefox SMP environment The system's M-bus and snoopy
caches are designed to present each processor a consistent view of a single, global memory and 1/0 space.
This feature, along with an interprocessor interrupt capability, provides the hardware functionality necessary to support an SMP operating system, under which the addition of each CPU improves the computational performance of the entire system.
Figure 6-1 is a functional block diagram of the KA60, with its two functionally independent CPUs that
share a single set of bus transceivers, clocks, and test-mode connections.

6.1. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 1

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTR1BL1ION

m
0

Processor A

Processor B

CVAXCPU
and level-I
cache

I
J

SMP

Floating

SMP

processor

point

processor

~gistcrs

accelerator

~gistcrs

Level-2

data

tag

store

Cache

control

M-bus

interface

t---

sto~

Diagnostic
ROM

1----1

'---

t-----1

data

store

Diagnostic
t--

ROM

t-----1

l
M-bus
transceivers

M-bus

Figure 6-1:

KA60 Procauor Module

Because of the symmetry of the K.A60, the following sections describe the functionality of either processor
A or processor B without specific reference to implementation.
6.1.1. CV AX CPU/FPA

The CV AX CPU is a : --bit, virtual-memory microprocessor that is implemented in custom CMOS technology and incorporates the following features:
•

A subset of VAX data types: byte, word, longword, quadword, character string, and variable-length
bit field

2 Firefox System Specification

December 21, 1987

Dual-CV AX Processor Module 6.1.1.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

•

The MicroVAX subset of the VAX instruction set: integer and logical, address, variable-length bit
field, con~l, procedure.. call, queue, MOVC3/MOVC5, and operating system support

•

Full VAX memory management

•

An on-chip, 1 Kbyte, 2-way associative mixed instruction- and data-stream cache

•

A 32-bit standard interface (CVAX pin-bus)

•

Large virtual- and physical-memory address spaces: 4-Gbyte virtual- and 1-Gbyte physical-address
space

To accelerate floating point operations, each CVAX bas a companion CFPA that performs the following:
•

Operates on F_fioating, D_fioating, and G_floating VAX data types

•

Supports VAX floating-point instructions for F_fioating, D_ftoating, and G_floating

•

Accelerates execution of MULL, DIVL, and EMUL integer instructions

Complete specifications for the CVAX and CFPA chips are available in the appendices to the Firefox
Designers' Guide.
VAX SRM values are as follow:
•

System ID for the CVAX processor: 10

•

System type for the KA60: 3

6.1.2. Twt>Level Caches

Firefox CPUs implement two-level caches. The level-I cache is internal to the CVAX processor. The
level-2 cache is the external snoopy cache. The Firefox snoopy caches both service their attached CV AX
processor and monitor the system M-bus to provide other caches with the contents of memory locations
they have cached. Because all caches monitor all memory-space bus transactions, they can continually
determine whether cache entries are exclusively in their cache or are shared by two or more caches. The
caches use this information to maintain systemwide cache consistency and to dynamically minimize M-bus
bandwidth, utilizing a write-back policy for unshared entries and a write-though policy for shared entries.
6.1.2.1. Level-1 Cache

Each CVAX CPU chip contains an internal, 1-Kbyte, 2-way, set-associative level-1 cache, with a quadword line size. The level-1 cache, which is always write-through, can be independently enabled/disabled
for instruction- and data-stream references.
In Firefox, the level-1 cache is enabled for both instruction- and data-stream references. To ensure
system-wide cache consistency the level-1 cache is maintained as a subset of the level-2 cache. This is
accomplished through CVAX cache-invalidate cycles in the following manner:
•

A line removed from the level-2 cache is invalidated in the level-1 cache.

•

A line updated from the M-bus in the level-2 cache is invalidated in the level-1 cache.

To maintain cache consistency, the level-1 cache must not be enabled while the level-2 cache is disabled.
6.1.2.2. Level-2 Snoopy Cache

All CVAX references to global memory or 1/0 space must go through a level-2 cache that implements the
snoopy cache protocol as defined by the Firefox System Specification. The level-2 cache provides longword access to a direct-mapped cache that has an octaword line size. At system powerup, a single-entry,
level-2 snoopy cache present in the Firefox M-bus interface is utilized. After self-test and proper software
initialization, a 64-Kbyte, level-2 snoopy cache is enabled for each processor on the KA60.
CV AX references to global I/0 space are never cached and always generate M-bus cycles. CV AX
memory space references are always cached by the level-2 cache and generate M-bus transactions

6.1.2.2. Dual-CV AX Processor Module

December 21. 1987

Firefox System Specification 3

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

dependent upon the values of state variables stored with each line.
Each entry in the !evel-2 cache.. consists of an octaword data store, a tag store, and two state variables. The
state variables represent the two independent properties of each entry: shared and dirty.
The shared property indicates whether the entry is present exclusively in this cache or resident in two or
more system caches. If a cache entry is unshared, writes to this location do not generate bus traffic. If a
cache entry is shared, write-hits initiate an M-bus write-through transaction to update the shared entries in
other caches.
The dirty property indicates that the entry has been modified in the level-2 cache but has not been updated
in memory. Dirty entries must be written back to memory if a miss requires use of the dirty entry's location in the cache; this is termed a victim write.
The snoopy caches monitor all M-bus memory space transactions and probe their tag store at the indicated
address. If a hit results, they assert the MSHARED signal on the M-bus, thus allowing update of the shared
status flags. For a write transaction, all monitoring caches with a hit update their data entries. For a read
transaction, data will be supplied either from memory or from a cache that shares the entry. Data is supplied by a monitoring cache only when it hits on a dirty entry.
The combination of shared and dirty properties results in four possible states for each cache entry. Figure
6-2 diagrams the transitions among these states.

CPU write
Not shared
Not dirty

Not shared
Victim write

Dirty

Miss (not shared

Victim write
M-bus read
M-bus write

Miss (shared)

U write (not shared)

M-bus rea

M-bus write
Shared

Shared

Not dirty

Dirty
CPU write (shaRid)
CPU read
CPU write (shared)
M-bus read

M-bus read
M-bus write
CPU read

Figure 6-2:

Snoopy Cache State Transitions

6.1.2.2.1. Cache Reads

When the CVAX issues a memory-space read, the level-2 cache probes its tag store to determine if the
requested location is present in the cache. If the tag probe results in a hit, the level-2 cache presents the
data from the level-2 data store to the CVAX. This is accomplished within one CVAX extemru cycle.

In the event of a tag-probe miss, the level-2 cache retries the CVAX, performs victim processing on the
data store, and fills the data store with the requested line. The CVAX hardware will retry the read cycle,
which now results in a read hit.

4 Firefox System Specification

December 21, 1987

Dual-CV AX Processor Module 6.1.2.2.2.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBL"TION

6.1.2.2.2. Cache Writes

When the CV~j$sues a memory-space write, the level-2 cache probes its tag store to determine whether
the requested location is present in the cache. If the tag probe results in a hit, the level-2 data store is
updated by the CVAX, and the dirty bit is set for the cache line.

If the tag had the shared bit asserted, a masked octaword write-through transaction is generated on the Mbus using the data and byte masks from the CVAX. When the write through is completed, the level-2
shared bit for the line is updated with the value of MSHARED. That is, if the line is no longer in other
caches, it reverts to being an unshared line.
If the tag probe results in a miss, the level-2 cache retries the CVAX, performs victim processing on the
data store, and fills the data store with the requested line. The CVAX hardware will retry the write cycle,
which now results in a write hit.
6.1.2.2.3. Victim Processing and Cache Fiiis

When a read or write miss occurs, victim processing is initiated before the data store is filled with the
requested line.

If the victim line is clean, the level-2 cache immediately issues an M-bus memory read to obtain the
requested line. While reading the data from memory, the cache controller performs a CVAX octaword
cache invalidate cycle to (possibly) remove the victim line from the level-1 cache. When the M-bus read is
complete, the data store is updated with the requested line. The tag store is updated with the new address,
the shared bit is set to the value of the MSHARED signal, and the dirty bit is cleared.

If the victim line is dirty, the level-2 cache performs a CV AX octaword cache invalidate cycle to (possibly)
remove the victim from the level-1 cache. The cache controller issues an M-bus memory victim write to
flush the dirty line back to memory. The cache controller issues an M-bus memory read to obtain the
requested line and fill the data store. The tag store is updated with the new address, the shared bit is set to
the value of the MSHARED signal, and the dirty bit is cleared.
6.1.2.2.4. Shared Reads

Whenever the level-2 cache observes the start of a memory read on the M-bus, it does a tag-store probe to
determine whether its data store contains the requested line. If a hit results, the cache controller sets its
shared bit and asserts the MSHARED signal so the module that issued the memory read can also mark the
cache line "shared." If the tag hit references a dirty line, the cache controller arbitrates to provide its data to
the M-bus master. This ensures that modified data resident in caches is used in place of stale memory data.
6.1.2.2.5. Shared Writes

Whenever the level-2 cache obseives the start of a memory write through on the M-bus, it does a tag-store
probe to detennine whether its data store contains the requested line. If a hit results, the cache controller
sets its shared bit and ~serts the MSHARED signal so the module that issued the memory write can also
mark the cache line "shared." The slave tag-store dirty bit is cleared because the initiator of the write
through has set its dirty bit.
6.1.2.3. Order for Enabllng/Dlsabllng Caches

To maintain systemwide cache coherence, the following procedure must be followed.
The system powers up with CV AX level-1 cache disabled and, by default, uses the hardware-initialized
FBIC single-entry level-2 snoopy cache. The CPU begins executing out of ROM and turns on the level-2
cache to enable access to its cache tags. The tags are initialized to 00000000#16. When this is completed,
the data entries are initialized to 00000000#16, following which the tags are rewritten to reflect shared,
dirty status 00006000#16. At this point, all caches contain the same entries and are consistent, and the
level-1 cache can be enabled as an instruction and data cache. The level-1 cache must not be enabled until
the level-2 cache is operational.

6.1.2.3. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 5

DIGITAL EQUIPMENT CORPORATION -RESTRICTED DISTRIBCTION

Once the external cache bas been enabled, it must not be turned off without first being manually flushed, or
modified cache ~a will be los.t. Should the level-2 cache need to be disabled, the level-1 cache must first
be disabled, and-the external cache manually flushed. To flush the external data store, read all tags through
I/O space and force victim writes of any dirty lines by reading memory at a congruent address without generating memory writes in the process. Turning off the external cache is not recommended.
6.1.3. M-Bus Interface

Each CPU implements an M-bus interface that conforms to the Firefox M-Bus Specification. The M-bus
provides a view of memory and 1/0 space that is consistent to all processors.
In addition to the M-bus snoopy cache protocol, the M-bus interface provides support for the following:
•

VAX interlocked instructions

•

VAX vectored interrupts

•

Slot-specific 1/0 mapping

The diagnostic/boot ROMs and FBIC device registers are mapped to the 32-Mbyte, M-bus slot-specific l/0
space. Processors A and Bon the KA60 module further subdivide the slot-specific 1/0 into two 16-Mbyte
regions.
M-bus arbitration and priority are managed in the M-bus interface. Subarbitration and priority determination between the two processors on the KA60 module are supported by the interface.
6.1.4. SMP Requirements

The KA60 supports required S:MP functionality in the following manner:
•

A global memory and 1/0 space is implemented through the hardware level-2 snoopy caches.

•

A single console is provided that can independently halt any processor. Console input can be
directed to any halted processor.

•

External processor registers (EPRs) that contain the processor-specific data structures CPUID and
WHAMI are implemented.

•

The KA60 supports interprocessor interrupts at four interrupt-priority levels (14, 15, 16, and 17).

•

A mechanism is provided to determine the number and status of processors in the system.

6.1.5. Diagnostic/Boot ROM

Each CPU has its own diagnostic/boot ROM. The ROM resides in local CVAX 1/0 space in the VAX restart address range of 2004000#16 to 2007FFFF#16. The ROM is also mapped to global slot-specific 1/0
space.
6.1.6. Teat-Mode Inputs and Status Indicators

The value of two manufacturing-mode input pins is available to both CPUs for use by test software. Both
CPUs on the K.A60 share a set of test-mode input and output connections. Each CPU is provided a four-bit
bank of red LEDs used to display per-processor status. A single bit from both CPUs' LED output latch is
logically ANDed to provide module-OK test output and drive a single, green LED status indicator.
6.1.7. External Connections

The KA60 module has no external connections. Two manufacturing-mode input pins and one module-OK
output pin are provided for attachment to manufacturing testers.

6 Firefox System Specification

December 21, 1987

Dual-CVAX Processor Module 6.2.

DIGITAL EQUIPMENT CORPORATION· RESTRICTED DISTRIBl.JTION

6.2. Implementation
The KA60 desi~makes exteDSive use of Surface Mount Device (S:MD) technology to fit two VAX CPUs
onto a single-sided module with 70 square inches of logic area. The KA60 uses the comet-on-a-quad PC
board outline defined for Firefox.
A block diagram of the KA60 module appears in Figure 6-3.
Processor A

Processor B

I CCLOCK I

CFPA

CVAX pin-bus ---

.,_ CV AX pin-bus

CVAX

ADAL<l5:0>
I
r-i
... -1.~
.

YIROM!i
!~1

t
Lcvel-2

I
t

BDAL<l5:0>

iLJT

•

I Level-2

~
BDAL<l5:4>

· ADAL<l5:4>

TCACHE< 15:0>

MDAL<l5:4>

M-bus

MDAL<l5:4>

1
M-bus
Transceivers

M-bus

Figure 6-3:

KA60 Block Diagram

The sections that follow describe the hardware implementation of the KA60. The FBIC is the multipurpose bus interface and cache controller for Firefox. It connects a CVAX pin-bus to the system M-bus
and supports the level-2 snoopy cache, S:MJ> registers, and the manufacturing mode interfaces for the
KA60. Subsets of the FBIC features are used in the implementation of each function described. The FBIC
device features used in support of a given function are described in the context of that functional unit.

6.2. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 7

DIGITAL EQUIPMENT CORPORATION -RESTRICTED DISTRIBlJTION

6.2.1. CV AX CPU

The CVAX CPU functi.onalityAf the KA60 is implemented in three custom CMOS chips produced by the
Semiconductor Engineering Group in Hudson:
•

CVAX CPU chip

•

CVAX Floating-Point Accelerator chip (CFPA)

•

CVAX Clock chip (CCLOCK)

Most of the circuitry required to configure the CVAX chipset as a Firefox S:rvtP computer is provided
through a single Firefox Bus Interface Chip (FBIC), which is an ASIC chip designed by the Workstation
Systems Engineering Group in Palo Alto.
The KA60-80 uses binned CMOS I chips with an 80-ns cycle time. K.A60-60 uses CMOS II chips with a
60-ns cycle time.
The CCLOCK chip is a CVAX support chip that generates all the precision MOS clock signals necessary
to operate the CVAX CPU, CFPA, and FBIC. One CCLOCK chip generates clocks and for both CPUs.
The different versions of the KA60 use oscillators of either a 50-Mhz, KA60-80 or a 66.6 MHz, KA60-60.
For module-level testing, the CCLOCK oscillator input can oe provided externally. The CCLOCK chip
contains no software-configurable registers.
A complete description of the chipset can be obtained by referring to the CVAX, CFPA, and CCLOCK
chip specifications provided in the appendices to the Firefox System Specification Designer's Guide.
Two CVAX CPU circuits are provided on the KA60, they share one CCLOCK chip.

6.2.2. Caches

The KA60 level-2 cache is physically implemented as eight 16K x 4-bit SRAMs for data store, four 16K x
1-bit SRAMs for data parity, and four 4K x 4-bit SRAMs for the tag and tag parity. Six 74FCT373As provide the necessary bus isolation and address latching. A 74FCT191A and a 74FCT374A provide the longword addressing of the octaword data store. The cache controller and CVAX pin-bus to M-bus interface
function is provided by the FBIC. A level-2 snoopy cache is provided for each CPU implemented on the
KA60.

6.2.2.1. CVAX Level-1 Cache

The CVAX level-1 cache is a 1-Kbyte, two-way set-associative, write-through cache with a quadword line
size. The level-1 cache stores both instruction and data-stream references. Level-1 cache organization is
depicted in Figure 6-4.
Figure 6-4:

Level-1 Cache Organization

64 Rows

Set2

I
93

8 Firefox System Specification

December 21, 1987

Dual-CVAX Processor Module 6.2.2.1.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

A line in the level-I cache is shown in Figure 6-5.

72 71

+--------------------------------------------------+
IPIVI TAG IPIB71PIB6IPIBSIPIB4iPIB31PiB21PIBliPiBOI
+--------------------------------------------------+
I I
I I
I I
I I
I I
I I
I I
+-1--+-1--+-1--+-1--+-1--+-1--+-1--+---> 8 Bytes
+----+----+----+----+----+----+----+------> Parity bits
+----------------------------------------------> 20-bit tag
+--------------------------------------------------> Valid bit
+----------------------------------------------------> Tag parity bit
Figure 6-5:

CVAX Level-1 Cache Une

In Firefox, the level-1 cache is enabled for both instruction- and data-stream references. 1bis is accomplished by programming the CVAX cache disable register bits CADR<5> and CADR<4> to Os. Under no
conditions are I/0-space references stored in the level-1 cache.
The level-1 cache is protected by byte parity on the data store and by a single parity bit on the tag store.
Internal CV AX error-checking signals parity errors on the level-1 cache.
The level-1 cache in Firefox is maintained as a consistent subset of the level-2 snoopy cache. Whenever a
line is removed from the level-2 cache, or whenever a shared entry in a level-2 cache is updated from the
M-bus, level-1 cache entnes may become inconsistent with level-2 and must be mvalidated. Level-1 cache
invalidate cycles are performed by the FBIC in response to either of these conditions.

6.2.2.2. Level-2 Cache

Both CV AX CPUs on the KA60 are supported externally by a level-2 snoopy cache. The level-2 cache is a
64-Kbyte, direct-mapped, write-back snoopy cache with an octaword line size. The level-2 cache stores
both instruction- and data-stream references. All CVAX instruction- and data-stream references that miss
the level-I cache will generate level-2 cache cycles. I/0-space references are never cached.
The level-2 cache has a data store 64 Kbytes in size and configured as 4K lines. Each cache entry has a
128-bit (octaword) data store, a 15-bit tag store, 16 data parity bits (byte parity), and 1 tag parity bit. The
level-2 cache is always longword accessed; M-bus accesses always reference four sequential longwords.
Figure 6-6 shows the organization of the level-2 cache.

4K x 144
Data

4Kx 16
Tag

159
Figure 6-6:

144 143

Level-2 Cache Organization

6.2.2.2.1. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 9

DIGITAL EQUIPMENT CORPORATION - RESTRJCTED DISTRIBUTION

6.2.2.2.1. Level-2 Data Store

The data store COl)Sists of 16K.Iongwords organized as 4K octawords. Each byte in the data store is errorprotected by a parity bit. Parity is calculated/evaluated either by the CVAX processor or by the FBIC,
dependent upon whether it is a CVAX pin-bus- or an M-bus-generated cycle.
The level-2 data store resides on the CVAX pin-bus side of the FBIC. Whenever an access to the data
store from the M-bus is necessary, a CVAX DMA cycle is performed to get control of the CVAX pin-bus.
The FBIC performs this function.
Figure 6-7 shows an entry in the level-2 data store.

143
0
jsFjBE!BD!BC!BBjsAjs9)Bsjs1)B6)B5js4\B3/B2js11so)
16 Bytes with byte parity
Figure 6-7:

Level-2 Data-Store Entry

6.2.2.2.2. Level-2 Tag Store

The tag store is comprised of the address tag field, the snoopy cache dirty and shared flags, and a tag parity
bit. A level-2 tag-store entry appears in Figure 6-8.
Name: TAG
1

TAG

/.\

TAG:

Parity
Shared
Dirty_bit
DAL<28:16>
Figure 6-8:

Level-2 Tag-Store Entry

Only the lowest 512 Mbytes of M-bus memory space are cached in the level-2 cache. This represents the
full physical address space of a VAX processor. The address tag field corresponds to address bits <28: 16>
of the physical address of the current cache en~; The level-2 cache is a direct-mapped physical cache;
address bits <15:0> represent the index into the 64-Kbyte cache.
The physical address of the entry in the data store is written to the address tag field whenever a line is allocated. The dirty flag is updated whenever a write occurs to the data store. The shared flag is updated
whenever there is a read or write hit in the level-2 data store. New tag parity is calculated/evaluated

10 Firefox System Specification

December 21, 1987

Dual-CVAX Processor Module 6.2.2.2.2.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

whenever the tag store is accessed. The FBIC is responsible for all access and updating of the level-2 tag
store.
Physical address bits CDAL<15:4> are used to index this field. The contents of this field are compared
against physical address bits CDAL<28:16> for equality. If there is a match, a cache hit occurs, and the
FBIC takes appropriate action. The physical address bits CDAL<3:2> are used to index the appropriate
longword in the cache. See Figure 6-9.
Name: DECODE

1 1
6 5

3 3 2 2

1 0 9 8
TAG

4 3 2 1 0
Tag_index

DECODE:

Length

-~I

I/O_space
Tag
Tag_index
Longword_ind.5*-~~~~~~~~~~~~~~~~~~~~~~~~~~~--'

Reseived
Figure 6-9:

Level-2 Cache Address Decode

6.2.2.3. FBIC Support of the Level-2 Cache

The FBIC implements both a single-entry level-2 snoopy cache to be used at powerup and the controller
for the external level-2 cache. The FBICSR <26> EXCAEN bit controls enable/disable of the level-2
cache.

6.2.3. M·Bus Interface

The M-bus interface consists of one FBIC per processor. Circuitry shared between the two CPUs includes
seven 74F245 M-bus transceivers, one 74AS760 open collector M-bus driver, and one 74AS808B for onboard arbitration.
The FBIC registers that support access to the M-bus and error logging are MODTYPE, BUSCSR,
BUSCTL, BUSADR, BUSDAT, FBICSR, RANGE, IADRl and IADR2.

6.2.4. SMP Requirements

All support for SMP is provided through the FBIC. Some of the registers that support the SMP functions
are implemented as external processor registers. All SMP registers are mapped to slot-specific I/O space.
The console program initializes the SMP-specific registers of the Firefox.

6.2.4.1. Halting Processors

Individual processors can be halted by writing to FBICSR<25>, HALTCPU, in each CPU's slot-specific
I/0 space. To enable halting, FBICSR<7> HALTEN must be set. All processors in the system may be
halted simultaneously by depressing the Halt switch on the RF distribution panel of the Firefox workstation.

6.2.4.1. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 11

DIGITAL EQU1PME."IT CORPORATION - RESTRICTED DISTRIBUTION

6.2.4.2. Interprocessor Interrupts

Delivery of inteiprocessor iiaterrupts is controlled by the FBIC IPDVINT register. Programming
IPDVINT<16>, IPUNIT, to a 1 with an appropriate vector in IPDVINT<15:0>, VECTOR, causes the
FBIC to function as an interprocessor interrupt unit.
Any processor in the system can interrupt another processor by writing to the target processor's
IPDVINT<27:24>, IPL, field in the other processors' slot-specific I/0 space. The FBIC will generate a
vectored interrupt to occur on the target processor at the requested IPL level. The level of an interprocessor interrupt is passed to the target processor via the bits <3 :2> of the vector.

6.2.4.3. SMP Processor Registers

The FBIC implements the SMP CPUID and WHA.MI external processor registers. CPUID is a read-only
register that specifies the hardware CPU identifier. CPlHD is accessible as EPR 14 and through slotspecific I/O space. WHA.MI (WHo AM I) is a read/write register that specifies a software CPU identifier,
typically a pointer to a CPU-specific data structure. WHAMI is accessible as EPR 15 and through slotspecific 1/0 space.

6.2.4.4. SMP Console Support

The FBIC implements the SAVGPR (Save General Purpose Register) external processor register for use by
the console program. SA VGPR is a read/write register that allows the console program to push the contents of one of the VAX GPRs without affecting processor context. SAVGPR is accessible as IPR 41 and
through slot-specific I/O space.

6.2.4.5. SMP CPU Census

The FBIC provides the hardware necessary to determine both the number of CPUs present in an SMP
configuration and their statuses.
At M-bus initialization, all FBICs assert their M-bus MBRQ lines. This provides a positive option-present
determination for all M-bus slots. The value of the :MBRQ lines is logged in the FBIC BUSCIL <6:0>,
:MBRM, register in each FBIC. By reading the value of the MODTYPE register of each FBIC for which a
positive option present was received, the number of CPUs in the system can be determined.
The console program is able to interrogate the status of each CPU. CPUs that fail the self-test can be
removed from the S~ system by the console program. This is accomplished by setting the TSTFNC bits
of the FBICSR <5:1> register to logically isolate the processor on that FBIC from the M-bus. In that all
CPUs run the self-test in parallel, the primary CPU needs only poll the status of each processor's CPU-OK
signal in FBICSR <12>, LEDS<4>, to determine whether a given processor is functioning.

6.2.5. Diagnostic/Boot ROM

Each CPU is configured with one 27210 64K by 16 EPROM. A total of 128 Kbytes is provided to each
CPU. The FBIC controls reading of the EPROMs that physically reside on the CVAX pin-bus. These
EPROMs are socketed
Diagnostic/boot ROM is mapped to the CVAX in local I/0 space in the VAX restart address range of
2004000#16 to 2007FFFF#l6. To the M-bus, the ROM appears in slot-specific I/0 space defined by the
FBIC.
Per the SRM, the system type identifier (value= 3 for the KA60) is programmed into the .:>QM at address
20040004.

12 Firefox System Specification

December 21, 1987

Dual-CVAX Processor Module 6.2.6.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBtrrION

6.2.6. Test-Mode Inputs and Status Indicators

Each processor mi.plements a·rect quad LED indicator on which to report per-processor status. A single
green LED indicator and a 74FOO provide the module-OK signal and indicator.
The FBIC provides the interface through which software reads the test-mode inputs and sets the indicator
outputs. The FBICSR<l 1:8> LEDS are connected to each processor's red quad LEDs. The FBICSR<12>
LEDS output of each CPU's FBIC are ANDed together externally to drive the single green LED. Setting
FBICSR<l2> in both FBICs will illuminate the green LED.
FBICSR<31:30> reflects the current status of the manufacturing-mode input pins to the module. The
manufacturing.mode inputs of both FBICs are wired together.

6.3. Programming
Tills section describes both required initialization of the KA60 and the address spaces seen by the processor. The examples used here are based on a KA60 in M-bus slot 1. All addresses are VAX physical
addresses.
6.3.1. Address Map

The Firefox M-bus supports a 2-Gbyte memory address space and a separate 2-Gbyte 1/0 address space.
VAX processors support a 512-Mbyte memory address space and a separate 512-Mbyte 1/0 address space.
Mapping of the 30-bit VAX address space into the 32-bit M-bus address space is provided by the FBIC. In
this section, this mapping is ignored, and address maps are presented as they are seen by the CV AX processors. The KA60 physical address map is shown in Figure 6-10.
3FFFFFFF

1/0 space
512 Mbytes

20000000
lFFFFFFF

Memory space
512 Mbytes

00000000
Figure 6-10:

KA60 Physical Address Map

Memory-space references are decoded for CVAX physical addresses in the range of 00000000
1FFFFFFF#l6. 1/0 space is accessed by addresses in the range of 20000000 ... 3FFFFFFF#16.

6.3.1. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 13

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

6.3.1.1. Firefox 1/0 Space

Firefox I/0 space,-as seen by the KA60, is broken into the following four sections:
•

M-bus global I/O

•

Module-specific ROM

•

Processor-local I/0

•

Slot-specific I/O

M-bus global I/O provides approximately 128 Mbytes of systemwide I/O space. This is implemented to
allow mapping of VLSI controllers with hardwired address decoders and encompasses the range of
addresses assigned to Unibus and Q-bus controllers. M-bus global 1/0 locations can be mapped to any system module through the programming of the FBlC range registers.
Module-specific ROM is decoded locally on the KA60 in the VAX restart address range 20040000 ...
2007FFFF. This allows each processor in a Fuefox system to perfonn a boot and self-test without requiring any M-bus references. The m0<1uie-specific ROM is also mapped by the FBlC to the slot-specific 1/0
region.
Processor-local 1/0 is provided for I/0-mapped coprocessors. Addresses in this range are fixed for each
processor and are not globally mapped. This allows processor access to coprocessor registers at a fixed
address that is independent of M-bus slot Symmettic configurations of CPU/coprocessor pairs require this
address mapping to allow the code that services them to be independent of the pair on which it executes.
Slot-specific I/0 provides a 32-Mbyte region for module-specific 1/0. This region contaim FBlC registers,
module ROM, tag stores, module CSRs, and 1/0 mapped buffer memory. At most, eight modules can
reside in an M-bus backplane. Each module is assigned a 32-Mbyte region of 1/0 space, as detennined by
the module ID decoding of its backplane slot
Figure 6-11 shows the Firefox 1/0 space as seen by a CVAX processor.

14 Firefox System Specification

December 21, 1987

Dual-CVAX Processor Module 6.3. l. l.

DIGITAL EQUIPMENT CORPORATION - RESTRJCTED DISTRIBUfION

3EOO<f000 I

"32-Mbyte Slot 7

3COOOOOO I

32-Mbyte Slot 6

3AOOOOOO I

32-Mbyte Slot 5

Slot-Specific I/O Space

I
I
I

38000000
I

36000000

32-Mbyte Slot 4
32-Mbyte Slot 3

.... ,
•I

II
I

34000000

32-Mbyte Slot 2

I
32000000 I
I

32-Mbyte Slot 1

30000000 I

32-Mbyte Slot 0

l.·····

FBIC reeisters
External tag store
Reserved
Module ROM

I 33F80000
33FOOOOO
33E80000
33EOOOOO

30Mbyte1/0

32000000

128-Mbyte
I, CVAX Pin-bus local 1/0
28000000
127.5-Mbyte
Global M-bus 1/0
20080000
20040000
20000000
Figure 6-11 :

256 KBytes
256 KBytes

VAX boot ROM
Global M-bus 1/0

Firefox 1/0 Space

6.3.1.2. Slot-Specific 1/0

Table 6-1 lists the base addresses for the various M-bus backplane slots. Accesses to devices mapped to
slot-specific I/0 must add the device offset to the slot base address.
Table 6-1:

Slot
0
1
2
3

4
5
6
7

M-Bus Backplane Slot BaH Addre•-

M-Bus Address

VAX Address

90000000#16
92000000#16
94000000#16
96000000#16
98000000#16
9AOOOOOO#l6
9COOOOOO#l6
9EOOOOOO#l6

30000000#16
32000000#16
34000000#16
36000000#16
38000000#16
3A000000#16
3COOOOOO#l 6
3EOOOOOO#l 6

The FBIC maps the upper two Mbytes of slot-specific 1/0 into four separate 512-Kbyte regions that contain
the following:

6.3 .1.2. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 15

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRIBUTION

•

FBIC registers

•

External lag. store (proe&ssor modules only)

•

Reserved

•

Module ROM

Table 6-2 lists the offsets for the upper two Mbytes of slot-specific I/0 space.
Table 6-2:

FBIC-Managed Offsets

Slot
FBIC registers
Tag Store
Reserved
Module ROM

Address Range
XXF80000 ... XXFFFFFF
XXFOOOOO ... XXF7FFFF
XXE80000 ... XXEFFFFF
XXEOOOOO ... XXE7FFFF

FBIC registers require only a 64-byte portion of the 512-Kbyte portion. The registers appear 8192 ti.mes
within this region. For compatibility with possible future extensions, software must access the registers
only at their designated addresses. Table 6-3 provides a description of the FBIC registers and the functions
they support.
Table 6-3:

Name
MODTYPE
BUSCSR
BUSCTL
BUSADR
BUSDAT
FBI CSR
RANGE
IPDVINT
WHAM!
CPUID
IADRl
IADR2
SAVGPR

FBIC Register Map

Address
XXFFFFFC#l6
XXFFFFF8#16
XXFFFFF4#16
XXFFFFFO#l6
XXFFFFEC#l6
XXFFFFE8#16
XXFFFFE4#16
XXFFFFE0#16
XXFFFFDC#l6
XXFFFFD8#16
XXFFFFD4#16
XXFFFFDO#l6
XXFFFFC4#16

R/W
R
R/W
R/W
R/W
R/W
R/W
R/W
R/W
R/W
R
R/W
R/W
R/W

Description
Module-type register
M-bus error status register
M-bus error control signal log register
M-bus error address signal log register
M-bus error data signal log register
FBIC control status register
I/0-space range decode register
Interprocessor/device interrupt register
Unique software ID register
Unique hardware ID register
Interlock 1 address register
Interlock 2 address register
Scratch register for halt code

Each external cache store tag appears in l/O space four times at each longword within the natlll'ally aligned
octaword. That is, the tag for external cache line 0 is accessible at offsets XXFOOOOO, XXF00004,
XXF00008, and XXFOOOOC. Software can access the tag for a cache line at any of the four addresses. The
entire set of tags reappears eight times within the 512-Kbyte tag space. For compatibility with larger external caches, software must access only the tags in the XXFOOOOO ... XXFOFFFF region.
The ROM appears at both the VAX restart address of 20040000 and XXEOOOOO. The FBIC responds only
to addresses 20040000 ... 2007FFFF (256 Kbytes) due to address-space assignments of the CQBIC.
Modules with 512 Kbytes of ROM can access only the upper 256 Kbytes of the ROM at XXE40000 ...
XXE7FFFF. A ROM smaller than its assigned region reappears multiple times within the region.
Multiple processor modules in a system communicate through slot-specific I/0 space. Because there are
two CV AX processors on each KA60 module, the slot-specific 1/0 space for a processor module is further
subdivided into two 16-Mbyte regions. Figure 6-12 shows the l/0 address map of another KA60 module in
M-bus slot 1.

16 Firefox System Specification

December 21, 1987

Dual-CV AX Processor Module 6.3.1.2.

DIGITAL EQUIPMENT CORPORATION - RESTRICTED DISTRlBtJTION

Processor A Slot I/0

~-Mbyte Slot 7

3EOOOOOO I

.{
32-Mbyte Slot 6

3COOOOOO I
i
3AOOOOOO I

./I

32-Mbyte Slot 5

14 Mbyte

Processor B Slot I/O
PROCB FBIC

32F80000

PROC B Tag store

32FOOOOO

Reserved

32E80000

PROC B Boot ROM

32EOOOOO

32-Mbyte Slot 0

30000000

·.
128 Mbyte
CVAX Pin-bus
local I/O

14 Mbyte

·.

28000000

PROCB
CVAX Pin-bus I/0

127.5 Mbyte
Global M-bus I/O

··.I

32000000

Slot Specific I/O -- Dual CPU

20080000

Figure 6-12:

I 33EOOOOO

33000000
32-Mbyte Slot 1

32000000

20040000
20000000

PROCA
CVAX Pin-bus I/0

32-Mbyte Slot 2

34000000

/ 33F80000
33FOOOOO
I 33E80000
i

32-Mbyte Slot 3

36000000

PROC A Tag store
Reserved
PROC A Boot ROM

32-Mbyte Slot 4

38000000

PROCA FBIC

•I

256KB
256KB

VAX boot ROM
Global M-bus I/O

KA60 Slot-SpecHlc 1/0-Sp•ce Map

6.3.1.3. External Processor Registers (EPRs)

KA60 modules provide external processor register access to the SMP registers in the FBIC. Table 6-4 summarizes the FBIC EPRs.
Table 6-4:

KA60 Extamal Processor Registers

Name

EPR Address

R/W

Description

CPUID
WHAM!
SAVGPR

R
R/W
R/W

Unique hardware ID register
Unique software ID register
Scratch register for halt code

15
41

6.3.2. Locating Modules

After a workstation reset (M-bus MRESET asserted) console and diagnostic software must determine the
workstation configuration. During MRESET, the FBIC saves the value of the M-bus MBRQ signals in its
BUSCTL register. Software may use this as a module-present indication to identify backplane M-bus slots
that contain Firefox modules. To interpret the BUSCTL<MBRM> bits, a module must first determine its

6.3.2. Dual-CV AX Processor Module

December 21, 1987

Firefox System Specification 17

DIGITAL EQt:IPME.~'T CORPORATION - R.ESTRlCTED DISTRlBl.l'TION

own M-bus slot by reading its FBIC CPUID<MID> register field. The FBIC CPUID register is mapped as
a CVAX EPR and is read through a move-from-processor-register instruction. At powerup, each CPU
must perform ariEPR read of CPUID to determine the base address of its slot-specific 1/0 space. Table 6-5
lists the interpretation of BUSCTL<MBRM> as a function of a module's M-bus slot (from its
CPUID<MID> ). Software must use or save the value of BUSCTL<MBRM> before it enables FBIC error
logging, or the information will be lost.
Table 6-5:

Interpretation of the FBIC BUSCTLcMBRM> Field

CPUID<MID>

<6>

<5>

<4>

<3>

<2>

<1>

<0>

0
l
2
3
4
5
6
7

7
7
7
7
7
7
7
6

6
6
6
6
6
6
5
5

5
5
5

4
4
4
4
3
3
3
3

3
3
3
2
2
2
2
2

2
2
1
1
1
1
1
1

1
0
0
0
0
0
0
0

5
5
4
4
4

If a module reads its CPUID<MID> register field and obtains 4, it is in M-bus slot 4. If it then reads its
BUSCTI..<6:0> register field and obtains 1011001#2, there are modules in M-bus slots 0, 3, 5, and 7.
To coo.firm presence of a module in each slot, software should read the MODTYPE register of each slot.
In the example just given, the MODTYPE registers would be at VAX addresses 31FFFFFC#l6,
37FFFFFC#l6, 3BFFFFFC#l6, and 3FFFFFFC#l6.
Computation of I/0 address is performed as follows:
:MFPR
ASIIl..
BICL2
ADDL2

#cpuid,m
#23,m,rx
#"X800000,rx
#"X30000000,rx

;read cpuid
;shift left 23
;drop LSB of cpuid
;=SLOT IO space add

6.3.3. lnltlallzatlon

After workstation reset, the KA60 requires initialization of the FBIC and level-2 cache before normal
operation can commence. Firmware will perform the required initialization sequences necessary to get to
base system ROM on the I/0 module and execute the console code. Details on the initialization sequence
can be found in Chapter 13, "Diagnostics."
6.3.4. Base Workstation ROM

The base workstation ROM is located on the workstation 1/0 module. It contains system self-test code and
workstation disk/tape/networlc bootstrap code. Access to the ROM from other modules is relatively slow-typically two microseconds per access. It is recommended that frequently used or time-critical code be
copied to memory.

18 Firefox System Specification

December 21, 1987

Dual-CV AX Proccssqr Module 6.3.4.