DIRECTORYLESS SHARED MEMORY COHERENCE USING EXECUTION MIGRATION

We introduce the concept of deadlock-free migration-based coherent shared memory to the NUCA family of architectures. Migration-based architectures move threads among cores to guarantee sequential semantics in large multicores. Using a execution migration (EM) architecture, we achieve performance comparable to directory-based architectures without using directories: avoiding automatic data replication significantly reduces cache miss rates, while a fast network-level thread migration scheme takes advantage of shared data locality to reduce remote cache accesses that limit traditional NUCA performance. EM area and energy consumption are very competitive, and, on the average, it outperforms a directory-based MOESI baseline by 1.3 and a traditional S-NUCA design by 1.2 . We argue that with EM scaling performance has much lower cost and design complexity than in directorybased coherence and traditional NUCA architectures: by merely scaling network bandwidth from 256 to 512 bit flits, the performance of our architecture improves by an additional 13%, while the baselines show negligible improvement.


Background
Current trends in microprocessor design clearly indicate an era of multicores for the 2010s.As transistor density continues to grow geometrically, processor manufacturers are already able to place a hundred cores on a chip (e.g., Tilera Tile-Gx 100), with massive multicore chips on the horizon; many industry pundits are predicting 1000 or more cores by the middle of this decade [1].Will the current architectures and their memory subsystems scale to hundreds of cores, and will these systems be easy to program?
The main barrier to scaling current memory architectures is the off-chip memory bandwidth wall [1,2]: offchip bandwidth grows with package pin density, which scales much more slowly than on-die transistor density [3].Today's multicores integrate very large shared last-level caches on chip to reduce the number of off-chip memory accesses [4]; interconnects used with such shared caches, however, do not scale beyond relatively few cores, and the power requirements of large caches (which grow quadratically with size) exclude their use in chips on a 1000-core scale-for example, the Tilera Tile-Gx 100 does not have a large shared cache.
For massive-scale multicores, then, we are left with relatively small per-core caches.Since a programming model that relies exclusively on software-level message passing among cores is inconvenient and so has limited applicability, programming complexity considerations demand that the per-core caches must present a unified addressing space with coherence among caches managed automatically at the hardware level.
On scales where bus-based mechanisms fail, the traditional solution to this dilemma is directory-based cache coherence: a logically central directory coordinates sharing among the per-core caches, and each core cache must negotiate shared (read-only) or exclusive (read/write) access to each line via a complex coherence protocol.In addition to protocol complexity and the associated design and verification costs, directory-based coherence suffers from three other problems: (a) directory sizes must equal a significant portion of the combined size of the per-core caches, as otherwise directory evictions will limit performance [5]; (b) automatic replication of shared data significantly decreases the effective total on-chip cache size because, as the core counts grow, a lot of cache space is taken by replicas and fewer lines in total can be cached, which in turn leads to sharply increased off-chip access rates; and (c) frequent writes to shared data can result in repeated cache invalidations and the attendant long delays due to the coherence protocol.
Two of these shortcomings have been addressed by S-NUCA [6] and its variants [7].These architectures unify the per-core caches into one large shared cache, in their pure form keeping only one copy of a given cache line on chip and thus steeply reducing off-chip access rates compared to directory-based coherence.In addition, because only one copy is ever present on chip, cache coherence is trivially ensured and a coherence protocol is not needed.This comes at a price, however, as accessing data cached on a remote core requires a potentially expensive two-message round-trip: where a coherence protocol would take advantage of spatial and temporal locality by making a copy of the block containing the data in the local cache, S-NUCA must repeat the round-trip for every access to ensure sequential memory semantics.Various NUCA and hybrid proposals have therefore leveraged data migration and replication techniques previously explored in the NUMA context (e.g., [8]) to move private data to its owner core and replicate read-only shared data among the sharers at the operating system (OS) level [9,2,10] or aided by hardware [11,12,13], but while these schemes improve performance on some kinds of data, they still do not take full advantage of spatio-temporal locality and require either coherence protocols or repeated remote accesses to access read/write shared data.
To address this limitation and take advantage of available data locality in a memory organization where there is only one copy of data, we propose to allow computation threads to migrate from one core to another at a finegrained instruction level.When several consecutive accesses are made to data assigned to a given core, migrating the execution context allows the thread to make a sequence of local accesses on the destination core rather than pay the performance penalty of the corresponding remote accesses.While computation migration, originally considered in the context of distributed multiprocessor architectures [14], has recently re-emerged at the single-chip multicores level, e.g., [15,16,17], for power management and fault-tolerance, we are unique in using migrations to provide memory coherence.We also propose a hybrid architecture that includes support for SNUCA-style remote access.
Specifically, in this paper we: 1. introduce the idea of using instruction-level execution migration (EM) to ensure memory coherence and sequential consistency in directoryless multicore systems with per-core caches; 2. combine execution migration (EM) with NUCA-style remote memory accesses (RA) to create a directoryless shared-memory multicore architecture which takes advantage of data locality; 3. utilize a provably deadlock-free hardware-level migration algorithm [18] to move threads among the available cores with unprecedented efficiency and generalize it to be applicable to the EM/RA hybrid.

Migration-based memory coherence
The essence of traditional distributed cache management in multicores is bringing data to the locus of the computation that is to be performed on it: when a memory instruction refers to an address that is not locally cached, the instruction stalls while either the cache coherence protocol brings the data to the local cache and ensures that the address can be safely shared or exclusively owned (in directory protocols) or a remote access is sent and a reply received (in S-NUCA).
Migration-based coherence brings the computation to the data: when a memory instruction requests an address not cached by the current core, the execution context (architecture state and TLB entries) moves to the core that is home for that data.As in traditional NUCA architectures, each address in the system is assigned to a unique core where it may be cached: the physical address space in the system is partitioned among the cores, and each core is responsible for caching its region.
Because each address can be accessed in at most one location, many operations that are complex in a system based on a cache coherence protocol become very simple: sequential consistency and memory coherence, for example, are ensured by default.(For sequential consistency to be violated, multiple threads must observe multiple writes in different order, which is only possible if they disagree about the value of some variable, for example, when their caches are out of sync.If data is never replicated, this situation never arises.)Atomic locks work trivially, with multiple accesses sequentialized on the core where the lock address is located.
In what follows, we first discuss architectures based purely on remote accesses and purely on migration, and then combine them to leverage the strengths of both.

Basic remote-access-only (RA) architecture
In the remote-access (RA) architecture, equivalent to traditional S-NUCA, all non-local memory accesses cause a request to be transmitted over the interconnect network, the access to be performed in the remote core, and the data (for loads) or acknowledgement (for writes) be sent back to the requesting core: when a core C executes a memory access for address A, it must 1. compute the home core H for A (e.g., by masking the appropriate bits); 2. if H = C (a core hit), (a) forward the request for A to the cache hierarchy (possibly resulting in a DRAM access); 3. if H = C (a core miss), (a) send a remote access request for address A to core H, (b) when the request arrives at H, forward it to H's cache hierarchy (possibly resulting in a DRAM access), (c) when the cache access completes, send a response back to C, (d) once the response arrives at C, continue execution.
To avoid interconnect deadlock, 1 the system must ensure that all remote requests must always eventually be served.This is accomplished by using an independent virtual network for cache to home core traffic and another for cache to memory controller traffic.Next, within each such subnetwork, the reply must have higher priority than the request.Finally, network messages between any two nodes within each subnetwork must be delivered in the order in which they were sent.In the hybrid EM/RA architecture, memory accesses to addresses not assigned to the local core cause the execution context to be migrated to the core, or may result in a remote data access.

Basic execution-migration-only (EM) architecture
In the execution-migration-only variant (EM), all non-local memory accesses cause the executing thread to be migrated to the core where the relevant memory address resides and executed there.What happens if the target core is already running another thread?One option is to allow each single-issue core to round-robin execute several threads, which requires duplicate architectural state (register file, TLB); another is to evict the executing thread and migrate it elsewhere before allowing the new thread to enter the core.Our design features two execution contexts at each core: one for the core's native thread (i.e., the thread originally assigned there and holding its private data), and one for a guest thread.When an incoming guest migration encounters a thread running in the guest slot, this thread is evicted to its native core.
Thus, when a core C running thread T executes a memory access for address A, it must As with the remote-access-only variant from Section 2.1, cache ↔ memory controller traffic (steps 3 and 4) travels on one virtual network with replies prioritized over requests, and migration messages travel on another.Because DRAM → cache responses arrive at the requesting core, a thread with an outstanding DRAM request cannot be evicted until the DRAM response arrives; because this will always eventually happen, however, the eviction will eventually be able to proceed.Eviction migrations will always complete if (a) each thread T has a unique native core N which will always accept an eviction migration,2 and (b) eviction migration traffic is prioritized over migrations caused by core misses.Since core-miss migrations can only be blocked by evictions, they will also always eventually complete, and the migration protocol is free of deadlock.Finally, to avoid migration livelock, it suffices to require each thread to complete at least one CPU instruction before being evicted from a core.
Because combining two execution contexts in one single-issue core may result in round-robin execution of the two threads, when two threads are active on the core they both experience a serialization effect: each thread is executing only 50% of the time.Although this seems like a relatively high overhead, observe that most of the time threads access private data and are executing on their native cores, so in reality the serialization penalty is not a first-order effect.

Hybrid architecture (EM/RA)
In the hybrid migration/remote-access architecture, each core-miss memory access may either perform the access via a remote access as in Section 2.1 or migrate the current execution thread as in Section 2.2.The hybrid architecture is illustrated in Figure 1.
For each access to memory cached on a remote core, a decision algorithm determines whether the access should migrate to the target core or execute a remote access.Because this decision must be taken on every access, it must be implementable as efficient hardware.In this paper, therefore, we consider and evaluate a simple heuristic scheme: the DISTANCE scheme.If the migration destination is the native core, the distance scheme always migrates; otherwise, it evaluates the hop distance to the home core.It migrates execution if the distance exceeds some threshold d else it makes a round-trip remote cache access.
In order to avoid deadlock in the interconnect, migrations must not be blocked by remote accesses and vice versa; therefore, a total of three virtual subnetworks (one for remote accesses, one for migrations, and one for memory traffic) are required.At the protocol level, evictions must now also wait for any outstanding remote accesses to complete in addition to waiting for DRAM → cache responses.

Migration framework
The novel architectural component we introduce here is fast, hardware-level migration of execution contexts between two cores via the on-chip interconnect network.
Since the core miss cost is dominated by the remote access cost and the migration cost, it is critical that the migrations be as efficient as possible.Therefore, unlike other thread-migration approaches (such as Thread Motion [19], which uses special cache entries to store thread contexts and leverages the existing cache coherence protocol to migrate threads), our architecture migrates threads directly over the interconnect network to achieve the shortest possible migration latencies.
Per-migration bandwidth requirements, although larger than those required by cache-coherent and remoteaccess-only designs, are not prohibitive by on-chip standards: in a 32-bit x86 processor, the relevant architectural state amounts, including TLB, to about 1.5 Kbits [19].In some cases, one may want to migrate additional state, such as branch prediction state and floating point registers, and therefore, we consider both a 1.5 Kbit and 4 Kbit context in Section 4.4.
Figure 2 shows the differences needed to support efficient execution migration in a single-threaded five-stage CPU core.When both context slots (native and guest) are filled, execution round-robins between them to ensure that all threads can make progress.Register files now require wide read and write ports, as the migration logic must be able to unload all registers onto the network or load all registers from the network in relatively few cycles; to enable this, extra muxing logic connects the register files directly with the on-chip network router.The greater the available network bandwidth, the faster the migration.As with traditional S-NUCA architectures, the memory subsystem itself is connected to the on-chip network router to allow for accesses to the off-chip memory controller as well as for reads and writes to a remote cache (not shown in the figure).

Data placement
The assignment of addresses to cores affects the performance of EM/RA in three ways: (a) because context migrations pause thread execution and therefore longer migration distances will slow down performance; (b) because remote accesses also pause execution and longer round trips will also limit performance; and (c) indirectly by influencing cache performance.On the one hand, spreading frequently used addresses evenly among the cores ensures that more addresses are cached in total, reducing cache miss rates and, consequently, off-chip memory access frequency; on the other hand, keeping addresses accessed by the same thread in the same core cache reduces migration rate and network traffic.
As in standard S-NUCA architectures, the operating system controls memory-to-core mapping via the existing virtual memory mechanism: when a virtual address is first mapped to a physical page, the OS chooses where the relevant page should be cached by mapping the virtual page to a physical address range assigned to a specific core.Since the OS knows which thread causes a page fault, more sophisticated heuristics are possible: for example, in a first-touch-style scheme, the OS can map the page to the thread's native core, taking advantage of data access locality to reduce the migration rate while keeping the threads spread among cores.
In EM/RA architectures, data placement is key, as it determines the frequency and distance of remote accesses and migrations.Although placement has been studied extensively in the context of NUMA architectures (e.g., [8]) as well as more recently in NUCA context (e.g., [2]), we wish to concentrate here on the potential of the EM/RA architecture and implement none of them directly.Instead, we combine a first-touch data placement policy [20], which maps each page to the first core to access it, with judicious profiling-based source-level modifications to our benchmark suite (see Section 3.3 and [21]) to provide placement and replication on par or better than that of available automatic methods.

Architectural simulation
We use Pin [22] and Graphite [23] to model the proposed execution migration (EM), remote-access (RA) and hybrid (EM/RA) architectures as well as the cache-coherent (CC) baseline.Pin enables runtime binary instrumentation of parallel programs, including the SPLASH-2 [24] benchmarks we use here; Graphite implements a tile-based multicore, memory subsystem, and network, modeling performance and ensuring functional correctness.
The default settings used for the various system configuration parameters are summarized in Table 1; any deviations are noted when results are reported.In experiments comparing EM/RA architectures against CC, the parameters for both were identical, except for (a) the memory directories which are not needed for EM/RA and were set to sizes recommended by Graphite on basis of the total cache capacity in the simulated system, and (b) the 2-way multithreaded cores which are not needed for cache-coherent baseline.
To exclude differences resulting from relative scheduling of Graphite threads, data were collected using a homogeneous cluster of machines.

On-chip interconnect model
Experiments were performed using Graphite's model of an electrical mesh network with XY routing with 256-bit flits.Since modern network-on-chip routers are pipelined [25], and 2-or even 1-cycle per hop router latencies [26] have been demonstrated, we model a 2-cycle per hop router delay; we also account for the appropriate pipeline latencies associated with loading and unloading a packet onto the network.In addition to the fixed per-hop latency, contention delays are modeled using a probabilistic model similar to the one proposed in [27].

Application benchmarks
Our experiments used a set of SPLASH-2 benchmarks: FFT, LU CONTIGUOUS, OCEAN CONTIGUOUS, RADIX, RAY-TRACE, and WATER-N 2 .For the benchmarks for which versions optimized for cache coherence exist (LU and OCEAN [28,24]), we chose the versions that were most optimized for directory-based cache coherence.It is important to note that these benchmarks have been extensively optimized to remove false sharing and improve working set locality, fitting our requirement for the best-case loads for directory coherence.
Application benchmarks tend not to perform well in RA architectures with simple striped data placements [2], and sophisticated data placement and replication algorithms like R-NUCA [2] are required for fair comparisons.We therefore used the modified SPLASH-2 benchmarks presented in [21] that represent a reference placement/replication scheme through source-level transformations that are limited to rearranging and replicating the main data structures.As such, the changes do not alter the algorithm used and do not affect the operation of the cache coherence protocol.In fact, the modified benchmarks are about 2 % faster than the originals when run on the cachecoherent baseline.
Each application was run to completion using an input set that matched the number of cores used, e.g., we used 4,000,000 keys for RADIX sort benchmark, a 4× increase over the recommended input size.For each simulation run, we tracked the total application completion time, the parallel work completion time, the percentage of memory accesses causing cache hierarchy misses, and the percentage of memory accesses causing migrations.While the total application completion time (wall clock time from application start to finish) and parallel work completion time (wall clock time from the time the second thread is spawned until the time all threads re-join into one) show the same general trends, we focused on the parallel work completion time as a more accurate metric of average performance in a realistic multicore system with many applications.

Directory-based cache coherence baseline selection
In order to choose a directory-based coherence (CC) baseline for comparison, we considered the textbook protocol with Modified/Shared/Invalid (MSI) states as well as two alternatives: on the one hand, data replication can be completely abandoned by only allowing modified or invalid states (MI); on the other hand, in the presence of data replication, off-chip access rates can be lowered via protocol  extensions such as an owned and exclusive states (MOESI) combined with cache-to-cache transfers whenever possible.
To evaluate the impact of these variations, we compared the performance for various SPLASH-2 benchmarks under MSI, MI, and MOESI (using parameters from Table 1).As shown in Figure 3a, MI exhibits by far the worst memory latency: although it may at first blush seem that MI removes sharing and should thus improve cache utilization much like EM/RA, in actuality eschewing the S state only spreads the sharing-and the cache pollution which leads to capacity misses-over time when compared to MSI and MOESI.At the same time, MI gives up the benefits of read-only sharing and suffers many more cache evictions: its cache miss rates were 2.3× greater than under MSI.The more complex MOESI protocol, meanwhile, stands to benefit from using cache-to-cache transfers more extensively to avoid writing back modified data to off-chip RAM, and take advantage of exclusive cache line ownership to speed up writes.Our analysis shows that, while cache-to-cache transfers result in many fewer DRAM accesses, they instead induce significantly more coherence traffic (even shared reads now take 4 messages); in addition, they come at a cost of significantly increased protocol, implementation and validation complexity.Neverthe-less, since our simulations indicate (Figure 3a) that MOESI is the best-performing coherence protocol out of the three, we use it as a baseline for comparison in the remainder of this paper.
Finally, while we kept the number of memory controllers fixed at 8 for all architectures, for the cachecoherence baseline we also examined several ways of distributing the directory among the cores via Graphite simulations: central, one per memory controller, and fully distributed.On the one hand, the central directory version caused the highest queueing delays and most network congestion, and, while it would require the smallest total directory size, a single directory would still be so large that its power demands would put a significant strain on the 256core chip (power demands scale quadratically with SRAM size).On the other end of the spectrum, a fully distributed directory would spread congestion among the 256 cores, but each directory would have to be much larger to allow for imbalances in accesses to cache lines in each directory, and DRAM accesses would incur additional network latencies to contact the relatively few memory controllers.Finally, we considered the case of 8 directories (one for each of the 8 memory controllers), which removed the need for network messages to access DRAM and performed as well as the best-case fully distributed variant.Since the 8-directory configuration offered best performance and a good tradeoff between directory size and contention, we used this design in our evaluation.

Remote-access NUCA baseline selection
To compare against an RA architecture baseline, we considered two approaches: the traditional S-NUCA approach where the L1 and L2 caches are shared (that is, a local L1 or L2 may cache only a subset of the address space), and a hybrid NUCA/coherence approach where private L1 caches are maintained via a coherence protocol.Although the hybrid variant offers some relief from remote accesses to frequently used locations, the L1 caches must keep very large full-map directories (significantly larger than total cache on the core [2]!): if the directories are too small, the L1's will suffer frequent invalidations due to directory evictions and the combined performance will revert towards a remote-access-only design.Based on these considerations we chose to compare our hybrid architecture to a fully shared L1/L2 remote-access-only baseline.

Cache size selection
We ran our SPLASH-2 simulations with a range of cache sizes under both an execution-migration design and the cache-coherent baseline.While adding cache capacity improves cache utilization and therefore performance for both architectures, cache miss rates are much lower for the migration-based approach and, with much smaller on-chip caches, EM/RA achieves significantly better results (Figure 3).When caches are very large, on the other hand, they tend to fit most of the working set of our SPLASH-2 benchmarks and both designs almost never miss the cache.This is, however, not a realistic scenario in a system concurrently running many applications: we empirically observed that as the input data set size increases, larger and larger caches are required for the cache-coherent baseline to keep up with the migration-based design.To avoid bias either way, we chose realistic 64 KB L2 data caches as our default configuration because it offers a reasonable performance tradeoff and, at the same time, results in 28 Mbytes of on-chip total cache (not including directories for CC).

Instruction cache
Since the thread context transferred in an EM architecture does not contain instruction cache entries, we reasoned that the target core might not contain the relevant instruction cache lines and a thread might incur an instruction cache miss immediately upon migration.To evaluate the potential impact of this phenomenon, we compared L1 instruction cache miss rates for EM and the cache-coherent baseline in simulations of our SPLASH-2 multithreaded benchmarks.
Results indicated an average instruction cache miss rate of 0.19% in the EM design as compared to 0.27% in the CC baseline.The slight improvement seen in EM is due to the fact non-memory instructions are always executed on the core where the last memory access was executed (since only another memory reference can cause a migration elsewhere), and so non-memory instructions that follow references to shared data are cached only on the core where the shared data resides.

Area and energy estimation
For area and energy, we assume 32 nm process technology and use CACTI [29] to estimate the area requirements of the on-chip caches and interconnect routers.To estimate the area overhead of extra hardware context in the 2way multithreaded core for EM, we used Synopsys Design Compiler [30] to synthesize the extra logic and registerbased storage.We also use CACTI to estimate the dynamic energy consumption of the caches, routers, register files, and DRAM.The area and dynamic energy numbers used in this paper are summarized in Table 2.We implemented several energy counters (for example the number of DRAM reads and writes) in our simulation framework to estimate the total energy consumption of running SPLASH-2 benchmarks for both CC and EM.Note that DRAM only models the energy consumption of the RAM and the I/O pads and pins will only add to the energy cost of going off-chip.

Results and analysis 4.1 Advantages over directory-based cache coherence
In the EM architecture, each address-even shared by multiple threads-is assigned to only one cache, leaving more total cache capacity for other data.Because the additional capacity arises from not storing addresses in many locations, cache miss rates naturally depend on the memory access pattern of specific applications; we therefore measured the differences in cache miss rates for several benchmarks between our EM/RA designs and the CC baseline.(Note that the cache miss rates are virtually identical for all our EM/RA designs).The miss rate differences in realistic benchmarks, shown in Figure 4, are attributable to two main causes.On the one extreme, the FFT benchmark does not exhibit much sharing and the high cache Cache miss rates illustrate the core potential advantage of EM/RA designs over CC: significantly lower offchip access rates given the same cache sizes.Although miss rates in CC architectures can be reduced by increasing the per-core caches, our simulation results (not shown here) indicate that, overall, the CC design would need in excess of 2× the L2 cache capacity to match the cache miss rates of EM/RA.

Advantages over traditional directoryless NUCA (RA)
Although RA architectures eschew automatic sharing of writable data and significantly lower cache miss rates, their main weakness lies in not being able to take advantage of shared data locality: even if many consecutive accesses are made to data on the same remote core, sequential consistency requires that each be an independent round-trip access.To examine the extent of this problem, we measured the run length for non-local memory access: the number of consecutive accesses to memory cached in a non-local core not interrupted by any other memory accesses.
Figure 5 shows this metric for one of our benchmarks.Predictably, the number of remote accesses with run length of one (a single access to a remote core followed by access to another remote core or the local core) is high; more significantly, however, a great portion of remote memory accesses in both benchmarks shown exhibit significant core To examine the real improvement potential offered by extending RA with efficient execution migrations, we next counted the core miss rates-the number of times a roundtrip remote-access or a migration to a remote core must be made-for the RA baseline and our EM architecture.
Figure 6a shows core misses across a range of benchmarks.As we'd expect from the discussion above (Section 4.2), OCEAN CONTIGUOUS and LU CONTIGUOUS show that migrations significantly lower core miss rates, and most other benchmarks also improve.The outlier here is FFT: most of the accesses it makes are to each thread's private data, and shared accesses are infrequent and brief.
Figure 6b shows how many overall core misses were handled by remote accesses and migrations in several EM/RA variants.In the EM/RA scheme that performed best, namely, EM(distance=11), see Figure 7a, both migrations and remote access play a significant role, validating our intuition behind combining them into a hybrid architecture.
It is important to note that the cost of core misses is very different under RA and under EM: in the first, each core miss induces a round-trip remote access, while in the second it causes a one-way migration (the return migration, if any, is counted as another core miss).Adding efficient migrations to an RA design therefore offers significant performance potential, which we examine next.

Overall area, performance and energy
The EM/RA architectures do not require directories and as can be seen from Table 2 are smaller than the CC baseline.
Figure 7a shows the parallel completion time speedup relative to the CC baseline for various EM/RA schemes: a remote-access-only variant, a migrations-only variant, and a range of hybrid schemes where the remote-access vs. migration decision is based on on hop distance.Overall, performance is very competitive with the cache coherent baseline and the EM and best EM/RA design EM(distance=11) show an average 1.3× and 1.25× improvement over the CC baseline, respectively.EM/RA with too small (large) a distance threshold does not perform as well because there are too many (few) migrations.For the chosen context size and hop latency, when distance ≥ 11 migrations become cheaper than data word round-trips resulting in the best performance.Additionally, EM is 1.2× higher performance than the RA baseline.
The benefits are naturally application-dependent: as might be expected from Figure 4, the benchmarks with the largest cache miss rate reductions (FFT and OCEAN CONTIGUOUS) offer the most performance improvements.At the other extreme, the WATER benchmark combines fairly low cache miss rates under CC with significant read-only sharing, and is very well suited for directory-based cache coherence; consequently, CC outperforms all EM/RA variants by a significant margin.
The result also shows the benefits of a combined EM/RA architecture: in some benchmarks (e.g., RADIX, LU CONTIGUOUS, OCEAN CONTIGUOUS), a migrationonly design significantly outperforms remote accesses, while in others (e.g., WATER-N 2 ) the reverse is true.On average, the best distance-based EM/RA hybrid performs better than either EM or RA, and renders the EM/RA approach highly competitive with directory-based MOESI cache coherence.
Since energy dissipated per unit performance will be a critical factor in next-generation massive multicores, we employed an energy model (cf.Section 3) to estimate the dynamic energy consumed by the various EM/RA variants and CC.On the one hand, migrations incur significant dynamic energy costs due to increased traffic in the on-chip network and the additional register file per core; on the other hand, dramatic reductions in off-chip accesses equate to very significant reductions in DRAM access energy.
As illustrated in Figure 7b, energy consumption depends on each application's access patterns.For FFT, for example, which incurs crippling rates of eviction invalidations, the energy expended by the CC protocol messages and DRAM references far outweighs the cost of energy used by remote accesses and migrations.On the other extreme, the fairly random patterns of memory accesses in RAYTRACE, combined with a mostly private-data and read-only sharing paradigm, allows CC to efficiently keep data in the core caches and consume far less energy than EM/RA.The high cost of off-chip DRAM accesses is particularly highlighted in the WATER-N 2 benchmark: although the trend in cache miss rates between CC and EM/RA is similar for WATER-N 2 and RAYTRACE, the overall cache miss rate is markedly higher in WATER-N 2 ; combined with the associated protocol costs, the resulting offchip DRAM accesses make the CC baseline consume more energy than the EM/RA architecture.
We note that our energy numbers for directory-based coherence are quite optimistic, since we did not include energy consumed by I/O pads and pins; this will result in higher energy for off-chip accesses which CC makes more of.

Performance scaling potential for EM designs
Finally, we investigated the scaling potential of the EM architecture.We reasoned that, while directory-based coherence is limited by cache sizes and off-chip bandwidth and RA performance is restricted by interconnect latencies, EM can be improved by increasing interconnect band- width (network link and router bandwidth can be scaled by widening data paths, allowing more bits to be transferred in a single cycle): with higher on-chip network bandwidth, the main effect is that messages carrying the thread's context consume fewer cycles.
With this is mind, we evaluated our 256-core system with a much larger context size of 4 Kbits and compared EM against CC at our default 256 bit-flit network as well as a higher bandwidth 512 bit-flit network.As illustrated in Figure 8, the performance of EM dropped from a 1.3× advantage over CC (using 1.5 Kbits context size) to a 1.1× advantage.When the network bandwidth was doubled to a 512 bit-flit size, EM outperformed CC by 1.25×.Since scaling of network bandwidth is easy-although buffers and crossbars must be made wider so area increases linearly, the fundamental design of the interconnect remains constant and the clock frequencies are not appreciably affected.Moreover, since the same amount of data must be transferred, dynamic energy consumption does not grow in tandem.Contrasted with the off-chip memory bandwidth wall and quadratically growing power requirements of large caches limiting directory-based architecture performance, and the difficulty in reducing electrical network hop counts limiting remote-access performance on the other hand, an EM or EM/RA architecture offers an attractive way to significantly increase performance at sublinear impact on cost.
5 Related work

Thread migration
Migrating computation to the locus of the data is not itself a novel idea.Hector Garcia-Molina in 1984 suggested moving execution to data in memory bound architectures [14].Nomadic threads reduce the number of messages needed to access data in multi-chip computers [31], and [32] uses migration to improve spatial locality of distributed array structures.
In recent years migrating execution context has reemerged in the context of single-chip multicores.Michaud shows the benefits of using execution migration to improve the overall on-chip cache capacity and utilizes this for migrating selective sequential programs to improve performance [15].Computation spreading [17] splits thread code into segments and assigns cores responsible for different segments, and execution is migrated to improve code locality.Kandemir presents a data migration algorithm to address the data placement problem in the presence of nonuniform memory accesses within a traditional cache coherence protocol [16].This work attempts to find an optimal data placement for cache lines.A compile-time program transformation based migration scheme is proposed in [33] that attempts to improve remote data access.Migration is used to move part of the current thread to the processor where the data resides, thus making the thread portion local; this scheme allows programmer to express when migration is desired.Dataflow machines (e.g., [34])-and, to some extent, out-of-order execution-are superficially similar as they allow an activated instruction to be claimed by any available execution unit, but cannot serve as a sharedmemory abstraction.The J-machine [35] ties processors to on-chip memories, but relies on user-level messaging and does not address the challenge of off-chip memory bandwidth.Our proposed execution migration machine is unique among the previous works because we completely abandon data sharing (and therefore do away with cache coherence protocols).In this paper we have proposed to rely on execution migration to provide coherence and consistency.

Remote-access NUCA and Directory Coherence
Remote memory access is performed in S-NUCA [6] and its variants [7]: these architectures unify the per-core caches into one large shared cache, in their pure form keeping only one copy of a given cache line on chip and thus steeply reducing off-chip access rates compared to directory-based coherence; in addition, because only one copy is ever present on chip, cache coherence is trivially ensured without a coherence protocol.This comes at the price of a potentially expensive two-message round-trip as mentioned in the introduction.Various NUCA and hybrid proposals have therefore leveraged data migration and replication techniques previously explored in the NUMA context (e.g., [8]) to move private data to its owner core and replicate read-only shared data among the sharers at the OS level [9,2,10,36] or aided by hardware [11,12,13], but while these schemes improve performance on some kinds of data, they still do not take full advantage of spatiotemporal locality and require either coherence protocols or repeated remote accesses to access read/write shared data.In this paper, we use source-level transformations in conjunction with a first-touch scheme to obtain good data placements, and instruction-level thread migration to take advantage of locality.
We have also compared against MOESI directorybased coherence which is state-of-the-art and includes cache to cache transfers to minimize off-chip memory accesses (e.g., [37], [38]).Some recent work has addressed reducing directory sizes required for cache coherence in large-scale multicores (e.g., [39], [40]).These schemes typically tradeoff performance or protocol complexity for reduced directory area.

Conclusion
In this paper, we have extended the family of directoryless NUCA architectures by adding efficient, hardwarelevel core-to-core thread migrations as a way to maintain sequential consistency and memory coherence in a large multicore with per-core caches.Taking advantage of locality in shared data accesses exhibited by many applications, migrations amortize the cost of remote accesses that limit traditional NUCA performance.At the same time, an execution migration design retains the cache utilization benefits of a shared cache distributed among many cores, and brings NUCA performance up to the level of directorybased cache-coherent designs.
We have demonstrated that appropriately designed execution migration (EM) and remote cache access (RA) hybrid designs do not cause deadlock.We have explored very straightforward hybrid EM/RA architectures in this paper; future work involves the development of betterperforming migration predictors.Perhaps most promisingly, we have shown that the performance of EM designs is relatively easy to improve with low area cost, little power overhead, and virtually no verification cost, allowing chip designers to easily select the best compromise for their application space.

Figure 1 .
Figure1.In the hybrid EM/RA architecture, memory accesses to addresses not assigned to the local core cause the execution context to be migrated to the core, or may result in a remote data access.
Figure 2. (a) A single-issue five-stage pipeline with efficient context migration; differences from a single-threaded pipeline are shaded.(b) For a context transfer, the register file of the originating core is unloaded onto the router, transmitted across the network and finally loaded onto the home core's register file via the router.

Figure 3 .
Figure 3. (a) Although parallel completion time for different coherence protocols varies somewhat across the benchmarks (notably, the high directory eviction rate in FFT leads to rampant invalidations in MSI and MOESI and favors MI), generally MOESI was the most efficient protocol and MI performed worst.(b) Cache hierarchy miss rates at various cache sizes show that, by eschewing replication, the EM/RA architecture achieves cache miss rates much lower than the CC baseline at all cache sizes.(Settings from Table 1.)

Figure 4 .
Figure 4.For our benchmarks, under our EM/RA designs the cache miss rates are on the average 3.2× lower because storing each cache line in only one location eliminates many capacity and coherence-related evictions and effectively increases the availability of cache lines.

Figure 5 .
Figure 5. Non-local memory accesses in our RA baseline binned by the number of surrounding contiguous accesses to the same remote core.Although, predictably, many remote operations access just one address before accessing another core, a surprisingly large number belong to streaks of 40-50 accesses to the same remote core and indicate significant data locality.locality and come in streaks of 40-50 accesses.Although core locality is not this dramatic in all applications, these examples show precisely where a migration-based architecture shines: the executing thread is migrated to a remote core and 40-50 now effectively "local" memory accesses are made before incurring the cost of another migration.To examine the real improvement potential offered by extending RA with efficient execution migrations, we next counted the core miss rates-the number of times a roundtrip remote-access or a migration to a remote core must be made-for the RA baseline and our EM architecture.Figure6ashows core misses across a range of benchmarks.As we'd expect from the discussion above (Section 4.2), OCEAN CONTIGUOUS and LU CONTIGUOUS show that migrations significantly lower core miss rates, and most other benchmarks also improve.The outlier here is FFT: most of the accesses it makes are to each thread's private data, and shared accesses are infrequent and brief.Figure6bshows how many overall core misses were handled by remote accesses and migrations in several EM/RA variants.In the EM/RA scheme that performed best, namely, EM(distance=11), see Figure7a, both migrations and remote access play a significant role, validating our intuition behind combining them into a hybrid architecture.It is important to note that the cost of core misses is very different under RA and under EM: in the first, each

Figure 6 .
Figure 6.The potential for improvement over the RA baseline.(a) When efficient core-to-core thread migrations are allowed, the number of memory accesses requiring transition to another core (core misses) significantly decreases.(b) The fraction of core miss rates handled by remote accesses and migrations in various migration/remote-access hybrids shows that the bestperforming scheme, EM(distance=11), has significant migration and remote access components.

Figure 7 .Figure 8 .
Figure 7. (a) The performance of EM/RA variants relative to CC.Although results vary significantly by benchmark, the best EM/RA scheme, namely, EM(distance=11) outperforms CC by 1.25× on average.(b) Dynamic energy usage for all EM/RA variants improve compared to CC.Note that for energy consumption, CC and RA assume a 128b flit size, while EM and EM/RA hybrid variant utilizes a higher bandwidth network with 256b flit sizes.

Table 2 .
Area and energy estimates since in the EM/RA design the caches are only subject to capacity misses, the cache miss rate falls to under 2%.At the other end of the spectrum, the OCEAN CONTIGUOUS benchmark does not incur many directory evictions but exhibits significant read-write sharing, which, in directory-based cache coherence (CC), causes mass invalidations of cache lines actively used by the application; at the same time, replication of the same data in many per-core caches limits effective cache capacity.This combination of capacity and coherence misses results in a 5% miss rate under MOESI; the EM/RA architecture eliminates the coherence misses and increases effective cache capacity, and only incurs a 0.8% miss rate.The remaining benchmarks fall in between these two extremes, with a combination of directory evictions and read-write sharing patterns.