Cell SPE Task Library

This project contains a thread-safe digital signal processing and computational microtask execution library for the IBM Cell Broadband Engine. It works in Linux and the IBM Cell SDK 2.1 on the Playstation3 and Cell Blade. The project consists not really of a library but of source code and header files that you can include in your Cell program. The license is GNU LGPL.

The SPE program is started once on 1..N SPEs, where it will stay running and waiting for tasks that can be sent to it from your PPU program. The running SPE program(s) can be sent different such computing tasks. Tasks can be single or a chained queue of tasks, for example of different types. The SPE program uses faster mailbox communication, and latencies from task start to end on the PPU side are quite low. There is an additional framework for PPU code that makes accessing the computing SPEs and their tasks threadsafe.

The set of computational tasks (commands) can be easily extended with your own funcs. The code allows more control over processing, DMAs and buffering than the IBM ALF framework.

Another point for this project is to collect a bag of computational tricks and fast DSP funcs, in the form of cut&paste source code snippets that you can can copy into your own SPE programs. These code pieces include very fast sine and cosine calculation, complex multiply accumlate, and others.

1. Status
2. Source code access
3. Source code files
4. Short usage description
5. General Cell programming tips

Currently all processing results are transferred out from the SPE to an EA address (RAM or other SPE local store). In the default tasks included in the code,

Source code access

No binaries are distributed, only source code. It is available from CVS and should be compiled on the target system. You will need the IBM Cell SDK 2.1 and a Playstation3 or Cell Blade. Other Cell platforms are untested.

Anonymous CVS checkout:
$ cvs -z3 -d:pserver:anonymous@cellspe-tasklib.cvs.sourceforge.net:/cvsroot/cellspe-tasklib co -P cellspe-tasklib

ViewCVS repository source:
http://cellspe-tasklib.cvs.sourceforge.net/cellspe-tasklib/cellspe-tasklib/

Source code files

The first files are all part of the Cell DSP Task Library and framework. The second batch are individual subprojects that actually do something, but may or may not be using that framework.

Short Cell DSP Task Library usage description

  291 // Execute sine cosine generator (oscillator), amplitude is not normalized!!
  292 int speSinCosGenerator(float startphase, float increment, fc32* cos_sin_out, int numiter)
  293 {
  294    // init
  295    CellDSPTaskClass scgen(CELLDSP_TASK_SINCOS_GENERTR);
  296    scgen.addOutVec((void*)cos_sin_out, numiter*sizeof(cf32));
  297    CellDSPParams_Phase_tt * aux = &(scgen.getSubcontext()->Phase);
  298    aux->argcount = numiter;
  299    aux->phase_start = startphase;
  300    aux->phase_delta = increment;
  301    // perform
  302    return celldsp_executeTask(scgen.getTask());
  303 }

IBM articles: Cell Broadband Engine processor DMA engines
Part 1: The little engines that move data
Part 2: From an SPE point of view

More IBM docs (especially see Cell Broadband Engine Architecture):
Cell Broadband Engine documentation

DMA direction: each of the MFC's can handle bidirectional 2x25GB/s. Each core has one MFC, utilize them all! For severe throughput improvement, instead of your PPU program DMAing data to SPE LS's, tell the SPE programs where the data is (EA address) and let the SPE's use their own MFC to DMA the data into LS. The EA address you can pass in a 32-bit mailbox message or in a task context.

DMA lists: mfc_getl(),mfc_putl(), useful if you need to transfer more than the 16kB length limit for a single DMA allows. The DMA list implementation on the Cell processor is a quite cut-down version of what other DMA engines have. Each of the maximally 2048 list entries tells the DMA engine the amount of data to transfer and a 32-bit EA (caveat, not 64-bit!) from where to get or where to put the data. The transfer is started mfc_getl/mfc_putl() with a single LS address and the head of the DMA list. Unfortunately the data array at the LS address must always be contiguous, only in EA/RAM the data can be scattered into blocks, so scatter-gather works only outside of the SPE. To fill several different SPE-side arrays with one List DMA transfer, place them next to each other in LS by e.g. putting them into the same struct. The list entries also have a nice Stall-and-notify flag.

DMA of code: program and data are in the same memory so there's no read-only program memory. Hence you can DMA data from RAM into an array on the SPE and start executing it. This is called overlay programming. For example vector char newCode[512]; void (*func)(int)=&newCode[0]; the transfer data into newCode[] and start running it with func(0);.

DMA from SPE to SPE: if SPE processing generates intermediate results that will be processed on another SPE, try to avoid placing results back into RAM in between. The slow and high latency Cell external RAM may cut down the performance a lot. Better utilize the fast 300GB/s on-chip EIB bandwidth by doing DMAs from one SPE to another. For this to work, in the PPU code the SPE programs have to be started with direct problem state access (SPE_MAP_PS flag). Then, still on the PPU, use spe_get_ls(speid_t id) to get the EA address at which this SPEs LS starts. Hand these EA's to your SPE program via e.g. your task context. If the SPEs are running the same binary program and you are using static/global arrays, then the LS addresses of arrays are the same on every SPE! If arrays are e.g. outbuf[], inbuf[] then simply do a DMA mfc_put(&outbuf[0], otherSPE_EA + (unsigned long long)&inbuf[0], length, tag, 0, 0) or similar style.

Atomic DMA: all transfers ≤128 bytes are atomic, unlike with larger transfers, where there is no guarantee that all the data is written in one go and in sequential order. Atomic DMA to some EA address is useful for example for inter-SPE synchronization. While the SPEs have zero cache (no L1, L2), the MFC has 5 cache entries of 128 bytes each (cache line size). The SPEs MFC are indirectly aware of what addresses are in the other SPE's MFC cache. The atomic transfers will land in or come from the cache of the SPU or PPU MFC first, no slow RAM access involved. If you access that same EA address from all SPEs, the data reads and writes will stay on-chip and be very fast, ideal for SPE signaling and synchronization. Writing to this shared EA is safe by using lock line reservation - get-lock-line-and-reserve e.g. getllar() and a conditional write e.g. putllc that fails if the lock was lost.

Floating point (FP) precision: early versions of Cell have comparably very slow (~1/10th) double-precision units while single-precision FP performs fastest. 'Double' is accurate. The 'float' is less accurate than expected - it does non- IEEE-754 rounding and you may loose roughly 2-3 effective bits compared to normal 32-bit IEEE 'float'. Newer Cell (eDP, PowerXCell, ...) typically have 'double' that is about half the speed of 'float' and is accurate. Consider what minimum FP accuracy your task requires. You may have to trade speed versus precision.

Code profiling: if you are lucky enough to have a Cell Blade, you can use oprofile to profile SPE code. Playstation owners have to resort to call spu_write_decrementer(), spu_read_decrementer() timing functions inside SPE code, because the kernel (as of 2.6.23) still does not support oprofile profiling for PS3 Cell. There is a free Cell emulator from IBM that you could in theory use to profile SPE code performance. The emulator is necessarily slow and not very useful to profile real program runs with large data. To profile your PPU code you can use oprofile or gprof as usual.

Efficient computation: the PPU has something akin to hyperthreading and has Altivec but the performance is very poor. The PPU is useful mainly for I/O and starting SPE tasks. The real computation you will have to run on the memory-limited SPEs. Every SPE has its own MFC DMA engine, so have the SPEs perform their own DMA transfers of data from/to main memory or other SPUs. Don't try to use the PPUs single MFC to distribute data to each of the SPEs.

Try to keep data on-chip as long as possible and use SPE to SPE DMA transfers if applicable - try to avoid writing back temporary results to the slow and high-latency RAM only to reload it from there later. Preferrably move only the really final results back into RAM, this way you get the most out of the 300 GB/s aggregate on-chip bus performance.

To keep the computation fluid, use double buffering on the input and output data buffers where possible. Essentially, during current computation you should have ongoing DMAs that transfer out the previous results and transfers in the next input data (from/to RAM or another SPE). If you have several inputs it may mean you have to make buffers small to fit into the small SPE memory, or reuse buffers for other input data (array, vector ariable) as soon as computation no longer requires the current buffer data.

There is the restrict C keyword for pointers, it can come in handy and generate faster code...

Some operations on large matrices and all-to-all computation such as long Fourier transforms where data does not fit into SPE memory are unfortunately hard to implement. It may be possible to use another library and tell it to use just n out of N SPE cores, like FFTW allows to. The other SPEs running your program can then fetch the result from main RAM. Or you can alternate loading the library and then your program onto the SPEs, though this obviously adds large SPE task switch overhead.

About branching, the SPE has no dynamic branch prediction nor does it extensively support static branch hints. Branch optimizations have to be done in the source code. You can use gcc's __builtin_expect() to give the compiler some branch hints, so it can generate hbr, hbra, hbrr assembler hint instructions. See the branch elimination tutorial on cellperformance.com for details. You can for exampleeliminate certain branches by doing the two (asssumed trivial) computations A, B inside the the if/else simultaneously, then select which of the two results A,B you want by using spu_select(A,B,m) and a bit mask m generated by spu_...() comparison functions. The trick can be applied in some cases, in many others not. There are more tricks, see the tutorial.

Make better use of the 128 registers on the SPU. When data is loaded from an address into the register, try to perform most computations on the data in the registers before storing the result back to SPU memory. In your C code, have a section where you place the data into temporaries, then do all computations on the temporaries and finally place the data back into the main variables or arrays. Also try to reduce loop tests and make use of the available registers by doing loop unrolling - this means you should insert the same code for example four times into the for loop and cut the max loop iterations into one fourth. There are two ways for unrolling:

#define UNROLL_BY_4(x) {x}{x}{x}{x} for (int i=0; i<N/4; i++) { UNROLL_BY_4( \ *out = spu_mul(*in, coeff); \ in++; out++; \ ); }

for (int i=0; i<N; ) { const int unroll_factor = 4; for (int j=0; j<unroll_factor; j++) { out[j] = spu_mul(in[j], coeff); } in +=unroll_factor; out +=unroll_factor; i += unroll_factor; }

The first generates less instructions, though the latter may be more readable. Note the requirement that N is evenly divisible by 4. You should experiment with different loop unroll factors (2,4,6,8,...) especially when your code is near final. During code changes an earlier good unroll factor may cause slowdown in newer code.

TODO: more

Project admin and developer: Jan Wagner,