Colin DP

• type
  • 0- bounce backs (bulk fluids nodes)
  • 1- bounce backs (pit geometry boundary fluid nodes)
  • 2- not in fluid
• orientation (number of bounce backs)
  • 0- no bounce back (couldn't this be used for bulk fluid?)
  • 3- not sure – all the diagonals on a face minus 1?
  • 4- not sure – two faces (inside edge) minus all furthest diagonals?
  • 5- against a face
  • 6- not sure – seems to be face and one of the opposite diagonals?
  • 8- against two faces (inside edge)
  • 10 - against three faces (inside corner)

• Master branch (fracnes/final-fixes tag/frances branch + one commit from Colin) is a subset of tyson branch.

• MPI branch only uncomments the transfer and exchange calls.

• Need to set comm_modify cutoff to 2.5x dx to have required particles
• [X] Periodic MPI deadlocks in first steps
• [X] Crash at end-of-simulation
• [X] Split along z is okay, split along y has agreement issues, and split along z looses atoms

• fixviscouslb wasn't initialized if unused so fluid force could occur at different times

• comm was being destructed before fixes (modify)
• Upstream pull request to fix – from discussion should switch to post_run hook

• Constructor sets bogus fluid distribution exchange in flight
• setup (called at the start of a run) finishes the one if flight, throws it away, computes the proper one for the run, and starts it in flight
• Step routines expert fluid distribution in flight at start and put it inflight at end
• Destructor finishes fluid distribution exchange in flight

• [X] Get particle dumping working to visualize in paraview
• [X] Particles are lost when splitting x
  • Code for pressurebcx looks like it incorrectly applies to the internal side too for boundary processes
• [X] Fix pressurebcx applying to internal side for boundary processes
• [X] Forces are off when splitting x or y [9/9]
  • [X] Test with individual atoms, bodyforce, and no pressurebcx: fail
    • Issue when split along fixed boundary side, otherwise perfect agreement
  • [X] Test with rigid sphere, bodyforce, and no pressurebcx: pass
  • [X] Test with rigid sphere, bodyforce, no pressurebcx, and pits: pass
  • [X] Test without particle interactions: fail (not caused by particle interaction code)
  • [X] Figure out how to do paraview visualization of differences
    • Append attributes filter on multiple inputs and then calculator for relative error
    • Minimal size test example
    • Error starts in split and then jumps to ends as well
  • [X] Double checked EDGE_Z{0,1} usage for acting on both sides
  • [X] Test with all edge code disabled: failed
  • [X] Test with 3 processors along z: error on both boundaries, starting on left outer one: error on both sides of ends
  • [X] Looks into end extra point old adjustment code
• [X] Figure out why splitting along fixed boundary gives different results for MPI [3/3]
  • [X] Add boundary dump code after each GPU routine call
  • [X] Create a serial vs parallel dump comparison program
  • [X] Extend boundary dump to entire field
• [X] Fix discovered geogrid issue (requires 3 boundary points and not 2)
  • [X] Add additional boundary point to calculation
    • [X] sublattice initialization code uses wholelattice wrap of Nbz: correct as only applies to boundary
    • [X] Update dump routines
    • [X] Update gpu routines to use 3 boundary offsets value and names

• [X] fluid_dist_eq_initial_3_kernel
• [X] fluid_dist_eq_next_0_kernel
• [X] fluid_dist_eqn_next_3_kernel
• [X] fluid_dist_initial_2_kernel
• [X] fluid_dist_next_2_kernel
• [X] fluid_param_initial_2_kernel
• [X] fluid_param_next_2_kernel
• [X] fluid_correctu_next_2_kernel
• [X] xboundaries_fluid_dist_eq_kernel
• [X] yboundaries_fluid_dist_eq_kernel
• [X] zboundaries_fluid_dist_eq_kernel
• [X] xboundaries_fluid_dist_kernel
• [X] yboundaries_fluid_dist_kernel
• [X] zboundaries_fluid_dist_kernel
• [X] xboundaries_fluid_force_kernel
• [X] yboundaries_fluid_force_kernel
• [X] zboundaries_fluid_force_kernel
• [X] remove_momentum_kernel

• [X] fluid_dist_3_1_interior_read
• [X] fluid_dist_eq_3_3_interior_read
• [X] fluid_force_2_exterior_read
• [X] fluid_force_2_accumulate_read
• [X] fluid_dist_3_1_exterior_write
• [X] fluid_dist_eq_3_3_exterior_write
• [X] fluid_force_2_accumulate_write

• [X] Break appart commits and bisect
  • Accidentally removed queue assignment in clCreateCommandQueue call

• [X] C++ template magic class to progressively push location information
  • Dreadful failure
• [X] Abstract with profile records: location, rank, info, value
• [X] Resolve race conditions/locking by dumping different ranks to different files
• [X] Switch to binary format for space
  • Compact everything into a location field by combining bits
  • Reduce data required to pass between functions
  • Replace strings with enums

• [X] GPU timer requires OpenCL 2.1 (no NVIDIA), use CPU timer instead
• [X] initial_integrate
• [X] pre_force
• [X] post_force
• [X] final_integrate
• [X] atom_read
• [X] atom_write
• [X] fluid_force_accumulate

• [X] Python code extract binary data format
• [X] Switch to R as python pandas apply functionality too slow
  • nearly 8 minutes in python pandas vs 2 seconds in R tidyverse
  • don't have a good way to load data: use python to save as a feather file
• [X] Initial plot of timestep
  • transfer and atom related routines seem to be taking most time

• [X] Distinguish between two uses of fluid_dist_read/write routines
• [X] Fix non-unique fluid_dist_read/write profile points (eq vs non-eq)

• [X] Plot of walltime for each step (group analysis)
  • Neighbour calculation is very expensive (x100 over regular step)
• [X] Distributions of walltime for each group of GPU calls (inter analysis)

• [X] Redo implicit ordering calculation code (no longer simple)
  • Add substep calculation as CPU and GPU out of order wrt substep
• [X] Syncronize CPU and GPU clocks from data
  • Have to correct for slight clock skew as well as offset (linear model)
• [X] Add special cases to handle missing events in CPU only profiles

• 2 nodes x 1 cpu & gpu/processor (p100): 1000 steps of in.polymer_sphere loop time 96.5 -> 84.3 (-12.6%)
• [X] Add memory for copy out space (put in end)
• [X] Adjust memory resize routines
• [X] Create new GPU function atom_unsort_position2_z and remove old ones
• [X] Add kernel variables and initialize and deinitialize
• [X] Add local and global size variables and initialize
• [X] Calls for initial arguments
• [X] Update all arguments (need to include force and index too!) on atom resize
• [X] Add profile enums and remove old ones
• [X] Call to invoke with final arguments
• [X] Update atom_read

• 2 nodes x 1 cpu & gpu/processor (p100): 1000 steps of in.polymer_sphere loop time 84.3 -> 73.5 (-12.8%)
• [X] Add memory for copy out space (put in end)
• [X] Adjust memory resize routines
• [X] Add new GPU function atom_resort_position2_z
• [X] Add kernel variable and initialize and de-initialize
• [X] Add local and global size variables and initialize
• [X] Calls for initial arguments
• [X] Update all arguments (need to include index too!) and local and global size on atom resize
• [X] Add profile enum
• [X] Call to invoke with final arguments
• [X] Add atom_rewrite

• 2 nodes x 1 cpu & gpu/processor (p100): 1000 steps of in.polymer_sphere loop time 73.5 -> 69.0 (-6.1%)
• If no forces added via fix_lb_viscous_gpu and or fix_lb_rigid_pc_sphere_gpu could skip a bunch of atom routines including correct_u
• The positions don't change so the sort mapping is fixed throughout
• The 1/2 step velocity is the correct one to be using and not the 1 step as is being done
• Push atom_force down down to post_force (more lammps standard and would like to be able to add forces into GPU routine)
• Technically correct_u in is a final_integrate sort of thing so put it there

• Steps overview

  Routine	Operation
  fluid_dist_exchange_finish-1	Recieve fluid_dist_3_new- exterior [G+S+F] (for previous step)
  fluid_dist_eq_exchange_finish	Recieve fluid_dist_eq_3_new exterior [1..G+S+F]
  fluid_dist_eqn_next_3	fluid_dist_eq_3_new [0..G+S+F] -> fluid_dist_eqn_3_new [0..G+S+F]
  fluid_dist_next_2	fluid_dist_~{eq,eqn}_3_{new,old} [0..G+S+F], ~fluid_dist_3_old [0..G+S+F], fluid_density_2 [..F] -> fluid_dist_3_new [0..G+S]
  restartWrite	fluid_dist_eq_3_new [0..G+S+F], fluid_dist_3_new [0..G+S] -> restart file (if requested)
  fluid_dist_exchange_start	Send fluid_dist_3_new interior [-G-S-F]
  fluid_param_next_2_start	Start fluid_dist_3_new [0..G+S] -> fluid_velocity_2 [0..G+S], fluid_density_2 [0..G+S]
  fluid_param_next_2_finish	End fluid_dist_3_new [0..G+S] -> fluid_velocity_2 [0..G+S], fluid_density_2 [0..G+S]
  atom_write	Atom [G] -> atom_position2 [G], atom_velocity [G], atom_type [G], atom_mass [G]
  atom_position2_z_index	atom_position2 [G] -> atom_position2_z [G], atom_index [G]
  atom_sort_position2_z	Sort by z atom_position2 [G], atom_velocity [G], atom_mass [G], atom_type [G], atom_position2_z [G], atom_index [G]
  atom_force	fluid_velocity_2 [0..G+S], fluid_density_2 [0..G+S], atom_position2 [G], atom_velocity [G], atom_mass [G], atom_type [G] -> atom_force [G]
  fluid_force_next_2	atom_position2 [G], atom_position2_z [G], atom_force [G] -> local fluid_force_2 [0..G+S]
  fluid_force_exchange_start	Send local fluid_force_2 exterior [0..G+S]
  fluid_force_exchange_finish	Recieve remote fluid_force_2 interior [-G-S..0]
  	Local fluid_force_2 [0], remote fluid_force_2 [0] -> fluid_force_2 [0]
  fluid_correctu_next_2	fluid_velocity_2 [0..G+S], fluid_density_2 [0..G+S], fluid_force_2 [0..G+S] -> fluid_velocity_2 [0..G+S]
  fluid_dist_eq_next_0+1	fluid_velocity_2 [0], fluid_density_2 [0..D], fluid_force_2 [0] -> ~fluiddisteq3new~+ [0] (for next step)
  fluid_dist_eq_exchange_start+1	Send ~fluiddisteq3new~+ interior [-G-S-F..-1] (for next step)
  atom_sort_index/atom_unsort_position2_z	Sort by index atom_force [G], atom_index [G]
  atom_read	atom_force [G] -> hydroF [G]

• Reorganization

  		Pre-fix_lb_rigid_pc_sphere_gpu	Post-fix_lb_rigid_pc_sphere_gpu	Post-fix_lb_rigid_pc_sphere_gpu	New
  Step	Lammps		(with fix_lb_viscous_gpu)	(without fix_lb_viscous_gpu)
  initial_integrate	coordinates 1	fluid_dist_exchange_finish-1	fluid_dist_exchange_finish-1	fluid_dist_exchange_finish-1	fluid_dist_exchange_finish-1
  	velocities 1/2	fluid_dist_eq_exchange_finish	fluid_dist_eq_exchange_finish	fluid_dist_eq_exchange_finish	fluid_dist_eq_exchange_finish
  		fluid_dist_eqn_next_3	fluid_dist_eqn_next_3	fluid_dist_eqn_next_3	fluid_dist_eqn_next_3
  		fluid_dist_next_2	fluid_dist_next_2	fluid_dist_next_2	fluid_dist_next_2
  		restartWrite	restartWrite	restartWrite	restartWrite
  		fluid_dist_exchange_start	fluid_dist_exchange_start	fluid_dist_exchange_start	fluid_dist_exchange_start
  		fluid_param_next_2_start	fluid_param_next_2_start	fluid_param_next_2_start	fluid_param_next_2_start
  post_integrate
  pre_exchange
  pre_neighbour
  post_neighbour
  pre_force		fluid_param_next_2_finish	fluid_param_next_2_finish	fluid_param_next_2_finish	fluid_param_next_2_finish
  		atom_write	atom_write	atom_write	atom_write
  		atom_position2_z_index	atom_position2_z_index	atom_position2_z_index	atom_position2_z_index
  		atom_sort_position2_z	atom_sort_position2_z	atom_sort_position2_z	atom_sort_position2_z
  		atom_force	atom_force	atom_force
  		fluid_force_next_2	fluid_force_next_2
  		fluid_force_exchange_start	fluid_force_exchange_start
  		fluid_force_exchange_finish	fluid_force_exchange_finish
  			fluid_correctu_next_2
  		fluid_dist_eq_next_0+1	fluid_dist_eq_next_0+1
  		fluid_dist_eq_exchange_start+1	fluid_dist_eq_exchange_start+1
  pre_reverse
  post_force	additional forces	atom_sort_index	atom_sort_index	atom_sort_index	atom_force
  		atom_read	atom_read	atom_read	fluid_force_next_2
  					fluid_force_exchange_start
  					fluid_force_exchange_finish
  					atom_unsort_position2_z
  					atom_read
  final_integrate	velocity 1				fluid_correctu_next_2
  					fluid_dist_eq_next_0+1
  					fluid_dist_eq_exchange_start+1
  end_of_step				atom_write
  				atom_position2_z_index
  				atom_sort_position2_z
  				atom_force
  				fluid_force_next_2
  				fluid_force_exchange_start
  				fluid_force_exchange_finish
  				fluid_correctu_next_2
  				fluid_dist_eq_next_0+1
  				fluid_dist_eq_exchange_start+1

• Defective riser card on GPU asserting power brake and cutting clocks to 1/3rd

Remove blocking OpenCL calls (all explicit dependencies)

• fluid_correctu_next_0 (now fluid_param_u_correct_0) adjusts fluid_velocity_2 [0]
  • doesn't adjust any of fluid_dist*_3 [0] (this is correct)
• fluid_momentum_remove (now fluid_vmc_remove_0) adjusts fluid_dist_3 [0] and fluid_dist_eq_3 [0]
  • should use corrected fluid_velocity_2 [0] instead of computing uncorrected version from fluid_dist*_3 [0]
  • need to also correct fluid_dist_eqn_3 [0..G+S+F] for fluid_dist_next_2 if using exponential integrator
  • can't compute border points [...G+S+F] (only have fluid_velocity_2 [0]) so need to trasfer these
  • need to delay fluid_dist_exchange_3_start until after this
  • need to send borders on other outputs too if using exponential integrator
• fluid_dist_eq_next_0 computes fluid_dist_eq_3 [0] (next)
  • from fluid_velocity_2 [0] (affected by fluid_correctu_next_0 but not affected by fluid_momentum_remove)
• fluid_dist_eqn_next_3 computes fluid_dist_eqn_3 [0..G+S+F] (next)
  • from fluid_dist_eq_3 [0] (affected by fluid_correctu_next_0 but not affected by fluid_momentum_remove)
  • could run earlier if fluid_vmc_remove_0 isn't being run this timestep
• fluid_dist_next_2 computes fluid_dist_3 [0..G+S] (next)
  • from fluid_dist_3 [0..G+S] (not affected by fluid_correctu_next_0 and not affected by fluid_momentum_remove)
  • from fluid_dist_{eq,eqn}_3 [0..G+S+F] (not affected by fluid_correctu_next_0 but affected by fluid_momentum_remove)
  • from fluid_dist_{eq,eqn}_3 [0..G+S+F] (next) (affected by fluid_correctu_next_0 but not affected by fluid_momentum_remove)

• have to change pointers to values

• As we are manually calculating from a 1D index, it is technically aligned now
• [X] cl_mem fluid_gridgeo_3_mem
• [X] cl_mem fluid_dist_3_mem
• [X] cl_mem fluid_dist_eq_3_mem
• [X] cl_mem fluid_dist_eqn_3_mem
• [X] cl_mem fluid_density_2_mem
• [X] cl_mem fluid_velocity_2_mem
• [X] cl_mem fluid_force_2_mem

• [X] print_fluid
• [X] print_internal
• [X] buffer_read_rectangle
• [X] buffer_write_rectangle
• [X] fluid_force_accumulate_rectangle
• [X] calc_mass_momentum
• [X] calc_MPT

• Map memory, directly initialize with strides, and drop sublattice

• [X] Test MPI boundary exchange
• [X] Test GPU boundary exchange

• Copied from sublattice via buffer_create with CL_MEM_COPY_HOST_PTR
• Wrong type in sizeof for memory map

• Hadn't updated even/odd offset to now be on k instead of i (largest step)

• Are old restart files still valid: no, now in Fortran order

• Wrong structure element count in MPI file view type declaration

• Reference code recomputes forces after full step when not used with fix_lbviscousgpu
  • this was introduced with fix_{lbrigidpcsphere}
  • recomputed fluid force used to recompute equilibrium distribution too
• Leaky borders
  • force stencils can extend into walls
  • pits derivative isn't one-sided along walls (is in CPU code, but doesn't matter as kappa_lb=0)
• pressurebcx treats duplicate end point differently causing ghost point divergence

• Use the same point on both sides with adjustwent on both left and right on both
• Update to the newer density adjustment method

• Need to save velocity as well as updated by fluid_param_u_correct_0
• Need to update distribution velocities to give final step restart (see cpu code)

• GPU bbf only bounces back diagonals on edges if both components are wall normal

  • orientation 7 isn't bouncing back directions: 9-13 (likely reversed with 6)
  • orientation 20 isn't bouncing back directions: 7-8, 13-14 (likely reversed with 19)

  ORI	Wall Normal Constructive Solid Geometry	Bounce Back Components	Original (if different)
  0	-	-
  1	(+1, 0, 0)	1, 7,10,11,14
  2	( 0,+1, 0)	2, 7, 8,11,12
  3	( 0, 0,+1)	5, 7, 8, 9,10
  4	(+1, 0, 0) and ( 0, 0,+1)	7,10	1, 5, 7,10
  5	(+1, 0, 0) or ( 0, 0,+1)	1, 5, 7, 8, 9,10,11,14
  6	( 0,+1, 0) and ( 0, 0,+1)	7, 8	2, 5, 7, 8, 9,10,11,12
  7	( 0,+1, 0) or ( 0, 0,+1)	2, 5, 7, 8, 9,10,11,12	2, 5, 7, 8
  8	(-1, 0, 0) or ( 0,-1, 0)	3, 4, 8, 9,10,12,13,14
  9	(-1, 0, 0) or ( 0,+1, 0)	2, 3, 7, 8, 9,11,12,13
  10	(+1, 0, 0) and ( 0, 0,+1) or ( 0,+1, 0) and ( 0, 0,+1)	7, 8,10
  11	(+1, 0, 0) and ( 0, 0,+1) or ( 0,-1, 0) and ( 0, 0,+1)	7, 9,10
  12	(-1, 0, 0) or ( 0,+1, 0) or ( 0, 0, +1)	2, 3, 5, 7, 8, 9,10,11,12,13
  13	(-1, 0, 0) or ( 0,-1, 0) or ( 0, 0, +1)	3, 4, 5, 7, 8, 9,10,12,13,14
  14	(-1, 0, 0)	3, 8, 9,12,13
  15	( 0,-1, 0)	4, 9,10,13,14
  16	( 0, 0,-1)	6,11,12,13,14
  17	(-1, 0, 0) and ( 0, 0, 1)	8, 9	3, 5, 8, 9
  18	(-1, 0, 0) or ( 0, 0, 1)	3, 5, 7, 8, 9,10,12,13
  19	( 0,-1, 0) and ( 0, 0, 1)	9,10	4, 5, 7, 8, 9,10,13,14
  20	( 0,-1, 0) or ( 0, 0, 1)	4, 5, 7, 8, 9,10,13,14	4, 5, 9,10
  21	( 1, 0, 0) or ( 0,-1, 0)	1, 4, 7, 9,10,11,13,14
  22	( 1, 0, 0) or ( 0, 1, 0)	1, 2, 7, 8,10,11,12,14
  23	(-1, 0, 0) and ( 0, 0, 1) or ( 0, 1, 0) and ( 0, 0, 1)	7, 8, 9
  24	(-1, 0, 0) and ( 0, 0, 1) or ( 0,-1, 0) and ( 0, 0, 1)	8, 9,10
  25	( 1, 0, 0) or ( 0, 1, 0) or ( 0, 0, 1)	1, 2, 5, 7, 8, 9,10,11,12,14
  26	( 1, 0, 0) or ( 0,-1, 0) or ( 0, 0, 1)	1, 4, 5, 7, 8, 9,10,11,13,14
  27	( 0, 1, 0) or ( 0, 0,-1)	2, 6, 7, 8,11,12,13,14
  28	( 0,-1, 0) or ( 0, 0,-1)	4, 6, 9,10,11,12,13,14
  29	( 1, 0, 0) and ( 0, 0, 1) or ( 0, 1, 0)	2, 7, 8,10,11,12
  30	( 1, 0, 0) and ( 0, 0, 1) or ( 0,-1, 0)	4, 7, 9,10,13,14
  31	(-1, 0, 0) and ( 0, 0, 1) or ( 0, 1, 0)	2, 7, 8, 9,11,12
  32	(-1, 0, 0) and ( 0, 0, 1) or ( 0,-1, 0)	4, 8, 9,10,13,14

• See CPU compute_vector and documented at end of initial comments
• [X] Scalar is temperature, isn't working 100% can skip
  • [X] Add atom_kinetic (temperature) mirroring atom_spread (allocation, etc.)
  • [X] Need to unsort these (don't actually as just need to sum)
  • [X] Compute degrees of freedom for atom_spread
• [X] Vector (length 4) is total mass and total momentum

• need to build the include file for the kernel code (could include it)

• Default selection
• Hostname based overrides

• Can do the weight interpolation in inital_integrate if we ensure nve goes first (need to warn user if not the case)
• Or just put it in the post_integrate (put it in pre_force for now as haven't hooked post_integrate)

Get forces onto GPU for spreading it between fluid and particles to tie velocities together

• Force on particle k = - hydroforce + m_particle/(m_particle + m_stencilfluid) F_{particle-particlek}
• Force on fluid from particle k = hydroforce + m_stencilfluid/(m_particle + m_stencilfluid) F_{particle-particlek}
• Can just store the first as the difference works out to F_{particle-particlek}
• stencil_density*area * dm_lb is m_stencil_fluid (at end of compute gamma interaction factor comment)
• The lbviscous routine then would just overwrite the force

https://www.sciencedirect.com/science/article/pii/S0010465522000364?via%3Dihub

• Need interpolated fluid mass for CPU calculation (both for force calculation and for temperature)
• Need to send LAMMPS forces to GPU for force calculation

• [X] Add IBM3
• [X] Replace Peskin with Keys

• [X] Only waits on OpenCL events (needs to correctly handle MPI)
• [X] fluid_momentum_remove should be using fluid_velocity_2 and fluid_density_2 so it sees the results of fluid_correctu_next_0
• [X] Fix should just set flag to enable an not call directly (to get in correct spot)
• [X] Verify step passed in is correct (should it be step or step+1 – former is correct)
• [X] Kernel defines dist_3_new = ... (step & 0x01) ... where new should actually be be step+1 & 0x01
• [X] Test periodic boundary conditions
  • call fix momentum, should leave it zero
  • add small body force, remove momentum every 10 steps, should see grow and then drop

• likely issue with `setup` `omega` calculation vs `initial/final<sub>integrate</sub>` one
• replaced by the standard rigid fix (gone in next iteration)

• verified with Colin and should only be calculating on interior fluid_correctu_next_0

• this would be applied in the first 1/2 step
• for consistency with other other lammps integrators, should be zero for this
• first calculation should occur in post force routine

• Working correctly, was limited by the 64 work groups per compute unit requirement

• Are actually as it is loading a single component from variable locations

• Need to compile with the -rdynamic option to not strip out static function names
• [X] Bug due to backtrace allocating one chunk for strings and pointers
• [X] Exception safety via C++11 unique_ptr rewrite
• [X] Figure out why function names are included

• Dump code (ntimesteps based) and fluid code (step based) are off by one in their
  • update->ntimestep is 0 in setup and then 1 for the first step
  • step is -1 in setup and then 0 for first step
• Dump code dumps in both setup and regular steps while fluid code only does regular steps

Make code cleaner by using post_run (can get rid of everything but send bit in setup)

• not doing as isn't cleaner due to setup (or init) mismatch with post_run
  • setup is always run the first time and then only if the run pre flag is set
  • post_run is always run
• how to handle calculation of forces in setup (1/2 step velocity is no longer available)
  • setup compute the forces
    • no need for momentum remaval (user will have disabled if they have introduce net momentum)
  • setup minimal, we are okay, no recomputations required

• Was renamed to fluid_vcm_remove_0

• Made in final version of action code

• Prefix with effect (wall, pressure) and
• Switch to global location condition inside of GPU

• [X] Switch from total runtimes to individual time points
• [X] Output profile information

• should we stick all the event handlers in a queue
• [X]

  Try on atom_sort_position2_z_* routines

  operation	wait between	wait at end	speedup
  sort_..._bb	7.57	5.98	21.0%
  sort_..._bm	6.82	5.32	22.0%
  combined wall	7.71	5.29	31.4%

• tracked down to broken riser card on gra986 asserting hardware power brake on GPU card