When you see this kind of issue in SAP world it means the users work process has used up all allocated memory and entered into HEAP Area which usually should not by design. If you really want to resolve this issue with high confidence you should start understanding how SAP memory is allocated and managed. Here are simple approaches at a higher level.
Approach 1: Kill such a process in PRIV status
Get user confirmation before doing this step
Approach 2: Request user to self logout of the system and log back in
This will free-up any held memory
Approach 3: Reduce the limit on the extended memory (defined in ztta/roll_extension), or increase the extended memory (see em/initial_size_MB).
Also tune abap/heap_area_total params too if needed.
When does work process switch to PRIV mode?
Tags: sm50 priv mode, sap heap private mode, memlimits, malloc, rz10, rz11,