a `e @sddlmZddlZddlZz ddlZWnddlZYn0ddlZddlZddlm Z dddZ dd d Z d d Z dddZ dS))virtual_memoryN)log?cCst|dtj|S)a  :param minimum minimum value to return (return value will never be lower than this) :param use_at_most use at most this fraction of the available memory. 0.5 = use at most 50% of available memory :return percentage value of memory to use (75 = 75%). d)maxrpercentminimumZ use_at_mostr /code/opendm/concurrency.pyget_max_memory sr rcCst|tjdd|S)a :param minimum minimum value to return (return value will never be lower than this) :param use_at_most use at most this fraction of the available memory. 0.5 = use at most 50% of available memory :return value of memory to use in megabytes. i)rr availabler r r r get_max_memory_mbsrcCstjSN)rtotalr r r r get_total_memorysrTc sdafddfdd}|dkrd}tgt|D]"}tj|d}||q>d}|D]}||f|d7}qjfd d }zj d krt d qWn*t yt d |tdYn0|tdur|rtdd}nd}|r|D]} | q dS)z Our own implementation for parallel processing which handles gracefully CTRL+C and reverts to single thread processing in case of errors :param items list of objects :param func function to execute on each object Ncs |dSrr )q)funcr r process_one*sz!parallel_map..process_onec sz\}}|dustdur&qvzBz |Wn(ty\}z|aWYd}~n d}~00Wq0qdSr)geterror task_done Exception)numre)pqrr r worker-s  zparallel_map..workerrF)targetcs2ttD]}dq D] }|q dS)N)N)rangelenputjoin)it)rthreadsr r stop_workersKs z"parallel_map..stop_workersrrzCTRL+C terminating...zCFailed to run process in parallel, retrying with a single thread...T)rqueue PriorityQueuer! threadingThreadstartappendr#unfinished_taskstimesleepKeyboardInterruptprintsysexitr ODM_WARNING) ritems max_workerssingle_thread_fallbackrZuse_single_threadr%r&r(rr )rrrr'r parallel_maps>          r:)rr)rr)rT)vmemrosr4Queuer)r+r0opendmrr rrr:r r r r s