开发者

Why do I have memory leakege using threads on Heroku?

I have the following setup on my flask heroku app to avoid timeout exception. Everything is working fine and I can pass the timeout exception.

For some reason after running the Threads multiple times I receive an error from Heroku logs

Process running mem=628M(122.6%)
heroku[web.1]: Error R14 (Memory quota exceeded)

I initially didn't have a join() statment, but this issue appearing even after adding the join() to the threads.

This is my /build route

   @app.route("/build",methods=["GET", "POST"])
    @login_required
    def build():
        if request.method == "POST":
            global global_dict
            @copy_current_request_context
            def operation(global_dict, session):
                ......
            global t1
            t1 = Thread(target=operation, args=[global_dict, session], name = str(session['user_id'])+'_operation_thread')
            t1.start()

            @copy_current_request_context
            def enter_sql_data(nasdaq_exchange_info, tickers):
               ....
            global t2
            t2 = Thread(target=enter_sql_data, args=[nasdaq_exchange_info, tickers], 
            name=str(session['user_id'])+'_sql_thread')
            t2.start()
       return render_template("loading.html")

This is where the global flag is being read

@app.route('/status')
def thread_status():
    userId = session['user_id']
    return jsonify(dict(status=('finished' if (global_dict[int(userId)]['finished'] == 'True') else 'running')))

This is the JS on the HTML page where the flag is checked every 1.5 seconds:

<script>
  $(document).ready(function() {
    var refresh_id = setInterval(function() {
        $.get(
          "{{ url_for('thread_status') }}",
          function(data) {
          开发者_开发问答  console.log(data);
            if (data.status == 'finished') {
              window.location.replace("{{ url_for('result') }}");
              clearInterval(refresh_id);
            }
          }
        )}
      , 1500);
  });
</script>

This is the result page I am showing at the end of the process

@app.route('/result')
def result():
    try:
        t1.join()
        t2.join()
        userId=session['user_id']
        return render_template("built.html", num_small=global_dict[int(userId)]['num_small'], plot_json_weights_min_vol_long=global_dict[int(userId)]['plot_json_weights_min_vol_long'], av_min_vol_long=global_dict[int(userId)]['av_min_vol_long'], leftover_min_vol_long=global_dict[int(userId)]['leftover_min_vol_long'], alloc_min_vol_long = global_dict[int(userId)]['alloc_min_vol_long'], plot_json_dist_min_vol_long=global_dict[int(userId)]['plot_json_dist_min_vol_long'], av = global_dict[int(userId)]['av'], leftover_min_vol_long_short=global_dict[int(userId)]['leftover_min_vol_long_short'], alloc_min_vol_long_short=global_dict[int(userId)]['alloc_min_vol_long_short'], ret=global_dict[int(userId)]['ret'],gamma=global_dict[int(userId)]['gamma'],volatility=global_dict[int(userId)]['volatility'], perf_L2=global_dict[int(userId)]['perf_L2'], perf_semi_v=global_dict[int(userId)]['perf_semi_v'], alloc_L2=global_dict[int(userId)]['alloc_L2'], alloc_semi_v=global_dict[int(userId)]['alloc_semi_v'], plot_json_graph=global_dict[int(userId)]['plot_json_graph'], plot_json_exp_cov=global_dict[int(userId)]['plot_json_exp_cov'], plot_json_Ledoit_Wolf=global_dict[int(userId)]['plot_json_Ledoit_Wolf'], plot_json_weight_min_vol_long_short=global_dict[int(userId)]['plot_json_weight_min_vol_long_short'], plot_json_L2_weights=global_dict[int(userId)]['plot_json_L2_weights'], plot_json_L2_port = global_dict[int(userId)]['plot_json_L2_port'], plot_json_semi_v = global_dict[int(userId)]['plot_json_semi_v'], leftover_L2=global_dict[int(userId)]['leftover_L2'], leftover_semi_v=global_dict[int(userId)]['leftover_semi_v'],listofna=(', '.join(global_dict[int(userId)]['listofna'])), min_cvar_rtn = global_dict[int(userId)]['target_CVaR_exp_rtn'], min_cvar_risk = global_dict[int(userId)]['target_CVaR_cond_val_risk'], var = global_dict[int(userId)]['var'], cvar = global_dict[int(userId)]['cvar'], plot_json_cvar=global_dict[int(userId)]['plot_json_cvar'], cvar_value=global_dict[int(userId)]['cvar_value'], alloc_cvar = global_dict[int(userId)]['alloc_cvar'], leftover_cvar = global_dict[int(userId)]['leftover_cvar'])
    except:
        try:
            t1.join()
            t2.join()
            return_error = str(global_dict[int(userId)]['error'])
            flash(return_error)
            return redirect("/build")
        except:
            t1.join()
            t2.join() 
            return redirect("/build")

Any idea how to avoid the memory leakage?

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜