Skip to content

Releases: LDSSA/heroku-model-deploy

Batch3

02 Nov 07:21
@hershaw hershaw
3
0f9acd2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Batch3 Latest
Latest

Tagging this before work starts for batch 4

Assets 2
Loading

Batch2

11 Aug 09:06
@hershaw hershaw
2
5923e12
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Batch2

This is an update to batch 1's heroku deploy BLU. It has the following updates:

-On the readme, it uses pipenv instead of virtualenv (since its the new "official" way of dealing with environments.

  • Fixes an issue with the latest category-encoders package (package version is fixed on requirements.txt
  • Uses joblib instead of pickle to serialize the pipeline
  • Fixes errors using pewee, both when performing a /predict request with an existing observation_id and when submitting an /update request when the id does not exist. Now there are rollbacks to avoid memory leaks with unclosed db connections.
  • Adds an error message in the requests response when applicable

Benchmarks

Regarding benchmarks, i did a couple with Apache Bench, i didnt go far mainly because I am using the heroku free tier app. Here are the results.

echo '{"id": 13, "observation": {"Age": 23.0, "Cabin": null, "Embarked": "S", "Fare": 7.25, "Parch": 0, "Pclass": 3, "Sex": "male", "SibSp": 3}}' > predict_point.json
  • 10 concurrent requests up to 2000 ( Was taking to long, stopped it after 325 requests)
manuel@manuel heroku-model-deploy (batch2) $ ab -p predict_point.json -T application/json -c 10 -n 2000 http://predictive-app.herokuapp.com/predict

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       47   50   2.3     49      66
Processing:  2358 2974 1963.7   2598   15072
Waiting:     2357 2973 1963.7   2597   15072
Total:       2407 3023 1965.4   2647   15138

Percentage of the requests served within a certain time (ms)
  50%   2647
  66%   2739
  75%   2793
  80%   2850
  90%   2944
  95%   3044
  98%  13458
  99%  14514
 100%  15138 (longest request)
  • 2 concurrent requests up to 1000:
manuel@manuel heroku-model-deploy (batch2) $ab -p predict_point.json -T application/json -c 2 -n 1000 http://predictive-app.herokuapp.com/predict
Server Software:        gunicorn/19.9.0
Server Hostname:        predictive-app.herokuapp.com
Server Port:            80

Document Path:          /predict
Document Length:        79 bytes

Concurrency Level:      2
Time taken for tests:   329.706 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      247000 bytes
Total body sent:        296000
HTML transferred:       79000 bytes
Requests per second:    3.03 [#/sec] (mean)
Time per request:       659.411 [ms] (mean)
Time per request:       329.706 [ms] (mean, across all concurrent requests)
Transfer rate:          0.73 [Kbytes/sec] received
                        0.88 kb/s sent
                        1.61 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       51   54   2.5     54     125
Processing:   542  605  58.6    588     985
Waiting:      541  605  58.5    588     985
Total:        595  659  58.6    642    1040

Percentage of the requests served within a certain time (ms)
  50%    642
  66%    663
  75%    682
  80%    694
  90%    738
  95%    779
  98%    823
  99%    877
 100%   1040 (longest request)
  • 1 concurrent request:
manuel@manuel heroku-model-deploy (batch2) $ ab -p predict_point.json -T application/json -c 1 -n 1000 http://predictive-app.herokuapp.com/predict
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking predictive-app.herokuapp.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
^C

Server Software:        gunicorn/19.9.0
Server Hostname:        predictive-app.herokuapp.com
Server Port:            80

Document Path:          /predict
Document Length:        79 bytes

Concurrency Level:      1
Time taken for tests:   500.769 seconds
Complete requests:      796
Failed requests:        0
Total transferred:      196612 bytes
Total body sent:        235912
HTML transferred:       62884 bytes
Requests per second:    1.59 [#/sec] (mean)
Time per request:       629.107 [ms] (mean)
Time per request:       629.107 [ms] (mean, across all concurrent requests)
Transfer rate:          0.38 [Kbytes/sec] received
                        0.46 kb/s sent
                        0.84 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       47   50   4.7     49     126
Processing:   538  579  36.5    566     759
Waiting:      538  579  36.5    565     759
Total:        587  629  37.1    615     808

Percentage of the requests served within a certain time (ms)
  50%    615
  66%    632
  75%    643
  80%    655
  90%    679
  95%    702
  98%    744
  99%    761
 100%    808 (longest request)
Assets 2
Loading