Get 500 response code on model invocation error

Models deployed on Baseten using Truss 0.7.1 or later can now send the 500 response code when there is an error during model invocation. This change only affects newly deployed models.

Any exception raised will result in a 500 response code. For example, this Truss code:

class Model:
    def predict(...):
         raise Exception("hello")

Will yield a response with the following content:

{
    "error": "Internal Server Error"
}

You can see details in the model logs for each exception raised, and errors are aggregated on the model metrics page.

✕

Inference volume showing 200, 400, and 500 response codes