Thanks for pointing me in the right direction, was hoping that something like that existed.
Also for that solution to work I had to modify AuthConfigurer.cs
so that the token is actually checked in url:
var path = context.HttpContext.Request.Path.Value;
if (path.Contains("/Chat/GetUploadedObject") || path.Contains("/Attachment/Download"))
{
return SetToken(context, false);
}