I don't believe there's a benefit for returning the CSP for any request other than a document request. From what I can tell, the CSP is returned by all MVC actions and Razor Page handlers. It can be restricted to just document requests using the following middleware code:

bool documentRequest = false;
if (httpContext.Request.Headers.TryGetValue("sec-fetch-dest", out StringValues dest)) documentRequest = dest.Equals("document");
else if (httpContext.Request.Headers.TryGetValue("accept", out StringValues accept)) documentRequest = accept.First().StartsWith("text/html");

The sec-fetch-dest header is implemented in all major browsers except Safari, so that's why there's a fallback to checking the preferred content type requested by the browser. That check isn't as precise because an ajax request for html will match it and not need it.