defaulting CLOSE_CONNECTION=DEFER for rasters leads to large memory consumption with FCGI · Issue #7459 · MapServer/MapServer · GitHub
Skip to content

defaulting CLOSE_CONNECTION=DEFER for rasters leads to large memory consumption with FCGI #7459

Description

@landryb

i have a 12-process FCGI pool w/ supervisord, servicing loads of WMS requests from large COGs (eg ~150 files, most around 40/70Go, some around 250Go). that's running mapserver 8.6 on debian 13, and is used as a backend for mapproxy which serves requests to end users.

i always had a large memory consumption for the mapserver pools, eg i have a supervisor plugin (superlance/memmon) that kills a worker when it reaches 8Gb, and i have workers getting killed every hour, where i have ~50k hits a day.

so far i attributed this large memory consumption to memleaks down the line in the libraries, but found out that setting PROCESSING "CLOSE_CONNECTION=NORMAL" in each layer config stopped the large memory consumption. Measuring the memory consumption of replaying 5000 requests from a production accesslog on the 12 workers, i noted a 80Gb growth with DEFER, and negligible with NORMAL

analysing closer with lsof, i noticed that each worker accumulated open handles (because CLOSE_CONNECTION defaults to DEFER in https://github.com/MapServer/MapServer/blob/main/src/mapraster.c#L697 ?) on all the COGs in my datadirs, not giving back memory to the system (eg not calling GDALClose() ?).

so my question is, would it make sense to default CLOSE_CONNECTION to NORMAL for long-running FCGI processes, when using raster data or vector files (i guess the same issue can happen with a bunch of large shps/gpkgs..) ? for postgis layers it makes sense to default to DEFER so that the connections are kept in the pool, but for files... is there a way in the code to know at runtime that we're running as FCGI and not old-school/traditional CGI ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions