i have a 12-process FCGI pool w/ supervisord, servicing loads of WMS requests from large COGs (eg ~150 files, most around 40/70Go, some around 250Go). that's running mapserver 8.6 on debian 13, and is used as a backend for mapproxy which serves requests to end users.
i always had a large memory consumption for the mapserver pools, eg i have a supervisor plugin (superlance/memmon) that kills a worker when it reaches 8Gb, and i have workers getting killed every hour, where i have ~50k hits a day.
so far i attributed this large memory consumption to memleaks down the line in the libraries, but found out that setting PROCESSING "CLOSE_CONNECTION=NORMAL" in each layer config stopped the large memory consumption. Measuring the memory consumption of replaying 5000 requests from a production accesslog on the 12 workers, i noted a 80Gb growth with DEFER, and negligible with NORMAL
analysing closer with lsof, i noticed that each worker accumulated open handles (because CLOSE_CONNECTION defaults to DEFER in https://github.com/MapServer/MapServer/blob/main/src/mapraster.c#L697 ?) on all the COGs in my datadirs, not giving back memory to the system (eg not calling GDALClose() ?).
so my question is, would it make sense to default CLOSE_CONNECTION to NORMAL for long-running FCGI processes, when using raster data or vector files (i guess the same issue can happen with a bunch of large shps/gpkgs..) ? for postgis layers it makes sense to default to DEFER so that the connections are kept in the pool, but for files... is there a way in the code to know at runtime that we're running as FCGI and not old-school/traditional CGI ?
i have a 12-process FCGI pool w/ supervisord, servicing loads of WMS requests from large COGs (eg ~150 files, most around 40/70Go, some around 250Go). that's running mapserver 8.6 on debian 13, and is used as a backend for mapproxy which serves requests to end users.
i always had a large memory consumption for the mapserver pools, eg i have a supervisor plugin (superlance/memmon) that kills a worker when it reaches 8Gb, and i have workers getting killed every hour, where i have ~50k hits a day.
so far i attributed this large memory consumption to memleaks down the line in the libraries, but found out that setting
PROCESSING "CLOSE_CONNECTION=NORMAL"in each layer config stopped the large memory consumption. Measuring the memory consumption of replaying 5000 requests from a production accesslog on the 12 workers, i noted a 80Gb growth withDEFER, and negligible withNORMALanalysing closer with
lsof, i noticed that each worker accumulated open handles (becauseCLOSE_CONNECTIONdefaults toDEFERin https://github.com/MapServer/MapServer/blob/main/src/mapraster.c#L697 ?) on all the COGs in my datadirs, not giving back memory to the system (eg not callingGDALClose()?).so my question is, would it make sense to default
CLOSE_CONNECTIONtoNORMALfor long-running FCGI processes, when using raster data or vector files (i guess the same issue can happen with a bunch of large shps/gpkgs..) ? for postgis layers it makes sense to default toDEFERso that the connections are kept in the pool, but for files... is there a way in the code to know at runtime that we're running as FCGI and not old-school/traditional CGI ?