Repository navigation
Dockerhub image from linuxserver works with runc but not cc-runtime #986
Description
When testing a popular docker hub image called linuxserver/radarr (10 million pulls) you can't connect to port 7878 from a browser when using cc-runtime but runc works as expected.
First create a config directory at /config
mkdir /config
Run the container and attempt to go to http://:7878 and it works under runc but won't connect under cc-runtime.
docker run -d --runtime=runc --name=radarr -v /config:/config -p 7878:7878 linuxserver/radarr
This works and you can go to http://localhost:7878
docker run -d --runtime=cc-runtime --name=radarr -v /config:/config -p 7878:7878 linuxserver/radarr
This does not work and http://localhost:7878 times out
Trying the same thing with an nginx container works fine but the nginx container monitors port 80 but passing in 7878 as the host port to use still works.
cc-runtime version 3.0.16
runc version 1.0.0-rc4+dev
docker version 17.09.1
Clear Linux version 20650
Activity
grahamwhaley commentedon Feb 22, 2018
Let's start with an ack - I can re-create the issue here as well.
Now, I wonder where the problem is - is it going to be inside the container/VM, or outside... my gut tells me there will either be maybe:
I'm going to look into how we check out and track the port mappings both in the container (which might mean we have to enable the VM OS debug shell), and on the host side (which might mean digging into docker namespaces).
@sboeuf @amshinde - any ideas from your side around the agent/networking/port mapping side?
sboeuf commentedon Feb 22, 2018
@grahamwhaley no idea on the top of my head, this needs further investigations.
jodh-intel commentedon Feb 23, 2018
Hi @eadamsintel - please can you:
Enable debug
https://github.com/clearcontainers/runtime#debugging
Run the command (as requested in the issue template), and paste the output here:
grahamwhaley commentedon Feb 23, 2018
I'm having a peek at this btw...
grahamwhaley commentedon Feb 23, 2018
OK, some more info.
I noticed inside the container that with
ccwe are cycling through pids forabc 2149 202 0 14:05 ? 00:00:00 mono --debug Radarr.exe -nobrowswhereas we don't with
runc.If you run the docker command with
-tiand drop the-d, then you find that forccwe get a repeatingPress enter to exit...prompt appearing over and over. I suspect therefore that something is upsetting and/or not working for the
monoinvocation, and it is stuck in a retry loop. Hence, the server is not up, so we cannot connect to the 7878 port. afaict, the port looks mapped on the host side btw - I think this is therefore likely not a portmap issue, but amonoexecution issue.grahamwhaley commentedon Feb 23, 2018
Not sure how much this is going to help somebody (I have yet to digest it), but...
/var/run/s6/services/radarrrunfile there to a backup, and then make thatrunbenign with something like atail -f /dev/nullto stop the system trying to restart the broken-nesscd /opt/radarr; mono --debug Radarr.exe --nobrowser -data=/configThen I end up with:
ah, ok, that is a 'database fail' on /config, which smells like 9pfs issues to me... let's try...
to place the db on a tmpfs (ramfs) in the container - and - voila - we don't get the catastrophic failure, and I can browse the container on 7878.
/cc @eadamsintel - I think there is the root of the issue ;-)
sboeuf commentedon Feb 23, 2018
@grahamwhaley oh nice and quick debug !
What's the next step ? Because it's 9p issue, does that mean we cannot expect this to work ?
grahamwhaley commentedon Feb 23, 2018
:-( I'd have to take the next step in debug to be decisive - we'd have to know exactly what failed with the 9pfs mounted files - I suspect it will be one of the 'unlink' related issues. Normally I use strace to find that, but for mono, which is a JIT'd VM, I wonder how well that will work? :-)
Short term, at least we know what the problem is.
Mid term, we could re-visit the 9p patch sets and also look at what runv is carrying and see if we can improve the situation.
Long term, we need a more POSIX compliant fs solution.
sboeuf commentedon Feb 23, 2018
@grahamwhaley using devmapper might solve this issue then (unless the file that needs to be accessed is passed through 9p as an extra mount on top of the rootfs).
grahamwhaley commentedon Feb 23, 2018
yeah, I considered that - it is a
-vvolume mapping, which I think always goes as a 9p mount, doesn't it? (/cc @amshinde ) Which, surprised me a couple of weeks ago, but having seen a recent conversation, I think we don't block mount volumes apart from the (readonly?) rootfs, as then the 'device' would be double mounted - once in host and once on the guest, and there could then be fs write races between the two that [cw]ould then corrupt the FS....sboeuf commentedon Feb 23, 2018
Oh yeah... I haven't realized this was a
-vassignment. In this case, we use 9p because we don't have the ability to package that into a block device that we could hotplug...amshinde commentedon Feb 23, 2018
@grahamwhaley Yes the
-vbindings are always passed using 9pfs. We havent implemented checks for verifying if the volume passed with-vis a mount backed by a block device. We do need to implement that, as we just handle this case with--device.Maybe we can try this out, loopmount an image and pass the loop device as
--device /dev/loop#/configand see if that helps.grahamwhaley commentedon Feb 23, 2018
That's an idea @amshinde - hmm, I wonder if that is viable as an interim 'hack' to mount volumes into the VMs as block devices, by a loopback and device mount. It's worth a try to see if it does work and fixes the issue initially anyhow... I'll add it to my list.
sboeuf commentedon Feb 23, 2018
This should work but don't expect good performances.