{{ message }}
Repository navigation
docs: rewrite README for both curators and maintainers - #86
Merged
Merged
Conversation
The README covered only local aCeDB, Grafana and the SES mailer setup, and said nothing about what the services actually are or how they are deployed. Split it into two parts: - Part 1, for curators: how the pieces fit together, the split between the public forms and the password-protected curator forms (each linked to its live URL, with a pointer to the self-generating site map), what the OA is and the datatypes it is configured for, and where curated data goes and on what schedule. - Part 2, for developers: the Compose service inventory, repository layout, the env-var groups and the Dotenv precedence trap, what needs a rebuild versus what is live through the bind mounts, deployment to the two AWS hosts, and operations (routing, backups, certs, monitoring, logs). The deployment section documents what was previously only institutional knowledge: there is no CI/CD, both hosts select their config through the env_files/.env symlink, and the code that is actually served comes from SRC_DIR_PATH rather than from the checkout Compose is invoked in. The existing AWS SES section is preserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Kimberly points out on #86 that citace is no longer built, since there are no new WormBase releases, so the .ace dump pipeline no longer has a consumer downstream. Remove it as a live destination: from the intro, the data-flow diagram, the "where curated data goes" table, and the note telling curators to wait for a monthly release. The Alliance is now the destination that matters. The citace_upload/ tree stays in the repository listing, marked as legacy, with a rough-edges entry recording the actual state: most per-datatype dumpers are already commented out in curation/crontab, and only the papers wrapper and Cecilia's wrapper still run on a schedule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
Ranjana asked where to find the files kept on the instance, the gene descriptions generated for WormBase over the years in particular. Document /usr/caltech_curation_files: that pub/ and priv/ are browsable through the /files alias, so most things can be fetched without server access; the gene_descriptions layout in detail (WS268-WS298, per-species txt/tsv/json plus the .ace and count files per run); and a table of the other trees with their sizes and contents. Sizes and layout surveyed on the production instance. Also records, under rough edges, that the disk is 86% full, with citace/ (370 GB, now consumerless) and the indefinitely kept monthly dumps (124 GB) the two largest holdings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The README said the hosting has "no CI/CD", which is wrong: the jenkins service in this same Compose stack polls the sibling projects' GitHub repositories, builds their images and replaces their running containers. What is actually true is narrower - no pipeline exists for this repository, so the curation service alone is deployed by hand. Add a Jenkins section under Operations with the jobs configured on production, the repository each tracks and the containers it runs, verified by reading the job configs. Note that Jenkins runs privileged as root with the host Docker socket mounted, which is how a job restarts containers and also means a job has full control of the daemon. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Jenkins section read as though CI/CD were a property of the curation system. It is not: those pipelines build a set of applications developed separately from curation, which happen to run on the same instance. Replace it with "Other services on this instance", listing each application, where it is reached and which containers it runs, and what it shares with the curation system: the reverse proxy, the disk, the curation hostname for anatomy function, and - verified in the job configs - the caltech_curation database itself, which the ACKnowledge and WormiCloud backends connect to. Jenkins is now described there as how those applications deploy, with the curation system explicitly outside it. Also stop calling them "Textpresso services"; the textpressolab.com hostnames stay, since those are the real domains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
acedb.org is a parked domain now: it answers 200 but serves only a redirect stub to a registrar lander, so a status check does not catch it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instance carries a checkout of alliance-genome/agr_genedescriptions that nothing in this repository refers to, so it was invisible in the README while its output - 39 GB of gene description files - was already documented. Record both halves. pipelines/wormbase/ is the copy installed here: it never ran automatically, there is no cron entry or Jenkins job for it, it was launched by hand per release, and it is no longer run at all now that WormBase is frozen, which is why WS298 is the last release on disk. pipelines/alliance/ is where generation happens now, running automatically on every Alliance build, with nothing on this instance involved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
azurebrd
approved these changes
Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README covered only local aCeDB, Grafana and the SES mailer setup. It said nothing about what the services actually are, or how they get deployed. This rewrites it around the two audiences who need it.
Part 1 — for curators. No shell commands. A diagram of how the pieces fit together, the split between the public forms and the password-protected curator forms (every one linked to its live URL, with a pointer to the site map that generates itself from the scripts on disk), what the OA is and the 21 datatypes it is configured for, and a table of where curated data goes — citace, the Alliance, e-mail — and on what schedule.
Part 2 — for developers. The Compose service inventory, repository layout, the env-var groups and the
Dotenv-does-not-overwrite trap, a table of what needs an image rebuild versus what is already live through the bind mounts, deployment to the two AWS hosts, and operations: routing and ports, restic backups, certbot, Prometheus/Grafana, logs.The existing AWS SES section is preserved essentially verbatim.
Review requests
@vanaukenk — Part 1 is the part that matters most, and I wrote it from the code rather than from doing the work. Please check that the descriptions of the forms and the OA datatypes match how you actually use them, and that the export schedule table is right: I read it off
curation/crontab, but whether that matches what curators expect to see in a release is your call, not the crontab's.@azurebrd — Part 2, particularly the deployment section. I worked it out by inspecting the running containers on both servers, so please confirm I have it right, especially: that the code served comes from
SRC_DIR_PATH(/home/azurebrd/git/caltech-curation-services) rather than from whichever checkout Compose is invoked in, and that a CGI or script change therefore needs only agit pullthere and no restart.Unrelated bug found while checking the links
@azurebrd — every URL in Part 1 was verified, and one came back broken:
pub/cgi-bin/forms/reagent_help.cgireturns 500 on production and 200 on dev.Both hosts serve the same bind-mounted checkout, so this looks environmental rather than a problem with the script — a data file missing under
/usr/caltech_curation_fileson prod, or one the Apache user cannot read. The Apache error log on the prod curation container should say which. Nothing in this PR touches it; the README still links the form, since that is its canonical URL.🤖 Generated with Claude Code