TODO list for hyrax:ngap on el9

[ndp-opendap]

TODO list for hyrax:ngap on el9

Overview

After a successful SIT deployment I started this issue by collecting the TODO remarks from our hyrax-docker/el9-builds directory. After discussing them with @jgallagher59701 I decided to reorganize this ticket into three categories:

• Staying Current (Making sure we are always using the latest dependencies)
• Docker Image Size Reduction
• MISC - File Hack for NGAP Deployment (That one stray issue)

I think the job here is to:

• Review/discuss/triage this collection of issues as identified by section
• Create a ticket for each of the sections that are deemed worthy of treatment, and link that ticket to this one.
• Work the resulting tickets.

I have labeled things that I think would shake out as tickets.

---

Staying Current: Installing the latest dependencies during the build

In our el9 ngap build there are two dependency libraries that we install from our own caches:

• Apache APR (from RPM)
• Redisson Session Manager libraries and related dependencies.

These are, in effect, frozen on the version bundle that we cached, and there is not currently an automated way to track and update them. For our Tomcat installation we determine the latest release of the specified major version and we download and install that.

I think we should be doing the same for these dependencies.

Apache APR RPMs (Ticket 165)

This package is a dependency for the Tomcat Native APR, and is currently installed from RPM files we have cached in our S3 bucket. I assert that we need to add code (probably to our hyrax-docker repo) so that each build acquires the "latest" version from Apache and then build and/or install it afresh every time we build a docker container.

Related TODOs from el9-builds/ngap/Dockerfile

• 127 # @todo Do we need this Apache APR RPM?
• 128 # @todo Is this duplicating the compile Tomcat Native APR??

Related TODOs from el9-builds/ngap/build-el9

W.r.t. installing the Apache APR RPMs

• 693 # @todo Why are we getting the Apache APR (Apache Portable Runtime) from the OPeNDAP S3 bucket? Shouldn't we be getting "latest" from Apache? Fix.

Related TODOs from el9-builds/ngap/build-hyrax-ngap.sh

• 55 # @todo Why are we getting Apache RPMs from OPeNDAP's build bucket? Why not lastest APR from Apache??

---

Redisson Session Manager Java Libraries (Ticket 166)

These are currently being retrieved by the hyrax-docker build from the OLFS GitHub repository by cloning the OLFS repo and copying the jar files from their location in that repository.
This package is pivotal to our session management going forward. Again, I assert that we should modify the hyrax-docker build code so that we get the "latest" version from Redis (or wherever it is we get this package) to be installed in our hyrax:ngap docker build.

Current files:

• redisson-all-3.50.0.jar
• redisson-tomcat-11-3.50.0.jar

Discussion

These Redisson libraries are not used by the OLFS directly and are only installed into the Tomcat deployment at the time we build the docker container. Because they are not used directly by the OLFS we might consider:

• Removing these libraries from the OLFS project repository
• Writing a wee build.gradle file that we could run from our .travis.yml job to find and pull the latest Redisson libraries and place them locally so the ngap Dockerfile build can just copy them into the correct place.

Related TODOs from el9-builds/ngap/Dockerfile

• 426 # @todo Figure out a better (less static) way to get these Redisson and related libraries.
• 427 # @todo Like maybe prefetch the "latest" Redisson stuff from wherever these were acquired and copy them in.

---

Managing/minimizing the size of the resulting Docker image.

Our current NGAP build is quite large:
opendap/hyrax:ngap-1.17.1-1211-el9: 3.69 GB (disk usage) (1.29 GB compressed on docker hub)

How does image size affect our situation? (Ticket)

• What are the size constraints on the opendap/besd used in the Cumulus task?
• Does the Cumulus deployment of the opendap/besd run in a lambda function? If so what are the size constraints of that deployment?
• How does the size of the NGAP hyrax image affect the memory resources available to our docker deployments in NGAP?

Installing the Tomcat Native APR (Ticket)

The Tomcat Native APR is needed for SSL and is claimed to improve Tomcat performance.

Currently, in the Dockerfile build we:

• Aquire the "latest" Tomcat-11 distribution.
• Unpack Tomcat-11
• Locate the Tomcat Native APR library bundle within said distribution.
• Use installed developer tools to compile and install the software.

This requires the docker image to have a developers tool suite for compiling/linking C/C++ source code. If we could do this outside of the docker container and bring the results in using COPY we would not have to have all the developer tools in the image.

Related TODOs from el9-builds/ngap/Dockerfile

• 48 # @todo Build the Tomcat Native APR outside of this Dockerfile and bring in the compiled result.
• 209 # @todo Remove this install stage when we stop compiling on in this build. (Installing Developer tools to enable Tomcat Native APR build.)
• 334 # @todo Building the Tomcat Native-APR library should be done elsewhere and copied in, if possible

How much cURL do we need (test this) (Ticket)

In the past we have always needed a full cURL suite because we were compiling code that needed those things. If curl-minimal is adequate as a runtime resource for use we should it (so. much. smaller.)

Related TODOs from el9-builds/ngap/Dockerfile

• 166 # @todo Can we live with curl-minimal? If so, drop this install stage.

---

MISC - File Hack for NGAP Deployment (Ticket)

This final TODO might be worth a look and a fix to harmonize the file names between these disparate parts of the build/deployment.

Related TODOs from el9-builds/ngap/entrypoint.sh

• 290 # @todo THis seems like a crappy hack, we should just change the source file in BitBucket to be correct

The Hack:

  echo "$BES_SITE_CONF" | sed -e "s+BES.LogName=stdout+BES.LogName=$BES_LOG_FILE+g" > "$BES_SITE_CONF_FILE"

[hannahilea]

drive-by github/markdown pro-tip: if you use

- [ ] thingy
- [x] thingy-completed

in place of of a normal bullet

- thingamabob

it will show up as a check box in the github UI:

• thingy
• thingy-completed
• thingamabob

...and then you can check off the boxes as items are addressed!

[ndp-opendap]

@hannahilea - What is the socially acceptable way to add people to this?
My brain says to add you, and James, and dan, and myself as Assignees. But I have this feeling that's a faux pas.

Thoughts?

[hannahilea]

If you tag us in a comment (like you did for me!) we'll get notifications of any conversation around it. I think that should be sufficient!

[ndp-opendap]

@jgallagher59701 & @dh-opendap

We should plan to discuss this, soon!

[dh-opendap]

It's possible we don't need Tomcat APR for the Redisson Session-Manager. This needs verification.

[jgallagher59701]

Overall, this plan seems like a good Idea. To say anything much about the details, I need to get my head around these pieces, so I have a few questions:

Tomcat Native APR

APR == Apache Portable Runtime?

Currently, in the Dockerfile build we:

• Aquire the "latest" Tomcat-11 distribution.
• Unpack Tomcat-11
• Locate the Tomcat Native APR library bundle within said distribution.
• Use installed developer tools to compile and install the software.

This requires the docker image to have a developers tool suite for compiling/linking C/C++ source code.

So The "latest" Tomcat-11 distribution holds the compiled Java code for Tomcat and some C++ source for the interface to Redis? I thought that Redisson was written in Java.

Apache APR RPMs

At this point I am not sure what needs this package, but I think it's dependency for the Tomcat Native APR.

This is a RedHat RPM of Tomcat? Minus the Redis interface code?

[ndp-opendap]

APR == Apache Portable Runtime?

Yes

So The "latest" Tomcat-11 distribution holds the compiled Java code for Tomcat and some C++ source for the interface to Redis? I thought that Redisson was written in Java.

This is not a Redisson thing, although it may be used by Redisson. The Tomcat-11 distribution contains c/c++ code that we have to build and install. I think it's an SSL thing which may (not?) require the Apache APR RPMs.

From the Gemini LLM:
Tomcat Native APR (Apache Portable Runtime) is an optional library that allows Apache Tomcat to use native C-based libraries for I/O operations and SSL/TLS, rather than relying solely on Java's networking stack. It significantly improves Tomcat's performance, scalability, and integration with native server technologies, particularly for handling high-concurrency web traffic and serving static content.

This is a RedHat RPM of Tomcat? Minus the Redis interface code?

No definitely not. If we still need this (see Dan's comment) it's because it's needed/expected by the Tomcat Native APR code.

From Gemini LLM:
Does the Tomcat Native APR depend on the Apache APR? Yes, absolutely. The Tomcat Native Library (libtcnative) is a JNI (Java Native Interface) wrapper around the Apache Portable Runtime library.

[ndp-opendap]

@jgallagher59701 - Based on our EOD conversation yesterday I have revised/rewritten this ticket's description. Please re-read.

[ndp-opendap]

• Writing a wee build.gradle file that we could run from our .travis.yml job to find and pull the latest Redisson libraries and place them locally so the ngap Dockerfile build can just copy them into the correct place.

I tried this on a branch. Here's the Ticket and Here's the PR. Pretty slick actually

[jgallagher59701]

We might be able to install tomcat, APR and openssl using dnf.

I looked at a few places and found packages for all of these. Before we do lots of work to build this from source, lets try that. For example: sudo dnf install apr apr-util tomcat-native openssl Maybe it will be that easy. I looked at RPMFind and RPM pbone (e.g., https://rpm.pbone.net/index.php3). There are many packages for this out there - I only spent about 5 minutes on this, so maybe we can't use them, but it would save so much work if we can.

(for the hyrax-deps, the next move will be to see how much of that stuff can be turned into package loads. Now that we no longer are building a shipping those nasty static libs, we just load the needed packages on the docker images and drop the huge build)

[jgallagher59701]

But, the package angle notwithstanding, this looks good and we should push onward with tickets and work, IMHO. @hannahilea do you agree?

[ndp-opendap]

We might be able to install tomcat, APR and openssl using dnf.

• We already install openssl from with dnf.
• We might be able to locate the Apache APR in the dnf catalog - I was not able to when we set this up. However, I asked the Goole LLM and it said that it is. We should definitely try it: sudo dnf install -y apr apr-devel apr-util apr-util-devel
• The Tomcat Native APR - I doubt it. (And the Google LLM says no, but EPEL might, depending on which Tomcat)

Tomcat from dnf/yum

We used to use Tomcat from dnf/yum but we stopped because of problems.

• The installation is an absolute mess. Nominal Tomcat distributions have an easy to understand structure, are located in the place we choose, and can be easily modified for our purposes. Tomcat installations from dnf/yum are not these things, They are decomposed and scattered throughout the system in the "linux" way and it is terrible. To go back to this will require significant work in our deployment to remove the parts that we do not want (and I think there are a few) and to find the places where we need to install the things we do want. Unless major changes have been made to the dnf/yum Tomcat installation I think that this will be a mess.

• Updates - Tomcat from dnf has historically been slow to receive updates. Right now we are very close to the Tomcat release process so we are able to very quickly get security updates onto our images.

• "Silent" patching - We also experienced numerous cases in which a CVE is announced on the Tomcat version we are have installed and then the maintainers of the Tomcat in dnf/yum "patch" the issue by making changes to the distribution files. But, they do not change the version number(s) of the Tomcat. This led to an endless series of conversations with various NASA security teams where we tried to explain why the version we had was good because it was "patched". This included spending our time crawling around in the various commit logs on the linux distort to find the exact places where they mentioned the various patched issues and then forwarding these to the security teams. We haven't had to do this kind of "No really it's fine" justification since we moved to our current Tomcat deployment model. Let's not put ourselves in that position again - it's stressful and a major waste of our time.

Since we made the change to using Tomcat distributions directly from Tomcat these issues have been resolved.
Let's not go backwards on this road.
I feel strongly that what we are doing now with Tomcat is a best practice.