How to share dependent classes between a main app and plugins in Java?

Question

Assume, I have a relatively large and complex application (100 MB .war file) with multiple dependencies. Some part of the code is defining an object factory, where similar objects (all inheriting from the same base class) are instantiated.

My goal is to convert these hard coded objects to plugins that can be loaded dynamically at run-time.

So, I wrote a plugin loader:

String localPath = "...";
String pluginName = "...";

File jarFile = new File(localPath);
ClassLoader pluginLoader = URLClassLoader.newInstance(new URL[]{jarFile.toURL()});

pluginLoader.loadClass(pluginName).newInstance();

I then went on to actually write the plugins, which is when I realized that each potential plugin has a large number of dependencies and creating a jar file for each of them would result in plugins of roughly 50MB in size.

I'm worried about the memory requirements this would entail, when loading 100 of these plugins.

Is there a way I can load plugins that depend on classes inside the main app without bundling them into the plugin itself?

Or how would I best share such common resources?

[UPDATE] For now I figured I could expose the already loaded classes to the ClassLoader, which I was told works like this:

ClassLoader mainLoader = PluginManager.class.getClassLoader();`
ClassLoader pluginLoader = URLClassLoader.newInstance(
                                 new URL[]{jarFile.toURL()},
                                 mainLoader);

The downside to this is obviously that any change in the main application can make the plugins fail. This will, however, help with my memory concerns and with proper error handling it will be easy to find no longer working plugins.

Answer 1

First, I should point out that external dependencies (not from source code of your project or it's subprojects, in Gradle terms) are not included into jars, unless you specifically create a "fat" jar (or "uberjar").

---

Probably you're looking for compileOnly dependencies (in Gradle terms) or provided dependencies (in Maven terms). Such dependencies are only used to compile your project, but are not included into resulting distribution.

The idea is that, in a plugin code, you specify your main project as a compileOnly dependency, so that you can implement interfaces from the inside of it.

Compare compileOnly with runtime:

• runtime: "these need to be there when this program executes, so that my code can use instances of them"
• compileOnly: "those are just needed for type-checking to compile this code, but it doesn't matter if they exist on runtime"

Since you only need some API interfaces to implement, the latter is your case.

Answer 2

Delaying of loading of the dependencies will not solve the memory problem; it will just delay it.

100MB war file is not really that big (depending on your server) and not everything is loaded into memory on startup. Classes are loaded into memory only if the instances/statics are created. If there is too much memory required, there are too many instances, not too many classes. I would suggest to look for a pooling solution for those object factories.

Answer 3

1. Identify the internal/external dependencies of the plugin, review them and create a jar out of the dependencies, Hope it gives an opportunity to refactor as well. There has to be something which is cohesive about it. This needs to be managed a separate project with its version. (Needed to track them with separate versions.)
2. Include this a compileOnly in case of plugin and include the jar in the main application.
3. Dynamic loading of the plugin in the main application as done in the application.

Answer 4

What you are describing is already handled by Java and web containers will often add more complex class loading (you are using a war file.)

Let's start with the basic class loading process (oversimplified.) In Java if you try to load a class or use a dependency, it will search the classpath for that dependency. Whatever instance of that dependency is found first, get's pulled in. So if I have two classes in separate jars that both depend on the same Logger class from Log4J, I don't need two copies of Log4J in my classpath. Only one will ever get used. But there's a problem here. What if one of your jars needs version 1 of com.foo.bar.Snafu and another one needs version 2 of com.foo.bar.Snafu? This simple model won't work for you.

Because the idea behind JEE is that you have a lot of different things running all together with an bazillion dependencies, this comes up a lot. So in my experience (which is not recent - more on that later) most containers offer a more complex classloading model. The first step is to search (recursively) through all parent class loaders for a class. If it is not found, it will be loaded from the web apps local class path. So if you want to have one copy sof Log4J for everyone, you put it on the containers main classpath. If everyone can use that version, you are done. Your wars or ears need not provide binaries for that dependency.

However, if you need different versions for different web apps under this model, that means you need to get it out of the main classpath and add the different versions at different levels of the classpath hierarchy. This is complex and fraught so containers offer the ability to flip that model on it's head in your web app. That is, it will look first in it's local classpath before looking in parent classpaths.

So regardless of which way you go, if you want to share these dependencies, just put them on the classpath at the appropriate level. If they are used by multiple apps, consider putting them at the root.

Another thing to consider is the fact that all of this nonsense kind of sucks to work with. I highly encourage anyone doing web development in Java to consider embedded approaches such as embedding Jetty or embedding Tomcat. It will make your life so much easier and it is much more compatible with containerization.

P.S. One of the comments mentions OSGi. This does, in theory, address a lot of these issues. However, I would not recommend this unless you are a masochist. It makes things far more complicated and won't necessarily work for you.